Skip to content

Feature/exponential backoff #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dgomesbr wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Feature/exponential backoff #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e7a4cd7
feat: implement batch processing to stabilize large library scans
dgomesbr Jan 1, 2026
6da8f27
Updating batch info on Readme
dgomesbr Jan 1, 2026
84f7544
Update Changelog
dgomesbr Jan 1, 2026
f82ab8e
Add exponential backoff retry for Radarr/Sonarr API clients
dgomesbr Jan 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 44 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,38 @@
### Added

#### Radarr/Sonarr Integration

- ✅ Created Radarr API client module (`internal/radarr/`) with full API support
- Movie search by title, year, TMDb ID, IMDb ID, and file path
- Automatic TMDb ID extraction from Radarr database
- Connection testing and system status endpoints

- ✅ Created Sonarr API client module (`internal/sonarr/`) with full API support

- TV series search by title, year, TMDb ID, TVDb ID, IMDb ID, and file path
- Episode fetching and file path matching
- Connection testing and system status endpoints

- ✅ Updated configuration system to support Radarr/Sonarr

- Added `USE_RADARR` and `USE_SONARR` environment variables
- Added `RADARR_URL`, `RADARR_API_KEY`, `SONARR_URL`, `SONARR_API_KEY` configuration
- Added validation for Radarr/Sonarr settings when enabled

- ✅ Enhanced TMDb ID extraction to use multiple sources

- Primary: Plex metadata (existing functionality preserved)
- Secondary: Radarr/Sonarr API matching (new)
- Fallback: File path regex matching (existing functionality preserved)
- Added source tracking to show where TMDb ID was found

- ✅ Updated media processor to integrate with Radarr/Sonarr

- Modified `extractMovieTMDbID` to query Radarr when enabled
- Modified `extractTVShowTMDbID` to query Sonarr when enabled
- Added multiple matching strategies: title/year, IMDb ID, TVDb ID, file path

- ✅ Updated main application to initialize Radarr/Sonarr clients

- Added connection testing on startup
- Graceful handling when Radarr/Sonarr are not configured

Expand All @@ -42,6 +47,7 @@
#### Verbose Logging & Debugging

- ✅ Added verbose logging feature

- New `VERBOSE_LOGGING` environment variable (default: false)
- Shows detailed TMDb ID lookup process for each item
- Displays all available Plex GUIDs (IMDb, TMDb, TVDb)
Expand All @@ -51,6 +57,7 @@
- Helps troubleshoot matching issues

- ✅ Added progress tracking for large libraries

- Shows percentage progress for libraries with >100 items
- Displays current processing status
- Shows summary of skipped items in verbose mode
Expand All @@ -61,6 +68,34 @@
- Shows current and new keywords being merged
- Confirms successful application to Plex

#### Batch Processing

- ✅ Added batch processing system to prevent API overwhelming

- New `BATCH_SIZE` environment variable (default: 100 items per batch)
- New `BATCH_DELAY_SECONDS` environment variable (default: 10 seconds between batches)
- **Fully backward compatible** - existing configurations work without changes
- Processes large libraries in manageable chunks instead of continuous loop
- Prevents Radarr/Sonarr SQLite database locks and API flooding
- Reduces memory usage and prevents container crashes
- Configurable batch size and delay for different system capabilities

- ✅ Enhanced processing flow for large libraries

- Sequential batch processing with progress reporting
- Batch-specific progress indicators (e.g., "Processing batch 5/42")
- Configurable delays between batches to allow system recovery
- Maintains existing per-item processing logic within batches
- No delay after final batch for efficiency
- **Smart messaging** - verbose batch logs only for large libraries or custom settings

- ✅ Added configuration validation and guidelines
- Validates positive batch sizes and non-negative delays
- Comprehensive documentation for different library sizes
- Performance tuning recommendations based on library size
- Troubleshooting guide for optimal batch configuration
- **Zero migration required** - defaults provide immediate stability improvements

#### Persistent Storage

- ✅ Added persistent storage for processed items
Expand All @@ -74,6 +109,7 @@
#### Error Handling & Connection Testing

- ✅ Added TMDb API connection testing on startup

- Validates API token before processing begins
- Provides clear error messages for authentication failures
- Shows detailed error responses for debugging
Expand All @@ -93,6 +129,7 @@
### Documentation

- ✅ Updated README.md with comprehensive documentation

- Added Radarr/Sonarr Integration section with benefits and configuration
- Added Verbose Logging section with examples
- Updated environment variables documentation
Expand All @@ -107,12 +144,14 @@
#### Keyword Normalization

- ✅ Added intelligent keyword normalization feature

- Automatically normalizes TMDb keywords for consistent formatting
- Pattern-based recognition for dynamic handling without hardcoding
- Smart title casing with proper article and preposition handling
- Automatic duplicate removal after normalization

- ✅ Pattern Recognition Features

- **Critical Replacements**: Known abbreviations (sci-fi → Sci-Fi, romcom → Romantic Comedy)
- **Acronym Detection**: Automatically uppercases known acronyms (FBI, CIA, DEA, etc.)
- **Agency Patterns**: Detects agency roles (dea agent → DEA Agent)
Expand All @@ -123,6 +162,7 @@
- **Credit Stinger Terms**: Expands compound terms (duringcreditsstinger → During Credits Stinger)

- ✅ Added comprehensive test suite

- 90+ test cases covering various normalization scenarios
- Tests for edge cases, mixed case preservation, and pattern matching
- Ensures consistent behavior across different keyword types
Expand Down Expand Up @@ -156,6 +196,7 @@
### Documentation

- ✅ Updated README.md with comprehensive documentation

- Added Radarr/Sonarr Integration section with benefits and configuration
- Added Verbose Logging section with examples
- Added Keyword Normalization section with pattern examples
Expand All @@ -171,10 +212,11 @@
- Technical implementation details

### Technical Details

- Radarr/Sonarr clients use API v3 endpoints
- Implemented robust error handling and fallback mechanisms
- No breaking changes - Radarr/Sonarr integration is fully optional
- Maintains backward compatibility with existing file path matching
- Verbose logging provides detailed insights without affecting normal operation
- Keyword normalization uses regex patterns for scalability
- All features are designed to be non-breaking and backward compatible
- All features are designed to be non-breaking and backward compatible
Loading