Add SQLite output module #3666
Add SQLite output module #3666
Conversation
Introduces a new State class that replaces in-memory state storage with SQLite persistence for node statistics, job tracking, and scheduling data. Includes automatic schema migration and WAL mode for better concurrency.
Tests cover all state management operations including node statistics, job tracking, mtime updates, cleanup operations, and concurrent access.
State is now initialized at application startup and accessible via Oxidized.state throughout the application.
Stats are now stored in SQLite instead of memory with caching to minimize database queries. Maintains backward compatibility with existing API.
Last job information is now persisted to SQLite and loaded on-demand. Maintains in-memory cache for performance.
Job durations used for scheduling are now persisted to SQLite and restored on application restart.
When nodes are reloaded, state for removed nodes is automatically cleaned up from the database to prevent data accumulation.
- Validate all input data (node names, jobs, numeric values) - Set secure file permissions (0600) on database files - Set directory permissions (0700) on state directory - Type coercion for Time and Float values - Bounds checking and finite number validation
Tests cover: - Input validation (nil, empty, oversized values) - Data type validation (infinite, NaN values) - File permission security - Unicode support - Numeric precision - Edge cases (large counters, rapid updates) - Various time formats and status symbols
Sequel is now required for persistent state management and must be available at runtime, not just for development and testing.
Documents architecture, security measures, configuration, troubleshooting, and API reference for the new SQLite-based state persistence system.
Documents all additions, changes, and removals related to the migration from in-memory to persistent SQLite-based state management, including migration notes for users upgrading.
Allows users to inspect and debug the persistent state database using sqlite3 command-line tool inside the container.
- Remove trailing whitespace from all modified files - Fix hash key alignment for consistent formatting - Convert symbol arrays to %i notation - Rename get_job_durations to job_durations (remove get_ prefix) - Simplify if statements to modifier form where appropriate - Fix numeric literal formatting (add underscores) - Refactor nested if-else to elsif in Jobs class
- Fix method chain indentation alignment (align with opening element) - Fix hash key/value alignment for consistent formatting - Convert remaining symbol arrays to %i notation - Refactor guard clauses (use early return pattern) - Remove redundant parentheses in test assertions - All 46 RuboCop offenses resolved
- New output type: sqlite (in addition to file, git, http) - Store all device configs in single SQLite database file - Automatic version history (up to 100 versions per device) - Support for groups and obsolete node cleanup - Secure file permissions (0600) and WAL mode - Implements store, fetch, version, get_version methods - Compatible with existing output interface
- Test store, fetch, and version operations - Test group support and obsolete node cleanup - Test database security (file permissions) - Test version history limits and ordering - Test concurrent access and WAL mode - 200+ lines of test coverage
- Add comprehensive SQLite output documentation - Include configuration examples - Document version history features - Explain database schema and security - List advantages and limitations - Add Docker deployment examples
- Add SQLite output to Added section - Document key features and capabilities - Reference documentation location
- Remove duplicate 'end' statement on line 380 - Fixes unexpected token kEND syntax error - File now passes Ruby syntax check
- Fix method chain indentation alignment - Remove redundant :: prefix in spec file - Use safe navigation operator (&.) - Replace sprintf with String#% operator - All RuboCop offenses resolved
- Align all method chains with opening receiver (@db) - Fix alignment in update_node_stats, update_mtime - Fix alignment in add_job_duration, trim_history - All 28 RuboCop offenses resolved - Code now passes RuboCop checks
|
Persistent state is something we definitely need. I'm just worried about the author's comprehension on what they are doing, as quick glance makes me feel this is very LLMy work. Quick glance didn't give me any obvious specific concerns, just LLMyness made me worry that lot of care will be needed to review. |
r2unit
changed the title
|
Thanks for the feedback, yes i've used claude for the boilerplate, rubocop fixes and also some other minor issues i encountered. |
|
An example:
Existing statistics are always lost on upgrade as they are being stored in memory. An LLM does not understand what it is doing. It does only outputs what statistically seams the best option. The only thing I could do with this PR is use it as an inspiration for implementing a persistent state or an SQL Output - they are separate things that should probably be implemented separately. I would probably prefer starting from scratch and think about the best way to do it. Both are not in my top priority, so I will not review this PR. |
3 participants
Pre-Request Checklist
Description
adds a new SQLite output module and fixes RuboCop violations to get CI passing.
SQLite Output Module
Adds support for storing device configurations in a SQLite database, similar to the existing file and git outputs. This is useful for deployments where you want versioned configs in a single portable database file.
Features:
Example config:
RuboCop Fixes
Fixed all style violations to make CI happy:
endstatement)Both features include tests and documentation updates.