Add SQL-based persistent state management #3702
Description
- I have read and understood docs/Issues.md.
Is your feature request related to a problem? Please describe.
- Oxidized needs a lot of memory to store its state. On an instance with 5k Devices, the Resident Set Size is 350 MB.
- When using the cached git history to display versions in oxidized-web, the Resident Set size goes up to 1,5 GB (git Size: 600MB).
- Compute intensive operations on oxidized block oxidized-web, and compute intensive operations on oxidized-web block oxidized (probably due to the thread lock). Having a SQL persistent Database could better decouple a front end from the back end.
- @ytti noted this in Issue Refactor #812 (first item)
- PR Add SQLite output module #3666 shows the need for an SQL-based persistent state (not directly usable because raw output of LLM)
Describe the solution you'd like
- Oxidized save and reads it states to an SQL backend
- The choice of the SQL backend is flexible
- The git history is cached in the SQL backend
- The SQL backend can be used as an interface to extensions (oxidized-web, Node.js based frontend...)
Describe alternatives you've considered
Keep the state in memory.
Cache git logs only to some extend (last 10 Versions...) and only get more with user interaction.
Additional context
Setting this to Milestone 1.0.0 and assigning it to myself. I'm not sure how much work this will need.