Post tree-states PM #7748
Description
After merging the main tree states changes
Lighthouse has massively reduced its disk footprint on both unfinalized and finalized data while keeping a similar performance profile and reducing I/O significantly.
There a few next steps we can do tho
Overall goals
- Make happy path faster when block processing needs to advance
- Almost never happens in mainnet
- LH not OOM during non-finality: review state caches
Caches
- Cache based on size, measure cache size
- Cache for HDiff buffers
- Improve and unify parellel state caches Improve & unify parallel de-duplication caches #5112
- Removing the pubkey cache is one less cache to build De-duplicate pubkey caches across
BeaconState/BeaconChain#6484
State cache issues / goals
- Needs to adapt the max count of states to not OOM
- Takes 1-2 seconds for the 128 states, so it's something we can run every epoch Measure state cache size in memory #7449
- Needs to be smarted about what to store
- Reduce the impact of a missed state, most of the time is spent rebuilding caches
Extras
- Abolish iterators that load states Abolish iterators that load states #7184
Concerns
- Block processing takes 10 seconds on Holesky.
- Database write takes 10 seconds
- Epoch processing takes 5 seconds sometimes
- The state advance timer sometimes is not useful
When processing a block that requires a skip, we need to advance the state a bunch of slots. For each epoch boundary we have to compute the diff, store it, and maybe run an epoch transition.
- Computing the diff for the block itself is not