Prune blobs can OOM

[michaelsproul]

Description

We have a bug report from AllNodes about a Lighthouse node using 77GB of RAM when switching from --prune-blobs false to --prune-blobs true.

They helpfully shared a jemalloc memory dump which shows that the allocations are within do_atomically_with_block_and_blobs_cache, in particular in the get_blobs call under the partition.

[michaelsproul]

The reason we're doing get_blobs is so we can undo the write to the blobs DB in the case where the write to the hot DB fails.

I would like to consider refactoring and deleting large parts of do_atomically_with_block_and_blobs_cache.

[michaelsproul]

Alternative strategy:

Do away with the oldest_blob_slot. It's kind of a pain anyway. Pruning could proceed by just iterating all the blobs and deleting ones prior to the data availability period. On nodes that are already pruning there aren't really that many blobs, and we can avoid having to hold too many in memory (load one at a time, check whether it needs to be deleted, stage for deletion). For the OP's case of pruning a DB with lots of blobs this will also work fine, and will only run slowly once (the first time switching from --prune-blobs false to true).

I think this will allow us to get rid of all the complexity around reverting blob DB transactions and coordinating writes across databases. I think the only other place it's currently used is when importing blocks. In this case we can write the blobs first, and then write the block, so that we maintain the invariant:

block in hot_db && blobs_in_da_period --> blobs for block in blobs_db

[eserilev]

i can pick this one up

I think itd be nice to add a delete_while method to the KeyValueStore trait, just like in the slasher, for stuff like this. It also allows us to use extract_if for redb which is optimized for deleting a range of values

[michaelsproul]

Fixed now by:

• Modularize beacon node backend #4718