Potential CPU oversubscription in `BeaconProcessor` due to unscoped `rayon` usage

[jimmygchen]

Currently, BeaconProcessor is initialised with max_workers set to the number of available CPUs:

lighthouse/beacon_node/beacon_processor/src/lib.rs

Lines 251 to 254 in f67084a

	impl Default for BeaconProcessorConfig {
	fn default() -> Self {
	Self {
	max_workers: cmp::max(1, num_cpus::get()),

Each task spawned by the processor consumes one worker. However, some tasks (e.g. Work::ColumnReconstruction) internally use rayon to parallelise computation. By default, rayon uses its global thread pool, which is also sized to the number of CPUs.

This likely results in CPU oversubscription: both Lighthouse and rayon independently assume full CPU availability, leading to degraded performance under load.

Proposed Solution

Consider allowing BeaconProcessor to allocate multiple workers to expensive tasks. When spawning such a task:

• Wait until n workers are free (e.g. 4),
• Spawn the task with a scoped rayon thread pool limited to those n threads,
• Release the workers once the task completes.

This would preserve control over CPU usage while still enabling parallelism within heavy tasks.

Additional Details

Discussion below with @michaelsproul from the experimental PR #7720:

BeaconProcessor

Async task

• Keep computation light
• Any blocking computation go through the beacon processor
• Not bounded by max workers (num_cpus) but probably still need a queuing system to make sure it doesn't go unbounded and cause memory issues

Blocking task

• Gets allocated 1 thread by default
• For heavy tasks that requires rayon, allowWorkTypes to acquire more than 1 worker (N) - when executing, create a scoped rayon pool and run the parallel tasks within the scope, and release the workers after it completes

CPU allocation

• tokio runtime: num_cpus thread (default)
• BeaconProcessor: running inside the tokio runtime, and maintains blocking workers max_blocking_workers = num_cpus
• rayon: scoped per task

Memory

• Both sync and async tasks are bounded by queues of each individual work type.

[michaelsproul]

Wait until n workers are free (e.g. 4)

This sounds like it could lead to starvation: if there are single-thread tasks continuously jumping in front of us, we may never get 4 free tasks. Maybe we could temporarily oversubscribe, and then avoid scheduling more work while we're oversubscribed. I.e. allow current_workers to go a small amount over max_workers. Maybe we could rename current_workers -> current_worker_threads.

lighthouse/beacon_node/beacon_processor/src/lib.rs

Line 828 in 999b045

pub current_workers: usize,

[dapplion]

Maybe we could temporarily oversubscribe, and then avoid scheduling more work while we're oversubscribed

Yeah I was thinking the same, reconstruction will just prevent new things from happening for a bit

[jimmygchen]

Had some discussion in the closed PR
#7720 (comment)

I'll move it to description here.

I think it's worth avoiding oversubscription (even if it's a short duration) if it's simple to do - i'm not sure if a 50ms task would only delay by 50ms if it's oversubscribed (delay may not be linear) - my guess is CPU heavy tasks would struggle more, and overall it would be slower.

[michaelsproul]

Plan:

• Add metrics for task start time so we can measure total task execution from event queueing to completion.

Test:

• Temporary oversubscription (current approach in Allocate multiple threads to column reconstruction task #7789) vs waiting to schedule (@jimmygchen's idea: don't schedule the 4 task until there are 4 worker slots available, and don't schedule any 1-tasks if the next task is a 4-task waiting for space).
• Scoped Rayon pools (more flexible) vs Global rayon
• Unbounded async vs other approaches

[michaelsproul]

Test:

• Beacon processor vs OS scheduler (try unbounded beacon processor threads 😱 ). Outcome: terrible. Hundreds of thousands of threads turned out to be overwhelming, everything was slow (10s+ for gossip verification of attestations).

[jimmygchen]

• Temporary oversubscription (current approach in Allocate multiple threads to column reconstruction task #7789) vs waiting to schedule (@jimmygchen's idea: don't schedule the 4 task until there are 4 worker slots available, and don't schedule any 1-tasks if the next task is a 4-task waiting for space).

I think my original suggestion of waiting is suboptimal, and oversubscription shouldn't be too bad if we don't allocate a large number of CPUs (current behaviour is rayon allocates num_cpus, which resulted in overall slow down)