Self rate limiting due to number of concurrent requests for `data_column_sidecars_by_range`

[jimmygchen]

When trying to sync on a PeerDAS-enabled devnet, we see a large number of self rate limiting logs. Range sync only sends up to 5 batches concurrently, but this changes with PeerDAS, as we may send more than 1 data column requests per batch - so might be worth investigating and see if the current limit makes sense for data column requests, as this will slow down sync:

Jul 05 18:08:02.446 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"
Jul 05 18:08:02.549 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"
Jul 05 18:08:02.651 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"
Jul 05 18:08:02.753 DEBUG Self rate limiting due to the number of concurrent requests  peer_id: 16Uiu2HAmACmiJYvqiMWDiVN8tbwTpHqFWxj3E9bPvnRq1jeokGhH, protocol: data_column_sidecars_by_range, service: "network"

[jimmygchen]

@pawanjay176 wonder if you are aware of this - given you were able to sync with up to 20 slots /s on your branch?

[dapplion]

We could address this by prioritizing peers with no active requests on the same protocol. With a similar function to this one but only for that specific topic

lighthouse/beacon_node/network/src/sync/network_context.rs

Line 406 in b9c1a2b

fn active_request_count_by_peer(&self) -> HashMap<PeerId, usize> {

[jimmygchen]

This isn't really a bug. The request is queued due to number of concurrent requests for the same protocol - this is part of the spec. The logs may be a bit noisy on a smaller network, but i might be fine after we tweak the peer maintainance logic in #7916 and less noisy on a real network with 100+ peers (200 from the next release)

Although i think it's worth implementing Lion's suggestion above. I'll make a PR.

[jimmygchen]

@pawanjay176 IIRC you were going to check with @ackintosh about this, anything we need to action on?

I don't know if we really benefit from changing the peer selection, because we already prioritise by request count, so if the same peer is selected, it likely means don't have any other options? Although if we leave it in AwaitingDownload, we might be able to request from another peer sooner if a peer joins the chain.

Still have to figure out why the span live for so long, doesn't seem to be related to the concurrent request self limiting

[jimmygchen]

Moving to v8.0.0