chore: update boxo for bitswap providing refactor

[Jorropo]

Keeps in sync with:

• Move providing responsabilities from bitswap to blockservice boxo#534
• Remove content routing from bitswap network boxo#535
• Remove provider query manager boxo#536

I also took time to cleanup how we handle offline dag service and offline blockservice (I added them to the fx 🪄✨) as now blockservice has the side effect of providing (instead of exchange (bitswap)).

[Jorropo]

There used to be a bug where files could take minutes if not hours to be accessible after ipfs add even when using accelerateddhtclient, this has been fixed because now we go through the reprovider which does batching.

[Jorropo]

After testing I think this also fixes #10011 because we don't keep CIDs providing queue in memory (as much), the queue is handled by badger or leveldb which writes to disk.

The providing also goes much faster.

Might not fix it completely, but it certainly help.

[Jorropo]

So after adding 15k blocks:

seq 15000 | parallel -j 1000 "bash -c 'echo {} | ipfs add --raw-leaves -Q'"

Before this patch: RSS is at 512MiB and climbing.
After this patch, RSS bounces between 225MiB and 300MiB, not climbing (the peak is actually reached right after starting the node)

I don't know if I fixed #10011 but whatever this is it's a great improvement.

[hacdias]

@Jorropo I can take a better look once you have the Go tests and the sharness passing

[Jorropo]

@hacdias I didn't because I'm doing more things (rewriting the server) and wanted to handle all of this in kubo at once.
I don't want to bother fixing bugs I might remove the feature for idk.

[aschmahmann]

There's a large set of interlinked PRs here. Trying to understand which are the independent sets of changes here and why we want/need them.

As I understand we have:

1. We need to fix the nopfs breakage regarding sessions and generally make sessions easier to use correctly for added efficiency. That's Difficult to use one session per request (e.g. gateway request) boxo#567
2. We should have the providing pipeline all go through the same batch provider to be able to leverage bulk providing when using the accelerated DHT client. While this could be fixed more indirectly, lifting the providing logic out of boxo/bitswap and at least into boxo/blockservice if not higher seems reasonable. That's the goal of Move providing responsabilities from bitswap to blockservice boxo#534
3. There's a claim that the part of the ProviderQueryManager in boxo/bitswap which limits the number of findprovider queries in flight isn't helping or is hurting the end user experience (certainly magic limits not configurable by users seems bad). The goal of Remove provider query manager boxo#536 is to remove that limit

A couple other things that got hit along the way in the PR chain:

1. Because the ContentRouting interface from go-libp2p includes providing a more limited interface should go into boxo/bitswap. Seems reasonable given a subset of the interface can be defined at the recipient per Go interface semantics.
2. There's a DX claim that switching the internal connection semantics in boxo/bitswap to use peer.AddrInfo rather than peer.ID is better since it aligns with go-libp2p's host.Connect semantics rather than host.NewStream semantics and we're discovering addresses from provider records (and relying on go-libp2p-kad-dht to insert addresses into the peerstore feels weird). . IIUC this is not particularly important given the set of things that would need to change for this to matter, but it should operate as an internal implementation detail anyhow.

@Jorropo does this look about right to you?

[aschmahmann]

If I've got this right it seems like the next steps should be:

1. I'll take the lead on Difficult to use one session per request (e.g. gateway request) boxo#567
2. @Jorropo should refactor Move providing responsabilities from bitswap to blockservice boxo#534 to not depend on other PRs (seems like it shouldn't be a problem but lmk if I'm wrong)
3. @Jorropo should refactor Remove provider query manager boxo#536 to not depend on other PRs (IIUC this should be fine too even without moving the code around sessions, but lmk if I'm wrong)

[lidel]

Closing - my understanding is that it aged quite a lot and this is no longer needed.

The dependent boxo PRs (ipfs/boxo#534, ipfs/boxo#535, ipfs/boxo#536) were closed and the providing refactor landed through a different approach:

• Bitswap: move providing -> Exchange-layer, providerQueryManager -> routing boxo#641 restructured providing/content-routing
• ipfs/boxo@5f2ff9fbc simplified it further

Kubo is now on boxo v0.36.0 which includes all these improvements. The memory issues mentioned in the comments (re: #10011) were addressed as part of that work.

Any follow-up work needs to happen in a fresh PR.