Caplin didn't use Mev-Boost for Block Production

[drjake007]

After recently updating to Erigon 3.0.2 with Caplin as my Consensus client, Teku 25.4.1 as my validator, and Mev-Boost 1.9, my 1st produced block appears to not have used Mev-Boost, and there were only 5 transactions in the block (with only a .0007 Block reward). Here's the logs from Erigon:
[INFO] [04-28|21:06:35.742] [Beacon API] Found BeaconState object for block production slot=11589631 duration=649.571437ms [INFO] [04-28|21:06:35.752] [ForkChoiceUpdated] BlockBuilder added payload=1 [INFO] [04-28|21:06:35.752] Building block... [INFO] [04-28|21:06:35.763] BlockProduction: GetSyncAggregate took duration=12.352337ms [INFO] [04-28|21:06:35.773] [1/4 MiningCreateBlock] Start mine block=22372222 baseFee=366191074 gasLimit=35999965 [INFO] [04-28|21:06:35.790] Filtration initial=50 no sender=0 no account=0 nonce too low=45 nonceTooHigh=0 sender not EOA=0 fee too low=0 overflow=0 balance too low=0 bad chain id=0 filtered=5 [INFO] [04-28|21:06:35.802] [3/4 MiningExecution] Done Commit every block blk=22372222 blks=2 blk/s=220.3 txs=7 tx/s=770 gas/s=16.83M buf=232.6KB/512.0MB stepsInDB=0.00 step=1810.9 inMem=true alloc=14.5GB sys=29.1GB [INFO] [04-28|21:06:35.802] FinalizeBlockExecution block=22372222 txn=5 gas=152885 receipt=5 payload=1 [INFO] [04-28|21:06:35.811] [4/4 MiningFinish] block ready for seal block=22372222 transactions=5 gasUsed=152885 gasLimit=35999965 difficulty=0 header="&{ParentHash:0xe3631fa05728b7298a38ef866b7e4b6e1889f1d524bc792da87931f5b09be22e UncleHash:0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 Coinb> [INFO] [04-28|21:06:35.821] Built block hash=0xd06bd29d64a8c454c3a29c22103c7eb80f01075bb9de5eff71a398739840b2ed height=22372222 txs=5 executionRequests=0 gas used %=0.425 time=69.094215ms [INFO] [04-28|21:06:35.822] BlockProduction: ForkChoiceUpdate&GetPayload took duration=70.75483ms [INFO] [04-28|21:06:36.119] BlockProduction: GetBlockOperations&findBestAttestations took duration=367.949479ms [INFO] [04-28|21:06:36.597] Check mev bid useLocalExec=true execValue=704571877651510 builderValue=13517663221254276 boostFactor=0 targetSlot=11589631 [INFO] [04-28|21:06:36.606] [Beacon API] Built block consensus-state slot=11589631 duration=8.936027ms [INFO] [04-28|21:06:36.608] [Beacon API] Computed state root while producing slot slot=11589631 duration=2.559657ms [INFO] [04-28|21:06:36.608] BlockProduction: Block produced proposerIndex=425386 slot=11589631 state_root=0xb490b6d6ee6a3953bc0aade850a0ae2db420365f04a7f14a9e2854bc5996441a attestations=128 execution_value=704571877651510 version=deneb blinded=false took=1.516032737s [INFO] [04-28|21:06:36.943] BlockPublishing: publishing block and blobs slot=11589631 blobs=0

I'm wondering if I've configured Erigon/Teku correctly for Mev-Boost?

Here's my Erigon Config:
--chain=mainnet \ --datadir=/var/lib/erigon \ --prune.mode=full \ --http \ --http.addr=0.0.0.0 \ --http.port=8545 \ --ws \ --ws.port=8546 \ --caplin.enable-upnp \ --caplin.discovery.addr=0.0.0.0 \ --caplin.discovery.port=4000 \ --caplin.discovery.tcpport=4001 \ --caplin.mev-relay-url=http://127.0.0.1:18550 \ --caplin.validator-monitor \ --beacon.api=beacon,validator,builder,config,debug,events,node \ --metrics

And my Teku Config:
--network=mainnet \ --data-path=/var/lib/teku \ --beacon-node-api-endpoint=http://127.0.0.1:5555 \ --validator-keys=/var/lib/teku/validator_keys:/var/lib/teku/validator_keys \ --validators-proposer-default-fee-recipient=0x3AD08c2eeB83e4C561B5CFDC62E7645AXXXXXXXX \ --validators-builder-registration-default-enabled=true \ --metrics-enabled=true

Didn't have any problem with Mev-Boost when using Teku as my consensus client.

[domiwei]

[INFO] [04-28|21:06:36.597] Check mev bid                          useLocalExec=true execValue=704571877651510 builderValue=13517663221254276 boostFactor=0 targetSlot=11589631

I noticed that builder_boost_factor (boostFactor) is zero in the logs, which means that the Teku validator client isn't including this value in its requests, so Caplin defaults it to zero. Then always choose local payload.
Could you take a look at how to configure this value in Teku VC?
Here's the relevant spec: https://ethereum.github.io/beacon-APIs/#/Validator/produceBlockV3

[Giulio2002]

what if it is just "unspecified" by default and we count it as 0?

[drjake007]

I've attached the available Teku configuration options. When running Teku as a validator-only client, only the options in the "Validator" section are allowed. I don't see a 'boostFactor' specific option. It also interesting that Caplin only assembled a block with 5 transactions - with MevBoost, I was averaging close to 150 transactions. Has anyone successfully produced a MevBoost block with Caplin using any Validator client?
teku_options.txt

[Giulio2002]

We do support mev-boost and it does work. It seems Caplin reads 0 as the boost factor which is the reason why it does not talk to mev boost. maybe teku does not send a boostFactor by default and expects us to default to 100

[Giulio2002]

builder_boost_factoris defaulted to 0, I will increase to 100 by default

@domiwei review #14913

anyway this is a configurable parameter in Teku VC

[domiwei]

It's merged.
@drjake007 In our experiment we use lighthouse and lodestar vc. These two both include builder_boost_factor in the request and mev-boost works.
Anyway now you can try latest main branch to see if everything works fine if you want.
Less transactions might be another issue

[domiwei]

I found a configurable param builder-bid-compare-factor in teku BN doc
Perhaps that's the reason why it works with teku BN

[drjake007]

Thank you @domiwei
builder-bid-compare-factor is not a valid teku option when running teku as a validator client only. I have a question into the teku team about builder_boost_factor so we'll see what they say.

On another note, after the Pectra upgrade, Caplin is now throwing a warning: [WARN] [05-07|07:24:02.834] [Beacon REST] failed to process attestation in attestation service err="committee index must be 0" so attestations are not being processed.

Went back to using teku as my consensus client until this gets resolved.

Also, with teku as the consensus client, now getting an Error from teku: teku[2510]: 08:51:03.037 FATAL - PLEASE CHECK YOUR ETH1 NODE | Encountered a problem retrieving deposit events from eth1 endpoint: Expected next deposit at index 1966766, but got 1983690 although it doesn't seem to affect publishing attestations.

[domiwei]

I meant builder-bid-compare-factor is a parameter when running for beacon node, which defaults to 90 according to the doc.
and with this error err="committee index must be 0" I guess you need to upgrade your teku vc to better support pectra upgrade?

[drjake007]

Why do you believe the Caplin warning: [WARN] [05-07|07:24:02.834] [Beacon REST] failed to process attestation in attestation service err="committee index must be 0" is an issue with the teku validator client? Running a full teku client (beacon + validator) after Pectra works fine as far as attestations go.

[Giulio2002]

@domiwei please test teku vc. @drjake007 switch away from Caplin for now. It is working with lodestar/lighthouse VC so this is weird

[domiwei]

If it'a latest version then that's weird. Because it's an important check in electra spec for a deprecated field to make sure it equals to zero.
Anyway let me do experiment for teku.

[drjake007]

@domiwei I set up a Hoodi testnet and was able to reliably repeat the issue with the teku vc:

May 15 08:24:26 ckubuntu2 erigon[21491]: [DBUG] [05-15|08:24:26.181] Produced Attestation                     app=caplin slot=425772 committee_index=38 cached=true beacon_block_root=0x79bbfc546b7a9b2b570777f29391933de014860bfc8ef8fb9aa8a9f3208c92f1 duration=24.576µs
May 15 08:24:26 ckubuntu2 erigon[21491]: [DBUG] [05-15|08:24:26.186] clstage finish                           app=caplin stage=ForkChoice in=911.782051ms next=SleepForSlot
May 15 08:24:26 ckubuntu2 erigon[21491]: [WARN] [05-15|08:24:26.194] [Beacon REST] failed to process attestation in attestation service err="committee index must be 0"

I then set up a Lighthouse vc and had no issues with the attestations:
May 15 18:53:28 ckubuntu2 erigon[21491]: [DBUG] [05-15|18:53:28.002] Produced Attestation app=caplin slot=428917 committee_index=57 cached=true beacon_block_root=0x1141dcd1cd195731c680ee3409dbab6cb7d9bd2f0202fc1ed0c3484eb42fa466 duration=15.55µs

Note that for both teku and lighthouse, the committee_index was non-zero, yet only teku failed.

[domiwei]

@drjake007 Thanks for your experiment on lighthouse.
I've tried using teku as vc and found some minor issues. I think we already fixed it but now only on main branch.
Will pick this fix to release branch, so maybe wait for our next release. :)

[drjake007]

@domiwei was the issue fixed in #15034 and if so, has it been promoted to main?

I built the main branch and ran it again on hoodi, and still getting the same error with attestations:

May 19 19:06:14 ckubuntu2 erigon[79796]: [DBUG] [05-19|19:06:14.814] Produced Attestation                     app=caplin slot=457781 committee_index=9 cached=true beacon_block_root=0x8012b917a7607e77cfd8afa367566d32537c7af386d21b1a1eb1c16f41ae5a22 duration=97.466µs
May 19 19:06:15 ckubuntu2 erigon[79796]: [DBUG] [05-19|19:06:15.161] [pubsubhandler] stream handler returned error protocol=/eth2/beacon_chain/req/goodbye/1/ssz_snappy peer=16Uiu2HAkxy3eQf71UYhzm1jU6XpwaVUdoquazgFbYaocUL48WDU3 err="unable to read varint from message prefix: stream reset"
May 19 19:06:18 ckubuntu2 erigon[79796]: [WARN] [05-19|19:06:18.956] [Beacon REST] failed to process attestation in attestation service err="committee index must be 0

Here's the Erigon build info:

erigon[79796]: [INFO] [05-19|19:02:57.309] Build info git_branch=main git_tag=v3.0.0-beta1-764-g9f9318b git_commit=9f9318bea8aa7ff2ec28c685559dd93256e0244e

Thanks

[domiwei]

That is weird. I'll look into it.
ok. I found we didn't clean the cache. It will be fixed soon

[drjake007]

@domiwei looks good!

erigon[87354]: [DBUG] [05-20|09:29:02.550] Produced Attestation app=caplin slot=462095 committee_index=51 cached=true beacon_block_root=0x64a1a0c39f06a5629ddc91a622b3be90d15c388f91f88a7787eeb3035fb70700 duration=81.234µs

teku[87269]: 09:29:07.132 INFO - Validator *** Published attestation Count: 1, Slot: 462095, Root: 64a1a0c39f06a5629ddc91a622b3be90d15c388f91f88a7787eeb3035fb70700

erigon[87354]: [INFO] [05-20|09:24:29.558] Build info git_branch=main git_tag=v3.0.0-beta1-775-ga1ae240 git_commit=a1ae2408fe382bbbeaa3db39e546803a1de13e72

[domiwei]

Thanks for the testing from your side.
Have you already tried mev again? We already changed boost_factor and defaulted it to 100, so it should work if payload from external relay gets a higher score.

[drjake007]

I have hoodi running with Caplin/Teku/Mev, but will have to wait for a block proposal before the boost_factor fix can be verified.

[drjake007]

@domiwei I got a block proposal on hoodi. Looks like Caplin is now properly processing Mev builder bids:

erigon[74834]: [DBUG] [06-06|19:35:36.001] [Beacon API] Producing block             slot=587528
erigon[74834]: [INFO] [06-06|19:35:36.433] [Beacon API] Found BeaconState object for block production slot=587528 duration=431.541ms
erigon[74834]: [INFO] [06-06|19:35:36.435] [ForkChoiceUpdated] BlockBuilder added   payload=1
erigon[74834]: [INFO] [06-06|19:35:36.435] Building block...
erigon[74834]: [INFO] [06-06|19:35:36.436] BlockProduction: GetSyncAggregate took   duration=1.222638ms
erigon[74834]: [DBUG] [06-06|19:35:36.454] [db] open                                label=temporary sizeLimit=512GB pageSize=4KB
erigon[74834]: [DBUG] [06-06|19:35:36.455] [1/4 MiningCreateBlock] Starting Stage run
erigon[74834]: [INFO] [06-06|19:35:36.455] [1/4 MiningCreateBlock] Start mine       block=551666 baseFee=914293711 gasLimit=59941351
erigon[74834]: [DBUG] [06-06|19:35:36.455] [1/4 MiningCreateBlock] DONE             in=349.225µs
erigon[74834]: [DBUG] [06-06|19:35:36.455] [2/4 MiningBorHeimdall] Starting Stage run
erigon[74834]: [DBUG] [06-06|19:35:36.455] [2/4 MiningBorHeimdall] DONE             in=6.242µs
erigon[74834]: [DBUG] [06-06|19:35:36.455] [3/4 MiningExecution] Starting Stage run
erigon[74834]: [DBUG] [06-06|19:35:36.467] [db] open                                label=temporary sizeLimit=512GB pageSize=4KB
erigon[74834]: [DBUG] [06-06|19:35:36.469] [txpool] Processing best request         last=551665 txRequested=50 txAvailable=32 txProcessed=32 txReturned=32
erigon[74834]: [INFO] [06-06|19:35:36.469] Filtration                               initial=32 no sender=0 no account=0 nonce too low=0 nonceTooHigh=0 sender not EOA=0 fee too low=0 overflow=0 balance too low=0 bad chain id=0 filtered=32
erigon[74834]: [DBUG] [06-06|19:35:36.469] Transaction adding was requested to stop payload=1
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0x29ee8dd40183707f18964aeba2e75abc988ea425d177c01a3f79ed9430928ae2 sender=0x3F63ae7cb6EF734387AA8223CfFa7D4e1a075847
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0x296187846670897584c5da64d06e7dc9075b39f9225ffc718b4283053119fc88 sender=0x4eB310da5bb44E287ebEC2DFbbE5C75A283504a6
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0x4f78296ced957f27c537fbf2aa2981487de929c137263d74ad800e44962467df sender=0x57514D6C07956Bf94311b9A2f98F08C6f59b705D
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0xcf02b2e2cd27c061de5211c477c7699512da6a9544500ec11a962b87b4e990b1 sender=0xB6d85Ba2388324aC0587B2Fd7021014a729a8038
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0x01766e592a9dc45be6ec9f8b710ee62fb99462d594c7cdc4b1ed217b8ae8e247 sender=0x9FBAB94921d3832bBB2Fd3b93a06f17be2d1845D
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0xabb862f44bde393abaad47ff1bb3ceb147278ffbfa8dbe9bf1c8b408c51abcaa sender=0xB77A7a4583e9dD6eAf2F1a845a3915EBDff2E395
erigon[74834]: [DBUG] [06-06|19:35:36.643] [3/4 MiningExecution] Gas limit exceeded for env block hash=0x34ba9a41cf9b97d46c80688da15a878c0bf9ed0d0883620db0b154811a7e12b6 sender=0x8cA045450dec3Cea547a8a00b7Abea7eFA4B7720
erigon[74834]: [DBUG] [06-06|19:35:36.643] SpawnMiningExecStage                     block=551666 txn=25 payload=1
erigon[74834]: [DBUG] [06-06|19:35:36.818] [commitment] store state                 block=551666 txNum=7466604 rootHash=f4d6a0c9841630ecbd4ab5554060a2ea94edb6e61bff9765d5d99d69e7ab6f0c
erigon[74834]: [INFO] [06-06|19:35:36.818] [3/4 MiningExecution] Done Commit every block blk=551666 blks=2 blk/s=11.7 txs=27 tx/s=157 gas/s=336.04M buf=312.6KB/512.0MB stepsInDB=0.00 step=4.8 inMem=true alloc=5.6GB sys=8.1GB
erigon[74834]: [INFO] [06-06|19:35:36.818] FinalizeBlockExecution                   block=551666 txn=25 gas=57563995 receipt=25 payload=1
erigon[74834]: [DBUG] [06-06|19:35:36.826] [3/4 MiningExecution] DONE               in=370.515878ms
erigon[74834]: [DBUG] [06-06|19:35:36.826] [4/4 MiningFinish] Starting Stage run
erigon[74834]: [INFO] [06-06|19:35:36.826] [4/4 MiningFinish] block ready for seal  block=551666 transactions=25 gasUsed=57563995 gasLimit=59941351 difficulty=0 header="&{ParentHash:0x2a48ba6417526d77416fb7f2473cf80d6b611a964c24251db8e49a14ab69fbfe UncleHash:0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 Coinbase:0x1fc253e00D50762D7A97b585188D87739651f7c5 Root:0xf4d6a0c9841630ecbd4ab5554060a2ea94edb6e61b>
erigon[74834]: [DBUG] [06-06|19:35:36.827] [4/4 MiningFinish] DONE                  in=818.539µs
erigon[74834]: [INFO] [06-06|19:35:36.835] Built block                              hash=0x62e4d9334db80e21c7f338ca8eaa614d96e383045172c82a35abde2f6bcf6c5e height=551666 txs=25 executionRequests=0 gas used %=96.034 time=399.663995ms
erigon[74834]: [INFO] [06-06|19:35:36.836] BlockProduction: ForkChoiceUpdate&GetPayload took duration=400.895ms
erigon[74834]: [DBUG] [06-06|19:35:37.051] error executing clstage                  app=caplin stage=ChainTipSync err="context deadline exceeded"
erigon[74834]: [DBUG] [06-06|19:35:37.051] clstage finish                           app=caplin stage=ChainTipSync in=25.050894786s next=ChainTipSync
erigon[74834]: [INFO] [06-06|19:35:37.054] BlockProduction: GetBlockOperations&findBestAttestations took duration=619.867517ms
erigon[74834]: [DBUG] [06-06|19:35:37.055] Produced BeaconBody                      app=caplin slot=587528 duration=620.158682ms
erigon[74834]: [DBUG] [06-06|19:35:37.102] waiting for blocks...                    app=caplin stage=ChainTipSync seenSlot=587525 targetSlot=587528 requestedSlots=3
erigon[74834]: [DBUG] [06-06|19:35:37.295] MevBoost                                 app=caplin slot=587528 duration=860.876641ms
erigon[74834]: [INFO] [06-06|19:35:37.295] Check mev bid                            useLocalExec=false execValue=4973958858699755 builderValue=104097659804127375 boostFactor=100 targetSlot=587528
erigon[74834]: [INFO] [06-06|19:35:37.307] [Beacon API] Built block consensus-state slot=587528 duration=11.283839ms
erigon[74834]: [INFO] [06-06|19:35:37.309] [Beacon API] Computed state root while producing slot slot=587528 duration=2.074972ms
erigon[74834]: [INFO] [06-06|19:35:37.309] BlockProduction: Block produced          proposerIndex=1142123 slot=587528 state_root=0x3b03132a59371967fab6f093a0a764582476ddfe7a9f21a2e72e4580645b9ce4 execution_value=104097659804127375 version=electra blinded=true took=1.307605373s
erigon[74834]: [DBUG] [06-06|19:35:37.726] Received blob sidecar via gossip         index=0 size=128KB
erigon[74834]: [DBUG] [06-06|19:35:37.975] Received block via gossip                slot=587528

[domiwei]

good 👍 Then I close the issue