pdpv0: stuck settlement transactions after confirmed on-chain but failed DB update

[TippyFlitsUK]

Environment

• Curio version - curio version 1.27.0+calibnet+git_0e56f82e_2026-01-22T18:51:12-05:00
• Node - calib.ezpdpz.net (ID 9)

Summary

Five settlement transactions became stuck in a loop causing repeated "Failed to process pending settle transactions" warnings. The transactions were confirmed on-chain
(tx_status='confirmed', tx_success=TRUE in message_waits_eth) but remained in filecoin_payment_transactions, causing the settle processor to repeatedly attempt processing them.

Root Cause

The transactions were successfully confirmed on-chain but the subsequent database cleanup (removing from filecoin_payment_transactions) failed or was interrupted. This left the
records in an inconsistent state where:

• message_waits_eth shows: tx_status='confirmed', tx_success=TRUE
• filecoin_payment_transactions still contains the records

Affected Records

  0x7d6bff4e0b0c3214d11c99b5599b1c719e26c5f500fe89ff3040763a4f21d0ef                                                                                                                       
  0x3e0ddefffc41e61f31e8be1e01fb945733cbc859899e08895f63269d04fd22f4                                                                                                                       
  0xf53ad83ef9f8226b1cb52c19f9c7405038e2dbc347b0cb50147d74bcc28c0539                                                                                                                       
  0x7ca5ec103015cbd74fbf8ac8944e96f0510216b57e60cc6b59ed1617e3b74168                                                                                                                       
  0xe7b4280f6f9b943e3da69b7e05c621013f24e7b58d0a75b2f899546d588c5ba8     

Resolution

Manual deletion of the stuck records:

DELETE FROM curio.filecoin_payment_transactions                                                                                                                                          
WHERE tx_hash IN (                                                                                                                                                                       
  '0x7d6bff4e0b0c3214d11c99b5599b1c719e26c5f500fe89ff3040763a4f21d0ef',                                                                                                                  
  '0x3e0ddefffc41e61f31e8be1e01fb945733cbc859899e08895f63269d04fd22f4',                                                                                                                  
  '0xf53ad83ef9f8226b1cb52c19f9c7405038e2dbc347b0cb50147d74bcc28c0539',                                                                                                                  
  '0x7ca5ec103015cbd74fbf8ac8944e96f0510216b57e60cc6b59ed1617e3b74168',                                                                                                                  
  '0xe7b4280f6f9b943e3da69b7e05c621013f24e7b58d0a75b2f899546d588c5ba8'                                                                                                                   
);    

Diagnostic Query

To find similar stuck records in the future:

  SELECT fpt.tx_hash, fpt.rail_ids                                                                                                                                                         
  FROM curio.filecoin_payment_transactions fpt                                                                                                                                             
  JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash                                                                                                                     
  WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;   

[ZenGround0]

Is it possible to get more logging info on the error? The thing is that

message_waits_eth shows: tx_status='confirmed', tx_success=TRUE
and
filecoin_payment_transactions still contains the records

are not inconsistent state, they are simply the precondition for processing a settlement tx. What is interesting is that the watcher is not fully processing and deleting the record. I think the only way you are going to see loops like this is if you are seeing failed to verify settle: .... It would be very helpful to see the error after failed to verify settle. Any subcall of verify settle failing is interesting and unexpected.

[TippyFlitsUK]

I have uploaded the full context logs here, @ZenGround0 - https://drive.google.com/file/d/1aUOKQFiiO996sa1EoKkPwh9ecdNK0m8a/view?usp=sharing
These are from calib2.ezpdpz.net 👍

[TippyFlitsUK]

Additional Affected Node Report

Environment

• Curio version: 1.27.0+calibnet+git_56ce8781_2026-02-09T22:38:27+13:00
• Node: main.ezpdpz.net

Summary

Issue still exists on this node with 176 stuck settlement transactions (significantly more than the original 5 reported).

Diagnostic Query Results

SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt
JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash
WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;
-- Result: 176 rows

Active Symptoms

Warnings appearing continuously every ~30 seconds in /var/log/curio/curio.log:

{"level":"warn","ts":"2026-02-10T12:55:30.079Z","logger":"filecoin-pay-settle","caller":"pay/watcher.go:40","msg":"Failed to process pending settle transactions: failed to verify settle: failed to get rail to data set: message execution failed (exit=[33], revert reason=[message failed with backtrace: 00: f0176641 (method 3844450837) -- contract reverted at 75 (33) 01: f0176641 (method 6) -- contract reverted at 513 (33) (RetCode=33)], vm error=[none])"}

Sample Affected Transaction Hashes (first 10 of 176)

0xc5903f3120bed177d13798d08352ea4613933df89dd875e28efbcf4cb63b37a0
0x104aff812c36a2bbb41f1fe410c611ef763968c0c69517102deaa35a3a460592
0x35c68fd3e2fe4172df4b0634de20ab59883638853713ff0fcf55a228de7978e5
0x842251792091a1b06963cab16f3f297f0d487459cf6e28e57957f1aa73e79420
0x0379f79a40cd0d32d87ebc5aa53f519631f117f6553a76b259c9581555c2d00f
0xad159fad4a7fd217d97776e5937d1393175d85a5dd0c772f7aa1479177265b1e
0xa72f8d1491cd1f6369e023d55847134a65d61229675d6a1f639e7718c305d80f
0x7a08c1ca311369c873f27269acf7a4b09058b8c8e14232dfd0281ecff097fdb4
0x53933747b8a475b1ba62bc54de1b61487c77b439178ab2f2bf6767fa073904e1
0x5fe37630d2b04e1099150c63cc07caae0f6aca6c24ad376608b06d1e9def855d

Impact

• Settle processor running in infinite loop attempting to process already-confirmed transactions
• Excessive log spam (~2 warnings per minute)
• Potential resource waste from repeated failed processing attempts