pdpv0: Settle watcher doesn't block on fwss railToDataSet errors

[ZenGround0]

Closes #922

This is a strange one because the rail to dataset mapping should

1. Never fail with a revert
2. Never fail with empty value until dataset is deleted which should ONLY be possible after we have processed settlement in the watcher

So in short I have no idea how we are in this state. But there is probably no reason to be stopping the world if we get in such a bad state.

@TippyFlitsUK it would be very helpful to see the railIds in your new log so I can try to understand what is going on with fwss contract to cause such errors in the first place. But I hope that after merging this your watcher will only log the error once and then move on.

[rjan90]

@ZenGround0 Beck also posted some RailIDs he is seeing here: #922 (comment)

[TippyFlitsUK]

BEFORE node update to fix/settle-failures

calib.ezpdpz.net - LOGS

"SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;"
 count 
-------
   182
(1 row)

calib2.ezpdpz.net - LOGS

"SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;"
 count 
-------
   262
(1 row)

main.ezpdpz.net - LOGS

"SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;"
 count 
-------
     5
(1 row)

[TippyFlitsUK]

POST 961 node update

Looking good! No new instances of the error occurring on any of the 3 nodes using:

tail -f /var/log/curio/curio.log|grep "failed to verify settle"

calib.ezpdpz.net - calib_961.log

SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;"
 count 
-------
     0
(1 row)

calib2.ezpdpz.net - calib2_961.log

"SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;"
  
 count 
-------
     0
(1 row)

main.ezpdpz.net - main_961.log

"SELECT COUNT(*) FROM curio.filecoin_payment_transactions fpt JOIN curio.message_waits_eth mwe ON fpt.tx_hash = mwe.signed_tx_hash WHERE mwe.tx_status = 'confirmed' AND mwe.tx_success = TRUE;"
 count 
-------
     0
(1 row)

[rvagg]

I started reviewing this, put it down for a while, came back and it was merged. But that exercise resulted in this: #973