feat: Add Ethereum Transaction Retry Logic with Receipt Confirmation

[salindne]

Problem

The transaction 0x1c1ea4... was submitted via Flashbots Protect but expired without being mined. Current code has three gaps:

1. No receipt confirmation - WithdrawFromCanton returns immediately after tx submission
2. No retry implementation - Reconciliation loop has TODOs but no actual retry logic
3. No gas bumping - Stuck transactions are not resubmitted with higher gas

The database already has retry_count and error_message fields, and config has max_retries/retry_delay - they are just unused.

---

Architecture

sequenceDiagram
    participant P as Processor
    participant E as EthClient
    participant DB as Store
    participant R as Reconciler
    
    P->>E: WithdrawFromCanton
    E->>E: Submit tx
    E->>E: WaitForReceipt (timeout)
    alt Receipt received
        E-->>P: txHash, nil
        P->>DB: UpdateStatus(completed)
    else Timeout/Error
        E-->>P: "", error
        P->>DB: UpdateStatus(pending), IncrRetry
    end
    
    R->>DB: GetRetryableTransfers
    loop For each pending transfer
        R->>E: ResubmitWithGasBump
        R->>DB: UpdateStatus
    end

Loading

---

Implementation Plan

1. Database Layer (pkg/db/store.go)

Add methods:

• GetRetryableTransfers(direction, maxRetries) - returns failed/pending transfers eligible for retry
• IncrementRetryCount(id, errMsg) - increments retry count and sets error message
• UpdateTransferPending(id) - resets a transfer to pending for retry

2. Ethereum Client (pkg/ethereum/client.go)

Add receipt confirmation:

• WaitForReceipt(ctx, txHash, timeout) - polls for transaction receipt with timeout
• WithdrawFromCantonWithConfirmation(ctx, ...) - submits withdrawal and waits for receipt

Add config fields to EthereumConfig:

• tx_timeout (default: 5m)
• gas_bump_percent (default: 20)

3. Reconciliation Engine (pkg/relayer/engine.go)

Replace the TODO in runReconciliation with actual retry logic:

• Get failed transfers eligible for retry
• For each: check on-chain status, resubmit with gas bump if needed
• Update database status

4. Handler Updates (pkg/relayer/handlers.go)

Update EthereumDestination.SubmitTransfer to:

• Wait for receipt confirmation
• Mark as pending (not failed) on timeout for retry eligibility

---

Key Design Decisions

Decision	Rationale
Receipt timeout: 5 min	Flashbots bundles have ~5 min lifetime
Gas bump: 20% per retry	Standard practice for unstuck transactions
Max retries from config	Already exists (default 3), just use it
On-chain check before retry	Prevents duplicate withdrawals if tx mined late

---

Files to Change

File	Changes
pkg/db/store.go	Add GetRetryableTransfers, IncrementRetryCount
pkg/ethereum/client.go	Add WaitForReceipt, receipt timeout, gas bumping
pkg/config/config.go	Add tx_timeout, gas_bump_percent fields
pkg/relayer/engine.go	Implement retry logic in runReconciliation
pkg/relayer/handlers.go	Update EthereumDestination.SubmitTransfer to use confirmation
config.example.yaml	Document new config options

---

Tasks

• Add GetRetryableTransfers and IncrementRetryCount to pkg/db/store.go
• Add WaitForReceipt and gas bump logic to pkg/ethereum/client.go
• Add tx_timeout and gas_bump_percent to EthereumConfig
• Implement retry logic in runReconciliation in pkg/relayer/engine.go
• Update EthereumDestination.SubmitTransfer to wait for confirmation
• Document new config options in config.example.yaml