Check for fully removed pieces if not found in deletion scheduling queue

[ZenGround0]

Addressing #924

I suspect that this will fix the problem but I could use your help seeing if this works @TippyFlitsUK

[rvagg]

lgtm, except for that log needing some nuance

[rjan90]

@rvagg With Zen OOO for the day(2026-02-06)/monday, any concerns with me committing the suggestion and merging this, so that we can cut a Curio tag?

[rvagg]

@rjan90 make the changes here and ping me on slack and I'll come and re-review

[TippyFlitsUK]

I am still seeing the errors in my logs after updating to this branch.

Note that these piece 80 alerts were also present prior to the update. They are not new.

{"level":"warn","ts":"2026-02-06T16:28:30.299Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-06T16:29:00.284Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-06T16:29:30.280Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-06T16:29:31.890Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}

[rvagg]

@TippyFlitsUK would you mind grabbing the latest from this branch and giving it a go to see if you get those errors again, or anything like it? I believe this is good to go but would like to see confirmation.

[TippyFlitsUK]

Still seeing these warning messages, I'm afraid:

curio version 1.27.0+calibnet+git_56ce8781_2026-02-09T22:38:27+13:00

{"level":"warn","ts":"2026-02-09T16:54:00.327Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-09T16:54:02.015Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-09T16:54:30.318Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-09T16:54:31.921Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-09T16:55:00.346Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-09T16:55:30.334Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}
{"level":"warn","ts":"2026-02-09T16:56:00.384Z","logger":"pdp","caller":"pdp/watch_piece_delete.go:39","msg":"Failed to process pending piece delete: piece 80 is not scheduled for removal"}

[rvagg]

Suggested change

	return xerrors.Errorf("piece %d is not scheduled for removal", piece.PieceID)
	log.Warnw("piece is live but not in scheduled removals despite successful delete tx; (possible chain reorg) clearing stale delete tracking",
	"dataSetId", piece.DataSetID, "pieceID", piece.PieceID, "txHash", piece.TxHash)
	_, err := db.Exec(ctx, `UPDATE pdp_data_set_pieces SET rm_message_hash = NULL
	WHERE data_set = $1 AND piece_id = $2 AND rm_message_hash = $3`,
	piece.DataSetID, piece.PieceID, piece.TxHash)
	if err != nil {
	return xerrors.Errorf("failed to clear stale rm_message_hash: %w", err)
	}
	continue

I believe we might be seeing this for only a single piece, so I'm going to go out on a limb and say this instance was due to a chain reorg. We got a successful delete tx but now the chain says it's not a delete.

@ZenGround0 plausible?

[ZenGround0]

oooh yeah that's a nice idea. Curio is executing these fast and reorging a block or two is pretty normal.

I'll file an issue to review curio for these more generally. In the past we were handing these pretty well but I suspect we haven't been as rigorous as we need to be.