Fix race condition causing bookmark queue items to accumulate during rapid succession resumes

[Copilot]

When multiple /tasks/{taskid}/complete requests arrive in rapid succession, duplicate queue items are created for the same bookmark. The original implementation only deleted queue items when responses.Count > 0, causing the second queue item to remain stuck after the bookmark was already processed.

Changes

Modified BookmarkQueueProcessor.ProcessItemAsync()

• Move queue item deletion to finally block—always executes regardless of bookmark presence or exceptions
• Use CancellationToken.None for deletion to ensure cleanup during shutdown
• Add explicit exception logging for workflow resumption failures

Modified BookmarkQueueProcessor.ProcessPageAsync()

• Add try-catch wrapper to prevent one failing item from blocking batch processing

Before/After

// Before: queue item only deleted if bookmark found
var responses = await workflowResumer.ResumeAsync(filter, options, cancellationToken);
if (responses.Count > 0) {
    await store.DeleteAsync(item.Id, cancellationToken);
}

// After: always deleted, exception-safe
try {
    var responses = await workflowResumer.ResumeAsync(filter, options, cancellationToken);
    // ... logging ...
}
catch (Exception ex) {
    logger.LogError(ex, "Error resuming workflow...");
}
finally {
    try {
        await store.DeleteAsync(item.Id, CancellationToken.None);
    }
    catch (Exception ex) {
        logger.LogWarning(ex, "Failed to delete queue item...");
    }
}

Safety

The distributed lock in WorkflowResumer.ResumeAsync() ensures bookmarks are only processed once. Queue items are ephemeral markers—safe to delete after processing attempt. Existing 1-minute TTL purge task serves as fallback for any deletion failures.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Bookmarks remain stuck in database when resuming multiple workflows in rapid succession</issue_title>
<issue_description>## Description
When attempting to resume multiple bookmarks in rapid succession, several records remain stuck in the Bookmarks table and are never processed, even though the API calls to /tasks/{taskid}/complete return a successful (200 OK) response.

Steps to Reproduce

1. Setup & Execution

   • Open the attached Visual Studio project.
   • Configure your connection string in the appsettings.Development.json file of the Server project.
   • Start the Server project.
   • Log into Elsa Studio using default credentials and import the attached Workflow JSON.
   • Publish the imported workflow.
   • Configure your connection string in the appsettings.json of the ConsoleTestApp project.
   • Build the ConsoleTestApp project.
   • Ensure the Bookmarks table is currently empty.
   • Open a command prompt (CMD) in the directory containing Sample.ElsaWorkflow.ConsoleTestApp.exe.
   • Execute the following command to initialize workflows:

     Sample.ElsaWorkflow.ConsoleTestApp.exe run 200

   • Wait for completion and verify that 200 records exist in the Bookmarks table.
   • Execute the following command to resume the workflows:

     Sample.ElsaWorkflow.ConsoleTestApp.exe resume 200

   • Wait for the resume process to finish.

2. Observation

   • Upon completion, several records remain "hanging" in the Bookmarks table.
   • No errors are reported in the server logs.

Expected Behavior

All bookmarks should be successfully resumed, and the corresponding workflows should proceed to completion.

Actual Behavior

A subset of bookmarks remains stuck in the database, leaving the workflows in a permanent Suspended state.

Environment

• Elsa Package Version: 3.5.3 and 3.6.0-rc2
• Operating System: Windows 11, Windows Server 2016, Windows Server 2019.
• Database Engine: SQL Server 15 Express, SQL Server 14 Developer.
• Dispatcher: Default Workflow Dispatcher.

test-input-parent.json
Sample.ElsaWorkflow.zip
</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Bookmarks remain stuck in database when resuming multiple workflows in rapid succession #7223

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[Copilot]

Pull request overview

Addresses a race condition where rapid successive resume requests can leave duplicate bookmark queue items stuck in the queue, leading to accumulation and unprocessed workflows.

Changes:

• Wrap per-item processing in ProcessPageAsync with error handling to keep batch processing moving.
• Update ProcessItemAsync to always delete the queue item via a finally block and add logging for resume failures.

[Copilot]

ProcessPageAsync catches all exceptions and continues, which will also catch OperationCanceledException if cancellation is requested (e.g., during shutdown) and ProcessItemAsync starts propagating it. This can lead to noisy error logs and the processor continuing work even though cancellation was requested. Handle OperationCanceledException separately (rethrow) and/or break out of the loop when cancellationToken.IsCancellationRequested is true.

[Copilot]

The catch (Exception ex) here will also catch OperationCanceledException when cancellationToken is canceled, log it as an error, and then continue processing more items. This makes shutdown/cancellation noisy and prevents normal cancellation flow. Add an OperationCanceledException catch/when-filter that rethrows when cancellationToken.IsCancellationRequested, and only log unexpected exceptions.

[Copilot]

Because queue items are now always deleted, the offset-based pagination in ProcessAsync can skip items: after deleting the first page, incrementing offset causes the next query to skip records in the remaining set, potentially leaving items unprocessed until the next trigger (or even until they are purged by TTL). Consider switching the loop to always read from offset 0 until no items remain, or use cursor-based paging (e.g., last CreatedAt/Id) instead of offsets when deleting during iteration.

[Copilot]

Using CancellationToken.None for the delete means this await can block indefinitely if the underlying store call hangs (e.g., DB connectivity issues), which is especially risky during shutdown. Prefer using a bounded token (e.g., a short CancelAfter timeout) so cleanup is attempted but can't stall the processor forever.