@effect/workflow: Effect.all with concurrency > 1 wrapping Activity.make causes deadlock during replay

[Necmttn]

What version of Effect is running?

• effect: 3.19.15
• @effect/workflow: 0.16.0
• @effect/platform: 0.94.2
• @effect/cluster: 0.56.1
• Runtime: Bun 1.3.0

What steps can reproduce the bug?

When a workflow uses Effect.all(activities, { concurrency: N }) where N > 1 and each element is an Activity.make(...), the workflow completes successfully on first execution but deadlocks during durable replay (e.g. when the workflow engine re-executes the workflow to recover state).

Minimal reproduction pattern:

import { Activity, Workflow } from "@effect/workflow";
import { Effect, Schema, Schedule } from "effect";

const MyWorkflow = Workflow.make({
  name: "DeadlockRepro",
  payload: Schema.Struct({ items: Schema.Array(Schema.String) }),
  success: Schema.String,
  execute: ({ items }) =>
    Effect.gen(function* () {
      // Create an array of activities, one per item
      const activities = items.map((item) =>
        Activity.make({
          name: `process:${item}`,
          success: Schema.String,
          error: Schema.String,
          execute: Effect.succeed(`done:${item}`),
        })
      );

      // This works with concurrency: 1, deadlocks with concurrency > 1 during replay
      const results = yield* Effect.all(activities, { concurrency: 3 });
      return results.join(",");
    }),
});

Steps to reproduce:

1. Execute the workflow with items: ["a", "b", "c", "d", "e"]
2. All 5 activities complete successfully on first execution
3. Trigger a replay (e.g. restart the workflow engine, or let durable execution recovery kick in)
4. During replay, the workflow hangs indefinitely — activities never resume

The same Effect.all({ concurrency: 3 }) pattern works perfectly outside of @effect/workflow (i.e. without Activity.make), confirming the issue is in the Activity replay mechanism interacting with concurrent suspension.

What is the expected behavior?

Effect.all(activities, { concurrency: N }) where N > 1 should work correctly both during first execution and during replay. Activities should be replayed from the durable log and the concurrent combinator should resume all branches.

What do you see instead?

The workflow hangs indefinitely during replay. No error is thrown, no timeout fires at the workflow level — the activities simply never complete. The workflow fiber appears to be suspended waiting for activity results that are never replayed.

With concurrency: 1 (sequential execution), the same workflow replays correctly.

Additional information

History:

• Designed concurrency: 3 to match provider API rate limits
• Upgraded to @effect/workflow 0.15.2 which included PR ensure no more Activites are attempted before suspending #5880 ("ensure no more Activities are attempted before suspending") — believed to fix this
• Upgraded to 0.16.0, restored concurrency: 3 — deadlock persisted during replay
• Reverted to concurrency: 1 as workaround

Investigation (2026-01-30): Created a standalone test that runs the same provider layer stack (CredentialService + RateLimitedHttpClient + NotionProvider) with Effect.all({ concurrency: 3 }) but without any @effect/workflow or Activity machinery. Result: 14/14 pages fetched successfully in both sequential and concurrent modes. No hang. This confirms the service layer is not the cause — the deadlock is specifically in the Activity replay mechanism when combined with concurrent Effect.all.

Workaround: Use concurrency: 1 (sequential) for all Effect.all/Effect.forEach calls that wrap Activity.make. This avoids the deadlock but sacrifices throughput.

Related: #5876 (different symptom — unhandled error logging — but same @effect/workflow Activity area)

[Necmttn]

Follow-up: reproduction attempt results

I wrote extensive reproduction scripts testing multiple scenarios with @effect/workflow@0.16.0:

Test 1: First execution with Effect.all({ concurrency: 3 }) + 5 activities

• Using ClusterWorkflowEngine.layer + TestRunner.layer
• Result: PASS — all activities complete, no deadlock

Test 2: Same idempotency key re-execution (cached result path)

• Both WorkflowEngine.layerMemory and ClusterWorkflowEngine + TestRunner
• Result: PASS — second execution returns cached result instantly

Test 3: DurableDeferred suspend → resume → replay with concurrent activities

• Phase 1: 5 concurrent activities complete
• Workflow suspends on DurableDeferred.await
• External signal via DurableDeferred.succeed triggers resume
• Engine re-runs workflow: phase 1 activities replayed from storage (no re-execution), deferred resolves from storage, phase 3 activities execute fresh
• Result: PASS — replay works correctly

Conclusion: The deadlock does not reproduce with TestRunner.layer and simple activities. In our production environment (Bun 1.3.0, SingleRunner.layer({ runnerStorage: "memory" }), real Notion API calls taking 300ms-10s each, rate limiter at 3 req/s), the hang is intermittent and occurs when workflows run for extended periods with many activities.

The production pattern has activities inside Effect.iterate (paged batches), each activity wrapped in Effect.retry({ times: 3 }), and the activities perform real HTTP calls with rate limiting. The timing-dependent nature suggests the issue may involve:

1. The wrapActivityResult latch interacting with activities that take varying durations
2. The cluster engine's activityExecute retry loop (line 305-334 in ClusterWorkflowEngine.js) when an activity entity returns Suspended before the activity handler is registered
3. Fiber scheduling under heavy async I/O load

Our workaround (concurrency: 1) remains effective. The investigation confirmed that the service layer (CredentialService, RateLimitedHttpClient, NotionProvider) works correctly under concurrency: 3 without the workflow engine — only when activities run through the workflow engine does the hang occur.

[Necmttn]

Follow-up #2: Exhaustive reproduction attempts — unable to reproduce

Built increasingly faithful reproductions mirroring the exact production pattern. All pass with concurrency: 3.

What was tested

Test	Storage	Replay	Registered workflows	Activity latency	Result
Flat Effect.all concurrent	In-memory (TestRunner)	No	1	20ms	✅ PASS
Effect.iterate + concurrent activities	In-memory	No	1	20ms	✅ PASS
+ Effect.retry wrapping Activity.make	In-memory	No	1	20ms	✅ PASS
+ Effect.catchTag → null + addFinalizer + health-check Activity	In-memory	No	1	20ms	✅ PASS
+ Varying latency (50–500ms)	In-memory	No	1	50–500ms	✅ PASS
Full pattern, SQL-backed	PostgreSQL (SingleRunner)	No	1	50–500ms	✅ PASS
Suspend → resume → replay	PostgreSQL	Yes	1	50–500ms	✅ PASS
15 workflows registered, 14 pages, 200ms–2s	PostgreSQL	No	15	200–2000ms	✅ PASS

The "full pattern" variants mirror our production workflow exactly:

// Sequential health-check activity
yield* Activity.make({ name: "healthcheck", ... });

// Effect.iterate with batched concurrent activities
yield* Effect.iterate(state, {
  while: (s) => !s.done,
  body: (state) => Effect.gen(function* () {
    const batch = pages.slice(state.offset, state.offset + BATCH_SIZE);
    const results = yield* Effect.all(
      batch.map((page) =>
        Activity.make({ name: `page:${page}:${state.iteration}`, ... }).pipe(
          Effect.retry({ times: 3, schedule: Schedule.exponential("1 second") }),
          Effect.catchTag("MyError", () => Effect.succeed(null)),
        ),
      ),
      { concurrency: 3 },
    );
    // ...
  }),
});

The suspend→resume→replay test forces the exact path we believed triggers the deadlock: phase 1 activities complete → DurableDeferred.await suspends → external signal resumes → engine replays all phase 1 activities from SQL storage → phase 3 activities execute fresh. This completed successfully.

The multi-workflow test registers 15 workflows (matching our production WorkflowsLayer) in a single SingleRunner backed by PostgreSQL, with realistic per-activity latency. Also passes.

What was NOT tested

• Real HTTP I/O inside activities (our activities do Notion API calls taking 300ms–10s each, with rate limiting)
• Pod restart / crash mid-execution followed by replay in a new process
• Concurrent workflow executions of the same workflow type (multiple Notion syncs running simultaneously)

Revised assessment

The deadlock is not caused by any code pattern we use (iterate, retry, catchTag, finalizer, concurrent Effect.all with Activity.make). It also doesn't require SQL-backed storage or explicit suspend/resume/replay to trigger.

This leaves a few possibilities:

1. The hang only manifests with real async I/O inside activities (actual HTTP calls, not Effect.sleep)
2. A process restart mid-execution creates a storage state that triggers the deadlock during replay in the new process
3. The issue was fixed in a recent @effect/workflow update and we observed stale behavior from prior runs

Our concurrency: 1 workaround remains in place. We'll test with concurrency: 3 against the real production workflow next. If we can reproduce it there, we'll bisect to isolate the exact trigger.

[Necmttn]

Update 3: Real workflow confirms hang — service layer isolated as safe

Test setup

Ran the real production NotionContentFetch workflow with CONCURRENCY=3 against the actual Notion API (14 pages, real HTTP I/O, real PostgreSQL-backed SqlMessageStorage).

Result: Workflow hangs reliably

Run 1 (with stale PG cluster_messages from previous runs):

• Two parallel workflow executions started simultaneously (stale messages caused duplicate entity creation)
• 6 pages fetched successfully in ~2s
• Workflow went completely silent after starting 3 more concurrent fetches
• "No healthy runners available" warnings started ~2 minutes later — workflow engine dead

Run 2 (clean PG state — all cluster_messages deleted):

• Single workflow execution (no duplicates)
• 6 pages completed in ~6s
• 3 concurrent page fetches started (pages 80ac, 80bc, 80c3) — never completed
• Workflow went completely silent, same "No healthy runners" pattern

Isolation: Service layer works perfectly

Ran the same 14 pages with same concurrency=3 using Effect.all({ concurrency: 3 }) directly — no @effect/workflow, no Activity.make, just NotionProvider.getPageContent():

Mode	Pages	Result	Time
Sequential	14/14	✅ All succeeded	20,454ms
Concurrent (3)	14/14	✅ All succeeded	7,857ms

All pages including the largest (18,001 chars, 183 lines, recursive blocks to depth 4) complete fine.

Conclusion

• The hang is definitively in @effect/workflow's Activity machinery when combined with real I/O
• Synthetic reproductions with Effect.sleep() as the Activity body all pass — the issue requires real async I/O (HTTP requests taking 300-8000ms) to trigger
• Our CredentialService, RateLimitedHttpClient, and NotionProvider are all safe under concurrent access
• Workaround: CONCURRENCY=1 remains correct until this is fixed upstream

Additional finding: stale cluster_messages

When a runner process restarts, it picks up unprocessed messages from cluster_messages (PostgreSQL) for entities from previous runner instances. This causes old workflow executions to resume simultaneously alongside new ones, compounding the issue. This is a separate concern but worth noting — it may indicate that workflow entity garbage collection / message expiry is needed.

[Necmttn]

Update 4: Workaround applied — takeaways

Workaround

We shipped the workaround described in Update 2: single Activity per batch iteration, concurrency inside the execute body.

Instead of:

// DEADLOCKS during replay
yield* Effect.all(
  pages.map(page => Activity.make({ name: `fetch:${page.id}`, ... })),
  { concurrency: 3 }
)

We use:

// Works — one Activity, concurrency inside execute body
yield* Activity.make({
  name: `batch:${iteration}`,
  execute: Effect.gen(function* () {
    return yield* Effect.all(
      pages.map(page => fetchAndSave(page)),
      { concurrency: 3 }
    )
  }),
})

The trade-off: if the Activity fails mid-execution, all pages in the batch are retried (no per-page durability). SurrealDB status tracking (pending_content → pending_chunks → processed) provides natural idempotency, so replayed pages that already succeeded are cheap no-ops.

Additional finding: child workflow execution from Activity deadlocks too

During the same investigation, we discovered a related pattern that also deadlocks:

// Parent workflow
yield* Activity.make({
  name: "derive",
  execute: Effect.gen(function* () {
    // This launches child workflows via the cluster runner
    yield* ChildWorkflow.execute({ ... })  // ← hangs forever
  }),
})

Calling Workflow.execute() from inside an Activity.make execute body blocks the runner from processing the child workflow. The fix is to call child workflows directly from the parent workflow body (not inside an Activity).

Summary of patterns to avoid with @effect/workflow

1. Effect.all({ concurrency > 1 }) wrapping Activity.make calls — deadlocks during replay
2. Workflow.execute() inside Activity.make execute body — deadlocks (runner can't process children while Activity is pending)
3. Long-running operations inside Activity.make execute body — works but consider that the entire execute body is atomic for replay purposes

[tim-smart]

The reproduction isn't valid code. Do you have one that compiles?