-
-
Notifications
You must be signed in to change notification settings - Fork 295
Description
Describe the bug
Durability agents for statically registered tenants in a multi-tenant PostgreSQL environment do not appear to start or process messages. While messages are correctly persisted to the tenant-specific wolverine_incoming_envelopes or wolverine_dead_letters tables (proving the Outbox/Inbox persistence works), they are never picked up for retries, scheduled execution, or Dead Letter Queue (DLQ) replays. These background tasks only function for the database designated as the "Main" message store.
To Reproduce
Steps to reproduce the behavior:
- Use Wolverine version
5.12.1with PostgreSQL persistence. - Configure a "master" database for coordination:
options.PersistMessagesWithPostgresql(masterConnection, schemaName: "wolverine"). - Register multiple static tenants with their own connection strings:
.RegisterStaticTenants(tenants => { tenants.Register("tenant-1", "Host=localhost;Port=5433;..."); tenants.Register("tenant-2", "Host=localhost;Port=5434;..."); })
- Enable durable policies:
options.Policies.UseDurableInboxOnAllListeners()andoptions.Policies.UseDurableOutboxOnAllSendingEndpoints(). - Send a message via
bus.SendAsyncfortenant-1that intentionally fails in the handler. - The message correctly lands in
tenant-1'swolverine_dead_letterstable. - Attempt to replay the message (e.g., using
MapDeadLettersEndpointsor manually setting it to replyable). - Observe that the message remains in the database indefinitely; the durability agent for
tenant-1never processes it. - Change the configuration so that
tenant-1's connection string is used as themasterConnection(Main store). - Restart the application and observe that the message is now successfully picked up and processed.
Expected behavior
Wolverine should start a Durability Agent for every registered tenant. These agents should independently monitor their respective tenant databases to handle scheduled messages, retries, and DLQ replays, regardless of which database is acting as the master coordination node.
Desktop (please complete the following information):
- OS: Windows
- .NET Version: 9.0
- Wolverine Version: 5.12.1
- Database: PostgreSQL
Additional context
- Crucial Infrastructure Detail: Each tenant uses a completely separate PostgreSQL database (Docker container) (distinct connection strings, with different ports).
- Scheduled Messages: The issue is identical for scheduled messages. They are correctly persisted to the tenant's database but are never "woken up" and processed because the durability agent for that tenant is not running.
- It appears that the durability agent assignment logic is failing to "see" or assign responsibilities for the tenant-specific message stores.
- Outbox/Inbox writing works perfectly across all tenants; only the background "durability" processing is missing.
- When a tenant database is temporarily promoted to be the "Master" database in the configuration, its durability agent starts working immediately, confirming that the issue is related to the multi-tenancy assignment.
Simplified configuration
builder.Host.UseWolverine(options =>
{
var masterConnection = builder.Configuration.GetConnectionString("wolverine")
?? throw new Exception("wolverine connection string not found");
var connectionStrings = builder.Configuration
.GetSection("ConnectionStrings")
.GetChildren()
.Where(connectionString => connectionString.Key != "wolverine")
.ToList();
options.PersistMessagesWithPostgresql(masterConnection, schemaName: "wolverine")
.RegisterStaticTenants(tenants =>
{
foreach (var cs in connectionStrings)
{
tenants.Register(cs.Key, cs.Value!);
}
});
options.Services.AddDbContextWithWolverineManagedMultiTenancy<CompanyDbContext>((b, cs, _) =>
{
b.UseNpgsql(cs.Value, x =>
{
x.MigrationsAssembly("Company.Infrastructure");
x.MigrationsHistoryTable("__EFMigrationsHistory", "Company");
});
}, AutoCreate.CreateOrUpdate);
options.UseEntityFrameworkCoreTransactions();
options.Policies.AutoApplyTransactions();
options.Policies.UseDurableOutboxOnAllSendingEndpoints();
options.Policies.UseDurableInboxOnAllListeners();
options.Policies.UseDurableLocalQueues();
options.Discovery.IncludeAssembly(typeof(ApplicationAssemblyMarker).Assembly);
});