JetStream: DeliverLastPerSubject with MaxMsgsPerSubject causes stream lock-up on sparse sequence ranges

[antonbarchukov]

Observed behavior

When creating a DeliverLastPerSubject consumer on a stream with MaxMsgsPerSubject set, if the stream has accumulated a large sequence range relative to its actual message count (common over time due to overwrites), the following happens:

• Consumer creation pins an entire vCPU core for the duration of the initial delivery
• All writes to the stream are blocked while the consumer is scanning
• nats stream info and other CLI commands time out
• The stream is effectively unusable until the consumer finishes initializing
  In our case we have ~400K live messages across a ~1.5M sequence range (first seq ~20K, last seq ~1.5M). Creating a consumer takes the stream down for the entire time it takes to linearly scan through that gap.

pprof confirms loadNextMsgLocked at ~50% cumulative CPU, with mapaccess2_fast64 (the map lookups inside the linear scan) right behind it at ~48%.

This happens because selectStartingSeqNo has a fast path for MaxMsgsPerSubject == 1 that skips building the lastSeqSkipList. Without the skip list, every message delivery goes through LoadNextMsg → loadNextMsgLocked, which iterates for nseq := fseq; nseq <= lseq doing map lookups across the entire sparse range.

We also found a secondary issue: when we patched the above locally to always build the skip list, delivery is fast (~140K msg/s) until the consumer reaches entries that were overwritten by concurrent producers during delivery.

Each stale entry causes getNextMsg to return ErrStoreMsgNotFound, which sends the consumer to waitForMsgs where it blocks on mch. It only wakes up when the next live message arrives in the stream, processes one skip list entry, and if that's also stale, goes back to sleep.

Delivery rate drops from ~140K msg/s to ~500 msg/s for the tail end of the initial delivery:
0-5s: ~140K msg/s (valid skip list entries)
5-19s: ~500 msg/s (stale entries, consumer sleeping between live publish signals)
19-20s: resumes and finishes

Without concurrent producers, delivery is smooth after building the skip list.

server/consumer.go

• Line ~5873 — selectStartingSeqNo: the mmp == 1 fast path that skips building the skip list
• Lines 4581-4599 — getNextMsg: skip list delivery that returns ErrStoreMsgNotFound on stale entries instead of looping to next
• Lines 5032-5048 — loopAndGatherMsgs: error handling that sends consumer to waitForMsgs on ErrStoreMsgNotFound
• Line 5187 — waitForMsgs select block where the consumer sleeps

server/memstore.go

• Lines 1798-1872 — loadNextMsgLocked: the linear scan for nseq := fseq; nseq <= lseq that pins CPU
• Lines 789-806 — allLastSeqsLocked: the efficient alternative that iterates the subject tree (O(N subjects) + sort)
• Lines 1987-2016 — removeSeqPerSubject: the write path that Terrible performance when catching up on a stream with large sequence number gap and MaxMsgPerSubject=1 #6687 fixed

Expected behavior

Consumer creation should not block writes to the stream or cause CLI timeouts. Delivery of initial state should scale with message count, not sequence range. Stale skip list entries should be skipped over immediately rather than causing the consumer to sleep.

Server and client version

nats-server built from main branch

Host environment

8-core ARM64 (AWS c7g.2xlarge, Graviton 3)
In-memory store (memstore)
Stream config: MaxMsgsPerSubject=1, ~400K subjects, ~1-2M sequence range

Tested with both raw streams using MaxMsgsPerSubject and KV buckets with History=1 — same behavior

Steps to reproduce

1. Create a stream with MaxMsgsPerSubject set (or a KV bucket with any history value)
2. Publish to many subjects repeatedly over time so the sequence range grows well beyond the live message count. We used a script that publishes to 400K unique subjects (though as this proved later, doesn't quite matter as much as the sheer number of sequencing numbers) to inflate the sequence range to ~1-2M while retaining ~400K live messages, but this also happens naturally over hours/days of production use.
3. Create a DeliverLastPerSubject consumer (or run nats kv watch) — stream locks up, writes block, CLI times out
4. pprof CPU profile shows loadNextMsgLocked pinning a core

Related: #6687 (fixed the write path in removeSeqPerSubject but the consumer read path still has the same class of issue)

Let me know if pprofs are useful, I can share in thread/pm.

[antonbarchukov]

antonbarchukov@a9b98b2

pushed to own fork for reference for impacted code.

[projector1z]

Bump

[sciascid]

@antonbarchukov thanks for reporting this issue!
I confirm: loadNextMsgLocked performs excessive map look ups. However, I'm not sure the suggested change is the correct fix. I found that there are cases where loadNextMsgLocked extends fseq and lseq unnecessarily. This will require some more investigation.

[antonbarchukov]

@sciascid Thanks for taking a look at this. Please let me know what you find.

[sciascid]

@antonbarchukov the proposed fix in selectStartingSeqNo will likely introduce performance regressions in other scenarios. For example, consider a stream with a large number of distinct subjects that is densely populated and configured with MaxMsgsPerSubject = 1. If you create a simple consumer without filtering, the upfront construction of the skip list is unnecessary overhead. In the worst case, building this list requires traversing the entire subject tree, collecting the last sequence number for each subject into a slice, and then sorting that slice.

[antonbarchukov]

@sciascid this is the exact scenario that we currently face. and building that skip-list even with the overhead of filtering across 400K-800K subjects is the only way we were able to get consumer creation and a full dump of a stream in under 5 seconds, versus previously the consumer creation alone locked up the stream for publishes and consumer creation by itself took over a minute.

maybe I'm misunderstanding the process, but the skip-list IS needed where there is a large sequence gap (regardless of number of subjects), otherwise the consumer is checking dead sequence numbers, no?

[antonbarchukov]

@sciascid misread the "densely" part;

perhaps there should be a way to do this conditionally, where a skip-list is built only for a large sequence gap (in our testing this proved to be around 400K-800K across 400K-600K subjects.

not fully sure what that condition should actually be or how that would even be measured, but there should be a way to accurately measure that gain vs regression intersection with direct testing, or perhaps another way to do that.

[sciascid]

@antonbarchukov a proper solution would take into account the ratio #messages / (last - first sequence), type of filters, memory / filestore, and maybe other things as well.

However, I think I also identified a bug in LoadNextMsg which causes excessive scanning in some cases. Not sure this applies to your case.

What subjects and filters are you using in the stream and consumer in your case?

[antonbarchukov]

@sciascid further looking at this,

the dense case regression that you mention, would imply a large subject count with all subjects loaded, and never updated, and where only the initial load performance matters. at which point, you might as well just do a DeliverAll, not DeliverLastPerSubject.

however, this doesn't account for anything that happens after those messages are loaded (KV in real world with mmp 1 and overwrites to them), if a fraction of your large initial subjects is updated, you get where we essentially require said skip list, as the first sequence value starts at the stalest message.