nsqd: there seems to be a possibility that the message may be lost

[383317916]

I did an experiment. The producer sent a message to nsqd, and the consumer did not finish after receiving the message. Then close the terminal where nsqd is located. Restart nsqd again and the message will not be sent to the consumer, that is, it is lost.

It seems that when nsqd exits abnormally and persistmetadata is not executed, messages in inflight state will be lost.

version: nsqd v1.2.1-alpha (built w/go1.13)
command: command: /nsqd --lookupd-tcp-address="0.0.0.0:4160" --max-req-timeout=168h0m0s --broadcast-address=0.0.0.0 --data-path=/data/ --mem-queue-size=0

[ploxiln]

Yup, that's true. Messages in-flight are only in memory. nsqd must exit gracefully for them to be saved to disk and then retried after nsqd starts up again.

[ploxiln]

A "fix" for this would be something like: #625 (although that effort has stalled and is unlikely to resume).

The NSQ system is resilient against some types of failures, but not others. For example, it is resilient against crashing consumers. Also, messages continue to flow, even if any nsqd or nsqlookupd node disappears, through other nodes. These were trade-offs that made sense for the original creator and user of NSQ (in exchange for relative simplicity).

There is at least one fork with a very different feature set, see #887