Queue crash due to bad filename

[embedded-bed]

Describe the bug

Our client should consume messages with a few 100ms interval, but sometimes it can take a couple of seconds between it's ready to consume a message. I think this causes RabbitMQ to save the message to a file on disk.

However, on some occurrences the file appears to get a garbage name that results in a error,enametoolong crash of the queue.

I have consulted documentation and searched the web, but have not found any other similar issues.

RabbitMQ runs in a docker container based on the image rabbitmq:4.2.2-alpine and the host OS is Debian 13.1.
Here is the error log:

[error] <0.753.0> closing AMQP connection <0.753.0> (172.18.0.20:56068 -> 172.18.0.3:5671, duration: '1H, 2M, 0s'):
[error] <0.753.0> missed heartbeats from client, timeout: 60s
[error] <0.671.0> ** Generic server <0.671.0> terminating
[error] <0.671.0> ** Last message in was {'$gen_cast',
[error] <0.671.0>                         {deliver,
[error] <0.671.0>                          {delivery,false,false,<0.7683.0>,
[error] <0.671.0>                           {mc,mc_amqpl,
[error] <0.671.0>                            {content,60,none,
[error] <0.671.0>                             <<0,0>>,
[error] <0.671.0>                             rabbit_framing_amqp_0_9_1,
[error] <0.671.0>                             [<<128,4,149,56,0,0,0,0,0,0,0,125,148,40,140,7,
[error] <0.671.0>                                97,122,105,109,117,116,104,148,71,64,89,71,
[error] <0.671.0>                                222,128,186,232,180,140,4,116,105,108,116,148,
[error] <0.671.0>                                71,63,242,127,251,92,97,95,14,140,4,122,111,
[error] <0.671.0>                                111,109,148,71,0,0,0,0,0,0,0,0,117,46>>]},
[error] <0.671.0>                            #{id =>
[error] <0.671.0>                               <<13,168,128,211,29,137,103,244,102,231,238,168,
[error] <0.671.0>                                 75,51,156,207>>,
[error] <0.671.0>                              d => false,x => <<"camera.update">>,
[error] <0.671.0>                              rk => [<<>>],
[error] <0.671.0>                              rts => 1769136282053}},
[error] <0.671.0>                           undefined,flow},
[error] <0.671.0>                          false}}
[error] <0.671.0> ** When Server state == {q,
[error] <0.671.0>                          {amqqueue,
[error] <0.671.0>                           {resource,<<"/">>,queue,
[error] <0.671.0>                            <<"consumer-client-channels-camera.update">>},
[error] <0.671.0>                           false,false,<0.595.0>,
[error] <0.671.0>                           [{<<"x-max-length">>,signedint,1},
[error] <0.671.0>                            {<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.671.0>                           <0.671.0>,[],[],[],undefined,undefined,[],[],live,0,
[error] <0.671.0>                           [],<<"/">>,
[error] <0.671.0>                           #{user => <<"O=client,CN=consumer-client">>},
[error] <0.671.0>                           rabbit_classic_queue,#{}},
[error] <0.671.0>                          none,true,rabbit_priority_queue,
[error] <0.671.0>                          {passthrough,rabbit_variable_queue,
[error] <0.671.0>                           {vqstate,
[error] <0.671.0>                            {0,[],[]},
[error] <0.671.0>                            {0,[],[]},
[error] <0.671.0>                            {delta,undefined,0,0,undefined},
[error] <0.671.0>                            {1,[],
[error] <0.671.0>                             [{msg_status,8683,
[error] <0.671.0>                               <<244,7,220,219,70,155,28,28,86,186,251,59,45,
[error] <0.671.0>                                 212,114,103>>,
[error] <0.671.0>                               {mc,mc_amqpl,
[error] <0.671.0>                                {content,60,none,
[error] <0.671.0>                                 <<0,0>>,
[error] <0.671.0>                                 rabbit_framing_amqp_0_9_1,
[error] <0.671.0>                                 [<<128,4,149,56,0,0,0,0,0,0,0,125,148,40,140,
[error] <0.671.0>                                    7,97,122,105,109,117,116,104,148,71,64,88,
[error] <0.671.0>                                    214,250,113,14,18,60,140,4,116,105,108,
[error] <0.671.0>                                    116,148,71,63,249,250,241,211,248,6,222,
[error] <0.671.0>                                    140,4,122,111,111,109,148,71,0,0,0,0,0,0,
[error] <0.671.0>                                    0,0,117,46>>]},
[error] <0.671.0>                                #{id =>
[error] <0.671.0>                                   <<244,7,220,219,70,155,28,28,86,186,251,59,
[error] <0.671.0>                                     45,212,114,103>>,
[error] <0.671.0>                                  d => false,x => <<"camera.update">>,
[error] <0.671.0>                                  rk => [<<>>],
[error] <0.671.0>                                  rts => 1769136281046}},
[error] <0.671.0>                               false,false,memory,false,queue_store,
[error] <0.671.0>                               {message_properties,undefined,false,67}}]},
[error] <0.671.0>                            {0,[],[]},
[error] <0.671.0>                            8684,8683,
[error] <0.671.0>                            #{8682 =>
[error] <0.671.0>                               {msg_status,8682,
[error] <0.671.0>                                <<137,171,120,241,217,211,174,188,107,79,110,
[error] <0.671.0>                                  123,123,110,137,92>>,
[error] <0.671.0>                                {mc,mc_amqpl,
[error] <0.671.0>                                 {content,60,none,
[error] <0.671.0>                                  <<0,0>>,
[error] <0.671.0>                                  rabbit_framing_amqp_0_9_1,
[error] <0.671.0>                                  [<<128,4,149,56,0,0,0,0,0,0,0,125,148,40,
[error] <0.671.0>                                     140,7,97,122,105,109,117,116,104,148,71,
[error] <0.671.0>                                     64,88,189,8,240,216,247,192,140,4,116,
[error] <0.671.0>                                     105,108,116,148,71,63,249,68,50,143,134,
[error] <0.671.0>                                     220,229,140,4,122,111,111,109,148,71,0,0,
[error] <0.671.0>                                     0,0,0,0,0,0,117,46>>]},
[error] <0.671.0>                                 #{id =>
[error] <0.671.0>                                    <<137,171,120,241,217,211,174,188,107,79,
[error] <0.671.0>                                      110,123,123,110,137,92>>,
[error] <0.671.0>                                   d => false,x => <<"camera.update">>,
[error] <0.671.0>                                   rk => [<<>>],
[error] <0.671.0>                                   rts => 1769136280034}},
[error] <0.671.0>                                false,true,memory,false,queue_store,
[error] <0.671.0>                                {message_properties,undefined,false,67}}},
[error] <0.671.0>                            #{},undefined,undefined,
[error] <0.671.0>                            {qi,
[error] <0.671.0>                             {resource,<<"/">>,queue,
[error] <0.671.0>                              <<"consumer-client-channels-camera.update">>},
[error] <0.671.0>                             <<"/var/lib/rabbitmq/mnesia/rabbit@43cc534806f8/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/CPN5X92SW3PPUQXAS40XY2AHD/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////">>,
[error] <0.671.0>                             #{},0,#{},#{},#{},#{},
[error] <0.671.0>                             #Fun<rabbit_variable_queue.2.128175942>,
[error] <0.671.0>                             #Fun<rabbit_variable_queue.3.128175942>},
[error] <0.671.0>                            {qs,
[error] <0.671.0>                             <<"/var/lib/rabbitmq/mnesia/rabbit@43cc534806f8/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/CPN5X92SW3PPUQXAS40XY2AHD/">>,
[error] <0.671.0>                             undefined,64,#{},0,#{},undefined,undefined},
[error] <0.671.0>                            {undefined,
[error] <0.671.0>                             {client_msstate,<0.441.0>,
[error] <0.671.0>                              <<250,231,179,214,190,144,236,69,119,220,190,11,
[error] <0.671.0>                                206,8,17,125>>,
[error] <0.671.0>                              undefined,#Ref<0.3681152174.3998089221.258093>,
[error] <0.671.0>                              <<"/var/lib/rabbitmq/mnesia/rabbit@43cc534806f8/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient">>,
[error] <0.671.0>                              #Ref<0.3681152174.3998089221.258094>,
[error] <0.671.0>                              #Ref<0.3681152174.3998089221.258095>,
[error] <0.671.0>                              #Ref<0.3681152174.3998089221.258096>,
[error] <0.671.0>                              {4000,800}}},
[error] <0.671.0>                            false,0,4096,1,67,67,0,0,0,infinity,1,0,0,134,3,4,
[error] <0.671.0>                            {rates,0.12948930171032805,0.12948930170978792,
[error] <0.671.0>                             0.12948930170978792,0.13002103955524963,
[error] <0.671.0>                             -576409845894912912},
[error] <0.671.0>                            #{},#{},#{},#{},2,3,103,112,0,default,2,#{},
[error] <0.671.0>                            <<"/">>,false}},
[error] <0.671.0>                          {state,
[error] <0.671.0>                           {queue,[],[],0},
[error] <0.671.0>                           {inactive,-576409831920158,906448,
[error] <0.671.0>                            0.9980147361170485}},
[error] <0.671.0>                          undefined,undefined,
[error] <0.671.0>                          {erlang,#Ref<0.3681152174.4117757955.189418>},
[error] <0.671.0>                          undefined,
[error] <0.671.0>                          {state,none,5000,undefined},
[error] <0.671.0>                          #{},undefined,undefined,undefined,
[error] <0.671.0>                          {state,
[error] <0.671.0>                           #{<0.7683.0> =>
[error] <0.671.0>                              #Ref<0.3681152174.4005298177.188558>},
[error] <0.671.0>                           delegate},
[error] <0.671.0>                          undefined,undefined,1,undefined,'drop-head',1,0,
[error] <0.671.0>                          running,false}
[error] <0.671.0> ** Reason for termination ==
[error] <0.671.0> ** {{case_clause,{error,enametoolong}},
[error] <0.671.0>     [{rabbit_file,open_eventually,3,[{file,"rabbit_file.erl"},{line,361}]},
[error] <0.671.0>      {rabbit_classic_queue_index_v2,new_segment_file,3,
[error] <0.671.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,610}]},
[error] <0.671.0>      {rabbit_classic_queue_index_v2,publish,8,
[error] <0.671.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,599}]},
[error] <0.671.0>      {rabbit_variable_queue,maybe_write_index_to_disk,3,
[error] <0.671.0>          [{file,"rabbit_variable_queue.erl"},{line,1847}]},
[error] <0.671.0>      {rabbit_variable_queue,publish1,6,
[error] <0.671.0>          [{file,"rabbit_variable_queue.erl"},{line,1707}]},
[error] <0.671.0>      {rabbit_variable_queue,publish,5,
[error] <0.671.0>          [{file,"rabbit_variable_queue.erl"},{line,537}]},
[error] <0.671.0>      {rabbit_priority_queue,publish,5,
[error] <0.671.0>          [{file,"rabbit_priority_queue.erl"},{line,213}]},
[error] <0.671.0>      {rabbit_amqqueue_process,deliver_or_enqueue,3,
[error] <0.671.0>          [{file,"rabbit_amqqueue_process.erl"},{line,824}]}]}
[error] <0.671.0> 
[error] <0.671.0>   crasher:
[error] <0.671.0>     initial call: rabbit_amqqueue_process:init/1
[error] <0.671.0>     pid: <0.671.0>
[error] <0.671.0>     registered_name: []
[error] <0.671.0>     exception exit: {{case_clause,{error,enametoolong}},
[error] <0.671.0>                      [{rabbit_file,open_eventually,3,
[error] <0.671.0>                           [{file,"rabbit_file.erl"},{line,361}]},
[error] <0.671.0>                       {rabbit_classic_queue_index_v2,new_segment_file,3,
[error] <0.671.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.671.0>                            {line,610}]},
[error] <0.671.0>                       {rabbit_classic_queue_index_v2,publish,8,
[error] <0.671.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.671.0>                            {line,599}]},
[error] <0.671.0>                       {rabbit_variable_queue,maybe_write_index_to_disk,3,
[error] <0.671.0>                           [{file,"rabbit_variable_queue.erl"},{line,1847}]},
[error] <0.671.0>                       {rabbit_variable_queue,publish1,6,
[error] <0.671.0>                           [{file,"rabbit_variable_queue.erl"},{line,1707}]},
[error] <0.671.0>                       {rabbit_variable_queue,publish,5,
[error] <0.671.0>                           [{file,"rabbit_variable_queue.erl"},{line,537}]},
[error] <0.671.0>                       {rabbit_priority_queue,publish,5,
[error] <0.671.0>                           [{file,"rabbit_priority_queue.erl"},{line,213}]},
[error] <0.671.0>                       {rabbit_amqqueue_process,deliver_or_enqueue,3,
[error] <0.671.0>                           [{file,"rabbit_amqqueue_process.erl"},{line,824}]}]}
[error] <0.671.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1174)
[error] <0.671.0>     ancestors: [<0.670.0>,<0.454.0>,<0.432.0>,<0.431.0>,
[error] <0.671.0>                   rabbit_vhost_sup_sup,rabbit_sup,<0.209.0>]
[error] <0.671.0>     message_queue_len: 0
[error] <0.671.0>     messages: []
[error] <0.671.0>     links: [<0.670.0>]
[error] <0.671.0>     dictionary: [{rand_seed,
[error] <0.671.0>                       {#{max => 288230376151711743,type => exsplus,
[error] <0.671.0>                          next => #Fun<rand.5.40079776>,
[error] <0.671.0>                          jump => #Fun<rand.3.40079776>},
[error] <0.671.0>                        [168873703042841010|119473941416470472]}},
[error] <0.671.0>                   {{ch,<0.604.0>},
[error] <0.671.0>                    {cr,<0.604.0>,#Ref<0.3681152174.3997958145.261145>,
[error] <0.671.0>                        {1,
[error] <0.671.0>                         [{8682,<<"ctag1.0f5b8ce6cda34df687b5d18be6cc5ff9">>}],
[error] <0.671.0>                         []},
[error] <0.671.0>                        1,
[error] <0.671.0>                        {queue,
[error] <0.671.0>                            [{<0.604.0>,
[error] <0.671.0>                              {consumer,
[error] <0.671.0>                                  <<"ctag1.0f5b8ce6cda34df687b5d18be6cc5ff9">>,
[error] <0.671.0>                                  true,1,[],
[error] <0.671.0>                                  <<"O=client,CN=consumer-client">>}}],
[error] <0.671.0>                            [],1},
[error] <0.671.0>                        {qstate,<0.603.0>,dormant,
[error] <0.671.0>                            {1,
[error] <0.671.0>                             {<<"ctag1.0f5b8ce6cda34df687b5d18be6cc5ff9">>,
[error] <0.671.0>                              {credit,0,auto},
[error] <0.671.0>                              nil,nil}}},
[error] <0.671.0>                        45,#{}}},
[error] <0.671.0>                   {'$logger_metadata$',#{domain => [rabbitmq,queue]}},
[error] <0.671.0>                   {process_name,
[error] <0.671.0>                       {rabbit_amqqueue_process,
[error] <0.671.0>                           {resource,<<"/">>,queue,
[error] <0.671.0>                               <<"consumer-client-channels-camera.update">>}}},
[error] <0.671.0>                   {guid,{{4209488854,3197168709,2010955275,3456635261},0}},
[error] <0.671.0>                   {credit_blocked,[]},
[error] <0.671.0>                   {{credit_to,<0.7683.0>},132}]
[error] <0.671.0>     trap_exit: true
[error] <0.671.0>     status: running
[error] <0.671.0>     heap_size: 28690
[error] <0.671.0>     stack_size: 29
[error] <0.671.0>     reductions: 74957994
[error] <0.671.0>   neighbours:
[error] <0.671.0> 
[error] <0.670.0>     supervisor: {<0.670.0>,rabbit_amqqueue_sup}
[error] <0.670.0>     errorContext: child_terminated
[error] <0.670.0>     reason: {{case_clause,{error,enametoolong}},
[error] <0.670.0>              [{rabbit_file,open_eventually,3,
[error] <0.670.0>                   [{file,"rabbit_file.erl"},{line,361}]},
[error] <0.670.0>               {rabbit_classic_queue_index_v2,new_segment_file,3,
[error] <0.670.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,610}]},
[error] <0.670.0>               {rabbit_classic_queue_index_v2,publish,8,
[error] <0.670.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,599}]},
[error] <0.670.0>               {rabbit_variable_queue,maybe_write_index_to_disk,3,
[error] <0.670.0>                   [{file,"rabbit_variable_queue.erl"},{line,1847}]},
[error] <0.670.0>               {rabbit_variable_queue,publish1,6,
[error] <0.670.0>                   [{file,"rabbit_variable_queue.erl"},{line,1707}]},
[error] <0.670.0>               {rabbit_variable_queue,publish,5,
[error] <0.670.0>                   [{file,"rabbit_variable_queue.erl"},{line,537}]},
[error] <0.670.0>               {rabbit_priority_queue,publish,5,
[error] <0.670.0>                   [{file,"rabbit_priority_queue.erl"},{line,213}]},
[error] <0.670.0>               {rabbit_amqqueue_process,deliver_or_enqueue,3,
[error] <0.670.0>                   [{file,"rabbit_amqqueue_process.erl"},{line,824}]}]}
[error] <0.670.0>     offender: [{pid,<0.671.0>},
[error] <0.670.0>                {id,rabbit_amqqueue},
[error] <0.670.0>                {mfargs,
[error] <0.670.0>                    {rabbit_amqqueue_process,start_link,
[error] <0.670.0>                        [{amqqueue,
[error] <0.670.0>                             {resource,<<"/">>,queue,
[error] <0.670.0>                                 <<"consumer-client-channels-camera.update">>},
[error] <0.670.0>                             false,false,<0.595.0>,
[error] <0.670.0>                             [{<<"x-max-length">>,signedint,1},
[error] <0.670.0>                              {<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.670.0>                             none,[],[],[],undefined,undefined,[],[],live,0,[],
[error] <0.670.0>                             <<"/">>,
[error] <0.670.0>                             #{user => <<"O=client,CN=consumer-client">>},
[error] <0.670.0>                             rabbit_classic_queue,#{}},
[error] <0.670.0>                         <0.669.0>]}},
[error] <0.670.0>                {restart_type,transient},
[error] <0.670.0>                {significant,true},
[error] <0.670.0>                {shutdown,600000},
[error] <0.670.0>                {child_type,worker}]
[error] <0.670.0> 
[error] <0.176320.0> Restarting crashed queue 'consumer-client-channels-camera.update' in vhost '/'.

The configuration file looks like this:

listeners.ssl.default = 5671
listeners.tcp = none
ssl_options.cacertfile = /run/secrets/rabbitmq_ca.crt
ssl_options.certfile = /run/secrets/rabbitmq_server.crt
ssl_options.keyfile = /run/secrets/rabbitmq_server.key
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = true
auth_mechanisms.1 = EXTERNAL
definitions.import_backend = local_filesystem
definitions.local.path = /run/secrets/rabbitmq_definitions.json

Reproduction steps

None, only running a RabbitMQ Dockerised server rabbitmq:4.2.2-alpine on a Debian 13.1 host.
The queue type is classic with a queue length of 1-message.
Message consumption varies between 100ms and a few seconds.

Expected behavior

Saved messages in the message store does not trigger an enametoolong error.

Additional context

No response

[michaelklishin]

rabbit_classic_queue_index_v2:new_segment_file/3 runs into an {error, enametoolong}.

That's all I can tell from the log.

@embedded-bed I do not understand what this means

Our client should consume messages with a few 100ms interval, but sometimes it can take a couple of seconds between it's ready to consume a message. I think this causes RabbitMQ to save the message to a file on disk.

We need example virtual host and queue names of equal lengths. Queue names are supported to be limtied to 255 bytes in the protocol.

I do not remember us ever seeing enametoolong responses from the filesystem calls that create files.
You can search for enametoolong in this repository (which has a 16 to 18 year old history depending on how you count).

Node name can also contribute to the overall path length but not the filename length.

@lhoguin what else would you need to narrow this behavior down, what other inputs?

[lukebakken]

Wow, the filename that is too long ends with 3980 / characters.

[lhoguin]

The paths in qi and qs are computed using the same functions and inputs yet one is correct and the other is not. The final character of the path is just a binary append of "/". I doubt there's a bug in binary appends. There could be a memory/corruption bug in any layers between Erlang/OTP and the hardware though.

[embedded-bed]

Hello Ihoguin, thank you for the response. I am still trying to find the source of this corruption. I have tested on two different servers (different hardware) and Debian 12/13 and still find the same bug occurs.

I will try to change my setup to run RabbitMQ as a systemd managed service instead of a Docker container. Maybe there could be some issues in how Docker manages the volume.

[lhoguin]

Since you can reproduce if you could provide the exact steps that you follow we could try to reproduce as well.