Azure rabbitmq docker instance dies after a certain (large) amount of messages

[Dib-za]

Hi,

I am using the rabbitmq resource through Azure and am encountering a rather frustrating problem.

After approximately 360000 (or 720000 on a bigger VM) messages rabbitmq freezes up and stops processing messages. The VM and the docker container show that they have free memory, storage and CPU % so i don't think it is related to the VM (or docker container) failing. I have tried slowing down the speed at which the messages are sent. It seems to freeze after the same approximate number of messages, regardless on how spread out (over time) those messages are.

The management portal is not working for me (possibly due to being inside the VM), but running docker exec containerID rabbitmqctl status gives me the following type of output as it reaches the max number of messages before it freezes:

Status of node 'HelloRabbitDocker@64db8c1f1c6d' ...
[{pid,111},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.6.3"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.3"},
      {webmachine,"webmachine","1.10.3"},
      {mochiweb,"MochiMedia Web Server","2.13.1"},
      {inets,"INETS  CXC 138 49","6.2"},
      {syntax_tools,"Syntax tools","1.7"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.3"},
      {amqp_client,"RabbitMQ AMQP Client","3.6.3"},
      {compiler,"ERTS  CXC 138 10","6.0.3"},
      {ssl,"Erlang/OTP SSL application","7.3"},
      {public_key,"Public key infrastructure","1.1.1"},
      {asn1,"The Erlang ASN1 compiler version 4.0.2","4.0.2"},
      {crypto,"CRYPTO","3.6.3"},
      {rabbit,"RabbitMQ","3.6.3"},
      {rabbit_common,[],"3.6.3"},
      {ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},
      {xmerl,"XML parser","1.3.10"},
      {os_mon,"CPO  CXC 138 46","2.4"},
      {mnesia,"MNESIA  CXC 138 12","4.13.3"},
      {sasl,"SASL  CXC 138 11","2.7"},
      {stdlib,"ERTS  CXC 138 10","2.8"},
      {kernel,"ERTS  CXC 138 10","4.2"}]},
 {os,{unix,linux}},
 {erlang_version,
     "Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:2:2] [async-threads:64] [hipe] [kernel-poll:true]\n"},
 {memory,
     [{total,3319586272},
      {connection_readers,14280},
      {connection_writers,5688},
      {connection_channels,147862968},
      {connection_other,47744},
      {queue_procs,915471272},
      {queue_slave_procs,0},
      {plugins,254328},
      {other_proc,1920883688},
      {mnesia,72272},
      {mgmt_db,488552},
      {msg_index,48992},
      {other_ets,1441488},
      {binary,202833928},
      {code,27819950},
      {atom,1000601},
      {other_system,101340521}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,2922468147},
 {disk_free_limit,50000000},
 {disk_free,28257247232},
 {file_descriptors,
     [{total_limit,1048476},
      {total_used,4},
      {sockets_limit,943626},
      {sockets_used,1}]},
 {processes,[{limit,1048576},{used,243}]},
 {run_queue,2},
 {uptime,83490},
 {kernel,{net_ticktime,60}}]

The connection_channels number was concerning me until i realized it was in bytes :/.

It's possible I am doing something wrong client side, but I have been struggling for the past few days to identify the issue. Any help would be appreciated.

Regards
Dylan