http://blog.gmane.org/gmane.comp.networking.rabbitmq.general hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23267 <pre>Hello, I've just upgraded from RabbitMQ 3.0.2 to RabbitMQ 3.1.1. When I went to the web-based management console the first time, I saw the Overview page. Clicking through the tabs, however, I noticed that sometimes the UI didn't update. Now, whenever I try to go to the Overview page, the UI does not update, and I see one or multiples of this error at the bottom of the page: "ReferenceError: queue_length is not defined". Even after a server restart I cannot see the overview page. I am on RedHat 6.2 using Erlang R14B04 from EPEL. There are no notable errors in the logs. Any thoughts? Bug? Thanks, Chris </pre> Chris 2013-05-21T20:04:59 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23265 <pre>Has anyone implemented an automated means of removing RabbitMQ cluster nodes from an F5 load balancer pool? If so, I'd be interested in how you did it... Cheers, -ronc </pre> Cordell, Ron L 2013-05-21T16:24:10 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23262 <pre>Hi, After reciently upgrading to RabbitMQ 3.1.0 we are now seeing a 'Processing Error' when (some) STOMP frames contain a 'reply-to' header. [ Sending the frame without a reply-to header, or using a /temp-queue as the reply-to queue both send successfully ] Any ideas as to how to interpret the stack trace below, or hints suggest what I'm doing wrong would be most appreciated. Here's a sample frame that causes the error (the queue has been created in the admin interface): === SEND destination:/amq/queue/test1 reply-to:/amq/queue/test1 Test message === The corresponding stack trace form the log is: =ERROR REPORT==== 21-May-2013::22:21:07 === STOMP error frame sent: Message: "Processing error" Detail: "Processing error" Server private detail: {badarg, [{erlang,list_to_binary, [[84,95,47,116,101,109,112,45,113,117,101,117, 101,47|none]], []}, {rabbit_stomp_util,internal_tag,1,[]}, {rabbit_stomp_processor,ensure_reply_queue,2,[]}, {rabbit_stomp_processor,ensure_reply_to,2,[]}, {rabbit_stomp_processor,do_send,4,[]}, {rabbit_stomp_processor,with_destination,4,[]}, {rabbit_stomp_processor,process_request,3,[]}, {gen_server2,handle_msg,2,[]}]} Thanks in advance, Russell -- Russell Jenkins Programmer, Strategic Data. </pre> Russell Jenkins 2013-05-21T12:53:44 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23261 <pre>The RabbitMQ team is pleased to announce the release of RabbitMQ 3.1.1. This release fixes a number of bugs in 3.1.0 and earlier versions. See the release notes at: http://www.rabbitmq.com/release-notes/README-3.1.1.txt for more information. The new release can be downloaded from: http://www.rabbitmq.com/download.html As always, we welcome any questions, bug reports, and other feedback on this release, as well as general suggestions for features and enhancements in future releases. Mail us via the RabbitMQ discussion list. Regards, The RabbitMQ Team (http://www.rabbitmq.com) _______________________________________________ rabbitmq-announce mailing list rabbitmq-announce-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8&lt; at &gt;public.gmane.org https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-announce</pre> Tim Watson 2013-05-21T13:34:31 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23251 <pre>I have 3 RabbitMQ nodes, namely rabbit&lt; at &gt;A, rabbit&lt; at &gt;b, rabbit&lt; at &gt;c in a cluster with active-mirrored queues HA. I am using RabbitMQ 3.1.0 with autoheal chosen for "cluster_partition_handling" and "net_ticktime" set to 2. I tried to shutdown the network interface instead of shutting down rabbitmq to observe the behavior and realized that it results in the temporary unresponsiveness of all the other available nodes for a short period of time. This was what I did: 1) All the 3 nodes are running as per normal 2) I shutdown the network interface for rabbit&lt; at &gt;B 3) The other 2 nodes become unresponsive for a short period of time and the messages sent during that period were mostly lost. I would like to find out if that is the intended behavior or possibly something wrong with my setup. Thanks in advance. -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/All-nodes-in-a-cluster-becomes-unresponsive-for-a-short-period-of-time-when-any-node-goes-down-tp26883.html Sent from the RabbitMQ mailing list archive at Nabble.com. </pre> thomas 2013-05-21T05:39:08 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23250 <pre>Hi All, I have a very simple scenario. I create a queue, exchange and bind the queue to exchange. Code : String queueName = new StringBuilder("LP.DATA.").append("TESTACCOUNT").append(".").append("subLP").toString(); channel.exchangeDeclare(ExchangeConstants.DATA_EXCG, "direct", true); channel.queueDeclare(queueName, true, false, false, null); channel.exchangeBind(queueName, ExchangeConstants.DATA_EXCG, queueName); I get the following error when exchangeBind() is called. com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method&lt;channel.close&gt;(reply-code=404, reply-text=NOT_FOUND - no exchange 'LP.DATA.TESTACCOUNT.subLP' in vhost '/', class-id=40, method-id=30), null, ""} My ExchangeConstants.DATA_EXCG is "DATA_EXCG". Using RabbitMq admin I see that the exchange and queue are all created. From the exception I don't understant why is it looking for exchange with name "LP.DATA.TESTACCOUNT.subLP", this is a queuename. It seems quite trivial, I am sure I am missing something. I am using java rabbitmq-client version 3.04. Thanks, Vikram </pre> Vikram Viswanathan 2013-05-21T05:15:07 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23249 <pre>I'm glad to be able to announce the Kerberos (note: not GSSAPI[1]) authentication plugin for RabbitMQ. The plugin makes it possible to use Kerberos principals to login to RabbitMQ 3.x both via AMQP and the management GUI. Since Kerberos is an authentication protocol you still need a plugin to provide authorization. By default that is done via rabbit_auth_backend_internal. The code is available from &lt;https://github.com/simmel/rabbitmq-auth-backend-kerberos&gt; and is built with help from the RabbitMQ plugin development guide &lt;https://www.rabbitmq.com/plugin-development.html&gt;. Code is licenced under the revised BSD license. Configuration and install documentation is also available from the above github page. The plugin works with both Heimdal and MIT Kerberos libs. I'd like to thank everyone who has helped this become a reality, especially Simon and Tim from VMware and Linus and Klas, my former co-workers. Finally I'd like to thank my employer, IT Services at Stockholm University, for giving me the opportunity to learn C (atleast, more properly) and Erlang. [1] While we would love to use GSSAPI instead of user/password via Kerberos, we weren't that keen on implementing GSSAPI in all RabbitMQ/AMQP libraries available. Br, - Simon ____________________________________ Simon Lundström Section for Infrastructure IT Services Stockholm University SE-106 91 Stockholm, Sweden www.su.se/it ____________________________________ </pre> Simon Lundström 2013-05-21T05:38:42 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23243 <pre>Hello, I've came across a problem, but unfortunately I have very little information about it. I use rabbitmq 2.8.4, along with clients such as pika (python side) and SimpleAmqpClient (c++ side). This issue was experienced in both those clients. What happened is simply that DeclareQueue or queue_declare wouldn't return at all (I mean being blocked for *days*). When the incident started, I get a lot of =WARNING REPORT==== 18-May-2013::03:09:18 === closing AMQP connection &lt;0.3256.1697&gt; (127.0.0.1:46099 -&gt; 127.0.0.1:5672): connection_closed_abruptly in the rabbitmq log, but nothing more. I fixed this by … restarting rabbitmq, but I'm kind of worry because I don't know how to detect this problem again (building a timeout mechanism on top of the clients is just not ok). If anyone has any idea about what could cause this, it would be greatly appreciated. </pre> Michael Journo 2013-05-20T15:26:42 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23240 <pre>There is a new feature in rabbitmq 3.1.0 - eager synchronization. What is this eager synchronization in detail? Should I explicitly set eager sync policy? </pre> Andrey Kolchanov 2013-05-20T13:56:30 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23238 <pre>Hello, I've came across a problem, but unfortunately I have very little information about it. I use rabbitmq 2.8.4, along with clients such as pika (python side) and SimpleAmqpClient (c++ side). This issue was experienced in both those clients. What happened is simply that DeclareQueue or queue_declare wouldn't return at all (I mean being blocked for days). When the incident started, I get a lot of =WARNING REPORT==== 18-May-2013::03:09:18 === closing AMQP connection &lt;0.3256.1697&gt; (127.0.0.1:46099 -&gt; 127.0.0.1:5672): connection_closed_abruptly in the rabbitmq log, but nothing more. I fixed this by … restarting rabbitmq, but I'm kind of worry because I don't know how to detect this problem again (building a timeout mechanism on top of the clients is just not ok). If anyone has any idea about what could cause this, it would be greatly appreciated. michael </pre> Michaël Journo 2013-05-20T13:44:34 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23230 <pre>Hi, We are in the process of deploying a Riak cluster, and are wondering if some of the lessons we learned monitoring Erlang in Riak should be applied to RabbitMQ. During testing we noticed a lot of busy_dist_port in the logs, the root was cause was a the Erlang kernel parameter zdbbl was set too low. We have adjusted the parameter and are now monitoring the logs for busy_dist_port errors. I have searched the RabbitMQ mailing list for cases of busy_dist_port or zdbbl, but cannot find any. I am wondering if RabbitMQ can suffer from problems with busy_dist_port or if zdbbl should be set to any particular value? Or I should I monitor the RabbitMQ logs for busy_dist_port? Thanks, Iain. </pre> Iain Hull 2013-05-20T11:15:41 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23225 <pre>Hi I want to use RabbitMQ to create a distributed task queues on aws. In order to kill/start ec2 instances, i would need to access the moving average of queued messages of the last 1/5 minutes. Since this is possible in the web ui I would guess it is possible as well using the HTTP api. After some extensive Googling and reading RabbitMQ in Action I am still not able to find how to retrieve this information. The best I can do is retrieve the following JSON - it might be in there but I am not sure where :-) Any idea ? Thanks Guillaume { "memory": 89664, "message_stats": { "ack": 44, "ack_details": { "rate": 0.0 }, "deliver": 45, "deliver_details": { "rate": 0.0 }, "deliver_get": 45, "deliver_get_details": { "rate": 0.0 }, "publish": 160, "publish_details": { "rate": 0.0 } }, "messages": 116, "messages_details": { "rate": 0.0 }, "messages_ready": 116, "messages_ready_details": { "rate": 0.0 }, "messages_unacknowledged": 0, "messages_unacknowledged_details": { "rate": 0.0 }, "idle_since": "2013-05-20 3:22:02", "policy": "", "exclusive_consumer_tag": "", "consumers": 0, "backing_queue_status": { "q1": 0, "q2": 0, "delta": [ "delta", "undefined", 0, "undefined" ], "q3": 0, "q4": 116, "len": 116, "pending_acks": 0, "target_ram_count": "infinity", "ram_msg_count": 116, "ram_ack_count": 0, "next_seq_id": 160, "persistent_count": 116, "avg_ingress_rate": 0.0, "avg_egress_rate": 0.0, "avg_ack_ingress_rate": 0.0, "avg_ack_egress_rate": 0.0 }, -- View this message in context: http://rabbitmq.1065348.n5.nabble.com/HTTP-monitoring-API-tp26859.html Sent from the RabbitMQ mailing list archive at Nabble.com. </pre> tog 2013-05-20T04:17:00 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23197 <pre>Multiple delete. I often have many queues that I no longer need, especially in testing. I would like to be able to select multiple queues for deletion from the management web console. Thanks </pre> Rob Woolfson 2013-05-19T09:03:46 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23190 <pre>Hi all, In case we have 2 or more rabbitmq nodes connected in cluster and each with VIP assigned, would we get more throughput if messages are sent on all nodes (using round-robin method) and received by workers connected on each node. If cluster is not goot way to get more throughput using additional rabbitmq nodes, what is solution? Thanks, Nikola </pre> nikolasavic77 2013-05-18T07:34:27 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23189 <pre>*Hi, all,* *I've been trying to install rabbitmq (ver 3.1.0) on my own vps ( linux distribution: debian, x86) for a long time, but I just failed again and again.* * * *First, I'd like to show the error message I got:* root&lt; at &gt;XXX:~/code# /etc/init.d/rabbitmq-server start [warn] Starting message broker: rabbitmq-server[....] FAILED - check /var/log/rabbitmq/startup_\{log, _err\} ... (warning). failed! root&lt; at &gt;XXX:~/code# rabbitmqctl status {error_logger,{{2013,5,18},{13,21,20}},"Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",epmd_close]} {error_logger,{{2013,5,18},{13,21,20}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,&lt;0.21.0&gt;},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,320}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[net_sup,kernel_sup,&lt;0.10.0&gt;]},{messages,[]},{links,[#Port&lt;0.93&gt;,&lt;0.18.0&gt;]},{dictionary,[{longnames,false}]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,785}],[]]} {error_logger,{{2013,5,18},{13,21,20}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},{mfargs,{net_kernel,start_link,[[rabbitmqctl12829,shortnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]} {error_logger,{{2013,5,18},{13,21,20}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} {error_logger,{{2013,5,18},{13,21,20}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,&lt;0.9.0&gt;},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[&lt;0.8.0&gt;]},{messages,[{'EXIT',&lt;0.10.0&gt;,normal}]},{links,[&lt;0.8.0&gt;,&lt;0.7.0&gt;]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]} {error_logger,{{2013,5,18},{13,21,20}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]} {"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}}"} Crash dump was written to: erl_crash.dump Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k root&lt; at &gt;XXX:~/code# ps xua | grep epmd rabbitmq 12470 0.0 0.1 2524 548 ? S 13:15 0:00 /usr/lib/erlang/erts-5.10.1/bin/epmd -daemon root 12866 0.0 0.1 4428 804 pts/0 S+ 13:21 0:00 grep --color=auto epmd root&lt; at &gt;XXX:~/code# ps xua | grep rabbitmq rabbitmq 12470 0.0 0.1 2524 548 ? S 13:15 0:00 /usr/lib/erlang/erts-5.10.1/bin/epmd -daemon root 12868 0.0 0.1 4428 812 pts/0 S+ 13:22 0:00 grep --color=auto rabbitmq *And I've tried many solving methods I've googled, such as changing hostname and writing rabbitmq.conf, but none worked.* * * *What's more, I even updated my Erlang, which is now R16B.* * * *Still, nothing could be solved, and I turn to your for help.* * * *Thank you in advance and I'm looking forward to your reply.* * * *Xiang Yu* </pre> Parlin 2013-05-18T05:26:18 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23188 <pre>Hello, What is expected value of consumed msg/s per CPU code running at 2.4GHz? We're using 4 core VM running only RabbitMQ. When under load of about 500msg/s, IDLE is only about 50%. This sounds like far away from high performance, since we're expecting much much more msg/s :( There is enough RAM, about 30-35 connections are reported, 32 channels, 61 queues. Thanks, Nikola </pre> nikolasavic77 2013-05-17T20:44:09 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23187 <pre>Hi All, I have a three node cluster running on VMs for testing purposes: node1 (disk) node2(disk) node3(ram) Today we had a power outage and all the nodes went down at once. I am having a serious issue recovering from this because node1 thinks it belongs to a cluster but node3 disagrees. I can not seem to do anything at all with rabbitmq on node1. If I try running running "rabbitmqctl reset" it can not connect because node1 is down. How can I recover from this gracefully? Thanks, Sam </pre> Sam Winchenbach 2013-05-17T19:32:26 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23185 <pre>Bunny 0.9.0.pre12 is released to rubygems.org [1]. This release restores Ruby 1.8 compatibility. Release notes: http://blog.rubyrabbitmq.info/blog/2013/05/18/bunny-0-dot-9-0-dot-pre12-is-released/ 1. http://rubygems.org/gems/bunny/versions/0.9.0.pre12 </pre> Michael Klishin 2013-05-18T14:54:23 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23179 <pre>Hi, I am running a RabbitMQ cluster of 4 nodes in AWS EC2 on *c1.xlarge *instances using Ubuntu 12.04 LTS (kernel 3.2.0-38). Periodically two of my nodes holding the largest queues will crash and require me to restart the rabbitmq-server service. Looking at the logs, I'm at a loss as to why the crash is occurring. I've have tried reproducing the crash manually by flooding my nodes with messages, but that doesn't seem to trigger the issue. Below is output from various RabbitMQ logs: # cat /var/log/rabbitmq/rabbit-72dRMgatfUn5amjFlPYPZiOxP4NdHAT3hG9boyeUdBc&lt; at &gt;public.gmane.org Notice how the *case_clause *is *empty*. The other rabbit log looks similar: # cat /var/log/rabbitmq/rabbit-72dRMgatfUn5amjFlPYPZgfGRAsWV3w2&lt; at &gt;public.gmane.org =ERROR REPORT==== 18-May-2013::01:41:48 === ** Generic server &lt;0.301.0&gt; terminating ** Last message in was {'$gen_cast', {ack, [27585,27584,27607,27630,27653,27676,27699, 27722,27745,27768,27791,27814,27837,27860, 27883,27906,27929,27952,27975,27998,28021, 28044,28067,28090,28113,28136,28159,28180, 28201,28222,28243,28264,28285,28306,28325, 28343,28361,28379,28397,28415,28433,28451, 28469,28487,28505,28523,28541,28559,28577, 28595,28613], &lt;0.676.0&gt;}} ** When Server state == {q, {amqqueue, {resource,&lt;&lt;"/"&gt;&gt;,queue, &lt;&lt;"document_queue"&gt;&gt;}, true,false,none,[],&lt;0.301.0&gt;,[],[],undefined,[]}, none,true,rabbit_variable_queue, {vqstate, {0,{[],[]}}, {0,{[],[]}}, {delta,undefined,0,undefined}, {0,{[],[]}}, {0,{[],[]}}, * ... [ thousands of lines of state ] ... * ** Reason for termination == ** {{case_clause,{empty,{[],[]}}}, [{rabbit_amqqueue_process,subtract_acks,3,[]}, {rabbit_amqqueue_process,subtract_acks,4,[]}, {rabbit_amqqueue_process,handle_cast,2,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,249}]}]} connection &lt;0.430.0&gt;, channel 22 - soft error: {amqp_error,not_found, "no queue 'document_queue' in vhost '/'", 'queue.declare'} connection &lt;0.384.0&gt;, channel 12 - soft error: {amqp_error,not_found, "no queue 'document_queue' in vhost '/'", 'queue.declare'} connection &lt;0.384.0&gt;, channel 15 - soft error: {amqp_error,not_found, "no queue 'document_queue' in vhost '/'", 'queue.declare'} connection &lt;0.373.0&gt;, channel 23 - soft error: {amqp_error,not_found, "no queue 'document_queue' in vhost '/'", 'queue.declare'} * ... [ repeated a bunch of times ] ...* I also see this in my */var/log/syslog* with regards to *beam.smp*: # cat /var/log/syslog There are no strange CPU spikes on my EC2 node based on AWS monitoring. The only thing that seems to stand out is that my writes to my EBS volume spike during the crash (~90 write OPs/sec, 4 MiB/sec write bandwidth, ~30ms/op write latency). My team and I are completely stumped and would appreciate any insight you might have to the crash reports. Is this a problem with writes hanging when going against the EBS volume since EBS volumes are essentially network mounted? Is there a way to configure rabbit to be more tolerant of these delays? For debugging purposes, below is the status output of one of the nodes when it is healthy (so you can see the versions of everything i'm running): # rabbitmqctl status Note that I am using Erlang v16 from the Erlang Solutions repo (esl-erlang package). I originally was using Erlang v14 from Ubuntu repos (erlang-base), but decided to upgrade hoping it would resolve the issue (which it did not). I have also tried running RabbitMQ 3.0.4 without success. This issue seems to affect both versions. Thanks, Karl </pre> Karl Rieb 2013-05-18T03:20:01 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23176 <pre>Hello, I've been working with several two node clusters running various versions of 3.0.x, hosted on m1.small instances on Ubuntu 12.04.2 LTS in EC2. The setup is essentially as described here http://karlgrz.com/rabbitmq-highly-available-queues-and-clustering-using-amazon-ec2/ with the main exception being that both of the RabbitMQ servers are in the same availability zone. A while back I observed a half dozen or so occurrences over the course of a week where the clusters would become partitioned, accompanied by a messages on each server such as: =ERROR REPORT==== 17-May-2013::01:56:45 === ** Node 'rabbit&lt; at &gt;oemsg-new-29b15241' not responding ** ** Removing (timedout) connection ** =INFO REPORT==== 17-May-2013::01:56:45 === rabbit on node 'rabbit&lt; at &gt;oemsg-new-29b15241' down Looking over the logs and EC2 metrics, I wasn't able to identify any other anomalies that coincided with these failures. In particular, the load balancers in front of the cluster nodes did not report any health check failures connecting to the amqp port (on a 30 second interval), suggesting that network connectivity was otherwise healthy, and there didn't appear to be any unexpected spikes in resource consumption (such as excessive cpu/disk/network activity). The rabbit servers were fairly lightly loaded with messaging traffic at the time, and running some load tests against the same servers afterwards didn't induce any further failures over the course of several days. I tried increasing the net_ticktime to something like 5 or 10 minutes, but still observed a failure with the new value. I left several clusters running over an extended period, most with little or no load, with one cluster running under an extended load test. Several of the clusters experienced no failures over the course of a couple of months, while others became partitioned after a while (though they seemed to survive for at least a few weeks before partition). Anyone experience anything similar in EC2, or have any ideas what else might be done to diagnose what's going on? Ray Maslinski Senior Software Developer, Engineering Valassis / Digital Media Cell: 585.330.2426 maslinskir-bVq8gIIoexFWk0Htik3J/w&lt; at &gt;public.gmane.org www.valassis.com&lt;http://www.valassis.com/&gt; Creating the future of intelligent media delivery to drive your greatest success _____________________________________________________________________________ This message may include proprietary or protected information. If you are not the intended recipient, please notify me, delete this message and do not further communicate the information contained herein without my express consent. </pre> Maslinski, Ray 2013-05-17T20:03:36 http://comments.gmane.org/gmane.comp.networking.rabbitmq.general/23173 <pre>Hello, I've been experimenting with the new autoheal mode for handling cluster partitions and automatic synchronization for mirrored queues using servers hosted at Amazon EC2. My test setup is a two node cluster, with queue policy set to ha-mode=all and ha-sync-mode=automatic. The servers are also configured with cluster_partition_handling set to autoheal, and sit behind an ELB for client access. The rest of the configuration is essentially the same as describe here http://karlgrz.com/rabbitmq-highly-available-queues-and-clustering-using-amazon-ec2/ with the main exception being that both of the RabbitMQ servers are in the same availability zone. The servers are hosted on m1.small instances running Ubuntu Linux 12.04.2 LTS. The client load is being generated from JMeter using a tunable number of client threads to send and receive messages at a steady rate to a mirrored queue. With the load balancer in place, the tests are run with enough threads to ensure that both servers have active connections for both sending and receiving. The test sequence verifies that all message sent without encountering an error are also received. To simulate a network partition failure, I've been using iptables to temporarily block inbound and outbound access on one of the nodes to the single port configured for cluster communications through inet_dist_listen_min and inet_dist_listen_max settings (min = max). Client access is not blocked during a simulated partition fault. I've observed two anomalies during testing that I wasn't expecting based on the documentation I've read: - At a sufficiently high message rate, some number of messages will be lost during the fault sequence, with the number lost tending to increase with message rate. No indication of a send error has been observed by the client program. Based on results obtained from test logs and an independent monitor listening on trace messages from each node, it appears that as soon as the port is blocked, both nodes continue to accept published messages, but (temporarily) stop delivering messages until the cluster heartbeat failure is detected, at which point the cluster is partitioned and the slave promotes itself to become master. In the sequences I've looked at, the messages that are lost all appear to be published to the original master (and final master after a winner is selected during autoheal). Neither the start nor the end of the lost message window appear to line up with any events in the logs, other than the start occurring sometime after the port connection is blocked but before the cluster heartbeat failure is detected, and the end occurring sometime after the detection of the cluster heartbeat failure and before the detection of the partitioned cluster after the connection is unblocked. Is message loss to be expected in this scenario? - Occasionally the autoheal loser node fails to rejoin the cluster after restart. I don't have a lot of data points on this one since it's only happened a handful of times during overnight test iterations. During one failure, the autoheal winner showed the log message below during recovery: =ERROR REPORT==== 17-May-2013::02:40:36 === Mnesia('rabbit&lt; at &gt;oemsg-new-27b1524f'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit&lt; at &gt;oemsg-new-29b15241'} =INFO REPORT==== 17-May-2013::02:40:36 === Autoheal request received from 'rabbit&lt; at &gt;oemsg-new-29b15241' =ERROR REPORT==== 17-May-2013::02:40:36 === ** Generic server rabbit_node_monitor terminating ** Last message in was {autoheal_msg, {request_start,'rabbit&lt; at &gt;oemsg-new-27b1524f'}} ** When Server state == {state, {dict,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[], [[{rabbit,'rabbit&lt; at &gt;oemsg-new-29b15241'}| #Ref&lt;0.0.65.255337&gt;]], [],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}, ['rabbit&lt; at &gt;oemsg-new-29b15241'], {dict,1,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[], []}, {{[],[], [[&lt;0.20953.18&gt;|#Ref&lt;0.0.65.232752&gt;]], [],[],[],[],[],[],[],[],[],[],[],[],[]}}}, undefined, {winner_waiting, ['rabbit&lt; at &gt;oemsg-new-29b15241'], ['rabbit&lt; at &gt;oemsg-new-29b15241']}} ** Reason for termination == ** {function_clause, [{rabbit_autoheal,handle_msg, [{request_start,'rabbit&lt; at &gt;oemsg-new-27b1524f'}, {winner_waiting, ['rabbit&lt; at &gt;oemsg-new-29b15241'], ['rabbit&lt; at &gt;oemsg-new-29b15241']}, ['rabbit&lt; at &gt;oemsg-new-29b15241']]}, {rabbit_node_monitor,handle_info,2}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} =INFO REPORT==== 17-May-2013::02:40:36 === Autoheal decision * Partitions: [['rabbit&lt; at &gt;oemsg-new-29b15241'],['rabbit&lt; at &gt;oemsg-new-27b1524f']] * Winner: 'rabbit&lt; at &gt;oemsg-new-27b1524f' * Losers: ['rabbit&lt; at &gt;oemsg-new-29b15241'] =INFO REPORT==== 17-May-2013::02:40:36 === Autoheal request sent to 'rabbit&lt; at &gt;oemsg-new-27b1524f' =INFO REPORT==== 17-May-2013::02:40:36 === Autoheal: I am the winner, waiting for ['rabbit&lt; at &gt;oemsg-new-29b15241'] to stop However, I've also seen at least one other instance where the message about the node monitor terminating appeared but did not coincide with an autoheal failure. Comparing the two occurrences shows a somewhat different log sequence around server shutdown and startup: No autoheal failure: =WARNING REPORT==== 17-May-2013::01:58:36 === Non-AMQP exit reason 'shutdown' =ERROR REPORT==== 17-May-2013::01:58:36 === Mnesia('rabbit&lt; at &gt;oemsg-new-29b15241'): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, 'rabbit&lt; at &gt;oemsg-new-27b1524f'} =INFO REPORT==== 17-May-2013::01:58:36 === Starting RabbitMQ 3.1.0 on Erlang R14B04 Copyright (C) 2007-2013 VMware, Inc. Licensed under the MPL. See http://www.rabbitmq.com/ Autoheal failure: =WARNING REPORT==== 17-May-2013::02:40:36 === Non-AMQP exit reason 'shutdown' =INFO REPORT==== 17-May-2013::02:44:51 === Starting RabbitMQ 3.1.0 on Erlang R14B04 Copyright (C) 2007-2013 VMware, Inc. Licensed under the MPL. See http://www.rabbitmq.com/ Note that rabbitmq-server is started through an upstart job which attempts to respawn, so the long delay between shutdown and startup message may be related to multiple restart attempts of the process (nothing of interest in the startup/shutdown logs). Any ideas around possible causes of this autoheal failure? Ray Maslinski Senior Software Developer, Engineering Valassis / Digital Media Cell: 585.330.2426 maslinskir-bVq8gIIoexFWk0Htik3J/w&lt; at &gt;public.gmane.org www.valassis.com&lt;http://www.valassis.com/&gt; Creating the future of intelligent media delivery to drive your greatest success _____________________________________________________________________________ This message may include proprietary or protected information. If you are not the intended recipient, please notify me, delete this message and do not further communicate the information contained herein without my express consent. </pre> Maslinski, Ray 2013-05-17T19:38:05 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.networking.rabbitmq.general