gmane.comp.networking.rabbitmq.general

Re: cluster memory alarm blocks producer too long

Simon MacMullen — 2013-06-19T10:01:13

What does "rabbitmqctl status" / management node details page say the 
memory is used by?

Cheers, Simon

On 19/06/13 06:16, Xuehua Hou wrote:

Re: Work Queues design

Tim Watson — 2013-06-19T09:58:58


You can open multiple channels on a single connection (i.e., an application running in the JVM can open many connections and many channels per connection - each channel usually being assigned to a thread). As for "take the first available one", that's exactly what the "fair dispatch" section in the tutorial describes.


That's totally application specific, depending on how many messages you're expecting to process and so on. You can `re-size' the pool any time you like by adding more consumers - the prefetch-count will ensure that rabbit doesn't send any one consumer more than the specified number of un-acked messages. 


Add more consumers, as I said above. 


You can set up a cluster and use HA to mirror (i.e., replicate) the queue contents across multiple nodes. See www.rabbitmq.com/ha.html for details. Of course, with HA you'll store all the queue's messages on all of the (mirrored) nodes. If you were thinking of 'sharding' the data across multiple nodes, then we don't offer a solution for that out of

Re: rabbitmq-c how many channels I need?

Tim Watson — 2013-06-19T09:51:44


Have you spent any time trying to understand why memory use grows in the face of this function call? Is that memory growth constant (due to allocations), or transient due to buffering and so on?


If you pre-declare queues/exchanges on the broker then calling amqp_<object>_declare isn't strictly necessary, though of course if the object turns out to be missing (on the broker) you'll have to deal with the channel error (i.e., release the existing channel objects, open a new channel, call amqp_queue_declare and so on).

Work Queues design

Flavio Pompermaier — 2013-06-19T09:41:14

Hi to everybody,
I'm new to RabbitMQ but from what I see it could be the solution for a 
buffer implementation in one of my project.
In my scenario there will be a big queue of tasks (i.e. document indexing) 
that should be served as fast as possible,

Looking at the tutorial I think that what I need is somehow similar to the 
Work queues (http://www.rabbitmq.com/tutorials/tutorial-two-python.html) 
plus fair dispatching.
However, the aforementioned constraint "served as fast as possible" is not 
clear how to design a correct size of workers, in a scalable way..
In every example, a worker is a standalone java process, whereas I would 
like to manage workers more like a pool and take the first available one..

So, I think that I can summarize my questions as:

1) What is the best RabbitMQ architecture to solve my problems?
2) If my poolable solution is correct, how can I set up a pool? How do I 
decide the best size of the pool?
3) What if I discover that my server is overloaded and I want to distribute 
the

Re: rabbitmq-c how many channels I need?

Jonas Grimfelt — 2013-06-19T07:23:20

Please remove operations-bU72ZnzJ2wRBDgjK7y7TUQ< at >public.gmane.org from this discussion.
Den 19 jun 2013 09:03 skrev "3k4b251" <314992959-9uewiaClKEY< at >public.gmane.org>:

set/getTimeout in RabbitMQ for PHP

Iliya Isd — 2013-06-19T06:40:50

Hello! Having the next problem, maybe somebody has issued that. 
I'm trying to use the next function: 
http://php.net/manual/ru/amqpconnection.settimeout.php 
but nothing changes. Timeout isn't working. Thereby there's no way to avoid 
hanging up the site when trying to send message to rabbitMQ... 
Any ideas?

Re: rabbitmq-c how many channels I need?

3k4b251 — 2013-06-19T07:01:23

Thank you !   you  help  me soo much !   and  I  still  face  the problem  of 
memory use.       I  need  to  send a  message  to   a  queue ,  before 
that,  I'd  like  to declare  the queue  to  make  sure  the queue  already
there ,just like  amqp_queue_declare(....) ,and  then  i call 
amqp_basic_consume(....) ,  but  when  I use  this  function
:amqp_queue_declare(....)  , the memory grow up  fast.   what  can  I do  to
avoid  this statement?   



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-how-many-channels-I-need-tp27448p27474.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

cluster memory alarm blocks producer too long

Xuehua Hou — 2013-06-19T05:16:41

Hi,
I'm testing rabbitmq cluster.
The problem same as 
https://groups.google.com/forum/?fromgroups#!searchin/rabbitmq-discuss/memory$20cluster/rabbitmq-discuss/85g_ScoeeJM/jMspuiJHD7sJ 
still exists.
I can reproduce the issue.

I runned rabbitmq-server-3.1.1 on two centos6 machines, the erlang version 
is R16B.
The two machines are 4 core and 8G ram.
I used mirrored durable queue for HA.
Two nodes are disc.

Reproduce step: 
1.start four producers  and publish a lot of messages(more than 1000000).
2.start four consumers 
Producers and consumers running more than one hour, memory alarm will go 
off.
The master memory slow down quickly,
But the slave node stay in memory alarm state too long(more than 10 min).

Re: rabbitmq-c how many channels I need?

Alan Antonuk — 2013-06-19T04:02:30

As Michael mentioned: you can either create a channel per consumer or put
all your consumers on the same channel. Note that if a channel exception
occurs it will kill all of the consumers that are on that channel.


On Tue, Jun 18, 2013 at 6:28 PM, 3k4b251 <314992959-9uewiaClKEY< at >public.gmane.org> wrote:

Re: rabbitmq-c how can I close consumer?

Alan Antonuk — 2013-06-19T03:57:51

The function you want is amqp_basic_cancel(...):

https://github.com/alanxz/rabbitmq-c/blob/master/librabbitmq/amqp_framing.h#L621


On Tue, Jun 18, 2013 at 7:53 PM, 3k4b251 <314992959-9uewiaClKEY< at >public.gmane.org> wrote:

rabbitmq-c how can I close consumer?

3k4b251 — 2013-06-19T02:53:46

I  USE  amqp_basic_consume(....)   to     create  consumer ,  but  I  don't 
konw   how  to  close  the consumer.... 



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-how-can-I-close-consumer-tp27470.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

Re: rabbitmq-c how many channels I need?

3k4b251 — 2013-06-19T01:28:26

I  read  the  "amqpcpp-master"   example.  I  find  It  Create  Channel  for  
every  queue.   should  I  follow  that?



--
View this message in context: http://rabbitmq.1065348.n5.nabble.com/rabbitmq-c-how-many-channels-I-need-tp27448p27469.html
Sent from the RabbitMQ mailing list archive at Nabble.com.

Re: |Spam| Re: Possible memory leak in the management plugin

Travis Mehlinger — 2013-06-18T22:23:24

Hi Simon,

Sure thing, here you go:
https://gist.github.com/tmehlinger/158bb003fe3d64618a4e

Best, Travis


On Tue, Jun 18, 2013 at 10:47 AM, Simon MacMullen <simon-mQ7lE4MOPXtWk0Htik3J/w< at >public.gmane.org>wrote:

Re: Using a local system account to run the RMQ Windows Service, where does Mnesia store it's data?

Matthias Radestock — 2013-06-18T21:52:51

Check the beginning of rabbit's log file.

Matthias.

Using a local system account to run the RMQ Windows Service, where does Mnesia store it's data?

Christopher Lefevre — 2013-06-18T20:16:07

We are using a Windows 2008 Server R2 for our RabbitMQ server.  With RabbitMQ 3.0.4 running as a service, under the local system account. This service, to be in a cluster needed the .erlang.cookie to be in the C:\Windows directory.

With that said, where would the MNESIA database files be located?

Currently I can only see them under my account on the server, however, those do not seem to be the correct files? I ask as we are trying to integrate with Hyperic HQ, and we can't seem to find the proper MNESIA DB files that would yield the proper monitoring/statistics.

I can't seem to find any way to show the file locations via Rabbitmqctl.bat or the Rabbit Management Plugin.

Thank you for your time,
-Christopher Lefevre

The information contained in this electronic message and any attached document(s) is intended only for the personal and confidential use of the designated recipients named above. This message may be confidential. If the reader of this message is not the intended recipient, you are hereby notif

Re: Mirrored queues behind a load balancer

Girish Gangadharan — 2013-06-18T18:00:26

Thank you.

On Sun, Jun 16, 2013 at 11:24 PM, Jason McIntosh <mcintoshj-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

Re: Mirrored queues behind a load balancer

Girish Gangadharan — 2013-06-18T18:00:36

Thank you.

On Mon, Jun 17, 2013 at 3:18 AM, Tim Watson <watson.timothy-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

Re: rabbitmq-c how many channels I need?

Alan Antonuk — 2013-06-18T16:58:32

On Tue, Jun 18, 2013 at 4:54 AM, Michael Klishin <
michael.s.klishin-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


rabbitmq-c doesn't synchronize use of any part of the connection. Using a
rabbitmq-c object from multiple threads requires external synchronization.
Additionally if you want to consume messages in one thread while executing
other AMQP methods in another using the same connection there is some
knowledge of the AMQP protocol required in order not to transmit a method
at inappropriate moment between frames.

Re: rabbitmq-c how many channels I need?

Alan Antonuk — 2013-06-18T16:51:38

On Tue, Jun 18, 2013 at 4:58 AM, Michael Klishin <
michael.s.klishin-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


The structs you'd care about in rabbitmq-c are:

amqp_basic_consume_ok_t (
https://github.com/alanxz/rabbitmq-c/blob/master/librabbitmq/amqp_framing.h#L394
)

amqp_basic_deliver_t (
https://github.com/alanxz/rabbitmq-c/blob/master/librabbitmq/amqp_framing.h#L427
)

The amqp_basic_deliver_t method is the first method delivered in this
message reading loop:
https://github.com/alanxz/rabbitmq-c/blob/master/examples/amqp_consumer.c#L81

You can do the following to get an amqp_basic_delivery_t object from the
frame:

amqp_basic_deliver_t *delivery = (amqp_basic_deliver_t
*)frame.method.decoded;

Re: |Spam| Re: Possible memory leak in the management plugin

Simon MacMullen — 2013-06-18T15:47:08

OK, that definitely looks like a leak. Could you also give me the output 
from:

rabbitmqctl eval '{[{T, ets:info(T,size), ets:info(T,memory)} || T <- 
lists:sort(ets:all()), rabbit_mgmt_db <- [ets:info(T, name)]], 
sys:get_status(global:whereis_name(rabbit_mgmt_db))}.'

to make sure I'm clear on which table is leaking.

Cheers, Simon

On 18/06/13 16:11, Travis Mehlinger wrote:

Re: Error: cannot_discover_cluster: The nodesprovided are either offline or not running

Tim Watson — 2013-06-20T08:10:26

Erlang will load the cookie from $HOME/.erlang.cookie so if you don't put it there the runtime will assign a random cookie and create the file itself.

In this context, HOME variable belongs to the user running the program, i.e., rabbitmq.

Cheers
Tim

On 19 Jun 2013, at 18:49, "k.madnani84" <k.madnani84-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote: