gmane.comp.lang.erlang.general

Re: Rabbit under small load

Max Bourinov — 2013-05-24T19:59:28

It is only 100-500 messages per second.

Best regards,
Max



On Fri, May 24, 2013 at 7:49 PM, Tim Watson <watson.timothy< at >gmail.com>wrote:

Re: Rabbit under small load

Tim Watson — 2013-05-24T17:49:20


I'm pretty sure what he said was that he's currently handling 100 M/s, which probably fits the "messages per second" guess, since as you point out, rabbit has not been demonstrated to go anywhere near 1millions messages per second.

Is your gen_server wrapper open source btw?

Re: Rabbit under small load

Tim Watson — 2013-05-24T17:18:30

Nonsense. He hasn't stated what his target throughput actually is, so how can you say that?

On 24 May 2013, at 18:04, Yogish Baliga <yogishb< at >yahoo.com> wrote:

Re: Rabbit under small load

Yogish Baliga — 2013-05-24T17:04:59

Single RabbitMQ server cannot handle your load requirement. You need to
setup multiple RabbitMQ servers and handle shard the messages across
multiple servers.

Re: Investigate an infinite loop on productionservers

Jesper Louis Andersen — 2013-05-24T11:17:59

emysql… check your MySQL query log for queries with large return results. Emysql has no cursor-support, so you have to build the result in memory and this can *easily* blow up your heap space if you do a large select.

Jesper Louis Andersen
  Erlang Solutions Ltd., Copenhagen



On May 24, 2013, at 10:34 AM, Morgan Segalis <msegalis< at >gmail.com> wrote:

Re: Rabbit under small load

Jesper Louis Andersen — 2013-05-24T11:15:25


On May 24, 2013, at 10:28 AM, Max Bourinov <bourinov< at >gmail.com> wrote:


Is that 100 * 1000 * 1000 messages per second or 100 messages per second?

RabbitMQ should easily be able to handle the latter load without much trouble. I have built a gen_server as a middle-man for RabbitMQ which handles the request/reply toward RMQ more or less like your approach and I can easily do 6000 messages per second with this setup and very little to no tuning. I don't know where the limit to that solution is, but there is a one-core bottleneck in the gen_server.

100 million messages is a different question however. Given the average message size, you are at 400 megabytes per second and hence you need 4 Gigabit of bandwidth to even hope hitting target. Apart from that, I don't think RMQ can handle that amount of load. You may need something else entirely to handle that.

Jesper Louis Andersen
  Erlang Solutions Ltd., Copenhagen

Re: Investigate an infinite loop on productionservers

Vance Shipley — 2013-05-24T09:27:43

It "freezes"? So it hasn't crashed at all.  In that case you just need to
be more patient and wait for it to either crash, and leave a crash dump, or
output to etop. Possibly setting process priorities would help. Give the
suspicious ones low priority.
If it's CPU resources which are being depleted you would want to observe
which have the most reductions. Use stop to order by reductions and see
who's the busiest.

Another way would be to run a debug emulator and interrupt  it while it's
frozen. Then inspect the backtrace to see what it has been doing.
On May 24, 2013 1:43 PM, "Morgan Segalis" <msegalis< at >gmail.com> wrote:

Re: Investigate an infinite loop on productionservers

Vance Shipley — 2013-05-24T09:05:43

Have you set the ERL_CRASH_DUMP_SECONDS environment variable?:
   It won't create one unless you set it to a positive value. Set it to 60
or more to be sure it completes.
 On May 24, 2013 1:43 PM, "Morgan Segalis" <msegalis< at >gmail.com> wrote:

Re: Investigate an infinite loop on productionservers

Dmitry Kolesnikov — 2013-05-24T09:06:03

Indeed the list of apps is simple, basically this is either ssl, emysql or your integration layer.

If you read a lot of data from mysql then that code might leak a memory by keeping ref to huge binaries. 
E.g. you you do select * from xxx then that data is returned as set of binaries. 
Whole binary hands in memory even if a first row is used. This is a one way to treat binaries received from somewhere: 

case binary:referenced_byte_size(X) of
   Large when Large > 2 * byte_size(X) -> 
      binary:copy(X);
   _ ->
      X
end

as long as I see emysql does not use binary:copy anywhere but of course they might dereference them in other way around.
If you could monitor a erlang:memory(binary) over a time then it might reveal the case.

BTW, is this specific to R16 or other release if so then some glitches at ssl?

- Dmitry

On May 24, 2013, at 11:34 AM, Morgan Segalis <msegalis< at >gmail.com> wrote:

Re: R14B04 and OpenSSL 1.0.1c

Bogdan Andu — 2013-05-24T08:48:22

in erts/configure.in there is a section:

openbsd*)
                DED_LD="$CC"

DED_LD_FLAG_RUNTIME_LIBRARY_PATH="$CFLAG_RUNTIME_LIBRARY_PATH"
                DED_LDFLAGS="-shared"
        ;;

Spoke with guys on the misc< at >openbsd.org maillist and they told me to run
autoreconf, to regenerate configure from configure.in .

And indeed that did the trick and now is working fine.

There is no need to specify :
DED_LDFLAGS="-shared /usr/lib/crtbeginS.o /usr/lib/crtendS.o"
because this generates conflict on symbol '__guard_local'


Based on discussions in this topic and in misc< at >openbsd.org maillist here is
a small recipe for those who need to compile erlang R14B04 from sources on
OpenBSD 5.3 amd64:

1. cd otp_src_R14B04
2. apply the patches founded in directory /usr/ports/lang/erlang/patches/
3. $ autoreconf
4. compile and install as usual

Thank you for your help,

Bogdan


On Thu, May 23, 2013 at 11:50 PM, Stanislav Sedov <stas< at >freebsd.org> wrote:

Re: Investigate an infinite loop on productionservers

Morgan Segalis — 2013-05-24T08:35:48

Thanks ! 

I'll try that…

will keep you posted.

Le 24 mai 2013 à 10:30, masked.prize< at >gmail.com a écrit :

Re: Investigate an infinite loop on productionservers

Morgan Segalis — 2013-05-24T08:34:37

Thank you, I'll look into it.

Heres what application:which_application() gives me : 

[{emysql,"Emysql - Erlang MySQL driver","0.2"},
 {ssl,"Erlang/OTP SSL application","5.2.1"},
 {public_key,"Public key infrastructure","0.18"},
 {crypto,"CRYPTO version 2","2.3"},
 {stdlib,"ERTS  CXC 138 10","1.19.1"},
 {kernel,"ERTS  CXC 138 10","2.16.1"}]

nothing fancy has you can see...

Le 24 mai 2013 à 10:31, Dmitry Kolesnikov <dmkolesnikov< at >gmail.com> a écrit :

Re: Investigate an infinite loop on productionservers

Dmitry Kolesnikov — 2013-05-24T08:31:17

Hello,

I am not aware of a single flag to limit the memory like in Java.
You can try to configure memory allocation 
http://www.erlang.org/doc/man/erts_alloc.html 

One of the freeze reason might be a huge crash_dump.
See the flags at bottom of page how to tune its behaviour 
http://www.erlang.org/doc/man/erl.html

If you switch off a swap it helps to observe OOM.

Would you share to the list app you running applications?
application:which_applications()


- Dmitry


On May 24, 2013, at 11:13 AM, Morgan Segalis <msegalis< at >gmail.com> wrote:

Rabbit under small load

Max Bourinov — 2013-05-24T08:28:54

Hello Dear Erlangers!

I have about 3000 erlang-processes which should send messages to AMQP-queue
of destination and receive replies from it.

The message rate at the moment is about 100 M/s
Average message size: 4 Kb

At the moment I have one locally registered gen_server that does all
communication with RabbitMQ. It consumes messages and routes them
to corresponding worker processes (via gproc). When worker process need to
send something, it cast message to RabbitMQ process via local name.  With
this approach I see some problems with message consumption speed.

What is the right way to interact with RabbitMQ server in Erlang
code? Should it be more consumer processes?

Best regards,
Max

Re: Investigate an infinite loop on productionservers

Dave Cottlehuber — 2013-05-24T08:17:38

You could use ulimit -v perhaps, I never got ulimit to work right with
resident memory.

A+
Dave

Re: Investigate an infinite loop on productionservers

Morgan Segalis — 2013-05-24T08:13:46

The problem is that the VM freezes completely, it does not generate a crash dump

Is there a way to limit the memory that a VM can allocate, so the server is not overwhelmed in order to create a crash dump ?

Le 24 mai 2013 à 02:00, Vance Shipley <vances< at >motivity.ca> a écrit :

Re: Investigate an infinite loop on productionservers

Morgan Segalis — 2013-05-23T21:30:10

Yeah you got that right ! leaking at a huge rate at some point !

- The number of Fd - I don't get close to the max
# cat /proc/sys/fs/file-nr
3264    0       6455368

- On the production server there is only the erlang node, no other service…
The beam.smp was through the roof at 300% CPU and 97% RAM
The weird thing is that it got there in a second, I was looking at it when it happens.

- It has happened with 2000 connections, 4000 connections, and 10000 connections… 5 min after start, 5hours after start.

I really can't find a pattern here…and I'm becoming a little desperate.

Thank you for your help again.

Morgan.

Le 23 mai 2013 à 23:20, Dmitry Kolesnikov <dmkolesnikov< at >gmail.com> a écrit :

Re: Investigate an infinite loop on productionservers

Dmitry Kolesnikov — 2013-05-23T21:20:30

You system definitely leaking some resources :-/
 - Check number of used FD(s) may be you exceeded limit there 
 - What was overall system memory / cpu utilisation before crash?
 - Check how many connections you got before crash, may be you can reproduce it at dev

- Dmitry

On May 24, 2013, at 12:13 AM, Morgan Segalis <msegalis< at >gmail.com> wrote:

Re: Investigate an infinite loop on productionservers

Morgan Segalis — 2013-05-23T21:13:29

Ok, it finally got into the infinite loop…

And of course, the node on which I was running etop could not give me anymore since it got disconnected from the production node.

So back to square one… no way to investigate correctly so far :-/

Morgan.

Le 23 mai 2013 à 16:34, Morgan Segalis <msegalis< at >gmail.com> a écrit :

Re: R14B04 and OpenSSL 1.0.1c

Stanislav Sedov — 2013-05-23T20:50:39


Did you try to link agains libcrtendS.o as well like openbsd guys do for CDE?
Also, you need to make sure that crypto.o linking actually uses these flags you
changes in erts/configure.in (also, btw, you need to rebuild configure if you
changes configure.in).

I don't have openbsd installed anywhere, so cannot provide more detailed help,
unfortunately.  I'd recommend you to play with the linking flags for crypto.o:
log the entire build, find where it links crypto.o and use the same command
from the shell.  It will be easier to try different flags combinations that
way.

--
ST4096-RIPE

Re: "New" vs. "old" console behavior: bug or feature?

Max Lapshin — 2013-05-23T20:25:55

I'm deploying some of erlang servers with capistrano, so I had to add:

default_run_options[:pty] = true

in deploy.rb to enable everything working as usual.

It really looks like a black magic for me: why does erlang behaviour 
differs if it is ssh or ssh -t =(