gmane.network.zeromq.devel

Re: [crossroads-dev] XS over serial port

Thomas Taranowski — 2012-05-26T03:18:02

Yes, quite a few actually for embedded systems.  There are quite a few
high speed rs422/485 serial interfaces still being developed.  The
primary reason for doing this is that I already have to define a
framing and messaging protocol, and resultant code, to interface with
those devices, so I'd rather just reuse the zmq protocol and pull
those into the fold so I can interface with them directly through and
get all the zmq benefits.

On Fri, May 25, 2012 at 3:41 PM, Pieter Hintjens <ph< at >imatix.com> wrote:

Re: [crossroads-dev] XS over serial port

Pieter Hintjens — 2012-05-25T22:41:49


Tom, the plan is just to make small, safe iterative changes to libzmq.
This is the main reason I didn't port the session changes back from xs
to zmq. So if you're more comfortable modifying the tty_engine, that's
fine. If you prefer to modify the codecs, that's also OK.

Are there actual use cases for a tty:// transport? (Flashback to 1990
and serial UART programming...:)

-Pieter

Re: [crossroads-dev] XS over serial port

Thomas Taranowski — 2012-05-25T21:43:48

Thanks for the patch Martin.  I am working on pulling this over into
the zmq fold, and it's pretty clean with one snag.  I notice that the
codecs(decoder_t, encoder_t) for crossroads take a session instance,
while zmq takes an inout.  While zmq has a session, it's not been
pushed into the codec interfaces.

My question:
Should I look at adding support to the codecs for taking a session
instance, or should I stick with the current inout interface?

For the time being I modified the tty_engine to use inout, rather than
codecs, but it's not clear to me yet which path is more consistent
with plan for zmq.

Thanks,
Tom

Re: 3.1 stability status

Pieter Hintjens — 2012-05-25T16:21:02


It's definitely stable enough to develop against, it passes all the
PyZMQ regression tests properly, and the remaining open issues are all
marginal in one way or another. Probably next week I'll cut a stable
release candidate.

-Pieter

Pub/Sub over PGM with many topics

Ambalu, Robert — 2012-05-25T14:00:07

Hi, I'm new to zeroMQ and am trying to understand how pub.sub over multicast would work in my scenario.
I want to have one publisher which publishes say 10k different topics.  I can have N subscriber processes, each of which can subscribe to multiple topics
So, use case example.  Say my publisher is publishing market data for 10k different instruments, my subscriber app only handles 100 stocks so it only want to get messages for those 100 stocks.  In addition, I don't want to have my app do any type of topic resolution, but if possible let zeromq handle the topic multiplexing.
So I was thinking on the subscriber side, for every topic I have it looks like I would have to create a separate SUB socket to the same pgm multicast channel, but set with a different SUBSCRIBE topic.  Then I would have to poll on all sockets ( essentially, all topics ) and then dish out to the appropriate pollitem ( which in this case would be my object, so say data for AAPL would be subscribed to my an AAPL pollitem handler which

PUSH/PULL not working in this example

Sam Zaroug — 2012-05-25T08:01:15

Hi Folks,


I ve the following client server executables that arent working on my machine (attached)


please, note: 
- replace boost  calls with std::cout if you dont use boost in your dev env.


the reciever seems to wait forever, and the sender exit without waiting for the message being processed.
if you uncomment the _sleep(1) on the sending side things works just fine.


I wondering if there is any logical explanation to this simple senario?




Kind Regards


Brownviper1966






       _______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: 3.1 stability status

Chuck Remes — 2012-05-25T15:01:26


3.1 will probably be released very soon (next week or two). You can see the remaining "big issues" by taking a look at the JIRA issues open for it. 

Based upon irc discussions and questions on this list, there are already quite a few people using it (github master is essentially 3.1 right now). I haven't heard of any major problems.

Give it a try and let us know how it works for you.

cr


_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: several binds on the same ipc

Ian Barber — 2012-05-25T14:38:42

In 2.2 I'm seeing the last bind win (at least within the same process),
which kind of makes sense as there is some code that just blows away the
IPC file if it already exists, which I believe is mainly there to allow
handling previous runs that did not exit properly and clean up after
themselves.

Ian
_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

3.1 stability status

2012-05-25T14:24:13

Hello,

I plan to bring ZeroMQ into my codebase to start testing out its pub/sub
capabilities. I had brought in ZeroMQ 2.1 in a former project and found
it a bit painful to resolve the uuid dependency on the embedded Linux
platform we use. I hear 3.1 no longer contains that dependency and am
interested in jumping to that revision if it is fairly stable. There are
also some other features in 3.1 that I could benefit from. Is there a
timeframe as to when 3.1 will be considered the new "stable release"?
Are there any big issues with it today?

 

/Jon

 

_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: several binds on the same ipc

Tarek Ziadé — 2012-05-25T07:27:55

So here's the C demo Benjamin wrote to demo this:

http://tarek.pastebin.mozilla.org/1649255

Please let us know if that looks correct (this program behaves as I 
described)


Unfortunately, I have zero knowledge of the zeromq code and I am quite 
rusty in c/c++ :)

If someone hints me where to look I can try...

Cheers
Tarek

Re: several binds on the same ipc

Pieter Hintjens — 2012-05-24T21:19:54


If this is indeed the behavior, it sounds like a bug. The code should
maybe check first off that the IPC endpoint isn't already used. Or
perhaps there's an error that's not being checked.

Either way, if you want to send a patch, that'd be welcome.

-Pieter

Re: Code-style question

MinRK — 2012-05-24T19:44:28

On Wed, May 23, 2012 at 4:42 AM, Marco Trapanese
<marcotrapanese< at >gmail.com>wrote:


In IPython, we abstract serialization into a single object, where
serialization is any function that turns a Message (in our case, dicts)
into a sequence of binary blobs.  This lets us switch between JSON, pickle,
msgpack, etc. as appropriate with a single line of config.  This lets us
use JSON for easy human-readable debugging, and msgpack (or blosc or
protobuf) for better performance.  The application code exclusively talks
high-level messages, and there are calls like `session.send(socket,
message)` and `message = session.recv(socket)`.

This sort of thing may be more trouble than it's worth to implement in C,
but it's dead easy in higher level languages, and I strongly encourage it,
as it makes the question of serialization entirely isolated from the rest
of your application, as it should be, and it also lets us put any
authentication, message metadata construction, etc. in the same place at a
later time, without touching

Re: several binds on the same ipc

Tarek Ziadé — 2012-05-24T17:23:49

Yeah, sorry for the false conclusion,

I am not sure what's the rule here but it appears that the first one to 
bind is the winner.

I am guessing this is a bug in zmq, right ?

Cheers
Tarek

_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: several binds on the same ipc

MinRK — 2012-05-24T16:24:23


As discussed there, it has been confirmed that pyzmq behaves no differently
from C in this regard, so it is not related to pyzmq.

You can bind multiple sockets to a single ipc channel, but only one will be
active at a time.

-MinRK


_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: several binds on the same ipc

Tarek Ziadé — 2012-05-24T14:17:20

After a bit of discussion on IRC, this seems to be a pyzmq issue, where 
the second
bind() call does not raise an error, as expected.

So I opened a bug there: https://github.com/zeromq/pyzmq/issues/209

Cheers
Tarek
_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: several binds on the same ipc

Tarek Ziadé — 2012-05-24T14:19:12

After a bit of discussion on IRC, this seems to be a pyzmq issue, where 
the second
bind() call does not raise an error, as expected.

So I opened a bug there: https://github.com/zeromq/pyzmq/issues/209

Cheers
Tarek
_______________________________________________
zeromq-dev mailing list
zeromq-dev< at >lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

several binds on the same ipc

Tarek Ziadé — 2012-05-24T12:17:58

Hey

I am trying to solve an issue I have with a broker that opens a zmq 
router socket and just relay calls to workers via a socket dealer.

The problem is that I don't know how to prevent a second broker to bind 
itself to the same ipc.

Consider the following scripts where I isolate the issue:

The server: http://tarek.pastebin.mozilla.org/1649175
The client: http://tarek.pastebin.mozilla.org/1649176

If I run them they happily interact. Now I can run a second server. It 
will bind the socket but won't do anything or even error out.

If I stop the first server, the client will simply lock, as it seems to 
have a sticky connection to the first server - and won't communicate
with the second server until I restart it.

As far as I understand, we can only bind() *once*, which make sense.

So my question is - shouldn't the second attempt to bind() the socket 
raise an error ? maybe this is related to the python binding ?

If not, how can I know if there's already a socket bound to the ipc ?

Cheers
Tarek

Re: exhausting limited FDs

Brian Smith — 2012-05-24T05:57:53



Only you can really decide for yourself if it's a problem. There are people
using Zeromq
for which it is not an issue, even well above the scale you mention in your
first post.
Everyone has different limitations. In some cases, no matter how
parsimonious you
are with your CPU and RAM, you'll never be able to reach even 5K clients
per process
anyway.

Even so, if you have ideas about how to improve the FD usage of Zeromq, I
believe the
standard response around here is: Pull requests accepted [gladly]!



Pull requests for documentation improvements also gladly accepted!
Having such things noted is a pretty good idea. I _think_ it's already on
the wiki, but, noting it specifically in the man pages provides another way
to find the information.



In a future when machines have 256 or more cores, and several hundred
gigs of RAM, figuring out how to use all those resources effectively will be
more challenging than dealing with the number of FDs in use.

-B


_______________________________________________
zero

Re: exhausting limited FDs

Chick Corea — 2012-05-24T05:45:28

Then you acknowledge the problem?  And the recommended solution is to
increase the FD-max instead of use FDs efficiently.  Something in the
docs about this particular limit would be helpful since many
programmers consider CPU and RAM but not FD exhaustion.

Thanks for the clarification.

Although I can't help but wonder about what this means for future
systems (5yrs; 10yrs) in which we might take for granted 1M FDs.  We
seem to always be trading one set of problems for another.

CZC

On Wed, May 23, 2012 at 10:06 PM, Pieter Hintjens <ph< at >imatix.com> wrote:

Re: exhausting limited FDs

Pieter Hintjens — 2012-05-24T05:06:14


Raise the per-process FD limits on your machines.

-Pieter

exhausting limited FDs

Chick Corea — 2012-05-24T03:02:25

I have a non-hypothetical concern re: FD usage with ZMQ.  Admittedly,
I am a typical programmer having implemented most inter-proc comm via
bare sockets.

As I understand the general use-model, a program would would use
multiple TCP ports for the various message-types to be sent between
servers and clients, PUB/SUB, REQ/REP, PUSH/PULL.  That being the
case, comm between two procs, a server could easily use 2 or more
sockets for each client.  If a proc is limited to 5k FDs then the
number of clients simultaneously connected is limited to less than
2.5k before it has to start managing those FDs with some kind of LRU
algorith with reconnects as necessary.

That seems to be the inevitable outcome of not being able to map an
endpoint to multiple sockets.  Is that correct?

What's the solution?   Is that actually acceptable?

Thank you.

FWIW, ZMQ seems very promising to use in my current project.  I have
two key concerns re: it; this is one of them.

CZC