gmane.comp.java.clojure.storm

Trident: Querying on a partitioned state.

2013-05-24T08:58:48

Hello,

I have a state partitioned into 3. When I query the state I find that my 
stream used for querying is automatically partitioned into 3. To illustrate 
this point here is the sample code.

TridentState state = topology.newStream("spout",new FixedbatchSpout())
                                          .partitionBy(new 
Fields("some_field"))
                                          .partitionAggregate(new 
Fileds("some_fields..",""...), new TimeSlotBasedAggregator(), new 
Fields("slot","value"))
                                          .partitionPersist(....)
                                          .parallelismHint(3);

topology.newStream("tick_stream",new TickSpout())
            .stateQuery(state, . . .. )
            .parallelismHint(4);

My question is, in how many parts the state will be partitioned?

Regards
Kshitiz

Re: autogen.sh: error: could not find pkg-config. pkg-config is required to run autogen.sh.

Michael Vogiatzis — 2013-05-24T08:30:19

sudo apt-get install pkg-config doesn't work? We had no problems installing 
JZMQ on Ubuntu 12.04

Regards,
Michael Vogiatzis
< at >mvogiatzis <https://twitter.com/mvogiatzis>

On Thursday, 23 May 2013 17:59:57 UTC+1, SAG_Developer wrote:

Re: Re: performance implication when trident merges multiple functions into a single bolt

Viral Bajaria — 2013-05-24T08:29:16

If you look at my GC logs, it is definitely happening very quickly. For
example at 9:21:59, GC kicked in 12 times in that 1 second timeperiod. And
I agree with your observation that even though it's concurrent there can be
thread contention. I run all my topologies with persistentAggregate. Not
all of them slow down (or don't appear to) but the one's that see a lot of
incoming data do slow down considerably over a period of time. I can try
and see if memory leak is a culprit in my case too.

If you don't mind me asking, how many messages per second are you
processing ? And are you storing it to a NoSQL store like Cassandra, HBase ?

On Fri, May 24, 2013 at 1:15 AM, Jason Jackson <jasonjckn-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Re: performance implication when trident merges multiple functions into a single bolt

Jason Jackson — 2013-05-24T08:15:28

I am noticing 3 different topologies that use persistentAggregate will work 
fine at first, but it looks like there's a memory leak somewhere because I 
see used memory fill up slowly, and after 2 hours of running, time spent in 
GC is 30+ seconds. Debugging this now. Do you think this could be the same 
issue you're hitting? I saw your GC stack traces, but they were for the 
concurrent collector (not stop-the-world type GC) although it could still 
be causing serious thread contention that could be misleading you and it's 
actually a GC problem. Definitely looks like a memory leak on my end.

I am also seeing batch retries, but I don't know if that's that's a cause 
or effect.  

On Friday, 24 May 2013 00:46:33 UTC-7, Viral Bajaria wrote:

Re: Re: performance implication when trident merges multiple functions into a single bolt

Viral Bajaria — 2013-05-24T07:46:33

I am not sure if I have solved it yet (see below). I had a brief period in
which I was able to get rid of these pauses but I feel I am hitting them
again.

I profiled my code base and noticed that a lot of it was waiting for a slot
in the LMAX ring buffer. After reading up on ring buffer, I figured my
bottleneck was the number of lines that I was emitting for each input line.
I bumped up the buffers and noticed significant throughput increase.

But as I type this, I noticed pauses in my production system again today. I
have done a new deployment recently so not sure if it's a side effect of
that but I will need to debug more before coming to a conclusion.

Have you hit similar issues ? Any pointers ?

Thanks for reaching out.

-Viral

On Fri, May 24, 2013 at 12:36 AM, Jason Jackson <jasonjckn-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: performance implication when trident merges multiple functions into a single bolt

Jason Jackson — 2013-05-24T07:36:53

Any update on this thread Viral Bajaria ?

On Tuesday, 12 March 2013 17:06:49 UTC-7, Viral Bajaria wrote:

Multiple topologies and worker count

2013-05-23T21:46:21

Hi, in our storm (0.8.2) cluster -- we have 8 supervisor machines and each 
machine by default configuration has 4 slots (supervisor.slots). So our 
total slots are 32. We are deploying two independent topologies and each 
topology has topology.workers set to 8. Does this mean that 16 out of the 
32 slots will be used across the clusters for both the topologies? 

In our environment, I run into the issue where once the first topology is 
submitted (and running fine), the second topology does not start. When I 
explicitly make the topology.workers=6 for 1st topology and 
topology.workers=2 for 2nd topology, they work fine together. I was under 
the impression that # of slots = # of JVMs = # of total workers in the 
cluster? Is that assumption right? If so, I would think that both 
topologies should be able to start independently on the same cluster with 
num of workers set to 8 each

Thanks,
Suyog

autogen.sh: error: could not find pkg-config. pkg-config is required to run autogen.sh.

SAG_Developer — 2013-05-23T16:59:57

I am trying to setup a storm cluster on Ubuntu 12.04 server by following 
this 
link https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster

I have ZeroMQ installed
Running autogen.sh in the jzmq source folder is throwing an error.* autogen.sh: 
error: could not find pkg-config.  pkg-config is required to run autogen.sh.
*
*
*
*What do I need on my server to by pass this issue?*

Re: Combining the output of two Trident streams

Nathan Marz — 2013-05-23T21:35:14

Use TridentTopology#merge


On Thu, May 23, 2013 at 1:55 PM, <andy-tM+69RX+o3rB10RR8pdPRgC/G2K4zDHf< at >public.gmane.org> wrote:

Combining the output of two Trident streams

2013-05-23T20:55:45

If I have two Trident streams that produce the same fields as output, but 
at different rates, is there a way to combine their output?

So S1 has fields X, Y and Z and tuples entering it: 
x1, y1, z1
x2, y2, z2
x3, y3, z3
.
.
.

And S2 has the same fields X, Y and Z, with tuples:
x10, y10, z10
x11, y11, z11
.
.

Can I combine them to a new stream containing:

x1, y1, z1
x2, y2, z2
x3, y3, z3
x10, y10, z10
x11, y11, z11
.
.
.

I can see there's the TridentTopology.multireduce method, but I'm not sure 
if that'll do the job, or what the implications are for the flow of tuples 
from
each input stream?

Re: trident batch question

Sunil Yarram — 2013-05-23T20:19:55

Thanks Nathan for answering my questions.

So trident guarantees the commits happening in partitionPersist or 
persistentAggregate to be sequential across batches?

Thanks,
Sunil.

On Wednesday, May 22, 2013 11:00:45 PM UTC-7, Nathan Marz wrote:

Ack/Fail counting when topology fails

phlp — 2013-05-23T17:21:31

We use BaseRichSpout and BaseRichBolt in our application.  

The spout counts emitted tuples and acks/fails to derive 
state information when emitted-count == sum(ack-count, fail-count).

When a tuple is emitted in one topology which is then restarted before the 
tuple is ack'd by the bolt, will the new spout's ack routine be called when 
the tuple is re-emitted by storm and processed and ack'd by the bolts?  

I.E. will a spout get an ack for a tuple that it did not directly emit?

I'm assuming the answer is yes so that (for instance) external state 
associated with the tuple is cleaned up.

Re: trident batch question

SKG — 2013-05-23T16:21:29

Nathan - Pl clarify the parallelismHint is for each batch and if 4 batches 
are running parallel then the no of tasks running in parallel will be 
4*parallelismHint.

On Thursday, May 23, 2013 11:30:45 AM UTC+5:30, Nathan Marz wrote:

Re: The JVM gets killed unexpectedly in local mode during Storm shutdown

San — 2013-05-23T10:27:25

Kyrill,

Did u get any solution/fix for this issue?

Regards,
--Sanjeev


On Thursday, 14 June 2012 23:21:06 UTC+5:30, kyrill007 wrote:

Re: trident batch question

Nathan Marz — 2013-05-23T06:00:45

1. Yes, use TOPOLOGY_MAX_SPOUT_PENDING to control the number of batches
that can be processing at once
2. Yes, that's correct.


On Wed, May 22, 2013 at 6:22 PM, Sunil Yarram <yvsunil-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Trident ParallelismHint Set But Only One Work

2013-05-23T02:54:42


On Wednesday, May 22, 2013 11:10:38 PM UTC+8, art...-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org wrote:

Re: About setting RotatingMap ExpireTime

2013-05-23T02:03:24

Thank Nathan Marz's help
   I found a case use RotatingMap in Storm source code and reviewed 
RotatingMap  Code, got its mechanism.

 


在 2013年5月23日星期四UTC+8上午6时05分52秒，Nathan Marz写道：

trident batch question

Sunil Yarram — 2013-05-23T01:22:19

Hi,
  I have few basic questions on trident processing:

1. The trident document mentions only one batch will be in commit phase, 
does that mean there can be many parallel batches doing processing but not 
doing the commit?
2. If trident is processing one batch at a time then the parallelism is 
achevied by dividing the batch into partitions?  and i am guessing the 
number of partitions are controlled by parallelismHint() method.

Thanks,
Sunil.

Re: About setting RotatingMap ExpireTime

Nathan Marz — 2013-05-22T22:05:52

Use tick tuples and manually rotate the map.


On Wed, May 22, 2013 at 12:57 AM, <thiswym-9Onoh4P/yGk< at >public.gmane.org> wrote:

Re: (Newbie) Is there a very simple example to get started?

Nathan Marz — 2013-05-22T22:05:37

https://github.com/nathanmarz/storm-starter


On Wed, May 22, 2013 at 1:03 PM, Miguel Durazo <miguel.durazo-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

Re: spout stopped send message, blocked in SpoutOutputCollector.emit()

Nathan Marz — 2013-05-22T22:04:51

Upgrade to the latest version of Storm


On Wed, May 22, 2013 at 1:15 AM, 陈竞 <cj.magina-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote: