gmane.comp.java.clojure.storm

Re: Q? How to declare output fields for a List of Strings ...

Noel Milton Vega — 2013-06-19T00:42:19

With apologies, I realized the answer to my question shortly after posting:


Rather than doing this:     
    collector.emit(myTupleBatch); // List<String>


which results in a Tuple with "myTupleBatch.size()" number of fields in it,
I should instead be doing this (for this case):

    collector.emit(new Values(myTupleBatch));

which results in a Tuple with a number of fields equal to exactly one
(i.e. a 1-way Tuple).

That one field, of course, is a Collection of Objects containing many
elements.

Thanks & Best Regards,
NMV

How do you perform cleanup when killing a topology?

2013-06-18T22:05:03

Hi,

I'm using storm to write output to hdfs, and I was wondering how I could 
prompt storm to close the connections when the topology is killed.  I'm 
expecting this topology to be running at all times, but if I need to shut 
it down I'd like to be able to clean up its connections.  It seems odd to 
me that killing a topology doesn't trigger any of the cleanup code.  If the 
kill command doesn't set it off, what other methods are there for 
triggering cleanup?

Thanks,
Brian

Porting Storm to Windows

Min Wei — 2013-06-18T18:21:51

Hi, all. 

We are impressed by Storm and use Storm internally for some prototyping 
work on Windows based clusters. 

During our usage, we have made a set of changes which we would like to 
share with the community. Please find the attached diff file to get an 
overall picture. Separately we will make a set of pull requests in Github. 
Any feedback on the changes will be greatly appreciated. 

Besides addressing Windows specific file separator and task kill issues, we 
added the following:
    * add a set of Windows cmd scripts to launch and stop the Storm cluster 
without the Python dependency 
    * add a pseudo distributed mode for single node development 
    * add an API to allow user code jar to be submitted from Web client/UI 
    * forward update ZooKeeper change to 3.4.5 (this will be in a separate 
pull request for sure). 

Please review the changes. 

Best regards,  --min

Question about Statefull tasks handling when Work redistribution occure (when a worker process dies).

2013-06-18T20:02:46

Hi
 
In the case where a bolt task is statefull how is the state maintained if 
the worker where the task is runnign fails or get distroyed (machine just 
dies for unnatural reasons)?
 
Is it up to the developer of the bolt funtionality to keep info in DB so 
that when the task is restored on another system, it continues the 
processing formt hat state? or does Storm manage this somehow via 
snapshopts on memory?
 
Thanks
 
Fares

storm-yarn

Xavier Jodoin — 2013-06-18T20:14:45

I'm trying to deploy storm on yarn but I need storm-0.9.0-wip17. 
The build exist in maven repository but I cannot find the zip?

Thanks

Xavier

Policy for retrying of a batch in a transaction-topology after a timout.

2013-06-18T20:11:11

Hi

If a transaction topology does not complete in 30 sec (default timeout) the 
batch will be retries (to guarantee execution)....but is there a limit on 
retries?

The concern is that if the system just cannot handle the amount of data in 
that batch in less than 30 sec, then either the timeout need to be 
increased or the batch needs to be reduced, or the system need to be 
optimized with less bolts...etc....
 
Retrying the same thign again and again will probbaly fail continuously and 
then block the system.
What is the recomended policy to handle that.....monitor alarms, CPU, check 
nb-retries or kill topology and re-launch with smaller batches or bigger 
timeouts? can storm auto-heal and kill the topology itself after x retries?
 
 
Thanks
 
Fares

Re: Q? How to declare output fields for a List of Strings ...

Noel Milton Vega — 2013-06-19T00:04:01

Hello again:

I have a follow-up question.

Let's say I do this in the Spout (or Bolt):

    collector.emit(myTupleBatch);

where myTupleBatch is a "List<String>" with 100, or 1,000
or 10,000 fields/elements (as determined bysome setting in a
".properties"file for batchingof tuples).


For the case above, do I then have to declare as many
named-fields, like so?:

   declarer.declare(new Fields(myTupleFields));

where myTupleFiends is also a "List<String>"that contains
100, or 1,000, or 10,000 (respectively) Strings for field-names?

This implies the need for aloop in the declareOutputFields() method
to initialize "myTupleFields" with, say, these strings:
      "1", "2", "3", "4", ..., "99", "100"...


:-o


Excluding Trident (too late for that for my demo), is there
a better way to do this?


Thank you. :)
NMV

Re: Calculating Top 10 Results.

Ted Dunning — 2013-06-18T19:07:15

See streamlib [1] and Brickhouse [2] as well.

[1] https://github.com/clearspring/stream-lib
[2] https://github.com/klout/brickhouse


On Tue, Jun 18, 2013 at 6:41 PM, Colin Surprenant <
colin.surprenant-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Kafka Opaque Transactional Topologies and exporting to S3

ryan medlin — 2013-06-18T18:31:46

We are trying to implement exactly one processing and we are not sure how 
to really go about this.

We are using Clojure fo our entire stack and processing up to 1 billion 
events per day in our Kafka and storm clusters.

There really aren't alot of Clojure examples out there for doing this but 
its easy enough to just use the Java examples and wrap them in Clojure.

We need to pull off all of the json event messages and then batch them up 
and upload them to Amazon s3 to then be processed from Amazon MapReduce and 
loaded into our analytics database.  Yes so we are basically using storm as 
a way to move data to various other external systems right now. 

This is not an analytical system but an accounting system so the counts 
need to be accurate.  We are working on doing some pre-aggregations as well 
but we mainly want to allow other teams to digest this stream in a clean 
and simple way the way they want and need to show them a few recipes on how 
to implement the exactly once processing.

We need to ba

Re: Q? How to declare output fields for a List of Strings ...

Noel Milton Vega — 2013-06-18T17:54:37

Perfect Andy... 
That "glue" what I was looking for. Much appreciated.

Regards,
NMV

Re: Calculating Top 10 Results.

Colin Surprenant — 2013-06-18T16:41:51

You can also take a look at how tweitgeist
https://github.com/colinsurprenant/tweitgeist works for computing the top
trending hashtags on Twitter. This project uses Ruby/JRuby with RedStorm
https://github.com/colinsurprenant/redstorm but you can adapt the idea to
your context.

Colin


On Mon, Jun 17, 2013 at 11:51 PM, Bhagwant Bhobe
<bhobe.bhagwant-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

Zookeeper NoNode exception w/ TransactionalTridentKafkaSpout in Local mode

Xinh Huynh — 2013-06-18T16:35:20

Has anyone encountered this problem? The "TransactionalState" is trying to 
delete a ZK node that is no longer there. This happened right after a bunch 
of ZK connection errors and a reconnect. Is it possible the 
"cleanupBefore()" method was called twice on the same ZK nodes? Would it 
make sense to add a check before the delete that checks if the node the 
exists? I'm only running one topology.

ERROR backtype.storm.util - Async loop died!

java.lang.RuntimeException: 
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
NoNode for /my-dev.com:9092:1/24830
        at 
storm.trident.topology.state.TransactionalState.delete(TransactionalState.java:75)
        at 
storm.trident.topology.state.RotatingTransactionalState.cleanupBefore(RotatingTransactionalState.java:99)
        at 
storm.trident.spout.PartitionedTridentSpoutExecutor$Emitter.success(PartitionedTridentSpoutExecutor.java:124)
        at 
storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:62)
        at

Re: storm doesn't emit all the tuples

2013-06-18T16:33:15

I would check your code for thread safety first - if you're removing things 
from an ArrayList at the same time as they're being added by another 
thread, you may get all sorts of side effects. I'm assuming that you've 
ensured that each worker reads a different file? Finally, add lots of debug 
- check how many items are being read, how many are stored in your array 
list, how many are removed and emitted. Then you can see where they are 
being lost.

Andy

On Tuesday, 18 June 2013 07:39:04 UTC+1, ayush choubey wrote:

Re: Re: __acker task doesn't start

Philip O'Toole — 2013-06-18T16:26:54

You've truncated the bottom of your main snapshots. It's missing what
I need to see.

On Tue, Jun 18, 2013 at 9:06 AM, Christian Würtz
<christian.wuertz-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: __acker task doesn't start

Christian Würtz — 2013-06-18T16:06:00

I've made two new screenshots. One of the main and one of the topology page.

On Tuesday, June 18, 2013 11:56:07 AM UTC+2, Christian Würtz wrote:

Re: __acker task doesn't start

Philip O'Toole — 2013-06-18T15:47:28

Show us the main page of the Storm UI too. 

Philip

On Jun 18, 2013, at 2:56 AM, Christian Würtz <christian.wuertz-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Re: Q? How to declare output fields for a List of Strings ...

2013-06-18T15:45:11

emit() and declare() do different things at different times.

When you are building a topology, declare() returns a name for each of the 
'fields' in the tuples that your bolt will produce. Those names can be used 
when connecting one bolt up to another.

When you run a topology, emit() returns a tuple that contains values for 
each of the 'fields'.

So you might create a bolt that declares it returns two values in each 
tuple, called name and count respectively:

.. 
return new Fields("Name", "Count");

And then emit could return values for those fields:

String name = "George";
int count = 43;

collector.emit( new Values(name, count));

(Obviously a very boring bolt, as it just emits a tuple with the values 
"George", 43 again and again). 

Andy

Re: Topology stop processing tuples after a while

Philip O'Toole — 2013-06-18T15:44:42

Bug in your code? Perhaps you're not acking every message. 

On Jun 18, 2013, at 7:18 AM, Sergey <sergey-aXRFzBXuJn4kmLvzuZlaBw< at >public.gmane.org> wrote:

Re: Q? How to declare output fields for a List of Strings ...

Noel Milton Vega — 2013-06-18T15:37:52

The missing concept finally came to me after digesting the hint that SKG provided below a little further, alongside the Javadoc. Thank you SKG.

Regards,
SKG

Re: connection refused error while submit a topology to a cluster

Aimee Cheng — 2013-06-18T15:17:25

If you want to submit a topology by a storm client, it use the storm.yaml 
file under ~/.storm/storm.yaml , but not conf/storm.yaml. ref: 
https://github.com/nathanmarz/storm/wiki/Setting-up-development-environment, 
you can add a storm.yaml under ~/.storm.

Aimee

On Tuesday, June 18, 2013 8:07:12 PM UTC+8, Allon Yao wrote:

Topology stop processing tuples after a while

Sergey — 2013-06-18T14:18:53

Hi everybody,

I was doing a sample topology 1 spout 1 bolt. Spout grabs messages from 
Rabbit cluster, and pass it to the bolt. Almost no proccessing in the bolt, 
just store the result in MySQL.  It was working fine for some time, but 
than after a while after I stored about 1 million messages, bolt just stops 
processing messages, spout reached its MaxSpoutPending(10);  and no 
processing happening. By the way I'm trying it on the cluster 1 nimbus, 1 
zookeeper, 1 supervisor ( m2.4xlarge with 64 G of memory and a lot of CPU ) 
. 

I checked the log files of everything (supervisor, nimbus, worker) looks 
fine.

When I restarted the topology, it starts working again