gmane.comp.python.amqp.celery.user

Re: Re: Running a worker in unittest

Ionel Maries Cristian — 2013-06-19T17:16:44

Eg, to assert that a task actually started, and not wait forever i use
something like this:

    def _wait_for_started(self, result, tries=15):
        for _ in range(tries):
            result.backend.reload_task_result(result.task_id)
            if result.status == 'PENDING':
                time.sleep(1)
            else:
                return
        raise Exception("Task %s did not start !" % result.task_id)

But don't use that on the rabbitmq result backend - it will delete your
results. You shouldn't use that backend anyway :)

When starting the suite you should wait for celeryd to be ready (eg, in
bash: while ! manage.py celery inspect active --timeout=2; do sleep 0.1;
done) because you should start celeryd with '--purge' option as your
previous test run could fail (and leave garbage behind).

For the cleanup try to be as gracious as possible (eg, don't leave leftover
processes around). Eg, bash:

    test_cleanup() {
        set +e
        celery_manage celeryd_multi kill $CELERY_OPTS
        cat

Re: Re: Running a worker in unittest

Ionel Maries Cristian — 2013-06-19T15:12:24

It's never cool anywhere, cause it usually ends up to be a horrible shell
script mess :)

You could do the tests in project B. Before you start the tests you start a
celeryd instance from A, wait for it to be ready (use events and inspect)
and then start the test suite. You need to have some cleanup (eg: bash
signal trap) to shutdown celeryd once tests are complete (or they fail, you
interrupt etc)


Thanks,

Re: Long running tasks, SQS transport, visibility timeout, Kombu and Celery

Andres Riancho — 2013-06-19T14:43:06

ping!

On Thursday, June 6, 2013 11:37:42 AM UTC-3, Andres Riancho wrote:

Re: Running a worker in unittest

Andres Riancho — 2013-06-19T14:32:17

Looks like integration testing it not cool in the celery-users mailing list 
;)

On Tuesday, June 18, 2013 4:54:42 PM UTC-3, Andres Riancho wrote:

Re: Distributed celerybeat (Was: Celery Beat and ZooKeeper)

Matt — 2013-06-19T14:25:47

I mentioned this to Lewis on the Kazoo mailing list, but as much as I like 
Zookeeper, I think this is better done natively in Celery via the 
worker-to-worker communication. There is a strong need for the ability to 
have a distributed coordinated celerybeat job distribution system. I havn't 
taken a close look at his implementation yet, but I love the idea and would 
be extremely happy to help test it out however you end up implementing it.

On Tuesday, June 18, 2013 9:15:22 AM UTC-7, Ask Solem wrote:

how to remove task from celery

timger timger — 2013-06-19T05:52:09

hi all
    i have add some incorrect task  to a redis broker
    but i can't find some way to remove the incorrect task

Running a worker in unittest

Andres Riancho — 2013-06-18T19:54:42

List,

    I have the following setup:

* Django-Celery project A registers task "foo"
* Project B: Uses Celery's send_task to call "foo"
* Project A and project B have the same configuration: SQS, msgpack
for serialization, gzip, etc.
* Each project lives on a different github repository

    I've unit-tested calls to "foo" in project A, without using Celery
at all, just "foo(1,2,3)" and assert the result. I know that it works.

    I've unit-tested that send_task in project B sends the right parameters.

    What I'm not testing, and would like your advice is the
integration between the two projects. I would like to have a unittest
that would:

* Start a worker in the context of project A
* Send a task using the code of project B
* Assert that the worker started in the first step gets the task, with
the parameters I sent in the second step, and that the "foo" function
returned the expected result.

    It seems to be possible to hack this by using python's subprocess
and parsing the output of the worker, b

Periodic Tasks Problem

Rafael Novello — 2013-06-18T16:22:42

Hi people! I would like to ask for help from you.

I have a project in Django which uses Celery to, among other things, 
perform some tasks from time to time, periodic tasks. For this I am using 
the decorator periodic_task like this:

< at >periodic_task (name='task_name' run_every=crontab(hour="*/2"))

In the past this same task was performed every minute (using crontab minute 
= *), and after a change in the system the schedule was changed to what is 
above, every two hours.

My problem is that even after the change in the code, the task continues to 
run every minute in the production environment. And, what is worse, 
sometimes it gets a whole hour without being executed, for example - the 
task is running every minute until 2:59 am and only run again 4 am.

I already restarted the Celery and RabbitMQ and yet this behavior does not 
change.

If you can give me any hint will be of great help!

PS: The whole environment is on the same server, Celery, RabbitMQ and the 
system itself. The Celery and django-celery

Re: Distributed celerybeat (Was: Celery Beat and ZooKeeper)

Ask Solem — 2013-06-18T16:15:22

On Jun 17, 2013, at 7:42 PM, Lewis Franklin <lewis.franklin-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


Great work :)

I will take a deeper look at it.  If the implementation is proven, it may
be possible to move it away from ZK using the workers for election.

Re: Manipulate Args, Kwargs or Add Custom Data

Ask Solem — 2013-06-18T16:05:45

On Jun 18, 2013, at 4:59 PM, Ask Solem <ask-oFFgF2hNjUOoClj4AeEUq9i2O/JbrIOy< at >public.gmane.org> wrote:




Btw, it's issue https://github.com/celery/celery/issues/1281

Re: Manipulate Args, Kwargs or Add Custom Data

Tony Barbieri — 2013-06-18T16:01:11

Great, thanks Ask!


On Tue, Jun 18, 2013 at 11:59 AM, Ask Solem <ask-oFFgF2hNjUOoClj4AeEUq9i2O/JbrIOy< at >public.gmane.org> wrote:

Re: Manipulate Args, Kwargs or Add Custom Data

Ask Solem — 2013-06-18T15:59:03

On Jun 18, 2013, at 5:03 AM, Tony Barbieri <greatrgb-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


There's a feature request for this already at https://github.com/celery/celery/issues

Re: Re: Running a worker in unittest

Ionel Maries Cristian — 2013-06-19T17:16:44

Eg, to assert that a task actually started, and not wait forever i use
something like this:

    def _wait_for_started(self, result, tries=15):
        for _ in range(tries):
            result.backend.reload_task_result(result.task_id)
            if result.status == 'PENDING':
                time.sleep(1)
            else:
                return
        raise Exception("Task %s did not start !" % result.task_id)

But don't use that on the rabbitmq result backend - it will delete your
results. You shouldn't use that backend anyway :)

When starting the suite you should wait for celeryd to be ready (eg, in
bash: while ! manage.py celery inspect active --timeout=2; do sleep 0.1;
done) because you should start celeryd with '--purge' option as your
previous test run could fail (and leave garbage behind).

For the cleanup try to be as gracious as possible (eg, don't leave leftover
processes around). Eg, bash:

    test_cleanup() {
        set +e
        celery_manage celeryd_multi kill $CELERY_OPTS
        cat

Re: Re: Running a worker in unittest

Ionel Maries Cristian — 2013-06-19T15:12:24

It's never cool anywhere, cause it usually ends up to be a horrible shell
script mess :)

You could do the tests in project B. Before you start the tests you start a
celeryd instance from A, wait for it to be ready (use events and inspect)
and then start the test suite. You need to have some cleanup (eg: bash
signal trap) to shutdown celeryd once tests are complete (or they fail, you
interrupt etc)


Thanks,

Re: Long running tasks, SQS transport, visibility timeout, Kombu and Celery

Andres Riancho — 2013-06-19T14:43:06

ping!

On Thursday, June 6, 2013 11:37:42 AM UTC-3, Andres Riancho wrote:

Re: Running a worker in unittest

Andres Riancho — 2013-06-19T14:32:17

Looks like integration testing it not cool in the celery-users mailing list 
;)

On Tuesday, June 18, 2013 4:54:42 PM UTC-3, Andres Riancho wrote:

Re: Distributed celerybeat (Was: Celery Beat and ZooKeeper)

Matt — 2013-06-19T14:25:47

I mentioned this to Lewis on the Kazoo mailing list, but as much as I like 
Zookeeper, I think this is better done natively in Celery via the 
worker-to-worker communication. There is a strong need for the ability to 
have a distributed coordinated celerybeat job distribution system. I havn't 
taken a close look at his implementation yet, but I love the idea and would 
be extremely happy to help test it out however you end up implementing it.

On Tuesday, June 18, 2013 9:15:22 AM UTC-7, Ask Solem wrote:

how to remove task from celery

timger timger — 2013-06-19T05:52:09

hi all
    i have add some incorrect task  to a redis broker
    but i can't find some way to remove the incorrect task

Running a worker in unittest

Andres Riancho — 2013-06-18T19:54:42

List,

    I have the following setup:

* Django-Celery project A registers task "foo"
* Project B: Uses Celery's send_task to call "foo"
* Project A and project B have the same configuration: SQS, msgpack
for serialization, gzip, etc.
* Each project lives on a different github repository

    I've unit-tested calls to "foo" in project A, without using Celery
at all, just "foo(1,2,3)" and assert the result. I know that it works.

    I've unit-tested that send_task in project B sends the right parameters.

    What I'm not testing, and would like your advice is the
integration between the two projects. I would like to have a unittest
that would:

* Start a worker in the context of project A
* Send a task using the code of project B
* Assert that the worker started in the first step gets the task, with
the parameters I sent in the second step, and that the "foo" function
returned the expected result.

    It seems to be possible to hack this by using python's subprocess
and parsing the output of the worker, b

Periodic Tasks Problem

Rafael Novello — 2013-06-18T16:22:42

Hi people! I would like to ask for help from you.

I have a project in Django which uses Celery to, among other things, 
perform some tasks from time to time, periodic tasks. For this I am using 
the decorator periodic_task like this:

< at >periodic_task (name='task_name' run_every=crontab(hour="*/2"))

In the past this same task was performed every minute (using crontab minute 
= *), and after a change in the system the schedule was changed to what is 
above, every two hours.

My problem is that even after the change in the code, the task continues to 
run every minute in the production environment. And, what is worse, 
sometimes it gets a whole hour without being executed, for example - the 
task is running every minute until 2:59 am and only run again 4 am.

I already restarted the Celery and RabbitMQ and yet this behavior does not 
change.

If you can give me any hint will be of great help!

PS: The whole environment is on the same server, Celery, RabbitMQ and the 
system itself. The Celery and django-celery

Re: Distributed celerybeat (Was: Celery Beat and ZooKeeper)

Ask Solem — 2013-06-18T16:15:22

On Jun 17, 2013, at 7:42 PM, Lewis Franklin <lewis.franklin-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:


Great work :)

I will take a deeper look at it.  If the implementation is proven, it may
be possible to move it away from ZK using the workers for election.