gmane.comp.python.twisted

Weekly Bug Summary

2012-05-20T04:05:33

_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Getting rid of "DirtyReactorWarning"

2012-05-18T20:21:39

I think you should talk to jml about his ideas for a new base 
`TestCase`-like class which is better factored with respect to its 
handling of Twisted-related reponsibilities.

I don't think just removing `DirtyReactorWarning` from all the places it 
is currently emitted is a good idea (if only because it is providing 
*us* value, and because new behavior merits new interfaces).

More generally, I don't think ReactorBuilder is the best approach for 
unit testing anything except reactor implementations, just as I don't 
think using a real reactor (with real time, real networking, etc) is the 
best way approach for unit testing application code that uses the 
reactor.

We need to keep improving our test doubles, documenting their usage, and 
porting our own test suite (the application-y parts, at least) to them.

Jean-Paul

Re: Synchronous calls using Twisted?

2012-05-18T20:17:07

If "the usual way" is as a WSGI application, then blockingCallFromThread 
is probably the best fit.

Django is intended to be used in a multi-threaded (or multi-process) 
environment, not an asynchronous one.  It can't take advantage of 
Twisted's single-threaded asynchronous APIs (eg Deferreds).

Based on your original email, I think you had a pretty good grasp of 
what needed to be done, but it sounds like you got hung up on unit 
testing.

The unit tests work the same way as the actual application works.  That 
is, in actual deployment, you have one thread running the reactor and N 
threads, each running the Django application.  You use 
blockingCallFromThread in any of those N threads to talk to the one 
reactor thread.

The most obvious way for your unit tests to work is for them to do the 
same thing.  Run the reactor in one thread, run your (Django-based, 
blockingCallFromThread-using) unit tests in a different thread.

The deadlock you mentioned will go away, because now the code is 
executing as ex

Re: Synchronous calls using Twisted?

Louis — 2012-05-18T16:32:04

Hello,
On 17/05/2012 02:00, Jasper St. Pierre wrote:
Once again, maybe I *cannot* make Django do asynchronous calls simply
because I am not used to Twisted and asynchronous calls.

 +----------+   +-------------+   +------------+   +-----------+
 | Database |---| Core Server |---| Web Server |---| Web client|
 +----------+   +-------------+   +------------+   +-----------+

A bit more context.
What I call "core server" is more or less a proxy to a database. It
exposes some high-level functions to access and manipulate the database,
doing some additional checks to ensure that user is allowed to do what
he is trying to do, and ensuring that he does not mess up the integrity
of the database. I am trying to make this server be a Twisted server.
The "web server" is built using Django. It has access to the database
only using the high-level functions exposed by the core server.
Until I got stuck with this synchronous/asynchronous questions, I
thought Twisted would be a nice tool to use to make the core serve

Re: Synchronous calls using Twisted?

gelin yan — 2012-05-18T04:06:55


Hi Andrew

    What is the advantage of using stackless with twisted?

Regards

gelin yan
_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Synchronous calls using Twisted?

Mike Winter — 2012-05-17T20:17:43

half async/half sync is not as obscure as you may think:

www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf

On May 17, 2012, at 12:06:04PM, Andrew Francis wrote:


_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Synchronous calls using Twisted?

Andrew Francis — 2012-05-17T19:06:04

Message: 1

Date: Thu, 17 May 2012 00:58:57 +0200
From: Louis <spalax< at >gresille.org>
Subject: [Twisted-Python] Synchronous calls using Twisted?
To: twisted-python< at >twistedmatrix.com
Message-ID: <4FB43131.2030805< at >gresille.org>
Content-Type: text/plain; charset=ISO-8859-1

I am not sure why inline generators won't work?

Over the years, I use Stackless Python with Twisted. I use Twisted for many of the reasons you describe. For myself there is another dimension: I find that a lot of the code I write are orchestrations - that is in order to compute something, I have to make a few network calls sequentially. 

 I use a technique that Christopher Armstrong (what happened to him?) called a blockOn. For the sake of being academic (roll your eyes here),  this is an example of an obscure design pattern called "Half-sync/Half-async."

Essentially one does the following:

def blockOn(deferred): 
    ch = stackless.channel() 
    def cb(result): 
        ch.send(result) 
    deferred.addBoth(cb)

Re: Synchronous calls using Twisted?

Jasper St. Pierre — 2012-05-17T00:00:03

The point of blockingCallFromThread is as a last resort when
interfacing with a synchronous API that can't be replaced, like the
DBAPI. It fools Twisted into thinking an external synchronous API is
asynchronous, by carefully punting it to a new thread.

I don't understand why you can't make Django do asynchronous calls.
Maybe a bit more context would help.

On Wed, May 16, 2012 at 6:58 PM, Louis <spalax< at >gresille.org> wrote:

Re: Commercial support contracts for Twisted - any interest?

Itamar Turner-Trauring — 2012-05-16T23:45:39

One final reminder - if you're interested in support contracts for 
Twisted, please fill out the questionnaire linked below.

Synchronous calls using Twisted?

Louis — 2012-05-16T22:58:57

  Hello,
  I am writing an application, which architecture is (I hope my
beautiful drawing is not messed up):

  +-------------+   +------------+   +-----------+
  | Core Server |---| Web Server |---| Web client|
  +-------------+   +------------+   +-----------+

  That is, I have a core server (itself interacting with a database, but
I think this is not important). I have a second piece of software, which
is both a client regarding the core server, and a web server. At last,
the web client is the web browser of your choice.
  The web server is a Django application, which means that (as far as I
know) I cannot (easily) make it use asynchronous calls.
  At last, for the communication between the core server and the web
server (which is, from this point of vue, a client), I was looking for
something that let me use remote objects as if they were local. That is
when I discovered Twisted.



  Twisted seems to be the kind of tool I want to use because it seems to
be powerful, can use SSL, authentication, transm

Re: Declaring AMP responders away from

Allen Short — 2012-05-16T15:07:48

Have a look at what I did in Frack:

http://bazaar.launchpad.net/~washort/frack/trunk/view/head:/frack/wiring.py#L27
http://bazaar.launchpad.net/~washort/frack/trunk/view/head:/frack/responder.py#L85

You can unbundle the locator and dispatcher bits. I only used this for
a single separate class, but with some imagination the principle could
be extended to multiple objects I bet.

Declaring AMP responders away from

Laurens Van Houtven — 2012-05-16T14:26:00

Hi,

I'm writing a service that exposes a bunch of methods over AMP.

These methods are easily logically grouped into separate modules (i.e. message-related, picture-related…). So, I'd like to define the behavior for those methods in those modules.

However, the amp.Command.responder decorator only works inside the class definition of an AMP subclass.  (http://twistedmatrix.com/documents/current/api/twisted.protocols.amp.Command.html#responder).

Does that mean I should have multiple AMP subclasses? How do I get all of them to listen on the same port? Is this what the mysterious "routes" feature I've been hearing about does?

Right now I'm doing it like this, which works but isn't very pretty or nice:

user.py:
-----
class Register(amp.Command):
    arguments = […]
    response = […]
    errors = […]


def register(self, …):
    # do actual registering behavior here
-----

api.py:
-----
from project import auth

class API(amp.AMP):
    register = auth.Register.responder(auth.register)
-----


cheer

Getting rid of "DirtyReactorWarning"

Itamar Turner-Trauring — 2012-05-15T10:26:24

I have a ticket open for allowing tests that look like this, where each 
run has a new reactor instance:


def test_somethingShouldHappen(self, reactor):
      result = []
      reactor.listenTCP(...)
      self.runReactor(reactor)
      self.assertEqual(result, ["hooray"])


Further thought suggests however that this doesn't add much, 
necessarily, over the current "return a Deferred" idiom for most users. 
Experience with ReactorBuilder framework that has similar testing 
mechanism suggests it's far too easy to fail to stop the reactor. 
Moreover, things like coiterate() don't work since they rely on 
pre-imported global reactor, and given the "from twisted.internet import 
reactor" idiom this testing style won't work well for many people's code.

But! ReactorBuilder tests are superior in one way: no 
DirtyReactorWarning. You don't have to wait until all connections are 
closed, which is difficult, or make sure to cancel every timeout. This 
makes test writing simpler.

Perhaps we should get rid of DirtyRe

Re: Cannot install Twisted version 8 through PIP

Pascal Briet — 2012-05-14T14:00:18

Ok,

Thanks for your help.
Best regards,

Re: Cannot install Twisted version 8 through PIP

Itamar Turner-Trauring — 2012-05-14T13:54:00

We have a strict backwards compatibility policy; anything that changes 
or is removed will have deprecation warnings for at least 2 releases and 
one year.

So the thing to do is, upgrade to version 9, run your program and make 
sure its tests pass, and fix any code that triggers deprecation warnings 
in Twisted. Repeat with each version until you're up to date. In most 
cases this may take as little as a couple of hours of work; possibly 
none if you didn't happen to be using any deprecated APIs.

Re: Cannot install Twisted version 8 through PIP

Pascal Briet — 2012-05-14T13:35:14

Hi,

Thanks for your answer.

Well, there is no specific reason that we use the 8.2.0 version.
It's just... history. We never needed to upgrade it.

Do you think we can easily upgrade to 9/10/11/12 ??
Thanks very much,

Re: Cannot install Twisted version 8 through PIP

Itamar Turner-Trauring — 2012-05-14T13:29:23

My understanding is that pip uses PyPI, and personally I know of no one 
who removed stuff from PyPI; more likely PyPI decided to remove old 
versions of software to save space.

Any reason you're using 8.2? It's almost 4 years old at this point, and 
we've fixed a lot of bugs and added a lot of features since then.

-Itamar

Cannot install Twisted version 8 through PIP

Pascal Briet — 2012-05-14T13:14:49

Hi,

We're having some troubles today, because it seems that old packages of
Twisted are not available anymore.
We are currently relying on Twisted 8.2.0.

pip install twisted==8.2.0  is failing  (first version available seems to
be 9.0.0)
The package was still available last Friday.

Is there a critical reason explaining that it has been removed from the
servers ?

Thanks very much,
Best regards,

Re: question : json.loads failed using LineReceiver data

2012-05-14T12:47:17

Read your log message.  This is not a valid JSON string:

'{"a_flw_fwd":12,"a_flw_rtr":5029,"a_rpm":519,"b_flw_fwd":23,"b_flw_rtr":8,"b_rpm":913}'

Presumably your objective is to receive something like this instead:

{"a_flw_fwd":12,"a_flw_rtr":5029,"a_rpm":519,"b_flw_fwd":23,"b_flw_rtr":8,"b_rpm":913}

The problem is unrelated to Twisted or the console.  The sender is 
sending the wrong data, or else your parsing of it is incorrect.

Jean-Paul

Re: question : json.loads failed using LineReceiver data

Itamar Turner-Trauring — 2012-05-14T10:47:38

Perhaps the JSON spans more than one line? Printing out the repr() of 
the lines you're receiving may be informative.

question : json.loads failed using LineReceiverdata

bino oetomo — 2012-05-14T08:53:53

Dear All ..

I'm trying to received json formated text using LineReceiver

Here is my code
----START----
class ArduinoReceiver(LineReceiver):
     def __init__(self):
         print 'INIT : ArduinoReceiver'

     def lineReceived(self, line):
         line = line.strip()
         print 'Received line = '+line
         if 'Arduino' in line :
             return
         self.linedict=json.loads(line)

----STOP-----
but I got error msg in the log as :
----START----
2012-05-14 15:42:00+0700 [-] Received line = 
'{"a_flw_fwd":12,"a_flw_rtr":5029,"a_rpm":519,"b_flw_fwd":23,"b_flw_rtr":8,"b_rpm":913}'
2012-05-14 15:42:00+0700 [-] Unhandled Error
     Traceback (most recent call last):
       File 
"/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/python/log.py", 
line 84, in callWithLogger
         return callWithContext({"system": lp}, func, *args, **kw)
       File 
"/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/python/log.py",