gmane.comp.python.twisted

Re: deferLater with trial issue

Conway, Nicholas J — 2012-05-24T17:22:24

task.Clock() definitely worked for making the test pass, by making the
clock a keyword parameter in the MyLoop.__init__  for a class variable.

Additionally, I also experimented with making the deferLater defer a class
variable and I do a d.cancel() in the teardown.  Even did an
self.addCleanup(self.cL.d.cancel).  All worked.

I get it now.  Gotta seriously clean up / cancel / shut down / stop
listening stuff in the tearDown steps.

Thanks for helping me over the conceptual hump guys.

-Nick

Simple ssh problem, please help

Kevin Ting — 2012-05-24T16:52:25

Hi everyone, I am a beginner to Twisted so any help would be appreciated.

I got an example code from the Twisted, Networking Programming Essentials
book:

from twisted.conch import error
from twisted.conch.ssh import transport, connection, keys, userauth,
channel, common
from twisted.internet import defer, protocol, reactor

class ClientCommandTransport(transport.SSHClientTransport):
    def __init__(self, username, password, command):
        self.username = username
        self.password = password
        self.command = command
        print "Transport Created"

    def verifyHostKey(self, pubKey, fingerprint):
        print "In Verify Host Key"
        return defer.succeed(True)

    def connectionSecure(self):
        print "In Connection Secure"
        temp = PasswordAuth(self.username, self.password,
ClientConnection(self.command))
        print "Created Everything"
        self.requestService(temp)
        print "Double Check"

class PasswordAuth(userauth.SSHUserAuthClient):
    def __init__(self, user, password, connection):
        userauth.SSHUserAuthClient.__init__(self, user, connection)
        self.password = password
        print "init password auth"

    def getPassword(self, prompt=None):
        print "In getPassword"
        return defer.succeed(self.password)

class ClientConnection(connection.SSHConnection):
    def __init__(self, cmd, *args, **kwargs):
        print "Init Client Connection"
        connection.SSHConnection.__init__(self)
        print "Next line of Client Connection"
        self.command = cmd

    def serviceStarted(self):
        print "Service Started"
        self.openChannel(CommandChannel(self.command, conn=self))

class CommandChannel(channel.SSHChannel):
    name = 'session'

    def __init__(self, command, *args, **kwargs):
        channel.SSHChannel.__init__(self, *args, **kwargs)
        self.command = command

    def channelOpen(self, data):
        print ("Channel was opened")
        self.conn.sendRequest(self,'exec',common.NS(self.command),wantReply
= True).addCallback(self._gotResponse)

    def _gotResponse(self,_):
        self.conn.sendEOF(self)

    def dataReceived(self, data):
        print data

    def closed(self):
        print("###### Entered Closed #####")
self.loseConnection()
        reactor.stop()
        print("##### Done with Closed ####")

class ClientCommandFactory(protocol.ClientFactory):
    def __init__(self, username, password, command):
        self.username = username
        self.password = password
        self.command = command

    def buildProtocol(self,addr):
        protocol = ClientCommandTransport(
            self.username, self.password, self.command)
        return protocol

if __name__ == "__main__":
    import sys, getpass
    server = sys.argv[1]
    command = sys.argv[2]
    username = raw_input("Username: ")
    password = getpass.getpass("Password: ")
    factory = ClientCommandFactory(username, password, command)
    reactor.connectTCP(server, 22, factory)
    reactor.run()



*************************************************************

But after running this, I get this Unhandled error.  Which is strange
because the reactor is being successfully stopped.  Does anyone have any
clue what this means and how I should go about fixing it?  Thanks!!!!!

Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py",
line 586, in _doReadOrWrite
    why = selectable.doRead()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line
199, in doRead
    rval = self.protocol.dataReceived(data)
  File "/usr/lib/python2.7/dist-packages/twisted/conch/ssh/transport.py",
line 438, in dataReceived
    self.dispatchMessage(messageNum, packet[1:])
  File "/usr/lib/python2.7/dist-packages/twisted/conch/ssh/transport.py",
line 460, in dispatchMessage
    messageNum, payload)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84,
in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69,
in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line
118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line
81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/twisted/conch/ssh/service.py",
line 44, in packetReceived
    return f(packet)
  File "/usr/lib/python2.7/dist-packages/twisted/conch/ssh/connection.py",
line 314, in ssh_CHANNEL_REQUEST
    channel = self.channels[localChannel]
exceptions.KeyError: 0
_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: deferLater with trial issue

Itamar Turner-Trauring — 2012-05-23T23:13:35

An excellent way to do that is to not use the reactor for scheduling 
time in tests; this has the additional benefit of letting you test e.g. 
2 hour timeouts without having to wait 2 hours.

http://twistedmatrix.com/documents/current/core/howto/trial.html#auto9

Re: deferLater with trial issue

Laurens Van Houtven — 2012-05-23T22:04:00

You're supposed to clean up whatever junk you left in the reactor after your test. Do that in the tearDown method.

cheers
lvh



On 23 May 2012, at 23:30, Conway, Nicholas J wrote:

deferLater with trial issue

Conway, Nicholas J — 2012-05-23T21:30:45

Hi,

I have a class that needs to kick off a method the repeatedly gets call every so many seconds, lets say 2 seconds.

I chose to use task.deferLater to do this, but seems like LoopingCall gives similar results..

I've boiled it down to the following example
from twisted.internet import reactor, task
from twisted.trial import unittest

class MyLoop(object):
    def __init__(self):
        self.updateParameters()

    def updateParameters(self):
        d = task.deferLater(reactor, 2, self.dosomething)

    def dosomething(self):
        print "cool"

class LoopTestCase(unittest.TestCase):
    def setUp(self):
        self.cL = MyLoop()

    def test_dummy(self):
        d = defer.Deferred()
        d.addCallback(lambda x: x)
        d.callback(None)
        return d  # return a deferred just in case but happens either way
MyLoop works when running normally with a reactor.run(), but when I try run this test, I get the following error:

[ERROR]
Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x1018fbb90 [1.99891901016s] called=0 cancelled=0 Deferred.callback(None)>

So I don't fully understand what's going on to cause this.  Any pointers would be appreciated

Thanks,

-Nick
_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Multiple transmissions via SMTPClient

Jason J. W. Williams — 2012-05-23T19:10:44

Hi Guys,

I've been banging my head on this for a few days so I thought I'd ping
folks here. Been trying to modify twisted.mail.smtp.ESMTPClient to
allow the sending of more than one message over a single TCP
connection. I've changed the  behavior of smtpState_data() so that it
sets the next state to smtpState_from if there is another message to
send, and this seems to work unless there's an error returned by the
server. In this case the connection is still closed on the client side
after smtpState_msgSent, and I need to suppress this. The subsequent
message should still be sent on the open TCP connection error or not,
according to my needs, however I'm having a heck of time tracking down
where the disconnection is occurring.

My current code (with debugging statements) is here:
https://gist.github.com/2777154

Any help is greatly appreciated. This client is being used to run
functional tests.

-J

Weekly Bug Summary

2012-05-20T04:05:33

_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Getting rid of "DirtyReactorWarning"

2012-05-18T20:21:39

I think you should talk to jml about his ideas for a new base 
`TestCase`-like class which is better factored with respect to its 
handling of Twisted-related reponsibilities.

I don't think just removing `DirtyReactorWarning` from all the places it 
is currently emitted is a good idea (if only because it is providing 
*us* value, and because new behavior merits new interfaces).

More generally, I don't think ReactorBuilder is the best approach for 
unit testing anything except reactor implementations, just as I don't 
think using a real reactor (with real time, real networking, etc) is the 
best way approach for unit testing application code that uses the 
reactor.

We need to keep improving our test doubles, documenting their usage, and 
porting our own test suite (the application-y parts, at least) to them.

Jean-Paul

Re: Synchronous calls using Twisted?

2012-05-18T20:17:07

If "the usual way" is as a WSGI application, then blockingCallFromThread 
is probably the best fit.

Django is intended to be used in a multi-threaded (or multi-process) 
environment, not an asynchronous one.  It can't take advantage of 
Twisted's single-threaded asynchronous APIs (eg Deferreds).

Based on your original email, I think you had a pretty good grasp of 
what needed to be done, but it sounds like you got hung up on unit 
testing.

The unit tests work the same way as the actual application works.  That 
is, in actual deployment, you have one thread running the reactor and N 
threads, each running the Django application.  You use 
blockingCallFromThread in any of those N threads to talk to the one 
reactor thread.

The most obvious way for your unit tests to work is for them to do the 
same thing.  Run the reactor in one thread, run your (Django-based, 
blockingCallFromThread-using) unit tests in a different thread.

The deadlock you mentioned will go away, because now the code is 
executing as expected, with the reactor in a different thread than the 
blockingCallFromThread usage.

Whether this is elegant or or beautiful should be addressed separately 
from whether it is technically possible or not.  It is technically 
possible, and there aren't any new technical drawbacks of this approach 
as compared to any system built using Django (or any system built using 
Twisted, for that matter).

Another way to look at it is to consider that Django (and most, if not 
all) synchronously written systems are intended to be used with threads 
to address concurrency.  Using Twisted with Django differs very little 
from using anything else with Django.  To achieve concurrency, you use 
threads.  The only minor difference Twisted brings with it is that all 
of your Twisted usage can happen in just *one* extra thread, whereas 
using a blocking library would probably be one extra thread per 
concurrent request.

Hope that helps clear things up.
Jean-Paul

Re: Synchronous calls using Twisted?

Louis — 2012-05-18T16:32:04

Hello,
On 17/05/2012 02:00, Jasper St. Pierre wrote:
Once again, maybe I *cannot* make Django do asynchronous calls simply
because I am not used to Twisted and asynchronous calls.

 +----------+   +-------------+   +------------+   +-----------+
 | Database |---| Core Server |---| Web Server |---| Web client|
 +----------+   +-------------+   +------------+   +-----------+

A bit more context.
What I call "core server" is more or less a proxy to a database. It
exposes some high-level functions to access and manipulate the database,
doing some additional checks to ensure that user is allowed to do what
he is trying to do, and ensuring that he does not mess up the integrity
of the database. I am trying to make this server be a Twisted server.
The "web server" is built using Django. It has access to the database
only using the high-level functions exposed by the core server.
Until I got stuck with this synchronous/asynchronous questions, I
thought Twisted would be a nice tool to use to make the core server and
web server communicate.


The way I plan mixing Django and Twisted here is having Django be run
the usual way (from a Django point of view), and do calls to the core
server using Twisted. This excludes mixing Django and Twisted that way
[1] where, as far as I understand it, Twisted is "simply" the web server
calling Django.
Django being run the usual way, I can define Django functions which,
when a client is requesting a web page, are called, process data, and
return the requested web page. If I want to do a Twisted call in this
function (to get some information from the database (remember: the
Twisted server is a proxy to the database)), I must wait for the Twisted
call to return before returning the requested page.

I hope this is more clear... (more clear? clearer? Sorry... Not native
English speaker...)

But maybe there is another way of doing this. Maybe there is a way to
asynchronously call the Twisted function, getting a deferred, and adding
the "returnDjangoWebPage()" function (which makes use of the MVC
functionality of Django) as a callback to this deferred.

Is there a proper way to mix Django and Twisted the way I want it to be
mixed ?

Regards,
Louis

[1]
http://www.clemesha.org/blog/Django-on-Twisted-using-latest-twisted-web-wsgi/

Re: Synchronous calls using Twisted?

gelin yan — 2012-05-18T04:06:55


Hi Andrew

    What is the advantage of using stackless with twisted?

Regards

gelin yan
_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Synchronous calls using Twisted?

Mike Winter — 2012-05-17T20:17:43

half async/half sync is not as obscure as you may think:

www.cs.wustl.edu/~schmidt/PDF/PLoP-95.pdf

On May 17, 2012, at 12:06:04PM, Andrew Francis wrote:


_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Synchronous calls using Twisted?

Andrew Francis — 2012-05-17T19:06:04

Message: 1

Date: Thu, 17 May 2012 00:58:57 +0200
From: Louis <spalax< at >gresille.org>
Subject: [Twisted-Python] Synchronous calls using Twisted?
To: twisted-python< at >twistedmatrix.com
Message-ID: <4FB43131.2030805< at >gresille.org>
Content-Type: text/plain; charset=ISO-8859-1

I am not sure why inline generators won't work?

Over the years, I use Stackless Python with Twisted. I use Twisted for many of the reasons you describe. For myself there is another dimension: I find that a lot of the code I write are orchestrations - that is in order to compute something, I have to make a few network calls sequentially. 

 I use a technique that Christopher Armstrong (what happened to him?) called a blockOn. For the sake of being academic (roll your eyes here),  this is an example of an obscure design pattern called "Half-sync/Half-async."

Essentially one does the following:

def blockOn(deferred): 
    ch = stackless.channel() 
    def cb(result): 
        ch.send(result) 
    deferred.addBoth(cb) 
    return ch.receive() 

Of course, there is a bit more, like the Twisted reactor is running in its own tasklet. But this has the effect of nicely allowing one to use Twisted in a synchronous fashion. If you don't feel like installing Stackless Python, then you can use stackless.py (not the new version that uses continuations) with the greenlet package.

Here is a link to a complete example

http://andrewfr.wordpress.com/2011/11/30/the-santa-claus-problem-with-join-patterns/

Cheers,
Andrew_______________________________________________
Twisted-Python mailing list
Twisted-Python< at >twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: Synchronous calls using Twisted?

Jasper St. Pierre — 2012-05-17T00:00:03

The point of blockingCallFromThread is as a last resort when
interfacing with a synchronous API that can't be replaced, like the
DBAPI. It fools Twisted into thinking an external synchronous API is
asynchronous, by carefully punting it to a new thread.

I don't understand why you can't make Django do asynchronous calls.
Maybe a bit more context would help.

On Wed, May 16, 2012 at 6:58 PM, Louis <spalax< at >gresille.org> wrote:

Re: Commercial support contracts for Twisted - any interest?

Itamar Turner-Trauring — 2012-05-16T23:45:39

One final reminder - if you're interested in support contracts for 
Twisted, please fill out the questionnaire linked below.

Synchronous calls using Twisted?

Louis — 2012-05-16T22:58:57

  Hello,
  I am writing an application, which architecture is (I hope my
beautiful drawing is not messed up):

  +-------------+   +------------+   +-----------+
  | Core Server |---| Web Server |---| Web client|
  +-------------+   +------------+   +-----------+

  That is, I have a core server (itself interacting with a database, but
I think this is not important). I have a second piece of software, which
is both a client regarding the core server, and a web server. At last,
the web client is the web browser of your choice.
  The web server is a Django application, which means that (as far as I
know) I cannot (easily) make it use asynchronous calls.
  At last, for the communication between the core server and the web
server (which is, from this point of vue, a client), I was looking for
something that let me use remote objects as if they were local. That is
when I discovered Twisted.



  Twisted seems to be the kind of tool I want to use because it seems to
be powerful, can use SSL, authentication, transmits exceptions, is meant
to be used with Trial (I want to write tests), etc. The problem is that
one of the strength of Twisted is its ability to deal with asynchronous
tasks, which I am not interested in.  Using blockingCallFromThread()
[1], I managed to make a small package which looks (from my
Twisted-ignorant point of vue) promising for my application, where the
server is simply defined by (in a Twisted .tac  file):

    # Server() is the object to be exported to the client
    application = server.get_application(Server(),
                  "tcp:8800:interface=localhost")

And the client can perform something like:

    remote = client.RemoteObject("localhost", 8800)
    # display() is a method executed on the server
    remote.display("Hello, world!")
    remote.disconnect()


  But then I noticed that I might have problem to test it using Trial
(because an deadlock happens when blockingCallFromThread() and the
reactor are waiting for each other in the same thread). I also had a
look at inlineCallbacks [2], but although code looks synchronous inside
the decorated function, the function itself still returns a deferred.  I
noticed quite a few other minor things. And I started to realize that
Twisted is not meant to be used that way (and I read a bunch of threads
of the Twisted mailing list with subjects containing "synchronous",
including this long one [3]).

  My question is: What is your reaction?

a) "Nice hack! Go on!"
b) "Go on if you like, but you won't be able to use advanced Trial stuff
(like timeouts), let alone Trial."
c) "Go on if you wish, but you are missing the beauty of Twisted."
d) "Stop it now. You have clearly misunderstood Twisted. You are wasting
your time and ours."
e) Any other...

  Do you think Twisted is the right tool for my use case, or am I doing
an ugly hack to do what I want, which means I should rather use another
tool?


  Sorry for the length of this post, and thanks in advance for your help.
  Louis

[1]
http://twistedmatrix.com/documents/current/api/twisted.internet.threads.html#blockingCallFromThread
[2]
http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#inlineCallbacks
[3]
http://twistedmatrix.com/pipermail/twisted-python/2005-October/thread.html#11831

Re: Declaring AMP responders away from

Allen Short — 2012-05-16T15:07:48

Have a look at what I did in Frack:

http://bazaar.launchpad.net/~washort/frack/trunk/view/head:/frack/wiring.py#L27
http://bazaar.launchpad.net/~washort/frack/trunk/view/head:/frack/responder.py#L85

You can unbundle the locator and dispatcher bits. I only used this for
a single separate class, but with some imagination the principle could
be extended to multiple objects I bet.

Declaring AMP responders away from

Laurens Van Houtven — 2012-05-16T14:26:00

Hi,

I'm writing a service that exposes a bunch of methods over AMP.

These methods are easily logically grouped into separate modules (i.e. message-related, picture-related…). So, I'd like to define the behavior for those methods in those modules.

However, the amp.Command.responder decorator only works inside the class definition of an AMP subclass.  (http://twistedmatrix.com/documents/current/api/twisted.protocols.amp.Command.html#responder).

Does that mean I should have multiple AMP subclasses? How do I get all of them to listen on the same port? Is this what the mysterious "routes" feature I've been hearing about does?

Right now I'm doing it like this, which works but isn't very pretty or nice:

user.py:
-----
class Register(amp.Command):
    arguments = […]
    response = […]
    errors = […]


def register(self, …):
    # do actual registering behavior here
-----

api.py:
-----
from project import auth

class API(amp.AMP):
    register = auth.Register.responder(auth.register)
-----


cheers
lvh

Getting rid of "DirtyReactorWarning"

Itamar Turner-Trauring — 2012-05-15T10:26:24

I have a ticket open for allowing tests that look like this, where each 
run has a new reactor instance:


def test_somethingShouldHappen(self, reactor):
      result = []
      reactor.listenTCP(...)
      self.runReactor(reactor)
      self.assertEqual(result, ["hooray"])


Further thought suggests however that this doesn't add much, 
necessarily, over the current "return a Deferred" idiom for most users. 
Experience with ReactorBuilder framework that has similar testing 
mechanism suggests it's far too easy to fail to stop the reactor. 
Moreover, things like coiterate() don't work since they rely on 
pre-imported global reactor, and given the "from twisted.internet import 
reactor" idiom this testing style won't work well for many people's code.

But! ReactorBuilder tests are superior in one way: no 
DirtyReactorWarning. You don't have to wait until all connections are 
closed, which is difficult, or make sure to cancel every timeout. This 
makes test writing simpler.

Perhaps we should get rid of DirtyReactorWarning for regular 
Deferred-returning tests as well, then. We have all the infrastructure 
we need, after all, for canceling scheduled events and disconnecting all 
descriptors. We might want to log when we do so, to help debug things, 
but we already have cleanup code mostly implemented in trial anyway, and 
it seems like DirtyReactorWarning doesn't usually make future tests fail 
anyway since we do run cleanup.

Threaded code might still be a problem, but that's the case for current 
tests now, and even switching to a new style of tests as above would 
only somewhat mitigate the issue.

Thoughts?

Re: Cannot install Twisted version 8 through PIP

Pascal Briet — 2012-05-14T14:00:18

Ok,

Thanks for your help.
Best regards,

Re: Cannot install Twisted version 8 through PIP

Itamar Turner-Trauring — 2012-05-14T13:54:00

We have a strict backwards compatibility policy; anything that changes 
or is removed will have deprecation warnings for at least 2 releases and 
one year.

So the thing to do is, upgrade to version 9, run your program and make 
sure its tests pass, and fix any code that triggers deprecation warnings 
in Twisted. Repeat with each version until you're up to date. In most 
cases this may take as little as a couple of hours of work; possibly 
none if you didn't happen to be using any deprecated APIs.