SaltyCrane: twistedhttps://www.saltycrane.com/blog/2010-08-25T20:09:00-07:00Twisted web POST example w/ JSON
2010-08-25T20:09:00-07:00https://www.saltycrane.com/blog/2010/08/twisted-web-post-example-json/<p>
This is an example of a simple asynchronous Python web server
using <a href="http://twistedmatrix.com/trac/">Twisted</a>.
This is a copy of Jp Calderone's
<a href="http://jcalderone.livejournal.com/49707.html">
Twisted Web in 60 seconds: handling POSTs example
</a> modified to accept a JSON payload in the POST request instead
of form data. It also uses his
<a href="http://labs.twistedmatrix.com/2008/02/simple-python-web-server.html">
Simple Python Web Server example</a>
to run the web server as a daemon with twistd.
</p>
<h4><code>webserver.py</code></h4>
<pre class="python">
"""
http://jcalderone.livejournal.com/49707.html
http://labs.twistedmatrix.com/2008/02/simple-python-web-server.html
usage:
$ twistd -y webserver.py
"""
from pprint import pprint
from twisted.application.internet import TCPServer
from twisted.application.service import Application
from twisted.web.resource import Resource
from twisted.web.server import Site
class FormPage(Resource):
def render_GET(self, request):
return ''
def render_POST(self, request):
pprint(request.__dict__)
newdata = request.content.getvalue()
print newdata
return ''
root = Resource()
root.putChild("form", FormPage())
application = Application("My Web Service")
TCPServer(8880, Site(root)).setServiceParent(application)
</pre>
<h4><code>test_post.py</code></h4>
<p>Here is a simple test client using
<a href="http://code.google.com/p/httplib2/">httplib2</a>
to send a POST request with some JSON data. I used Mark Pilgrim's
<a href="http://diveintopython3.org/http-web-services.html#beyond-get">
Dive Into Python 3 Section 14.6</a> as a reference.
</p>
<pre class="python">
import httplib2
from datetime import datetime
import simplejson
TESTDATA = {'woggle': {'version': 1234,
'updated': str(datetime.now()),
}}
URL = 'http://localhost:8880/form'
jsondata = simplejson.dumps(TESTDATA)
h = httplib2.Http()
resp, content = h.request(URL,
'POST',
jsondata,
headers={'Content-Type': 'application/json'})
print resp
print content
</pre>
<h4>Run the web server</h4>
<pre class="console">$ twisted -y webserver.py </pre>
<h4>Run the test POST</h4>
<pre class="console">$ python test_post.py </pre>
<h4>twistd.log</h4>
<p>Here are the results stored in <code>twistd.log</code>.</p>
<pre>
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] {'_adapterCache': {},
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'args': {},
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'channel': <twisted.web.http.HTTPChannel instance at 0x7fb409dc8248>,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'client': <twisted.internet.address.IPv4Address object at 0x1b48f50>,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'clientproto': 'HTTP/1.1',
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'content': <cStringIO.StringO object at 0x1b4c068>,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'cookies': [],
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'headers': {'date': 'Thu, 26 Aug 2010 03:02:37 GMT', 'content-type': 'text/html', 'server': 'TwistedWeb/10.0.0'},
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'host': <twisted.internet.address.IPv4Address object at 0x1b48fd0>,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'method': 'POST',
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'notifications': [],
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'path': '/form',
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'postpath': [],
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'prepath': ['form'],
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'queued': 0,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'received_cookies': {},
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'received_headers': {'host': 'localhost:8880', 'content-type': 'application/json', 'accept-encoding': 'identity', 'content-length': '70', 'user-agent': 'Python-httplib2/$Rev$'},
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'requestHeaders': Headers({'host': ['localhost:8880'], 'content-type': ['application/json'], 'accept-encoding': ['identity'], 'content-length': ['70'], 'user-agent': ['Python-httplib2/$Rev$']}),
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'responseHeaders': Headers({'date': ['Thu, 26 Aug 2010 03:02:37 GMT'], 'content-type': ['text/html'], 'server': ['TwistedWeb/10.0.0']}),
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'site': <twisted.web.server.Site instance at 0x1b419e0>,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'sitepath': [],
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'stack': [],
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'transport': <HTTPChannel #1 on 8880>,
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 'uri': '/form'}
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] {"woggle": {"updated": "2010-08-25 20:02:37.449333", "version": 1234}}
2010-08-25 20:02:37-0700 [HTTPChannel,1,127.0.0.1] 127.0.0.1 - - [26/Aug/2010:03:02:36 +0000] "POST /form HTTP/1.1" 200 - "-" "Python-httplib2/$Rev$"
</pre>
Quick notes on trying the Twisted websocket branch example
2010-05-24T14:43:32-07:00https://www.saltycrane.com/blog/2010/05/quick-notes-trying-twisted-websocket-branch-example/<p>Here are my quick notes on trying out the
<a href="http://twistedmatrix.com/trac/browser/branches/websocket-4173-2/doc/web/examples/websocket.py">websocket
example</a> in
<a href="http://twistedmatrix.com/trac/">Twisted</a>'s
<a href="http://twistedmatrix.com/trac/browser/branches/websocket-4173-2">websocket
branch</a>. The
<a href="http://twistedmatrix.com/trac/export/29073/branches/websocket-4173-2/doc/web/howto/websocket.xhtml">documentation
is here</a>. The Twisted <a href="http://twistedmatrix.com/trac/ticket/4173">
ticket is here</a>. This came about after some conversation with
<a href="http://twitter.com/clemesha">@clemesha</a> on Twitter.
</p>
<p>(A <a href="http://en.wikipedia.org/wiki/Web_Sockets">Web Socket</a> is a new,
still-in-development technology, introduced in
<a href="http://en.wikipedia.org/wiki/HTML5">HTML5</a>, and may be used
for real-time web applications. It provides a simple (maybe better) alternative
to existing <a href="http://en.wikipedia.org/wiki/Comet_%28programming%29">Comet</a>
technology.)
</p>
<p>(Note: The
<a href="http://dev.w3.org/html5/websockets/">WebSocket API</a> is still changing.
Google Chrome
<a href="http://blog.chromium.org/2009/12/web-sockets-now-available-in-google.html">supports
(a version of) it</a>.
Firefox as of version 3.6,
<a href="http://hacks.mozilla.org/2010/04/websockets-in-firefox/">does
not support it yet</a>.)
</p>
<p>(I am no expert on Web Sockets. I just think
they are cool and want to start using them.)
</p>
<h4 id="install">Install Twisted websocket branch</h4>
<ul>
<li><a href="http://www.saltycrane.com/blog/2009/05/notes-using-pip-and-virtualenv-django/#install-pip">Install
pip and virtualenv</a>
</li>
<li>Install the Twisted websocket branch in a virtualenv
<pre class="console">$ cd ~/lib/python-environments
$ virtualenv --no-site-packages --distribute twisted-websocket-branch
$ pip install -E twisted-websocket-branch/ -e svn+svn://svn.twistedmatrix.com/svn/Twisted/branches/websocket-4173-2</pre>
</li>
</ul>
<h4 id="client"><code>~/wsdemo/index.html</code></h4>
<pre class="html">
<html>
<head>
<title>WebSocket example: echo service</title>
</head>
<body>
<h1>WebSocket example: echo service</h1>
<script type="text/javascript">
var ws = new WebSocket("ws://127.0.0.1:8080/ws/echo");
ws.onmessage = function(evt) {
var data = evt.data;
var target = document.getElementById("received");
target.value = target.value + data;
};
window.send_data = function() {
ws.send(document.getElementById("send_input").value);
};
</script>
<form>
<label for="send_input">Text to send</label>
<input type="text" name="send_input" id="send_input"/>
<input type="submit" name="send_submit" id="send_submit" value="Send"
onclick="send_data(); return false"/>
<br>
<label for="received">Received text</label>
<textarea name="received" id="received"></textarea>
</form>
</body>
</html>
</pre>
<h4 id="server"><code>~/wsdemo/demo.py</code></h4>
<pre class="python">import sys
from twisted.python import log
from twisted.internet import reactor
from twisted.web.static import File
from twisted.web.websocket import WebSocketHandler, WebSocketSite
class Echohandler(WebSocketHandler):
def frameReceived(self, frame):
log.msg("Received frame '%s'" % frame)
self.transport.write(frame + "\n")
def main():
log.startLogging(sys.stdout)
root = File(".")
site = WebSocketSite(root)
site.addHandler("/ws/echo", Echohandler)
reactor.listenTCP(8080, site)
reactor.run()
if __name__ == "__main__":
main()
</pre>
<h4 id="try-it">Try it</h4>
<ul>
<li>Activate virtualenv
<pre class="console">$ source ~/lib/python-environments/twisted-websocket-branch/bin/activate</pre>
</li>
<li>Run server
<pre class="console">$ cd ~/wsdemo
$ python demo.py
</pre>
</li>
<li>Visit <a href="http://localhost:8080/">http://localhost:8080/</a>
in your WebSocket-enabled browser (e.g. Google Chrome)
<a href="http://picasaweb.google.com/lh/photo/nUbWY5E8n0_PrPm-M7aU1Q?feat=embedwebsite"><img src="http://lh6.ggpht.com/_WnP2PKiLI14/S_yoqRKU9NI/AAAAAAAAALY/q-JqmoBjy90/s800/twisted_websocket_example.png" /></a>
</li>
<li>Here's the console output:
<pre>2010-05-25 21:47:46-0700 [-] Log opened.
2010-05-25 21:47:46-0700 [-] twisted.web.websocket.WebSocketSite starting on 8080
2010-05-25 21:47:46-0700 [-] Starting factory <twisted.web.websocket.WebSocketSite instance at 0x94243ac>
2010-05-25 21:47:56-0700 [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [26/May/2010:04:47:56 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.2 (KHTML, like Gecko) Chrome/5.0.342.9 Safari/533.2"
2010-05-25 21:47:56-0700 [HTTPChannel,1,127.0.0.1] 127.0.0.1 - - [26/May/2010:04:47:56 +0000] "GET /favicon.ico HTTP/1.1" 404 145 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.2 (KHTML, like Gecko) Chrome/5.0.342.9 Safari/533.2"
2010-05-25 21:48:16-0700 [HTTPChannel,2,127.0.0.1] Received frame 'hello'
2010-05-25 21:48:25-0700 [HTTPChannel,2,127.0.0.1] Received frame 'Twisted+Websocket!'</pre>
</li>
</ul>
<h4 id="see-also">See also</h4>
<ul>
<li><a href="http://bitbucket.org/rushman/tx-websockets/src">
http://bitbucket.org/rushman/tx-websockets/src</a></li>
<li><a href="http://github.com/gleicon/txwebsockets">
http://github.com/gleicon/txwebsockets</a></li>
<li><a href="http://github.com/fiorix/cyclone">
http://github.com/fiorix/cyclone</a></li>
<li><a href="http://boothead.github.com/rpz.websocket/">
http://boothead.github.com/rpz.websocket/</a></li>
</ul>
Running a Twisted Perspective Broker example with twistd
2008-10-27T00:26:00-07:00https://www.saltycrane.com/blog/2008/10/running-twisted-perspective-broker-example-twistd/<p>I've been using Twisted's Perspective Broker to manage networking for
my Python program. Perspective Broker allows me to run a Python program
on a remote machine and perform remote method calls on an object in the
Python program. It also allows me to serialize objects and transfer them
over TCP.</p>
<p>Once I got a Perspective Broker server and client running, I wanted
to create a "Twisted Application" and run it using <code>twistd</code>,
the <a href="http://twistedmatrix.com/projects/core/documentation/howto/basics.html#auto1">
Twisted Daemon</a>. Two major options are:
<a href="http://twistedmatrix.com/projects/core/documentation/howto/application.html">
creating a .tac file</a> and
<a href="http://twistedmatrix.com/projects/core/documentation/howto/tap.html">
creating a twistd plugin</a>. Below, I show the steps I took to create
a .tac application script and a twistd plugin using a simple Perspective
Broker example.
</p>
<h4>Basic Perspective Broker server and client</h4>
<p>Here is a simple Perspective Broker server and client example which
I adapted from Twisted's
<a href="http://twistedmatrix.com/projects/core/documentation/examples/#auto3">
examples page</a>.
The PB server code lives in pbsimpleserver.py:
</p>
<pre class="python">from twisted.spread import pb
from twisted.internet import reactor
class Echoer(pb.Root):
def remote_echo(self, st):
print 'echoing:', st
return st
if __name__ == '__main__':
serverfactory = pb.PBServerFactory(Echoer())
reactor.listenTCP(8789, serverfactory)
reactor.run()</pre>
<p>Here is a simple PB client, pbsimpleclient.py:</p>
<pre class="python">from twisted.spread import pb
from twisted.internet import reactor
class EchoClient(object):
def connect(self):
clientfactory = pb.PBClientFactory()
reactor.connectTCP("localhost", 8789, clientfactory)
d = clientfactory.getRootObject()
d.addCallback(self.send_msg)
def send_msg(self, result):
d = result.callRemote("echo", "hello network")
d.addCallback(self.get_msg)
def get_msg(self, result):
print "server echoed: ", result
if __name__ == '__main__':
EchoClient().connect()
reactor.run()</pre>
<p>This code connects the client to the server
using port 8789. It sends a message by calling the server's <code>remote_echo</code>
method with an argument of <code>"hello network"</code>. The server
prints the
message and returns the same message to the client.
The client then prints the message.</p>
<p>To test the code, I ran both client and server on my local machine.
I ran <code>python pbsimpleserver.py</code>
in one terminal shell, then ran <code>python pbsimpleclient.py</code>
in another shell. In the server shell, I got:</p>
<pre>echoing: hello network</pre>
<p>and in the client shell I got:</p>
<pre>server echoed: hello network</pre>
<p>To stop the client and server, I hit CTRL-C in both shells</p>
<h4>Converting to a Twisted Application (.tac script)</h4>
<p>To create a Twisted application, I used the
<code>twisted.application.service.Application</code> object.
When converting
the example code above to an Application, I replaced
<code>twisted.internet.reactor.listenTCP</code> with
<code>twisted.application.internet.TCPServer</code> and
<code>twisted.internet.reactor.connectTCP</code> with
<code>twisted.application.internet.TCPClient</code>.
Below is my server application, pbsimpleserver_app.py.
(Note: the two files below are considered .tac files even
though the filename doesn't end in .tac. From the documentation,
<em>.tac files are Python files which configure an Application object
and assign this object to the top-level variable "application".</em>)
</p>
<pre class="python">from twisted.spread import pb
from twisted.application.internet import TCPServer
from twisted.application.service import Application
class Echoer(pb.Root):
def remote_echo(self, st):
print 'echoing:', st
return st
serverfactory = pb.PBServerFactory(Echoer())
application = Application("echo")
echoServerService = TCPServer(8789, serverfactory)
echoServerService.setServiceParent(application)</pre>
<p>Here is my client application, pbsimpleclient_app.py:</p>
<pre class="python">from twisted.spread import pb
from twisted.application.internet import TCPClient
from twisted.application.service import Application
class EchoClient(object):
def send_msg(self, result):
d = result.callRemote("echo", "hello network")
d.addCallback(self.get_msg)
def get_msg(self, result):
print "server echoed: ", result
e = EchoClient()
clientfactory = pb.PBClientFactory()
d = clientfactory.getRootObject()
d.addCallback(e.send_msg)
application = Application("echo")
echoClientService = TCPClient("localhost", 8789, clientfactory)
echoClientService.setServiceParent(application)</pre>
<p>To run these as daemons with <code>twistd</code>, I
executed:</p>
<pre>twistd -l server.log --pidfile server.pid -y pbsimpleserver_app.py</pre>
<p>then:</p>
<pre>twistd -l client.log --pidfile client.pid -y pbsimpleclient_app.py</pre>
<p>This created the log file, server.log:</p>
<pre>2008-10-27 00:08:35-0700 [-] Log opened.
2008-10-27 00:08:35-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 00:08:35-0700 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2008-10-27 00:08:35-0700 [-] twisted.spread.pb.PBServerFactory starting on 8789
2008-10-27 00:08:35-0700 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x83c236c>
2008-10-27 00:08:53-0700 [Broker,0,127.0.0.1] echoing: hello network</pre>
<p>and client.log:</p>
<pre>2008-10-27 00:08:53-0700 [-] Log opened.
2008-10-27 00:08:53-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 00:08:53-0700 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2008-10-27 00:08:53-0700 [-] Starting factory <twisted.spread.pb.PBClientFactory instance at 0x84f488c>
2008-10-27 00:08:53-0700 [Broker,client] server echoed: hello network</pre>
<br>
<h4>Creating a twistd plugin</h4>
<p>In order to pass command line arguments to my Twisted application daemon,
I need to create a twistd plugin. The following is how I implemented
my plugin after reading the
<a href="http://twistedmatrix.com/projects/core/documentation/howto/tap.html">
Writing a twistd plugin</a> documentation.
Here is my directory structure. EchoProj is located in <code>~/Projects</code>.</p>
<pre>EchoProj
|-- echoproj
| |-- __init__.py
| |-- pbsimpleclient.py
| `-- pbsimpleserver.py
`-- twisted
`-- plugins
|-- echoclient_plugin.py
`-- echoserver_plugin.py</pre>
<p>pbsimpleserver.py:</p>
<pre class="python">from twisted.spread import pb
class EchoServer(pb.Root):
def remote_echo(self, st):
print 'echoing:', st
return st</pre>
<p>pbsimpleclient.py:</p>
<pre class="python">class EchoClient(object):
def send_msg(self, result):
d = result.callRemote("echo", "hello network")
d.addCallback(self.get_msg)
def get_msg(self, result):
print "server echoed: ", result</pre>
<p>echoserver_plugin.py:</p>
<pre class="python">from zope.interface import implements
from twisted.python import usage
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker
from twisted.application.internet import TCPServer
from twisted.spread import pb
from echoproj.pbsimpleserver import EchoServer
class Options(usage.Options):
optParameters = [["port", "p", 8789, "The port number to listen on."]]
class MyServiceMaker(object):
implements(IServiceMaker, IPlugin)
tapname = "echoserver"
description = "Echo Server"
options = Options
def makeService(self, options):
serverfactory = pb.PBServerFactory(EchoServer())
return TCPServer(int(options["port"]), serverfactory)
serviceMaker = MyServiceMaker()</pre>
<p>echoclient_plugin.py:</p>
<pre class="python">from zope.interface import implements
from twisted.python import usage
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker
from twisted.application.internet import TCPClient
from twisted.spread import pb
from echoproj.pbsimpleclient import EchoClient
class Options(usage.Options):
optParameters = [["port", "p", 8789, "The port number to connect to."],
["host", "h", "localhost", "The host machine to connect to."]
]
class MyServiceMaker(object):
implements(IServiceMaker, IPlugin)
tapname = "echoclient"
description = "Echo Client"
options = Options
def makeService(self, options):
e = EchoClient()
clientfactory = pb.PBClientFactory()
d = clientfactory.getRootObject()
d.addCallback(e.send_msg)
return TCPClient(options["host"], int(options["port"]), clientfactory)
serviceMaker = MyServiceMaker()</pre>
<p>I set the PYTHONPATH to include the top-level project directory:</p>
<pre>export PYTHONPATH="$HOME/Projects/EchoProj:$PYTHONPATH"</pre>
<p>Running <code>twistd --help</code> now showed "echoserver"
and "echoclient" in the list of commands. To run my server and
client as daemons using port 8790 on my local machine, I executed:</p>
<pre>twistd -l server.log --pidfile server.pid echoserver -p 8790</pre>
<p>and</p>
<pre>twistd -l client.log --pidfile client.pid echoclient -p 8790</pre>
<p>This produced the logfiles, server.log:</p>
<pre>2008-10-27 11:49:12-0700 [-] Log opened.
2008-10-27 11:49:12-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 11:49:12-0700 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2008-10-27 11:49:12-0700 [-] twisted.spread.pb.PBServerFactory starting on 8790
2008-10-27 11:49:12-0700 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x847fc2c>
2008-10-27 11:49:17-0700 [Broker,0,127.0.0.1] echoing: hello network</pre>
<p>and client.log:</p>
<pre>2008-10-27 11:49:17-0700 [-] Log opened.
2008-10-27 11:49:17-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 11:49:17-0700 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2008-10-27 11:49:17-0700 [-] Starting factory <twisted.spread.pb.PBClientFactory instance at 0x847fc2c>
2008-10-27 11:49:17-0700 [Broker,client] server echoed: hello network</pre>
Twisted links
2008-10-21T17:01:20-07:00https://www.saltycrane.com/blog/2008/10/twisted-links/<h4>Twisted tutorials</h4>
<ul>
<li><a href="http://jessenoller.com/2009/02/11/twisted-hello-asynchronous-programming/">
Twisted - hello, asynchronous programming</a><br>
<em>by Jesse Noller, February 11, 2009</em><br>
Overview of Twisted concepts
</li>
<li><a href="http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html">
Twisted inlineCallbacks and deferredGenerator</a><br>
<em>by Marcin Kasperski, August 13, 2008</em><br>
Examples comparing raw deferreds, deferredGenerator, and inlineCallbacks.
</li>
<li><a href="http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough.html">
Async Batching with Twisted: A Walkthrough</a><br>
<em>by Duncan McGreggor, June 20, 2008</em><br>
This article features 8 easy to understand examples demonstrating
Deferreds, callbacks, DeferredLists, DeferredSemaphores, and
task.Cooperator.
</li>
<li><a href="http://www.artima.com/weblogs/viewpost.jsp?thread=230001">
Concurrency with Python, Twisted, and Flex</a><br>
<em>by Bruce Eckel, May 3, 2008</em></li>
<li><a href="http://www.artima.com/weblogs/viewpost.jsp?thread=156396">
Grokking Twisted</a><br>
<em>by Bruce Eckel, April 15, 2006</em></li>
<li><a href="http://www.linuxjournal.com/article/7871">
Event-Driven Programming with Twisted and Python</a><br>
<em>Linux Journal, January 26, 2005</em></li>
<li><a href="http://gnosis.cx/publish/programming/twisted_1.html">
The Twisted Matrix Framework: Part One, Understanding Asynchronous Networking</a><br>
<em>by David Mertz, May 2003</em> (See also
<a href="http://gnosis.cx/publish/programming/twisted_2.html">
Part 2</a>,
<a href="http://gnosis.cx/publish/programming/twisted_3.html">
Part3</a>, and
<a href="http://gnosis.cx/publish/programming/twisted_4.html">
Part 4</a>)</li>
</ul>
<h4>Twisted philosophy</h4>
<ul>
<li><a href="http://glyph.twistedmatrix.com/2008/07/constructive-criticism.html">
Constructive Criticism</a><br>
<em>by Glyph Lefkowitz, July 2, 2008</em></li>
<li><a href="http://oubiwann.blogspot.com/2008/06/so-you-want-your-code-to-be-asynchonous.html">
So You Want Your Code to Be Asynchronous? A Twisted Interview</a><br>
<em>by Duncan McGreggor, June 27, 2008</em></li>
<li><a href="http://glyf.livejournal.com/40037.html">
Knowing Santa Claus is Fake Doesn't Ruin Christmas</a><br>
<em>by Glyph Lefkowitz, August 19, 2005</em></li>
<li><a href="http://oubiwann.blogspot.com/2005/06/thinking-in-twisted.html">
Thinking in Twisted</a><br>
<em>by Duncan McGreggor, June 13, 2005</em></li>
</ul>
Can't block for a Deferred in Twisted
2008-10-20T14:51:29-07:00https://www.saltycrane.com/blog/2008/10/cant-block-deferred-twisted/<p>Despite the existence of the promising
<a href="http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html#deferredGenerator">
<code>waitForDeferred</code>/<code>deferredGenerator</code></a> and the newer
<a href="http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html#inlineCallbacks">
<code>inlineCallbacks</code></a>, it appears there is no way to block
while waiting for a Deferred. Brian Granger
<a href="http://www.twistedmatrix.com/pipermail/twisted-python/2006-March/012664.html">
described the problem</a> on the Twisted mailing list:
<blockquote>I have a function that returns a Deferred. I need to have the result
of this Deferred returned in a (apparently) blocking/synchronous
manner:
<pre>def myfuncBlocking():
d = myfuncReturnsDeferred()
...
result =
return result</pre>
I need to be able to call this function like:
<pre>result = myfuncBlocking()</pre>
The question is how to get the result out of the Deferred() and make
it *look* like myfuncBlocking() has blocked.</blockquote>
glyph provided the <a href="http://www.twistedmatrix.com/pipermail/twisted-python/2006-March/012665.html">
succinct answer</a> (as well as an interesting
<a href="http://glyf.livejournal.com/40037.html">commentary on using Twisted
the wrong way</a>).
<blockquote>This issue has been discussed repeatedly - long story short, it's just a bad idea.
</blockquote>
</p>
<p>Hmmm, maybe learning Twisted will be
<a href="http://www.saltycrane.com/blog/2008/10/running-functions-periodically-using-twisteds-loopingcall/#c79">
harder than I thought</a>.</p>
<p><em>Update 2008-10-20</em>: Marcin Kasperski wrote a
<a href="http://blog.mekk.waw.pl/archives/14-Twisted-inlineCallbacks-and-deferredGenerator.html">
good example comparing raw deferreds, deferred generators, and inline callbacks</a>.
</p>
Running functions periodically using Twisted's LoopingCall
2008-10-14T16:00:12-07:00https://www.saltycrane.com/blog/2008/10/running-functions-periodically-using-twisteds-loopingcall/<p>
<a href="http://twistedmatrix.com/trac/">Twisted</a> is pretty
cool-- it is very powerful, but I haven't had the easiest time
learning it.
Here is a simple example that runs a couple functions
periodically (at different
rates) using <code>LoopingCall</code>.</p>
<p>For more information, here are the <a href="http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.task.LoopingCall.html">Twisted docs for LoopingCall</a>.</p>
<pre class="python">from datetime import datetime
from twisted.internet.task import LoopingCall
from twisted.internet import reactor
def hyper_task():
print "I like to run fast", datetime.now()
def tired_task():
print "I want to run slowly", datetime.now()
lc = LoopingCall(hyper_task)
lc.start(0.1)
lc2 = LoopingCall(tired_task)
lc2.start(0.5)
reactor.run()</pre>
<p>Results:</p>
<pre>
I like to run fast 2008-10-14 15:51:02.449537
I want to run slowly 2008-10-14 15:51:02.449915
I like to run fast 2008-10-14 15:51:02.551972
I like to run fast 2008-10-14 15:51:02.652013
I like to run fast 2008-10-14 15:51:02.752006
I like to run fast 2008-10-14 15:51:02.852008
I like to run fast 2008-10-14 15:51:02.952487
I want to run slowly 2008-10-14 15:51:02.952681
I like to run fast 2008-10-14 15:51:03.052012
I like to run fast 2008-10-14 15:51:03.152012
I like to run fast 2008-10-14 15:51:03.252010
I like to run fast 2008-10-14 15:51:03.352009
I like to run fast 2008-10-14 15:51:03.452008
I want to run slowly 2008-10-14 15:51:03.452206
I like to run fast 2008-10-14 15:51:03.552009
I like to run fast 2008-10-14 15:51:03.652013</pre>
<h4 id="timerservice">Using TimerService with twistd</h4>
<p>To create a daemon with twistd that achieves the same effect,
use TimerService. TimerService runs LoopingCall under the hood.
It is meant to be used with the
<a href="http://twistedmatrix.com/documents/current/core/howto/application.html">
Twisted Application infrastructure</a>. See also the
documentation on <a href="http://twistedmatrix.com/documents/10.1.0/api/twisted.application.internet.TimerService.html">TimerService</a>.
</p>
<p>timerservice_ex.py</p>
<pre class="python">from datetime import datetime
from twisted.application import service
from twisted.application.internet import TimerService
def tired_task():
print "I want to run slowly", datetime.now()
application = service.Application("myapp")
ts = TimerService(0.5, tired_task)
ts.setServiceParent(application)</pre>
<p>Run it:</p>
<pre class="console">$ twistd -y timerservice_ex.py </pre>
<p>Console output is stored in twistd.log:</p>
<pre>2010-09-20 18:53:50-0700 [-] Log opened.
2010-09-20 18:53:50-0700 [-] using set_wakeup_fd
2010-09-20 18:53:50-0700 [-] twistd 10.1.0 (/home/saltycrane/.virtualenvs/default/bin/python 2.6.5) starting up.
2010-09-20 18:53:50-0700 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2010-09-20 18:53:50-0700 [-] I want to run slowly 2010-09-20 18:53:50.896477
2010-09-20 18:53:51-0700 [-] I want to run slowly 2010-09-20 18:53:51.397043
2010-09-20 18:53:51-0700 [-] I want to run slowly 2010-09-20 18:53:51.897087
2010-09-20 18:53:52-0700 [-] I want to run slowly 2010-09-20 18:53:52.397047
2010-09-20 18:53:52-0700 [-] I want to run slowly 2010-09-20 18:53:52.897068
2010-09-20 18:53:53-0700 [-] I want to run slowly 2010-09-20 18:53:53.397073
2010-09-20 18:53:53-0700 [-] I want to run slowly 2010-09-20 18:53:53.897032
2010-09-20 18:53:54-0700 [-] I want to run slowly 2010-09-20 18:53:54.397083</pre>
Notes on parallel processing with Python and Twisted
2008-09-12T17:23:17-07:00https://www.saltycrane.com/blog/2008/09/notes-parallel-processing-python-and-twisted/<p><a href="http://twistedmatrix.com/trac/">Twisted</a> is a networking engine
written in Python, that among many other things, can be used to do parallel
processing. It is very big, though, so I had a hard time finding what I needed.
I browsed through the <a href="http://twistedmatrix.com/projects/core/documentation/howto/">
Twisted Documentation</a> and the
<a href="http://search.safaribooksonline.com/0596100329?tocview=true">
Twisted O'Reilly book</a>. There is also a
<a href="http://search.safaribooksonline.com/0596007973/pythoncook2-CHP-15-SECT-7">
Recipe in the Python Cookbook</a>. However, I found Bruce Eckel's article,
<a href="http://www.artima.com/weblogs/viewpost.jsp?thread=230001">Concurrency
with Python, Twisted, and Flex</a> to be the most helpful.
(See also Bruce Eckel's initial article on Twisted:
<a href="http://www.artima.com/weblogs/viewpost.jsp?thread=156396">Grokking Twisted</a>)
</p>
<p>Here are my notes on running Bruce Eckel's example. I removed the Flex part
because I didn't need or know anything about it. This example runs a Controller
which starts a number of separate parallel processes running Solvers
(a.ka. workers).
It also allows for communication between the Controller and Solvers. Though this
example only runs on one machine, the article said extending this to multiple
machines is not difficult. For a good explanation of how this works, please
see <a href="http://www.artima.com/weblogs/viewpost.jsp?thread=230001">the
original article</a>.
<p>Here is <code>solver.py</code> which is copied from the original article.
The actual "work" is done in the <code>step</code> method.
I only added some debugging print statements for myself.
</p>
<pre class="python">"""
solver.py
Original version by Bruce Eckel
Solves one portion of a problem, in a separate process on a separate CPU
"""
import sys, random, math
from twisted.spread import pb
from twisted.internet import reactor
class Solver(pb.Root):
def __init__(self, id):
print "solver.py %s: solver init" % id
self.id = id
def __str__(self): # String representation
return "Solver %s" % self.id
def remote_initialize(self, initArg):
return "%s initialized" % self
def step(self, arg):
print "solver.py %s: solver step" % self.id
"Simulate work and return result"
result = 0
for i in range(random.randint(1000000, 3000000)):
angle = math.radians(random.randint(0, 45))
result += math.tanh(angle)/math.cosh(angle)
return "%s, %s, result: %.2f" % (self, str(arg), result)
# Alias methods, for demonstration version:
remote_step1 = step
remote_step2 = step
remote_step3 = step
def remote_status(self):
print "solver.py %s: remote_status" % self.id
return "%s operational" % self
def remote_terminate(self):
print "solver.py %s: remote_terminate" % self.id
reactor.callLater(0.5, reactor.stop)
return "%s terminating..." % self
if __name__ == "__main__":
port = int(sys.argv[1])
reactor.listenTCP(port, pb.PBServerFactory(Solver(sys.argv[1])))
reactor.run()</pre>
<p>Here is <code>controller.py</code>. This is also copied from the original
article but I removed the Flex interface and created calls to
<code>start</code> and <code>terminate</code> in the Controller
class. I'm not sure if this makes sense, but at least this allowed
me to run the example. I also moved the <code>terminate</code> method
from the FlexInterface to the Controller.
</p>
<pre class="python">"""
Controller.py
Original version by Bruce Eckel
Starts and manages solvers in separate processes for parallel processing.
"""
import sys
from subprocess import Popen
from twisted.spread import pb
from twisted.internet import reactor, defer
START_PORT = 5566
MAX_PROCESSES = 2
class Controller(object):
def broadcastCommand(self, remoteMethodName, arguments, nextStep, failureMessage):
print "controller.py: broadcasting..."
deferreds = [solver.callRemote(remoteMethodName, arguments)
for solver in self.solvers.values()]
print "controller.py: broadcasted"
reactor.callLater(3, self.checkStatus)
defer.DeferredList(deferreds, consumeErrors=True).addCallbacks(
nextStep, self.failed, errbackArgs=(failureMessage))
def checkStatus(self):
print "controller.py: checkStatus"
for solver in self.solvers.values():
solver.callRemote("status").addCallbacks(
lambda r: sys.stdout.write(r + "\n"), self.failed,
errbackArgs=("Status Check Failed"))
def failed(self, results, failureMessage="Call Failed"):
print "controller.py: failed"
for (success, returnValue), (address, port) in zip(results, self.solvers):
if not success:
raise Exception("address: %s port: %d %s" % (address, port, failureMessage))
def __init__(self):
print "controller.py: init"
self.solvers = dict.fromkeys(
[("localhost", i) for i in range(START_PORT, START_PORT+MAX_PROCESSES)])
self.pids = [Popen(["python", "solver.py", str(port)]).pid
for ip, port in self.solvers]
print "PIDS: ", self.pids
self.connected = False
reactor.callLater(1, self.connect)
def connect(self):
print "controller.py: connect"
connections = []
for address, port in self.solvers:
factory = pb.PBClientFactory()
reactor.connectTCP(address, port, factory)
connections.append(factory.getRootObject())
defer.DeferredList(connections, consumeErrors=True).addCallbacks(
self.storeConnections, self.failed, errbackArgs=("Failed to Connect"))
print "controller.py: starting parallel jobs"
self.start()
def storeConnections(self, results):
print "controller.py: storeconnections"
for (success, solver), (address, port) in zip(results, self.solvers):
self.solvers[address, port] = solver
print "controller.py: Connected; self.solvers:", self.solvers
self.connected = True
def start(self):
"controller.py: Begin the solving process"
if not self.connected:
return reactor.callLater(0.5, self.start)
self.broadcastCommand("step1", ("step 1"), self.step2, "Failed Step 1")
def step2(self, results):
print "controller.py: step 1 results:", results
self.broadcastCommand("step2", ("step 2"), self.step3, "Failed Step 2")
def step3(self, results):
print "controller.py: step 2 results:", results
self.broadcastCommand("step3", ("step 3"), self.collectResults, "Failed Step 3")
def collectResults(self, results):
print "controller.py: step 3 results:", results
self.terminate()
def terminate(self):
print "controller.py: terminate"
for solver in self.solvers.values():
solver.callRemote("terminate").addErrback(self.failed, "Termination Failed")
reactor.callLater(1, reactor.stop)
return "Terminating remote solvers"
if __name__ == "__main__":
controller = Controller()
reactor.run()</pre>
<p>To run it, put the two files in the same directory and run <code>python
controller.py</code>. You should see 2 CPUs (if you have 2) go up to 100%
usage. And here is the screen output:</p>
<pre>controller.py: init
PIDS: [12173, 12174]
solver.py 5567: solver init
solver.py 5566: solver init
controller.py: connect
controller.py: starting parallel jobs
controller.py: storeconnections
controller.py: Connected; self.solvers: {('localhost', 5567): <twisted.spread.pb.RemoteReference instance at 0xb786bb0c>, ('localhost', 5566): <twisted.spread.pb.RemoteReference instance at 0xb786b98c>}
controller.py: broadcasting...
controller.py: broadcasted
solver.py 5566: solver step
solver.py 5567: solver step
controller.py: checkStatus
solver.py 5566: remote_status
Solver 5566 operational
solver.py 5567: remote_status
controller.py: step 1 results: [(True, 'Solver 5567, step 1, result: 683825.75'), (True, 'Solver 5566, step 1, result: 543177.17')]
controller.py: broadcasting...
controller.py: broadcasted
Solver 5567 operational
solver.py 5566: solver step
solver.py 5567: solver step
controller.py: checkStatus
solver.py 5566: remote_status
Solver 5566 operational
solver.py 5567: remote_status
controller.py: step 2 results: [(True, 'Solver 5567, step 2, result: 636793.90'), (True, 'Solver 5566, step 2, result: 335358.16')]
controller.py: broadcasting...
controller.py: broadcasted
Solver 5567 operational
solver.py 5566: solver step
solver.py 5567: solver step
controller.py: checkStatus
solver.py 5566: remote_status
Solver 5566 operational
solver.py 5567: remote_status
controller.py: step 3 results: [(True, 'Solver 5567, step 3, result: 847386.43'), (True, 'Solver 5566, step 3, result: 512120.15')]
controller.py: terminate
Solver 5567 operational
solver.py 5566: remote_terminate
solver.py 5567: remote_terminate</pre>