Running a Twisted Perspective Broker example with twistd

I've been using Twisted's Perspective Broker to manage networking for my Python program. Perspective Broker allows me to run a Python program on a remote machine and perform remote method calls on an object in the Python program. It also allows me to serialize objects and transfer them over TCP.

Once I got a Perspective Broker server and client running, I wanted to create a "Twisted Application" and run it using twistd, the Twisted Daemon. Two major options are: creating a .tac file and creating a twistd plugin. Below, I show the steps I took to create a .tac application script and a twistd plugin using a simple Perspective Broker example.

Basic Perspective Broker server and client

Here is a simple Perspective Broker server and client example which I adapted from Twisted's examples page. The PB server code lives in pbsimpleserver.py:

from twisted.spread import pb
from twisted.internet import reactor

class Echoer(pb.Root):
    def remote_echo(self, st):
        print 'echoing:', st
        return st

if __name__ == '__main__':
    serverfactory = pb.PBServerFactory(Echoer())
    reactor.listenTCP(8789, serverfactory)
    reactor.run()

Here is a simple PB client, pbsimpleclient.py:

from twisted.spread import pb
from twisted.internet import reactor

class EchoClient(object):
    def connect(self):
        clientfactory = pb.PBClientFactory()
        reactor.connectTCP("localhost", 8789, clientfactory)
        d = clientfactory.getRootObject()
        d.addCallback(self.send_msg)
        
    def send_msg(self, result):
        d = result.callRemote("echo", "hello network")
        d.addCallback(self.get_msg)

    def get_msg(self, result):
        print "server echoed: ", result
        
if __name__ == '__main__':
    EchoClient().connect()
    reactor.run()

This code connects the client to the server using port 8789. It sends a message by calling the server's remote_echo method with an argument of "hello network". The server prints the message and returns the same message to the client. The client then prints the message.

To test the code, I ran both client and server on my local machine. I ran python pbsimpleserver.py in one terminal shell, then ran python pbsimpleclient.py in another shell. In the server shell, I got:

echoing: hello network

and in the client shell I got:

server echoed:  hello network

To stop the client and server, I hit CTRL-C in both shells

Converting to a Twisted Application (.tac script)

To create a Twisted application, I used the twisted.application.service.Application object. When converting the example code above to an Application, I replaced twisted.internet.reactor.listenTCP with twisted.application.internet.TCPServer and twisted.internet.reactor.connectTCP with twisted.application.internet.TCPClient. Below is my server application, pbsimpleserver_app.py. (Note: the two files below are considered .tac files even though the filename doesn't end in .tac. From the documentation, .tac files are Python files which configure an Application object and assign this object to the top-level variable "application".)

from twisted.spread import pb
from twisted.application.internet import TCPServer
from twisted.application.service import Application

class Echoer(pb.Root):
    def remote_echo(self, st):
        print 'echoing:', st
        return st

serverfactory = pb.PBServerFactory(Echoer())
application = Application("echo")
echoServerService = TCPServer(8789, serverfactory)
echoServerService.setServiceParent(application)

Here is my client application, pbsimpleclient_app.py:

from twisted.spread import pb
from twisted.application.internet import TCPClient
from twisted.application.service import Application

class EchoClient(object):
    def send_msg(self, result):
        d = result.callRemote("echo", "hello network")
        d.addCallback(self.get_msg)

    def get_msg(self, result):
        print "server echoed: ", result

e = EchoClient()
clientfactory = pb.PBClientFactory()
d = clientfactory.getRootObject()
d.addCallback(e.send_msg)        

application = Application("echo")
echoClientService = TCPClient("localhost", 8789, clientfactory)
echoClientService.setServiceParent(application)

To run these as daemons with twistd, I executed:

twistd -l server.log --pidfile server.pid -y pbsimpleserver_app.py

then:

twistd -l client.log --pidfile client.pid -y pbsimpleclient_app.py

This created the log file, server.log:

2008-10-27 00:08:35-0700 [-] Log opened.
2008-10-27 00:08:35-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 00:08:35-0700 [-] reactor class: 
2008-10-27 00:08:35-0700 [-] twisted.spread.pb.PBServerFactory starting on 8789
2008-10-27 00:08:35-0700 [-] Starting factory 
2008-10-27 00:08:53-0700 [Broker,0,127.0.0.1] echoing: hello network

and client.log:

2008-10-27 00:08:53-0700 [-] Log opened.
2008-10-27 00:08:53-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 00:08:53-0700 [-] reactor class: 
2008-10-27 00:08:53-0700 [-] Starting factory 
2008-10-27 00:08:53-0700 [Broker,client] server echoed:  hello network

Creating a twistd plugin

In order to pass command line arguments to my Twisted application daemon, I need to create a twistd plugin. The following is how I implemented my plugin after reading the Writing a twistd plugin documentation. Here is my directory structure. EchoProj is located in ~/Projects.

EchoProj
|-- echoproj
|   |-- __init__.py
|   |-- pbsimpleclient.py
|   `-- pbsimpleserver.py
`-- twisted
    `-- plugins
        |-- echoclient_plugin.py
        `-- echoserver_plugin.py

pbsimpleserver.py:

from twisted.spread import pb

class EchoServer(pb.Root):
    def remote_echo(self, st):
        print 'echoing:', st
        return st

pbsimpleclient.py:

class EchoClient(object):
    def send_msg(self, result):
        d = result.callRemote("echo", "hello network")
        d.addCallback(self.get_msg)

    def get_msg(self, result):
        print "server echoed: ", result

echoserver_plugin.py:

from zope.interface import implements
from twisted.python import usage
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker
from twisted.application.internet import TCPServer
from twisted.spread import pb
from echoproj.pbsimpleserver import EchoServer

class Options(usage.Options):
    optParameters = [["port", "p", 8789, "The port number to listen on."]]

class MyServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echoserver"
    description = "Echo Server"
    options = Options

    def makeService(self, options):
        serverfactory = pb.PBServerFactory(EchoServer())
        return TCPServer(int(options["port"]), serverfactory)

serviceMaker = MyServiceMaker()

echoclient_plugin.py:

from zope.interface import implements
from twisted.python import usage
from twisted.plugin import IPlugin
from twisted.application.service import IServiceMaker
from twisted.application.internet import TCPClient
from twisted.spread import pb
from echoproj.pbsimpleclient import EchoClient

class Options(usage.Options):
    optParameters = [["port", "p", 8789, "The port number to connect to."],
                     ["host", "h", "localhost", "The host machine to connect to."]
                     ]

class MyServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    tapname = "echoclient"
    description = "Echo Client"
    options = Options

    def makeService(self, options):
        e = EchoClient()
        clientfactory = pb.PBClientFactory()
        d = clientfactory.getRootObject()
        d.addCallback(e.send_msg)
        return TCPClient(options["host"], int(options["port"]), clientfactory)

serviceMaker = MyServiceMaker()

I set the PYTHONPATH to include the top-level project directory:

export PYTHONPATH="$HOME/Projects/EchoProj:$PYTHONPATH"

Running twistd --help now showed "echoserver" and "echoclient" in the list of commands. To run my server and client as daemons using port 8790 on my local machine, I executed:

twistd -l server.log --pidfile server.pid echoserver -p 8790

and

twistd -l client.log --pidfile client.pid echoclient -p 8790

This produced the logfiles, server.log:

2008-10-27 11:49:12-0700 [-] Log opened.
2008-10-27 11:49:12-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 11:49:12-0700 [-] reactor class: 
2008-10-27 11:49:12-0700 [-] twisted.spread.pb.PBServerFactory starting on 8790
2008-10-27 11:49:12-0700 [-] Starting factory 
2008-10-27 11:49:17-0700 [Broker,0,127.0.0.1] echoing: hello network

and client.log:

2008-10-27 11:49:17-0700 [-] Log opened.
2008-10-27 11:49:17-0700 [-] twistd 8.1.0 (/usr/bin/python 2.5.2) starting up
2008-10-27 11:49:17-0700 [-] reactor class: 
2008-10-27 11:49:17-0700 [-] Starting factory 
2008-10-27 11:49:17-0700 [Broker,client] server echoed:  hello network