TUTORIAL: Real-time chat with Django, Twisted and WebSockets – Part 3

[Part 1] - [Part 2] - [Part 3] - [Addendums] - [Source]

[Table of Contents]

Preamble

You might wonder – why add long-polling client, connecting to a http service, to a functional websocket client-server implementation? The quick answer:

• A lot of people seem to want to know how to implement long polling clients against a twisted web server
• It’s a useful skill to have in your toolbelt
• The interaction between a blocking service (ie, one handling http requests, where long-polling might be used to emulate a continuous connection) and a non blocking service (ie, the websocket chat protocol) can be interesting to get right.
• Websockets are the future, but not (yet) the now

Django components

We’ll reuse the server we finished in Part 2, and have it serve a chat room interface which connects to our server with an http connection, relying on long-polling to emulate a continuous, “real-time” connection.

First, add an entry to your chat/urls.py file, telling Django we where we want it to serve our view:

urlpatterns = patterns('',
        url(r'^$', views.index, name='index'),
        url(r'^(?P<chat_room_id>\d+)/$', views.chat_room, name='chat_room'),
        url(r'^long_poll/(?P<chat_room_id>\d+)/$', views.longpoll_chat_room, name='longpoll_chat_room'),
)

Next, create the relevant view in chat/views.py:

def longpoll_chat_room(request, chat_room_id):
  chat = get_object_or_404(ChatRoom, pk=chat_room_id)
  return render(request, 'chats/longpoll_chat_room.html', {'chat': chat})

You can see from the view definition, that we’re going to be using a template (since we’re passing a template reference into the render call), and, since the template reference is a path – ‘chats/longpoll_chat_room.html’ – that’s where Django is going to expect it to be on disk. So, go ahead and open up chat/templates/chats/longpoll_chat_room.html, and write the following chunk in it:

{% load staticfiles %}</pre>
<h1>{{ chat.name }}</h1>
<div id="message_list"><ul></ul></div>

You’ll notice that this file is almost identical to the websocket chat. The reference to the graceful.webSocket library isn’t here, since it’s not needed, and all of the javascript client connection work is now going to be in a separate file called (instead of inside of a <script></script> tag like the chat_room.html implementation).

So, let’s create the javascript client, then. Edit chat/static/long_poll.js file, and write the following code into it:

//Numeric representation of the last time we received a message from the server
var lastupdate = -1;

$(document).ready(function(){
    getData();

    var inputBox = document.getElementById("inputbox");

    inputbox.addEventListener("keydown", function(e) {
      if (!e) { var e = window.event; }

      if (e.keyCode == 13) {
        e.preventDefault(); // sometimes useful
        postData(inputbox.value);
        inputbox.value="";
      }
    }, false);

});

var getData = function() {
    $.ajax({
        type: "GET",
        // set the destination for the query
        url: 'http://127.0.0.1:1025?lastupdate='+lastupdate+'&callback=?',
        dataType: 'jsonp',
        // needs to be set to true to avoid browser loading icons
        async: true,
        cache: false,
        timeout:1000,
        // process a successful response
        success: function(response) {
            // append the message list with the new message
            var message = response.data;
            $("#message_list ul")
                .prepend($('<li>'+message+'</li>'));
            // set lastupdate
            lastupdate = response.timestamp;
         },
         complete: getData(lastupdate),
    });
};

var postData = function(data) {
   $.ajax({
        type: "POST",
        // set the destination for the query
        url: 'http://127.0.0.1:1025',
        data: {new_message: data},
        // needs to be set to true to avoid browser loading icons
        async: true,
        cache: false,
   });
}

There’s a lot going on here.

We first add an almost-identical looking event listener to the one in the websocket based chat, to our input box, telling it to send messages when a user presses return/enter.

We then define two functions – getData and postData – which handle the actual communication with the chat server.

postData is the simpler of the two – it uses functionality defined by jQuery ($.ajax) to build, and then send, a post request to our chat server, with the contents of a message as the only argument. You can read the documentation for that command to learn more about how it does what it does. Note that we labeled sent information as  “new_message” – the server-side api component is going to have to unpack that by correctly referring to new_message when it’s received.

The more complex function is the one relying on long-polling to simulate real-time communication – getData. We’re doing a few interesting things here: First, we set the dataType to ‘jsonp’. This is necessary, since the javascript file is served by Django on one port (8000), and the chat interface is served by twisted on another (1025). When successful (when the server responds with a new message for us), we perform the same basic function as the websockets message receive function did – we add the message to our chat room.

The “long-polling” component is implemented by setting a function to execute on “complete”, and setting the timeout variable. With a timeout of 1000, we’re instructing jQuery to give the server at least 1 second to respond to our call. If either the server or at least 1 second has gone by, jQuery will terminate the request and call the complete function.

This completes the long-polling loop: once every second, we open a server connection asking it “do you have any new messages for me?”, handling any messages as they come.

For a production level client, 1 second is probably not appropriate – probably we’d want an exponentially decaying time delay, to be more efficient in network use. For now, this’ll do though.

Now we have a functional long-polling chat client. You can test it by starting Django if it’s not yet running, and opening up one of your chat rooms like so:

http://127.0.0.1:8000/long_poll/1

Since the chat-server components aren’t implemented yet, you might see javascript connection errors in your browser’s console, and actual message sending/receiving won’t (quite) work. So, let’s fix that:

Twisted based Blocking (http) chat server

We’re going to perform some delicate bits of surgery on the existing twisted chat server, to add a in a second, blocking, http-based, chat protocol. We’ll also want the two protocols to share data – so that they’ll provide a single set of chat rooms for people to connect to.

So. Open up your twisted server file (chatserver.py) and edit it to add the following to the top:

from twisted.web.websockets import WebSocketsResource, WebSocketsProtocol, lookupProtocolForFactory

import time, datetime, json, thread
from twisted.web.resource import Resource
from twisted.internet import task
from twisted.web.server import NOT_DONE_YET

These are references to the libraries and functions we’re going to be using. Next, we’ll define the chat protocol. Insert the following chunk into the file (replacing the existing ChatFactory definition):

from twisted.internet.protocol import Factory
class ChatFactory(Factory):
    protocol = WebsocketChat
    clients = []
    messages = {}

class HttpChat(Resource):
    #optimization
    isLeaf = True
    def __init__(self):
        # throttle in seconds to check app for new data
        self.throttle = 1
        # define a list to store client requests
        self.delayed_requests = []
        self.messages = {}

        #instantiate a ChatFactory, for generating the websocket protocols
        self.wsFactory = ChatFactory()

        # setup a loop to process delayed requests
        # not strictly neccessary, but a useful optimization,
        # since it can force dropped connections to close, etc...
        loopingCall = task.LoopingCall(self.processDelayedRequests)
        loopingCall.start(self.throttle, False)

        #share the list of messages between the factories of the two protocols
        self.wsFactory.messages = self.messages
        # initialize parent
        Resource.__init__(self)

    def render_POST(self, request):
        request.setHeader('Content-Type', 'application/json')
        args = request.args
        if 'new_message' in args:
            self.messages[float(time.time())] = args['new_message'][0]
            if len(self.wsFactory.clients) > 0:
                self.wsFactory.clients[0].updateClients(args['new_message'][0])
            self.processDelayedRequests()
        return ''

    def render_GET(self, request):
        request.setHeader('Content-Type', 'application/json')
        args = request.args

        if 'callback' in args:
            request.jsonpcallback =  args['callback'][0]

        if 'lastupdate' in args:
            request.lastupdate =  float(args['lastupdate'][0])
        else:
            request.lastupdate = 0.0

        if request.lastupdate < 0:
            return self.__format_response(request, 1, "connected...", timestamp=0.0)

        #get the next message for this user
        data = self.getData(request)

        if data:
            return self.__format_response(request, 1, data.message, timestamp=data.published_at)

        self.delayed_requests.append(request)
        return NOT_DONE_YET

    #returns the next sequential message,
    #and the time it was received at
    def getData(self, request):
        for published_at in sorted(self.messages):
            if published_at > request.lastupdate:
                return type('obj', (object,), {'published_at' : published_at, "message": self.messages[published_at]})();
        return

    def processDelayedRequests(self):
        for request in self.delayed_requests:
            data = self.getData(request)

            if data:
                try:
                    request.write(self.__format_response(request, 1, data.message, data.published_at))
                    request.finish()
                except:
                    print 'connection lost before complete.'
                finally:
                    self.delayed_requests.remove(request)

    def __format_response(self, request, status, data, timestamp=float(time.time())):
        response = json.dumps({'status':status,'timestamp': timestamp, 'data':data})

        if hasattr(request, 'jsonpcallback'):
            return request.jsonpcallback+'('+response+')'
        else:
            return response

There is a lot going on here.

Let’s break it down some. We’ve added a “messages” structure to ChatFactory(); this structure is going to function as a shared repository for all of the messages we receive over both protocols – to make it possible for users at either protocol to see the same contents for a chat room.

Beyond the standard setup, the initialization function (__init__(self)) instantiates a ChatFactory(), and creates a reference to its messages, so that the two protocols now access the same list of messages, and effectively share a chat room. We also have the initialization function start a loop that runs the processDelayedRequests function once a second. This is not strictly necessary for sending out messages – as you’ll see when you read render_GET – but it helps optimize the use of server resources, since, besides sending out messages as quickly as possible after they’re received, it also has the side effect of freeing up resources dedicated to dropped connections.

We define a render_POST function. The function name conforms to twisted conventions – twisted will attempt to call a function by this name every time a HTTP POST request comes in. Since we know that only message sends perform posts for now, we assume that we’re receiving a message, and go ahead and process it.

First, we add a message to our list of messages. Then, we send the message out to all of the websocket based clients by calling the (soon to be implemented) updateClients method on the first websocket client we can find.

Finally, we call processDelayedRequests, to update any waiting httpclients with the new message.

We also define a render_GET function. This function responds to requests to new messages. Since the initial request is going to have a lastupdate time of -1 (this is hard-coded in the long_polling.js client), we check if the lastupdate is below 0, and, if it is, we send out a message to let the user know he’s connected, and to request updates at time 0 or higher.

We then check to see if there’s any data waiting, which this user should see – the getData function gets the next message that this user should see in his chat room; if there is a message for this user to see, we send it out, together with the time it was received at (so the user knows to ask for the next message in the sequence next time).

This creates a request-loop, with the user requesting each message, one by one, until he’s up to date with the chat. Note that in a production application, you’ll probably want to send messages back to the user in batches (since creating/closing http connections is a an inefficient use of network and server resources).

Once we’ve run out of messages to send out to the user, we append the request to a list of clients waiting for an update, and use the twisted shorthand NOT_DONE_YET to ensure that the connection is not closed when the render_GET function returns (twisted, by default, closes the http connection if we return any other value from this function). processDelayedRequests performs much the same function as the render_GET function, only it performs it for requests currently waiting for an update.

Once a message is sent out, the related connection is closed with a request.finish() function call, and all server resources allocated to it are freed as a side effect of removing it from the delayedRequest list. getData and __format_response are helper functions, which are fairly readable. Note that getData is dynamically creating/instantiating a python object from constructed text (the syntax is a bit weird, but I like being able to do this in python, so I take any excuse to teach people that it’s possible).

We should probably also update our websocket based chat protocol, and have it send messages out to any of the http/blocking based clients. To do that, replace the existing dataReceived function with the following two (we’re factoring updateClients out of dataReceived to make it easier to call it from the POST function we wrote above):

    def dataReceived(self, data):
        self.factory.messages[float(time.time())] = data
        self.updateClients(data)

    def updateClients(self, data):
        for c in self.factory.clients:
            c.message(data)

Next, we’ll have to tell twisted that we’re now running two resources, on two different ports, and give it some directions on how to construct its infrastructure for supporting them. We’ll do that by replacing the last bit in the file with the following:

#resource = WebSocketsResource(lookupProtocolForFactory(ChatFactory())) #this line can be removed

from twisted.web.resource import Resource
from twisted.web.server import Site

from twisted.internet import protocol
from twisted.application import service, internet

resource = HttpChat()
factory = Site(resource)
ws_resource = WebSocketsResource(lookupProtocolForFactory(resource.wsFactory))
root = Resource()
root.putChild("",resource) #the http protocol is up at /
root.putChild("ws",ws_resource) #the websocket protocol is at /ws
application = service.Application("chatserver")
internet.TCPServer(1025, Site(root)).setServiceParent(application)

We now have the infrastructure for a twisted webserver running two chat protocols – one http based one at http://127.0.0.1:1025/, and another which is websocket based at http://127.0.0.1:1025/ws. Let’s test and see if things work! Restart twisted if it’s running, and make sure Django is still up (if not, start it):

bash: python manage.py runserver &
bash: twistd -n -y chatserver.py

Once that’s done, in a flurry of keystrokes, open up two browser windows here on one of your long_polling chat rooms, say http://127.0.0.1:8000/long_poll/1, and another two windows on the websocket client for the same window: http://127.0.0.1:8000/chat_room/1 – and chat away!

Source code

I’ve posted the source code for this tutorial to a git repository. Get the version up to this point here, or by running the following at a command line (explore the git repo for other work I’ve done beyond this tutorial):

git clone https://github.com/aausch/django_twisted_chat.git
git checkout tags/v0.1.2

What next?
You might notice a few things still need doing, for this chat room system to work – you can try to implement fixes yourself. In rough increasing order of difficulty:

• try creating two chat rooms, and posting some messages into each. What happens? Warning: when trying to fix this problem, avoid trying to get the twisted and the Django server to communicate directly.
• there are delays in some of the updates to the long_poll version of the chat rooms. Since things are running locally, the delays don’t have to be as long as they are (some of them don’t really have to be there at all!). As an exercise, try removing/reducing the delays
• the websocket chat rooms don’t update chat history on disconnect (if you close a window and you open it, you won’t get back-chat history). If you want a fun next exercise, try adding that in!
• messages don’t persist, if the server goes down (and take up more and more memory the longer the server is up for!). Try modifying your code to write messages to the database, as they are received, and only store a limited (fixed) number of them in memory at any point in time
• if you send a lot of messages, quickly, in a websocket client, they won’t all make it over to the http/long polling clients [correction: it’s almost certain you won’t notice messages being lost, on account of the GIL]. Oh no! There’s a race condition somewhere in the system. Fix this bug! (solution)