In the previous article we saw some of the basics of the XMPP protocol. In this second installment I hope to expand over what XMPP’s strong suits are and how it proved instrumental in the development of LiveBox. In this regard we will discuss the birth of our first Openfire plugin, aptly named Push Servlet.
I would like to introduce you to the architecture we developed in LiveBox using XMPP. Last time we described the advantages of the Push over Polling based systems, however in case you didn’t get a chance to read the first article I’ll summarize the basic findings to get you up to speed.
The story so far
Polling based systems expect clients to regularly contact the server “Poll” for changes at regular intervals, this quickly builds up into a linearly increasing static load on the server as the number of clients increases. This is a very inefficient approach, and it scales poorly, usually resulting in throwing increasing hardware resources uselessly at the problem as performance drops and response times escalate. A Push based system only requires communication when there are changes to be notified, both by the server and by the client. Meaning that the average load of your servers will be much lower and only peak user activity will ever significantly load your servers, of course this approach requires the server to be able to contact clients on it’s own, but that can easily be achieved. To Summarize the Summary, Polling bad, Push good.
Diving into Push Notifications
Now that the gods of formalities and the priests of introductions have been appeased, we can get down to business. The train of thoughts developers initially reasoning out how to develop push based systems find themselves on usually goes on somewhat like this: “ok, fine, sounds great, but… how do we go about it? If the server has to contact clients, especially if these clients are dispersed all over the internet, and very few of them will have stable/known location… How will we keep track of their changing IPs? Oh god firewalls and NAT, UPNP hell… How will we ensure delivery? …”. Ok, that should suffice, it quickly becomes a gloomy and sad place of mammoth implementations rapidly eroding the programmer’s self esteem and life prospects.
The choice of XMPP
However, in this day and age we benefit from the accidental growth of two very important fields, Online and mobile chat, and the so-called IoT (Internet of Things) and M2M (Machine to Machine) worlds. These paved the way towards the development of lightweight, scalable, stable, and easy to implement communication protocols. One such protocol is XMPP, a TCP based XML stream protocol. Easily extensible, widely supported by open source libraries on practically every platform known to man.
As I stated previously we looked at XMPP to solve notification delivery problems. It may seem overkill to deploy a chat application infrastructure to implement notifications for anything other than well… A chat application. However, it is far simpler to set up than many competing technologies, and allows you to reach multiple platforms (Windows, OSX, iOS, Android…) with relative ease.
In our specific case, the server had to be able to run on Linux, and (at least initially) the client had to be compatible with QT so basically C++ (The Mobile apps, as stated previously, still didn’t exist). For the web server we chose on Openfire, a Java XMPP server developed by the good folks at igniterealtime, it ticked all the right boxes: Easy to deploy (Java), Easy to Manage (It has a very friendly Web Admin interface) , Easy to tailor to our needs (Java + Open source + plugin friendly), Support for ActiveDirectory (Easy account management). And for the QT client we used the very good QXMPP library.
But, I hear you clamor quite rightfully, what did you exactly need it to do? Well, alright then, we needed it to send the clients JSON notifications regarding file synchronization, and even text messages to inform the user of the latest and greatest changes to his data. Possibly with delivery receipts. How did we go about this?
Down to business
First we configured Openfire to get user account information from our own Active Directory sources, check! In parallel we set up QXMPP (A QT based XMPP library) to connect to the Openfire server. After that we wrote a simple Openfire plugin that would listen for HTTP calls from the Main LiveBox server. This plugin, upon receiving a message from the server, would look for viable recipients among its registered users and would then send out the message to those eligible.This would solve almost pretty much any issue with routing messages to the LiveBox clients. The clients would then receive the notification wrapped inside an XMPP Message, would parse the JSON notification, and would then promptly contact the server or notify the user as appropriate. This resulted in near instant change propagation, easy deployment, it looked good on all fronts.
I think I know what you’re thinking right now, and yes, of course, this was a first implementation and it had lots of potential weak spots, for example, it required us to ask our customers to also open the XMPP communication ports on their company firewalls (5222 and 5223), connection difficulties could force clients back into polling, the initial implementation was also known to have problems with same-named users (worry not, we nipped that one in the bud). Also, initially we had these messages piggyback on XMPP Message stanzas (A stanza is a self contained block of XML sent over an XML stream) and normally this channel is used for actual chat messages between humans rather than the IQ (Information Query) stanzas that are instead tailored specifically to Data Exchange between XMPP clients and servers.
A typical push notification:
<message from="livebox.com" to="firstname.lastname@example.org/desktop" id="jkl5jb45" type="normal"> <livebox-extension> <!-- Encoded Notification Payload with unique ID inside --> </livebox-extension> </message>
This is a sample XMPP Message stanza, sent from our Openfire server to a LiveBox client application, the sender and recipient are identified respectively by the “from” and “to” fields, and the payload is set inside a packet extension “livebox-extension” in this case. The type can be “Normal” or “Chat” in our case normal was more appropriate as notifications did not require thread grouping, this changes how the server handles messages.
But let’s not jump bases. That was in the very early stages of LiveBox itself and there were no proper chat clients nor a very stringent protocol, but we knew quick change propagation and low idle system load were precious features and so we pushed on.
This plugin was aptly named “Push Servlet”, a matter of fact name for a very specifically developed plugin. The first step in its evolution was defining a tiny protocol through which LiveBox clients could “sign-up” for notifications with the plugin communicating over XMPP IQ packets, this reduced the number of messages aimed at offline devices or at users that hadn’t registered any devices yet. In this second iteration for each message, the Push Servlet plugin would cross the list of online recipients with registered recipients.
IQ Sign-up example:
<iq from="email@example.com" to="livebox.com" type="set" id="skf462"> <query xmlns="xmpp:agent"> <action actiontype="0"> <log>true</log> <notification type="message"/> </action> </query> </iq>
In the above snippet a LiveBox client informs the Push Servlet plugin it accepts notifications. Let’s examine it in a little more detail.
The stanza is of the IQ (Information Query) type, IQ packets can have 4 relatively straightforward types, “set”, “get”, “result”, “error” (see RFC3920 section 9 for details on valid stanzas and semantics), and its payload will be delivered to our plugin based on the xmlns namespace. and inside we find the action element (the kind of operation, being sign-up or sign-out), and whether to log. We also find the notification element, whose type attribute determines whether notifications will be sent over XMPP Message or XMPP IQ (the plugin will fall back to XMPP Message if it is not specified, to ensure backward compatibility with old client versions) Note: IQ packets can only have one child element, whereas messages can have as many as they please.
This was of course still a little off the mark, what would happen if you went briefly offline and then came back online? Any messages sent during this time frame would be lost (Openfire does save messages for delayed delivery but it is an internally managed cache and cannot be relied upon that much). The consequence of course was far worse, since we could not reconstruct missing operations that occurred in this time frame, clients would have to call the server to retrieve a large dataset to cross check for changes.
So we went about extending Push Servlet with the ability to store messages until delivery was possible, Push Servlet would now save messages on a MySQL table (just like it saved Registered Clients on another), when a client came back online and notified Push Servlet, it could request the backlog of lost messages , and Push Servlet would diligently wrap them up and send them over.
However, a much bigger improvement in reliability was achieved when we added a unique push id in the notifications, clients could now acknowledge message receipts, which would then be written off as delivered by Push Servlet, otherwise a resend would be attempted after a short timeout.
An example of Receipt acknowledgement
<iq from="firstname.lastname@example.org" to="livebox.com" id="s23e9f" type="set"> <query xmlns="xmpp:agent"> <action actiontype="3"> <messagereceived> 146979 </messagereceived> </action> </query> </iq>
This is a typical message acknowledgement stanza for a message delivered over XMPP Message (A newer shorter result form is available for newer clients), you may notice the “actiontype” 3 (Acknowledgement) and the Push ID in the “messagereceived” element.
This meant we now had a stable delivery system towards Windows and OSX clients, with next to no load on the LiveBox server, and a negligible load on the Openfire server. Of course, in case of dire network issues LiveBox clients still do fall back to polling, but you know, Hard Times call for Hard Measures. Most of the time we have snappy notifications and light loads.
Further developments of the Push Servlet plugin included error reporting, sending messages as IQs (Gaining automatic receipt Acknowledgments and send error management in the process), performance reporting functionalities, for example, even on our smallish development server where most of us work delivered an average of 10-20 thousand notifications per month, of course, the plugin counts deliveries to several computers of a given account as separate events).
In the next installment I hope to discuss the birth of the mobile apps and LiveBox chat. I also know very well some of you will be thinking Push Notifications have more to do with GCM (Google Cloud Messaging) and APNS (Apple Push Notification System), I can safely reveal that… yes, of course they do, stay tuned, we’ll get there within a couple of articles.
As always, comments, requests for clarification and insights are very welcome.