Performance of Fudge Persistence in MongoDB

Here at OpenGamma we make considerable use of MongoDB, in particular as a persistent store for data which we either don't want to spend the time on normalizing to an RDBMS Schema, or where we positively value the schema-free design approach taken by MongoDB.

We also make extensive use of the Fudge Messaging project for virtually all of our distributed object management needs (whether using Fudge Proto files, or manually doing the Object/Document encoding ourselves). Luckily, the two work extremely well together.

Because of the way that we defined the Fudge encoding specification, and designed the major Fudge Reference Implementation classes and interfaces, it's extremely easy to map the worlds of Fudge, JSON, and XML (in fact, Fudge already supports streaming translations to/from XML and JSON). We've actually had support for converting in between Fudge objects and the BasicDBObject that the MongoDB Java driver uses since Fudge 0.1, and we use it extensively in OpenGamma: anywhere you have a Fudge object, you can seemlessly persist it into a MongoDB database as a document, and load it back directly into Fudge format later on.

So with that in mind, I decided to try some performance tests on some different approaches that you can take to go from a Fudge object to a MongoDB persisted document.

Benchmark Setup

The first dimension of testing is the type of document being persisted. I had two target documents:

Small Document

This document, intended to represent something like a log file entry, consists of 3 primitive field entries, as well as a single list of 5 integers.

Large Document

This document, intended to represent a larger concept more appropriate to the OpenGamma system, consists of several hundred fields in a number of sub-documents (sub-DBObject in MongoDB, sub-FudgeFieldContainer in Fudge), across a number of different types, as well as some large byte array fields.

I considered had three different approaches to doing the conversion between the two types of objects:

MongoDB Native

In this case I just created BasicDBObject instances directly and avoided Fudge entirely as a baseline.

Fudge Converted

Created a Fudge message, and then converted to BasicDBObject using the built-in Fudge translation system

Fudge Wrapped

This one wasn't built in to Fudge yet (and won't be until I can clean it up and test it properly). I kept a Fudge data structure, and just wrapped it in an implementation of the DBObject interface, which delegated all calls to the appropriate call on FudgeFieldContainer.

Additional parameters of interest:

• Used a remote MongoDB server running on Fedora 11 (installed from Yum, mongo-stable-server-20100512-mongodb_1.fc11.x86_64 RPM) running on a VM with reasonably fast underlying disk.
• Local MongoDB server was 1.4.3 x86_64 running on Fedora 13 on a Core i7 with 8GB of RAM and all storage on an Intel SSD
• MongoDB Java Driver 1.4 (pulled from Github)
• JVM was Sun JDK 1.6.0_20 on Fedora 13 x86_64

Benchmark Results

Test Description	MongoDB Native	Fudge Converted	Fudge Wrapped
Creation of 1,000,000 Small MongoDB DBObjects	539ms	1,603ms	839ms
Persistence of 1,000,000 Small MongoDB DBObjects	41,188ms	46,201ms	92,866ms
Creation of 100,000 Large MongoDB DBObjects	15,351ms	23,956ms	15,785ms
Persistence of 100,000 Large MongoDB DBObjects (remote DB)	57,207ms	60,511ms	56,236ms
Persistence of 100,000 Large MongoDB DBObjects (local DB)	66,557ms	74,763ms	58,816ms

Results Explanation

The first thing to point out is that for the small DBObject case, the particular way in which MongoDB encodes data for transmission on the wire matters a lot. In particular, there's one decision that the driver has made that changes everything: it does a whole lot of random lookups.

A BasicDBObject extends from a LinkedHashMap, and so doing object.get(fieldName) is a very fast operation. However, because Fudge is a multi-map, we don't actually do that in Fudge, and by default we store fields as a list of fields (JSON stores lists as a, well, list; Fudge stores them as repeated values with the same field name). Because this makes point lookups slow, we intentionally do whole-message operations as often as we can, and just iterate over all the fields in the message.

The MongoDB driver code does the same thing, but instead of doing a for(Entry entry : entrySet()) style of operation, it iterates over the keys and does a separate get operation for each key. In Fudge, this is potentially a linear search through the whole message.

To work around this, in my wrapper object I built up a map where there was only a single value per field. This works well, but the small document case has 1/6 of the fields be a list, making this test thrash in CPU on doing the document conversion (which explains why the small document persistence test is more than twice as fast with the wrapper as just rebuilding the objects). Yes, I could do this optimization further, but it would be difficult to improve on the combined setup (document construction) and runtime (persistence) performance of just building up a BasicDBObject, which is what the Fudge conversion does anyway.

The wrapped Fudge object wins in every case for the large document test, no matter how many times I run them (and I've done it quite a few times for both local and remote, with all outliers eliminated). Moreover, I actually get faster performance running on a remote DB than on a local DB (which surprised me quite a bit).

The only things that I can conclude from this are:

• FudgeMsg limits the data size on insertion into the message (when you do a msg.add() operation, not on serialization) for small integral values (if you put in a long but it's actually the number 6, Fudge will convert that to a byte). However, the ByteEncoder which converts values in MongoDB to the wire representation will never do this optimization, and will actually upscale small values to at least a 32-bit boundary. This means that if you put data into a FudgeMsg first and then put it into the MongoDB wire encoding, you shrink the size of the message. Given the number of pseudo-random short, int and long values in this message, it's a clean win.
• The object churn for the non-wrapped form (where we construct instances of BasicDBObject from a FudgeFieldContainer) causes CPU effects that the wrapped form doesn't suffer from.

Conclusion

One of the things that was really pleasant for me in running this test is just how nice it is to take a document model that's designed for efficient binary encoding (Fudge), and persist it extremely quickly into a database that's designed for web-style data (MongoDB). The sum total of the actual persistence code is all of about 10 lines; I spend far more lines of code building the messages/documents themselves.

The wrapped object form definitely wins in a number of cases. My current code isn't production-quality by any means, but I think it's a useful thing to add to the Fudge arsenal. That being said, I think the real win is to rethink the way in which we get data into MongoDB in the first place.

Given the way the MongoDB Java driver iterates over fields, it seems to me that a far better solution is to cut out the DBObject system entirely, and write a Fudge persister that speaks the native MongoDB wire protocol directly, and take advantage of the direct streaming capabilities of the Fudge protocol. When we've done that, we should be going just about as fast and efficiently as we can and Fudge will have a seamless combination of rich code-level tools, efficient wire-level protocol for binary serialization, great codecs for working with text encodings like JSON and XML, and a fantastic document/message database facility using MongoDB.

Don't Host Crowd and Jira in the same Servlet Container

This took up quite a bit of my Saturday figuring out, so I figured I'd add some pointers for other people to find.

Atlassian doesn't recommend that you host their applications in the same Tomcat instance, rather encouraging you to deploy them in different Tomcat instances and JVMs through the "Standalone" distributions. However, recommendations never stopped me before, so at OpenGamma we have two different sets of Atlassian infrastructure:

• One set of Confluence, Crowd and Jira for the FudgeMsg project running in one Tomcat container on one VM
• One set of Crowd and Jira for our corporate use running in one Tomcat container on one VM
• Bamboo and FishEye for OpenGamma corporate use running behind our firewall in their own standalone implementations in their own VMs.

Yesterday I tried to upgrade Crowd and Jira for the OpenGamma corporate installation. It wasn't pretty.

First of all, I ran into this KnowledgeBase issue, where Confluence, Crowd, and Jira all ship with different versions of the Felix jar for plugin management. This stopped the plugin system from starting for Jira (since Tomcat launches the Crowd application first), so Jira was pretty borked.

Then I ran into something far more pernicious, which other people should be aware of.

Reindexing Requires Crowd

If you're running your applications with Crowd, the application delegates user-related information to Crowd and uses a RESTful approach to loading the data. So far, so good.

However, when Jira attempts to in-situ upgrade an installation (which again they don't recommend), it will do its database wrangling, and then reindex the system with the new functionality (in our case, going from 4.0 to 4.1.1 to allow searches based on votes and watches on issues). All this upgrading it does in the application startup logic, which happens in the Tomcat main thread.

When it gets to reindexing, it then attempts about 10 different RESTful calls to your Crowd instance, as it doesn't have any caches populated on user data. However, while Tomcat has opened up port 80 (or 8080 or whatever) for Crowd, it hasn't enabled application dispatch to the Crowd servlets yet.

This means that all 10 remote calls (which are actually local to the single Tomcat instance since both apps are co-hosted) hang, and the entire server startup process fails.

The only workaround is to launch Crowd in its own servlet container, change your Jira's crowd.properties to refer to the new one, startup, and then undo what you've done.

The Moral Of The Story

When your software vendor recommends that you don't do something, don't do it unless you have an exceptional reason to do so.

If you're a software vendor and you support something but don't recommend it, still test it as there will be customers who ignore your recommendations and do it anyway.

The Difference Engine: Give Up And Move To London

So I came about an article about The Difference Engine, which is attempting to be the YCombinator of Europe. I'm going to have to completely and utterly disagree with Mike Butcher here when I give some pretty pointed advice: they should give up right now, this minute, this very round of startups and escape Middlesbrough and move to London.

Editorial Note: If you're offended by some soft Southerner capping on The North, or, worst of all, an American bashing everywhere in the UK that Isn't London, just skip ahead to the comments and start bashing. You won't like the rest of this article, and you'll just start flaming the comments anyway, so save yourself some time.

Mike, you're 100% wrong that Europe is somehow Exceptional when it comes to siting of startups.

First of all, let's deal with the whole "YCombinator Of Europe" thing. YCombinator started out in Cambridge, Massachusetts, which already had a pretty large startup tech cluster. Then they started doing the program half-and-half with Silicon Valley. And then they finally gave up and moved the whole thing to Silicon Valley. They already had experience with several rounds of startups, and several rounds of exits. They saw that the #2 tech cluster in the US, and possibly the world, simply didn't have a big enough ecosystem. Yet somehow Middlesbrough does?

Second, let's deal with the whole "Don't Have To Be In London" thing. You're right, there are a number of European startups that are coming from NotLondon. Mike references Dopplr (Helsinki, but then also has a big London-based office) and Spotify (Stockholm, though how much of a scrappy startup they can be with the most opaque ownership and funding sources in the world is debatable). I'd add a number of Baltic-based startups to that list (like Erply). None of them started out in London, that's very true. But they all did start out in their country's gravity well for top talent.

One thing that strikes me as an American about Europe is that the vast majority of countries have one super-dominant central city that acts as a major pull for talent throughout the country, and which is usually the capital city: London (the UK); Dublin (Ireland); Paris (France); Copenhagen (Denmark); Prague (Czech); Stockholm (Sweden); Helsinki (Finland); Amsterdam (Netherlands, though this almost deserves an asterisk since the whole Utrecht-Rotterdam-Amsterdam cluster is one massive metropolitan area). Probably the place with the least sort of concentration is Germany, with its multiple nearly co-equal metropolitan areas.

All of these places are the dominant suck of talent from their respective nations, and produce the types of network effects that you need for a wide variety of knowledge-based industrial sectors. To me, it's no wonder that you're getting non-London-based startups out of these cities, rather than also-ran cities in each of the countries (how many startups are coming from Marseilles, Brno, Arhus, Den Haag, Valencia? Probably not more than a few). You need to be where the best pool of talent is.

So let's focus on what you need for a successful startup:

• A Pool of Skilled Talent: While The Difference Engine may be able to get these people on the ground in Middlesbrough, what happens when the team needs to go from 2 to 3? What happens when it needs to go from 3 to 10? Where are they going to get the talent? They'll have to leave Middlesbrough, that's what.
• Sources of Funding: This is a longer-term thing for an incubator, but you need to have the sources involved relatively early and often to make it a success; YCombinator flew the Cambridge guys to Silicon Valley just to give them experience with the funding scene, and Cambridge already has quite a few big name VCs. Middlesbrough? How many European VCs have ever been there?
• Entrepreneurial Culture and Mentors: You've got the guys directly involved with The Difference Engine, but what about other people who have run the whole cycle a few times and can act as passive or active mentors?

I contend that none of this is present in Middlesbrough, and quite simply never will be. All of them are present in London today. The single best thing that The Difference Engine could do in order to help its startup founders is to move to London next week.

And here's the best part: it's not going to stay in Middlesbrough. That's right, it's moving to Sunderland. Which means it'll end up with the types of back-and-forth that mean neither place ever sets up any type of cluster effects. Smacks to me of the type of thinking that's led to the European Parliament splitting its time between Brussels and Strasbourg.

Here's what this is, plain and simple: Yet Another Regional Development Agency Thinking One Industrial Park Will Rebuild Its Economy. How many cities and regions say "well, if we just setup an industrial park, we'll all of a sudden be the Silicon Valley Of X"? How many cities, regions, and countries say "well, if we just setup a small venture capital fund we'll build a new tech cluster?" I've seen it all before. And it turns out that the top three sponsors of The Difference Engine are all councils and a regional development agency.

Tech clusters happen organically. They can't be willed into existence by any government or development agency.

Look, I give Middlesbrough and Sunderland councils props for trying to convert their cities from the Grim Northern spots that they're known to be. But they're being, I think, disingenuous to their founding participants.

If you care about the actual founders, rather than just about wasting development funds, get your founders out of the North and into London immediately.

If you're a founder considering The Difference Engine, my personal recommendation is to ignore it. Move to London, get yourself a desk in a shared office in the Shoreditch or Bankside area, and go to town. If you're the type of person ready to do a startup, you'll probably learn more hanging around the vibe than you would sitting in the Grim North for a few months.

Twitter For Messaging: Encoding Binary In Unicode

This week at Chirp, Twitter announced Annotations, which is the Twitter-specific way of saying "you can assign arbitrary metadata to individual twitter updates." After a very quick Twitter (how self-referential) conversation with Alexis from Rabbit Technologies, we agreed that with this, Twitter is essentially trying to build a much more general-purpose pub-sub messaging technology. I wanted to talk about that.

History Repeats: Twitter Is MOM

The use of annotations is extremely familiar to anybody with a background in traditional messaging technologies. In general, publishing a message in a traditional environment requires:

• A destination. In traditional pub-sub, this is a topic name; in Twitter parlance it's the publisher's twitter handle.
• Message metadata. In traditional pub-sub, this is a set of properties (typesafe name/value pairs attached to a message); in Twitter parlance it's your annotations (full definition still forthcoming).
• Message content. In traditional pub-sub, this is in general a byte array (though specs like JMS allow for code-level specifications that ultimately resolve to a byte array); in Twitter parlance this is your 140-character tweet.

Now it looks like we've got some pretty good equivalencies; every major headline element is covered by Twitter. So how do you adapt to an All Twitter world?

I would say the starting point is the message content. I live in a world of machine-to-machine communication (people are messy). Byte arrays don't match up with character data.

Or do they?

An Historical Diversion: BinHex64

Let's consider a problem that impacted technology professionals back before most Ruby programmers were alive: how do you transmit binary data over the internet?

You had two options:

• Use a binary protocol, written from scratch or using something like RPC. This worked, but required endpoints that understood the protocol.
• Transmit data over a text encoding, like email or usenet. This allowed for the greatest amount of interim-stage compatibility, but had serious interoperability issues.

The primary interoperability problem with transmitting binary data over text protocols was a pretty simple one: most Internet protocols from the Dawn of Time were written by ignorant Americans and thus only supported 7-bit ASCII. Binary data inherently is 8-bit: you're trying to transmit a byte array, and each byte has 8 bits. How do you fit a square peg (8-bit binary) into a round hole (7-bit ASCII)?

The solution is BinHex encoding. The basic idea is that you attempt to represent a high-fidelity source dataset (8-bit binary) in a low-fidelity target encoding (7-bit ASCII) by increasing the size of the encoded message to fit the target encoding.

This seems stupid and archaic, but it's still the way binary attachments go into email.

Enter The Reverse BinHex

At first glance it might not seem that way, but Twitter represents the world in a reverse form to BinHex encoding. With BinHex encoding you're trying to fit 8-bit bytes into a 7-bit world; with Twitter tweets you're trying to fit 8-bit bytes into a "character" world. The only thing that's germane is "what is a character?"

Twitter is quite clear: A Twitter character is a Unicode code point. If it weren't so, Twitter wouldn't be able to handle localized tweets as well as it does.

Right, now we're cooking with gas.

A Unicode code point is, in the most broad brushstrokes possible, drawn from one of two distinct sets of planes:

• The Basic Multilingual Plane, consisting of code points in the numerical region from 0x0000-0xFFFF.
• The Astral Planes, currently allowing code points in the numerical region from 0x100000-0x10FFFF.

According to their character count page, Twitter uses UTF-8 in its internal representation. Let's consider two distinct possibilities:

• Twitter only handles the Basic Multilingual Plane. If that's the case, in general, one Twitter character can handle 2 8-bit bytes.
• Twitter handles the full theoretic range of Unicode including the Astral Planes. If that's the case, one Twitter character can handle 2 and a half 8-bit bytes (ignoring the Supplementary Private Use B plane, to make things simpler).

If we ignore all complicating factors, what can we thus store in a Twitter 140-character tweet if we're trying to encode machine-readable byte arrays?

• 280 bytes if Twitter only supports the Basic Multilingual Plane; or
• 350 bytes if Twitter supports the vast majority of the Astral Planes.

These don't seem like a lot, but if you allocate a few bytes (Twitter Encoded of course) for message sequence number and chaining, and you use a compact binary representation like Avro or FudgeMsg, you can get a lot of data into 280/350 bytes.

So if you use this theoretical reverse-BinHex encoding system to expand byte arrays into Twitter Messages (after full Annotation support is released), you can get arbitrary metadata for routing decisions, and a 280/350-byte binary payload. Enough clearly for a lot of uses.

Twitter As The New Machine-to-Machine Cloud Service

Don't be daft. This is entirely a thought experiment about how you could encode Real Data into a Tweet. If you attempt to hook multiple machine processes up through Twitter as a data distribution mechanism you are a moron.

If you're interested in that type of functionality, you should talk to Rabbit or another Cloud Messaging Provider (I'm sure there will be competition forthcoming). Cloud Messaging makes sense; using Twitter as your Cloud Messaging Provider is completely stupid.

Seriously. There are certainly use cases where you can see machines, people, and other machines communicating over Twitter. But if you're going to the point of converting binary data into arbitrary Unicode codepoints for transmission over Twitter, you completely fail at asynchronous communication, and should be required to spend at least 6 months doing nothing but implementing sections from EIP as penance.

Looking For a WordPress Theme Developer

In case you hadn't noticed, the OpenGamma web site doesn't really have a lot of content on it. We intend to change that.

We've got content broadly ready to go.

We've got a designer working on HTML templates for the content.

We're looking for a freelance designer to convert everything to a WordPress theme for a (mostly static) site that will definitely include a blog.

Nope, you don't need to be in London. You could be virtually anywhere in the world. Send an email to jobs or info or even kirk, all at opengamma.com.

My MP's View On The Digital Economy Bill

Bases on the tools provided by 38 Degrees, I contacted my MP to urge proper debate on the Digital Economy Bill facing the current parliament here in the UK.

Apparently I wasn't the only one, as his office had a full form response prepared and ready to go. Here's what he sent.

Full Text:

Thank you for contacting me about the Digital Economy Bill.

For nearly twelve years, the Government has neglected this crucial area of our economy. We believe a huge amount needs to be done to give the UK a modern regulatory environment for the digital and creative industries. Whilst we welcome aspects of the Bill, there are other areas of great concern to us.

We want to make sure that Britain has the most favourable intellectual framework in the world for innovators, digital content creators and high tech businesses. We recognise the need to tackle digital piracy and make it possible for people to buy and sell digital intellectual property online. However, it is vital that any anti-piracy measures promote new business models rather than holding innovation back. This must not be about propping up existing business models but creating an environment that allows new ones to develop. That is why we were opposed to the original Clause 17 and are still opposed to CLause 29, which props up ITV regional news with License-Fee-payer's money.

The Government's failure to introduce the bill until the eleventh hour of this Parliament has given rise to considerable concern that we no longer have the time to scrutinise many controversial measures it contains. We believe they should be debated in the House of Commons, and only if we are confident that they have been given the scrutiny that they deserve will we support them. My colleagues in the Shadow Culture, Media and Sport and Shadow Business, Innovation and Skills teams will do everything in their power to work towards legislation that strengthens our digital sector and provides the security that our businesses and consumers need.

Once again, thank you for taking the time to write to me.

OpenGamma is Looking for a Browser-Based Software Engineer

We've already put this up on our jobs page, but I wanted to highlight the job posting to my readers.

OpenGamma is now looking for someone to build up our browser-based software engineering efforts. We've got a super-strong set of server-side software engineers who are well versed in building the back-ends of applications (and delivering data to front-ends in browser-friendly ways). We've got people who are very familiar with extending well-defined applications to support new functionality. What we don't have is someone who lives and breathes the browser.

That's where you'd come in.

We want someone who can come in, and make sure that we present the platform that we're building in the best way possible to end users, delivered through browser-based mechanisms. You'd get a clean slate to work with, and the chance to work against what we know will be the limits of what browsers can do. And at no point ever will we ask you to support IE 6.

While we're building financial technology, we don't think you need to know a single thing about the financial services industry to take on this role; in fact, we think our ideal candidate isn't coming from finance at all (judging by the quality of web applications we've all seen in finance). Anything you need to know you'll pick up on pretty darn quickly.

We'd prefer someone local to London (our new offices are in the Bankside area, with views of the Tate Modern). If you're based outside the M25 and need accommodation for telecommuting, we don't need you in the office every single day.

We're a well-funded startup, we're building technology that has the potential to disrupt an entire industry, we have exceptional people to work with, we have a no-bullshit stance on bureaucracy, and we all have an equity stake in the firm.

Take a look at the more comprehensive spec on our web site, and if you think you or someone you know would be perfect, contact us (jobs at opengamma dot com).

Recruitment Agencies: We are not open to unsolicited profiles or CVs for this role without an existing MSA signed by OpenGamma.