Whence Fudge? Why Not Just Use/Extend Avro/GPB/Thrift?

Or, In Which I Make The Case For Fudge Existing At All

tl;dr: Fudge is binary for speed and self describing to do interesting things when you don't have schema support at the time of processing.

After I announced the Fudge Messaging 0.2 release I realized that we had a back-link from @evan on Twitter with a quite pithy characterization of the situation. I've followed up with Evan quite a bit over twitter, and I think this requires a bit more detailed description of why Fudge is different from other encoding systems.

What Do You Transmit On The Wire?

Let's start with a very simple question: if you're trying to transmit data in between machines, whether temporally concurrent or not, how do you encode said data? Broadly speaking, you have a number of options:

Hand-Rolled Binary Protocol

Yep, you could directly emit the individual bytes in the appropriate byte order and hand-roll the encoders and decoders, and it'll probably be pretty fast. It'd be a complete and utter waste of development resources, but fast it would undoubtedly be.

Compact Schema-Based Representation

The next option is that you use a system based on a schema definition, and use that schema definition and the underlying encoding system to automatically do your encoding/decoding. Examples of this type of system are Google Protocol Buffers, Thrift, and Avro. This is going to be fast, and an efficient use of development resources, but the messages are useless without access to the schema.

Object Serialization

Every major Object-Oriented language has some built-in serialization system for object graphs, and it's usually dead easy to use. However, they all suck in a number of ways, and don't work at all if you don't have matching code objects on both sides of the communications channel.

Text-Based Representation

You could just say "I don't care about performance at all" and just use a text-based encoding like XML, JSON, or YAML. The messages are at least marginally human readable, and you can do stuff with them without any schema (which may or may not be used at all), but it's going to be slow, slow, slow.

Compact Schema-Free Representation

The final option is to say "I like having my data in a compact binary representation, but I don't want to have to have access to any type of schema in order to work with messages." The most well known example of this is TibrvMsg from Tibco Rendezvous, but it's not a very good implementation, and it's proprietary. This is what we created Fudge to solve.

Hey, Isn't Avro Schema-Free?

No, no it's not. The Avro encoding system requires access to a schema in order to be able to decode messages at all, because all metadata is lost during encoding. However, it has two characteristics which are similar to schema-free encoding systems:

• It decodes into a schema-free representation. What I mean by this is that if you did have the schema for a message, you can decode it into an arbitrary data structure which isn't tightly bound to the schema (think generic structure vs. schema-generated code).
• The communications protocols involve out-of-band schema transmission. When you start communicating using RPC Avro semantics, you communicate the schema of the messages that you're going to be transmitting using a JSON-encoded schema representation. But once you've done that handshake, the actual messages are schema-bound.

Thus you can view "Avro the communications protocol" as schema-free, as you don't have to have a compile-time-bound schema to be able to communicate effectively, but "Avro the encoding system" as schema-bound.

Why Is Schema-Free Encoding Useful?

If you're looking at temporally concurrent point-to-point communications (think: RPC over a socket), a schema-bound system is pretty useful. You can squeeze every single unnecessary byte out of the communications traffic, and that's pretty darn useful from an efficiency perspective. But not all communications are temporally concurrent and point-to-point.

Let's consider the other extreme: temporally separate publish/subscribe communications (think: market data ticks, log messages, entity update messages). In that case, you have to make sure that every single point where you want to process the message has access to the schema used at the point of message serialization; if you don't have it, all you have is an opaque byte array. That's a logistical nightmare to keep everything in sync over time, particularly when you start doing logging and replay of messages for debugging and auditing.

Moreover, there's a whole host of things that you can do in between producer and consumer if you have a schema-free encoding:

• You can visually inspect the contents of messages. Not in a text editor unless you're using a text-based encoding system, but in some type of general purpose tool. This is an unbelievably useful development, debugging, and system support tool.
• You can do content-based routing on them. It's quite easy using XML, for example, to create XPath rules that route messages through the system ("send messages where /Company/Ticker = AAPL to Node 5; send other messages to Node 7"). You don't have to compile each little rule into native code, you can do it based purely on configuration. Heck, you could even build a custom routing system for ActiveMQ or RabbitMQ and put the functionality right in your message oriented middleware brokers.
• You can automatically convert them to other representations. Have data in XML but need it in JSON? No problem; you can auto-convert it. Have data in Fudge but need it in XML? No problem; you can auto-convert it.
• You can store and query them efficiently. You can take any arbitrary document and put it into a semi-structured data store (think: XML Database, MongoDB), and then query it in an efficient way ("find me all documents where /System/Hostname is node-5 and /Log/Type is AUDIT or SECURITY").
• You can transform the content using configuration. You can take an arbitrary message, and just using configuration (so no more custom code) change the contents to take into consideration renames, schema changes, add stuff, remove stuff, do whatever you want.

The SOA guys have known about these advantages for years, which is why they use XML for everything. The only primary flaw there is XML is, quite frankly, rubbish in every other way.

That's Why We Created Fudge

Most of the network traffic in the OpenGamma code isn't actually point-to-point RPC; it's distributed one-to-(one or more), and we use message oriented middleware extensively for communications. We needed a compact representation where given an arbitrary block of data, we could "do stuff" with it, no matter when it was produced.

None of the compact schema-free systems support this type of operation. If someone can point me to an Open Source system which covers these types of use cases, and it's better than ours, we'll gladly merge projects. But Avro ain't it. Avro's good for what it does, but what it does isn't what we need.

But Your Messages Must Be Huge

The Fudge Encoding Specification was very carefully designed to scale both down and up in terms of functionality. At the most verbose, for each field in a message/stream (of course Fudge supports streaming operations; if it didn't, do you think we could efficiently do all those auto-transformations?), the Fudge overhead consists of:

1-byte Field Prefix

This has the processing directives to say what else is in the stream for that field.

1-byte Type ID

We have to know at the very minimum the type of data being transmitted for that field.

The Field Name

The UTF-8 encoded name ("Bid", "Ask", "System", "UserName", etc.)

A 2-byte Field Ordinal

A numeric code for the field (nope, not the index into the message, just a numeric ID like the name is a textual ID)

Here's the thing though: only the field prefix and type ID are required. Although right now Fudge-Proto doesn't do so, you could easily use Fudge in a form where you rely on the ordering of fields in the message to determine the contents (which is exactly what Avro does). In that case, you pay a 2-byte penalty per field. Yes, if you're transmitting a lot of very small data fields that's a relatively high penalty, but if you're transmitting 8-byte numbers and text, it's really not a whole heck of a lot.

You want names so that you can have something like Map<String, Object> in your code, but you don't want to transmit those on the wire? That's what we invented Fudge Taxonomies for. 2-byte ordinal gives you full name data at runtime, without each message having to include the name.

So in one encoding specification, we can support very terse metadata as well as very verbose metadata.

Conclusion

So ultimately, we created Fudge to be binary (for speed and efficiency), self-describing (for schema-free operation even when a schema exists), and flexible (so you can only use the bits you need).

Yes, we could have started with an Avro or a GPB or a Thrift, but there's no way that you could have done what we've done without breaking existing implementations. And I think if you started with an Avro and worked down all the use cases we've run into in the past, you'd end up with Fudge in the end anyway; we just skipped a few steps.

The binary schema-bound systems are great and they define things like RPC semantics which Fudge doesn't. But hopefully it's clear that us creating Fudge wasn't just a case of NIH, it's that we had very specific use cases that they can't support naturally.

Fudge Messaging 0.2 Release

After I initially announced the first release of the Fudge Messaging project, we've had a lot of feedback on the code and on the specification, and we've been using it in anger quite a bit here at OpenGamma.

We've also been making a lot of enhancements, and are pleased to announce the 0.2 Release of the combined Fudge Messaging project.

While there have been a number of enhancements throughout the code, the primary focus of this release has been on making it easier for you to start using Fudge in your applications. With that focus has come a number of significant new features that you should be aware of:

• The Fudge Proto sub-project has been released for the first time. Taking our cue from Google Protocol Buffers, we have a .proto file format which can then generate out Java and C# language bindings for automatic conversion of objects to and from the Fudge encoding specification.
• We've made it substantially easier to build up an Object Encoding/Decoding Dictionary so that if you want to hand-write your mappings between Fudge messages and Objects, you can still access the auto-conversion functionality in the Fudge Proto project.
• We've built up a number of Automatic Object Conversion systems, so that even if you have nothing other than Java Beans/POJOs/PONOs, you can still automatically convert those to and from Fudge messages.
• We've developed Streaming Access APIs so that you can consume and write Fudge fields without having to work with an entire message as a bulk operation (we'll be using this for our automatic conversions to and from XML and JSON).
• A C Reference Implementation has been written. Already tested on a number of different Unix-like platforms and Windows, this will be the foundation of our eventual C++ version. As well, it can be used to support Fudge messaging in your favorite language which allows easy C bindings.

Our next release (0.3), will really be a release candidate release for 1.0. Our primary focus there is to work through any features that have been submitted by the community, and to finalize the one piece of the binary encoding that hasn't been completed (moving from Modified UTF-8 to true UTF-8, which won't affect you unless you want to encode Unicode null or certain high-bit multi-byte Unicode characters).

If you've thought having a self-describing, compact, binary encoding system ideally suited to distributed applications was interesting, but didn't know if we were ready for you to use in anger, don't wait. It's ready.

OpenGamma Set To Welcome Stephen Colebourne To The Team

As you can read from Stephen's blog post on the subject, he's coming to join the OpenGamma team in the beginning of March.

On JSR-310

Just to reiterate what he said in his post, one of his first duties at OpenGamma will be to get JSR-310 to at least reference draft status. When we started building our platform we made a very early bet that JSR-310 would be the ultimate future of dates and times in Java. That decision has brought a lot of power on working with dates and times for us (for example, we haven't had to build our own system on top of java.util.Date that every financial firm I've ever worked for has done), but we think that with a strong, concerted effort we (as a community) still have a chance to get JSR-310 as the defacto and dejure future of dates and times in Java.

Stephen mentioned earlier on the mailing list that a lack of time outside work has contributed to the slow-down of JSR-310. We're giving him time and resources during the working day, and waiving ownership of the code that he produces.

We're not hiring him just to work on JSR-310, but it seems obvious to me: financial software probably depends more than other software on having strong tools for working with the multiplicity of date and time rules that proliferate in finance. If giving Stephen the time he needs to finish off JSR-310 gives us those tools in a standardized way that the whole community can benefit from, that's far better than building Yet Another Proprietary Date/Time System.

On Non-Financial Hiring

When I mentioned in passing to a few people that Stephen was joining the team, they were quite surprised, because they thought from my original blog post on OpenGamma hiring that we were only looking for financial industry professionals. That was true at the time I posted it, but it's not true today.

If you take a look at the current OpenGamma Jobs page, you'll see that we put far more prominence on pure programming skill these days than financial industry expertise. Sure, if we have a choice between two candidates, identical in every way and one comes from the financial industry and one doesn't, we'll choose the one from the financial industry. But people are unique. We never see two candidates that are identical in every other respect.

And that's why we're eager to hire people who don't come from finance. We're building software that's complicated enough that we need the best across the entire software development industry. Unlike a common attitude I encountered by recruiters when I first moved to the UK, we're not under some assumption that the only people who are any good are the ones working at a Tier-1 bank or a hedge fund.

We made an offer to Stephen because he's a fantastic software developer and architect that we think will make an immediate and long-term impact on our success. We're thrilled that he's decided to join us.

We're Still Hiring

So if you were on the fence before, or you thought that we only wanted someone with N years of experience at some bank somewhere, think again. We want the best, no matter what their background. Email us your CV.

Content Management System Advice?

Dear Lazyweb,

If you've not seen the OpenGamma website yet, you'll note that it could best be described as a "holding page." It says that we exist and are hiring, but not a whole heck of a lot more.

We're currently writing content for a Real Web Site where, amongst other things, we'll actually say what is that we're doing. For that, we need some way to manage content.

Our current four contenders is:

Confluence

We like Confluence as a wiki, and are using it for our internal wiki as well as the Fudge Messaging web site. Other people we know use it as their primary web site management platform. Should we follow them?

WordPress

Not just a blogging system, but a whole content framework! What's not to love! Aside from the performance of course, but just bung more VMs at it and we'll be fine.

Movable Type

Movable Type differs from WordPress in one primary way that we can see: it's primarily static in nature, which we like as it simplifies the hosting and performance quite a bit. Should we roll that out?

Drupal

Personally I've never actually used it in any way (unlike everything else above), but there are lots of very happy users.

We're trying to figure out which one we're going to roll with, because much will then guide our choice of the person/people who actually do the web site generation for us (we're financial technologists, not web technologists, and we're not stupid enough to try to do it ourselves).

With that in mind, Internet Flamewar/Fanboi War ON.

Love,
Kirk

Chris Dixon's Startup Requirements: My Perspective

I like Chris Dixon. I've never met him, though I hope to, but his writing across Twitter, his Tumblr log and his blog are so evocative of his personality that I feel as though I know a few things about him already. Okay, enough of the fellow-startup-blogger-gushing.

Onto the posting.

Some of these are right on. Some of them aren't. Some of them are culturally relative. I wanted to pick out a few of them.

Stuff Chris Says You Need

Beer on Fridays

Come to London. We have good pubs. Everywhere. Beer everyday.

In fact, Thursday Is The New Friday here in London. So many people get out of town for the weekend we seldom go out together on Friday, because that's really intruding on Family/Friends/Personal Time (keep reading for that).

So in summary: Beer Good; Beer Not Only For Friday.

Proximity to Public Transportation

Again, come to London. Yep, our transport system largely sucks. But it sucks in the same way that the NHS "sucks", in that it transports more people more miles every day than virtually any other system in the world.

Nobody would even think to have a startup in London without being close to public transport, although Shoreditch is pretty much the worst Zone-1 location I could think of, and that's where most startups have located. C'est la vie.

Proximity to Park

This is where we kick ass over Manhattan. Y'all have like 2 parks. We have parks on virtually every other corner it seems. Almost no matter where you are, there's a park right around the corner. Sometimes it's big, sometimes it's small, but you can't really go a half mile radius without seeing a tree the way you can in Manhattan.

Mac laptops with second monitors

I like this idea, and when I saw the Cloudera offices in person, they had this policy. OpenGamma is at the "employees choose what they want" stage, so I'd say that from our perspective, if you're not doing a pure web startup, the better phrasing here is "whatever computers are necessary for developers to succeed." I prefer to let them decide that [1]. The key takeaway here to me is "don't have stupid Corporate policies regarding computers; focus on nonstop productivity."

Health care plans for everyone

OpenGamma is in London. We have the NHS. We love it. By virtue of merely residing in the United Kingdom of Great Britain and Northern Ireland, we don't ever have prospects saying "I'd come to work for you, but I'm worried about health care and COBRA and blah blah blah." Every single startup in the US should be pushing for a single-payer system.

Stuff Chris Says You Don't Need

Fancy (Aeron) chairs

Okay, lemme get on my high-horse here. Ergonomics matters. A lot. If you lose 2 days per quarter due to someone having back or shoulder pain, you're net negative on the cost of the chairs.

Don't get me wrong, I completely understand where Chris is coming from where the Aeron is, in essence, the very symbol of DotCom excess. I get it. But saying "buy the cheapest chairs you can" is completely counterproductive. Aerons have actually come down in price (and you can get them second-hand and reconditioned for pretty competitive prices against second-tier new chairs), so you're looking at a price that if you search out a new chair significantly cheaper, you're basically buying something from Office Depot that's going to hurt everybody's backs.

Don't go crazy on the chair front, but don't just say "good chairs were a sign of DotCom excess and therefore you shouldn't have them at all." I actually bought an Aeron for home; I don't expect that my employees or I are going to work on something worse than I'd be willing to work on for my home-time hacking. Will OpenGamma get Aerons? Maybe. Will we get the cheapest chairs on the market to show how hard/31337 we are? No.

Vacation policy

I completely agree that you shouldn't track it, but at least here in the UK your employment contract will stipulate a minimum holiday policy. We don't currently track it; we trust people not to abuse it. At some point we'll grow to the point where that doesn't work anymore, as it will at every startup.

But don't excuse that with saying "our Vacation Policy is that nobody should take one." That's short-sighted and stupid. Startups are a marathon, not a sprint, and nobody can maintain a massive long-term effort without taking time off to rejuvinate. You pressure people to never take proper time off and you'll end up with zombies who aren't functioning at 100%. And I at least would much rather have people at 100% for 90% of the time [2] than tapering down to 70% for 95% of the time. Because I can do math [3].

Business cards

I thought this too. But OpenGamma sells into a vertical, and we know we're going to be an enterprise sale. Tried to postpone it as long as possible, but if you're not consumer and need an enterprise presence you actually have to do this and stationary and the rest of the things that make you seem more than a few guys and a serviced office. It sucks, but you have to suck it up.

Phone system

See "Business cards." Plus, I would never expect an employee to have their personal mobile number as their work number. See "Vacation policy" and switching off: you work 100% when you're at work; when you leave, it's family/personal time.

I'd be tempted to try an experiment where we all have company-provided mobile phones, but here in the UK thanks to Caller Party Pays you can tell instantly if you're calling a mobile phone just based on the number. It doesn't seem professional. So if I have to setup a whole call forwarding system, I may as well setup a VoIP system.

Central air conditioning

You ever tried to work in an office in London (or Manhattan for that matter) in August with a pretty high concentration of workstations and 30" monitors? Windows opening doesn't help when all air is stagnant, high humidity, and hot.
Update 5 minutes after posting. Chris was originally talking about the types of systems where the building controls your A/C and you don't have per-tenant control. Now I agree with him 100%. You need A/C. You also need 24/7 control over it to the point of individual zones in your offices.

Carpeting

OpenGamma's moving to a permanent office shortly. It has hardwood floors. We asked them to put in carpeting. Because we care about people having the ability to work quietly, and hardwood floors and open plan offices don't work well to minimize noise. Carpeting is great for sound reduction, which is critical to allow people to work effectively in an open-plan environment. That and monstrous headphones.

Conclusion

Chris, you rock for giving me a good excuse to blog about a whole bunch of small matters that I agree are important to startups (it's just that my list is somewhat different to yours).

Next time you're in London, let me buy you a non-Friday beer. Next time I'm in NYC, let me buy you a Friday beer.

Footnotes

[0]: Yep, I used the dl tag. Bringing it back old-skool, yo. werd
[1]: Interestingly, we offered all employees the choice. 2/3 of current OpenGamma employees are running Linux on the desktop for development, even when offered the choice of a Mac Laptop or Mac Pro. 1 runs a Mac Pro, 1 runs Windows 7. Of the 3 company owned laptops, 2 run OS X and one runs Linux.
[2]: Yep, we get 24 days plus public holidays here at OpenGamma. We're typical of London.
[3]: In that I can do simple sums. We have a quant team at OpenGamma because I can't even fathom the mathematics necessary to drive our system. Have I mentioned that I'm the only founder without a PhD? Yeah....

Confluence, iSCSI, NetApp, Flexiscale, Fail

We've been hosting the FudgeMsg website using Confluence (and mad props for the free Open Source license by the way!) on a VM hosted by Flexiscale. We chose Flexiscale for the following reasons:

• Confluence is ridiculously tricky to cluster, meaning that you can't benefit from the scale-out capabilities of the Amazon EC2 model. (Edit 2009-12-24 - We're not attempting to cluster it; this statement is to indicate that because it's so tough to cluster, we're not trying to. Confluence is thinking we might be, and crashing as a result).
• Getting your AMI+EBS+ElasticIP for such a single-point vendor app is very difficult and time consuming, again meaning that the Amazon EC2 model isn't ideal.
• We didn't want to go for dedicated hardware for a low-volume application stack.
• We don't have a stable enough internet connection to host the application stack ourselves.

We thought we were pretty darn clever to be honest, but then we noticed that things started going wrong. In particular, we started getting this error a lot:

We were running a relatively complicated Java setup (jsvc to run as chrooted user, on Tomcat, with multiple web applications running), so we thought we had done something wrong. After all, the first rule in software engineering is always, always, assume you have caused the break.

We were wrong.

What we've since found out is that the entire Flexiscale model is that your VMs have effectively no local storage whatsoever, and everything is iSCSI hosted off of a NetApp cluster. That means that your storage is fully persistent, and survives migrations of your VMs across physical hardware. Which is a great concept when it works.

Except that the Flexiscale NetApp cluster is borked. Essentially, sporadically the iSCSI services will completely shut down. Sometimes for a second, sometimes for 4 hours. Your processes will still be running, but if they attempt to hit disk for any reason, they'll hang and ultimately timeout. Your processes are still running, but they can't talk to disk.

In the case of Confluence, which was actually trying to talk to PostgreSQL, which itself was trying to talk to disk, Confluence detects the complete timeout hang of PostgreSQL as a cluster violation, and puts itself into crash mode.

You want to know the best part of this? When you're in this state, there's nothing you can do. You can't SSH into the VM, because it can't read the password file to let you in. The Flexiscale tools won't even let you hard bounce the machine. And they won't tell you when it's back up directly, so you just have to keep trying until you can finally get into the VM, to restart your servlet container.

This has caused no fewer than 20 instances where Confluence has died for us, sometimes lasting hours until we can actually recover the VM, and twice now in 2 days. It makes us look like morans who can't even run Confluence, much less guide development of a message encoding system.

So until we manage to get off of Flexiscale (haha, can't even get in to back up the data at the moment), if you see that error when going to the Fudge Messaging website, now you'll know why.

There are two morals of the story:

• If you get a Non Clustered Confluence : Database is being updated by another Confluence instance error, and you know that's not the case, make sure your storage is responsive;
• Don't host anything on Flexiscale until they sort their hardware out.

2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!