Cartoon Characters Discuss Web-Scale Asynchronous Communications

I couldn't resist joining in on the MongoDB/MySQL/DevNull Xtranormal meme. Many of these things I've blogged about before. But now with cartoon characters!

Twitter For Messaging: Encoding Binary In Unicode

This week at Chirp, Twitter announced Annotations, which is the Twitter-specific way of saying "you can assign arbitrary metadata to individual twitter updates." After a very quick Twitter (how self-referential) conversation with Alexis from Rabbit Technologies, we agreed that with this, Twitter is essentially trying to build a much more general-purpose pub-sub messaging technology. I wanted to talk about that.

History Repeats: Twitter Is MOM

The use of annotations is extremely familiar to anybody with a background in traditional messaging technologies. In general, publishing a message in a traditional environment requires:

• A destination. In traditional pub-sub, this is a topic name; in Twitter parlance it's the publisher's twitter handle.
• Message metadata. In traditional pub-sub, this is a set of properties (typesafe name/value pairs attached to a message); in Twitter parlance it's your annotations (full definition still forthcoming).
• Message content. In traditional pub-sub, this is in general a byte array (though specs like JMS allow for code-level specifications that ultimately resolve to a byte array); in Twitter parlance this is your 140-character tweet.

Now it looks like we've got some pretty good equivalencies; every major headline element is covered by Twitter. So how do you adapt to an All Twitter world?

I would say the starting point is the message content. I live in a world of machine-to-machine communication (people are messy). Byte arrays don't match up with character data.

Or do they?

An Historical Diversion: BinHex64

Let's consider a problem that impacted technology professionals back before most Ruby programmers were alive: how do you transmit binary data over the internet?

You had two options:

• Use a binary protocol, written from scratch or using something like RPC. This worked, but required endpoints that understood the protocol.
• Transmit data over a text encoding, like email or usenet. This allowed for the greatest amount of interim-stage compatibility, but had serious interoperability issues.

The primary interoperability problem with transmitting binary data over text protocols was a pretty simple one: most Internet protocols from the Dawn of Time were written by ignorant Americans and thus only supported 7-bit ASCII. Binary data inherently is 8-bit: you're trying to transmit a byte array, and each byte has 8 bits. How do you fit a square peg (8-bit binary) into a round hole (7-bit ASCII)?

The solution is BinHex encoding. The basic idea is that you attempt to represent a high-fidelity source dataset (8-bit binary) in a low-fidelity target encoding (7-bit ASCII) by increasing the size of the encoded message to fit the target encoding.

This seems stupid and archaic, but it's still the way binary attachments go into email.

Enter The Reverse BinHex

At first glance it might not seem that way, but Twitter represents the world in a reverse form to BinHex encoding. With BinHex encoding you're trying to fit 8-bit bytes into a 7-bit world; with Twitter tweets you're trying to fit 8-bit bytes into a "character" world. The only thing that's germane is "what is a character?"

Twitter is quite clear: A Twitter character is a Unicode code point. If it weren't so, Twitter wouldn't be able to handle localized tweets as well as it does.

Right, now we're cooking with gas.

A Unicode code point is, in the most broad brushstrokes possible, drawn from one of two distinct sets of planes:

• The Basic Multilingual Plane, consisting of code points in the numerical region from 0x0000-0xFFFF.
• The Astral Planes, currently allowing code points in the numerical region from 0x100000-0x10FFFF.

According to their character count page, Twitter uses UTF-8 in its internal representation. Let's consider two distinct possibilities:

• Twitter only handles the Basic Multilingual Plane. If that's the case, in general, one Twitter character can handle 2 8-bit bytes.
• Twitter handles the full theoretic range of Unicode including the Astral Planes. If that's the case, one Twitter character can handle 2 and a half 8-bit bytes (ignoring the Supplementary Private Use B plane, to make things simpler).

If we ignore all complicating factors, what can we thus store in a Twitter 140-character tweet if we're trying to encode machine-readable byte arrays?

• 280 bytes if Twitter only supports the Basic Multilingual Plane; or
• 350 bytes if Twitter supports the vast majority of the Astral Planes.

These don't seem like a lot, but if you allocate a few bytes (Twitter Encoded of course) for message sequence number and chaining, and you use a compact binary representation like Avro or FudgeMsg, you can get a lot of data into 280/350 bytes.

So if you use this theoretical reverse-BinHex encoding system to expand byte arrays into Twitter Messages (after full Annotation support is released), you can get arbitrary metadata for routing decisions, and a 280/350-byte binary payload. Enough clearly for a lot of uses.

Twitter As The New Machine-to-Machine Cloud Service

Don't be daft. This is entirely a thought experiment about how you could encode Real Data into a Tweet. If you attempt to hook multiple machine processes up through Twitter as a data distribution mechanism you are a moron.

If you're interested in that type of functionality, you should talk to Rabbit or another Cloud Messaging Provider (I'm sure there will be competition forthcoming). Cloud Messaging makes sense; using Twitter as your Cloud Messaging Provider is completely stupid.

Seriously. There are certainly use cases where you can see machines, people, and other machines communicating over Twitter. But if you're going to the point of converting binary data into arbitrary Unicode codepoints for transmission over Twitter, you completely fail at asynchronous communication, and should be required to spend at least 6 months doing nothing but implementing sections from EIP as penance.

2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!

Want to know what a really fast MOM system has to handle?

Take a look at what you have to be able to consume if you want to consume live market data from major exchange feeds: April 2009 Capacity Statistics.

These statistics are the aggregate peak message flow on various industry feeds of live market data. In other words, if you want to keep up with the flow, you have to be able to consume at least that many messages per second during peak periods.

The most interesting ones (to me at least) are the ones that are multi-exchange aggregated feeds for options (Siac OPRA and NYSE ArcaBook Options). OPRA hit 869,109 mps peak in April, and NYSE ArcaBook Options hit 565,522 mps peak. If you want to consume all the major feeds listed in one box, you'd need to be able to handle more than 2MM mps.[1] That's a lot of individual messages for any software-only product to handle.

Yet another reason why I believe systems like Tervela and Solace are going to be key to the next generation of market feed aggregation: at levels like that you need something hardware accelerated in order to handle the first round of aggregation. Furthermore, using AMQP (which both currently have, or at least have plans for), you can then integrate your favorite second-tier tick distribution system (RabbitMQ, qPid, 29West [2]). Ingress handling hardware feeds to egress handling software. [3] Who says you have to choose?

Footnotes

[1]: Yes, I know you wouldn't, because they peak at different times of the day, and there is overlap, particularly between the ArcaBook and Siac feeds.
[2]: I know they're a member of the working group, but I can't see anything explicit about AMQP support on their web site as of yet, so I presume it's forthcoming.
[3]: Assuming you don't just want to go hardware-only end-to-end, which is the approach that BarCap is taking for example.

Best Effort Delivery Fails At The Worst Time

In an early post in the blog, I discussed the two fundamental types of MOM infrastructure: broker-based and distributed. A particular thing I wanted to address is the best-effort delivery model inherent in distributed MOM infrastructure.

Background on Best Effort Systems

As a reminder, distributed systems operate a series of broker instances, where each one is located in the same process, VM, or OS as the code participating in the message network. For example, it might be a library that's bound into your process space (like 29West), or it might be a daemon that is hosted on the same machine (like Rendezvous). These distributed components then communicate over a distributed, unreliable communications channel (such as Ethernet Broadcast or IP Multicast). [1]

Conventionally, publishers in these systems have no knowledge of the actual consumers, and no access control is thus possible (how can you enforce AuthN or AuthZ if you're just bit-blasting onto a wire?). They simply fire-and-forget in the most literal form, as they're sending over a network protocol that has no concept of acknowledgment. So they issue bits onto the wire, and hope they get to any recipients. Management of these systems involves setting up your networking infrastructure to handle IP multicast properly, or making sure you have sufficient ethernet subnet-to-subnet bridges (rvrds if you're working with Rendezvous) to make sure that publishers and consumers can communicate.

The only guarantee you have is one of "best-effort." What this means is that publishers and consumers will do everything reasonable to ensure that messages get from publisher to consumer, without sacrificing speed or adding unnecessary latency into the network. The moment you have a slow consumer, or you have a network glitch, or a publisher goes down, that's it; messages get lost or thrown away, and you don't know about it (most of the time; some systems have an optional "you just lost messages" signal you can receive, but in the worst case your client can't tell that it's lost anything at all). That's because in these scenarios, anything that you can do to increase the application semantics will cost you in performance. If you don't need it, why pay for it?

This is very different from guaranteed-delivery semantics, where the middleware is responsible for throttling and caching such that messages aren't lost unless something really drastic happens; simply throwing away a message because an internal queue was starting to fill up isn't something you expect from those systems.

And what you get out of this is a lot of raw speed. No TCP stacks getting in the way, no sliding windows to deal with ACK messages, no central routing brokers. Fast, low-latency messages. Sure, they're unreliable, but do some interesting stuff with the protocol, and consumers who miss messages can even request the ones that they missed from the publisher (so best-effort can be pretty-darn-good-indeed effort).

This is seductive, and it's particularly seductive to developers. Setting up a simple Rendezvous (or other low-latency, best-effort delivery system) network is something you can do without your IT staff's involvement. There's no discussion of ports and broker hardware and networking topologies and involvement with your companies MOM Ops Team or anything else. Just start banging messages around, and look! It all works! And that's when things get problematic.

Best Effort = Development Win, Production Fail

Because the problem is that Best Effort systems seem much better than that in development. In development, you can run days without dropping a single message, no matter what size it is. In testing, you can run many test iterations without being able to force a message drop [2]. More importantly, you really can't force a message drop, at least not without implementing a lossy Decorator on top of the MOM interfaces. And that's a problem.

Developers easily self-justify both of these situations:

• Yeah, so 25% of that message handling code isn't covered by my functional tests. Who cares? That's just because like RuntimeException handling, it's going to be so rare that there's no point forcing the test. If it was going to be more common, the functional and integration tests would already expose it.
• Why would I spend 40% of my time working on 25% of the code that I can't get to naturally execute? That's completely retarded. I need to get back to making my Traders happy by adding more functionality to the system and sticking more data into Excel. Get back to me when the Big Bad Error Case actually happens.

And why wouldn't you accept those scenarios? If you don't, you're stealing from the company through bad prioritization.

The problem is that development infrastructure in no way represents the production user's infrastructure, particularly in a Financial Services context. And the causes of message drop tend to only happen under serious load, which usually means significant market activity, which usually means it's the absolute worst time to drop a message and have to engage in some form of retry behavior that's thoroughly untested in anger.

Consider these differences:

• In development, you're the only app running that is using the specified MOM system (because like most developers, you work on one system at a time). In production, your users probably have three different apps all stressing the underlying networking infrastructure for the same MOM system.
• In development, your networking system is probably under minimal load; aside from accessing your SCM system and streaming MP3s to your colleagues and reading tech blogs and tweeting [3], you're not really taxing it; your production users are pushing their networking hardware as hard as they can (mostly loading and saving 100MB Excel files to network stores, but that still eats bandwidth).
• In development, you typically have single use cases: either big messages or small ones, but not mixed; your production users have them mixed all around.
• Production users and systems have a lot more running on them (if traders, they'll have Bloomberg and Reuters and 8 Excel processes and 2 trading systems and your apps all on the same machine). Under peaks, everything suffers simultaneously as resources that otherwise are plentiful become quite scarce.
• In development, if your machine crashes, you have a lot more to worry about than whether that test case ran successfully; in production, machines crash and die all the time and the only thing your users care about is getting themselves up and running as fast as possible. [4]
• For intra-data-center communications, your development machines are used for coding and compiling and testing; your production machines are churning through messages 24x7 and are in a much noisier set of switches usually. That has effects that you can't evaluate, because you're not allowed to play on the production hardware.

As a result, expect at least 500% more packets to be dropped in production than in development. [5] But if you've been properly prioritizing your development, you've never tested this situation. So you're hosed.

Effective Use of Best Effort Systems

So how do you effectively use distributed/low-latency/best-effort MOM systems?

• Always make sure your application has a non-best-effort delivery mechanism. Most of the problems with best-effort systems only affect you if you think your application should only have one MOM system, and you know that parts of your application are best suited to a distributed MOM infrastructure. If you are ever using a best-effort MOM system, start with a guaranteed-delivery MOM system and downgrade certain traffic accordingly. Never start in the other direction.
• Don't ever use a best-effort system for a human-interacting environment (though the back-end processors for a human front-end system is fine). Seriously. You don't need it (humans can't detect latency below 100ms anyway), and you're just going to thrash their PC. Don't do it.
• Make sure messages are as idempotent as possible. While tick delivery systems based on deltas are extremely useful, try to make them as prone to interpretation assuming individual message loss as possible. For example:
  • Base all delta ticks on deltas from start of day, not from last tick (because you might lose the previous tick)
  • When shipping only those fields which have changed, also ship the most important other fields (so if shipping just a change to the Ask Size, also ship the basic Bid/Ask/Trade sextuplet of data)[6]
• Every time you publish or subscribe using a best-effort system, ask yourself "What happens if this message disappears completely?" If the answer is not "whatever," don't use that system and/or upgrade your semantics.
• Any time you are using these systems, have a dedicated network completely shut off from other networking uses. If you have multiple major use cases (think: equities and fixed income, or tick delivery and analytics bundles) on the same machine, use completely different hardware for those as well. Combining traffic on the same network interface is the biggest cause of message loss that I've seen. If your networking team can't/won't do this, get a new networking team.
• Have those network interfaces (one per message context) hooked up to the same physical switching fabric wherever possible to minimize the amount of potential loss in inter-switch communications.

These guidelines essentially amount to the following:

If you think you want to use a distributed MOM system, you probably don't. If you know you do, only use it for server-to-server communications, and make your physical layout match your logical layout as much as possible.

Conclusion

This essay will probably seem pretty down on best-effort systems; that's intentional. I think these systems have their place still, but for most systems their place is back in history, when switches were unnecessary latency additions and broker-based MOM systems added yet more latency and hardware-based TCPoEs didn't exist. If you're giving up 500ms of latency by putting a broker in between publisher and consumer, you'd be stupid to do so if you can avoid it. But that's not the case anymore. Modern brokers can measure their additional latency in microseconds with far better than broadcast/multicast-based best-effort delivery; if you can do that well with a broker in the middle, why sacrifice application semantics to remove it?

That's where I think the future in this space is with systems like 29 West and not with systems like Rendezvous: 29 West is adding interesting functionality that you want, and better (think: non-publisher-host-based) resilience, while still having a non-broker-based infrastructure. While I still think it's fringe, Rendezvous is just stalled as a product, and it's not going anywhere. [7]

So the next time you're sitting there coding against Rendezvous and thinking "you know, let's face it, this best-effort thing is a sham; all my messages show up!" just remember: the moment you assume that, it's going to bite you in the ass. And it'll probably happen on the last day of the quarter when volatility is up 25% and your head trader is having a down day. And then you'll wish you had sacrificed those 10us of latency for better-than-best-effort delivery.

Footnotes:

[1]: In JMS terms, imagine an embedded JMS driver which communicates without a broker, just through IP broadcast and multicast with every other JMS driver on the network.
[2]: Even worse, because many distributed tests will launch all components on one machine, you never exercise the network effects that distributed MOM systems entail. This makes sense from a development efficiency perspective, but is particularly problematic in a distributed MOM situation, because the chances of dropping a message fall to pretty much 0 in such a scenario.
[3]: Assuming you don't work for Big Bank B, where all such forms of non-authorized-communications are Strictly Prohibited.
[4]: This is a particular worry if your Best Effort to Better Than Best Effort upgrade is based on disk or other publisher-side mechanisms: what happens when the publisher's machine goes down completely? Or, what happens if it's mostly down but the fast way to resolve the situation to your ops team is to swap it out before it fully fails?
[5]: No, I'm not going to specify this more fully, because it's so prone to variation based on the runtime environment. Just trying to say that if you drop one message a day in dev, assume at least 5 per endpoint per day. Or more.
[6]: Having periodic snapshot ticks (so adding the sextuplet to the tick that happens every 5 seconds, or manually pumping a snapshot every 5 seconds under low activity) is one alternative here, but what happens if you lose the snapshot tick?
[7]: Except, bizarrely, into the realm of broker-based MOM by virtue of the Solace-powered Tibco Messaging Appliance.

AMQP: Why Do I Care?

UPDATE 2009-08-18: I've been told that I signed an AMQP Reviewer Agreement, which is a different document from full licensees. I've removed the footnote as it turns out it was completely incorrect.

Aside from the fact that this blog is called "Kirk's Rants" for a reason, I get especially strident when I start talking about AMQP. So much so that in addition to a comment on one of my recent Red Hat posts, I've been asked by an online magazine (as part of an interview process) why in the world I care so much. It's simple: I care because I've been bitten in the past.

This blog's history with AMQP goes way back. In fact, it's my #2 post label (only immediately under the more generic Messaging). Let's take a trip down memory lane, shall we?

My first ever post on the subject was a lamentation. I had been having problems with the C++ client libraries for SonicMQ (an otherwise fantastic product, I must say). I grabbed the JMS client for qPid Java, and RabbitMQ, and they just didn't work together. Surely, however, that was the whole point of AMQP as an exercise? And I've been involved ever since.

I've read drafts of specifications. I've signed a Reviewer Agreement with the AMQP Working Group (though I'm not a member of it). I've met John O'Hara and Pieter Hintjens and Alexis Richardson and Carl Trieloff and gotten about as involved as you can be and not work for an MOM producer. I've gotten involved.

But why? Because it matters.

I've worked in anger with:

• IBM MQ
• SonicMQ
• Tibco Rendezvous
• ActiveMQ
• An in-house developed MOM broker at Derivatives Group A
• An in-house developed MOM broker at Big Bank B

In addition, I've done formal evaluations on:

• FioranoMQ
• Tibco EMS

Every single one of them has bitten me in the ass before, usually around client libraries. For that reason, most applications are written against one broker technology, whether it's the appropriate one for all use cases in the application or not. Or, a whole firm's MOM infrastructure will be written against one broker technology, to limit all the problems that come in when applications have to support multiple client libraries, or have to select them for each application.

I view this as something that holds back MOM and keeps it in a Financial Services Ghetto. I don't view it as odd that the "Messaging Is Not Just For Investment Banks" presentation at qCon attracted more people than the AMQP presentation. Getting your head around what you can do with fully asynchronous fire-and-forget APIs is hard enough; getting your head around the fact that if you're not adhering perfectly to the JMS spec [2] you have to learn and work with completely different client APIs, half of which are broken at any given time is just too much. It's why so many people just jump into using XMPP: it may not have the best semantics, but at least it interoperates. This needs to end.

For MOM to reach completely widespread adoption, I need to tie my clients to my programming language and operating environment, not to what the provider of my broker technology is willing and capable of supporting. In order to make that a viable proposition, I need to consolidate to one protocol. Therefore, in order to get to a situation where enough engineering work is going into the client libraries to make them really really good, I need to have very few of them, namely 1.

Imagine if the web didn't have HTTP. Imagine if instead of having one protocol that would talk to everything I had to have separate client libraries for Apache, and Netscape Web Server, and IIS, and Zeus, and anything else that I wanted to support. Imagine, further, that Apache is 2% of the market. That's the situation we're in with messaging at the moment. To break out of that situation, we need the HTTP of messaging (only with less text and more binary, natch). That's AMQP.

Again, remember the Apache being 2% of the market scenario. In order for AMQP to be a success, it has to embraced by the 98% of the market using proprietary solutions, so that applications which rely on those solutions can be converted. Once those applications are converted to AMQP, users (not developers) can try different brokers and see if there are perhaps ones that suit their applications better. Once that happens, we'll have proper innovation in the MOM space. I want to live to see that day.

I've spent a lot of time with Alexis Richardson and the rest of the RabbitMQ team. Given that they're in London, I'm in London, I personally get along with them down t'pub, and I spent a fair amount of time hanging out while looking for a Cash Money Gig, this shouldn't be surprising. Furthermore, the fact that I originally discovered AMQP through Steve Vinoski discussing that Erlang is ideally suited to Message Oriented Middleware implementation and thus RabbitMQ was a really good use of Erlang means of course I'm going to be interested in hanging out with them. However, let me be very clear about something: I have never received a single pound from any provider of message oriented middleware solutions [3]. Since starting this blog ferreals I've received money from exactly two sources: Derivatives Company A, and Big Bank B.

In sum, I care about this stuff because proprietary MOM clients have bitten in me in the ass way too many times, and because fundamentally I hate closed protocols. I'm a practitioner in this space, who never (again) wants to:

• deal with another application that can send out messages to Broker Technology A, where I'm using (or want to use) Broker Technology B;
• be stopped from using Interesting Technology C because the Broker Technology I have to connect to doesn't support it;
• be nearly into production with a system, only to find out that the client library has a hideous bug that I can't fix because it's a proprietary protocol.

In short, I care because I'm experienced enough in the space that I've learned to care.

Every practitioner should care so much about this type of thing. If you don't care anymore about the fundamental constraints of the space in which you develop, why are you still in the game?

Footnotes:
[1]: REMOVED (see update at top of post).
[2]: Not using Java? So sorry, there's nothing equivalent in any other language, and each vendor decides how to map JMS onto C++ in a completely different way.
[3]: Okay, I let Hans Jesperson from Solace buy me a hot chocolate. Guilty!

USPTO #20090063418 - Yes, I'm Aware

This has hit Slashdot, so given my audience I have to respond. Yes, I've been aware of this for several days. In the interest of fairness, I'm waiting for Red Hat to make a comment that might be accessible to me.

I'm not willing to wait forever, and I'm not pleased about the situation. Considering I met Carl Trieloff for the first time at qCon this week, I'm going to let them respond, but my patience isn't eternal.

In summary, assume the following:

• I know of the patent application;
• I'm apoplectic with rage (like 3 on the Kirk anger scale[1]);
• I'm swallowing that anger until I hear a good explanation for the patent application.

Footnotes:
[1]: A scale invented by ex-coworkers. Most people register in milli-Kirks at best. Yes, this is "Kirk's Rants" for a reason. :-)

Let A Million Brokers Bloom

To me, one of the most promising things about AMQP is that it actually has the potential to radically expand the amount of feature-based broker differentiation in the MOM space, by eliminating a lot of the "grunt" stuff that you have to go through in order to define a new broker model.

Right now, if you want to do something interesting (meaning not commercially supported) with asynchronous message-oriented communications, you have to:

• Define your basic messaging model
• Define the wire representation of that model
• Define your protocol
• Write up the server-side handling of the protocol
• Write up the client-side handling of the protocol for each programming language you need to support

Only when you're done with all that do you actually get to do "the fun stuff," which is to write up the unique piece of functionality that meant you wanted to start on the project in the first place.[1]

Consider a particularly germane example. In my current day job I'm working for a firm (and if you're reading this blog you've heard of the firm) that has a proprietary MOM system with (what I can see as) one USP.[2] Aside from that one USP (which I'm not going to identify), it's just a simple pubsub-based MOM broker. But nobody in the market has that feature, so they can't go with a commercial vendor. And there are facets of that USP that means that no commercial vendor is likely to add exactly that feature to their product.

But in order to get that one USP feature, they have to face every single other hurdle above, and constantly deal with refining every single bit over years and years of development. None of which is about that one USP feature, it's all just plumbing. Everything other than implementing that feature is a waste of valuable development effort.

The kicker? That USP is just a custom Exchange type in AMQP 0-X terminology. Nothing more, nothing less.

And that's the beauty of AMQP.

Once we reach the promised land, someone like my current cash money provider could have just grabbed an existing framework (all the AMQP client work and server-side protocol handling in its language of preference), and implemented the custom exchange (or broken existing ones, or got freaky with queue implementations, or whatever it wanted): no need to reinvent the wheel. AMQP acts as the standardization of all the bits that you don't want or need to care about to provide custom message broking behavior, allowing you to focus all your time and attention to the parts that you really need to implement yourself.

This makes the barrier to entry for unique MOM functionality only the cost to implement the unique functionality itself; you don't have to face the hurdle of implementing everything else required for asynchronous message-oriented communications to showcase or provide one single feature.

I consider this to be the messaging version of the story behind the Eigenbase project: a framework for writing custom brokers. A few people I know (big shout out to my homies John Sichi and Julian Hyde[3]) are excellent at writing "interesting" databases (with special sauce for special contexts). However, doing that from scratch each time means that you're constantly reinventing/rewriting the same old tired stuff: a SQL parser, a transactional data store, a wire protocol, blah blah blah. So they started the Eigenbase project as a way of taking care of all that cruft that you just Have To Have so that they could focus on the stuff that's special about Their Database.

Are there equivalently architected Framework Brokers yet for AMQP? Not that I'm aware of (please pipe up in the comments if you think I'm wrong). If nothing else, in the message broker/routing space, there are a lot of things that you might want to do to speed up adhering-to-standards mode that break the general programming model you'd want from a framework-based system. That leads me to believe that just as MySQL didn't go framework-based to become Eigenbase, we'll probably see a set of AMQP brokers focusing on ease of programming custom variants, just like Eigenbase is designed specifically around programming custom variants in the database space.

And at that point, I think we'll start to see a lot of new and interesting implementations, some in-house and some exposed to the world.

Footnotes

[1] Alternatively, you can horribly abuse an open source project, but that's going to do nothing but confuse everybody involved (including yourself) and leave you liable to abandonware if you rip out the insides of a system that you're replacing key components on (so if your open source basis isn't designed as a framework, and you replace some of the key message handling parts, the moment the developers change the internal processes you're effectively stuck).

[2] Unique Selling Proposition. But you knew that of course.

[3] Broadbase In Da Hizzy!

Tervela Joins AMQP Working Group

The Press Release. No, it's not on the formal AMQP Working Group web site yet as of my writing this.

I think this is really promising, as it's the first hardware vendor to have formally signed up to join the AMQP Working Group (despite my prodding of Solace to do so). I hope they manage to produce a full hardware implementation of the spec!

It also continues the industry's relentless march to where I believe is its destiny:

• Open Source
• Cheap Appliances
• Custom Hardware
• Cloud Provision
• IBM

If you're not one of those 5, figure out how you can be, because otherwise you're in the downward spiral. If you're paying someone not one of those five, start working on your exit strategy before it's too late and you're paying ground rent until the end of time for a second-rate solution. Don't migrate yet, but have your plan in place.

Update: Here's the Official Tervela site press release.

BarCap Standardizes On Solace For MOM

(via Hans Jesperson at Solace)

Usually I don't regurgitate press releases, but I think this one is particularly interesting to my fair readers: Solace To Supply Barclays Capital with Global Standard for Messaging Middleware.

Barclays Capital will use Solace 3260 Content Routers to integrate applications spanning the front, middle and back office. By consolidating all messaging needs onto the new platform Barclays Capital will simplify the current infrastructure while saving on development, datacenter and support costs.

Congratulations to Solace Systems for such a big win! Now if they can get content routing that's based on something less stupid than XML.........

I've said it before, but fundamentally I don't believe that existing commercial broker software has much of a future, as long as a hardware platform (e.g. Tervela, Solace) prices out to roughly the same as Tier-1 hardware with per-core licensing of any of the commercial players. Hardware will win by every metric as long as you're paying the Tibco/Progress/IBM/Fiorano licensing costs. If you're BarCap, why in the world would you pay to roll out Tibco EMS on Tier-1 hardware, when for the same price per node you get something far faster in every single metric, and without the OS maintenance overhead?

The world is going to come down to five major players:

• Hardware Vendors (Solace, Tervela, et. al)
• Open Source Software (RabbitMQ, qPid, et. al)
• Low-Cost Appliances (someone offer it for sale and you'll get customers I think)
• Middleware Services (Amazon SQS and future players)
• IBM (because large banks will never ever in a million years be able to move off of MQ Series, almost exclusively because of the Mainframe problem).

If you're not one of those, figure out how you can compete, because the buckets are pretty clear to me.

I Use Product X, AMQP Is Irrelevant To Me

As I've gotten more involved with the greater AMQP community, and as I've talked more and more about it with people who already "get" messaging, I keep hearing the following comments:

I've made a significant investment in Product X [MQ Series (or whatever IBM's calling it these days), SonicMQ, Tibco EMS/Rendezvous, FioranoMQ] and I use it for as much as I can. I'm happy with it. I'm happy with vendor support, all my apps "just work." Why in the world should I care at all about this AMQP thing?

It's a legitimate sentiment. A lot of the AMQP hype has been driven by next-generation pure-play AMQP broker platforms trying to eat into those existing install bases. I still think it's relevant to you, the Happy Commercial MOM Broker Customer.

It's The Ecosystem, Stupid!
Remember back when I initially ranted about AMQP. One point I made there, quite aside from my "My commercial vendor is annoying me" ranting, was about the overall ecosystem. Here's where I make my AMQP pitch: AMQP Will Help You Leverage Your Existing Investment.

Let's assume that you're completely happy with your WebSonicFioEMQ platform and you've changed all your code to be able to use it. You're still not leveraging it to its maximum level, because no matter which one you chose (or if you chose them all), you can't always get third party products to integrate properly.

Consider one pretty Enterprise Software product: Sybase Replication Server. Last time I checked, it could pump out event messages on "interesting" events through three mechanisms: JMS, Rendezvous, and MQSeries. But I'm sure it's actually a lot more specific than that (I can't check directly as it's firewall protected, but could someone who has access fill us in in the comments), down to the version of individual JMS Brokers and versions of Rendezvous and MQSeries and all that that it supports.

Consider some more emerging products like Continuous Integration platforms. All the modern ones can send alerts using email and Jabber.

They're astute in choosing the most dominant messaging mechanisms in their target environment: commercial MOM systems in the case of Sybase Rep Server, and open communications standards in the case of Continuous Integration. And therein lies the rub:

• Unless your MOM platform is truly ubiquitous in your domain, every piece of software you want to integrate into it will result in custom effort;
• Your MOM platform isn't ubiquitous in every domain.

It doesn't matter what you're using for asynchronous communications: it's not ubiquitous enough.

AMQP Promises Ubiquity
Let's assume that the promise of AMQP really gets going. I wholeheartedly believe that the sheer engineering effort that application vendors can save through only supporting one protocol, and thus one client library package, without any interoperability concerns, will force it through.

Consider someone like Atlassian. You're an Atlassian customer, and you want your Confluence and Jira and Bamboo and FishEye instances to fire off messages to your existing MOM infrastructure when "interesting" things happen. So you lobby and say "Hey, can you support SonicMQ for message publication?" and they say "hey, that looks pretty interesting."

But the moment someone else catches wind of this, pretty soon they have a many-to-many compatibility story as every customer with a different infrastructure pops up and says "if you support X, you should also support my MOM infrastructure as well." Math is Hard, Atlassian Goes Shopping and says "write your own plugin, yo."

What if Atlassian could satisfy every single major MOM infrastructure with exactly one protocol? They'd do it. Interop testing by the broker developers (and not Atlassian) will take care of everything else.

AMQP Helps Your Commercial Investment
If you think this all seems like it makes sense (and if it doesn't, let me know precisely what about it doesn't in the comments), your immediate response should be to get on the phone to your salesperson and say:

• I want AMQP support in a (near-)future release of your MOM Broker;
• If you don't do it, I will roll out someone else in parallel and bridge them.

Trust me, this will get them thinking. Because I'll guarantee that within 6 months of AMQP 1.0 coming out, there will be open source bridges between every major commercial MOM infrastructure and AMQP. And the absolute last thing your sales person wants to hear is that you're thinking of adding something to reduce your existing lockin.

Once this is done, you get the best of both worlds:

• You can leverage the entire organizational and technical infrastructure you've already built up to do MOM properly without adding more "products" to manage; and
• You can integrate yet more applications and participate in a far richer ecosystem than Your Product could ever support on its own, helping drive that investment farther.

So get on the phone to your sales guy. Now.

2009 Predictions

Just a quick set of predictions for 2009. I'll revisit nearer the end of the year to see how I've done.

Messaging Breaks Out
Right now asynchronous systems, particularly ones using message oriented middleware, are still pretty fringe. However, even though I disagree with the technology choice, techniques like XMPP are starting to bring asynchronous communications to the masses. I predict that the extension of XMPP, the forthcoming standardization of AMQP, new infrastructure being developed (like RabbitMQ, OpenAMQ, and Qpid), and extension to new programming languages and application execution environments will all drive more and more developers to finally end polling for updates.

Cloud Will Become Less Buzzy
Right now Cloud/Utility computing is really too buzzy to make heads or tails of, and I think it's suffering from that: when you say "Cloud Computing", it means both nothing and anything. I think 2009 will be the year that people start to solidify what works best in a utility computing space versus a local hosted space, and where hosting providers get mature enough and technology becomes mainstream enough that there's an assumption of utility over local hosting.

Java Will Stagnate
I predict that Java 7, if it even hits by the end of the year, will be pretty anemic and, quite frankly, lame. There will be some interesting enhancements to the JVM, but that's pretty much all that's going to be noteworthy.

C# Will Over-Expand
The entire Microsoft ecosystem will expand and grow beyond the ability of typical Microsoft stack developers to keep up. We're already seeing it with WinForms vs. WPF, LINQ vs. ADO.NET Entity Framework, and whole new technologies like WCF. I predict by the end of the year the Microsoft ecosystem has become so bewildering that while leaders in the community push for more change, day-to-day developers push for a halt to just absorb what's been happening.

Non-Traditional VM Languages Will Break Out
Right now most developers in enterprise systems are coding against the "core" languages (C#, VB.Net, Java) against their respective VMs. This reflects in no small part the maturity and power of their underlying runtimes (the CLR and JVM respectively), but the leading edge of the communities have already started to explore other languages that offer concrete advantages (F#, Scala, Groovy) on the tight runtimes that are already available. This will be the year that those actually break out of the leading edge and into mainstream use.

No-One In Social Networking Makes Money
I still think, by the end of 2009, there won't be a player in the social networking space that is cashflow positive, and I don't think it'll be by choice. I think the technology is so disruptive that we still haven't seen the "right way" to monetize it yet.

Sun Radically Restructures
Sun can't keep burning through cash the way they have been, and they can't continue to have such a chaotic story. At some point in 2009, Sun will relatively radically restructure itself in a bid for survival. Hopefully they'll have read my analysis (part 1, part 2). At the very least, they'll change their ticker away from JAVA.

I think you'll note that all my predictions are fuzzy. That's so that in December, 2009 when we revisit them, we can determine that I was partially right on nearly all of them.

Scoble Joins The Real-Time Web Conversation

Well, maybe "joins" isn't the proper word, but this showed up on my FriendFeed account today: RSS Shows Its Age. As I've posted about already, I think polling for updates is pretty lame, and asynchronous messaging is the right approach. But yet again, XMPP isn't it.

Enter the next contender: SUP. Took me a while to figure out what precisely it is, but it's essentially a JSON-encoded consolidated feed URL for a batch of URLs. So if you are a service that has a million individual RSS URLs, you provide one SUP URL, which allows clients to make a request and see all URLs that have been updated "recently". The magic sauce is:

• Instead of individual RSS URLs, they get encoded into a nonce (an SUP-ID) to save on transfer costs;
  
• JSON rather than XML;
• Definition of "recently" is up to the publisher so that it can be statically served and cached and changed depending on publisher server load.
  

The end results are you can poll the SUP feed for a whole site, and then find out what you have to hit the individual RSS for. So this makes it faster to poll the million RSS URLs, particularly since at any given time a tiny fraction are actively updating. And it makes FriendFeed feel better about polling Flickr every second to reduce latency.

But it's still polling! It still ultimately doesn't scale. You can make polling suck less, but it will always suck.

Again, asynchronous message oriented middleware systems are the ultimate solution to this, and anything you do in the interim is only postponing the inevitable.

REST Requires Asynchronous Notification

I recently (well, compared to people like Steve Vinoski) got converted to the RESTful architecture movement, but one of the first things that I recognized is that it suffers from the same generic limitation of every HTTP-based component architecture that I've seen: there's no defined way to get updates on content.

In fact, when I started getting more immersed in the REST culture, I realized that a lot of the nuances that are ideal to master in order to provide a great RESTful interface (caching, timeouts, advanced status codes) are there specifically to limit the number of repolls that are necessary against the terminal service by improving the ability of caching proxies upstream to handle requests.

But no matter what, there are clearly latency issues involved in any polling system, particularly one which is using fixed content caching timeouts to optimize performance (because this puts the latency entirely under the control of the publishing system which generated the result of the initial GET command). How do we minimize those?

Pull RSS Is No Substitute
RSS/Atom (note that I'm going to use RSS as a shorthand for the whole RSS/RDF/Atom spectrum of pull-based syndication lists) seems like a reasonable way to minimize the actual content that needs to be polled by batching updates: instead of having clients constantly polling every URL of interest, you batch them up into groups (perhaps one for each resource domain) and provide a single feed on each group. The RSS feed then indicates the state of all the resources in its domain that might have changed in the last period of interest. So now we've reduced hundreds of GETs down to one, but we're still polling.

My old colleague Julian Hyde posted some stuff recently which was precisely along the lines of what we're talking about here (post #1, post#2), in that he's building a live querying system against streams, and finding that the polling nature of RSS is particularly noxious for the type of system they're building. I'm surprised it took him so long. RSS is actually a pretty terrible solution to this problem: it combines polling with extra data (because an RSS feed is generic, it publishes more data than any particular consumer needs) in an inefficient form and still allows for the RSS publisher to put a caching timeout. Wow. Talk about a latency effect!

It gets worse. Let's assume that you're trying to do correlations across feeds. Depending on the cache timeout settings of each individual feed, you might have a latency of 1sec on one feed and 15min on another feed. How in the world do you try to do anywhere near real-time correlations on those two feeds? Answer: you can't. At best you can say "hey, dear user, 15 minutes ago something interesting happened." 15 minutes is a pretty short cache timeout for an RSS feed to be honest, but an eternity to many applications.

The only real advantage that RSS/Atom provides is that because it's based on HTTP, and we have all these skills built up as a community to scale HTTP-based queries (temporary redirects, CDNs, caching reverse proxies), we can scale out millions of RSS queries pretty well. Problem? All those techniques increase latency.

Enter AMQP
Anyone with any experience with asynchronous message oriented middleware can see that fundamentally you have a one-to-many pub/sub problem here, and that smells like MOM to me. More specifically, you have a problem where ideally you want a single publisher to be able to target a massive number of subscribers, where each subscriber gets only the updates that they want without any polling. Wow. That looks like exactly what MOM was designed for: you have a broker which collects new messages, maintains subscription lists to indicate which consumers should get which messages, and then routes messages to their consumers.

It's so clear that some type of messaging interface is the solution here that people are starting to graft XMPP onto the problem, and using that to publish updates to HTTP-provided data. Which would be nice, except that XMPP is a terrible protocol for asynchronous, low-latency MOM infrastructure. (If nothing else, the XML side really will hurt you here: GZipping will shrink XML quite a bit, at a computational and latency cost; a simple "URL changed at timestamp X" is going to be smaller in binary form than GZipped XML and pack/unpack much faster).

But we've already got another solution brewing here: AMQP, which simultaneously provides:

• Binary protocol (for performance)
• Open protocol (for compatibility)
• Open Source implementations
• Built by MOM experts

Looks like a winner. Just about the only thing that I can think the REST crowd not liking is that it's not over HTTP (The One True Protocol), but every reasonable MOM system has support for HTTP-based subscriptions (at a pretty serious performance penalty).

Combining REST with AMQP
At my previous employer, I actually built this exact system. In short, resources also had domain-specific MOM topics to which the REST HTTP system would publish an asynchronous message when any update to a resource happened. Clients would hit the HTTP endpoint to initialize their state, and immediately setup a subscription to updates to the separate MOM system.

Without the MOM component, it's just pure REST. Once you add it, you eliminate any resource or RSS polling at all. The MOM brokers are responsible for making sure that clients get updates that they care about, and there's only the latency that the broker introduces to updates. Worked brilliantly: all the benefits of a RESTful architecture, with none of the update latency effects.

We had it easy though: this was all purely internal. We didn't have to operate at Internet Scale, and we didn't have to interoperate with anybody else (no AMQP for us, this was all SonicMQ as the MOM broker).

In order to get this to work as a general principle, we as a community need to do the following:

1. Finish AMQP 1.0 (I've been told the specs are nearly ready and then just comes the implementation and compatibility jams);
2. Decide on a standard pattern for REST updates (topic naming, content encoding, AMQP broker identification in HTTP response headers);
3. Develop easy to use publisher and subscriber libraries (so that services and client packages can easily include this functionality across all the major modern languages);
   
4. Develop AMQP brokers and services that can operate at Internet Scale.

Once we've done this, people like Julian can hook up SQLStream directly up to AMQP endpoints and consume all the data they want without invoking the horror that is polling.

Real Time Linux Kernel Smacktown Comments

I read El Reg's writeup of Red Hat and Novell attempting to out-gun each other on their RT-esque kernel patch versions of their main Linux distros. Just a few quick comments.

First of all, they're testing against RMDS 6 (something I have some experience with). When you see anything about an RMDS 6 P2PS, what you're really looking at is an ancient branch of RVD running the TCP form of the Rendezvous protocol. That means that a P2PS esssentially acts as a middleware broker in a bizarre mixture of a protocol designed for unreliable broadcast/multicast usage (Rendezvous) but running on a single machine. It's a quite strange beast to be fair. Just a bit of background detail for you there.

Secondly, where things get really strange in this is when Infiniband comes in on the Novell SLERT side. Uhm, huh? They're combining RMDS with IB? In what form? Are they doing IP/IB? Uhm, why? I'm sorry, but if you're going to roll out IB, without using something IB specific like OpenFabrics that allows you to leverage all that nice IB RDMA action, there's something seriously wrong. Yes, you might shave off a few microseconds off your latency compared to IP/Ethernet, but you'd get much better by coding raw against IB. Based on that, I'm actually surprised that Novell included this at all, since I really doubt that anybody is going to attempt to improve the performance of their RMDS infrastructure by running it over IB.

So all that being said, this type of benchmark is really just a proxy for something testing how fast the kernel patches can get networking packets off the wire and into user-space for processing. That's a pretty interesting point and good to know, but if you really care about that and don't have to run RMDS for everything in the first place, why not use something like Solace which doesn't have user-space in the first place? Why not just code against IB using OpenFabrics? This is far more interesting to me in the client side (where you're going to have user-space code ultimately doing your processing) than on the broker (e.g. P2PS) side.

So that being said, I think a very interesting benchmark would be:

• Similar workload in general (the STAC workloads actually are representative of the general workload you get in financial market data processing);
• Hardware distribution to the clients (Tervela, Solace, or OpenFabrics/IB);
• Clients different between the RT-patch version of the kernel and stock kernel.

The client is where I think the action needs to be here, not the broker.

Meeting With Solace Systems: Hardware MOM

After my post on wanting an AMQP appliance, Hans Jespersen, Principal Systems Engineer with Solace Systems, got in touch with me and we met up yesterday when we were both in San Francisco, and I had a chance to talk tech with him on what precisely they do (as opposed to the marketing speak on their website). Hans is a pretty smart guy (ex-Tibco on the Rendezvous side rather than the EMS side), and seemed really technical and up on some of the other fringe technical topics we talked about.

Company Background And Overview
This is an Ottawa-based company, which is interestingly drawing from a group of ex-Telco engineers for the hardware side and a bunch of ex-MOM people (particularly Tibco, which gave them a big OEM win I'll talk about below). That's interesting to me because the two worlds haven't really interacted that much that I've seen, but there's a lot in common and a lot that I think both sides could probably learn from each other. I didn't ask about revenue or staff or anything else, I just had time to cover the technology itself.

The real goal of the company is to focus on low-latency messaging, which means that they do everything with a no-general-purpose-OS passthrough. As you'll see below, messages are delivered without ever hitting a general purpose OS or CPU, meaning that there's no chance to introduce random amounts of latency. But the same hardware approach allows them to also hit the high-volume and persistent messaging cases.

Base Technology Platform
The base platform is a pair of chassis, each running a customized motherboard, which has a number of PCIe slots in it (5 slots in the 2u 3230 and 10 slots in the 4u 3260). The motherboard itself has an embedded Linux system (with 8 cores actually), but at this point is only being used for management tasks and doesn't get involved in the message delivery path (though this wasn't true in the past, which is why there's such a beefy box to handle things like SNMP and web-based administration). By itself, the chassis is useless.

Each of the "blades" (which I'm going to call cards, as they're really just PCIe cards) provides a particular function to the box, but they're very strangely named on their website. These guys, rather than running general purpose chips, are all FPGA based. Here's what they do.

Network Acceleration Blade
This is the strangest name ever for what this thing does, because it's actually the primary message delivery platform for the box, as well as the only card that has any NICs (4 or 8 1GbE links) that can be used for messaging (the NIC on the chassis is just for management). This consists of:

• The network ports themselves
• A TCPoE on board
• Hardware (I'm not clear whether this is ASIC or FPGA based; everything else is FPGA, but these you can get off-the-shelf ASICs for, so I'm not sure what they did here) to do SSL and GZIP
• A multi-gigabyte RAM cache to store messages before they're delivered on
• An FPGA to do the protocol management
  

Interestingly, this is useless in and of itself, because the network card specializes in receiving messages, unpacking portions of the message to send over the bus to the other cards, and delivering them out. To actually route messages, you'll need one of the next one.

Topic Routing Blade
This is another FPGA-based card, which manages subscription lists of clients and has FPGA-accelerated regex-based subscription management. The way this works is that the Network Acceleration Blade extracts the topic name for a message from the message, and sends that over the internal bus to the Topic Acceleration Blade, which then responds to the NAB with the list of subscription channels (TCP sockets) to which the message should be forwarded (the message contents never travel over the internal bus in this case). Handles both RV-style wildcarding (.>) and JMS-style wildcarding (.#).

This isn't the only option for routing, which I'll get to separately, but assume that it's the only useful one for the time being.

These two cards are the only things that you need in a chassis to support basic transient store-and-forward MOM functionality. For persistence, you need the next one.

Assured Delivery Blade
This is a card which acts to ensure guaranteed delivery of messages, both in the persistent-sending case (don't acknowledge to publisher until the message can survive broker failure) and the disconnected-client case (if I'm down, store messages until I get back up). It's quite an interesting beast actually. This one has its own buffer of RAM, and two FC HBAs, and you would deploy this in a HA pair of chassis with a crossover between the two assured delivery cards. It's probably easiest to describe how it functions and you'll get the picture of what's in there:

• Message contents get delivered from the NAB to the Assured Delivery Blade over the chassis PCIe bus (yes, in this case the whole message body has to be transferred).
• The ADB sticks the contents into its internal RAM buffer
• The ADB sends the contents over the interconnect to the passive ADB in the other chassis
• The ADB acknowledges the message back to the NAB, which then is able to acknowledge back to the client that the message is persisted (more on why this works in a second)
• The ADB then batches writes (if necessary) to the underlying disk subsystem (they've only tested with EMC, he wasn't sure how high-end you had to go up the EMC product line) to clear the RAM buffer in a more optimized form.

I was a little sceptical about acknowledging the message just after the passive node acknowledges it (this in SonicMQ terms is DeliveryMode.NON_PERSISTENT_REPLICATED), but here's where having a pure hardware platform helps here. Each card has a big capacitor and a Compact Flash slot, and on failure, the capacitor has sufficient charge to flush the entire contents of the RAM buffer to the CF card, guaranteeing persistence.

It's a pretty clever model, and pretty indicative of what you can do with a dedicated hardware platform.

Marketing-Driven Blades
This is where things got a bit more sketchy. Note that in the above stack there's no way to do header-based message delivery (JMS Message Selectors). That's a pretty important thing if they're going to try to hit existing JMS customers who will have message selector-based systems. So they worked out a way to do that.

The problem here is that they added it with their Content Routing Blade, which is entirely XML based (the message selector language is XPath-based, rather than SQL-92-based). This is where he lost me and I told him that. While I'm sure this is great from a marketing perspective because it lets them sell into the Service Oriented Architecture Solution space, I think that space is rubbish, and the types of people who buy into it are not the types of people who are going to evaluate bespoke hardware from a minor vendor to increase message throughput. It also doesn't help porting existing applications, which I hit on in my initial AMQP analysis is one of the most important things you can do to try to drive early adoption of new technology.

They also have an XSLT-in-Hardware card (the Content Transformation Blade), but I'm so uninterested in hardware XSLT that I didn't bother talking to him about it.

Protocol Considerations
Given my writing about this, I did hit him pretty hard on the fact that right now they only have a proprietary protocol with binary-only drivers (C, C#, Java). The fact that it's a C and not C++ client library makes it much easier to port because you get to avoid the name mangling issues that are the worst in intermingling native compiled code, but there's nothing else you can use to talk to the box. They have a JMS driver you can use, but given that they don't support all of even the topic-half of JMS (like message selectors), I'm not sure of how much utility that is at the moment.

That being said, the fact that they've rolled with FPGAs and not ASICs means that they can flash the NAB to support more protocols (AMQP was specifically mentioned here). In fact, they've already done this by providing bespoke versions of the NAB and Topic Routing Blade to support the Rendezvous protocol natively under the Tibco Messaging Appliance brand name to Tibco. In that case, (if you're familiar with Tibrv), rather than using multicast or broadcast RV, you make a remote daemon (remote RVD) connection over TCP to the Solace box, which speaks remote RVD natively. Pretty cool, and Hans is pretty sure they're going to support AMQP once it standardizes.

Analysis
The things I particularly like here:

• Clever use of hardware. The use of FPGAs rather than ASICs is pretty good going, as is the custom motherboard with lots of fast PCIe interconnects. I also like that they've taken a really close look at the persistent messaging case, and leveraged the fact that they're in control of their hardware to ensure that they can optimize far more than a pure software solution could.
• Pure hardware data path. I like the use of external controller cards (the two routing cards) to minimize message flow over the bus, and that there's no general purpose CPU or OS touching the messages as they flow through the system.
• Speed. If his numbers are credible (and I have no reason to think they wouldn't be), they're hitting over 5MM 100b messages/second/NAB using topic routing, and 10MM subscription processes/second/card on the topic routing cards themselves. Persistence is over 100k persistent messages/second. That's pretty good and the types of numbers I'd expect from a hardware solution.

Things I don't like:

• The Content Routing/Transformation stuff. Pure marketing claptrap, and utterly silly. If you know you need a low-latency hardware MOM system, are you actually going to take your nice binary messages with a custom protocol and add the Slow of XML for routing? I can see throwing compressed XML around as your body content, but routing based on it? Taking binary fields and converting them to text just so that you can have XPath-based message selectors? That doesn't seem right to me. I'd be much happier if they gave a key-(binary) value map-based system, which would map much more naturally onto AMQP, JMS, and Tibrv models that we're currently coding against. That makes it hard to port existing systems, which makes it hard to get adoption.
• Proprietary Protocol. Yes, I hate these with a passion as they keep biting me in the ass every time I have to use one. You're a hardware company. If you actually think that giving me source-code access to your C client library is going to expose so much of your mojo that someone can hire up a bunch of FPGA guys and replicate your hardware that quickly, then someone will just reverse engineer it anyway. Lame decision.

Conclusion
Am I going to rush to buy some? No. I actually don't live in the ultra-low-latency space in my day job, and the technical decisions they've made would make porting my existing applications sufficiently painful that I'm not willing to go down that path for an eval when my current solution works.

Should you? I'm not 100% sold. The core direction of the product looks pretty darn good to be honest. Without seeing their proprietary client library, I can't tell you how easy it would be to port an existing JMS/Tibrv/AMQP application to their programming model for an evaluation. If I knew I was going to be working in super-low-latency space and I didn't mind an extended POC, I'd definitely add it into my evaluation stack.

But if I had an existing Tibrv application that was killing me in production (which most eventually will), I'd definitely take a look at the Tibco Messaging Appliance as fast as I possibly could. Anything that will kill off multicast and broadcast-based RV is ace in my books.

Sell Me An AMQP Appliance

I've been doing some thinking recently on the bevy of AMQP broker software which is being created, and I think it's a really positive thing that we're definitely seeing a new swathe of innovation in the broker-based middle ware space. In particular, the space seemed to many to be a fossilized space just a year or two ago, but now with AMQP and XMPP PubSub, is getting a lot of new development momentum. But ultimately, I'm starting to think that perhaps much of the innovation will be wasted on a large category of users.

I believe that we're going to see the broker-based MOM space divided into three categories:

• Edge Distribution. Imagine that you have 10 processes on the same machine that are all receiving the same data. The most optimal way to distribute the data is for that machine to be running its own broker and connecting with the hub broker to distribute each message once for redelivery to the 10 processes on the edge system.
• Embedded. In this case, an application is being written against the MOM protocols in question, but everything's happening inside the application's configuration, so that there are no external dependencies. Atlassian's Bamboo is a good example of this: they embed ActiveMQ so that their customers don't have to have an existing JMS infrastructure in place, and you don't even necessarily know you're running JMS at all.
• Central Distribution. This is every other case, and generally means a node that is configured just to act as a broker, and isn't really doing other stuff.

The thing I've come to realize is that for the third case, Central Distribution or Dedicated Broker, I don't want to buy or run "software" at all. I want to buy an appliance.

Why An Appliance?
I want to buy an appliance because I don't want to have to deal with the complexity of optimizing my installation (and by that I mean the entire hardware/OS/software stack) for the purposes of running a MOM broker. Let's start anecdotizing.

When my company was planning on rolling out SonicMQ across our entire infrastructure, we spent a lot of time working on our proposed infrastructure. This included, but is not limited to:

• Base Hardware Configuration. What processor? How fast? How many?
  
• Failover Network Connections. For our High Availability Pairs, what network connection(s) did we want to use for the replication traffic? How precisely were they to be configured?
• The OS and Patch Level. This got particularly hairy with some of the hardware options for the failover network connections, and we had to do a lot of patch wrangling to test out various hardware options.
• Filesystem Layouts. For the various types of filesystem storage (WAL for persistent message storage, message database for persistent topic and queue storage), how did various options of disk partitioning, disk speed, filesystem layout and options affect performance?
• RAM. How much? What GC settings for the JVM? Swap configuration?
  
• SonicMQ Configuration. How did any of the above affect SonicMQ? Were there any things we had to tweak to get it to perform better?

How much of that do I think was really part of our jobs? None. None at all. I think it was completely wasted time. And that time costs money, in that I was working with our systems staff on testing and configuration and we don't work for cookies (cookies are merely a bonus that endear us to vendors).

If Progress had sold a 1U SonicMQ Appliance, which had 4 external NICs and a pair of HA connectors, we would have bought it. Even at a premium over the software+hardware, because that premium couldn't have cost more than our time did.

Things Are Getting Worse
Now, things are starting to move really quickly on a lot of fronts that would affect an MOM appliance:

• Storage is changing rapidly. An ideal platform here would have some flash in it somewhere. But where? Fusion-IO-style PCIe card? Sun-style ZFS log acceleration? I have no idea, and I don't want to have to know precisely how to make your software go faster with Flash added. That's your job, Mr. Vendor.
• Networks are also changing rapidly. What to use for an interconnect? 10 GbE? Infiniband? Something else? You figure it out for me.
• Chips are opening up. Do you work best with Xeon? Opteron? Niagara? Does it matter? How fast do they have to be to saturate all network connections? I don't know. That's your job.
• Can you go faster with TCPoE? Or does it clog you down? What about custom FPGAs for routing tables? Any way to leverage GPGPUs? Again, you're the vendor, figure out the most cost effective way for me to go as fast as possible.
• Can you move into the kernel? If so, what parts? Remember, on a general purpose system my systems team will never allow a vendor to muck with the kernel. Once you're selling an "appliance", you can do what you want.
  

Just to pick on one of these that caused me no end of woe, even today, the choice of 10GbE HBAs matters: different cards have different bandwidth potentials per-socket vs. per-NIC, so this type of stuff really does start to matter more than it did before, and I don't want to have to test it all out for you. Which one do I use for an HA interconnect? What about for my connections to the outside world? I don't want to have to make that decision. I want you to do it for me based on what's optimal for your broker software.

Why Hasn't This Happened Yet?
I think the primary reason why this hasn't happened for things like MOM providers as of yet is that we, customers, just plain don't trust vendors. At all.

We don't trust them because:

• They don't cycle hardware quickly. Our SonicMQ boxes are running on dual dual-core Opterons. It's monstrously overkill, and they're mostly idle. But generically most customers don't trust vendors, particularly of second-tier appliances, to keep up with industry trends, and that costs you in terms of overall performance. They have to buy in bulk to get the discounts from the chip vendors, and that means that they're going to push their stock as long as they have to.
• They monstrously overcharge. Anyone who buys storage (and a perisistent MOM system is a storage beast as much as a network beast) knows this. Buy a disk drive for $X. Put it in a slide which costs $Y. Sell it for $Z where Z ~= 5(X+Y).
  Look, we get it, you need to charge a premium. But don't sell me ordinary equipment and charge a monstrous premium over what I can get myself from my reseller, or by going directly to the OEM. Clue Phone's Ringing: We're Not Idiots.
  You can charge a premium where it's warranted. But being an "appliance" vendor is not an excuse to jack prices up to a ridiculous level just because you can. You do that, and we'll start asking for generic software and configure it ourselves again.
  
• They're Another Hardware Vendor. Each hardware vendor we have means another stupid type of disk slide and another "certified" RAM module which is the same as your standard PC RAM. It's all extra stock we have to hold, it's all extra work our purchasing department has to do, it's all extra validation we have to perform, it's all extra overhead.
• They Don't Manage Well. Each appliance vendor seems to think that Their Approach is the Right One. So instead of saying "we're going to support getting machine statistics for Linux boxes" and just deploying that, we customers end up having a myriad of different SNMP MIBs and other little tweaks that means that for each new appliance, we have to change a whole lot of our management infrastructure. Blech.
  (Anecdote Time! NetApp provide SNMP MIBs on their network adapters. But for total bits, they don't provide a 64-bit counter. Meaning that if you have a 1Gbps network connection, it'll roll over in like 7 minutes. So if you're not running your stat gathering MRTG/Whatever every 5 minutes, you'll have to factor in rolls of the 32-bit counter. I will assume that you, as an "appliance" vendor, will have similar levels of Stupid).
  

Solution Study: LeftHand
LeftHand Networks had an extremely interesting approach to this, which is that they had software (SAN/iQ) that really benefitted from close hardware integration, but they partnered with Tier-1 system vendors (Dell, HP) to certify the LeftHand software on the system hardware. Brilliant option. They're now part of HP, but the idea was really sound. But take it a step further, and do what a lot of companies are now doing.

OEM the platform of a Tier-1 system vendor (Dell, HP, IBM, Sun). Sell us that platform as is. Don't bloody pretend it's something magical (for example, don't even bloody think of not allowing me to swap out one of your Special Magic Drives for a stock Dell/HP/IBM/Sun drive). Just sell it as an appliance with your software.

But even more than that, be quite open that what you're selling is:

• Tier-1 Hardware, running
• A customized Linux/OpenSolaris/*BSD operating system, with
• Your MOM software, with
• Perfect configuration, and
• Special Sauce (optional).
  

Don't stop me from going in an poking around in a read-only way at what you're doing, make sure that I can replace your hardware with stuff from the Tier-1 vendor itself, and make sure standard Linux software (like the standard SNMP MIB and SSH and top) are all available to me in some minimal way.

This covers all the hardware vendor objections (even if you just choose one Tier-One vendor, they all have programs for just this, and we all have experience with all of them), and still allows you to optimize for the OS and hardware like mad.

The Open Source Problem
But what if you want to run things all open-source on your own hardware? Then have the project, rather than distributing an RPM, distribute a full bootable image that you load. Or have your favorite Linux/OpenSolaris distribution come up with an installer that rather than building a general purpose system, builds a system dedicated to just running that particular application (e.g. Anaconda, but custom RPMs and boot images).

I think this is probably going to start being a more common paradigm going forward, and the move to cloud computing is probably going to take things farther as people use something like CohesiveFT or some other mechanism to grab VM instances or instances for their favorite cloud computing provider for infrastructure software.

Why I don't think you're going to want a Cloud Computing solution to this in particular (e.g. an EC2 cluster) is that something like MOM done right in Central Distribution mode is pretty network topology sensitive: it doesn't make sense to virtualize it or run it as a VMWare image. It wants to be close to the hardware. Run it in another type of workload, but don't try to push tens of thousands of persistent messages through it per second; it's never going to work. But the same general approach has a lot of merit for other types of projects.

Conclusion
I think there's a big option for someone to provide an AMQP appliance for the centralized distribution case and push it heavily. There's no reason why I should be running what essentially amounts to a persistent network appliance on generic off-the-shelf hardware and operating systems, and there's no reason at all why I should have to do your performance optimization yourself.

But again, AMQP is the secret ingredient: I'll trust an appliance running custom software to provide a standard network service. I wouldn't like an appliance running custom software for a custom protocol.

POSIX Message Queues: Useful, but Limited

Since my previous post on looking for VMS-style Mailbox semantics for more modern operating systems, I've done some research into POSIX Message Queues based on a comment in the entry. I think they're pretty darn cool, but it seems like they might be targetting a different audience than what I think I was looking for.

It also appears that the reason why I've not really seen much (any?) actual code using POSIX or SysV message queues is that support at least for POSIX Message Queues is relatively recent in the Linux kernel timeframe (2.6.6), and thus hasn't really been around for long enough as a system-primitive to find itself worked into much other software.

Disclaimer: It's entirely possible that at this point I'm largely inventing what I'd love from an internal queuing system, and it's quite probable that VMS Mailboxes don't do any of this. From here on, assume this is Kirk's Dream Land rather than "VMS Rulez".

Message Size
The default Linux implementation has the maximum message size set to 8192 bytes. That's not a lot of data to be honest, although it would be more than enough for an bridge between Erlang processes for most data. Unfortunately, this is a kernel option, and so to increase it you need to muck with your runtime kernel parameters (/proc/sys/fs/mqueue/msgsize_max).

Maximum Queue Size
The maximum number of messages you can put into the queue before it starts to block the sender is 10 messages. Again, kernel configurable (/proc/sys/fs/mqueue/msg_max), but 10 messages is pretty darn small.

I think these two combine to give a pretty clear indication of the type of message semantics they're going for: very fast, small messages. I could see a pretty clear analogue to most tick data in a financial services environment, or audio packets, or Erlang tuples. But this is really clearly designed for super-latency-critical scenarios, and the 10 message maximum is surprisingly small.

In doing some research into this, I came across this comment on a KernelTrap post, which I think gives some pretty clear insight into what's going on here, which is that this is designed for ultra-fast communications between processes (trying to make sure that everything is staying inside the same core and L1 cache if at all possible I would presume).

Persistence
This all makes sense once you realize that POSIX message queues aren't persistent at all: they're like a Priority-queue channel between currently running processes on the same machine, they don't appear to survive machine restarts in any way. This is great as a low-level IPC mechanism, but doesn't really seem to apply to more failsafe long-running messaging scenarios.

When the kernel bounces, all messages and queues disappear.

Initial Reaction
After all this, it appears to me that POSIX Message Queues are really intended to be used for IPC at a very low level of interaction within a particular system, rather than for longer processes with more data. I'd love to see them used for an interaction channel through which a disk-based queue system could be built though.

For that reason, I probably wouldn't attempt to bridge them using AMQP: it would break the semantic model of super-low-latency messaging far too much to try to bridge them over anything slower/higher-latency than Infiniband (though that would be an excellent non-MPI way to use IB, particularly if you have RDMA for the message contents). A persistent queue implementation using disk leveraging POSIX Message Queues for control? That would probably be a much better candidate for AMQP bridging to me.

SonicMQ: Constructing New MessagePublishers or not?

Just as a quick follow-on to my previous mini-benchmark, I've now tried the same basic benchmark but testing publishing each message to its own Topic, and tested creating a new MessagePublisher per message, versus creating a unbound (Session.createProducer(null)) MessageProducer and using that to send all messages (producer.send(destination, message)).

For the lazy and link-click-shy, this is a tiny benchmark of how to publish small messages as fast as possible against SonicMQ 7.5.1 using the 7.5 JMS driver.

There is a difference, but a pretty small one.

• Creating a single MessageProducer and using producer.send(destination, message) gave me 32352mps sent.
• Creating a new MessageProducer for each new message/topic gave me 31847mps sent.
• This was definitely reproducible, and I ran each test many times to make sure I got a pretty consistent result.
  

So there's a minor performance enhancement (1.5%), but it's pretty small, and judging from network traffic, it's not actually hitting the wire for the new MessageProducer, so it's entirely a local driver optimization.