2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!

Want to know what a really fast MOM system has to handle?

Take a look at what you have to be able to consume if you want to consume live market data from major exchange feeds: April 2009 Capacity Statistics.

These statistics are the aggregate peak message flow on various industry feeds of live market data. In other words, if you want to keep up with the flow, you have to be able to consume at least that many messages per second during peak periods.

The most interesting ones (to me at least) are the ones that are multi-exchange aggregated feeds for options (Siac OPRA and NYSE ArcaBook Options). OPRA hit 869,109 mps peak in April, and NYSE ArcaBook Options hit 565,522 mps peak. If you want to consume all the major feeds listed in one box, you'd need to be able to handle more than 2MM mps.[1] That's a lot of individual messages for any software-only product to handle.

Yet another reason why I believe systems like Tervela and Solace are going to be key to the next generation of market feed aggregation: at levels like that you need something hardware accelerated in order to handle the first round of aggregation. Furthermore, using AMQP (which both currently have, or at least have plans for), you can then integrate your favorite second-tier tick distribution system (RabbitMQ, qPid, 29West [2]). Ingress handling hardware feeds to egress handling software. [3] Who says you have to choose?

Footnotes

[1]: Yes, I know you wouldn't, because they peak at different times of the day, and there is overlap, particularly between the ArcaBook and Siac feeds.
[2]: I know they're a member of the working group, but I can't see anything explicit about AMQP support on their web site as of yet, so I presume it's forthcoming.
[3]: Assuming you don't just want to go hardware-only end-to-end, which is the approach that BarCap is taking for example.

Rise Of The Appliances

I've posted before [1] on my desire for someone to sell me an AMQP Appliance, and while nobody has yet (probably the fact that 1.0 is in the final stages of getting finalized is impacting this), I'm encouraged that somebody will.

Memcached Appliances Ahoy!

I'm particularly encouraged by the last week's frenzy of Appliance Startup Announcements:

• Schooner Information Technologies coming out of stealth with MySQL and Memcached appliances.
• Gear6 launching their own Memcached appliances.
• Virident launching their MySQL and Memcached appliances.

I've read more about the Schooner appliances than the other two [2], but they appear to be roughly what I said my ideal AMQP appliance would be: Tier-1 hardware, linux kernel, custom software, heavily tuned to the particular bits of hardware in the box itself. I'm a little surprised at the $45k price tag, but I suppose it's good to hit the high-end of the market first and work your way down over time. $45k would be justified with the amounts of RAM and SSDs in the Schooner box, but I'm sure they can scale things down to less Flash and less RAM and still have a reasonable margin.

Open Protocols Make It Possible

What makes this possible (particularly that everybody's targetting the currently very-much-in-vogue Memcached) is that there is a standard protocol. While the memcached protocol isn't a de jure standard, it is a de facto standard, and one that at this point is unlikely to change given the amount of software doing interesting things with it as a protocol.

It's only a matter of time before we see the same types of innovation in messaging or anything else for that matter if you can define a stable network interface to it. If you can define an open protocol (whether de facto or de jure), you can build an appliance to serve it. And it would probably be a better quality of service than rolling your own on top of the same hardware and OS combination. Systems which are heavily I/O bound (messaging, databases, network caches) benefit greatly from advanced tuning of the entire stack from hardware through to application code. Thus, they make sense to optimize holistically, whether you're using general purpose hardware or specialized hardware. Appliance vendors can do this. Generic software vendors have to rely on teaching their customers how to do it.

Cloud Hardware Combination

The one problem with this move going forward is that you can't do it with Cloud Computing, because you don't have access to the physical environment and can't put your own hardware in place. And there's a lot of stuff you can't do as a result:

• Appliances like Schooner, much less ultra-high performance messaging brokers like Tervela or Solace
• Direct network connections to the proprietary or specific network of your choice.
• Storage optimization through the use of any type of Flash acceleration
• Anything requiring significant local bandwidth on a single box

That leads me to question whether the current vogue of completely ignoring any type of CapEx whatsoever is actually optimal for your core infrastructure. For example, if you can spend $5k on a machine to do your MOM load perfectly, or your memcached big and fast, or whatever, how much extra are you going to have to spend in overall VM slice hours to equal that performance (for example, are you going to have to run 10 VM slices running memcached and an extra 5 app server slices because you can't hit the performance of a single box)? This is entirely possible for something as memory bandwidth intensive as memcached, or as network bandwidth intensive as an AMQP broker. Where's the crossover point?

The bright spot here, though, is Open Protocols. I know I keep banging on about it, but where this all merges together brilliantly is that you don't have to have your own appliances or hardware in your Cloud Computing vendor: you can just talk open protocols and allow the utility provider to sell it to you. You want memcached? They'll sell you an IP/port that has access to X gigabytes and guaranteed response type of Y for $Z/[request|month|core time]. Open network protocols thus act as the disconnected version of open APIs in the software development world: you can swap out the underlying implementation without substantial changes to the rest of your environment.

Furthermore, working with Open Protocols in this case insulates you from PaaS lock-in. If your utility provider is providing Caching As A Service using their own custom, proprietary protocol, in order to move to a different PaaS vendor you'll have to change your app stack in some pretty substantial ways; if it's both sides providing an equivalent service (meaning protocol and semantics), you can migrate with minimal efforts (particularly if it's a transient service).

Footnotes

[1]: Meta-note: haven't been posting recently, even with the AMQP F2F and Solace announcements, because I've been busy working on some big stuff that I can't talk about at all yet. If it happens, I'll let my adoring public know.
[2]: Because Red Point, which backed Radik in the past, funded them in their Series A. I like to see what they're investing in.

AMQP: Community, Patents, Openness

(Background: My initial rant on the Red Hat patent situation; my rant on the Red Hat press release).

There's been a lot of public discussion recently regarding AMQP and patents, specifically USPTO 20090063418. For the record, I will be formally objecting to the USPTO about this particular patent; regardless of who filed it or when or on what, it appears to fail the Obviousness test and I have been provided with quite relevant Prior Art that indicates that the patent should not be granted. This has nothing to do with the authors of the patent, but with the fact that I don't believe the patent is worthy of grant by the USPTO.

And for the record, I personally do not believe that software should be allowed to be patented, or that building up "defensive" patents are a good corporate response to the current insanity. However, I also understand that this is a decision that is made at board level, and no single employee below board level is responsible for their firm's patent policy, whether it be Microsoft, Red Hat, IBM, Cisco, or Some Random Startup.

Do I think that Red Hat behaved in a manner that I would deem admirable given their action in patenting an extension to an open standard? No. Do I think it's the fault of employees of Red Hat who are quite clearly feeling attacked by me? No. Just as these employees had no control over Red Hat's position on patents, they also had no control over whether they were allowed to speak to any particular person at any given time. Their lawyers were in control, and part of being an employee means following board-level rules. I don't know why their lawyers said that they couldn't disclose anything about the patent or issue particular statements at any given time, and neither do you (unless you're one of them).

Yesterday's AMQP PMC meeting was the first time that the PMC members had a chance to talk since the patent was unveiled and all the bad blood was unleashed. Here's my current take on the situation, and constructive suggestions to move forward.[1]

What Is The Specification?
First of all, a lot of the confusion and requirements of clarification come down to the precise nature of the patent grants that are part of joining the AMQP Working Group. Here's the problem: they're not public. More importantly, the definition of the term "the Specification," which appears to cause the biggest problems, isn't public. I searched on the AMQP web site, and they're not there. If I'm wrong, please let me know, but I sure as heck don't see them anywhere.

That makes it very difficult for anyone on the outside to verify any of the claims that people are making. I can't imagine that this is that commercially sensitive, heck, the W3C license grants are all public. Putting this into the public would solve a lot of problems here.

You Can't Steal People's Patents
Someone commented to me "well, why not just expand the spec every time a patent comes out?" This is prone to the precise "proprietary vendor fear" that the existing patent would be prone to. Let's look at the hardware vendors.

People like Tervela and Cisco, who are members of the AMQP Working Group, and Solace, who I've urged to become a member, have patents. And unlike the AMQP+XQuery patent, some of theirs actually are things that I think would be worthy of patenting. Having a policy that says "If you join the AMQP Working Group, we can render any of your patents moot by simply saying 'what your patent covers is now an Optional Extension to the AMQP Specification'" is just as much a kiss of death as saying "here's a patent that we filed that means if you add AMQP to your proprietary software product we'll sue you."

This matters because, as I've said repeatedly, if AMQP does not end up encompassing the existing proprietary MOM vendors, it's not going to win.

Disclose What You Can't Disclose
The Red Hat press release, in particular, was taken by many people (some who had nothing to do with AMQP and were neutral third-parties) as unnecessarily hostile, mostly because of what wasn't said. And I know that Red Hat employees have been completely absent from all public communications (excluding anonymous ones). These will be carefully orchestrated by the lawyers. That's a matter of board-level policy.

Here's what the board of any FOSS vendor who's dealing with patents needs to do: lay out the ground rules for what their employees can say and do, and tell the rest of us. Unless you do that, when you don't make a comment about a particular area, the rest of the world doesn't know why. Is it because there is malicious intent? Is it because your lawyers won't let you? Is it because you didn't deem it necessary? We don't know. We want to know, and I can't see a single rational basis for saying "we cannot tell you what we can and cannot talk about, even in the most generic terms." Meet us half-way.

Open Standards Are Exactly That
People participating in this process should expect that this is an Open Standard, and Open Standards are messy. They're delightfully, angrily, bloggingly, twitteringly messy. You think this little incident was unacceptably messy? You been following the closure debates in Java? What about HTML 5 or XHTML 1? ODF/OOXML?

My point here is that open standards are, by their very nature, subject to a large group of people piping up from the sidelines. They won't always say what you want, and they won't always behave the way you want. And that's still valuable, because they add content. This probably just happens to be the first time that I know of that AMQP has been prone to a big public fight. Get used to it.

This is actually a Good Thing. How many people do you think actually know what AMQP is now versus a week ago? How many people do you think are following what's happening in the community now versus a week ago? Public fights are signs that a community is healthy. Otherwise you end up with an ANSI Committee of the Good And Worthy And Irrelevant, and nobody listens to them anymore (I have my "pet committees" there, and not all of them are like this, but we all know some like these). They're closed, they deliberate in private for years, and only then do they hand something down from on high that the industry ignores because it's moved on in its absence. Debate, unruly as it is, means you're doing something right.

A Way Forward
I'd like to be so bold as to propose a constructive suggestion on a way to move forward. These are entirely my ideas and I've not discussed them with anybody yet.[2]

1. Publish key licensing definitions. I know that some components of membership may be legally or commercially sensitive. However, if we (as users and developers) are going to trust that everything is 100% on the level, we have to know the key terms under which patents are handled. In particular, we need to know core definitions for things like "the Specification."
   While many of these terms may seem self evident from someone in the PMC, they aren't for those of us who aren't. And that information disparity can lead to a lot of argumentation as people argue from different levels of knowledge. Level the playing field, even if only a little bit.
2. Determine an optional but interoperable feature policy. This should be for things that are not part of the required specification (and for the record, I strongly favor a minimal specification), where you still want interoperable implementations of common scenarios. This is where the Red Hat AMQP+XQuery patent would lie, because if I want to do XML-based CBR, I most certainly don't want to have 8 versions that are all just a tiny bit different to avoid each other's patents. Same with geospatial routing, dynamic multi-hop routing, whatever.
3. Assemble a clearing house for prior art. There's a lot of prior art in messaging that can hopefully develop into a massive database that can be used for the community to collectively fight against patents from non-members. Let's get this going. I don't know how to do this, but I'm sure there's a way.
4. Communicate more openly. Usually in a situation like this from another spec or working group, people from the spec committee would be talking constantly with the community in a constructive fashion. That didn't happen here. There was nothing public from the PMC for days while the process deteriorated until members of the PMC started making very inflammatory statements in public.
   As I mentioned earlier, you have to expect stuff like this is going to happen, and at least one or two people have to be empowered to talk openly and early, and need to do so. AMQP is turning the corner from something that was in a vendor niche to something that people are starting to Really Care About, and that means that you have to start dealing with people who aren't on the PMC on a regular basis, whether you like it or not.
   And some of those will be differences of opinions between PMC members which will spill out into the public domain. This is normal and healthy, and as long as people can remain civil and constructive, there's no reason why all communications should be limited to PMC-only. That's not constructive, because you will never effectively present a public face, so you have to allow for some public discussions when the PMC doesn't have a standard, quickly available response.

I want AMQP to work. I want the spec to be adopted. I want asynchronous MOM-based communications to flourish. I think we can move past this and AMQP will be stronger as a result.

Footnotes:
[1]: This is all from purely publicly available information.
[2]: For the first and last time, I am not a motherfarking sock puppet. If you think there's ever been anyone who can control what I say, you're sorely mistaken. Employers (e.g. Big Bank B) can tell me to not talk at all, but I'm not the type of person who will just repeat a party line with a straight face if I don't agree.

AMQP: Why Do I Care?

UPDATE 2009-08-18: I've been told that I signed an AMQP Reviewer Agreement, which is a different document from full licensees. I've removed the footnote as it turns out it was completely incorrect.

Aside from the fact that this blog is called "Kirk's Rants" for a reason, I get especially strident when I start talking about AMQP. So much so that in addition to a comment on one of my recent Red Hat posts, I've been asked by an online magazine (as part of an interview process) why in the world I care so much. It's simple: I care because I've been bitten in the past.

This blog's history with AMQP goes way back. In fact, it's my #2 post label (only immediately under the more generic Messaging). Let's take a trip down memory lane, shall we?

My first ever post on the subject was a lamentation. I had been having problems with the C++ client libraries for SonicMQ (an otherwise fantastic product, I must say). I grabbed the JMS client for qPid Java, and RabbitMQ, and they just didn't work together. Surely, however, that was the whole point of AMQP as an exercise? And I've been involved ever since.

I've read drafts of specifications. I've signed a Reviewer Agreement with the AMQP Working Group (though I'm not a member of it). I've met John O'Hara and Pieter Hintjens and Alexis Richardson and Carl Trieloff and gotten about as involved as you can be and not work for an MOM producer. I've gotten involved.

But why? Because it matters.

I've worked in anger with:

• IBM MQ
• SonicMQ
• Tibco Rendezvous
• ActiveMQ
• An in-house developed MOM broker at Derivatives Group A
• An in-house developed MOM broker at Big Bank B

In addition, I've done formal evaluations on:

• FioranoMQ
• Tibco EMS

Every single one of them has bitten me in the ass before, usually around client libraries. For that reason, most applications are written against one broker technology, whether it's the appropriate one for all use cases in the application or not. Or, a whole firm's MOM infrastructure will be written against one broker technology, to limit all the problems that come in when applications have to support multiple client libraries, or have to select them for each application.

I view this as something that holds back MOM and keeps it in a Financial Services Ghetto. I don't view it as odd that the "Messaging Is Not Just For Investment Banks" presentation at qCon attracted more people than the AMQP presentation. Getting your head around what you can do with fully asynchronous fire-and-forget APIs is hard enough; getting your head around the fact that if you're not adhering perfectly to the JMS spec [2] you have to learn and work with completely different client APIs, half of which are broken at any given time is just too much. It's why so many people just jump into using XMPP: it may not have the best semantics, but at least it interoperates. This needs to end.

For MOM to reach completely widespread adoption, I need to tie my clients to my programming language and operating environment, not to what the provider of my broker technology is willing and capable of supporting. In order to make that a viable proposition, I need to consolidate to one protocol. Therefore, in order to get to a situation where enough engineering work is going into the client libraries to make them really really good, I need to have very few of them, namely 1.

Imagine if the web didn't have HTTP. Imagine if instead of having one protocol that would talk to everything I had to have separate client libraries for Apache, and Netscape Web Server, and IIS, and Zeus, and anything else that I wanted to support. Imagine, further, that Apache is 2% of the market. That's the situation we're in with messaging at the moment. To break out of that situation, we need the HTTP of messaging (only with less text and more binary, natch). That's AMQP.

Again, remember the Apache being 2% of the market scenario. In order for AMQP to be a success, it has to embraced by the 98% of the market using proprietary solutions, so that applications which rely on those solutions can be converted. Once those applications are converted to AMQP, users (not developers) can try different brokers and see if there are perhaps ones that suit their applications better. Once that happens, we'll have proper innovation in the MOM space. I want to live to see that day.

I've spent a lot of time with Alexis Richardson and the rest of the RabbitMQ team. Given that they're in London, I'm in London, I personally get along with them down t'pub, and I spent a fair amount of time hanging out while looking for a Cash Money Gig, this shouldn't be surprising. Furthermore, the fact that I originally discovered AMQP through Steve Vinoski discussing that Erlang is ideally suited to Message Oriented Middleware implementation and thus RabbitMQ was a really good use of Erlang means of course I'm going to be interested in hanging out with them. However, let me be very clear about something: I have never received a single pound from any provider of message oriented middleware solutions [3]. Since starting this blog ferreals I've received money from exactly two sources: Derivatives Company A, and Big Bank B.

In sum, I care about this stuff because proprietary MOM clients have bitten in me in the ass way too many times, and because fundamentally I hate closed protocols. I'm a practitioner in this space, who never (again) wants to:

• deal with another application that can send out messages to Broker Technology A, where I'm using (or want to use) Broker Technology B;
• be stopped from using Interesting Technology C because the Broker Technology I have to connect to doesn't support it;
• be nearly into production with a system, only to find out that the client library has a hideous bug that I can't fix because it's a proprietary protocol.

In short, I care because I'm experienced enough in the space that I've learned to care.

Every practitioner should care so much about this type of thing. If you don't care anymore about the fundamental constraints of the space in which you develop, why are you still in the game?

Footnotes:
[1]: REMOVED (see update at top of post).
[2]: Not using Java? So sorry, there's nothing equivalent in any other language, and each vendor decides how to map JMS onto C++ in a completely different way.
[3]: Okay, I let Hans Jesperson from Solace buy me a hot chocolate. Guilty!

Red Hat's AMQP+XML Patent Press Release

Red Hat has finally come out with a statement on the AMQP Patent (USPTO 20090063418) application. I have written about this already, prior to the press release coming out.

First of all, the people who are the most rabidly angry about this are other AMQP leaders (leaders of other projects, members of the AMQP Working Group). This isn't a case where (to the best of my knowledge) Tibco is coming out and saying "don't trust those crazy AMQP people, look, they've got patent issues too!" Rather, this is a bunch of people who feel quite betrayed by someone they wanted to view as a positive member of the community who shared similar desires and goals to the rest of the community. FUD doesn't apply here, as those of us with legitimate concerns are hardly going to be spreading FUD about the thing (AMQP) that we are really pushing for.

Next, the Patent Promise and Estoppel issues notwithstanding, the AMQP working group is a mixture of commercial and open source vendors. Red Hat's Patent Promise only applies to Open Source vendors. Given that AMQP, in order to be a success at all, must attract support from commercial vendors who have no safe harbour under the Red Hat Patent Policy, this patent has the ability to poison the entire community. This is a key part of my original argument, and there's been no response from Red Hat on this.

Finally, let's take one paragraph which is, quite frankly, fictitious:

Although there have been some recent questions about one of our patent applications relating to the AMQP specification, they appear to originate in an attempt to spread FUD. There’s no reasonable, objective basis for controversy. In fact, the AMQP Agreement, which we helped to draft, expressly requires that members of the working group, including Red Hat, provide an irrevocable patent license to those that use or implement AMQP. In other words, even if we were to abandon our deeply held principles (which we will not), the AMQP Agreement prevents us from suing anyone for infringing a patent based on an implementation of AMQP specification. Moreover, our Patent Promise applies to every patent we obtain. Red Hat’s patent portfolio will never be used against AMQP, and Red Hat will support any modifications to the specification needed in future to verify that fact.

The factual inaccuracy here comes from the fact that the AMQP Agreement covers an irrevocable patent license for any patents necessary to implement the specification. AMQP XQuery Exchanges aren't part of the single document AMQP Specification. Therefore, the AMQP Agreement doesn't factor in. Therefore, this patent is not subject to an automatic grant. Remember, this patent doesn't cover AMQP itself (and nobody's ever claimed that it does). Rather, it patents a quite obvious extension to AMQP. That's the problem: it's extension rather than core.

The only way around this at the moment given the licensing of the AMQP specification would be to add XQuery-based Exchanges to the AMQP specification (make it core rather than extension). However, since most work on the AMQP specification is to make it smaller and focus on standardized/interoperable extensions, this isn't a great solution.

In short, not impressed with the response. I didn't expect more, but this isn't a great state of play for those of us who care about efficient, vendor neutral message oriented middleware.

I am aware that the AMQP Working Group is working within themselves to try to resolve this situation, and I wish them all the best luck in that!

Red Hat's AMQP+XML Patent Application: Stupid, Shifty, Short-Sighted

UPDATE at bottom of post
I've finally gotten word back from my moles that Red Hat has essentially stated that Patent Application 20090063418 is a defensive patent on an extension to AMQP to work around the broken US patent system.

This patent was filed for in 2007 [1]. The current state of it is that it's been opened up to public review, which is part of the new patent system process for allowing public feedback on "controversial" patents. Essentially, as of now, infringement rights have started (so if the patent actually is granted, if you infringe as of now, they can go after you), but the patent has not been granted. This process is set up so that you can object to the patent, usually via prior art (so if, for example, you were running AMQP with XML before 2007, or you have documentation in the public space that this was a known concept for extension, please let me or the patent office know ASAP).

The way the IP rights in the AMQP working group operate is that each member must agree to disclose and grant rights on any IP that they might hold that would be necessary to implement the AMQP specification, so that the specification isn't prone to stealth patents after ratification. So if Red Hat had gone for a patent on Direct Exchange Routing, they would have had to disclose that and license anybody else in the working group, rendering the patent effectively useless. However, because XML-based exchanges, while the most common example for a custom Exchange type in the AMQP literature, are not necessary for implementation of the spec, Red Hat were quite freely able to stealth file this patent.

While legally they are thus in their rights to file the patent, this is a really bad maneuver on the part of Red Hat.

Let's assume that it is in fact a defensive patent, as Red Hat are claiming. If it is, why not disclose it? And who are they attempting to defend, the wider AMQP community or themselves in a patent war with IBM or someone else? If they're worried that someone else was going to come in and patent this precise thing, why not just put prior art into the wild so that the community could oppose anybody attempting to patent just this thing?

Let's take worst-case scenario here:

• Solace Systems decides to implement AMQP in hardware
• They already have a hardware XML-based routing system (equivalent to the patent when not applied to AMQP)
• They release the product into the wild
• Red Hat realizes that they're not selling much MRG, because people prefer the Solace solution
• Red Hat sues Solace for damages

Do I think Red Hat is likely to do this? No. Do I think if I'm a commercial vendor I have to assume that Red Hat might do this at some point over the 17-year lifespan of the patent (assuming it actually is granted)? Yes, I have to. For all I know, Red Hat will sell the patent to a litigation firm and lose control over it in 13 years time, and then sue me for infringement 13 years in the past.

Yes, the US patent system is horribly, horribly broken. But looking for opportunities to build a defensive patent portfolio around an open standard is exactly what's wrong with the US patent system. More importantly, it's really bad for AMQP itself, and thus bad for Red Hat.

As we discovered at the qCon AMQP talk last week, one standard concern that potential adopters have is "when are we gonna get a 'real' company implementing this?" Not many companies are currently eager to run something that doesn't have companies with deep pockets behind it (some are because it's open source, some aren't). Why would I be an early adopter of any AMQP implementation when the spectre of stealth patents filed by AMQP implementers on any obvious extension to the spec may be lurking around? Why adopt anything if doing so opens me up to the threat of a lawsuit, when there isn't anybody around with deep pockets able to offer me indemnification as of yet?

The main factor here is that AMQP is not going to get beyond the early-adopter ghetto until and unless it is embraced by commercial vendors of MOM technology. To get past the initial early adopter hurdle, you need to eventually draw in Progress, IBM, Oracle, Tibco, Fiorano, all these guys. And virtually all of them have XML-based routing technology. What Red Hat has effectively done is said "the moment you support AMQP in your product, without neutering the XML-based routing you already have, we can sue you." What incentive do those firms have to adopt AMQP and thus build the community?

The blowback is already starting even in the Open Source world. Martin Sustrik has stated to me:

We are suspending any work on AMQP support in 0MQ till the issue is resolved.

In effect, this completely poisons the well of the AMQP community, and I fail to see how, short of Red Hat granting control over the patent to some other entity, they can recover from this. It would not surprise me if this effectively kills AMQP, and with it the entire basis for the messaging component of MRG.

In sum, this was a really, really, really short-sighted move on the part of Red Hat, and I highly encourage the community to fight this patent on grounds of Obviousness and Prior Art, both of which I think it fails in. And then to never trust Red Hat again in any standardization process.

Footnotes:
[1] : This effectively means that pretty much the first thing they did on getting involved with AMQP was to file patents on it. What a lovely approach!

UPDATE 2009-03-17
Red Hat has come out with a press release on this issue. Link to the press release and my response in my follow-up post on the subject.

USPTO #20090063418 - Yes, I'm Aware

This has hit Slashdot, so given my audience I have to respond. Yes, I've been aware of this for several days. In the interest of fairness, I'm waiting for Red Hat to make a comment that might be accessible to me.

I'm not willing to wait forever, and I'm not pleased about the situation. Considering I met Carl Trieloff for the first time at qCon this week, I'm going to let them respond, but my patience isn't eternal.

In summary, assume the following:

• I know of the patent application;
• I'm apoplectic with rage (like 3 on the Kirk anger scale[1]);
• I'm swallowing that anger until I hear a good explanation for the patent application.

Footnotes:
[1]: A scale invented by ex-coworkers. Most people register in milli-Kirks at best. Yes, this is "Kirk's Rants" for a reason. :-)

Let A Million Brokers Bloom

To me, one of the most promising things about AMQP is that it actually has the potential to radically expand the amount of feature-based broker differentiation in the MOM space, by eliminating a lot of the "grunt" stuff that you have to go through in order to define a new broker model.

Right now, if you want to do something interesting (meaning not commercially supported) with asynchronous message-oriented communications, you have to:

• Define your basic messaging model
• Define the wire representation of that model
• Define your protocol
• Write up the server-side handling of the protocol
• Write up the client-side handling of the protocol for each programming language you need to support

Only when you're done with all that do you actually get to do "the fun stuff," which is to write up the unique piece of functionality that meant you wanted to start on the project in the first place.[1]

Consider a particularly germane example. In my current day job I'm working for a firm (and if you're reading this blog you've heard of the firm) that has a proprietary MOM system with (what I can see as) one USP.[2] Aside from that one USP (which I'm not going to identify), it's just a simple pubsub-based MOM broker. But nobody in the market has that feature, so they can't go with a commercial vendor. And there are facets of that USP that means that no commercial vendor is likely to add exactly that feature to their product.

But in order to get that one USP feature, they have to face every single other hurdle above, and constantly deal with refining every single bit over years and years of development. None of which is about that one USP feature, it's all just plumbing. Everything other than implementing that feature is a waste of valuable development effort.

The kicker? That USP is just a custom Exchange type in AMQP 0-X terminology. Nothing more, nothing less.

And that's the beauty of AMQP.

Once we reach the promised land, someone like my current cash money provider could have just grabbed an existing framework (all the AMQP client work and server-side protocol handling in its language of preference), and implemented the custom exchange (or broken existing ones, or got freaky with queue implementations, or whatever it wanted): no need to reinvent the wheel. AMQP acts as the standardization of all the bits that you don't want or need to care about to provide custom message broking behavior, allowing you to focus all your time and attention to the parts that you really need to implement yourself.

This makes the barrier to entry for unique MOM functionality only the cost to implement the unique functionality itself; you don't have to face the hurdle of implementing everything else required for asynchronous message-oriented communications to showcase or provide one single feature.

I consider this to be the messaging version of the story behind the Eigenbase project: a framework for writing custom brokers. A few people I know (big shout out to my homies John Sichi and Julian Hyde[3]) are excellent at writing "interesting" databases (with special sauce for special contexts). However, doing that from scratch each time means that you're constantly reinventing/rewriting the same old tired stuff: a SQL parser, a transactional data store, a wire protocol, blah blah blah. So they started the Eigenbase project as a way of taking care of all that cruft that you just Have To Have so that they could focus on the stuff that's special about Their Database.

Are there equivalently architected Framework Brokers yet for AMQP? Not that I'm aware of (please pipe up in the comments if you think I'm wrong). If nothing else, in the message broker/routing space, there are a lot of things that you might want to do to speed up adhering-to-standards mode that break the general programming model you'd want from a framework-based system. That leads me to believe that just as MySQL didn't go framework-based to become Eigenbase, we'll probably see a set of AMQP brokers focusing on ease of programming custom variants, just like Eigenbase is designed specifically around programming custom variants in the database space.

And at that point, I think we'll start to see a lot of new and interesting implementations, some in-house and some exposed to the world.

Footnotes

[1] Alternatively, you can horribly abuse an open source project, but that's going to do nothing but confuse everybody involved (including yourself) and leave you liable to abandonware if you rip out the insides of a system that you're replacing key components on (so if your open source basis isn't designed as a framework, and you replace some of the key message handling parts, the moment the developers change the internal processes you're effectively stuck).

[2] Unique Selling Proposition. But you knew that of course.

[3] Broadbase In Da Hizzy!

Tervela Joins AMQP Working Group

The Press Release. No, it's not on the formal AMQP Working Group web site yet as of my writing this.

I think this is really promising, as it's the first hardware vendor to have formally signed up to join the AMQP Working Group (despite my prodding of Solace to do so). I hope they manage to produce a full hardware implementation of the spec!

It also continues the industry's relentless march to where I believe is its destiny:

• Open Source
• Cheap Appliances
• Custom Hardware
• Cloud Provision
• IBM

If you're not one of those 5, figure out how you can be, because otherwise you're in the downward spiral. If you're paying someone not one of those five, start working on your exit strategy before it's too late and you're paying ground rent until the end of time for a second-rate solution. Don't migrate yet, but have your plan in place.

Update: Here's the Official Tervela site press release.

I Use Product X, AMQP Is Irrelevant To Me

As I've gotten more involved with the greater AMQP community, and as I've talked more and more about it with people who already "get" messaging, I keep hearing the following comments:

I've made a significant investment in Product X [MQ Series (or whatever IBM's calling it these days), SonicMQ, Tibco EMS/Rendezvous, FioranoMQ] and I use it for as much as I can. I'm happy with it. I'm happy with vendor support, all my apps "just work." Why in the world should I care at all about this AMQP thing?

It's a legitimate sentiment. A lot of the AMQP hype has been driven by next-generation pure-play AMQP broker platforms trying to eat into those existing install bases. I still think it's relevant to you, the Happy Commercial MOM Broker Customer.

It's The Ecosystem, Stupid!
Remember back when I initially ranted about AMQP. One point I made there, quite aside from my "My commercial vendor is annoying me" ranting, was about the overall ecosystem. Here's where I make my AMQP pitch: AMQP Will Help You Leverage Your Existing Investment.

Let's assume that you're completely happy with your WebSonicFioEMQ platform and you've changed all your code to be able to use it. You're still not leveraging it to its maximum level, because no matter which one you chose (or if you chose them all), you can't always get third party products to integrate properly.

Consider one pretty Enterprise Software product: Sybase Replication Server. Last time I checked, it could pump out event messages on "interesting" events through three mechanisms: JMS, Rendezvous, and MQSeries. But I'm sure it's actually a lot more specific than that (I can't check directly as it's firewall protected, but could someone who has access fill us in in the comments), down to the version of individual JMS Brokers and versions of Rendezvous and MQSeries and all that that it supports.

Consider some more emerging products like Continuous Integration platforms. All the modern ones can send alerts using email and Jabber.

They're astute in choosing the most dominant messaging mechanisms in their target environment: commercial MOM systems in the case of Sybase Rep Server, and open communications standards in the case of Continuous Integration. And therein lies the rub:

• Unless your MOM platform is truly ubiquitous in your domain, every piece of software you want to integrate into it will result in custom effort;
• Your MOM platform isn't ubiquitous in every domain.

It doesn't matter what you're using for asynchronous communications: it's not ubiquitous enough.

AMQP Promises Ubiquity
Let's assume that the promise of AMQP really gets going. I wholeheartedly believe that the sheer engineering effort that application vendors can save through only supporting one protocol, and thus one client library package, without any interoperability concerns, will force it through.

Consider someone like Atlassian. You're an Atlassian customer, and you want your Confluence and Jira and Bamboo and FishEye instances to fire off messages to your existing MOM infrastructure when "interesting" things happen. So you lobby and say "Hey, can you support SonicMQ for message publication?" and they say "hey, that looks pretty interesting."

But the moment someone else catches wind of this, pretty soon they have a many-to-many compatibility story as every customer with a different infrastructure pops up and says "if you support X, you should also support my MOM infrastructure as well." Math is Hard, Atlassian Goes Shopping and says "write your own plugin, yo."

What if Atlassian could satisfy every single major MOM infrastructure with exactly one protocol? They'd do it. Interop testing by the broker developers (and not Atlassian) will take care of everything else.

AMQP Helps Your Commercial Investment
If you think this all seems like it makes sense (and if it doesn't, let me know precisely what about it doesn't in the comments), your immediate response should be to get on the phone to your salesperson and say:

• I want AMQP support in a (near-)future release of your MOM Broker;
• If you don't do it, I will roll out someone else in parallel and bridge them.

Trust me, this will get them thinking. Because I'll guarantee that within 6 months of AMQP 1.0 coming out, there will be open source bridges between every major commercial MOM infrastructure and AMQP. And the absolute last thing your sales person wants to hear is that you're thinking of adding something to reduce your existing lockin.

Once this is done, you get the best of both worlds:

• You can leverage the entire organizational and technical infrastructure you've already built up to do MOM properly without adding more "products" to manage; and
• You can integrate yet more applications and participate in a far richer ecosystem than Your Product could ever support on its own, helping drive that investment farther.

So get on the phone to your sales guy. Now.

RabbitMQ/AMQP OSGi Integration

I had a chance to meet up with Neil Bartlett today, and got to talk to him about his RabbitMQ OSGi integation (note that I believe that everything he's done thus far is actually applicable to anyone using an AMQP broker that is compatible with the RabbitMQ Java client, so I'm treating it as a generic AMQP/OSGi module). While it's a work in progress, it's incredibly cool. It allows you to:

• Declare production/consumption parameters (exchanges, queues, hosts, credentials) in a declarative way;
• Have your OSGi container connect the dots so that your declarative bundles automatically match up with concrete connections to your AMQP broker of choice;
• (If you're using Equinox) Use the OSGi shell to interact with the RabbitMQ/AMQP client library.

The first two parts were the types of things that I had worked on in the past (having a declarative way to specify messaging endpoints, down to the broker connectivity level) so that you could change them at execution-time. The third was the real WOW-worthy point.

Assume that you're coding against an OSGi runtime. Assume that you're working with bundles, and dynamically loading and unloading them as you're compiling your code (meaning that you're probably at an osgi> prompt). Assume that want to see very quickly whether your code is interacting properly. Now you can easily setup your bindings and publish and consume messages as your real code is in flight, and see the effects in your debugger.

How amazing is that? Imagine the human cycles that you'll save being able to do all this from one area, that you're already familiar with?!?!

This is a work in progress, and hopefully Ben Hood and I had a chance to give him some places to go from here, but I think this is exactly what the promise of AMQP offers: the chance to do amazing things based on not having to support a million different client libraries, but to code against a standard and make great stuff happen.

If you're using the RabbitMQ Java Client and any type of OSGi container, I highly encourage you to take a look at this and get involved. I see great things happening here.

2009 Goals

Following on from my predictions for 2009, let me follow Joel's lead and try to turn Seth's comment into a meme. So without further ado, here are my goals for 2009 (whether they're SMART or not is an exercise to the reader and yours truly).

Find Gainful Employment
As those of you who are faithful readers know, I found myself affected by the Credit Earthquake and am currently seeking gainful employment. My #1 goal is thus to find myself once again part of the gainfully employed section of society (and if you think you can help that goal, feel free to contact me).

Extend Asynchronous Communications To The Masses
My #2 goal is to seek an end to polling; it's inefficient, it doesn't scale, it doesn't smell right, and it's not necessary. Expect more posts here about this, as well as ceaseless and boring arguments elsewhere about how we, as software engineers, have alternatives, and should seek and embrace them.

Help AMQP 1.0 Along
Many of you found me through my ramblings on about AMQP 0-10. I believe in AMQP, and believe proprietary protocols are doomed to the dustbin of history. As such, I am setting my #3 goal to be to do everything in my effort to give us, in the community, a workable, interoperable AMQP specification that we can use to achieve my #2 goal above. This means that I'm actually being less ranty and more constructive in my relationship with people helping to get AMQP 1.0 along; have no fears, my innate lack of tact hasn't disappeared, I'm just subsuming it towards the greater good.

Fix My Flat
Usually this would go on my (much-neglected) non-technical blog, but most of the readers there read this as well. My flat is a vast collection of Fail. We've spent two years (since we moved in) trying to resolve the Fail into a cascade of Win. My #4 goal is thus to have a fully functioning flat, with all major services working.

Get Zoo to Cash-Flow Positive
Again, a non-technical subject, but many readers know that we setup and run a registered UK animal conservation charity (the Rare Species Conservation Trust), in part to operate the zoo that my partner operates, the Rare Species Conservation Centre. After a massive, multi-year financial and time investment, I believe that this is the year that the zoo will pay for itself and (as we originally envisaged):

• Help us fund the in-situ wildlife conservation work that we've devoted so much of our lives to;
• Continue our mission of educating the UK public in the plight of the animals that have touched our lives and are worthy of protection in the wild.

We're trying to help undo the trauma that man has inflicted on some of the rarest, most endangered, most misunderstood animals on Earth, and I hope that this is the year that our efforts to educate the UK public are rewarded and we can turn back to our main goal: helping in-situ wildlife conservation efforts.

It's only a few goals, and several of them are under way, so maybe I'll actually hit my goals this year after all.

2009 Predictions

Just a quick set of predictions for 2009. I'll revisit nearer the end of the year to see how I've done.

Messaging Breaks Out
Right now asynchronous systems, particularly ones using message oriented middleware, are still pretty fringe. However, even though I disagree with the technology choice, techniques like XMPP are starting to bring asynchronous communications to the masses. I predict that the extension of XMPP, the forthcoming standardization of AMQP, new infrastructure being developed (like RabbitMQ, OpenAMQ, and Qpid), and extension to new programming languages and application execution environments will all drive more and more developers to finally end polling for updates.

Cloud Will Become Less Buzzy
Right now Cloud/Utility computing is really too buzzy to make heads or tails of, and I think it's suffering from that: when you say "Cloud Computing", it means both nothing and anything. I think 2009 will be the year that people start to solidify what works best in a utility computing space versus a local hosted space, and where hosting providers get mature enough and technology becomes mainstream enough that there's an assumption of utility over local hosting.

Java Will Stagnate
I predict that Java 7, if it even hits by the end of the year, will be pretty anemic and, quite frankly, lame. There will be some interesting enhancements to the JVM, but that's pretty much all that's going to be noteworthy.

C# Will Over-Expand
The entire Microsoft ecosystem will expand and grow beyond the ability of typical Microsoft stack developers to keep up. We're already seeing it with WinForms vs. WPF, LINQ vs. ADO.NET Entity Framework, and whole new technologies like WCF. I predict by the end of the year the Microsoft ecosystem has become so bewildering that while leaders in the community push for more change, day-to-day developers push for a halt to just absorb what's been happening.

Non-Traditional VM Languages Will Break Out
Right now most developers in enterprise systems are coding against the "core" languages (C#, VB.Net, Java) against their respective VMs. This reflects in no small part the maturity and power of their underlying runtimes (the CLR and JVM respectively), but the leading edge of the communities have already started to explore other languages that offer concrete advantages (F#, Scala, Groovy) on the tight runtimes that are already available. This will be the year that those actually break out of the leading edge and into mainstream use.

No-One In Social Networking Makes Money
I still think, by the end of 2009, there won't be a player in the social networking space that is cashflow positive, and I don't think it'll be by choice. I think the technology is so disruptive that we still haven't seen the "right way" to monetize it yet.

Sun Radically Restructures
Sun can't keep burning through cash the way they have been, and they can't continue to have such a chaotic story. At some point in 2009, Sun will relatively radically restructure itself in a bid for survival. Hopefully they'll have read my analysis (part 1, part 2). At the very least, they'll change their ticker away from JAVA.

I think you'll note that all my predictions are fuzzy. That's so that in December, 2009 when we revisit them, we can determine that I was partially right on nearly all of them.

REST Requires Asynchronous Notification

I recently (well, compared to people like Steve Vinoski) got converted to the RESTful architecture movement, but one of the first things that I recognized is that it suffers from the same generic limitation of every HTTP-based component architecture that I've seen: there's no defined way to get updates on content.

In fact, when I started getting more immersed in the REST culture, I realized that a lot of the nuances that are ideal to master in order to provide a great RESTful interface (caching, timeouts, advanced status codes) are there specifically to limit the number of repolls that are necessary against the terminal service by improving the ability of caching proxies upstream to handle requests.

But no matter what, there are clearly latency issues involved in any polling system, particularly one which is using fixed content caching timeouts to optimize performance (because this puts the latency entirely under the control of the publishing system which generated the result of the initial GET command). How do we minimize those?

Pull RSS Is No Substitute
RSS/Atom (note that I'm going to use RSS as a shorthand for the whole RSS/RDF/Atom spectrum of pull-based syndication lists) seems like a reasonable way to minimize the actual content that needs to be polled by batching updates: instead of having clients constantly polling every URL of interest, you batch them up into groups (perhaps one for each resource domain) and provide a single feed on each group. The RSS feed then indicates the state of all the resources in its domain that might have changed in the last period of interest. So now we've reduced hundreds of GETs down to one, but we're still polling.

My old colleague Julian Hyde posted some stuff recently which was precisely along the lines of what we're talking about here (post #1, post#2), in that he's building a live querying system against streams, and finding that the polling nature of RSS is particularly noxious for the type of system they're building. I'm surprised it took him so long. RSS is actually a pretty terrible solution to this problem: it combines polling with extra data (because an RSS feed is generic, it publishes more data than any particular consumer needs) in an inefficient form and still allows for the RSS publisher to put a caching timeout. Wow. Talk about a latency effect!

It gets worse. Let's assume that you're trying to do correlations across feeds. Depending on the cache timeout settings of each individual feed, you might have a latency of 1sec on one feed and 15min on another feed. How in the world do you try to do anywhere near real-time correlations on those two feeds? Answer: you can't. At best you can say "hey, dear user, 15 minutes ago something interesting happened." 15 minutes is a pretty short cache timeout for an RSS feed to be honest, but an eternity to many applications.

The only real advantage that RSS/Atom provides is that because it's based on HTTP, and we have all these skills built up as a community to scale HTTP-based queries (temporary redirects, CDNs, caching reverse proxies), we can scale out millions of RSS queries pretty well. Problem? All those techniques increase latency.

Enter AMQP
Anyone with any experience with asynchronous message oriented middleware can see that fundamentally you have a one-to-many pub/sub problem here, and that smells like MOM to me. More specifically, you have a problem where ideally you want a single publisher to be able to target a massive number of subscribers, where each subscriber gets only the updates that they want without any polling. Wow. That looks like exactly what MOM was designed for: you have a broker which collects new messages, maintains subscription lists to indicate which consumers should get which messages, and then routes messages to their consumers.

It's so clear that some type of messaging interface is the solution here that people are starting to graft XMPP onto the problem, and using that to publish updates to HTTP-provided data. Which would be nice, except that XMPP is a terrible protocol for asynchronous, low-latency MOM infrastructure. (If nothing else, the XML side really will hurt you here: GZipping will shrink XML quite a bit, at a computational and latency cost; a simple "URL changed at timestamp X" is going to be smaller in binary form than GZipped XML and pack/unpack much faster).

But we've already got another solution brewing here: AMQP, which simultaneously provides:

• Binary protocol (for performance)
• Open protocol (for compatibility)
• Open Source implementations
• Built by MOM experts

Looks like a winner. Just about the only thing that I can think the REST crowd not liking is that it's not over HTTP (The One True Protocol), but every reasonable MOM system has support for HTTP-based subscriptions (at a pretty serious performance penalty).

Combining REST with AMQP
At my previous employer, I actually built this exact system. In short, resources also had domain-specific MOM topics to which the REST HTTP system would publish an asynchronous message when any update to a resource happened. Clients would hit the HTTP endpoint to initialize their state, and immediately setup a subscription to updates to the separate MOM system.

Without the MOM component, it's just pure REST. Once you add it, you eliminate any resource or RSS polling at all. The MOM brokers are responsible for making sure that clients get updates that they care about, and there's only the latency that the broker introduces to updates. Worked brilliantly: all the benefits of a RESTful architecture, with none of the update latency effects.

We had it easy though: this was all purely internal. We didn't have to operate at Internet Scale, and we didn't have to interoperate with anybody else (no AMQP for us, this was all SonicMQ as the MOM broker).

In order to get this to work as a general principle, we as a community need to do the following:

1. Finish AMQP 1.0 (I've been told the specs are nearly ready and then just comes the implementation and compatibility jams);
2. Decide on a standard pattern for REST updates (topic naming, content encoding, AMQP broker identification in HTTP response headers);
3. Develop easy to use publisher and subscriber libraries (so that services and client packages can easily include this functionality across all the major modern languages);
   
4. Develop AMQP brokers and services that can operate at Internet Scale.

Once we've done this, people like Julian can hook up SQLStream directly up to AMQP endpoints and consume all the data they want without invoking the horror that is polling.

Sell Me An AMQP Appliance

I've been doing some thinking recently on the bevy of AMQP broker software which is being created, and I think it's a really positive thing that we're definitely seeing a new swathe of innovation in the broker-based middle ware space. In particular, the space seemed to many to be a fossilized space just a year or two ago, but now with AMQP and XMPP PubSub, is getting a lot of new development momentum. But ultimately, I'm starting to think that perhaps much of the innovation will be wasted on a large category of users.

I believe that we're going to see the broker-based MOM space divided into three categories:

• Edge Distribution. Imagine that you have 10 processes on the same machine that are all receiving the same data. The most optimal way to distribute the data is for that machine to be running its own broker and connecting with the hub broker to distribute each message once for redelivery to the 10 processes on the edge system.
• Embedded. In this case, an application is being written against the MOM protocols in question, but everything's happening inside the application's configuration, so that there are no external dependencies. Atlassian's Bamboo is a good example of this: they embed ActiveMQ so that their customers don't have to have an existing JMS infrastructure in place, and you don't even necessarily know you're running JMS at all.
• Central Distribution. This is every other case, and generally means a node that is configured just to act as a broker, and isn't really doing other stuff.

The thing I've come to realize is that for the third case, Central Distribution or Dedicated Broker, I don't want to buy or run "software" at all. I want to buy an appliance.

Why An Appliance?
I want to buy an appliance because I don't want to have to deal with the complexity of optimizing my installation (and by that I mean the entire hardware/OS/software stack) for the purposes of running a MOM broker. Let's start anecdotizing.

When my company was planning on rolling out SonicMQ across our entire infrastructure, we spent a lot of time working on our proposed infrastructure. This included, but is not limited to:

• Base Hardware Configuration. What processor? How fast? How many?
  
• Failover Network Connections. For our High Availability Pairs, what network connection(s) did we want to use for the replication traffic? How precisely were they to be configured?
• The OS and Patch Level. This got particularly hairy with some of the hardware options for the failover network connections, and we had to do a lot of patch wrangling to test out various hardware options.
• Filesystem Layouts. For the various types of filesystem storage (WAL for persistent message storage, message database for persistent topic and queue storage), how did various options of disk partitioning, disk speed, filesystem layout and options affect performance?
• RAM. How much? What GC settings for the JVM? Swap configuration?
  
• SonicMQ Configuration. How did any of the above affect SonicMQ? Were there any things we had to tweak to get it to perform better?

How much of that do I think was really part of our jobs? None. None at all. I think it was completely wasted time. And that time costs money, in that I was working with our systems staff on testing and configuration and we don't work for cookies (cookies are merely a bonus that endear us to vendors).

If Progress had sold a 1U SonicMQ Appliance, which had 4 external NICs and a pair of HA connectors, we would have bought it. Even at a premium over the software+hardware, because that premium couldn't have cost more than our time did.

Things Are Getting Worse
Now, things are starting to move really quickly on a lot of fronts that would affect an MOM appliance:

• Storage is changing rapidly. An ideal platform here would have some flash in it somewhere. But where? Fusion-IO-style PCIe card? Sun-style ZFS log acceleration? I have no idea, and I don't want to have to know precisely how to make your software go faster with Flash added. That's your job, Mr. Vendor.
• Networks are also changing rapidly. What to use for an interconnect? 10 GbE? Infiniband? Something else? You figure it out for me.
• Chips are opening up. Do you work best with Xeon? Opteron? Niagara? Does it matter? How fast do they have to be to saturate all network connections? I don't know. That's your job.
• Can you go faster with TCPoE? Or does it clog you down? What about custom FPGAs for routing tables? Any way to leverage GPGPUs? Again, you're the vendor, figure out the most cost effective way for me to go as fast as possible.
• Can you move into the kernel? If so, what parts? Remember, on a general purpose system my systems team will never allow a vendor to muck with the kernel. Once you're selling an "appliance", you can do what you want.
  

Just to pick on one of these that caused me no end of woe, even today, the choice of 10GbE HBAs matters: different cards have different bandwidth potentials per-socket vs. per-NIC, so this type of stuff really does start to matter more than it did before, and I don't want to have to test it all out for you. Which one do I use for an HA interconnect? What about for my connections to the outside world? I don't want to have to make that decision. I want you to do it for me based on what's optimal for your broker software.

Why Hasn't This Happened Yet?
I think the primary reason why this hasn't happened for things like MOM providers as of yet is that we, customers, just plain don't trust vendors. At all.

We don't trust them because:

• They don't cycle hardware quickly. Our SonicMQ boxes are running on dual dual-core Opterons. It's monstrously overkill, and they're mostly idle. But generically most customers don't trust vendors, particularly of second-tier appliances, to keep up with industry trends, and that costs you in terms of overall performance. They have to buy in bulk to get the discounts from the chip vendors, and that means that they're going to push their stock as long as they have to.
• They monstrously overcharge. Anyone who buys storage (and a perisistent MOM system is a storage beast as much as a network beast) knows this. Buy a disk drive for $X. Put it in a slide which costs $Y. Sell it for $Z where Z ~= 5(X+Y).
  Look, we get it, you need to charge a premium. But don't sell me ordinary equipment and charge a monstrous premium over what I can get myself from my reseller, or by going directly to the OEM. Clue Phone's Ringing: We're Not Idiots.
  You can charge a premium where it's warranted. But being an "appliance" vendor is not an excuse to jack prices up to a ridiculous level just because you can. You do that, and we'll start asking for generic software and configure it ourselves again.
  
• They're Another Hardware Vendor. Each hardware vendor we have means another stupid type of disk slide and another "certified" RAM module which is the same as your standard PC RAM. It's all extra stock we have to hold, it's all extra work our purchasing department has to do, it's all extra validation we have to perform, it's all extra overhead.
• They Don't Manage Well. Each appliance vendor seems to think that Their Approach is the Right One. So instead of saying "we're going to support getting machine statistics for Linux boxes" and just deploying that, we customers end up having a myriad of different SNMP MIBs and other little tweaks that means that for each new appliance, we have to change a whole lot of our management infrastructure. Blech.
  (Anecdote Time! NetApp provide SNMP MIBs on their network adapters. But for total bits, they don't provide a 64-bit counter. Meaning that if you have a 1Gbps network connection, it'll roll over in like 7 minutes. So if you're not running your stat gathering MRTG/Whatever every 5 minutes, you'll have to factor in rolls of the 32-bit counter. I will assume that you, as an "appliance" vendor, will have similar levels of Stupid).
  

Solution Study: LeftHand
LeftHand Networks had an extremely interesting approach to this, which is that they had software (SAN/iQ) that really benefitted from close hardware integration, but they partnered with Tier-1 system vendors (Dell, HP) to certify the LeftHand software on the system hardware. Brilliant option. They're now part of HP, but the idea was really sound. But take it a step further, and do what a lot of companies are now doing.

OEM the platform of a Tier-1 system vendor (Dell, HP, IBM, Sun). Sell us that platform as is. Don't bloody pretend it's something magical (for example, don't even bloody think of not allowing me to swap out one of your Special Magic Drives for a stock Dell/HP/IBM/Sun drive). Just sell it as an appliance with your software.

But even more than that, be quite open that what you're selling is:

• Tier-1 Hardware, running
• A customized Linux/OpenSolaris/*BSD operating system, with
• Your MOM software, with
• Perfect configuration, and
• Special Sauce (optional).
  

Don't stop me from going in an poking around in a read-only way at what you're doing, make sure that I can replace your hardware with stuff from the Tier-1 vendor itself, and make sure standard Linux software (like the standard SNMP MIB and SSH and top) are all available to me in some minimal way.

This covers all the hardware vendor objections (even if you just choose one Tier-One vendor, they all have programs for just this, and we all have experience with all of them), and still allows you to optimize for the OS and hardware like mad.

The Open Source Problem
But what if you want to run things all open-source on your own hardware? Then have the project, rather than distributing an RPM, distribute a full bootable image that you load. Or have your favorite Linux/OpenSolaris distribution come up with an installer that rather than building a general purpose system, builds a system dedicated to just running that particular application (e.g. Anaconda, but custom RPMs and boot images).

I think this is probably going to start being a more common paradigm going forward, and the move to cloud computing is probably going to take things farther as people use something like CohesiveFT or some other mechanism to grab VM instances or instances for their favorite cloud computing provider for infrastructure software.

Why I don't think you're going to want a Cloud Computing solution to this in particular (e.g. an EC2 cluster) is that something like MOM done right in Central Distribution mode is pretty network topology sensitive: it doesn't make sense to virtualize it or run it as a VMWare image. It wants to be close to the hardware. Run it in another type of workload, but don't try to push tens of thousands of persistent messages through it per second; it's never going to work. But the same general approach has a lot of merit for other types of projects.

Conclusion
I think there's a big option for someone to provide an AMQP appliance for the centralized distribution case and push it heavily. There's no reason why I should be running what essentially amounts to a persistent network appliance on generic off-the-shelf hardware and operating systems, and there's no reason at all why I should have to do your performance optimization yourself.

But again, AMQP is the secret ingredient: I'll trust an appliance running custom software to provide a standard network service. I wouldn't like an appliance running custom software for a custom protocol.

AMQP, Exchanges, and Routing Thread on rabbitmq-discuss

For those of you who have been following my various ramblings on AMQP, specifically on the whole Exchange-Routing semantics (which overflowed into Ben Hood's blog), on Ben and Alexis' request I've started a thread over on rabbitmq-discuss to try to hash things out in a more direct format rather than duelling bloggers. If you think you have something to add, or just want to watch things unfold, I recommend you take a gander over there.

And if nobody takes me up on it, I'll just assume that I've won and the new AMQP spec will have everything I want in it, including a gift certificate to Millie's Cookies. And beer.

Disclaimer: I don't actually use RabbitMQ, my kind hosts just thought it would be as good a place as any for a public MOM-protocol-routing-semantics smackdown.

RabbitMQ talk at LShift Last Night

I attended the AMQP+RabbitMQ talk last night at LShift, and it was a pretty good time. Apparently the talk was largely a rehash of the talk that Alexis & company did at Google UK a few weeks ago, and was pretty interesting.

The basic pattern was:

• Talk about MOM (messaging is good) and AMQP (AMQP is also good)
• Talk about RabbitMQ (RabbitMQ is good) and Erlang/OTP (they're also good and ideal for implementation of an AMQP broker)
• Demonstration of some cool stuff you can do tying it all together

To me the biggest parts that were new and quite interesting and well worth the time was the latter half, where they showed how a lot of the features in Erlang are ideally suited to developing an AMQP broker in under 5000 lines of human-written code (a little more is auto-generated from the AMQP spec itself), and some of the neat things that Erlang gives you as a platform for maintenance and monitoring.

Specifically, in case you haven't been looking at it yet, Erlang gives you a natural model to represent a lot of concepts in building large asynchronous systems. Since there's a lot of processing that goes on in message-oriented systems which is by its very nature asynchronous, it's pretty easy to map that onto Erlang in a pretty trivial way, which drastically reduces the amount of code that you need to write yourself.

Since the major concepts you're working with (Exchanges, Queues, Sockets) map directly to low-level Erlang actors^H^H^H^H^H^Hprocesses, Erlang gives you a really easy way to automatically introspect the state of those things, since they're being mapped to processes. This for a Java guy was pretty cool, because essentially you can introspect into the state of a running Erlang system and see in one fell swoop a snapshot of a process which shows you the equivalent of (for Java):

• The stack trace
• Values of all local variables across the stack
• Values of all elements in the heap directly addressible from the bound variables

Doing that even with the new tools available in Java 6 and Terracotta is nowhere near as easy to do, and it was pretty darn powerful to witness.

And then, down t' pub.

One of the most interesting things to me was that quite a few of the people there really weren't part of the converted trying to hear things to make them feel all happy about themselves. Rather, there were a lot of people who aren't currently using MOM systems (or have basically done stuff that's core to MOM themselves, because the basic concepts are so intuitive that if you don't know what's out there, you'll end up writing your own half-baked MOM implementation yourself to do almost any non-trivial asynchronous system) who are eager to look into them now.

That's pretty nice to hear, and I think is one of the major medium-term benefits of AMQP: expansion of the overall MOM space from one dominated by MQSeries which people use because they have to, not because they want to, to one where people actively choose to use MOM because it's seen as a benefit rather than a hurdle. So AMQP really will involve growing the space, rather than cannibalizing it. And that to me is a great thing.

IPC: Where My Queues At, Yo?

Updated 2008-10-20: Adrian Found Where My Queues At. Unbeknownst to me, they were there (somewhat all along) in POSIX Message Queues, more in the comments. I've changed this to be part of LazyWeb, because I really didn't know that they existed and haven't actually seen anybody using them. M4D props to Adrian.

Updated 2008-10-20 #2: POSIX Message Queues are close, but not quite it. I'll be doing another post on this, but turns out POSIX Message Queues aren't quite what I think mailbox semantics are, and some differences turn out to be key. Also, they've only been around since Linux kernel 2.6.6, which would explain why I've not seen them in more widespread use.

Not that long ago, my firm was interviewing someone from the Czech Republic who had been working in Frankfurt for the Deutsche Boerse, and some interesting stuff came out of that. Not the least interesting thing was that their entire infrastructure was still based on OpenVMS, and it was working extremely well. The secret? Queues. Or, rather, Mailboxes.

What a lot of people know about VMS, aside from the fact that it's Old and Not UNIX, is that it spawned Windows NT (WNT == VMS+1). What they don't realize is that it's actually quite a reasonable operating system that's kinda gotten a bad name because of the WNT thing, and because it's Not UNIX, but I want to address the thing that it's really gotten right, and that's queuing as an intrinsic form of IPC.

As I understood from the interviews, Deutsche Boerse uses this pretty extensively to distribute work: work is pulled from mailboxes, posted to other mailboxes, in a pretty standard modern MOM pattern. The recipients of messages may be on the same machine, or on another machine. They don't know; they don't care. Sounds like what you'd use an MOM system for, right? Only they don't use an MOM system, they use their operating system.

A Mailbox in VMS essentially forms the basis of a fully asynchronous queuing system. Applications can define mailboxes, to which you can publish messages, and the recipient will be notified at some point in the future that a message is available. And because VMS has a lot of clustering facilities, mailboxes can either be on the local machine or on another machine, and because it's all fully guaranteed delivery (within reasons of course), you have full fire-and-forget semantics.

Contrast all this with your UNIX forms of IPC: Pipes, Files, Signals. What do the former two have in common? They're either synchronous or near-enough in that they involve polling. Signals may be asynchronous, but they can't really carry a payload: they're just a number. These are enough to form the basis for almost all forms of common IPC, except for fully asynchronous ones. What do you do in a UNIX IPC model if you want to send a message and at some point in the future make sure that something else picks it up? Put it in a spool file? That seems pretty clunky, even today.

Sure, you can expand the UNIX model to involve asynchronous IO, but that's really just an asynch vaneer on top of traditional synchronous IPC. It doesn't change the fundamental basis that your basic units of system-system IPC are fundamentally synchronous. This all smells funny.

Given that there are a lot of AMQP people who read my blog apparently, where's the topicality? How about this: why is it necessary that I even have to use something like broker-based MOM when the operating system could realistically do this for me? Why is this all so resolutely in user space when operating systems were doing this decades ago? Why do I need to communicate using sockets (another system-system synchronous IPC system) to another process to do basic message-based IPC?

Why can't I take all the various implementations of this pretty basic concept (fire-and-forget, guaranteed messaging, asynchronous notification, single consumer delivery) and pull them out of their various programming-language (Scala, Erlang) and application-semantic specific models (JMS, WCF), and pull them up a level? Why can't I get this out of my operating system? Why can't we just break out of the UNIX model of decades ago?

Maybe we've accepted that the micro-kernel model of operating systems has really won. And by that, I don't mean the Mach-model of operating systems, but, rather, the model that system-level APIs are pretty much fixed in stone by history, and won't be expanded (yes, they might be expanded in the particulars, but they're not going to be significantly expanded in the general concepts that people are willing to defer to the kernel to do). People do everything they have to in user-space.

But something to me is wanting. I think this really simple form of IPC, which is so insanely beautiful that people are willing to program in actor-based concurrency systems like Erlang just to get it, really should be given a second thought. Maybe not in kernel space, but definitely as a standard that you can assume will be around on any system on which you program.

And the interviewee? A (hopefully) happy employee of my firm. We hire quality when we can find it.

Interop News on AMQP: Huh?

Sorry, I don't usually call out at other writers, but this article makes utterly no sense whatsoever. I mean, m4d pr0p2 for giving a call out to AMQP, but WTF?

Some random stuff in there:

• Right off the bat, the author (Jeff Gould) calls out Rendezvous as a guaranteed message delivery system. But that's actually the opposite of Rendezvous's raison d'etre. Rendezvous is best as a super-low-latency tick distribution system where its default best-effort semantics are good enough, not a guaranteed message delivery system. That's because it's an inherently distributed system. Yes, there are advanced features of Rendezvous that make it nearly guaranteed, but they're pretty ugly to be fair.
  
• He conflates bridging with heterogenous client connectivity. In general, the JMS API-driven approach that most vendors do with interoperability actually is good enough, because there's going to be something that has to bridge them no matter what, and many different JMS-based systems allow for automatic bridging from a broker to another system quite easily. Just because the code wasn't all written by one vendor (the broker by Vendor A, the client library to interact with the other broker by Vendor B), it isn't a massive problem.
  Furthermore, most of the time when you have such an interop problem, someone has to write some bloody code to actually implement all the tiny little differences between the two systems (such as ID issues: if both companies are using the same software, there's probably a conflicting ID namespace somewhere that requires translation). That involves a bridge.
  Imagining that there will be a world where two companies running two ever-so-slightly different versions of the same software package will be able to just exchange messages willy-nilly and all will be well and the programmers can all just go home is rubbish.
  
• He assumes that AMQP is a replacement for the two major products that he directly calls out to: IBM MQ (just calling it that since I will always think of it as MQSeries) and Rendezvous. And then refers to Sonic. But here's the thing: none of the three has almost anything to do with each other, and each is used in pretty different spaces.
  • IBM MQ is largely used in older systems (think flat-out legacy). It's big, it's slow, it runs on mainframes, and I really don't know of a single person doing work similar to me that's actually deployed it out of choice in ages. And you know what? AMQP will do absolutely 0 to that market. It's very legacy and very entrenched and IBM has absolutely no incentive to change it and never will.
  • Rendezvous is a completely decentralized MOM system ideally suited to low-latency best-effort systems (tick distribution, enterprise system monitoring). It does something completely different to IBM MQ, and you would never replace it with MQ. And it has competitors (29 West is my favorite to give a call-out to).
  • SonicMQ and other pure-play JMS vendors (Fiorano, Tibco EMS, ActiveMQ, ...) have a broker-based model with guaranteed delivery and best-effort built in. They're both a superset of the IBM MQ and Rendezvous systems, with some special sauce out of having at least a marginally intelligent approach to broker-based middleware worked in.

The point of my third bullet up there is to say that JMS attempted to define an semantic layer inside the application that you can use to develop broker-based messaging. You couldn't actually use Rendezvous underneath the JMS layer because you'd end up having to throw a broker in the middle (Rendezvous can't support the JMS semantic model), and then you'd just end up with Tibco EMS at the end of the day, but you can do best-effort delivery in JMS if you're willing to put a broker in the middle.

AMQP does a similar thing at the network protocol layer: it defines a semantic layer that assumes a centralized broker-based middleware system and wraps a network protocol around that. It really competes more with the JMS pure-plays than either IBM MQ or Rendezvous. But I don't think it as a protocol provides a compelling reason for people who actually know and want to be on a distributed system like Rendezvous to shift, and people on IBM MQ are there because they have to be.

None of this is to say that the core of his message isn't correct, which is that AMQP is a great concept for a variety of reasons, but his introduction is setting himself up for something he didn't need to.

I still stand by my original point:

• Vendors cannot support every possible programming language, OS, chip, or compiler with a closed/proprietary client driver.
• Programming language experts are superior at doing that and can provide open source versions.
• This naturally and inevitably leads to possible interoperability gains.

That's the eventual victory of AMQP.