Thursday, February 26, 2009

Let A Million Brokers Bloom

To me, one of the most promising things about AMQP is that it actually has the potential to radically expand the amount of feature-based broker differentiation in the MOM space, by eliminating a lot of the "grunt" stuff that you have to go through in order to define a new broker model.

Right now, if you want to do something interesting (meaning not commercially supported) with asynchronous message-oriented communications, you have to:

  • Define your basic messaging model
  • Define the wire representation of that model
  • Define your protocol
  • Write up the server-side handling of the protocol
  • Write up the client-side handling of the protocol for each programming language you need to support
Only when you're done with all that do you actually get to do "the fun stuff," which is to write up the unique piece of functionality that meant you wanted to start on the project in the first place.[1]

Consider a particularly germane example. In my current day job I'm working for a firm (and if you're reading this blog you've heard of the firm) that has a proprietary MOM system with (what I can see as) one USP.[2] Aside from that one USP (which I'm not going to identify), it's just a simple pubsub-based MOM broker. But nobody in the market has that feature, so they can't go with a commercial vendor. And there are facets of that USP that means that no commercial vendor is likely to add exactly that feature to their product.

But in order to get that one USP feature, they have to face every single other hurdle above, and constantly deal with refining every single bit over years and years of development. None of which is about that one USP feature, it's all just plumbing. Everything other than implementing that feature is a waste of valuable development effort.

The kicker? That USP is just a custom Exchange type in AMQP 0-X terminology. Nothing more, nothing less.

And that's the beauty of AMQP.

Once we reach the promised land, someone like my current cash money provider could have just grabbed an existing framework (all the AMQP client work and server-side protocol handling in its language of preference), and implemented the custom exchange (or broken existing ones, or got freaky with queue implementations, or whatever it wanted): no need to reinvent the wheel. AMQP acts as the standardization of all the bits that you don't want or need to care about to provide custom message broking behavior, allowing you to focus all your time and attention to the parts that you really need to implement yourself.

This makes the barrier to entry for unique MOM functionality only the cost to implement the unique functionality itself; you don't have to face the hurdle of implementing everything else required for asynchronous message-oriented communications to showcase or provide one single feature.

I consider this to be the messaging version of the story behind the Eigenbase project: a framework for writing custom brokers. A few people I know (big shout out to my homies John Sichi and Julian Hyde[3]) are excellent at writing "interesting" databases (with special sauce for special contexts). However, doing that from scratch each time means that you're constantly reinventing/rewriting the same old tired stuff: a SQL parser, a transactional data store, a wire protocol, blah blah blah. So they started the Eigenbase project as a way of taking care of all that cruft that you just Have To Have so that they could focus on the stuff that's special about Their Database.

Are there equivalently architected Framework Brokers yet for AMQP? Not that I'm aware of (please pipe up in the comments if you think I'm wrong). If nothing else, in the message broker/routing space, there are a lot of things that you might want to do to speed up adhering-to-standards mode that break the general programming model you'd want from a framework-based system. That leads me to believe that just as MySQL didn't go framework-based to become Eigenbase, we'll probably see a set of AMQP brokers focusing on ease of programming custom variants, just like Eigenbase is designed specifically around programming custom variants in the database space.

And at that point, I think we'll start to see a lot of new and interesting implementations, some in-house and some exposed to the world.


[1] Alternatively, you can horribly abuse an open source project, but that's going to do nothing but confuse everybody involved (including yourself) and leave you liable to abandonware if you rip out the insides of a system that you're replacing key components on (so if your open source basis isn't designed as a framework, and you replace some of the key message handling parts, the moment the developers change the internal processes you're effectively stuck).

[2] Unique Selling Proposition. But you knew that of course.

[3] Broadbase In Da Hizzy!

blog comments powered by Disqus