RabbitMQ/AMQP OSGi Integration

I had a chance to meet up with Neil Bartlett today, and got to talk to him about his RabbitMQ OSGi integation (note that I believe that everything he's done thus far is actually applicable to anyone using an AMQP broker that is compatible with the RabbitMQ Java client, so I'm treating it as a generic AMQP/OSGi module). While it's a work in progress, it's incredibly cool. It allows you to:

• Declare production/consumption parameters (exchanges, queues, hosts, credentials) in a declarative way;
• Have your OSGi container connect the dots so that your declarative bundles automatically match up with concrete connections to your AMQP broker of choice;
• (If you're using Equinox) Use the OSGi shell to interact with the RabbitMQ/AMQP client library.

The first two parts were the types of things that I had worked on in the past (having a declarative way to specify messaging endpoints, down to the broker connectivity level) so that you could change them at execution-time. The third was the real WOW-worthy point.

Assume that you're coding against an OSGi runtime. Assume that you're working with bundles, and dynamically loading and unloading them as you're compiling your code (meaning that you're probably at an osgi> prompt). Assume that want to see very quickly whether your code is interacting properly. Now you can easily setup your bindings and publish and consume messages as your real code is in flight, and see the effects in your debugger.

How amazing is that? Imagine the human cycles that you'll save being able to do all this from one area, that you're already familiar with?!?!

This is a work in progress, and hopefully Ben Hood and I had a chance to give him some places to go from here, but I think this is exactly what the promise of AMQP offers: the chance to do amazing things based on not having to support a million different client libraries, but to code against a standard and make great stuff happen.

If you're using the RabbitMQ Java Client and any type of OSGi container, I highly encourage you to take a look at this and get involved. I see great things happening here.

2009 Predictions

Just a quick set of predictions for 2009. I'll revisit nearer the end of the year to see how I've done.

Messaging Breaks Out
Right now asynchronous systems, particularly ones using message oriented middleware, are still pretty fringe. However, even though I disagree with the technology choice, techniques like XMPP are starting to bring asynchronous communications to the masses. I predict that the extension of XMPP, the forthcoming standardization of AMQP, new infrastructure being developed (like RabbitMQ, OpenAMQ, and Qpid), and extension to new programming languages and application execution environments will all drive more and more developers to finally end polling for updates.

Cloud Will Become Less Buzzy
Right now Cloud/Utility computing is really too buzzy to make heads or tails of, and I think it's suffering from that: when you say "Cloud Computing", it means both nothing and anything. I think 2009 will be the year that people start to solidify what works best in a utility computing space versus a local hosted space, and where hosting providers get mature enough and technology becomes mainstream enough that there's an assumption of utility over local hosting.

Java Will Stagnate
I predict that Java 7, if it even hits by the end of the year, will be pretty anemic and, quite frankly, lame. There will be some interesting enhancements to the JVM, but that's pretty much all that's going to be noteworthy.

C# Will Over-Expand
The entire Microsoft ecosystem will expand and grow beyond the ability of typical Microsoft stack developers to keep up. We're already seeing it with WinForms vs. WPF, LINQ vs. ADO.NET Entity Framework, and whole new technologies like WCF. I predict by the end of the year the Microsoft ecosystem has become so bewildering that while leaders in the community push for more change, day-to-day developers push for a halt to just absorb what's been happening.

Non-Traditional VM Languages Will Break Out
Right now most developers in enterprise systems are coding against the "core" languages (C#, VB.Net, Java) against their respective VMs. This reflects in no small part the maturity and power of their underlying runtimes (the CLR and JVM respectively), but the leading edge of the communities have already started to explore other languages that offer concrete advantages (F#, Scala, Groovy) on the tight runtimes that are already available. This will be the year that those actually break out of the leading edge and into mainstream use.

No-One In Social Networking Makes Money
I still think, by the end of 2009, there won't be a player in the social networking space that is cashflow positive, and I don't think it'll be by choice. I think the technology is so disruptive that we still haven't seen the "right way" to monetize it yet.

Sun Radically Restructures
Sun can't keep burning through cash the way they have been, and they can't continue to have such a chaotic story. At some point in 2009, Sun will relatively radically restructure itself in a bid for survival. Hopefully they'll have read my analysis (part 1, part 2). At the very least, they'll change their ticker away from JAVA.

I think you'll note that all my predictions are fuzzy. That's so that in December, 2009 when we revisit them, we can determine that I was partially right on nearly all of them.

RabbitMQ talk at LShift Last Night

I attended the AMQP+RabbitMQ talk last night at LShift, and it was a pretty good time. Apparently the talk was largely a rehash of the talk that Alexis & company did at Google UK a few weeks ago, and was pretty interesting.

The basic pattern was:

• Talk about MOM (messaging is good) and AMQP (AMQP is also good)
• Talk about RabbitMQ (RabbitMQ is good) and Erlang/OTP (they're also good and ideal for implementation of an AMQP broker)
• Demonstration of some cool stuff you can do tying it all together

To me the biggest parts that were new and quite interesting and well worth the time was the latter half, where they showed how a lot of the features in Erlang are ideally suited to developing an AMQP broker in under 5000 lines of human-written code (a little more is auto-generated from the AMQP spec itself), and some of the neat things that Erlang gives you as a platform for maintenance and monitoring.

Specifically, in case you haven't been looking at it yet, Erlang gives you a natural model to represent a lot of concepts in building large asynchronous systems. Since there's a lot of processing that goes on in message-oriented systems which is by its very nature asynchronous, it's pretty easy to map that onto Erlang in a pretty trivial way, which drastically reduces the amount of code that you need to write yourself.

Since the major concepts you're working with (Exchanges, Queues, Sockets) map directly to low-level Erlang actors^H^H^H^H^H^Hprocesses, Erlang gives you a really easy way to automatically introspect the state of those things, since they're being mapped to processes. This for a Java guy was pretty cool, because essentially you can introspect into the state of a running Erlang system and see in one fell swoop a snapshot of a process which shows you the equivalent of (for Java):

• The stack trace
• Values of all local variables across the stack
• Values of all elements in the heap directly addressible from the bound variables

Doing that even with the new tools available in Java 6 and Terracotta is nowhere near as easy to do, and it was pretty darn powerful to witness.

And then, down t' pub.

One of the most interesting things to me was that quite a few of the people there really weren't part of the converted trying to hear things to make them feel all happy about themselves. Rather, there were a lot of people who aren't currently using MOM systems (or have basically done stuff that's core to MOM themselves, because the basic concepts are so intuitive that if you don't know what's out there, you'll end up writing your own half-baked MOM implementation yourself to do almost any non-trivial asynchronous system) who are eager to look into them now.

That's pretty nice to hear, and I think is one of the major medium-term benefits of AMQP: expansion of the overall MOM space from one dominated by MQSeries which people use because they have to, not because they want to, to one where people actively choose to use MOM because it's seen as a benefit rather than a hurdle. So AMQP really will involve growing the space, rather than cannibalizing it. And that to me is a great thing.

RabbitMQ at Google UK

Turns out the RabbitMQ guys have been busy, and Alexis and company went over to Victoria and gave a talk to the Google UK guys. Can't wait for the video to show up!