Confluence, iSCSI, NetApp, Flexiscale, Fail

We've been hosting the FudgeMsg website using Confluence (and mad props for the free Open Source license by the way!) on a VM hosted by Flexiscale. We chose Flexiscale for the following reasons:

• Confluence is ridiculously tricky to cluster, meaning that you can't benefit from the scale-out capabilities of the Amazon EC2 model. (Edit 2009-12-24 - We're not attempting to cluster it; this statement is to indicate that because it's so tough to cluster, we're not trying to. Confluence is thinking we might be, and crashing as a result).
• Getting your AMI+EBS+ElasticIP for such a single-point vendor app is very difficult and time consuming, again meaning that the Amazon EC2 model isn't ideal.
• We didn't want to go for dedicated hardware for a low-volume application stack.
• We don't have a stable enough internet connection to host the application stack ourselves.

We thought we were pretty darn clever to be honest, but then we noticed that things started going wrong. In particular, we started getting this error a lot:

We were running a relatively complicated Java setup (jsvc to run as chrooted user, on Tomcat, with multiple web applications running), so we thought we had done something wrong. After all, the first rule in software engineering is always, always, assume you have caused the break.

We were wrong.

What we've since found out is that the entire Flexiscale model is that your VMs have effectively no local storage whatsoever, and everything is iSCSI hosted off of a NetApp cluster. That means that your storage is fully persistent, and survives migrations of your VMs across physical hardware. Which is a great concept when it works.

Except that the Flexiscale NetApp cluster is borked. Essentially, sporadically the iSCSI services will completely shut down. Sometimes for a second, sometimes for 4 hours. Your processes will still be running, but if they attempt to hit disk for any reason, they'll hang and ultimately timeout. Your processes are still running, but they can't talk to disk.

In the case of Confluence, which was actually trying to talk to PostgreSQL, which itself was trying to talk to disk, Confluence detects the complete timeout hang of PostgreSQL as a cluster violation, and puts itself into crash mode.

You want to know the best part of this? When you're in this state, there's nothing you can do. You can't SSH into the VM, because it can't read the password file to let you in. The Flexiscale tools won't even let you hard bounce the machine. And they won't tell you when it's back up directly, so you just have to keep trying until you can finally get into the VM, to restart your servlet container.

This has caused no fewer than 20 instances where Confluence has died for us, sometimes lasting hours until we can actually recover the VM, and twice now in 2 days. It makes us look like morans who can't even run Confluence, much less guide development of a message encoding system.

So until we manage to get off of Flexiscale (haha, can't even get in to back up the data at the moment), if you see that error when going to the Fudge Messaging website, now you'll know why.

There are two morals of the story:

• If you get a Non Clustered Confluence : Database is being updated by another Confluence instance error, and you know that's not the case, make sure your storage is responsive;
• Don't host anything on Flexiscale until they sort their hardware out.

2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!

Rise Of The Appliances

I've posted before [1] on my desire for someone to sell me an AMQP Appliance, and while nobody has yet (probably the fact that 1.0 is in the final stages of getting finalized is impacting this), I'm encouraged that somebody will.

Memcached Appliances Ahoy!

I'm particularly encouraged by the last week's frenzy of Appliance Startup Announcements:

• Schooner Information Technologies coming out of stealth with MySQL and Memcached appliances.
• Gear6 launching their own Memcached appliances.
• Virident launching their MySQL and Memcached appliances.

I've read more about the Schooner appliances than the other two [2], but they appear to be roughly what I said my ideal AMQP appliance would be: Tier-1 hardware, linux kernel, custom software, heavily tuned to the particular bits of hardware in the box itself. I'm a little surprised at the $45k price tag, but I suppose it's good to hit the high-end of the market first and work your way down over time. $45k would be justified with the amounts of RAM and SSDs in the Schooner box, but I'm sure they can scale things down to less Flash and less RAM and still have a reasonable margin.

Open Protocols Make It Possible

What makes this possible (particularly that everybody's targetting the currently very-much-in-vogue Memcached) is that there is a standard protocol. While the memcached protocol isn't a de jure standard, it is a de facto standard, and one that at this point is unlikely to change given the amount of software doing interesting things with it as a protocol.

It's only a matter of time before we see the same types of innovation in messaging or anything else for that matter if you can define a stable network interface to it. If you can define an open protocol (whether de facto or de jure), you can build an appliance to serve it. And it would probably be a better quality of service than rolling your own on top of the same hardware and OS combination. Systems which are heavily I/O bound (messaging, databases, network caches) benefit greatly from advanced tuning of the entire stack from hardware through to application code. Thus, they make sense to optimize holistically, whether you're using general purpose hardware or specialized hardware. Appliance vendors can do this. Generic software vendors have to rely on teaching their customers how to do it.

Cloud Hardware Combination

The one problem with this move going forward is that you can't do it with Cloud Computing, because you don't have access to the physical environment and can't put your own hardware in place. And there's a lot of stuff you can't do as a result:

• Appliances like Schooner, much less ultra-high performance messaging brokers like Tervela or Solace
• Direct network connections to the proprietary or specific network of your choice.
• Storage optimization through the use of any type of Flash acceleration
• Anything requiring significant local bandwidth on a single box

That leads me to question whether the current vogue of completely ignoring any type of CapEx whatsoever is actually optimal for your core infrastructure. For example, if you can spend $5k on a machine to do your MOM load perfectly, or your memcached big and fast, or whatever, how much extra are you going to have to spend in overall VM slice hours to equal that performance (for example, are you going to have to run 10 VM slices running memcached and an extra 5 app server slices because you can't hit the performance of a single box)? This is entirely possible for something as memory bandwidth intensive as memcached, or as network bandwidth intensive as an AMQP broker. Where's the crossover point?

The bright spot here, though, is Open Protocols. I know I keep banging on about it, but where this all merges together brilliantly is that you don't have to have your own appliances or hardware in your Cloud Computing vendor: you can just talk open protocols and allow the utility provider to sell it to you. You want memcached? They'll sell you an IP/port that has access to X gigabytes and guaranteed response type of Y for $Z/[request|month|core time]. Open network protocols thus act as the disconnected version of open APIs in the software development world: you can swap out the underlying implementation without substantial changes to the rest of your environment.

Furthermore, working with Open Protocols in this case insulates you from PaaS lock-in. If your utility provider is providing Caching As A Service using their own custom, proprietary protocol, in order to move to a different PaaS vendor you'll have to change your app stack in some pretty substantial ways; if it's both sides providing an equivalent service (meaning protocol and semantics), you can migrate with minimal efforts (particularly if it's a transient service).

Footnotes

[1]: Meta-note: haven't been posting recently, even with the AMQP F2F and Solace announcements, because I've been busy working on some big stuff that I can't talk about at all yet. If it happens, I'll let my adoring public know.
[2]: Because Red Point, which backed Radik in the past, funded them in their Series A. I like to see what they're investing in.

Bamboo 2.2 - This is a Game Changer

Earlier this week, Atlassian announced Bamboo 2.2 had been released. Due to qCon pressures I hadn't had a chance to do a writeup on it until now, but I definitely wanted to mention it. It's a doozy.

The headline feature in 2.2 (and this is so big that I'm shocked that they didn't bump up to 3.0) is Elastic Bamboo. In essence, this allows you to use Amazon EC2 instances as your remote build agents, seamlessly. That's huge. Really huge.

Let's assume you're a developer trying to work out your build and test strategy. Your options right now are to:

• Run all your build agents under the Bamboo process. Not an option if you have to work with a heterogeneous environment, so I'll consider this the Hobbyist approach.
• Stick build agents on your various development machines as-and-when they match the exact configuration that you want to be able to test under. Doing this, though, will unpredictably mess with the machine's performance as Bamboo decides to take away all the resources to run a build-and-test.
• Push VMWare deep into the organization so that you can do your own VMWare instances to match the configurations you want to test against. Doing this, though, means that you have to have these instances up and running pretty much all the time, whether you're testing against them or not.
• Buy hardware just for testing. Uhm, haven't you heard? Capex is SOOO 2007, particularly for development infrastructure. Spend your Capex on 30" displays, yo.

Those are all fine options if you have friendly IT people who love doing things for developers as fast as possible. I seldom run into those. Elastic Bamboo gives you the option of running your build agents in EC2, starting them up to run a build and shutting them down when the build is done. Just think about that. You now have the option to have unlimited build-and-test configurations, completely outside the control of your IT department, and only pay for them when you're actually testing against them! Have a configuration you only supported for an old release that only a few customers are using? No worries, leave the instance in place, and it'll only launch when you change that really old branch. Have a configuration that only one customer is stupid enough to use in production? No worries, just leave the instance configured and run it every week or two. You now have no excuses for not exhaustively testing against your matrix all the time (I'm talking to you, Sonic!).

Along the way, they ended up realizing precisely what I realized when I tried to run build agents in between London, New York and Hong Kong, which is that Bamboo is a hog when it comes to log and artifact shipping over high latency links (and often low latency ones as well). They've fixed that, as if you're developing in Sydney and the closest AWS site you have is US East, you can't rely on Lan speeds anymore. All in all Atlassian resolved 5 issues that I was the original filer for [1]. That's a great track record, and I don't know very many commercial software companies that I could find that out for in the public record. Yet another reason I shill for Atlassian constantly. [2]

In addition to the whole "game-changing" thing, this is also extremely cool from a utility computing play. You probably could have done this yourself already (subject to the artifact/log shipping problems). But they're doing it for you. I think we're going to see that more and more from commercial software vendors, as they add support for major cloud computing platforms into their in-house software networks. Don't be surprised if you start to see anything that could benefit from cloud CPU resources start automatically coming AWS enabled.

I got cornered down t' pub at qCon by The Build Doctor and gave an only slightly inebriated rant about Electric Bamboo. For those of you who don't know what I look or sound like, here I am in all my gesticulating glory:

Major Caveat: My old employers, who are nothing if not early adopters for anything, upgraded an existing infrastructure with a lot of MSBuild, and it's not going particularly well for them. It appears that the officially closing off of BAM-1413 (MSBuild support [3]) actually broke the MSBuild support that was part of the NAnt plugin (which already shipped with Bamboo and supported MSBuild to great success). This is causing massive problems for the dozens of MSBuild-based projects at my old cash money gig. When it gets resolved I'll update.

UPDATE:: If you upgrade to the newest (2.1.9) NAnt plugin it'll fix the problems.

Minor Caveat: The Pre-Post Build Command Plugin doesn't work with 2.2. My old coworkers hacked a local version to get it to work, but there was an API change that broke the plugin.

The worst thing? I don't even get to use Bamboo at my current cash money gig. We're stuck with Electric Commander [4]. Sigh.

Footnotes:
[1]: For those counting: BAM-3177, BAM-1831, BAM-2989, BAM-2852, and BAM-2983
[2]: Fair disclosure: after I published that original mega-shill, they sent me an insanely great laptop bag, which is now my daily turtle shell. Moral: suck-up to cool companies and they send you great swag.
[3]: Oh, look, there I am again in the comments. I'm everywhere in this release.
[4]: Product by company founded by John Ousterhout, inventor of Tcl. I shall assume my gentle readers all have a strongly held opinion about Tcl one way or the other and say no more. But EC is no Bamboo, even though it's been around a lot longer. Or maybe it's just badly run here.

Designed for Deployment Reliability

I was reading an interesting article on Designed for Reliability on The Daily WTF [1]. And then I started reading the comments. I think a lot of people don't get it.

The real story here (at least to me) is that because the system is so reliable, deployment rules are never done and enforced assuming that it might not be. It's a classic problem, because good deployment rules are hard. If you can keep things up and running, you're better off treating the problem as just maintaining the system, but eventually that just plain old fails. It always will. [2]

Ultimately, any system that hasn't had constant redeployment baked in is going to face this exact problem: the machine exploded, OMFG, how do I get it up and running? For one of your larger distributed systems, could you do that particularly easily? More importantly, have you actually tried?

My contention is this: if you don't design your processes such that you always have a reliable deployment story, your super-expensive hardware/system is worthless. Things always, eventually fail. You have to be ready for it.

Annual Power Down

One practice that I think is useful in a variety of ways is the annual power-down. In some areas of London (and in this I don't know whether it's a requirement all the time, but it's happened to me at two different locations, one in the City and one in Westminster Council, so it appears to be widespread), you have to switch off all electrical power to your office block once a year so that it can be tested and inspected. And that's absolute: no UPS, no diesel generators, you really do have to switch the machines off.

Luckily for all, this is scheduled. Beforehand, everybody who maintains a system that's run in the affected data centre scrambles to make sure that the stuff they support is likely to get up and running again without them doing any work. Mostly this is because the power's gonna come back on early Sunday morning, and if they want to party the previous night, they better make damn sure they're not getting woken up at 9:00 am Sunday morning. Startup scripts are checked, internal consistency of the system is checked, checklists are updated, everybody does their part.

It still won't work of course, which is why there are so many people on call and ready to help nudge their particular part of the world back into operation (if nothing else, hardware that runs full-bore 364 days of the year often picks power-up as a time to fail, so make sure you've got a lot of spare disks and power supplies at hand), but the discipline forces you to, at least once a year, make sure that you can get up and running before Monday morning.

Thus if something catastrophic happens, your scripts are on average 6 months behind (and we all know that's worst case: everybody pays attention to the lessons learned for 3 months after the power down, and everybody starts to get ready 3 months before hand, so worst case is you get caught right in the middle well of apathy), but you should still be able to get up and running with a minimum of fuss.

In this case, the annual power down is part of your methodology to ensure that you have a reproducible deployment system.

Can The System Help You?

So let's say you have a system that is designed to make it extremely easy to make runtime changes that would affect your ability to bring the system up and running again in the case of catastrophic failure. What might you do to help fight against this tendency? I think the only solution here is one of two cases:

• Run every change through an external system that keeps track of what is done and potentially allows replay. Note that for this to work, every change (no matter how minor) must go through The System; this essentially prevents developers or support staff from being able to log into the box at all, because everything has to be tracked (though you could maybe get around this these days by just having some type of logging SSH shell or something).
• Have the system self-maintain. In this case, the system maintains in some stable form (backed up disk for example) the exact configuration that it currently has, as well as perhaps the last N changes, so that it can be brought back to a consistent state quickly in the case of catastrophe.
  

The former has the advantage that the system doesn't have to know what's going on, the latter has the advantage that the system can be written to handle just this situation and not have to log unnecessary trivia.

The latter is actually easier than you might think:

• Your VM (in the case of Erlang/OTP) might write out the state of all directly launched processes (as opposed to software launched ones) so that it knows the location and state of all code used to kick off the system;
• Your container (like OSGi) might silently copy all bundles to a WORM location when loading so that there's an accurate record of what was running at any given point of time;
• Your OS (ZFS FTW) might snapshot the state of the system any time something interesting happens so that you can go back and recover gracefully.
  

The Cloud/Utility Computing Take

So given that I think most systems are ultimately moving to Utility Computing (in-house or out in teh Cloudz), does that help you or hurt you in this case? Help, majorly.

Here's a good test: If You're Ready To Run On EC2, You're Done. Whether or not you're going to deploy on EC2, surviving the semantics of EC2 means you're ready.

The rationale here is simple: the semantics of EC2 require that your system be able to deploy at any time. Sure, you can log into a machine and poke around with it, but you're designing for N-way scalability, remember? If you do that, and you don't fork a new AMI with your changes, you're screwed.

Now that means that you have to spend a lot of time working on your AMIs to make sure that for any given production iteration you've got a reproducibly deployable set that represents the state of the universe for that iteration. This is a problem, and I know that it's caused some people trying to integrate with EC2 no end of woe, because it adds a lot of troublesome, time-consuming grunt-work that they're not used to and can't be made to go faster easily. [3]

But by going through that effort, you know that your system is always ready to handle any given node getting cycled at any time. You know that you're consistently ready to deploy.

Think of it as the deployment equivalent of a test-infected culture:

• Testing and Cloud Deployment both institute costs (development costs and configuration costs accordingly)
• Testing lets you develop faster by making sure you're confident that your system is always working
• Cloud Deployment lets you support faster by making sure you can always bring the system up consistently in a crisis

See the parallels here? Up-front costs (development and configuration) lead to long-term reduction in those same costs by making the process consistent and reproducible.

So even if you're not going to deploy on anything other than your local hardware in a non-virtualized environment, pretending that you are can still reap some benefits. [4]

Conclusion

Design your systems and processes for reproducible deployment. If you don't, I guarantee you that the time you discover that your system can't come up without 3 days of hacking will be less than 24 hours before a major regulatory driven deadline.

Footnotes and Tangents

[1] Yes, this is an old article, but I started writing this a long time ago and am only now finishing my writeup. Yes, I have a whole lot of stuff that I think of a take on and never get around to finishing.
[2] In the Erlang/OTP community, this is definitely seen as an advantage: OTP makes it really easy to replace existing code in a running VM without taking anything down, so you can conceivably have a particular VM that's running for years. Read this post on Vinoski's blog for some classic Erlang insight.
[3] Systems like ElasticServer (from CohesiveFT) can help a lot here, because you can do your round-trip testing on VMWare on your workstation, and then just dump out the resulting images in AMI format rather than VMWare format when you're done.
[4] This isn't something that I've actually run a proper evaluation on, aside from designing distributed systems designed to come up and go down hundreds of times at a go for continuous-integration-driven testing. I'd want to run a formal evaluation of the benefits of virtualization configuration as opposed to test-driven rapid configuration to make this more concrete than just a rant.

Powered By Cloud Rundown

I had the privilege of chairing the Powered By Cloud Privacy, Regulation, and Security in Cloud Computing session last week.

While the snow shut down all of London, and the entire schedule had to be juggled to handle the complete failure of London's transport system, Tim Jackson and the rest of the crew did an amazing job putting together a two-day conference over the two worst weather days of the year.

I had the privilege of chairing the session with:

• Miranda Mowbray of HP Bristol
• Paul Massey of K&L Gates
• Brian Reale of Colosa, and
• Richard Sykes of Intellect

We each had our own take on the subject (Paul talked about regulation and international boundaries, Miranda talked about general concerns, Brian talked about licensing, Richard gave some general comments), and I think it was a pretty great conversation for all involved. I could tell by some of the questions at the end from the audience that this particular subject really is critical to the Cloud adoption community, and hopefully we shed some light on the concerns and the state of the art in the space.

My talk in particular was about how regulations, particularly in financial services, don't explicitly state that you can't use any particular Cloud-Sourced service, but, rather, that interpretations of principles-based regulation cause your internal or external auditor to disallow things that you really should be able to do. Basically, you can use Cloud Computing in IT, if you're willing to stand up to the auditors who are just focused on bureaucratic box-checking.

Slides (short, as we each had 10-15 minutes on our own):

Powered By Cloud 2009-02-03

View more presentations from KirkWylie.

Appearing at Powered By Cloud

Those of you who are in the UK and interested in Cloud/Utility Computing might find it interesting that I'll be appearing on a panel at the Powered By Cloud conference in London.

I'll be on a panel addressing a subject close to my heart, regulatory issues in cloud computing in finance. I'll be part of Session 7: Privacy, Regulation, Security and Cloud Computing (no worries, my name isn't there because I haven't given them a bio and face pic yet).

Key Details
Date:
2-3 February, 2009 (I'll be on day 2, 3 February 2009)
Location:
Lewis Media Centre
Millbank
London
SW1P 4RS

2009 Predictions

Just a quick set of predictions for 2009. I'll revisit nearer the end of the year to see how I've done.

Messaging Breaks Out
Right now asynchronous systems, particularly ones using message oriented middleware, are still pretty fringe. However, even though I disagree with the technology choice, techniques like XMPP are starting to bring asynchronous communications to the masses. I predict that the extension of XMPP, the forthcoming standardization of AMQP, new infrastructure being developed (like RabbitMQ, OpenAMQ, and Qpid), and extension to new programming languages and application execution environments will all drive more and more developers to finally end polling for updates.

Cloud Will Become Less Buzzy
Right now Cloud/Utility computing is really too buzzy to make heads or tails of, and I think it's suffering from that: when you say "Cloud Computing", it means both nothing and anything. I think 2009 will be the year that people start to solidify what works best in a utility computing space versus a local hosted space, and where hosting providers get mature enough and technology becomes mainstream enough that there's an assumption of utility over local hosting.

Java Will Stagnate
I predict that Java 7, if it even hits by the end of the year, will be pretty anemic and, quite frankly, lame. There will be some interesting enhancements to the JVM, but that's pretty much all that's going to be noteworthy.

C# Will Over-Expand
The entire Microsoft ecosystem will expand and grow beyond the ability of typical Microsoft stack developers to keep up. We're already seeing it with WinForms vs. WPF, LINQ vs. ADO.NET Entity Framework, and whole new technologies like WCF. I predict by the end of the year the Microsoft ecosystem has become so bewildering that while leaders in the community push for more change, day-to-day developers push for a halt to just absorb what's been happening.

Non-Traditional VM Languages Will Break Out
Right now most developers in enterprise systems are coding against the "core" languages (C#, VB.Net, Java) against their respective VMs. This reflects in no small part the maturity and power of their underlying runtimes (the CLR and JVM respectively), but the leading edge of the communities have already started to explore other languages that offer concrete advantages (F#, Scala, Groovy) on the tight runtimes that are already available. This will be the year that those actually break out of the leading edge and into mainstream use.

No-One In Social Networking Makes Money
I still think, by the end of 2009, there won't be a player in the social networking space that is cashflow positive, and I don't think it'll be by choice. I think the technology is so disruptive that we still haven't seen the "right way" to monetize it yet.

Sun Radically Restructures
Sun can't keep burning through cash the way they have been, and they can't continue to have such a chaotic story. At some point in 2009, Sun will relatively radically restructure itself in a bid for survival. Hopefully they'll have read my analysis (part 1, part 2). At the very least, they'll change their ticker away from JAVA.

I think you'll note that all my predictions are fuzzy. That's so that in December, 2009 when we revisit them, we can determine that I was partially right on nearly all of them.

The Forthcoming Ultra-Low-Latency Financial Apocalypse

This is a follow-on from my previous post on utility computing in financial services, and came out of a discussion between Nolan and me when he was helping review an early draft of the post (yes, I actually do review the blog posts, unlike my email rants). He brought up a good point which I think is going to result in the ultimate apocalypse of much of the ultra-low-latency space in financial services infrastructure.

Ultra-Low-Latency Background
There are a number of strategies to make money in the financial markets which are extraordinarily sensitive on the time it takes to deliver an order to an exchange based on incoming market data. Some of these are in the pure arbitrage space (such as an equity which trades in euros on Euronext and pounds on the LSE), but the vast majority of them are more complex (a trivial example: Cisco is part of the NASDAQ index; when the NASDAQ index moves 1 point, index funds will need to buy Cisco to keep up with the index, so buy Cisco if the NASDAQ moves up a point), to the point that you can't possibly describe them in this type of format. (Disclaimer: I've never worked in an ultra-low-latency trading environment; this is entirely based on conversations I've had with those who do).

These guys specialize in minimizing the time it takes for them to get their orders into the exchanges: if they don't get there before any other company trading using essentially the same strategy, they don't make money (and could in fact lose money). For that reason, they'll spend whatever it takes to make sure that they can get orders in faster than their competitors. Specific stuff they're known to do includes:

• Locating their computers in the same facility as the exchange (<sarcasm>because the speed of light in between midtown and Secaucus makes a big difference</sarcasm>).
• Buying their networking hardware based on the number of microseconds latency for packet delivery.
• Buying MOM from ultra-low-latency specialists who focus at the microsecond level (like Solace Systems, who I've already profiled, and 29West, who I haven't)
• Almost completely ignoring Infiniband for no earthly reason that Nolan or I can figure out
• Sometimes using Java, even though it contradicts every single above-mentioned point

It's an arms race here, and the firms that play in it are constantly on the bleeding edge with startups to get every advanced technology (new hardware, new software versions) before their competitors, because it's a game where they believe microseconds matter.

The Utility Computing Play
Ultimately, this looks like a market that you actually can hit with a utility/cloud computing play, because if you look at what they're doing, every single tool in their arsenal amounts to one key metric: How fast can you deliver a new piece of market data from its original source to the code which must process it. All the tricks and techniques above are designed to optimize this one single metric.

That to me seems like a metric that a utility computing provider could easily optimize for, and do so in ways that individual firms wouldn't due to resources. For example, most people in this space aren't using Infiniband, even though it's the clear latency winner for the types of distribution that they're doing. Why not? Probably because they don't have the experience in coding against it and deploying it and maintaining it. But why should they care? They want the data into user space as fast as possible. Whether that's done over IB or Ethernet doesn't matter. The key metric for optimization is user-space data delivery.

I think rather than 10 banks and 30 hedge funds all playing the same game here, you're going to see 3-4 utility computing plays which are making an offer roughly along these lines:

• We're going to sell you computing power in some form (bear in mind that these guys usually want lots and lots of CPU time to run analytic models)
• We're going to have an SLA guaranteeing how fast we're going to deliver new data into user-space of your code
• We're going to have an SLA guaranteeing how fast we can deliver data between your various VM slices (and you pay more for lower-latency delivery)
  

It's such a clear value proposition once the regulatory issues are worked out that lots of firms would jump at the chance to offload their own infrastructure.

Not all of them of course. There will be a period where there are firms which believe, correctly or not, that they can do better themselves. But this will largely be because they've built up lines of business that won't survive the transition and they want to try to defend them as long as they can. Eventually even they will give up.

The Aftermath
First of all, the hosting providers will be able to achieve a level of scale that no single financial services player would. This will allow them to pull R&D in house rather than relying on startups pushing the boundaries of technology. This means that they'll be able even to afford to do custom silicon (maybe working through design agencies, but clearly proprietary hardware configurations).

Second, all trading strategies which rely on acting on market data faster than competitors are going to disappear into Zero Arbitrage Assumption land: no firm will ever be able to outgun the hosting providers. Any hosting provider which ever allowed a single market participant (as opposed to another hosting provider) to get a latency advantage would go out of business quite shortly thereafter.

Third, more players will actually get in the algo trading space. Right now there's a high fixed cost in building out an infrastructure capable of supporting low-latency automatic trading systems. However, the fixed cost allows you to add additional apps easily once you build out your data centres and switches and messaging brokers and all that. Once you reduce that cost to 0, then more people will start dabbling in low-latency algo trading. Hopefully the exchanges are able to handle it, because it's going to increase the number of trades dramatically and introduce some people with some really stupid systems pretty quickly.

Fourth, Silicon Valley/Startup activity in the space will change. A traditional path for a startup looking at the low-latency space, particularly in hardware, is to produce some prototypes, sell them to each of the low-latency customers in financial services one-by-one, exhaust the market, and sell out to Cisco or Juniper or somebody and have them (hopefully) include the underlying technology in their existing product range. This gets disrupted by moving to utility vendors as the only remaining clients; I'm not sure quite what this does to the startup prospects, except that it will definitely limit the number of potential reference customers that a startup can acquire. Each one will be a much harder sell, and require much more time. (Note that this is not unique to the low-latency space; in general, the utility computing transition is shrinking the number of potential customers for most infrastructure hardware, and that's changing the market accordingly. I single this out because specifically targetting the financial services industry's latency-sensitive applications has been a well-known path for Silicon Valley startups, and it's going to convert).

Fifth, more things viewed as an in-house operation in low-latency processing are going to be cloudsourced. Once you have your computational nodes outsourced, and you have your market data delivery outsourced, why not keep going? Why not outsource your live analytics? Or your historical time series storage/lookups? Or your mark-to-market golden copies for end-of-day P&L/Risk? More and more of this is going to be cloudsourced, and very little of it is going to be provided directly by the utility companies: it'll be provided by companies that can span all the utility computing companies.

Conclusion
I think very soon vendors will resolve the regulatory issues and start launching low-latency expert cloud computing solutions specifically targeting the low-latency automatic trading community. This will result in:

1. Hosting providers clearly winning the latency war.
2. The elimination of most profitable low-latency trading strategies.
3. Algo trading will grow as a financial services space.
4. Silicon Valley will have to view the market differently.
5. More market data services are going to be cloudsourced.

Utility Computing in Finance: Regulatory Arbitrage

Financial Services companies (particularly those doing derivatives work) are quite big consumers of CPU power for all the computations that are necessary to support the business (and as some derivatives contracts amount to running millions of Monte Carlo simulations to capture low probability events, they can be pretty expensive to compute). For that reason, financial firms spend a lot of time working out how to minimize their computational costs and eke out every cycle that they can.

I had a conversation with my friend Nolan who knows quite a bit about the utility computing space, and thought it would be useful to the general public. I'm defining utility computing as renting out CPU time on other people's hardware (such as EC2). I'm not talking about in-house grid computing here, as you'll see when you look at some of the points below.

Broadly speaking, financial services computation boils down into the following areas:

• Latency Sensitive. These are computations that you need to do as quickly as possible, and are normally internally parallelized where possible. But within this, you have two more distinctions
• Market Latency Sensitive. These systems are usually automatically trading based on algorithms, and need to be able to execute trades as fast as possible with no human interaction. These guys are the ones buying ultra-low-latency MOM systems and networking switches.
• Human Latency Sensitive. These systems are presenting data to a human user who responds in some way, and you broadly have to keep up with the market and a human's reaction time (hint: updating a screen any faster than once per 100 milliseconds ignores basic human reaction time).
• Latency Insensitive. These are the computations that you have to run periodically, usually once a day as part of your P&L and risk cycles. They run overnight, and your basic job is to do all your computations within your computation window using as little expensive hardware as you can.

Of these, Market Latency Sensitive systems aren't a candidate for the current generation of utility computing: they operate within tolerances that current utility computing platforms can't possibly cope with. These are the guys who rent space at the same data center that their dominant exchange is located in to try to save every last millisecond (and are the reason why Lehman sold to Barclays for less than the value of its Manhattan data center). They are definitely candidates for a new generation of latency-sensitive utility computing (where the cloud guarantees that data will be delivered to your software within N microseconds of it arriving from the exchange), but until general purpose utility computing is prepared for financial services it isn't a starter because they would still face every issue below.

Human Latency Sensitive may be candidates for current approaches (once you're considering humans, you can start to think about acceptable delays for computations, and farm some things out to a utility computing cloud). Latency Insensitive definitely are (you just need the cycles once a night; as long as you finish within your window, you really don't care about anything else). Yet, financial services firms seldom use utility computing facilities of any kind.

That's actually a lie. They use them for everything that they can. The problem is that the regulators don't like them, so those uses are extremely limited.

As near as I can tell, here are the things that the regulators don't like about them:

• The machine isn't under your control. This means that, in theory, the utility company could be doing all types of nasty things with your execution flow and you wouldn't necessarily know anything about it.
• The data path isn't under your control. This means that you're sending (potentially) sensitive data outside your network, which someone might intercept and do something with (modify it, sell it to your competitors).
• You have no availability guarantees. If you own your own machines, you know if you have the CPU cycles to deal with all your available trades. Utility computing companies may lie to you and you wouldn't be able to deal with critical time periods.

These are not insurmountable! Rather, it requires a discipline and auditing routine similar to what financial services firms work with all the time. While it may be good enough for your Web 2.0 startup to just assume that Amazon isn't up to anything nefarious, it isn't good enough for the regulators.

The solution here I think is two-fold:

1. Regulators provide standards and guidelines. This would include the rules that they would require utility computing providers to adhere to, and facilities for independent audits of those procedures, precisely as financial services firms already have to for internal operation. I can think of a number of potential rules, but I'm not a regulator. They should come up with what they would be happy with, and they should be as equivalent as they can be between the four major financial services markets: London, USA (specifically New York and Chicago), Hong Kong and Tokyo.
2. Companies build utility computing platforms to those standards. I don't expect Amazon to start providing auditable logs of everything they do to EC2. I don't expect Google to adhere to external audits of their processes and procedures and security standards every 6 months. But those are value-add services that I'm sure somebody would be willing to do, because they could charge much more than Amazon or Google do. And those would be the firms that financial services firms could use for their utility computing needs.
   

Financial services firms are used to working within constraints. We're used to paying for products and services that can satisfy these constraints. But in this case, I think the regulators need to step up and provide guidance about what they would expect in this space so that the market can progress.