2009 Predictions

Just a quick set of predictions for 2009. I'll revisit nearer the end of the year to see how I've done.

Messaging Breaks Out
Right now asynchronous systems, particularly ones using message oriented middleware, are still pretty fringe. However, even though I disagree with the technology choice, techniques like XMPP are starting to bring asynchronous communications to the masses. I predict that the extension of XMPP, the forthcoming standardization of AMQP, new infrastructure being developed (like RabbitMQ, OpenAMQ, and Qpid), and extension to new programming languages and application execution environments will all drive more and more developers to finally end polling for updates.

Cloud Will Become Less Buzzy
Right now Cloud/Utility computing is really too buzzy to make heads or tails of, and I think it's suffering from that: when you say "Cloud Computing", it means both nothing and anything. I think 2009 will be the year that people start to solidify what works best in a utility computing space versus a local hosted space, and where hosting providers get mature enough and technology becomes mainstream enough that there's an assumption of utility over local hosting.

Java Will Stagnate
I predict that Java 7, if it even hits by the end of the year, will be pretty anemic and, quite frankly, lame. There will be some interesting enhancements to the JVM, but that's pretty much all that's going to be noteworthy.

C# Will Over-Expand
The entire Microsoft ecosystem will expand and grow beyond the ability of typical Microsoft stack developers to keep up. We're already seeing it with WinForms vs. WPF, LINQ vs. ADO.NET Entity Framework, and whole new technologies like WCF. I predict by the end of the year the Microsoft ecosystem has become so bewildering that while leaders in the community push for more change, day-to-day developers push for a halt to just absorb what's been happening.

Non-Traditional VM Languages Will Break Out
Right now most developers in enterprise systems are coding against the "core" languages (C#, VB.Net, Java) against their respective VMs. This reflects in no small part the maturity and power of their underlying runtimes (the CLR and JVM respectively), but the leading edge of the communities have already started to explore other languages that offer concrete advantages (F#, Scala, Groovy) on the tight runtimes that are already available. This will be the year that those actually break out of the leading edge and into mainstream use.

No-One In Social Networking Makes Money
I still think, by the end of 2009, there won't be a player in the social networking space that is cashflow positive, and I don't think it'll be by choice. I think the technology is so disruptive that we still haven't seen the "right way" to monetize it yet.

Sun Radically Restructures
Sun can't keep burning through cash the way they have been, and they can't continue to have such a chaotic story. At some point in 2009, Sun will relatively radically restructure itself in a bid for survival. Hopefully they'll have read my analysis (part 1, part 2). At the very least, they'll change their ticker away from JAVA.

I think you'll note that all my predictions are fuzzy. That's so that in December, 2009 when we revisit them, we can determine that I was partially right on nearly all of them.

SNOS: Quit Angering Andrew Tanenbaum

Recently, someone forwarded me a blog post about a Social Network Operating System and what that might mean. I'll tell you what it means: nothing. Much like the Chewbacca Defense, it doesn't make sense.

An operating system is a system designed to manage low-level hardware resources on behalf of software processes. Let's look at the examples from the original blog post on the evolution of history:

• Unix: An operating system. Huzzah! We've correctly identified one!
  
• J2EE: Not an operating system. Probably not even the right example for this entry, but nevertheless...
• "Something in the browser": Not an operating system. Lest we forget history, Ted Dziuba already told us what we need to know about this subject.

Here's a cheat sheet:


The irony is that what Chris is talking about early in the post is quite a useful thing to have, which is a set of underlying services that will allow people to develop new social applications easily that combine features and data from other social applications. But that's not an Operating System.

It's also not a Relational Database, a B-Tree, a Hash Table, a Compiler, or a Micro-Kernel. All of these things have pretty precise meanings that everybody understands, and the precision facilitates conversation. Let's keep the conversation back on precise terminology. (And inventing your own terminology is fine as well if there's no term that precisely describes your use case [that's how we got Micro-Kernel for example], just don't reuse one).

Scoble Joins The Real-Time Web Conversation

Well, maybe "joins" isn't the proper word, but this showed up on my FriendFeed account today: RSS Shows Its Age. As I've posted about already, I think polling for updates is pretty lame, and asynchronous messaging is the right approach. But yet again, XMPP isn't it.

Enter the next contender: SUP. Took me a while to figure out what precisely it is, but it's essentially a JSON-encoded consolidated feed URL for a batch of URLs. So if you are a service that has a million individual RSS URLs, you provide one SUP URL, which allows clients to make a request and see all URLs that have been updated "recently". The magic sauce is:

• Instead of individual RSS URLs, they get encoded into a nonce (an SUP-ID) to save on transfer costs;
  
• JSON rather than XML;
• Definition of "recently" is up to the publisher so that it can be statically served and cached and changed depending on publisher server load.
  

The end results are you can poll the SUP feed for a whole site, and then find out what you have to hit the individual RSS for. So this makes it faster to poll the million RSS URLs, particularly since at any given time a tiny fraction are actively updating. And it makes FriendFeed feel better about polling Flickr every second to reduce latency.

But it's still polling! It still ultimately doesn't scale. You can make polling suck less, but it will always suck.

Again, asynchronous message oriented middleware systems are the ultimate solution to this, and anything you do in the interim is only postponing the inevitable.

The Forthcoming Ultra-Low-Latency Financial Apocalypse

This is a follow-on from my previous post on utility computing in financial services, and came out of a discussion between Nolan and me when he was helping review an early draft of the post (yes, I actually do review the blog posts, unlike my email rants). He brought up a good point which I think is going to result in the ultimate apocalypse of much of the ultra-low-latency space in financial services infrastructure.

Ultra-Low-Latency Background
There are a number of strategies to make money in the financial markets which are extraordinarily sensitive on the time it takes to deliver an order to an exchange based on incoming market data. Some of these are in the pure arbitrage space (such as an equity which trades in euros on Euronext and pounds on the LSE), but the vast majority of them are more complex (a trivial example: Cisco is part of the NASDAQ index; when the NASDAQ index moves 1 point, index funds will need to buy Cisco to keep up with the index, so buy Cisco if the NASDAQ moves up a point), to the point that you can't possibly describe them in this type of format. (Disclaimer: I've never worked in an ultra-low-latency trading environment; this is entirely based on conversations I've had with those who do).

These guys specialize in minimizing the time it takes for them to get their orders into the exchanges: if they don't get there before any other company trading using essentially the same strategy, they don't make money (and could in fact lose money). For that reason, they'll spend whatever it takes to make sure that they can get orders in faster than their competitors. Specific stuff they're known to do includes:

• Locating their computers in the same facility as the exchange (<sarcasm>because the speed of light in between midtown and Secaucus makes a big difference</sarcasm>).
• Buying their networking hardware based on the number of microseconds latency for packet delivery.
• Buying MOM from ultra-low-latency specialists who focus at the microsecond level (like Solace Systems, who I've already profiled, and 29West, who I haven't)
• Almost completely ignoring Infiniband for no earthly reason that Nolan or I can figure out
• Sometimes using Java, even though it contradicts every single above-mentioned point

It's an arms race here, and the firms that play in it are constantly on the bleeding edge with startups to get every advanced technology (new hardware, new software versions) before their competitors, because it's a game where they believe microseconds matter.

The Utility Computing Play
Ultimately, this looks like a market that you actually can hit with a utility/cloud computing play, because if you look at what they're doing, every single tool in their arsenal amounts to one key metric: How fast can you deliver a new piece of market data from its original source to the code which must process it. All the tricks and techniques above are designed to optimize this one single metric.

That to me seems like a metric that a utility computing provider could easily optimize for, and do so in ways that individual firms wouldn't due to resources. For example, most people in this space aren't using Infiniband, even though it's the clear latency winner for the types of distribution that they're doing. Why not? Probably because they don't have the experience in coding against it and deploying it and maintaining it. But why should they care? They want the data into user space as fast as possible. Whether that's done over IB or Ethernet doesn't matter. The key metric for optimization is user-space data delivery.

I think rather than 10 banks and 30 hedge funds all playing the same game here, you're going to see 3-4 utility computing plays which are making an offer roughly along these lines:

• We're going to sell you computing power in some form (bear in mind that these guys usually want lots and lots of CPU time to run analytic models)
• We're going to have an SLA guaranteeing how fast we're going to deliver new data into user-space of your code
• We're going to have an SLA guaranteeing how fast we can deliver data between your various VM slices (and you pay more for lower-latency delivery)
  

It's such a clear value proposition once the regulatory issues are worked out that lots of firms would jump at the chance to offload their own infrastructure.

Not all of them of course. There will be a period where there are firms which believe, correctly or not, that they can do better themselves. But this will largely be because they've built up lines of business that won't survive the transition and they want to try to defend them as long as they can. Eventually even they will give up.

The Aftermath
First of all, the hosting providers will be able to achieve a level of scale that no single financial services player would. This will allow them to pull R&D in house rather than relying on startups pushing the boundaries of technology. This means that they'll be able even to afford to do custom silicon (maybe working through design agencies, but clearly proprietary hardware configurations).

Second, all trading strategies which rely on acting on market data faster than competitors are going to disappear into Zero Arbitrage Assumption land: no firm will ever be able to outgun the hosting providers. Any hosting provider which ever allowed a single market participant (as opposed to another hosting provider) to get a latency advantage would go out of business quite shortly thereafter.

Third, more players will actually get in the algo trading space. Right now there's a high fixed cost in building out an infrastructure capable of supporting low-latency automatic trading systems. However, the fixed cost allows you to add additional apps easily once you build out your data centres and switches and messaging brokers and all that. Once you reduce that cost to 0, then more people will start dabbling in low-latency algo trading. Hopefully the exchanges are able to handle it, because it's going to increase the number of trades dramatically and introduce some people with some really stupid systems pretty quickly.

Fourth, Silicon Valley/Startup activity in the space will change. A traditional path for a startup looking at the low-latency space, particularly in hardware, is to produce some prototypes, sell them to each of the low-latency customers in financial services one-by-one, exhaust the market, and sell out to Cisco or Juniper or somebody and have them (hopefully) include the underlying technology in their existing product range. This gets disrupted by moving to utility vendors as the only remaining clients; I'm not sure quite what this does to the startup prospects, except that it will definitely limit the number of potential reference customers that a startup can acquire. Each one will be a much harder sell, and require much more time. (Note that this is not unique to the low-latency space; in general, the utility computing transition is shrinking the number of potential customers for most infrastructure hardware, and that's changing the market accordingly. I single this out because specifically targetting the financial services industry's latency-sensitive applications has been a well-known path for Silicon Valley startups, and it's going to convert).

Fifth, more things viewed as an in-house operation in low-latency processing are going to be cloudsourced. Once you have your computational nodes outsourced, and you have your market data delivery outsourced, why not keep going? Why not outsource your live analytics? Or your historical time series storage/lookups? Or your mark-to-market golden copies for end-of-day P&L/Risk? More and more of this is going to be cloudsourced, and very little of it is going to be provided directly by the utility companies: it'll be provided by companies that can span all the utility computing companies.

Conclusion
I think very soon vendors will resolve the regulatory issues and start launching low-latency expert cloud computing solutions specifically targeting the low-latency automatic trading community. This will result in:

1. Hosting providers clearly winning the latency war.
2. The elimination of most profitable low-latency trading strategies.
3. Algo trading will grow as a financial services space.
4. Silicon Valley will have to view the market differently.
5. More market data services are going to be cloudsourced.

REST Requires Asynchronous Notification

I recently (well, compared to people like Steve Vinoski) got converted to the RESTful architecture movement, but one of the first things that I recognized is that it suffers from the same generic limitation of every HTTP-based component architecture that I've seen: there's no defined way to get updates on content.

In fact, when I started getting more immersed in the REST culture, I realized that a lot of the nuances that are ideal to master in order to provide a great RESTful interface (caching, timeouts, advanced status codes) are there specifically to limit the number of repolls that are necessary against the terminal service by improving the ability of caching proxies upstream to handle requests.

But no matter what, there are clearly latency issues involved in any polling system, particularly one which is using fixed content caching timeouts to optimize performance (because this puts the latency entirely under the control of the publishing system which generated the result of the initial GET command). How do we minimize those?

Pull RSS Is No Substitute
RSS/Atom (note that I'm going to use RSS as a shorthand for the whole RSS/RDF/Atom spectrum of pull-based syndication lists) seems like a reasonable way to minimize the actual content that needs to be polled by batching updates: instead of having clients constantly polling every URL of interest, you batch them up into groups (perhaps one for each resource domain) and provide a single feed on each group. The RSS feed then indicates the state of all the resources in its domain that might have changed in the last period of interest. So now we've reduced hundreds of GETs down to one, but we're still polling.

My old colleague Julian Hyde posted some stuff recently which was precisely along the lines of what we're talking about here (post #1, post#2), in that he's building a live querying system against streams, and finding that the polling nature of RSS is particularly noxious for the type of system they're building. I'm surprised it took him so long. RSS is actually a pretty terrible solution to this problem: it combines polling with extra data (because an RSS feed is generic, it publishes more data than any particular consumer needs) in an inefficient form and still allows for the RSS publisher to put a caching timeout. Wow. Talk about a latency effect!

It gets worse. Let's assume that you're trying to do correlations across feeds. Depending on the cache timeout settings of each individual feed, you might have a latency of 1sec on one feed and 15min on another feed. How in the world do you try to do anywhere near real-time correlations on those two feeds? Answer: you can't. At best you can say "hey, dear user, 15 minutes ago something interesting happened." 15 minutes is a pretty short cache timeout for an RSS feed to be honest, but an eternity to many applications.

The only real advantage that RSS/Atom provides is that because it's based on HTTP, and we have all these skills built up as a community to scale HTTP-based queries (temporary redirects, CDNs, caching reverse proxies), we can scale out millions of RSS queries pretty well. Problem? All those techniques increase latency.

Enter AMQP
Anyone with any experience with asynchronous message oriented middleware can see that fundamentally you have a one-to-many pub/sub problem here, and that smells like MOM to me. More specifically, you have a problem where ideally you want a single publisher to be able to target a massive number of subscribers, where each subscriber gets only the updates that they want without any polling. Wow. That looks like exactly what MOM was designed for: you have a broker which collects new messages, maintains subscription lists to indicate which consumers should get which messages, and then routes messages to their consumers.

It's so clear that some type of messaging interface is the solution here that people are starting to graft XMPP onto the problem, and using that to publish updates to HTTP-provided data. Which would be nice, except that XMPP is a terrible protocol for asynchronous, low-latency MOM infrastructure. (If nothing else, the XML side really will hurt you here: GZipping will shrink XML quite a bit, at a computational and latency cost; a simple "URL changed at timestamp X" is going to be smaller in binary form than GZipped XML and pack/unpack much faster).

But we've already got another solution brewing here: AMQP, which simultaneously provides:

• Binary protocol (for performance)
• Open protocol (for compatibility)
• Open Source implementations
• Built by MOM experts

Looks like a winner. Just about the only thing that I can think the REST crowd not liking is that it's not over HTTP (The One True Protocol), but every reasonable MOM system has support for HTTP-based subscriptions (at a pretty serious performance penalty).

Combining REST with AMQP
At my previous employer, I actually built this exact system. In short, resources also had domain-specific MOM topics to which the REST HTTP system would publish an asynchronous message when any update to a resource happened. Clients would hit the HTTP endpoint to initialize their state, and immediately setup a subscription to updates to the separate MOM system.

Without the MOM component, it's just pure REST. Once you add it, you eliminate any resource or RSS polling at all. The MOM brokers are responsible for making sure that clients get updates that they care about, and there's only the latency that the broker introduces to updates. Worked brilliantly: all the benefits of a RESTful architecture, with none of the update latency effects.

We had it easy though: this was all purely internal. We didn't have to operate at Internet Scale, and we didn't have to interoperate with anybody else (no AMQP for us, this was all SonicMQ as the MOM broker).

In order to get this to work as a general principle, we as a community need to do the following:

1. Finish AMQP 1.0 (I've been told the specs are nearly ready and then just comes the implementation and compatibility jams);
2. Decide on a standard pattern for REST updates (topic naming, content encoding, AMQP broker identification in HTTP response headers);
3. Develop easy to use publisher and subscriber libraries (so that services and client packages can easily include this functionality across all the major modern languages);
   
4. Develop AMQP brokers and services that can operate at Internet Scale.

Once we've done this, people like Julian can hook up SQLStream directly up to AMQP endpoints and consume all the data they want without invoking the horror that is polling.

Purity vs. Pragmatism

I was being interviewed for a new job last week (yes, I am actively on the market), and had a very interesting, frustrating run-in with one of the interviewers (disclaimer: by mutual acclaim, the role and I decided we weren't right for each other; you'll figure out why shortly) at a large bank.

We Frustrate Each Other
The frustrating part of the interview came when I realized that The Interviewer (we'll abbreviate it to TI, not to be confused with the hip-hop artist T.I.) and I were disagreeing on virtually every point of philosophical substance. But that was just a manifestation of a broader disagreement that I think went to the core of why the two of us must never work together: we fundamentally disagree with the core principle of software engineering.

Let me explain with a few examples of where we differed to try to explain what was going on (no, this is nowhere near an exhaustive list):

• He believed that checked exceptions in Java are always wrong; I believe that sometimes they're useful and sometimes they're not.
• He believed that you should only accept or return interfaces (and never a concrete class) from a module; I believe that you pick and choose between interfaces and POJOs depending on the context.
• He believed that setter-based Dependency Injection is always wrong and that only constructor-based DI should be used; I believe that you pick the right one for the context.
• He believed that you can only ever use DI with a DI container like Spring or PicoContainer; I believe that it's an architectural principle that can applied with or without DI-specific tools.
• He believed that you cannot consider yourself a practitioner of agile methodology without rigidly adopting a Formal Agile Methodology (Scrum, XP, whatever); I believe that the whole point of agile methodologies is that you pick amongst the parts that help your team develop better software faster.

What's the major differentiation here? Purity.

Purity
TI's approach to every major difference between the two of us fell down on the side of rigid, unbending application of a principle in the interests of purity. My approach is far more fluid and contextual.

Purity in any endeavor (art, design, architecture, music, religion, software engineering) is attractive because it strips away all thought and all decisions, and in doing so, pushes a concept to its ultimate expression. I can understand the sentiment. When I moved into my flat, every single surface was either white (floors, walls, ceilings, some doors) or gray metal (stairs, shelves, other doors). It's a minimalist, pure aesthetic, and it removes all distraction and makes it very simple to make decisions: there's nothing subjective about additions.

Sometimes, purity is exactly what you want. It focuses you, and allows you to fully explore one concept to its extreme (how white can you get the walls and floors? can you get them the same white even though you have to use different paints for different surfaces? can you keep the floor white even though people walk on it and put furniture on it?). Even the exploration of pure silence has begat its own groundbreaking work.

Purity in Software Engineering
Taking a purist approach to a software engineering matter allows you to nail your banner on the church for all to see: I believe in X, therefore X is always correct; by applying X to every situation, I prove the superiority of X and validate my initial conclusion that X is superior. This comes up a lot, particularly in architectural discussions:

• Asynchronous Messaging is great! Let's use it for everything!
• An RDBMS is great! Let's use it for everything!
• REST is great! Let's use it for everything!
• Google Protocol Buffers is great! Let's use it for everything!
• Cubes are great! Let's use them for everything!
  
• Lisp is great! Let's use it for everything!
  

The converse also happens:

• XML is crap! Let's banish it from the world!
• RPC is crap! Let's banish it from the world!
• Solaris is crap! Let's banish it from the world!
• RDBMSes are all crap! Let's banish them from the world!
• TCL is crap! Let's banish it from the world!
  
• Lisp is crap! Let's banish it from the world!
  

Purist decisions are easy. They don't require thought. They don't require constant critical evaluation.

And that's why I view them as intellectual cowardice: by limiting your choices intentionally, by limiting the scope of tools at your disposal, by closing your mind off, you reduce the amount of thought you have to put in. But as engineers, as craftsmen, we live by our minds: we are paid to think, and the more we're paid, in general, the more thinking we're expected to do.

TI could replace himself with someone far less experienced by simply writing his own DSL that works on the JVM that doesn't allow you to do any of the things that he thinks are wrong, and forces you to do all the things that he thinks are right. It wouldn't be that hard. Then he can simply hand that off to someone and say "here you are; a language that obeys every one of my edicts perfectly; you are sure to do an excellent job now that I've handcuffed you to my beliefs."

[Aside: if TI's beliefs realy were so universal as to allow him to view me with revulsion for not sharing them, why isn't there already the TI programming language? Are you to tell me that there isn't a programming language which targets some target environment (JVM, CLR, raw x86 machine code) that forces the practices that he likes and forbids the practices that he doesn't? Really? Does that tell you something perhaps? Because it's not like we have an absence of programming languages these days. And it's not like it's that particularly hard to write a language and target one or more existing VM environments, so all the hard work is taken care of already.]

In Defence Of Pragmatism
I'm impure. I'm about as tainted as you can get. And you know what? I think that makes me more effective, rather than less. Because what I get out of a lack of purity is pragmatism. The two can't coexist particularly effectively: I think you fundamentally agree with one or the other.

Pragmatism allows me to look at a problem, carefully consider the advantages and disadvantages to each potential solution, and then determine the right approach to each particular situation.

Pragmatism allows me the flexibility to do things that I know in other circumstances would be wrong, but in that particular one would be right.

Pragmatism allows me to have a bag of tricks rather than one, and pull them out as I see fit.

Pragmatism allows me to gradually refine my beliefs by using both my favored and unfavored approaches in a variety of circumstances so that I have evidence and backup behind me when I express my opinion.

Pragmatism allows me to work with a variety of code written by a variety of people and engage with it constructively as it is, rather than seeking to rewrite it before I'd be willing to touch it.

Pragmatism allows me to decide where to focus development efforts based on what's most important that day: sometimes rushing a feature into production to make more money, sometimes spending time testing the bejebus out of a small change before I make it.

Pragmatism forces me to build software which is easy to maintain and consistent in its internal architecture and clean in its modularization and consummately tested, because that's the only way to maintain a complex system over time. Pragmatism also tells me that if I know the piece of development is only there to run a single time, focusing on all of that is wasted time better spent elsewhere.

Pragmatism allows me to assess the strengths and weaknesses of my team, and the constraints of my customers, before assessing the correct development practices that will result in that unique group of people producing the best software possible with the fewest resources in the shortest possible time. Pragmatism forces me to understand that no one methodology could possibly work for all engineers, all customers, and all projects.

Pragmatism allows me to focus on one and one thing only: getting the job done well. Do what it takes, but get it done, and make sure it works properly. And that's precisely what we're here to do as software engineers: engineer software. And engineering, unlike art, design, music, or any purely creative endeavor, requires getting something done that works. We're not in this business to craft a purist expression of an idea. We're here to build things that work.

The irony is that many purists get that way out of a misguided belief that they're being pragmatic: buy choosing a single technology or technique or methodology, they've chosen the One True Solution and don't have to consider the same decision with over and over. Furthermore, they've made sure that those people working for/with them (particularly the lesser skilled ones) don't do The Wrong Thing. But that means that they've closed their minds off to the chance that they've chosen incorrectly. Or, even more appropriately, that there is no one right decision.

That's why purity and pragmatism can't coexist: purity requires that you ignore the results in favor of an ideological ideal; pragmatism is all about pursuing results for their own sake.

And that's why I think TI and I couldn't work together: I'm a pragmatist and he's a purist. The two of us were bound to clash, and at least we discovered it quickly.

But in the end, I know my approach is right. I'm proud to be a pragmatist.

Except when it comes to TCL. Man, I wish I could banish it from the world.....

Meaningless, Programming, and Innate Affinity

Clay Shirky guest post on BoingBoing called "Comfort with meaninglessness the key to good programmers". The comments are filled with "my programming language r0x0rZ" and suchlike, but it's another perspective on the often-espounded (not least by yours truly) theory that being an excellent programmer is an innate characteristic. You can be trained to be a software engineer of some skill (and large firms are stocked with hundreds of the type, who are trained to implement business logic in a programming language, or in pixel-pushing GUIs), but there appears to be something innate in making a programmer.

Or, as I was recently told, "so what you're saying is that all of you are naturally freaks."

Real Time Linux Kernel Smacktown Comments

I read El Reg's writeup of Red Hat and Novell attempting to out-gun each other on their RT-esque kernel patch versions of their main Linux distros. Just a few quick comments.

First of all, they're testing against RMDS 6 (something I have some experience with). When you see anything about an RMDS 6 P2PS, what you're really looking at is an ancient branch of RVD running the TCP form of the Rendezvous protocol. That means that a P2PS esssentially acts as a middleware broker in a bizarre mixture of a protocol designed for unreliable broadcast/multicast usage (Rendezvous) but running on a single machine. It's a quite strange beast to be fair. Just a bit of background detail for you there.

Secondly, where things get really strange in this is when Infiniband comes in on the Novell SLERT side. Uhm, huh? They're combining RMDS with IB? In what form? Are they doing IP/IB? Uhm, why? I'm sorry, but if you're going to roll out IB, without using something IB specific like OpenFabrics that allows you to leverage all that nice IB RDMA action, there's something seriously wrong. Yes, you might shave off a few microseconds off your latency compared to IP/Ethernet, but you'd get much better by coding raw against IB. Based on that, I'm actually surprised that Novell included this at all, since I really doubt that anybody is going to attempt to improve the performance of their RMDS infrastructure by running it over IB.

So all that being said, this type of benchmark is really just a proxy for something testing how fast the kernel patches can get networking packets off the wire and into user-space for processing. That's a pretty interesting point and good to know, but if you really care about that and don't have to run RMDS for everything in the first place, why not use something like Solace which doesn't have user-space in the first place? Why not just code against IB using OpenFabrics? This is far more interesting to me in the client side (where you're going to have user-space code ultimately doing your processing) than on the broker (e.g. P2PS) side.

So that being said, I think a very interesting benchmark would be:

• Similar workload in general (the STAC workloads actually are representative of the general workload you get in financial market data processing);
• Hardware distribution to the clients (Tervela, Solace, or OpenFabrics/IB);
• Clients different between the RT-patch version of the kernel and stock kernel.

The client is where I think the action needs to be here, not the broker.

Sun: Split Up, Yo

Since the whole pick-on-sun meme has really taken off, I figured I'd post some more on the current situation, because El Reg's take led me to believe that the only really positive outcome is complete dismemberment.

Look back at my previous posting. Logically, you've got several distinct companies floating about in that mess of Fail:

• Legacy SPARC, including Niagara. Sell it to Fujitsu and give up. Although the Niagara stuff may be cool, if you can't get it price/performance competitive with x86 just give up and use it for having a forward migration path for your existing customers.
• Storage. This would include the StorageTek stuff, Thumper, FishWorks, and the Open Storage stuff. Separate company, which might even be able to stand on its own. Otherwise, find someone like Dell who's sick of OEMing EMC stuff and sell the business to them.
• Software. Take all the open source stuff you have (including and especially Solaris) and give it to a new company which doesn't have anybody working for it who has ever worked for the hardware business in any form (in fact, it should probably be run by non-Sun people). Allow them to create an alternative to Novell and Red Hat as a multi-product open source company, completely ignoring your Sparc lines of business. I guarantee that this would result in the software changes that the community wants (ZFS with a Linux-friendly license) because it's in the best interest of such a company, but not in the best interest of the Sun hardware business. All those retarded decisions you've made in software seem to be to try to push your legacy hardware; free the company from that, and it'll do some great stuff.
  
• Other Stuff. This is the x86 stuff and the HPC stuff (Constellation, blades) and everything else. Leave this as a rump hardware company, allowed and encouraged to compete with the Fujitsu-owned Sparc business. Alternatively, sell it to one of the Asian manufacturers that wants to bootstrap a server business (Asus?). I think it'd do quite well actually.

The simple fact is that if you look at it this way, Sun makes logical sense as 4 different companies, and really they're not getting any synergies out of having them all as one company, so why keep pretending? Split it up completely and maybe the world will be a better place.

If nothing else, we could all get on with our lives and stop blabbing on about it.

Programming Problems and Technical Interviews

I've been involved in quite a few technical interviews over the years, and one thing that invariably (and rightfully) comes up is the programming problem. I view this as an opportunity to see if someone can actually code, rather than just talking about it. The real issue here is that the interview room is not the same environment as a developer will be using: you don't have a keyboard in front of you, you don't have all your tool suite available, at best it's a first order approximation.

That being said, I think any technical interview which doesn't provide the interviewer with a chance to evaluate the candidate's actual coding skill is just wasted time; while we need to think, write, and discuss, software engineers are paid to provide working code. Your job as an interviewer is to determine whether the candidate can do that subject to your standards and requirements. Your job as a candidate is convince the interviewer of that fact.

So what are the basic ways in which I've seen this hurdle laid out?

Read Me The Code
This is the most obscure one, but it was part of my phone screen for a job at Google back in 2002 (disclosure: I was offered the job, and like an idiot, turned it down). Essentially, I was given a programming problem over the phone, given about 10 minutes while I was on the phone to write down a solution, and then had to read it out to the developer on the other end.

I didn't really get this as a concept, and I still don't. Is he mentally picturing the code as I write it? Writing it down himself to make sure that I'm getting the semicolons and braces correct? Why even bother? I'm not a fan of this one because I don't think it actually tells you anything about the candidate other than whether he can get his hand around describing syntactic elements of programming concisely over the phone.

I've chalked it up to an obscure Google-ism, or an interviewer experimenting with a new interviewing style (we've all done it).

Whiteboard/Pen-and-Paper
This is the most common form, and in it the candidate and the interviewer get together in a room and the candidate writes some code longhand, whether on paper or a whiteboard or anything else. I think that this is a relatively valuable exercise for simple programming problems, because if you can't write out something that works elegantly for a simple problem, then you probably don't have an excellent grasp of your underlying language.

But the key thing here is that you have to engage in a dialog with the candidate about their solution, particularly if you think that there are tools or techniques or language features that they didn't employ. I've done this several times when someone had a strange solution only to find out that they didn't remember the exact syntax for using a language/library feature, and so they wanted to make sure that they did something that they knew to be correct. Because you don't know what the interviewer is looking for (and how much they'll mark off for mixing a brace or semicolon), you are at a loss to know what precisely to optimize for.

Furthermore, you have to be clear about what you're after here. Some problems are more conceptually difficult, and ideally the interviewer should be looking for your thought processes, and whether you can come up with a coded solution to the problem (damn the curly braces). Other problems are far more simple conceptually, and the interviewer should be seeing if you can write up a simple routine that is production quality longhand. Where things go awry is when interviewers want you to simultaneously come up with a solution to a really complex problem, and have every single syntactical element and edge case perfect in the first go. Gotcha-problems fit into this area as well (As a candidate, I get it; you've solved this problem in the past; you're ever so clever, and I should be thrilled to work with someone so brilliant as you; did this really teach you whether I can work with you effectively?).

The main problem with both of these approaches, though, is that they're not realistic. You don't write code by hand anymore. You definitely don't read it over the phone. These exams test (to some extent) your raw programming ability and familiarity with the syntax and libraries of your programming languages, but they don't show how well you can actually engineer software. For that, you need one of the following methodologies.

In Advance Programming Problem
I employed this as a low-pass filter while interviewing candidates at a previous employer, and it worked out pretty well. Rather than doing a phone screen at all, we had a relatively simple programming problem (in our case, XML walking) that we wanted a compilable, runnable solution to. Shouldn't have taken anybody competent more than an hour to do, and we wanted to see the style and nature of their code. What tools did they use? What language features? How did they document? How did they break up the problem? How did they package up the result?

This served two purposes: firstly, it weeded out the chaff that didn't really want to meet with us at all (if you think you're doing us a favor by even talking to us, you're not going to want to program in advance); secondly, it showed us a real-world example of what you can do with your chosen tool suite. Because it was so small, there were no excuses for providing something that didn't compile, didn't run, didn't produce correct output, or was ugly (all of which we got). Real software engineering involves spending time on niceties that aren't about the basic code execution path. Did you care enough to provide us with something that shows that you can do those?

Pair Programming Problem
Another financial services firm used this with me, and was the first time I saw it. Essentially, I came over to the interviewers computer, was asked which IDE I wanted to use (this was for a Java job, so I had my choice of Eclipse or IDEA), and he opened it up empty (new workspace for you Eclipse people). He then gave me the programming problem, and watched and talked to me as I worked through the entire end-to-end process in front of him.

I really liked this at the time. It was far more realistic than pen-and-paper could have possibly been, and also showed him how optimized I had gotten my workflow with my tool suite. It also allowed us to explore far more serious software engineering concepts like unit testing (I wrote the tests as I went along) and test coverage (making sure that my tests covered all inputs and results). I think this is actually an extremely strong way to check how well a programmer can work in a real-world scenario, and is far better than pen-and-paper development for languages like C#, Python, and Java, which have far simpler project setup times (though I think you could do it for a simple C/C++ test as well). That's the key thing here: you're watching a developer work with a real-world tool suite.

Programming Death Race
This was the most extreme form of any of these that I've ever had to do, and was extremely intense. I was given a Java file which contained four empty method bodies with documentation about what they were supposed to do, and told which ones I had to fill in. I then had 60 minutes to do two programming problems, returning the resulting Java source file, which they had an automated system to compile and run through a test suite.

These were not simple, reverse-a-linked-list problems. Each of them could easily have taken far more than an hour to do. And they were conceptually difficult as well (I'm not going to disclose the precise problems because I don't want to ruin their methodology), meaning that you have to actually think. A lot. There's not a lot of time to properly think and code a really complex solution when you know you're under a deadline!

While I completed the test successfully and got past that stage of interviews, I couldn't help thinking that it was all a bit much. I ended up providing a solution to one of the problems that was fully functional, but didn't actually look as elegant as I would have liked, because of the serious time pressure. Perhaps part of their goal was to see how fast you can get your head around a complex mathematical problem, but I felt like what it was rewarding was how fast you can churn out a solution, regardless of the underlying efficiency or elegance.

The firm was quite open that they would far rather have a hundred false negatives than a single false positive, which is why they intentionally made it so difficult, but something to me still said that proving someone can function at that level of intensity isn't necessarily the same as that someone can function on a consistent marathon pace. Then again, in financial services, if you can't function that quickly when things are going wrong and you're losing money every minute, it's probably a bonus for the employer to know that.

My Recommendation
I think I like the In Advance Programming Problem as a general concept (low-pass filter, and weeding out people who aren't really interested in the job), but I'd probably mix it with some of the stuff from the Programming Death Race: fixed time limits and automated testing. I just wouldn't make it as conceptually tough. That's better spent in person working through someone's thought processes.

But for the in-person work, I would definitely choose the Pair Programming Problem over the Pencil-and-Paper one any day. It serves the same purpose, but it also ensures that you're seeing how someone actually works. We don't code on paper anymore, we shouldn't assume that tells us whether someone's a competent programmer anymore.

If you want to test someone's thought processes, run through a conceptual problem. That's an extremely useful thing. But if you're going to ask for code, make sure that you either specify that you're looking for pseudocode, or that you don't care about the details and just want to see the rough order code. Otherwise you're at risk of taking a very valuable test (can the candidate think about problems in a reasonable way) and turning it into a trivia test.

And if you're not doing a real-world programming test, you should. Too many candidates have gotten good at the depth-of-a-tree or nth-Fibonacci-number level of problem without having any real concept of proper software construction as a discipline. Does your interviewing methodology go beyond that and try to determine whether the candidate is someone you'd want to share code with? If not, figure out how you would. Pencil-and-paper coding doesn't.

TBray Response: Sun Should Stop Sucking

(Talking about Tim Bray's opinion on what Sun should do).

As someone who's used a lot of Sun's products, here's my response as to what you can do, but more importantly, what you probably actually will do.

Price Your Hardware Less
Let's look at the Niagara vs. x86

• T5140 base (dual 4-core T2+ processors, 8GB of RAM, some disks) is $15k.
• X4140 (dual 4-core Opteron processors, 8GB of RAM, lots more disks) is $5.6k.
• X4150 (dual 4-core Xeon processors, 8GB of RAM, same disks as the X4140) is $7.3k.

Lemme get this straight, Tim: you think that the web application deployment crowd are willing to spend about 2-3 times the price for your magical CMT platform? Really? Have you met your typical hosting company? Or have you been spending so long at Sun you don't know what people actually care about?

Here's the thing: you actually make good hardware. The X4140? Great server. Your Niagara processors? Probably pretty good (never had a chance to play with one yet). Constellation? Great IB switch. The multithreaded 10GbE NICs? Pretty good hardware if you have an app that is multi-socket based. But people aren't going to run web applications on something that's more than twice the price; web applications are all about horizontal scaleout. Unless your hardware is 2x the performance for 2x the price, you're going to fail.

Quit Confusing Your Branding
We get it. You invented Java. Good on you. I like Java. That's not an excuse for:

• Changing your stock ticker to JAVA.
• Calling every single thing you can, even when it doesn't involve Java at all (Sun Java System Messaging Server, a product which contains precisely 0% Java)
• Grouping completely unrelated products under completely confusing banners (Sun Java Communications Suite, Sun Java System Messaging Server, Sun Java System Application Platform; these are all on your web site as of right now).

When you come up with your next branding exercise, please stop rebranding every single thing you make into one big bag of branding Fail.

Solaris Will Never Beat Linux
Like it or not, Solaris will never beat Linux at this point. You had lots of opportunities to make this not be the case, and you failed. Mostly because of your own stupid decisions, but the simple fact is that at this point, Solaris will never beat Linux for anything other than specialized systems. These are:

• Applications that require it because they were written once 10 years ago and can never change. Milk these guys for as much as you possibly can; it's the Computer Associates business plan and they seem to do okay out of it.
• Storage appliances (CIFS integration and ZFS are good and better than the equivalents in Linux-land).

No matter what Tim says, Solaris will never defeat Linux in the general web application deployment space, and there is absolutely nothing you can ever do to change this. Give up. Give up now. You're way too far behind, you don't get those developers, and you'll never be able to catch up with the state of the world.

The thing is that one of the points that Tim raises, Solaris having such a stable ABI, actually causes them problems in general worldview and software engineering, because it means that they can never actually change anything to make it better. But more than that, it indicates that their core focus is really about all the legacy applications which are tied to their platform, and not about driving new customers to the platform.

What Sun Could Do
Divide yourself logically into the following divisions:

• Legacy Systems. Sparc IV, Solaris, all the old software packages nobody uses, existing StorageTek hardware. Your job is to keep these customers from spending the effort to migrate to something cheaper; no more, no less.
  
• Modern Hardware. Your x86 hardware, IB hardware, networking chipsets, Niagara. Your job is to do advanced development and be technologically advanced, but at least marginally cost competitive.
  
• Open Storage. OpenSolaris, the new Open Storage hardware. Your job here is to provide a new path off all the storage dead ends that you've gone down, and try to eviscerate the big storage vendors who are insanely overpriced at this point.
  
• Goodwill Software. All the stuff you're never really going to make proper money off of, and probably shouldn't have gotten involved in in the first place. MySQL, Java, Glassfish, NetBeans, StarOffice. Your job here is to try to stem the loss that all of these systems are costing you, and keep from allowing their marketing teams from ruining the rest of your branding on profitable products.
  

Note that there are two growth markets in there (Modern Hardware and Open Storage), and the rest is all irrelevant tangents and legacy. The growth markets are where your future lies, and keeping the others around gives you the chance to migrate existing customers to the new platform, keeping your vision of a one-stop-shop IBM killer intact. But you have to be completely honest with yourselves: the existing stuff is legacy and will never go anywhere, and you need to pile resources into the growth areas without confusing your branding or customers.

What Sun Will Do
Here's my predictions:

• Sun will continue to price all their proprietary hardware so absolutely above the costs of generic hardware that only people under serious lockin to their platform even think about buying it, never allowing them to achieve any types of economy of scale.
• Sun will continue to give software products stupid, confusing names. I predict the Sun Java System Enterprise Database Suite being the new name for MySQL.
• Sun will continue to try to drive Solaris to everything through a neverending sequence of initiatives, confusing anybody even considering deploying it, so that you only ever hit the legacy market and Solaris die-hards.
• Sun will continue to invest in stuff that will never ever drive any meaningful revenue to them, but sap massive amounts of engineering resources. To try to justify this to their shareholders, they will come up with confusing branding and marketing initiatives to try to tie everything together.

In short, Sun, I have no fear that you will find some way to drag failure from the claws of oh-so-close. Just like you have for years.

Utility Computing in Finance: Regulatory Arbitrage

Financial Services companies (particularly those doing derivatives work) are quite big consumers of CPU power for all the computations that are necessary to support the business (and as some derivatives contracts amount to running millions of Monte Carlo simulations to capture low probability events, they can be pretty expensive to compute). For that reason, financial firms spend a lot of time working out how to minimize their computational costs and eke out every cycle that they can.

I had a conversation with my friend Nolan who knows quite a bit about the utility computing space, and thought it would be useful to the general public. I'm defining utility computing as renting out CPU time on other people's hardware (such as EC2). I'm not talking about in-house grid computing here, as you'll see when you look at some of the points below.

Broadly speaking, financial services computation boils down into the following areas:

• Latency Sensitive. These are computations that you need to do as quickly as possible, and are normally internally parallelized where possible. But within this, you have two more distinctions
• Market Latency Sensitive. These systems are usually automatically trading based on algorithms, and need to be able to execute trades as fast as possible with no human interaction. These guys are the ones buying ultra-low-latency MOM systems and networking switches.
• Human Latency Sensitive. These systems are presenting data to a human user who responds in some way, and you broadly have to keep up with the market and a human's reaction time (hint: updating a screen any faster than once per 100 milliseconds ignores basic human reaction time).
• Latency Insensitive. These are the computations that you have to run periodically, usually once a day as part of your P&L and risk cycles. They run overnight, and your basic job is to do all your computations within your computation window using as little expensive hardware as you can.

Of these, Market Latency Sensitive systems aren't a candidate for the current generation of utility computing: they operate within tolerances that current utility computing platforms can't possibly cope with. These are the guys who rent space at the same data center that their dominant exchange is located in to try to save every last millisecond (and are the reason why Lehman sold to Barclays for less than the value of its Manhattan data center). They are definitely candidates for a new generation of latency-sensitive utility computing (where the cloud guarantees that data will be delivered to your software within N microseconds of it arriving from the exchange), but until general purpose utility computing is prepared for financial services it isn't a starter because they would still face every issue below.

Human Latency Sensitive may be candidates for current approaches (once you're considering humans, you can start to think about acceptable delays for computations, and farm some things out to a utility computing cloud). Latency Insensitive definitely are (you just need the cycles once a night; as long as you finish within your window, you really don't care about anything else). Yet, financial services firms seldom use utility computing facilities of any kind.

That's actually a lie. They use them for everything that they can. The problem is that the regulators don't like them, so those uses are extremely limited.

As near as I can tell, here are the things that the regulators don't like about them:

• The machine isn't under your control. This means that, in theory, the utility company could be doing all types of nasty things with your execution flow and you wouldn't necessarily know anything about it.
• The data path isn't under your control. This means that you're sending (potentially) sensitive data outside your network, which someone might intercept and do something with (modify it, sell it to your competitors).
• You have no availability guarantees. If you own your own machines, you know if you have the CPU cycles to deal with all your available trades. Utility computing companies may lie to you and you wouldn't be able to deal with critical time periods.

These are not insurmountable! Rather, it requires a discipline and auditing routine similar to what financial services firms work with all the time. While it may be good enough for your Web 2.0 startup to just assume that Amazon isn't up to anything nefarious, it isn't good enough for the regulators.

The solution here I think is two-fold:

1. Regulators provide standards and guidelines. This would include the rules that they would require utility computing providers to adhere to, and facilities for independent audits of those procedures, precisely as financial services firms already have to for internal operation. I can think of a number of potential rules, but I'm not a regulator. They should come up with what they would be happy with, and they should be as equivalent as they can be between the four major financial services markets: London, USA (specifically New York and Chicago), Hong Kong and Tokyo.
2. Companies build utility computing platforms to those standards. I don't expect Amazon to start providing auditable logs of everything they do to EC2. I don't expect Google to adhere to external audits of their processes and procedures and security standards every 6 months. But those are value-add services that I'm sure somebody would be willing to do, because they could charge much more than Amazon or Google do. And those would be the firms that financial services firms could use for their utility computing needs.
   

Financial services firms are used to working within constraints. We're used to paying for products and services that can satisfy these constraints. But in this case, I think the regulators need to step up and provide guidance about what they would expect in this space so that the market can progress.

Meeting With Solace Systems: Hardware MOM

After my post on wanting an AMQP appliance, Hans Jespersen, Principal Systems Engineer with Solace Systems, got in touch with me and we met up yesterday when we were both in San Francisco, and I had a chance to talk tech with him on what precisely they do (as opposed to the marketing speak on their website). Hans is a pretty smart guy (ex-Tibco on the Rendezvous side rather than the EMS side), and seemed really technical and up on some of the other fringe technical topics we talked about.

Company Background And Overview
This is an Ottawa-based company, which is interestingly drawing from a group of ex-Telco engineers for the hardware side and a bunch of ex-MOM people (particularly Tibco, which gave them a big OEM win I'll talk about below). That's interesting to me because the two worlds haven't really interacted that much that I've seen, but there's a lot in common and a lot that I think both sides could probably learn from each other. I didn't ask about revenue or staff or anything else, I just had time to cover the technology itself.

The real goal of the company is to focus on low-latency messaging, which means that they do everything with a no-general-purpose-OS passthrough. As you'll see below, messages are delivered without ever hitting a general purpose OS or CPU, meaning that there's no chance to introduce random amounts of latency. But the same hardware approach allows them to also hit the high-volume and persistent messaging cases.

Base Technology Platform
The base platform is a pair of chassis, each running a customized motherboard, which has a number of PCIe slots in it (5 slots in the 2u 3230 and 10 slots in the 4u 3260). The motherboard itself has an embedded Linux system (with 8 cores actually), but at this point is only being used for management tasks and doesn't get involved in the message delivery path (though this wasn't true in the past, which is why there's such a beefy box to handle things like SNMP and web-based administration). By itself, the chassis is useless.

Each of the "blades" (which I'm going to call cards, as they're really just PCIe cards) provides a particular function to the box, but they're very strangely named on their website. These guys, rather than running general purpose chips, are all FPGA based. Here's what they do.

Network Acceleration Blade
This is the strangest name ever for what this thing does, because it's actually the primary message delivery platform for the box, as well as the only card that has any NICs (4 or 8 1GbE links) that can be used for messaging (the NIC on the chassis is just for management). This consists of:

• The network ports themselves
• A TCPoE on board
• Hardware (I'm not clear whether this is ASIC or FPGA based; everything else is FPGA, but these you can get off-the-shelf ASICs for, so I'm not sure what they did here) to do SSL and GZIP
• A multi-gigabyte RAM cache to store messages before they're delivered on
• An FPGA to do the protocol management
  

Interestingly, this is useless in and of itself, because the network card specializes in receiving messages, unpacking portions of the message to send over the bus to the other cards, and delivering them out. To actually route messages, you'll need one of the next one.

Topic Routing Blade
This is another FPGA-based card, which manages subscription lists of clients and has FPGA-accelerated regex-based subscription management. The way this works is that the Network Acceleration Blade extracts the topic name for a message from the message, and sends that over the internal bus to the Topic Acceleration Blade, which then responds to the NAB with the list of subscription channels (TCP sockets) to which the message should be forwarded (the message contents never travel over the internal bus in this case). Handles both RV-style wildcarding (.>) and JMS-style wildcarding (.#).

This isn't the only option for routing, which I'll get to separately, but assume that it's the only useful one for the time being.

These two cards are the only things that you need in a chassis to support basic transient store-and-forward MOM functionality. For persistence, you need the next one.

Assured Delivery Blade
This is a card which acts to ensure guaranteed delivery of messages, both in the persistent-sending case (don't acknowledge to publisher until the message can survive broker failure) and the disconnected-client case (if I'm down, store messages until I get back up). It's quite an interesting beast actually. This one has its own buffer of RAM, and two FC HBAs, and you would deploy this in a HA pair of chassis with a crossover between the two assured delivery cards. It's probably easiest to describe how it functions and you'll get the picture of what's in there:

• Message contents get delivered from the NAB to the Assured Delivery Blade over the chassis PCIe bus (yes, in this case the whole message body has to be transferred).
• The ADB sticks the contents into its internal RAM buffer
• The ADB sends the contents over the interconnect to the passive ADB in the other chassis
• The ADB acknowledges the message back to the NAB, which then is able to acknowledge back to the client that the message is persisted (more on why this works in a second)
• The ADB then batches writes (if necessary) to the underlying disk subsystem (they've only tested with EMC, he wasn't sure how high-end you had to go up the EMC product line) to clear the RAM buffer in a more optimized form.

I was a little sceptical about acknowledging the message just after the passive node acknowledges it (this in SonicMQ terms is DeliveryMode.NON_PERSISTENT_REPLICATED), but here's where having a pure hardware platform helps here. Each card has a big capacitor and a Compact Flash slot, and on failure, the capacitor has sufficient charge to flush the entire contents of the RAM buffer to the CF card, guaranteeing persistence.

It's a pretty clever model, and pretty indicative of what you can do with a dedicated hardware platform.

Marketing-Driven Blades
This is where things got a bit more sketchy. Note that in the above stack there's no way to do header-based message delivery (JMS Message Selectors). That's a pretty important thing if they're going to try to hit existing JMS customers who will have message selector-based systems. So they worked out a way to do that.

The problem here is that they added it with their Content Routing Blade, which is entirely XML based (the message selector language is XPath-based, rather than SQL-92-based). This is where he lost me and I told him that. While I'm sure this is great from a marketing perspective because it lets them sell into the Service Oriented Architecture Solution space, I think that space is rubbish, and the types of people who buy into it are not the types of people who are going to evaluate bespoke hardware from a minor vendor to increase message throughput. It also doesn't help porting existing applications, which I hit on in my initial AMQP analysis is one of the most important things you can do to try to drive early adoption of new technology.

They also have an XSLT-in-Hardware card (the Content Transformation Blade), but I'm so uninterested in hardware XSLT that I didn't bother talking to him about it.

Protocol Considerations
Given my writing about this, I did hit him pretty hard on the fact that right now they only have a proprietary protocol with binary-only drivers (C, C#, Java). The fact that it's a C and not C++ client library makes it much easier to port because you get to avoid the name mangling issues that are the worst in intermingling native compiled code, but there's nothing else you can use to talk to the box. They have a JMS driver you can use, but given that they don't support all of even the topic-half of JMS (like message selectors), I'm not sure of how much utility that is at the moment.

That being said, the fact that they've rolled with FPGAs and not ASICs means that they can flash the NAB to support more protocols (AMQP was specifically mentioned here). In fact, they've already done this by providing bespoke versions of the NAB and Topic Routing Blade to support the Rendezvous protocol natively under the Tibco Messaging Appliance brand name to Tibco. In that case, (if you're familiar with Tibrv), rather than using multicast or broadcast RV, you make a remote daemon (remote RVD) connection over TCP to the Solace box, which speaks remote RVD natively. Pretty cool, and Hans is pretty sure they're going to support AMQP once it standardizes.

Analysis
The things I particularly like here:

• Clever use of hardware. The use of FPGAs rather than ASICs is pretty good going, as is the custom motherboard with lots of fast PCIe interconnects. I also like that they've taken a really close look at the persistent messaging case, and leveraged the fact that they're in control of their hardware to ensure that they can optimize far more than a pure software solution could.
• Pure hardware data path. I like the use of external controller cards (the two routing cards) to minimize message flow over the bus, and that there's no general purpose CPU or OS touching the messages as they flow through the system.
• Speed. If his numbers are credible (and I have no reason to think they wouldn't be), they're hitting over 5MM 100b messages/second/NAB using topic routing, and 10MM subscription processes/second/card on the topic routing cards themselves. Persistence is over 100k persistent messages/second. That's pretty good and the types of numbers I'd expect from a hardware solution.

Things I don't like:

• The Content Routing/Transformation stuff. Pure marketing claptrap, and utterly silly. If you know you need a low-latency hardware MOM system, are you actually going to take your nice binary messages with a custom protocol and add the Slow of XML for routing? I can see throwing compressed XML around as your body content, but routing based on it? Taking binary fields and converting them to text just so that you can have XPath-based message selectors? That doesn't seem right to me. I'd be much happier if they gave a key-(binary) value map-based system, which would map much more naturally onto AMQP, JMS, and Tibrv models that we're currently coding against. That makes it hard to port existing systems, which makes it hard to get adoption.
• Proprietary Protocol. Yes, I hate these with a passion as they keep biting me in the ass every time I have to use one. You're a hardware company. If you actually think that giving me source-code access to your C client library is going to expose so much of your mojo that someone can hire up a bunch of FPGA guys and replicate your hardware that quickly, then someone will just reverse engineer it anyway. Lame decision.

Conclusion
Am I going to rush to buy some? No. I actually don't live in the ultra-low-latency space in my day job, and the technical decisions they've made would make porting my existing applications sufficiently painful that I'm not willing to go down that path for an eval when my current solution works.

Should you? I'm not 100% sold. The core direction of the product looks pretty darn good to be honest. Without seeing their proprietary client library, I can't tell you how easy it would be to port an existing JMS/Tibrv/AMQP application to their programming model for an evaluation. If I knew I was going to be working in super-low-latency space and I didn't mind an extended POC, I'd definitely add it into my evaluation stack.

But if I had an existing Tibrv application that was killing me in production (which most eventually will), I'd definitely take a look at the Tibco Messaging Appliance as fast as I possibly could. Anything that will kill off multicast and broadcast-based RV is ace in my books.

Some Interactions Are Only Indirectly Profitable

I met with Ari Zilka from Terracotta Technologies yesterday in their offices in San Francisco for a follow-up meeting from a series of meetings I had had with Ari and our sales representative over the course of the past year. My interaction with Terracotta has largely been one of "This technology really is a game-changer; I just have to find my personal game that it changes," and I've been working with Ari ever since on the first entrance path for Terracotta into my company's infrastructure.

Ari's a busy guy. He's a founder and CTO of a software startup, and having been there, I know how difficult it is for him to devote time to anyone. And I've now had three meetings with him over the course of about 9 months. And my employer has not given them a single dollar as of yet, and there's no guarantee that even if we end up rolling into production on top of Terracotta, that my employer will stump up cash to them for an open source technology (FTR, this is the company that inspired the original Open Source Cookies post). This is probably a frustrating situation for the sales guy, because sales guys have numbers and targets and need to make money, and the sales guy needs to dole out his fraction of Ari's time in the way that's going to maximize his commission. I'm clearly not that as of yet. And yet there I was for the third time.

Why?

What's the rationale for his wasting yet more time talking to me?

Having discussed some similar issues with Laura Khalil from Atlassian yesterday when I met with her earlier in the day (more on that meeting anon), I think things started to gel in a more concrete way, because they face these problems as well from a non-Open Source perspective.

The rationale here is that some interactions are only indirectly profitable, but the indirect benefits potentially vastly outweigh the direct ones. So you pursue them anyway if you're an open company; a closed company won't.

Traditional Sales Model
Consider the traditional Enterprise Software (deal supporting direct sales force, or > $100k licensing) sales model:

1. Company sends out feelers to vendors
2. Vendors send representatives
3. Company goes ahead with due diligence/POC with one or more vendors
4. Company starts negotiations
5. Deal is signed
6. All vendors move on

In this model, you have a discrete lifecycle of a particular sale, and most players are only around for the sale; after the cash changes hands, you're into support land, which is at least partially an insurance business. The company projects to the vendors its rough budget, the vendors determine how to maximize their portion of the budget and whether the deal is even worth pursuing. Knowledge is largely gained by the customers interacting directly with the vendors, or maybe checking with some research firms.

Note here that the vendors themselves pull out of the conversation the moment they realize that they don't want the particular deal: it's too small, they're not the right solution, whatever. Then they go back to their closed external appearance, and go back into information embargo. If they can't make a sale, why waste anybody's time on the interaction?

I think any company that thinks this way and is trying to sell to technologists is going to fail and fail hard. And I think split open/closed source companies are best suited to be able to leverage this.

I'm A Bad Customer
For any commercial software company, from a sales perspective, my employer is not their ideal customer. We're not that big for a financial services company (our parent company is, but we have our own technology stack and purchasing departments). We don't buy way more than we need. We don't like shelfware. We like best-of-breed, and don't buy whole software stacks (we will never buy a Service Oriented Architecture Solution). Our technologists are massively involved in sales decisions, even when it's really a business-facing application. We constantly evaluate software in build-vs-buy mentality (and we as a culture like writing software). We're small fry, and we're an expensive (from the vendor's perspective) sale.

But, that being said, we probably are a good candidate for a sale that influences others. We are passionate about technology. We have lots of technical contacts (friends, ex-coworkers) with people at much larger companies. We have people who are involved with lots of online technical communities. We have people who do open source work in their spare time. We have people who blog, both positively and negatively. We go to user groups. We engage vendors constantly on product improvements. We take betas and alphas and developer cuts all the time.

That means that ultimately getting us on your side (even when we don't give you a single dollar in revenue) ends up influencing a lot more people than even a single larger sale would.

So when you look at the overall picture, it starts to make sense for Ari to spend time talking with me, even without a Big Ticket Sale right in front of him. And I think that would be true of any open source technology.

Turn Indirect Profit To Direct Revenue
If you're selling anything commercial having to do with an open source product, I think you'll find that a significant proportion of your most technically savvy users are ones who will never pay you anything. They're working at home; they're in academia; they're smart but working for a poor company; they come from a less developed country. They're a massive source of improvements and knowledge and they get passionate about what they're doing, but they're not going to pay you. But they're a pretty good reason why you'll eventually get revenue from other people: they may go work for a bigger/richer company, or one which prefers Buy in Build-vs-Buy; they talk with the world constantly about what they're doing; they speak at conferences and write books and make your platform far more compelling than it otherwise would be. And so if you want to be successful, you view supporting them and interacting with them as indirectly profitable: no revenue comes from it, but it increases the profit potential of your ecosystem dramatically.

So where's the relevance with Atlassian here? They're not open source. They sell software. How have they leveraged these principles to end up with Laura wasting her time meeting with me?

• They give away licenses all the time. You're open source? Free license (this is how I originally found out about Jira way back in the day). You're working for a non-profit? Free license. You just want to use it for your own personal stuff? Free license. This builds a passionate ecosystem and doesn't stop you extracting revenue from companies that will pay you.
  
• They work with open source programmers. They have a plug-in ecosystem that they actively nurture, and many of those people are doing it open source. Those same passionate people making their ecosystem more attractive and more conducive to extracting revenue from others.
• They engage their customers. Laura knew I'd blog about at least part of what we talked about (that's how she found me in the first place). That increases the sum knowledge that the world has about Atlassian products, and makes for free marketing.
• They differentiate their customers. Some customers are a lot of dumb money, and some customers are a small amount of smart money. I'd like to think my firm is more of the latter, given the amount of time we devote to trying in any way to help make the products we use better.

If you're familiar with the classical Tactical/Strategic Sale quadrant, there are a whole host of people who are so tactically worthless sales wise that they're going to give you nothing. But strategically they're useful. Pursue them.

If you're a technology company and you're trying to play the old closed-information game, the every-interaction-must-be-profitable game, the "no you can't have the manuals unless you're a customer" game, the "no you can't download our whitepapers from a gmail.com email address" game, you're losing out a lot. Engage people who are only indirectly profitable and you'll find more that are directly profitable.

Perforce, FishEye, CC.Net, Labels, Oh My!

My employer is a big Perforce shop. Although we've got some CVS repositories still lying around for legacy reasons, over the past few years almost all of our source code has managed to make its way into four Perforce instances (separated for geographical and organizational reasons). We use it for pretty much everything, and we've got our entire software development methodology based around its use.

That includes a number of tools that we use that are integrated with it:

• Code review tools (all built in-house)
• Software release and distribution tools (also all built in-house)
  
• Bamboo, as our current continuous integration system
• FishEye, as our SCM web-based visualization system
• Jira, as our issue tracking system

In addition, in the past, before we moved to Bamboo, we had three other CI systems hitting it, CruiseControl, CruiseControl.NET, and Hudson (all but one retired, and that one is being retired shortly).

All of them are hitting it on a regular basis pretty hard to pull metadata (what happened when) and data (to pull the actual SCM data). This is a tale of where it started to go wrong and how we fixed it.

Problem Diagnosis
We noticed that FishEye was behaving pretty badly when it was rescanning a repository. We do this on a regular basis, because FishEye won't automatically rescan changelist descriptions (at least for Perforce), and sometimes we go back and edit changelist descriptions to hook them up to Jira issues or end-user requests, or just to provide more clarity on what someone did in the past. Since FishEye won't pick up those changes (if it's already processed changelist 123456, why would it process it again?), we have to periodically completely rescan our master Perforce server to pull out all changes.

Our master Perforce installation is relatively big for a non-gaming house (I can't give numbers for this in a public forum), and rescanning it was taking several days. In fact, it was taking so long, that we couldn't even plan on doing it over a weekend: if we kicked off the process on Friday night after New York went home, it wouldn't be done by the time Hong Kong and Tokyo came in on Monday. This was a problem.

Also, when Fisheye was doing this, the whole repository was noticably slower. Since the rescans started to take place over normal business hours when people were trying to work, this made the problem doubly bad: not only was Fisheye not available, it was making Perforce slow for users during the rescan.

So I started diagnosing what was going on, and the process that was taking the longest was processing labels. This alone was taking over a day, and because of the way Fisheye does this, and because forking to query Perforce on Solaris 10 is a painful experience, we needed to get this number down. We had way too many labels covering way too many revisions.

Perforce Is Not CVS
The metadata table that holds this data in Perforce (db.label) was absolutely massive: roughly 7GB, or about 60% of our entire Perforce metadata storage. This wasn't good, and it's far from ordinary. When I started investigating, we had over 12000 labels. That's a lot for the number of projects we're hosting and the number of releases we've done, but it turns out that 10000 of them were created by CruiseControl.NET builds.

This was largely done from a misconception of what labels are good for, and is basically a remnant of CVS-style thinking. In CVS, because revisions of files can be interleved together, if you want to reference the state of a particular subsection of the repository as of a particular point in time, you have to add a Tag to every revision of every file involved. This actually goes in and adds metadata for that revision to say it's part of BUILD_5 or some such thing.

A Perforce Label is different. Perforce has atomic, monotonically increasing changelist numbers, where each number uniquely identifies the state of every single revision in every single file in the entire server. And I can use them in all types of contexts. In particular, I can pull down the state of a particular project as of a particular changelist number: "Give me the Fibble project as of changelist 12345." This is how Perforce-integrating CI systems work: they query Perforce to say "tell me all the submitted changelists that I haven't seen", and then sync up particular areas as of those changelist numbers. Therefore, a changelist number is the equivalent of a tag applied to every revision of every file in the whole server.

A Label, on the other hand, is there for cases where you need to refer to revisions of files across multiple changelists. The key use case here is patching. Let's say that you've released version 1.2.0 of your software, and then you start adding changes to support 1.2.1. But in the meantime, you discover a complete showstopper bug that requires you to put out a special release with only that bug fix in, and not any of the other changes you've got prepared for 1.2.1. Since the 1.2.1 features have already started going in, if you try to pull all the source code as of the point where the critical bug fix went in, you'll get the 1.2.1 changes as well. In this case, you create a Label, and you put in the label all the 1.2.0 revisions, as well as the revisions just for the showstopper fix, but none of the rest of the 1.2.1 changes. This gives you a way to refer to a collection of revisions of files across different changelists.

What our CC.Net server was doing (and as I didn't install it, I don't know if this was default or intentional on our part), was for every single build of every single project, creating a new label which contained the state of all the files for that build. But you don't need to do that: since it was pulling everything down as of a particular changelist number, all that accomplished was saying "The revisions for build 30 of the Fibble project are the same as the files as of changelist 12345," which Perforce changelist numbers already give you. So it was unnecessary metadata that was clogging up the server intolerably.

Solution
We deleted all of those 10000 labels (the fact that we had already moved all those projects to Bamboo made this a no-brainer, as we were no longer using the CC.Net provided builds at all). But the size of the db.label table didn't actually shrink. In fact, it actually got slightly bigger during that time.

This is because Perforce as an optimization assumes that you're constantly increasing the amount of metadata that you're putting in as time goes forward, and so doesn't prune the size of the tables. So they were still too big, and sparse, so although we didn't have to do as many queries across them, it was hurting the OS caching of the files.

The solution there is to restore from a checkpoint (a Perforce checkpoint is a gzipped text file containing every binary record in your metadata in plain text; it acts as your primary backup mechanism for the metadata records that it keeps along with your RCS files). Before we did this we went through a pruning process for old workspaces that people hadn't deleted (removing several hundred) to get the db.have file down in size as well.

After this was done, the size of our db.* tables went from 11GB to 3.0GB (our gzipped checkpoints went from 492MB to 221MB). And the FishEye scans went from 3 days to 5 hours. Job done.

Morals
So after all that, what we can draw from this is:

• Don't label unnecessarily. They're relatively expensive in terms of metadata, and you probably don't need to do it.
• Shrink your metadata. Remove anything that you no longer need, including old workspaces.
• When you prune significantly, restore from checkpoint. This is the only way to get Perforce to really shrink down your metadata tables on disk.