2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!

My Open Letter to the European Competition Commissioner

Monty urged me to help save MySQL. I couldn't possibly refuse such an offer.

Dear Competition Commissioner:

I am the Chief Executive and Technology Officer for OpenGamma, a financial technology startup located in the United Kingdom. I am writing to urge you to immediately and unconditionally approve the merger of Oracle and Sun.

I have a long standing history with both the Open Source and Database communities, having worked for a number of database startups in the United States of America, as well as working at early-stage companies making use and refining a number of Open Source technologies. I also have experience working at Oracle during a summer internship while I was attending the University of California at Berkeley.

Furthermore, in my more recent career as a consumer (rather than producer) of database technologies, I have setup and managed numerous, large MySQL installations, including one with more than 10 nodes and 150GB of active data (and several terabytes of archive data) in the financial services industry. I have also been a customer of Oracle's database technology for systems even larger.

In my mind, there is no logical reason to reject this merger based on considerations for the MySQL technology.

First of all, as a consumer of database technologies, I can tell you that Oracle and MySQL simply do not compete in the marketplace. While customers have replaced Oracle with MySQL, the applications based on Oracle that were ported to MySQL were never good candidates for Oracle and would have been ported to another database engine in due course as Oracle moves to the highest end of the market. Customers have numerous options for porting their applications off of Oracle onto a lower-cost database engine; MySQL simply has the most brand recognition in this space. Furthermore, MySQL has been used as a pricing lever by Oracle customers rather than an active option for migration.

That implies that Oracle's ownership of MySQL might see a reduction in competition for Oracle's core product, but there is a flourishing Open Source database market these days (which there wasn't when MySQL was originally created): Ingres, Firebird, LucidDB and PostgreSQL are all far more applicable to the Oracle customer base than MySQL is. Even if MySQL development were to come to an immediate halt, this wouldn't harm consumers in a such an extremely competitive environment.

However, the continued uncertainty over the Sun acquisition is potentially far more anticompetitive for consumers of technology as a whole: Sun has a number of competitive products with far greater applicability than MySQL (including their storage, networking, and computer chip technology). Allowing those products to die because Sun ran out of cash during this phase of its life would be extremely and permanently damaging to the overall computer industry inside Europe, and reduce competition significantly. Delaying this merger over the matter of MySQL would result in far greater anticompetitive results to European consumers of computing technology than even the worst case arguments of biased, self-interested advocates in this matter.

Thank you for your time, and once again, I urge you to approve this merger unconditionally and without further delay.

Sincerely Yours,

Kirk Wylie
Chief Executive Officer
OpenGamma

Document Stores: Please Give Me A Standard API

Although I'm a long-standing RDBMS guy (having worked on Broadbase, Kidar, and Eigenbase/LucidDB [indirectly through Kidar and Broadbase]), I'm quite excited by the emerging document-oriented database movement. While I'm not a pure "SQL is bad and old-economy and you should throw it away" guy, I do think that document stores, like their Hierarchical Database precedents, have their uses in modern architectures. In particular, practical (as opposed to theoretical) aspects of the use of a hierarchical/document model allow for advanced scalability and performance optimizations to be made for modern scale-out architectures.

That being said, even though we're at early stages, I think the major proponents of the technique need to learn from the RDBMS guys in one important aspect: unified APIs are key to widescale adoption.

If I'm writing an application that's going to be backed by a relational database, if I'm in a sensible programming language, I've got a standard API that I can code against: JDBC, ODBC, ADO.NET, et. al. It doesn't shield me entirely from differences in the underlying database implementation (or else there would be no opportunities for product differentiation), but it makes those differences minimal and relatively easy for a software developer to abstract.

Ditto for message oriented middleware: I can use JMS at the code layer, or AMQP at the network layer (and thus at the code layer as well). While different MOM implementations have underlying differences, which are particularly obvious if I want to push the technology to its limits, the product-specific differences are noticeable in the breach rather than in the general.

This isn't the case right now for document stores. I know that MongoDB and Riak and CouchDB and SDB and others (which I'm sure commenters will point out below) are pretty darn similar in their functionality. I know that the conceptual models are relatively similar. I know this logically. But I still have to do custom code for each one for my own application.

With multiple implementations out there, and with users (e.g. me) looking at the different systems and seeing them logically similar, it appears that it's probably time for the teams to start working together and come up with a code-level API that I can code against, much like JDBC or JMS. While this might seem like early stages for such an effort, trust me, it'll greatly lead to increased adoption because the perceived costs of evaluating different implementations will be greatly reduced.

It doesn't mean that you can't differentiate; it doesn't mean that you can't be superior or inferior to other implementations. But it does mean that I, as a consumer of these systems, can more easily support multiple implementations. And if I can do that, I'm more likely to move to one in the first place.

Giving Thanks 2009

I wrote this on a plane over the Atlantic heading to Chicago to visit my family for Thanksgiving, followed by a trip to San Francisco (my spiritual, if not familial, home). Maybe it's the maudlin sentiments from watching Julie and Julia, but I realized that I've got a lot to be thankful for this year.

The last time I was coming into Chicago was on a flight from San Francisco, all of 6 hours after I had found out I didn't have a job to come home to. It wasn't a particularly easy trip for me, and it was a pretty bittersweet trip to see my family not really knowing what I was going to do when I returned back to the UK (other than show up to be formally made redundant). Really, I didn't feel like I had a whole heck of a lot to be thankful for this time last year.

This year, I feel like things are completely different.

When I got back to London I immediately set about taking the idea which was OpenGamma and converting it to a Real Company. I pulled in two amazing cofounders, we started doing formal market research, refined the business model and strategy, and started working with potential funding sources. As I've mentioned, we closed a Series A round of investment, have started building the system that was little more than an idea and prototypes, and are pushing as hard as we can to make this a successful company.

It wasn't clear this time last year that it would happen, and a lot could have gone wrong along the way (more than actually did). I'm in a particularly rant-free mood, so let me give some thanks here.

First of all, I want to thank Jim and Elaine for joining me on the OpenGamma journey. Startups really are about founders, and I want to thank Jim and Elaine for trusting me when OpenGamma was nothing but my late-night emails and wildly-gesticulating concepts outside The Audley. You guys took a leap of faith and there's no way that we'd be where we are without all three of us working well together.

Second, I want to thank everybody who helped us pre-incorporation. We talked to a lot of people, and bounced a lot of different ideas off a lot of different people. Without people spending the time to talk to a bunch of crazy guys looking to disrupt financial technology, we never would have come down with a vision capable of being executed into a company. You guys spent time with us when you didn't need to, and when there wasn't any clear payoff to you. Thank you for spending the time and helping us refine our ideas.

I want to thank my family for standing by me and having faith in me and what we were doing. It hasn't been an easy year for all of you guys, but you've been rocks when I've been wavering and had doubts and went through difficult times. It's tough for me to keep in touch from 5,000 miles and 6 time zones, and I don't see my nieces and nephews anywhere near as much as I should do, but thanks for being there for me.

For the people at Big Bank B who took yet another leap of faith, thanks for not only giving me the opportunity to work for you while I was cementing OpenGamma, but for supporting me while I was doing it. I know I wasn't the typical candidate you were looking for, and I only hope that despite my abrasive nature and mercurial temperament I managed to perform at the level you were hoping. Your trust in bringing me in for a contract that gave me the freedom to actually launch OpenGamma hasn't been forgotten.

For my investors and board members, thanks for committing your time, energy, intellect, and capital in OpenGamma. Together, I think we'll create a fantastic business, and I thank you for joining Jim, Elaine and I on the journey.

Finally, I want to thank my husband. I know I've been all over the place over the past 12 months; I know at times I've been completely impossible to be around. I also know the sacrifices you're making giving up your partner to that harshest of mistresses, the early stage startup. But at the most critical time, when OpenGamma was one phone call from not happening, you were there for me. Sometimes I might seem like I don't appreciate you as much as I do, but trust me, I do. Thank you most of all for just being there. Without you, I doubt I would have had anything to be thankful for over the last year.

So yeah, I guess I do have a lot to be thankful for this year.

For all my USAmerican readers, eat some extra Thanksgiving dinner for me. For all my non-USAmerican readers, get yourself some turkey.

And don't worry, I'll be back to my usual abrasive, obnoxious, sarcastic, snarky, angry, ranty self soon enough.

More Adventures In Unethical Recruiting

Let's run through an entirely hypothetical situation for a second.

You're a recruiter. You find out that there might be a newly funded financial technology startup that's hiring, perhaps by looking at Blog Posts or LinkedIn Profiles. You think "hey, that's a great opportunity to get some business."

You then proceed to start calling engineers and telling them that there's a recently funded financial technology startup in stealth mode that's hiring, and they'd be brilliant to work for, and get some strong expressions of interest, including CVs.

You then contact the CEO of said company through LinkedIn and say "Hey, I see you're hiring, and I've got some candidates that are already interested!" The CEO might in theory point out to you that you in fact do not have a Master Services Agreement or any other contractual relationship with said startup, and that he's not interested in any candidates that have come through that might trigger a financial payout with a firm with which the startup lacks a pre-existing business relationship.

Now you're in a bind. The CEO flat-out says that he's not willing to sign any type of contract with you, but you've already told the candidates that you can get them introduced to the firm. You've been caught out in your lie.

I suppose what you could do is come clean with the candidates and just tell them the name of the firm and its CEO so that they can get in touch directly. In fact, the CEO might have even suggested this as a way forward for you. But I doubt you're going to do this. If you were interested in acting ethically, you might have contacted the CEO before lying to candidates.

I think more likely what you're going to do is tell the candidates that the CEO isn't interested in them in particular. And for that, and the situation that led to this mess, you earn a merit badge in Unethical Recruiting to add to your stack. You've also earned the wrath of said CEO, who will proceed to make sure that the reputations of both yourself and your firm are effectively ruined with everybody said CEO knows.

Hypothetically, of course.

Old posts now have comment registration

After a prolonged attack on a few of my older posts (and not the ones with the most traffic either; literally just three of them, but over 10 spam comments a day), I've turned on Registration to comment on the older ones.

Newer posts, all covered by Disqus, won't be affected.

Announcing the Release of the Fudge Messaging Project

Along with the rest of the OpenGamma team, I'm pleased to announce the first open release of a project we're calling Fudge Messaging [1].

What Is Fudge?

Fudge came out of a desire for a system to encode data which is:

• Hierarchical so that you can represent common document idioms (XML or JSON) using it
• Typesafe by storing the type of data along with the payload.
• Binary for compactness on the wire or on disk.
• Self-Describing in its contents so that you can introspect a message without knowing anything about any schema used for its production

In the past, we've looked at a number of different encoding schemes, and rejected them for various reasons:

• Google Protocol Buffers and Thrift are great if you have access to to the schema, and know what the type of messages on a particular communications channel are. But because they're not inherently self-describing, you can't build interesting middleware around their flow.
• XML and JSON are big (if you don't gzip them) and slow (if you do). Plus by being text-based, you have to convert everything to and from text.
• ASN.1 is conceptually nice in a lot of respects, but it's become such a niche that the tooling is utterly rubbish, and I don't see it getting any better.

Ultimately, for those of you familiar with Tibco Rendezvous, we really liked the TibrvMsg system, but couldn't use it because it's proprietary, and the implementations are surprisingly slow for something designed for low-latency messaging.

What Is The Fudge Project?

The Fudge project consists of the following major components:

• An Encoding Specification, conceptually similar to ASN.1 BER, DER, or PER, which specifies how to encode data in a linear sequence of bytes.
• A set of Reference Implementations, which adhere strictly to the encoding specification and provide access to Fudge-encoded data in a number of programming languages.
• Tools to make it easy to work with data encoded in the Fudge format.

OpenGamma is sponsoring the Fudge project, but in the classical Open Source way: we found something that was missing out there, built it to suit our needs, and have the capacity to help it grow while we use it.

Project Status

I'm not going to kid you: this is a very early stage project. We have Java and C# reference implementations (and the C# one, for example, is more advanced than the Java one in a lot of respects, mostly because it's leveraging relatively new features in the C# language). We have a wiki and Jira installation. We have code up in Github. But that's about it.

Don't let me mislead you: the Java reference implementation, for example, is rock-solid in stability and very good in performance; we use it daily here at OpenGamma. But we don't have polished releases and distributions and everything else as of yet. That will come, but we wanted to get the code out there as soon as we reasonably could.

License

We're releasing the code under the APLv2 because we believe that it's the best type of license for this type of work: it's developer friendly, as well as employer friendly.

The specification is free to use for everyone. If you're going to use the ideas but not the full specification we'd ask you to not call your project Fudge Compliant [2], but anyone can create their own Fudge implementation using whatever license they want.

Next Steps

If you're interested, take a look at the wiki, grab the code, and take a look. I hope you find it useful.

From our perspective, we're going to continue to develop the system and grow it.

Footnotes

[1]: Fast Unstructured Data Generic Encoding
[2]: Fair Disclosure: we are considering getting a trademark on the use of the word Fudge for messaging to allow us to ensure that only compliant implementations can use the name.