Confluence, iSCSI, NetApp, Flexiscale, Fail

We've been hosting the FudgeMsg website using Confluence (and mad props for the free Open Source license by the way!) on a VM hosted by Flexiscale. We chose Flexiscale for the following reasons:

• Confluence is ridiculously tricky to cluster, meaning that you can't benefit from the scale-out capabilities of the Amazon EC2 model. (Edit 2009-12-24 - We're not attempting to cluster it; this statement is to indicate that because it's so tough to cluster, we're not trying to. Confluence is thinking we might be, and crashing as a result).
• Getting your AMI+EBS+ElasticIP for such a single-point vendor app is very difficult and time consuming, again meaning that the Amazon EC2 model isn't ideal.
• We didn't want to go for dedicated hardware for a low-volume application stack.
• We don't have a stable enough internet connection to host the application stack ourselves.

We thought we were pretty darn clever to be honest, but then we noticed that things started going wrong. In particular, we started getting this error a lot:

We were running a relatively complicated Java setup (jsvc to run as chrooted user, on Tomcat, with multiple web applications running), so we thought we had done something wrong. After all, the first rule in software engineering is always, always, assume you have caused the break.

We were wrong.

What we've since found out is that the entire Flexiscale model is that your VMs have effectively no local storage whatsoever, and everything is iSCSI hosted off of a NetApp cluster. That means that your storage is fully persistent, and survives migrations of your VMs across physical hardware. Which is a great concept when it works.

Except that the Flexiscale NetApp cluster is borked. Essentially, sporadically the iSCSI services will completely shut down. Sometimes for a second, sometimes for 4 hours. Your processes will still be running, but if they attempt to hit disk for any reason, they'll hang and ultimately timeout. Your processes are still running, but they can't talk to disk.

In the case of Confluence, which was actually trying to talk to PostgreSQL, which itself was trying to talk to disk, Confluence detects the complete timeout hang of PostgreSQL as a cluster violation, and puts itself into crash mode.

You want to know the best part of this? When you're in this state, there's nothing you can do. You can't SSH into the VM, because it can't read the password file to let you in. The Flexiscale tools won't even let you hard bounce the machine. And they won't tell you when it's back up directly, so you just have to keep trying until you can finally get into the VM, to restart your servlet container.

This has caused no fewer than 20 instances where Confluence has died for us, sometimes lasting hours until we can actually recover the VM, and twice now in 2 days. It makes us look like morans who can't even run Confluence, much less guide development of a message encoding system.

So until we manage to get off of Flexiscale (haha, can't even get in to back up the data at the moment), if you see that error when going to the Fudge Messaging website, now you'll know why.

There are two morals of the story:

• If you get a Non Clustered Confluence : Database is being updated by another Confluence instance error, and you know that's not the case, make sure your storage is responsive;
• Don't host anything on Flexiscale until they sort their hardware out.

2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!

My Open Letter to the European Competition Commissioner

Monty urged me to help save MySQL. I couldn't possibly refuse such an offer.

Dear Competition Commissioner:

I am the Chief Executive and Technology Officer for OpenGamma, a financial technology startup located in the United Kingdom. I am writing to urge you to immediately and unconditionally approve the merger of Oracle and Sun.

I have a long standing history with both the Open Source and Database communities, having worked for a number of database startups in the United States of America, as well as working at early-stage companies making use and refining a number of Open Source technologies. I also have experience working at Oracle during a summer internship while I was attending the University of California at Berkeley.

Furthermore, in my more recent career as a consumer (rather than producer) of database technologies, I have setup and managed numerous, large MySQL installations, including one with more than 10 nodes and 150GB of active data (and several terabytes of archive data) in the financial services industry. I have also been a customer of Oracle's database technology for systems even larger.

In my mind, there is no logical reason to reject this merger based on considerations for the MySQL technology.

First of all, as a consumer of database technologies, I can tell you that Oracle and MySQL simply do not compete in the marketplace. While customers have replaced Oracle with MySQL, the applications based on Oracle that were ported to MySQL were never good candidates for Oracle and would have been ported to another database engine in due course as Oracle moves to the highest end of the market. Customers have numerous options for porting their applications off of Oracle onto a lower-cost database engine; MySQL simply has the most brand recognition in this space. Furthermore, MySQL has been used as a pricing lever by Oracle customers rather than an active option for migration.

That implies that Oracle's ownership of MySQL might see a reduction in competition for Oracle's core product, but there is a flourishing Open Source database market these days (which there wasn't when MySQL was originally created): Ingres, Firebird, LucidDB and PostgreSQL are all far more applicable to the Oracle customer base than MySQL is. Even if MySQL development were to come to an immediate halt, this wouldn't harm consumers in a such an extremely competitive environment.

However, the continued uncertainty over the Sun acquisition is potentially far more anticompetitive for consumers of technology as a whole: Sun has a number of competitive products with far greater applicability than MySQL (including their storage, networking, and computer chip technology). Allowing those products to die because Sun ran out of cash during this phase of its life would be extremely and permanently damaging to the overall computer industry inside Europe, and reduce competition significantly. Delaying this merger over the matter of MySQL would result in far greater anticompetitive results to European consumers of computing technology than even the worst case arguments of biased, self-interested advocates in this matter.

Thank you for your time, and once again, I urge you to approve this merger unconditionally and without further delay.

Sincerely Yours,

Kirk Wylie
Chief Executive Officer
OpenGamma

Document Stores: Please Give Me A Standard API

Although I'm a long-standing RDBMS guy (having worked on Broadbase, Kidar, and Eigenbase/LucidDB [indirectly through Kidar and Broadbase]), I'm quite excited by the emerging document-oriented database movement. While I'm not a pure "SQL is bad and old-economy and you should throw it away" guy, I do think that document stores, like their Hierarchical Database precedents, have their uses in modern architectures. In particular, practical (as opposed to theoretical) aspects of the use of a hierarchical/document model allow for advanced scalability and performance optimizations to be made for modern scale-out architectures.

That being said, even though we're at early stages, I think the major proponents of the technique need to learn from the RDBMS guys in one important aspect: unified APIs are key to widescale adoption.

If I'm writing an application that's going to be backed by a relational database, if I'm in a sensible programming language, I've got a standard API that I can code against: JDBC, ODBC, ADO.NET, et. al. It doesn't shield me entirely from differences in the underlying database implementation (or else there would be no opportunities for product differentiation), but it makes those differences minimal and relatively easy for a software developer to abstract.

Ditto for message oriented middleware: I can use JMS at the code layer, or AMQP at the network layer (and thus at the code layer as well). While different MOM implementations have underlying differences, which are particularly obvious if I want to push the technology to its limits, the product-specific differences are noticeable in the breach rather than in the general.

This isn't the case right now for document stores. I know that MongoDB and Riak and CouchDB and SDB and others (which I'm sure commenters will point out below) are pretty darn similar in their functionality. I know that the conceptual models are relatively similar. I know this logically. But I still have to do custom code for each one for my own application.

With multiple implementations out there, and with users (e.g. me) looking at the different systems and seeing them logically similar, it appears that it's probably time for the teams to start working together and come up with a code-level API that I can code against, much like JDBC or JMS. While this might seem like early stages for such an effort, trust me, it'll greatly lead to increased adoption because the perceived costs of evaluating different implementations will be greatly reduced.

It doesn't mean that you can't differentiate; it doesn't mean that you can't be superior or inferior to other implementations. But it does mean that I, as a consumer of these systems, can more easily support multiple implementations. And if I can do that, I'm more likely to move to one in the first place.

Giving Thanks 2009

I wrote this on a plane over the Atlantic heading to Chicago to visit my family for Thanksgiving, followed by a trip to San Francisco (my spiritual, if not familial, home). Maybe it's the maudlin sentiments from watching Julie and Julia, but I realized that I've got a lot to be thankful for this year.

The last time I was coming into Chicago was on a flight from San Francisco, all of 6 hours after I had found out I didn't have a job to come home to. It wasn't a particularly easy trip for me, and it was a pretty bittersweet trip to see my family not really knowing what I was going to do when I returned back to the UK (other than show up to be formally made redundant). Really, I didn't feel like I had a whole heck of a lot to be thankful for this time last year.

This year, I feel like things are completely different.

When I got back to London I immediately set about taking the idea which was OpenGamma and converting it to a Real Company. I pulled in two amazing cofounders, we started doing formal market research, refined the business model and strategy, and started working with potential funding sources. As I've mentioned, we closed a Series A round of investment, have started building the system that was little more than an idea and prototypes, and are pushing as hard as we can to make this a successful company.

It wasn't clear this time last year that it would happen, and a lot could have gone wrong along the way (more than actually did). I'm in a particularly rant-free mood, so let me give some thanks here.

First of all, I want to thank Jim and Elaine for joining me on the OpenGamma journey. Startups really are about founders, and I want to thank Jim and Elaine for trusting me when OpenGamma was nothing but my late-night emails and wildly-gesticulating concepts outside The Audley. You guys took a leap of faith and there's no way that we'd be where we are without all three of us working well together.

Second, I want to thank everybody who helped us pre-incorporation. We talked to a lot of people, and bounced a lot of different ideas off a lot of different people. Without people spending the time to talk to a bunch of crazy guys looking to disrupt financial technology, we never would have come down with a vision capable of being executed into a company. You guys spent time with us when you didn't need to, and when there wasn't any clear payoff to you. Thank you for spending the time and helping us refine our ideas.

I want to thank my family for standing by me and having faith in me and what we were doing. It hasn't been an easy year for all of you guys, but you've been rocks when I've been wavering and had doubts and went through difficult times. It's tough for me to keep in touch from 5,000 miles and 6 time zones, and I don't see my nieces and nephews anywhere near as much as I should do, but thanks for being there for me.

For the people at Big Bank B who took yet another leap of faith, thanks for not only giving me the opportunity to work for you while I was cementing OpenGamma, but for supporting me while I was doing it. I know I wasn't the typical candidate you were looking for, and I only hope that despite my abrasive nature and mercurial temperament I managed to perform at the level you were hoping. Your trust in bringing me in for a contract that gave me the freedom to actually launch OpenGamma hasn't been forgotten.

For my investors and board members, thanks for committing your time, energy, intellect, and capital in OpenGamma. Together, I think we'll create a fantastic business, and I thank you for joining Jim, Elaine and I on the journey.

Finally, I want to thank my husband. I know I've been all over the place over the past 12 months; I know at times I've been completely impossible to be around. I also know the sacrifices you're making giving up your partner to that harshest of mistresses, the early stage startup. But at the most critical time, when OpenGamma was one phone call from not happening, you were there for me. Sometimes I might seem like I don't appreciate you as much as I do, but trust me, I do. Thank you most of all for just being there. Without you, I doubt I would have had anything to be thankful for over the last year.

So yeah, I guess I do have a lot to be thankful for this year.

For all my USAmerican readers, eat some extra Thanksgiving dinner for me. For all my non-USAmerican readers, get yourself some turkey.

And don't worry, I'll be back to my usual abrasive, obnoxious, sarcastic, snarky, angry, ranty self soon enough.

More Adventures In Unethical Recruiting

Let's run through an entirely hypothetical situation for a second.

You're a recruiter. You find out that there might be a newly funded financial technology startup that's hiring, perhaps by looking at Blog Posts or LinkedIn Profiles. You think "hey, that's a great opportunity to get some business."

You then proceed to start calling engineers and telling them that there's a recently funded financial technology startup in stealth mode that's hiring, and they'd be brilliant to work for, and get some strong expressions of interest, including CVs.

You then contact the CEO of said company through LinkedIn and say "Hey, I see you're hiring, and I've got some candidates that are already interested!" The CEO might in theory point out to you that you in fact do not have a Master Services Agreement or any other contractual relationship with said startup, and that he's not interested in any candidates that have come through that might trigger a financial payout with a firm with which the startup lacks a pre-existing business relationship.

Now you're in a bind. The CEO flat-out says that he's not willing to sign any type of contract with you, but you've already told the candidates that you can get them introduced to the firm. You've been caught out in your lie.

I suppose what you could do is come clean with the candidates and just tell them the name of the firm and its CEO so that they can get in touch directly. In fact, the CEO might have even suggested this as a way forward for you. But I doubt you're going to do this. If you were interested in acting ethically, you might have contacted the CEO before lying to candidates.

I think more likely what you're going to do is tell the candidates that the CEO isn't interested in them in particular. And for that, and the situation that led to this mess, you earn a merit badge in Unethical Recruiting to add to your stack. You've also earned the wrath of said CEO, who will proceed to make sure that the reputations of both yourself and your firm are effectively ruined with everybody said CEO knows.

Hypothetically, of course.

Old posts now have comment registration

After a prolonged attack on a few of my older posts (and not the ones with the most traffic either; literally just three of them, but over 10 spam comments a day), I've turned on Registration to comment on the older ones.

Newer posts, all covered by Disqus, won't be affected.