Confluence, iSCSI, NetApp, Flexiscale, Fail

We've been hosting the FudgeMsg website using Confluence (and mad props for the free Open Source license by the way!) on a VM hosted by Flexiscale. We chose Flexiscale for the following reasons:

• Confluence is ridiculously tricky to cluster, meaning that you can't benefit from the scale-out capabilities of the Amazon EC2 model. (Edit 2009-12-24 - We're not attempting to cluster it; this statement is to indicate that because it's so tough to cluster, we're not trying to. Confluence is thinking we might be, and crashing as a result).
• Getting your AMI+EBS+ElasticIP for such a single-point vendor app is very difficult and time consuming, again meaning that the Amazon EC2 model isn't ideal.
• We didn't want to go for dedicated hardware for a low-volume application stack.
• We don't have a stable enough internet connection to host the application stack ourselves.

We thought we were pretty darn clever to be honest, but then we noticed that things started going wrong. In particular, we started getting this error a lot:

We were running a relatively complicated Java setup (jsvc to run as chrooted user, on Tomcat, with multiple web applications running), so we thought we had done something wrong. After all, the first rule in software engineering is always, always, assume you have caused the break.

We were wrong.

What we've since found out is that the entire Flexiscale model is that your VMs have effectively no local storage whatsoever, and everything is iSCSI hosted off of a NetApp cluster. That means that your storage is fully persistent, and survives migrations of your VMs across physical hardware. Which is a great concept when it works.

Except that the Flexiscale NetApp cluster is borked. Essentially, sporadically the iSCSI services will completely shut down. Sometimes for a second, sometimes for 4 hours. Your processes will still be running, but if they attempt to hit disk for any reason, they'll hang and ultimately timeout. Your processes are still running, but they can't talk to disk.

In the case of Confluence, which was actually trying to talk to PostgreSQL, which itself was trying to talk to disk, Confluence detects the complete timeout hang of PostgreSQL as a cluster violation, and puts itself into crash mode.

You want to know the best part of this? When you're in this state, there's nothing you can do. You can't SSH into the VM, because it can't read the password file to let you in. The Flexiscale tools won't even let you hard bounce the machine. And they won't tell you when it's back up directly, so you just have to keep trying until you can finally get into the VM, to restart your servlet container.

This has caused no fewer than 20 instances where Confluence has died for us, sometimes lasting hours until we can actually recover the VM, and twice now in 2 days. It makes us look like morans who can't even run Confluence, much less guide development of a message encoding system.

So until we manage to get off of Flexiscale (haha, can't even get in to back up the data at the moment), if you see that error when going to the Fudge Messaging website, now you'll know why.

There are two morals of the story:

• If you get a Non Clustered Confluence : Database is being updated by another Confluence instance error, and you know that's not the case, make sure your storage is responsive;
• Don't host anything on Flexiscale until they sort their hardware out.

2009 Predictions Revisited

As promised, I'm coming back to revisit my predictions for 2009 to see how I've done. As predicted by virtue of my fuzziness, I was mostly right on most subjects. Let's go into particulars!

Messaging Breaking Out
I think I did pretty well there. We've still not got a final AMQP 1.0, but just following Twitter and blogs I'm seeing a lot more people, particularly from the non-financial world, starting to use messaging in their applications.

Cloud Becoming Less Buzzy
Complete strike-out here. The same people arguing amongst themselves over what is "cloud" is still going on. That being said, the use of utility computing (as I prefer to call it) is on the rise, and Amazon has come up with so many innovations in the space that it's hard to keep track.

Java Stagnating
Mixed bag on this one I have to say. Java has definitely stagnated, and we still don't have a Java 7. That being said, it looks like the delayed Java 7 may actually give us the chance to see JSR-310 and closures coming into the language, which would be a very positive development.

Java stagnating leads nicely into my next subject (yes, I'm going out of order now):

Non-Traditional Languages Breaking Out
I think I hit this one right on the head. The stagnation of Java, and prominent proponents of systems like Scala and Groovy, are seeing people being more willing than ever to consider these languages the "next Java". Scala in particular has gone from an interesting programming language to one which is seeing mass adoption in enterprises and in web shops.

C# Over-Expanding
To be honest, I have no idea how I did here. I've found myself completely and utterly outside of the C# ecosystem, so I'll have to leave it to one of my faithful readers to fill me in on how I did here.

Social Networking Losing Money
Yep, I failed here. Twitter's probably profitable, Facebook is almost certainly gearing up for an IPO. I was completely wrong here.

But it's not just the social networks themselves, social gaming has gone from an interesting idea to one that makes lots of money, indicating that the space of social networking has started to turn profitable not just for network providers, but also for network ecosystem partners.

Sun Radically Restructuring
I think I can say I was right here, in that they're radically restructuring themselves into Snoracle.

Next year's predictions on the way!

My Open Letter to the European Competition Commissioner

Monty urged me to help save MySQL. I couldn't possibly refuse such an offer.

Dear Competition Commissioner:

I am the Chief Executive and Technology Officer for OpenGamma, a financial technology startup located in the United Kingdom. I am writing to urge you to immediately and unconditionally approve the merger of Oracle and Sun.

I have a long standing history with both the Open Source and Database communities, having worked for a number of database startups in the United States of America, as well as working at early-stage companies making use and refining a number of Open Source technologies. I also have experience working at Oracle during a summer internship while I was attending the University of California at Berkeley.

Furthermore, in my more recent career as a consumer (rather than producer) of database technologies, I have setup and managed numerous, large MySQL installations, including one with more than 10 nodes and 150GB of active data (and several terabytes of archive data) in the financial services industry. I have also been a customer of Oracle's database technology for systems even larger.

In my mind, there is no logical reason to reject this merger based on considerations for the MySQL technology.

First of all, as a consumer of database technologies, I can tell you that Oracle and MySQL simply do not compete in the marketplace. While customers have replaced Oracle with MySQL, the applications based on Oracle that were ported to MySQL were never good candidates for Oracle and would have been ported to another database engine in due course as Oracle moves to the highest end of the market. Customers have numerous options for porting their applications off of Oracle onto a lower-cost database engine; MySQL simply has the most brand recognition in this space. Furthermore, MySQL has been used as a pricing lever by Oracle customers rather than an active option for migration.

That implies that Oracle's ownership of MySQL might see a reduction in competition for Oracle's core product, but there is a flourishing Open Source database market these days (which there wasn't when MySQL was originally created): Ingres, Firebird, LucidDB and PostgreSQL are all far more applicable to the Oracle customer base than MySQL is. Even if MySQL development were to come to an immediate halt, this wouldn't harm consumers in a such an extremely competitive environment.

However, the continued uncertainty over the Sun acquisition is potentially far more anticompetitive for consumers of technology as a whole: Sun has a number of competitive products with far greater applicability than MySQL (including their storage, networking, and computer chip technology). Allowing those products to die because Sun ran out of cash during this phase of its life would be extremely and permanently damaging to the overall computer industry inside Europe, and reduce competition significantly. Delaying this merger over the matter of MySQL would result in far greater anticompetitive results to European consumers of computing technology than even the worst case arguments of biased, self-interested advocates in this matter.

Thank you for your time, and once again, I urge you to approve this merger unconditionally and without further delay.

Sincerely Yours,

Kirk Wylie
Chief Executive Officer
OpenGamma

Document Stores: Please Give Me A Standard API

Although I'm a long-standing RDBMS guy (having worked on Broadbase, Kidar, and Eigenbase/LucidDB [indirectly through Kidar and Broadbase]), I'm quite excited by the emerging document-oriented database movement. While I'm not a pure "SQL is bad and old-economy and you should throw it away" guy, I do think that document stores, like their Hierarchical Database precedents, have their uses in modern architectures. In particular, practical (as opposed to theoretical) aspects of the use of a hierarchical/document model allow for advanced scalability and performance optimizations to be made for modern scale-out architectures.

That being said, even though we're at early stages, I think the major proponents of the technique need to learn from the RDBMS guys in one important aspect: unified APIs are key to widescale adoption.

If I'm writing an application that's going to be backed by a relational database, if I'm in a sensible programming language, I've got a standard API that I can code against: JDBC, ODBC, ADO.NET, et. al. It doesn't shield me entirely from differences in the underlying database implementation (or else there would be no opportunities for product differentiation), but it makes those differences minimal and relatively easy for a software developer to abstract.

Ditto for message oriented middleware: I can use JMS at the code layer, or AMQP at the network layer (and thus at the code layer as well). While different MOM implementations have underlying differences, which are particularly obvious if I want to push the technology to its limits, the product-specific differences are noticeable in the breach rather than in the general.

This isn't the case right now for document stores. I know that MongoDB and Riak and CouchDB and SDB and others (which I'm sure commenters will point out below) are pretty darn similar in their functionality. I know that the conceptual models are relatively similar. I know this logically. But I still have to do custom code for each one for my own application.

With multiple implementations out there, and with users (e.g. me) looking at the different systems and seeing them logically similar, it appears that it's probably time for the teams to start working together and come up with a code-level API that I can code against, much like JDBC or JMS. While this might seem like early stages for such an effort, trust me, it'll greatly lead to increased adoption because the perceived costs of evaluating different implementations will be greatly reduced.

It doesn't mean that you can't differentiate; it doesn't mean that you can't be superior or inferior to other implementations. But it does mean that I, as a consumer of these systems, can more easily support multiple implementations. And if I can do that, I'm more likely to move to one in the first place.

Giving Thanks 2009

I wrote this on a plane over the Atlantic heading to Chicago to visit my family for Thanksgiving, followed by a trip to San Francisco (my spiritual, if not familial, home). Maybe it's the maudlin sentiments from watching Julie and Julia, but I realized that I've got a lot to be thankful for this year.

The last time I was coming into Chicago was on a flight from San Francisco, all of 6 hours after I had found out I didn't have a job to come home to. It wasn't a particularly easy trip for me, and it was a pretty bittersweet trip to see my family not really knowing what I was going to do when I returned back to the UK (other than show up to be formally made redundant). Really, I didn't feel like I had a whole heck of a lot to be thankful for this time last year.

This year, I feel like things are completely different.

When I got back to London I immediately set about taking the idea which was OpenGamma and converting it to a Real Company. I pulled in two amazing cofounders, we started doing formal market research, refined the business model and strategy, and started working with potential funding sources. As I've mentioned, we closed a Series A round of investment, have started building the system that was little more than an idea and prototypes, and are pushing as hard as we can to make this a successful company.

It wasn't clear this time last year that it would happen, and a lot could have gone wrong along the way (more than actually did). I'm in a particularly rant-free mood, so let me give some thanks here.

First of all, I want to thank Jim and Elaine for joining me on the OpenGamma journey. Startups really are about founders, and I want to thank Jim and Elaine for trusting me when OpenGamma was nothing but my late-night emails and wildly-gesticulating concepts outside The Audley. You guys took a leap of faith and there's no way that we'd be where we are without all three of us working well together.

Second, I want to thank everybody who helped us pre-incorporation. We talked to a lot of people, and bounced a lot of different ideas off a lot of different people. Without people spending the time to talk to a bunch of crazy guys looking to disrupt financial technology, we never would have come down with a vision capable of being executed into a company. You guys spent time with us when you didn't need to, and when there wasn't any clear payoff to you. Thank you for spending the time and helping us refine our ideas.

I want to thank my family for standing by me and having faith in me and what we were doing. It hasn't been an easy year for all of you guys, but you've been rocks when I've been wavering and had doubts and went through difficult times. It's tough for me to keep in touch from 5,000 miles and 6 time zones, and I don't see my nieces and nephews anywhere near as much as I should do, but thanks for being there for me.

For the people at Big Bank B who took yet another leap of faith, thanks for not only giving me the opportunity to work for you while I was cementing OpenGamma, but for supporting me while I was doing it. I know I wasn't the typical candidate you were looking for, and I only hope that despite my abrasive nature and mercurial temperament I managed to perform at the level you were hoping. Your trust in bringing me in for a contract that gave me the freedom to actually launch OpenGamma hasn't been forgotten.

For my investors and board members, thanks for committing your time, energy, intellect, and capital in OpenGamma. Together, I think we'll create a fantastic business, and I thank you for joining Jim, Elaine and I on the journey.

Finally, I want to thank my husband. I know I've been all over the place over the past 12 months; I know at times I've been completely impossible to be around. I also know the sacrifices you're making giving up your partner to that harshest of mistresses, the early stage startup. But at the most critical time, when OpenGamma was one phone call from not happening, you were there for me. Sometimes I might seem like I don't appreciate you as much as I do, but trust me, I do. Thank you most of all for just being there. Without you, I doubt I would have had anything to be thankful for over the last year.

So yeah, I guess I do have a lot to be thankful for this year.

For all my USAmerican readers, eat some extra Thanksgiving dinner for me. For all my non-USAmerican readers, get yourself some turkey.

And don't worry, I'll be back to my usual abrasive, obnoxious, sarcastic, snarky, angry, ranty self soon enough.

More Adventures In Unethical Recruiting

Let's run through an entirely hypothetical situation for a second.

You're a recruiter. You find out that there might be a newly funded financial technology startup that's hiring, perhaps by looking at Blog Posts or LinkedIn Profiles. You think "hey, that's a great opportunity to get some business."

You then proceed to start calling engineers and telling them that there's a recently funded financial technology startup in stealth mode that's hiring, and they'd be brilliant to work for, and get some strong expressions of interest, including CVs.

You then contact the CEO of said company through LinkedIn and say "Hey, I see you're hiring, and I've got some candidates that are already interested!" The CEO might in theory point out to you that you in fact do not have a Master Services Agreement or any other contractual relationship with said startup, and that he's not interested in any candidates that have come through that might trigger a financial payout with a firm with which the startup lacks a pre-existing business relationship.

Now you're in a bind. The CEO flat-out says that he's not willing to sign any type of contract with you, but you've already told the candidates that you can get them introduced to the firm. You've been caught out in your lie.

I suppose what you could do is come clean with the candidates and just tell them the name of the firm and its CEO so that they can get in touch directly. In fact, the CEO might have even suggested this as a way forward for you. But I doubt you're going to do this. If you were interested in acting ethically, you might have contacted the CEO before lying to candidates.

I think more likely what you're going to do is tell the candidates that the CEO isn't interested in them in particular. And for that, and the situation that led to this mess, you earn a merit badge in Unethical Recruiting to add to your stack. You've also earned the wrath of said CEO, who will proceed to make sure that the reputations of both yourself and your firm are effectively ruined with everybody said CEO knows.

Hypothetically, of course.

Old posts now have comment registration

After a prolonged attack on a few of my older posts (and not the ones with the most traffic either; literally just three of them, but over 10 spam comments a day), I've turned on Registration to comment on the older ones.

Newer posts, all covered by Disqus, won't be affected.

Announcing the Release of the Fudge Messaging Project

Along with the rest of the OpenGamma team, I'm pleased to announce the first open release of a project we're calling Fudge Messaging [1].

What Is Fudge?

Fudge came out of a desire for a system to encode data which is:

• Hierarchical so that you can represent common document idioms (XML or JSON) using it
• Typesafe by storing the type of data along with the payload.
• Binary for compactness on the wire or on disk.
• Self-Describing in its contents so that you can introspect a message without knowing anything about any schema used for its production

In the past, we've looked at a number of different encoding schemes, and rejected them for various reasons:

• Google Protocol Buffers and Thrift are great if you have access to to the schema, and know what the type of messages on a particular communications channel are. But because they're not inherently self-describing, you can't build interesting middleware around their flow.
• XML and JSON are big (if you don't gzip them) and slow (if you do). Plus by being text-based, you have to convert everything to and from text.
• ASN.1 is conceptually nice in a lot of respects, but it's become such a niche that the tooling is utterly rubbish, and I don't see it getting any better.

Ultimately, for those of you familiar with Tibco Rendezvous, we really liked the TibrvMsg system, but couldn't use it because it's proprietary, and the implementations are surprisingly slow for something designed for low-latency messaging.

What Is The Fudge Project?

The Fudge project consists of the following major components:

• An Encoding Specification, conceptually similar to ASN.1 BER, DER, or PER, which specifies how to encode data in a linear sequence of bytes.
• A set of Reference Implementations, which adhere strictly to the encoding specification and provide access to Fudge-encoded data in a number of programming languages.
• Tools to make it easy to work with data encoded in the Fudge format.

OpenGamma is sponsoring the Fudge project, but in the classical Open Source way: we found something that was missing out there, built it to suit our needs, and have the capacity to help it grow while we use it.

Project Status

I'm not going to kid you: this is a very early stage project. We have Java and C# reference implementations (and the C# one, for example, is more advanced than the Java one in a lot of respects, mostly because it's leveraging relatively new features in the C# language). We have a wiki and Jira installation. We have code up in Github. But that's about it.

Don't let me mislead you: the Java reference implementation, for example, is rock-solid in stability and very good in performance; we use it daily here at OpenGamma. But we don't have polished releases and distributions and everything else as of yet. That will come, but we wanted to get the code out there as soon as we reasonably could.

License

We're releasing the code under the APLv2 because we believe that it's the best type of license for this type of work: it's developer friendly, as well as employer friendly.

The specification is free to use for everyone. If you're going to use the ideas but not the full specification we'd ask you to not call your project Fudge Compliant [2], but anyone can create their own Fudge implementation using whatever license they want.

Next Steps

If you're interested, take a look at the wiki, grab the code, and take a look. I hope you find it useful.

From our perspective, we're going to continue to develop the system and grow it.

Footnotes

[1]: Fast Unstructured Data Generic Encoding
[2]: Fair Disclosure: we are considering getting a trademark on the use of the word Fudge for messaging to allow us to ensure that only compliant implementations can use the name.

Monty's Almost Certainly Looking For Investment

Remember how I predicted that Monty was attempting to create such an untenable situation for Oracle that they had to dispose of the MySQL IP and put it into his hands? Remember how all of Monty's protestations were based on the fact that he doesn't have the money to buy it? Remember how I said that he might not have it now, but could probably raise it?

Yeah. He's trying.

From Reuters, Florian Mueller is touring Wall Street:

EU strategist who is former MySQL shareholder and adviser announces Silicon Valley press conference (26 October) and New York City analyst briefing (27 October) -- Florian Mueller wants to "explain positions of critics of proposed transaction in the lion's den" -- New York event due to "strong Wall Street interest in the matter."

There haven't been any protestations from Wall Street that I've seen that they're unhappy with Oracle acquiring MySQL. Rather, this has been a European affair as Monty tries to get the EU to interfere in industrial policy for his own personal benefit. So why in the world would you send your sockpuppet to Wall Street to explain any positions? What interest would Wall Street have in the matter other than to provide funding?

You might argue that he's going there to talk about the position they're taking with the EU competition commission because there are lots of people who own Oracle and/or Sun stock, and might be interested in the matter. But I think a much simpler story is at least as likely: Florian is attempting to drum up a capital raise to acquire the MySQL IP to make the problem go away for Oracle, and to convince Oracle and Sun shareholders that Monty and Florian will do whatever it takes to block the acquisition so that they'll tell Larry to let go.

Add to the Look at the Balls on that Guy category: Florian Mueller.

Java APIs and Ivy: Click-Through Agreements Suck

We've been using Ivy for project dependency management at OpenGamma since day-one. It was one of the earliest decisions we made, and I'm quite happy that we made it. Furthermore, the work that the Ivyroundup guys have done to package up Ivy-specific files has been fantastic.

That being said, there's one big, honkin problem with it: Sun and their click-through agreements to download almost any of the "interesting" APIs that you need for enterprise software development. These packages (like the JMS or JavaMail APIs) are necessary for pretty much anything that you might want to do on a real system, and you can't legally put them up for download. And unfortunately, there are a lot of them.

The workaround, of course, is that someone in your organization downloads them all, clicks-through on the agreement that is completely and utterly unenforceable, and then sticks them on a private Ivy repository and you configure your ivysettings.xml file to refer to that location, but that's not particularly scalable. And for open source projects, it just doesn't work at all.

Yeah, we've got this Ivy thing so that it automatically identifies all the dependencies and downloads them so you don't have to do anything but have Ant installed. Except for the Java stuff, which you have to manually download and put into these particular locations, and that's a 20 minute task minimum.

Kinda takes the shine off the whole experience, doesn't it?

Now given that everybody seems to know that you can handle licensing restrictions in different ways (by a disclaimer in each file and a copy of the license included), why does Sun still live in the past? Why do they require a clickthrough for something that is essentially just an explicit disclaimer of warranties and liability? Why can I download Eclipse or NetBeans without a clickthrough but not the JMS 1.1 API interface files?

Man, I hope the EU finally lets Oracle buy Sun and put this type of stupidity out of its misery.

Monty, Stallman, MySQL, Oracle, and Sun: Open Letter Wars

I've tried to confine my ranting about the current state of the Sun/Oracle/MySQL debates to my Twitter feed, but I think I need to do more than the 140 character limit allows.

Background On Recent Moves

In case you haven't been following the state of play, we've got two recent open letters sent to the EU competition commissioner:

• RMS, Knowledge Ecology International, and the Open Rights Group came out as strongly opposed to allowing the Sun/Oracle acquisition to proceed as long as MySQL was part of the acquisition.
• Monty suggested that Oracle be forced to sell MySQL as part of the Sun acquisition.

If you've been following either of my posts on the subject, you'll know I'm not a dispassionate observer in this matter, particularly where Mr. Widenius is involved.

Competition and Acquisition

First of all, let's directly address the core matter at hand, which is that Monty, RMS, and the various others appear to believe that the Database market is hopelessly consolidated and were Oracle to get its hands on the copyright to the MySQL source code that would be bad for competition.

This, to be honest, completely and utterly disregards the actual history of the database market, which has always been one of consolidation and benefits to the consumer:

• Illustra, a Berkeley spin-out, was bought by Informix
• Informix was bought by IBM
• RedBrick was bought by IBM
• RDB was bought by Oracle

While this consolidation has reduced the number of vendors in the market, as of 2007 there was still pretty hefty competition with Oracle even then only with a 44.1% share of the paid database market. As someone who has had to work professionally with Oracle, DB/2, Sybase, and Microsoft SQL/Server, I can say this is almost certainly because it's the best overall product.

Furthermore, those numbers in terms of the database revenue are completely suspect (with the exception of Microsoft's). There's a huge amount of revenue for IBM and Oracle which are tied to services and software sitting on top of the database (such as Oracle applications and IBM services), and realistically the CFOs of each company can tune the percentage of the deal that goes to the underlying database based on what numbers they want to report. The overall deal may be $1MM, but the sales person has a lot of discretion on how they price the database component.

Is there so little competition in the market that it's hurting consumers? Hardly. The recent squabble over Oracle's TPC-C pseudo-announcements indicates that the vendors actively compete with each other. Furthermore, the rate of feature expansion has been truly dramatic. Finally, the ability of firms like Vertica to rapidly jump into the market indicates that this isn't a market that requires significant levels of competitive concern on the part of regulators.

The technology industry is based on larger firms buying smaller ones. Competition authorities should rightfully be concerned only if it harms consumers in general, not whether it harms a particular subset of users.

But MySQL Is Special

With all due respect, no it isn't.

Let's consider the pseudo-market for Open Source databases. We've got:

• MySQL and its various derivatives (like Drizzle)
• PostgreSQL and its various commercial/supported versions
• Ingres
• Firebird
• LucidDB

And that's just considering the relational ones. When you consider the NoSQL movement as potential competitors (which I, for example, most certainly do), MySQL just isn't that special anymore.

While it is possible that an Oracle acquisition might be bad for MySQL consumers, it doesn't follow that MySQL is so special and perfect and pure that it harms any general category of consumers. While databases aren't perfectly replaceable, if someone found that Oracle's stewardship of MySQL was so onerous that they wanted to move off of it, it wouldn't be impossible to move to another database, either commercial or Open Source.

What that means is that you're in a classic case where the acquisition of a particular company might be harmful to consumers of that company's products, but it doesn't generically affect the market in a negative way. IBM and Microsoft will continue to compete in the commercial space, and PostgreSQL, Ingres, and LucidDB will continue to compete in the Open Source space. There's no net harm to consumers as a whole from an acquisition, even if the result of the acquisition was the complete shutdown of all commercial support for MySQL.

Oracle Is A Bad Acquirer

First of all, let's get the obvious out of the way: Oracle bought BerkeleyDB, and continued to enhance it; Oracle bought InnoDB, and continued to enhance it. At no point did they crush them to drive Oracle database revenues, or change the licenses, or stop forward momentum. So when you look at the actual track record of the company, they're in the clear.

But they might do, because they're an evil, scary corporation that MySQL turned down once before (from the Stallman piece):

Oracle made an earlier effort to buy MySQL in 2006, but the management rejected Oracle's offer, in part because Oracle would not disclose its plan for MySQL, and some members of the MySQL management team were concerned that Oracle was only acquiring MySQL to curb its advances in the marketplace.

I know a number of people involved with MySQL when it was an independent organization. While there were people who worried about that fact, senior management wasn't. More importantly, Monty was willing to sell MySQL to Oracle in 2006 for the right price. The use of the words "in part" there are telling, because the primary consideration that MySQL's senior management had wasn't some happy-clappy love for the Libre Software Movement, it was money.

I'm sorry, but I fail to see what's changed in between 2006 and 2009 except that Monty is a whole heck of a lot richer. Why in 2005 and 2006 were offers ultimately rejected from Oracle based primarily on money, but now Oracle is an evil corporation that can't be trusted with MySQL? Larry's the same guy he was then, Oracle has bought BEA but they don't compete in any way with MySQL, it's the same company. Why would Monty trust Oracle back in 2006 but not now?

Force Oracle to Sell MySQL

This is Monty's solution. And it's cunning. It's particularly cunning that he says repeatedly that the obvious Monty-connected acquirer, Monty Program AB, lacks the funds to do such a purchase. Again, a half-truth.

MySQL was worth $1Bn in early 2008. Since then markets globally have tanked, but MySQL has had some good commercial strength recently within the Sun organization. So let's conservatively say that it's still worth $1Bn. Let's then say that Oracle values the acquisition of Sun highly enough to let MySQL go for less, and do a 20% haircut to $800MM. Who's got that kind of money to acquire?

• Microsoft. You think Stallman and Monty would be happy with that? No.
• IBM. #2 in the database market. Erm, raises same issues that Oracle would.
• Sybase has the market cap (super-recently) but not the cash.
• Red Hat has the market cap, but not the cash.
• Novell lacks the market cap and the cash.
• Computer Associates has the market cap and the cash, but is the place technology goes to die. They also have Ingres to work with.
• VMWare has the market cap and the cash and an acquisitive streak, but would MySQL really fit into their product strategy? I can see Spring driving people to vCloud, but can't even fathom the same kind of strategic benefit for MySQL.
• Symantec has the market cap and the cash, but their storage work has been pretty solidly focused on backup and low-level storage these days.

It doesn't look to me like there are that many companies out there that could really buy MySQL in cash and make Stallman and Monty happy.

But you don't actually need to have the cash yourself: you can use private equity money. It's happened before: BEA was funded by private equity originally to consolidate the Tuxedo market. That's why Monty's protestations ring hollow: his statement is explicitly "we don't have the money." But I think he could probably come up with it, and if he doesn't, then he needs to work with better financiers.

Finally, let's assume that Oracle really wants the rest of Sun, and considers carefully Monty's open statement that Sun is hemorrhaging $100MM of cash per month. Wouldn't it make sense for Oracle to actually just donate MySQL to pretty much anybody to make the EU issue go away? Oh, and lo and behold, Monty has two of those ready to go: Monty Program AB, and the Open Database Alliance.

To me, the current situation amounts to blackmail: we'll keep blocking your acquisition of Sun until you do what we want.

Consider The Sources

So let's look at the motivations of the major current players.

Stallman is irrelevant to any commercial discussion. His press release essentially says "I don't like the GPLv2 anymore, even though I wrote it, and it would be better if MySQL was under the GPLv3." Tough. Furthermore, RMS has no commercial experience of any kind. I fail to see how someone who has never even worked for a profitable commercial enterprise could be considered knowledgeable about how an acquisition would affect the marketplace in an anti-competitive way that harms consumers.

Furthermore, RMS' press release completely belies his previous positions regarding the possibilities for commercialization of GPL projects. He's stated in the past that offering dual licensing is only one of many ways that you can make money with the GPL being the dominant licensing model. Why all of a sudden does he believe that this is the only possibility for MySQL? Why is he so adamant that without that ability, there's no ability to derive commercial revenue from MySQL?

Monty has been slinging FUD about this acquisition for months. He was such a disruptive element inside Sun that they released him from his Non-Compete just to make him go away. Given that he's an extremely rich, disgruntled ex-employee and project founder, he has personal reasons and financial ones (under the Monty Program AB umbrella) to cause as much disruption to this deal as possible.

I've said it before and I'll say it again: Monty has been playing a long game here, and I think he'll be obstructive to any potential move that Oracle would make with MySQL until the IP is under his control.

Personal Opinions Should Not Drive Competition Policy

Ultimately, you can sum up the entire argument against the Oracle/Sun acqusition due to the MySQL situation as:

• We don't like Oracle owning the MySQL IP
• Therefore, don't let Oracle own the MySQL IP

Unfortunately, saying that you personally dislike something doesn't provide a valid reason to block an acquisition on competition grounds. Saying that you don't trust Oracle doesn't alter the marketplace in a way that disadvantages customers as a whole. Saying that nobody else could make money by selling commercial licenses for MySQL doesn't mean someone else must be allowed to.

The moment the MySQL founders, who have been handsomely rewarded, took VC money they turned MySQL from being a hobby project/company, and into a major technology company and an asset. The change happened years ago, it's just that they're only starting to wise up now.

But it's happened. It's done. It's no longer anybody's pet project; it's an asset that can and should be used by whomever is willing to pay the most for the IP. As a customer, under the GPLv2, you still have rights, including the right to fork. But don't go whining that a company that made massive amounts of money for its shareholders by commercializing a technology is no longer under your control.

OpenGamma Is Hiring

People who have been following the blog for a while, or my twitter feed, will know that I've co-founded a startup called OpenGamma, that we've received a significant equity investment, that we're working in the financial services technology space, and that we're so stealth we don't even have a web site. And now I'll add another nugget of information into the mix: we're hiring.

Immediately, we have head count for one person, and I'm describing what I'd like for that one hire below. However, we're always open to finding someone good, so no matter when you find this post, if you think you might be a good fit for the company, send your CV over; if we can't hire you right away we'll let you know right away, but our headcount requirements are changing constantly. We might not be able to hire you right now, but if we think we might have room later on, we'll let you know.

About OpenGamma

Let's completely ignore whether you're a good candidate for the role that we're looking to fill. The #1 question you should be asking yourself, without even knowing what we do, is do I want to work with these guys? Hopefully this section can fill you in about who we are.

OpenGamma was founded by three people: your humble author, Elaine McLeod, and Jim Moores. Of the three, I'm the only without a PhD, and Elaine and Jim remind me of this on a regular basis. I'm the CTO and Acting CEO [1]; Elaine is our resident quant; Jim works on our core engine. We've known each other for years now, and bring a combined 30 years of technology experience (13 in finance) to the table.

The team has already grown beyond the three of us, and by the time you read this we'll have one more person working on-site and one more who's agreed to join. So you'd be coming on board when we're still quite small, but that's one of the major attractions for a lot of people. We believe that since you'll likely be spending more waking hours with us than your family or friends, it should be an enjoyable, social atmosphere (one of the earliest decisions we had to make was which local pub to adopt [2]).

We're well capitalized, having just closed a Series A equity investment by a major international Venture Capital group [3]. We have an office in the London Bridge area [4], from whence we can easily make lunchtime trips to Borough Market [5]. We have fast computers and big monitors and good (unfiltered) network connections and tons of Diet Coke and all the types of things that mean nobody's banging his or her head against "if only I had X, I could code better/faster."

What we don't have is meaningless policy, bureaucracy, procedures, politics, or other stupidity. We don't have a dress code. We don't have any paperwork other than expense claim forms and what we're required by law to fill out. We believe that the only policies we need are "do the right thing." We believe the only procedures that are worth having are the ones that help you, and us, execute better. We believe in hiring the best, enabling them to perform, and trusting them.

About You

I probably don't know you. But I can guess a lot about who you are, if you're going to be a good fit for the team at such a critical early stage in OpenGamma's life.

First of all, you know something about Front Office or Risk technology in capital markets. Ideally, you've worked for a broker-dealer or investment bank in your past, and might even be working there now. While this isn't critical for every hire we make, it is for this one. You get a thrill out of working under the constraints that modern trading software requires, and you like producing software not just for its own sake, but to solve business problems.

However, you're not 100% happy working for a Big Bank. You've beat your head against policies, procedures, internal audit, external audit, anything that stops you from doing your job effectively, one too many times. You're sure that it doesn't have to be that way. Maybe you started out trying to change things, thinking that one person could make a difference to the organization, and gave up. Maybe you just thought "that's how it's got to be" until you discovered the wide world of technology outside your bank. But you know that you don't want to work there anymore.

You like technology; it's not just a 9-5 job with you. You read tech blogs [6], user-contributed sites like Proggit, Hacker News, and DZone. You post answers on Stack Overflow. You research new technologies even though you know your employer would never let you use them, because you find them interesting. You go to events like CloudCamp, Pub-Sub, and BarCamp. You probably code in your spare time.

You live within a reasonable commute of Central London. While at some point in our future we'll be happy for full remote working, we're not there now. You might need reasonable accommodation for your personal or family situation (and we're not just legally obligated, but happy to do so), but you have to be able to make it to the office on a regular basis.

Most importantly, you've gotten this far and thought to yourself "Wow; this sounds great! Kirk, tell me more!"

About The Role

So about this role. I can't tell you that much about it without bypassing the Great Shield Of The Stealth Startup, but here's your buzzword bingo section:

• You need to know Java. It doesn't have to be your preferred language, but you need to know it.
  • +1 if you came to Java from a C/C++ background.
  • +1 if you got sick of Java-the-language and started working with other JVM-based languages.
  • +1 if you've worked with low-level APIs like the post-1.5 concurrency libraries and the IO and NIO systems.
  • -5 if it's confined solely to web applications.
  • -10 if you've only ever used a traditional full J2EE stack.
• You need to be conversant with modern software engineering, architecture, and methodologies.
  • +1 for every agile technique you like to use
  • +1 if you understand the difference between agile and Agile, and prefer the former
  • +1 for IoC and use of Dependency Injection containers like Spring
  • +1 for rigorous use of automated test suites
  • +1 for dedication to continuous integration
  • +1 for every functional language you can still code in (Scheme from your SICP days doesn't count here)
• You need to be familiar with a standard Front Office/Risk Technology stack. By this I mean message oriented middleware, enterprise-class databases, distributed systems.
  • +1 for every Trading System you've worked with
  • +1 for every source of market data you've worked with (Reuters, Bloomberg, Exchange Feeds, etc.)
  • +1 if you've setup your own MOM infrastructure
  • -2 if you used XMPP rather than a real MOM infrastructure.
  • +5 if you've worked with AMQP
• You need to have some familiarity with analytic libraries for financial models. We don't need you to be a quant, but you should have worked with pricing models in the past.
  • +1 for every chapter in Hull you've managed to get through (and +2 for every problem you managed to do successfully without resorting to the solution book)
  • +1 for every pricing model you've seized from the quants to improve performance or stability
  • +10 if you understand why Correlation is a rubbish analytic for pricing credit derivatives (even if you can't work through the math)
• You might be familiar, even if you haven't used them in anger, with some advanced technologies not every bank has rolled out yet:
  • Clustering and Data Grid technologies like Terracotta or Coherence
  • Hardware accelerated computation like GPGPUs and FPGA-based acceleration cards
  • Shared-memory or shared-flash clustering systems like Violin
  • Key-Value stores like Project Voldemort and CouchDB

Risk/Reward

Here's what OpenGamma can offer you:

• The chance to work for an organization designed to allow you to produce great technology.
• No bureaucracy.
• An equity stake in the business.
• The chance to learn about working for an early-stage startup first-hand.
• A tech team filled with the smartest, best people we can get. In fact, if we can't find the right person, the role stays unfilled.

However, there are some things OpenGamma can't offer you:

• Job Security. We're well capitalized, but we're still a startup. If you want/need a guaranteed job for several years, we're not for you.
• The Chance To Hide. What you work on is going to be seen by everybody else in the firm on a daily basis. Moreover, you're going to interact with customers. You're going to interact with the public at large. No room for shrinking violets, and no room to coast on your teammates, I'm afraid.
• Hedge Fund Cash Compensation. I believe in fair compensation for excellent staff; London is an expensive city, and if you're good enough to work for OpenGamma, your fixed expenses have largely grown commensurate with your banking salary income. We can offer good compensation, but we can't and won't match what you'll earn in a good year for an Investment Bank or a Hedge Fund: your equity stake is to make you whole over your employment at OpenGamma, and to make sure that everybody has aligned interests.

How To Apply

If you still think we might be a good match, send an email to jobs at opengamma dot com with your CV, and a description of why you think you're a good fit for OpenGamma, and we're a good fit for you.

Note: Recruiters/Agencies Not Welcome To Apply. We're not working with recruiters as of yet, so please don't contact us with a story about a candidate whose "profile" you're "exclusively representing." Any agency-provided CVs will be deleted unopened and unread.

Update 2009-10-28: We are now working with external recruiters, so if you're a candidate who just read that after a recruiter contacted you, ask to see the job specification. If they have it, they're definitely working for us.

Footnotes

[1]: I'm CEO during this, the most formative period of OpenGamma's lifetime; I'm not wedded to the role, and it will become quite clear to myself and our board of directors when it's time to bring in a permanent CEO. Until then, I'm it.
[2]: The Trinity. Best one around, even though it's farther from London Bridge station than the office.
[3]: Yes, I'm coy about who it is. No, I'm not going to tell you unless you get to the phone discussion stage. Yes, we have valid reasons for that.
[4]: If Old Street is Silicon Roundabout, can we be Silicon Bridge or something?
[5]: Shame that it's 1/3 shut down these days...
[6]: Like this fine screed. That guy's really good, if a little opinionated for my liking. He should calm down a little and go back to writing about tech stuff rather than politics and personal attacks on industry figures.

Startup Visa : The Sausage Is Half Stuffed

In my first, very personal post about what's now become the Startup Visa movement, I referred to the sausageworks nature of modern USAmerican Immigration policy. Brad Feld has come out with a straw-man policy, and asked for ways the system can be "gamed". Check the comments for my took-me-10-seconds way to game it.

More than that, though, I think the simplistic nature of the argument is ignoring the types of detailed policy issues that come out whenever you attempt to meddle in immigration policy. Let me try to come up with a few that came to my mind right off the bat.

Families

So let's say your founders aren't 23-year-old Ramen-eating YCombinator founders. Let's say they've got a family of any kind. How do you deal with the family members?

• First of all, how do you even define family member? Child? Adopted child? Step-child without adoption? Spouse? Parent? Brother/Sister? There are clauses in immigration law in the US to handle all those types of situations. [1]
• Can they move to the US? While you might at first guess "of course," this isn't always guaranteed.
• Can the spouse work? What about the children [2]? Do they have any restrictions on what types of employment they can take?
• What about benefits for the family? Can the children go to school? Are they eligible for school lunches and after-school programs?
• What about someone who wants to get married after they move here? Can they bring their fiance in from their original country?

Families are messy, and you can't just wave your hands and say "well, just do whatever it is you do for Class X," because all visas have a different way of dealing with family members.

Exit and Employment

Okay, so you've managed to get a startup going, and it's been successful, and now it looks like some big, rich company is going to buy it. Great! What happens to the founder?

At that point, the founder will, no doubt, have a big set of golden handcuffs keeping him or her with the purchasing company. That will require that the founder convert from being a founder to an employee. What does that do to the immigration status under a Startup Visa? Does it auto-convert? Under what terms? After the acquisition can the employee then leave to another firm? If not, can they go back to VCs and found another company, or has their Startup Visa status altered irrevocably?

Because if it auto converts, here's a very easy loophole in the system:

1. Borrow $500k
2. Give it to a trusted source
3. Trusted source #1 invests the $500k in your firm, and takes 100% ownership. Firm now has $500k.
4. You get your visa and move to the states
5. Trusted source #1 sells the firm for $1 to a shell company run by Trusted source #2 (who now has the $500k)
6. Trusted source #2 gives the $500k back to the founder as a sign-on bonus, who now has a work permit and the original $500k.
7. Founder now takes work permit and moves somewhere else (possibly by selling the shell corporation to the ultimate employer).

Sure, there would be transaction costs along the way, but it's another way to game the system. And I'm pretty sure that you can't come up with ways to stop this gaming of the system without stopping the type of constant acquisition flow that happens in the real world.

If you don't give the founders conversion rights to an employee, their startups are effectively defunct before they get started, because they're essentially acquisition-proof. If you do, the system can be gamed. [3]

Long-Term Conversion

So let's say that you've got your two-year renewable visa. What does that eventually translate into? Because at some point you're going to want to actually settle down, and you might not have had a successful exit yet. Do you have to keep being a founder of a company until you hit the jackpot? You can never join an existing startup?

You're going to need some means of long-term visa normalization. Does the visa translate directly to a Green Card? Would be difficult given the number of years it takes to live in the US before you can apply for a Green Card under other statuses. What about converting to an H1-B? Oh, wait, that's full of issues at the best of times, and now, you have to factor in the annual lottery (how would an H1-B conversion affect the numbers in the lottery?).

So you're going to have to come up with some means of normalizing the long-term immigration status of the founder, and his family, and I don't think there's an existing mechanism for that that works for this type of visa.

"My Turn For Reform."

So you've got a bill that's sensible, and covers all the various bits and pieces and makes sense from a legislative perspective. Here's the problem: there are people who have vested interests in immigration reform that have been waiting for longer than you have, and they're not going to be happy if you skip to the head of the immigration reform queue.

For example, let's consider gays and lesbians, who have UAFA to correct the circumstances that led to my leaving the USA, which has been pending in the house since 2000. That's a long time. And as you can tell, I don't like some upstarts coming in and trying to jump to the head of the queue.

Yep, I'm bitter, and biased, and ranty. I'm also the exact type of person likely to go ballistic and get my representative to try to attach UAFA to your bill, which will then tank it [4]. And if you don't think there are at least 5 other aggrieved groups in America who will do the same, you're wrong.

Because immigration policy affects people so intimately, it really REALLY touches people's buttons, and they will fight tooth and nail for their own pet reform, at the expense of anything and everything else, which is one reason why this is such a morass.

Summary

Again, I have vested interests here: I'm bitter, and I want London to succeed as an independent tech hub outside of the US. But I think if you read this far, you should be convinced that a proposal for a visa that fits into a single paragraph isn't sufficiently worked through to be practical.

I repeat myself from several comments on Other People's Blogs: getting a Startup Visa is going to require a lot of long, hard policy debates. Why not spend that time working on more meaningful immigration reform that solves the underlying problem, rather than just a symptom? And why not push for the UAFA while you're at it?

Footnotes

[1]: Note that for the sake of simplicity I'm just going to assume that the US continues to have its "no gays desired" immigration policy and not even go into that minefield.
[2]: If you've got teenage children, they might want to have a summer job so they can fit in. Heck, they might even want to have an after-school job.
[3]: I don't want people to game the system. I merely want to point out that if I see it, others in lawmaking and the immigration lobby will as well.
[4]: Because if there's something that kills bills instantly, it's not being suitably discriminatory to teh gheys. Heck, just add something discriminatory to the Startup Visa bill and it'll probably get through first try!

Have finally resolved comments and layout

After several attempts in the past, I've finally managed to move Kirk's Rants to Disqus for comments, and fixed the layout.

While I attempted to use their snazzy-dazzy comment importing system, it reported that there was an error with the import, so I had to fall back to only enabling for new posts. Let's see if it actually works now!

I've also got a template now that actually fills out the screen, which is pretty important when you write as verbosely as I do, and want to include code snippets.

Hopefully this postpones the time before I finally move off Blogger and onto some other platform.

My qCon London 2009 Presentation

It's hit the interwebs. Here's the video and slide deck all in one.

I haven't actually watched the whole thing yet (it's 2am here in Blighty), but if you missed the talk, have a watch. At least the first 5 minutes isn't terribly embarrassing.

The Founder Visa Ignores Immigration Reality

My fearless readers are, no doubt, already familiar with the fact that I'm an American born and bred, but transplanted here to blighty to face my future amongst the whinging poms. I feel, based on my Silicon Valley-meets-London personal experience, I have something to add to the "Founder Visa" debate.

The notion of a "Founder Visa", originally coined by Paul Graham, has spread far and wide: Brad Feld, Slate, Eric Ries and a whole wave of twitches picked up the meme. The basic idea is that you'd get a visa to go and live in the US if you founded a new venture-backed company in the US. It sounds so simple a policy prescription that it must be a fantastic idea!

Exasperated Sigh

In Which Our Hero Visits Locations Unknown

I didn't move to the UK because I was a huge fan of the weather, or the food [1], or the transportation [2], or the dentistry. I moved here for immigration reasons.

When people ask me why in the world I left San Francisco to move to England, the typical answer I give is "family reasons." Which is true, but not incredibly descriptive. I think it's time to give a fuller story, and in the process out myself. [3]

When I moved here I was in a long term relationship with a British citizen. He had moved to the US with an L-1E-1 Treaty Trader visa [4], and was hoping, nay praying, that this could be upgraded to an H1-B. Unfortunately, although he was an extremely experienced financial software pre-sales consultant, with a BS from a quite reasonable UK university, and paid quite significant taxes for precious little in return, this wasn't going to happen. We lived for several years under the threat that if his firm decided to let him go for whatever reason, he had two weeks to leave the country forever. Given that his firm wasn't doing particularly well at the time, this put a major strain on his life, and thus our relationship.

After New Labour was elected here in the UK, one of their first actions on taking power was to rationalize gay immigration rights. Essentially, they said that if you were in a same-sex domestic partnership akin to marriage, that would be considered as though you were married for immigration purposes. Thus, I was able to move to the United Kingdom based on our relationship [5]; the converse was not true. We didn't have the option to get married in the US or the UK, and even if California had same-sex marriage at the time, it didn't matter for immigration purposes. [6]

So I gave up a career in Silicon Valley, where I had already been a founder of one firm, and worked in senior technical roles in several others. No doubt I would have started at least one firm in the Valley since then. I was a net win for the US in every single respect: I paid a lot of tax, I didn't use very many services, and I created a lot of high-tech jobs. The US lost me because certain Americans view my sexual orientation as, well, wrong.

Recently, I've started a venture-backed technology startup in London. Even though we've only been incorporated one month, I'm already benefiting the local economy and tax base. I'm creating jobs, hiring service providers who employ local staff, and spending money that creates jobs out of work. I pay personal taxes, and pay corporate taxes that help raise the tax base of the country as a whole. From a purely economic perspective, it's absurd to alienate someone like me from the US, but that's precisely what's happened. I don't plan to move back anytime soon. [7]

In Which Immigration Policy Becomes A Morass

Fundamentally, US immigration policy is a complete nightmare. All the vested interests, all the religious/social/conservative arguments, all the animosity, it's all a nightmare. And it's much more the third-rail of US politics than even health care or social security. Even though the Shrub had massive majorities in both houses of congress, he still couldn't push through even the most minor (to me) immigration reforms. What hope do a bunch of VCs have?

The UK, Canada, and Australia have solved this in a very clean way. They take a person, weigh them up on purely objective criteria, and determine whether the country believes that they will be a net-win to the country; if so, they get in. The UK calls this the Highly Skilled Worker program; Australia calls it the General Skills Migrant program. Why can't the US do the same thing?

Because when I hear about things like a Founder Visa program, what I really hear is a general denunciation of US immigration policies and procedures. What I really hear is "We can't hire the people that are necessary for the industries that are important to the country, and we're picking the edge case that we understand the most." That's not good enough. The edge case isn't the problem, the system is the problem.

In Which VCs Run A Chocolate Factory

Let's say that we had a Founders Visa. Let's say that we had this single exception to the insane US immigration policy morass. Here's my worry from a policy perspective: it changes the power dynamic for early stage investment with a non-American founding team so drastically it disturbs me. Top-tier VCs I'm not worried about; it's the less scrupulous sources of money that trouble me. [8]

Right now you have a power dynamic that, in general, involves equity investment by a risk capital firm in a firm founded by one or more entrepreneurs. Now let's say that in the middle of those discussions, you also had the dynamic where an offer of funding would also change the founder(s) immigration rights dramatically. Don't you think that would be detrimental?

Because I could easily see situations where bad sources of capital would essentially give a term sheet, wait until near closing and the founder(s) had started to make plans on living in the US, and then change the parameters of the deal in the docs stage. Because if you've already taken a lease on a home, started moving your stuff, started thinking you're going to live there, what's a few extra percent off the top if it preserves your visa?

Personally, I'm against the Founders Visa because it conflates immigration policy with risk capital investment. I'm in favor of rationalizing US immigration policy because it's the right thing to do.

Epilogue

If you start thinking that you can address US immigration policy without addressing my personal reasons for emigrating, think again. You have to reform the system as a whole, and you have to convince Americans in places that don't have a startup scene that immigration of people Not Like Them can be good for the country. If you don't start from first principles, you're doomed to failure.

I was forced to leave the US because there are people in the US who believe that the country would be better off without me, and those people have legislative power. I accept that, and it's why I'm not planning on moving back. Unless you manage to convince those same people that your plans to allow immigration are better for the country, and for them, you'll never succeed.

I wish you all the best of luck. I won't help you, because I think you're targeting an edge case but disregarding the fundamental iniquities in immigration policy. Furthermore, your policies wouldn't have kept me in the Bay Area, and I'm still bitter.

However, if you manage to fix things, I might just move back to Silicon Valley for the next startup.

Footnotes

[1]: I kid, I kid. Seriously, the food in London is fantastic, if you can afford to dine at the fantastic restaurants.
[2]: My commute in the San Francisco bay area: 45 miles; 45 minutes. My commute in London: 5.5 miles; 45 minutes. Then again, I've never been allowed to drive here, and I've never missed it, so it's a mixed bag.
[3]: I've been careful to never allow my sexuality to enter my public persona. I feel strongly enough about this matter, and it has affected my life so personally, I can't remain silent about it.
[4]: You've likely never met someone with this visa; it's exclusively for sales professionals of a foreign firm selling into the US, and nothing to do with the more common L-1 for consulting purposes.
[5]: The fact that I was obviously a net tax win for the UK made the application a formality, but it wasn't part of the official evaluation at the consulate.
[6]: People seem to forget about immigration in the US state-by-state same-sex marriage debate. It's a pure federal matter, so as long as DOMA is in effect, single-state same-sex marriage doesn't affect immigration rights.
[7]: I don't plan to ever move back as long as the official stance of the US government is that I am a lesser citizen by virtue of my sexual orientation.
[8]: And don't pretend they're not around. We all know that for every Accel, every Benchmark, every Index and KPCB and Red Point and Sequoia, there at least 2 firms that prey on entrepreneurs.
Updates since original posting:
I was corrected in the type of visa my partner had. It was an E-1 Treaty Trader, not an L-1 as I originally said.

The Ivy Tutorial Strikes Back

We're continuing on with the Jim Moores Guest Post series, as Ivy proceeds to destroy his source code directory.

After my previous problem with the build.xml file supplied in the Ivy tutorial I hit another issue that I thought I'd share. The default ant build file given in the Ivy tutorial generates an example source file within the ant file and then builds it. I, and I presume many others, based their new ivy-aware projects on that file. It has the default targets for downloading and installing ivy. Yeah. Problem is that one of the targets looks like this:

Can you see where this is going? Note where it says to delete the entire source directory. Yeah. That just happened to me.

I lost about half a week's work. It was my own fault. Because I'm a Perforce refugee and we've decided to try using git, I was more hesitant than usual about checking non-working code into my own code branch just because i wasn't so familiar with the git style of doing things. So while I was trying to get the ivy problems ironed out I ran ant clean which promptly deleted all my source code. All of it. Well, except the unit tests.

Please, let this be a lesson that I've learned so that you don't have to. While I accept that this was my fault, I can't help thinking that as a rule no build file should ever delete the entire contents from the src directory, particularly not one shipped as part of a tutorial that one would attempt on existing code.

Ant, Ivy 2.0.0, and java.net.UnknownHostException

This is a Very Special Guest Post by Jim Moores, another one of the OpenGamma team. We thought this would be useful to the interwebs, and until we have the OpenGamma blog ready to go, we're all ranting as one.

Ivy is a great way of avoiding shipping all the dependent libraries with your code and making upgrading dependencies smoother and easier, but it can still be a bit rough around the edges. For those of you new to Ivy, it's a dependency manager that is built on top of Ant that goes off and downloads and installs all your dependent libraries given a pretty simple list in an xml file. It hit version 2.0 a few months back and started to be really useful with more and more projects switching over to it. Of course Maven has done many of the same things for much longer, but Ivy doesn't require the whole head-warping change of viewpoint that Maven does. To be fair to Maven, I've only used it in passing when building the MyFaces JSF source code and while I did get it working, and was impressed, it took a bit of fiddling. My general understanding of Maven is that it's more declarative than 'normal' build systems - you tell it what you need and you have to rely on it figuring the whole thing out and it requires you let go of the idea of targets and explicit flow control (apologies to Maven fans if any of this is inaccurate). Ivy requires less of a leap of faith because it fits in with the Ant style that is more familiar to many but still brings the advantages of automatic dependency tracking. It can even install itself.

When I first tried Ivy three or four months ago I quickly found that some of the packages I wanted weren't available in the default ibiblio maven repository, in my case it was BerkeleyDB. I soon came across the excellent Ivy RoundUp repository. Ivy RoundUp doesn't actually host the packages themselves, it's a meta-repository that hosts information on how to download extract and repackage artifacts on demand. Unfortunately I couldn't just switch over to RoundUp because I needed some packages that were only on ibibio, so I had to delve deeper into the ivysettings.xml file to set up what is called a chain resolver. Chain resolvers do just what they sound like they do - they try one repository and then fall back to another if the package can't be found in the first. Here is my settings file:

Unfortunately this didn't work at all. It turned out that the example ivy build.xml in the tutorial contains the line:

  <property name="ivy.install.version" value="2.0.0-beta1">
</property>

And that 2.0.0-beta1 has a bug in it that means that chain resolvers don't work. Once I updated the ivy.install.version property to the latest version (2.0.0 at the time) it worked fine. This is still a problem with the tutorials, so be aware.

Since then I've needed to start using Hibernate in my application so I thought adding it would be easy as it's one of the most widely used Java packages. Err, no. After finding Hibernate in RoundUp easily enough (the latest ibibio version is really old), my build hung for some time, apparently unpacking a file, and then went totally exception-tastic on me.

[ivy:cachepath]
[ivy:cachepath] download.1.N65558:
[ivy:cachepath]
[ivy:cachepath]       [get] Getting: file://C:/DOCUME~1/Jim/LOCALS~1/Temp//jta-1_1-classes.zip
[ivy:cachepath]
[ivy:cachepath]       [get] To: C:\Documents and Settings\Jim\.ivy2\packager\cache\jta-1_1-classes.zip
[ivy:cachepath]
[ivy:cachepath]       [get] Error getting file://C:/DOCUME~1/Jim/LOCALS~1/Temp//jta-1_1-classes.zip to C:\Documents and Settings\Jim\.ivy2\packager\cache\jta-1_1-classes.zip
[ivy:cachepath]
[ivy:cachepath] C:\Documents and Settings\Jim\.ivy2\packager\build\javax.transaction\jta\1.1\build.xml:53: The following error occurred while executing this line:
[ivy:cachepath] C:\Documents and Settings\Jim\.ivy2\packager\build\javax.transaction\jta\1.1\packager-output.xml:28: java.net.UnknownHostException: C
[ivy:cachepath]         at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:541)
[ivy:cachepath]         at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:418)
[ivy:cachepath]         at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
[ivy:cachepath]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ivy:cachepath]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[ivy:cachepath]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

and another ten pages of related stuff which I'll spare you. There was clearly some problem in downloading the JTA 1.1 classes, which are a dependency of Hibernate. I tried upgrading to the latest version of ivy - didn't work. I then assumed it was an ivy package file and went digging around. When I looked to see the temporary file it was trying to open it wasn't actually there. Hmm. Just before that it implies that it's downloading the file in question. Could it be deleting it? I eventually decided it must be a problem with the file in the repository, so I went looking for help there. That's when I came across this thread on the roundup mailing list archives, and then onto the Roundup discussion of manually downloaded software.

It seems some of the software DOES NOT get downloaded automatically by ivy. Unfortunately, instead of giving you a nice error message telling you where to put it, you get a java.net.UnknownHostException. It turns out you need to download the artifacts manually from the Sun website and put them in the temporary directory ivy uses to store package downloads.

Once it did this it all works magically. Apparently the reason for this is that the download requires ticking a license agreement box which can't be done automatically. I hope this knowledge helps someone save some time. I think in the long run it'll probably mean me having a private repository for those artifacts, but for now I'll leave that for another day.

Dress Code Wars Episode IV: The Suits Strike Back

Wow. What a bit of a crazy couple of days since my last rant on the absurdity of forcing developers to work under dress codes. It hit Proggit, Hacker News, was tweeted by some high-follower-count people, and had more page views than anything else I've done. For those of you new to the feed, welcome!

I've been following the conversation on as many sites as I've been aware of, to try to keep the conversation going, and I was quite chuffed when there were more words on the subject written by !Kirk as by me. Particularly given my verbose writing style, that to me is a sign of a good conversation, which was the primary reason for the rant.

I just wanted to take some time to address some of the common arguments that kept coming up.

Kirk, you're a pompous Prima Donna and you're really not that good, either. Just shut up.

Ahhh, Argumentum ad Hominem. Where would the internet be without you? Just for the record though, I don't claim I'm that good a developer, and while I've been accused in the past of being a Prima Donna, I really don't let it get to me. I think it's more telling about the accuser than the accused.

I like dressing nicely

Bully for you! I'm sure you look lovely in your bespoke suit and custom made shirt. I'm positive you're quite the looker, and are attractive to both genders. You're a straight shooter with Management Potential written all over you.

If you like dressing nicely so much, why are you worried about a lack of a dress code? Why not have no dress code and still wear a suit? If you're just saying that you like dressing well, why not stay dressing well without forcing your sartorial choices on others?

I like South-East Asian food a lot. I think you should like it. But I sure as heck wouldn't have a company cafeteria only sell Thai food, and force you to eat in it every day.

It's better for your career to dress well

Thanks for looking out for your fellow software engineers, and incidentally, I actually agree with this one (if you desire a management role, or a role where you're working very closely with the business on a daily basis, you should dress up a little). But it should be your choice.

This one is an arms race as well. The only way this works is if you dress just a little bit better than everybody else, no matter what they're wearing. Norm is jeans and T-shirt? You wear chinos and a shirt. Norm is slacks and a shirt? You wear a suit. Norm is a suit? You wear a bespoke suit and a custom-made shirt and Hermes tie. No matter what, you always have to dress just a little bit better. You should actually be the most in favor of loosening dress codes, because then you don't have to dress that well/expensively/uncomfortably to still stand out.

Without a dress code, chaos will ensue

"Dress Appropriately" is the most powerful dress code you can have, because it completely eliminates all these arguments:

• People will dress like ragamuffin street tramps. Someone goes too far and you think it affects others? Take them aside and say "that's inappropriate."
• People won't dress up for meetings where formal attire is required. I dress down when I code, but when I meet with investors or customers, I dress up. That's just common sense, and if you actually hire people with common sense, they'll use it.

Fundamentally, we're not children, we're professional software developers. Removing arbitrary rules won't result in a Lord of the Flies situation breaking out.

I don't like seeing your manky feet

Believe it or not, this came up several times. As long as women can wear open toed shoes or nice sandals or whatever, it's just a rubbish argument. You don't like seeing a bloke's feet because you don't usually see a bloke's feet. Believe it or not, there are whole countries where very few people of either gender wear shoes on a regular basis. Get over your foot phobia.

My family needs to eat

Look, I completely understand that your family needs to eat, and if, on balance, the best job on offer is one which dictates a dress code, and you're aware of the likely attitude of the firm based on the existence of said dress code, you should take it. I didn't want a job with a dress code when I took the contract with Big Bank B, but my family needed to eat as well. Pragmatism is the primary virtue of the successful professional software developer.

Just be aware that if the firm has a dress code, in my opinion, that signals many things about the firm and its relationship with its technical staff. Go into it with your eyes open. And in my opinion, if you have a choice between a job with no dress code, and a slightly higher-paid job with a dress code, you should pick the one without the dress code.

Software Developers Should Never Have Dress Codes

On Saturday, I visited my dry cleaner. [1] Normally this wouldn't be a particularly momentous occasion [2], except that it was the first time in three weeks that I had gone. Up until the end of August, I had to make sure that like clockwork, at least every 2 weeks my Better Half or I made the trek to the dry cleaners, to make sure that I had suitable attire for my Cash Money Day Job. But it hit me on Saturday that this was a thing of the past: I no longer work for a company foolish enough to have a dress code for its technical staff.

When I first joined Big Bank B on my 6-month contract, it was the first time I had ever in my life had to dress up for work. I'd managed to work full-time for over 10 years, inside financial services and technology firms, and never had a dress code. In fact, my previous company (Investment Bank A) was so notoriously dress-down for financial services that there was a catty reference in the New York Times regarding the fact that people would (gasp! shock! horror!) show up to meetings in shorts and flip flops. At Hedge Fund V (the previous employer), while traders would usually show up in business casual, nobody cared what the geeks wore. However, the week before I started at Big Bank B, I had to go out and shell out a few hundred pounds on all the things that I didn't have enough of: work shirts, proper shoes, non-denim trousers. All this just to go into an office and code all day.

There's no justification for a dress code policy in this day and age if you view the primary function of a software developer to be the development of software. Right now I'm wearing shorts, sandals, and a T-shirt. Can anyone argue that I'd be more productive (in quantity or quality of technical output) if I was wearing chinos and a blue shirt? Or, perish the thought, a suit and tie? Definitely not.

However, the big question is Why? Why in the world do people still persist with this gross stupidity in this day and age? And why is it that big banks, in particular, seem to be so obsessed with dress codes?

In The Trenches, Fighting Bugs

A few years ago, I interviewed with Lehman Brothers. I noticed that all the technologists interviewing me (and it was an all-tech interview process) were wearing suits. Some of them had ties on, some of them didn't. I didn't wear a suit to the interview, and felt a little self conscious about it. [3] I asked one of the guys there why in the world they were so dressed up to write code all day.

Apparently this had come up in the past, and one of the Really Big American Bosses justified the policy with a phrase that has stuck in my mind ever since: "When a soldier goes to war, he wears a uniform."

There are so many things wrong with that statement that I can't even enumerate them, but let's start with the most glaring: Programmers Are Not Soldiers. Soldiers go to boot camp, a process designed specifically to break down their sense of self and recraft the mentality of the individual to mindlessly obey orders, even unto death. Soldiers go to war; soldiers fight people with weapons; soldiers live in barracks and eat in mess halls and, you know, act like they're in the military.

That's not what I do as a software engineer/craftsman: I think; I discuss; I code; I maintain. I don't live in close quarters with my coworkers; I get to eat what I want when I want; and while whiteboard discussions may get pretty heated, they're seldom settled with bayonets. I don't write code while firing an automatic weapon, and I don't engage in hand-to-hand combat with bugs. How in the world does a standard set by a military organization apply to a private sector financial technology job? [4]

Developers Should Be Heard But Not Seen

Second justification that I've heard for these ridiculous policies actually came from the hiring manager at one of the firms that bought out the carcass of Lehman, maintaining the dress code insanity: "if you ever have to go to the trading floor, there might be customers on there, and we won't want them to see you dressed down." That's right, because someone might see a derivatives trading floor, and figure out that there are actual programmers who make the software that keeps the 8 LCD displays each of the traders have full of blinkenlights.

It reminds me of firms that I've heard of that actually have separate entrances for the IT staff: customers and well-dressed individuals in Door A, all the plebes in Door B (heaven forfend that the twain shall meet). Big Bank B effectively had this policy, in putting all customer-facing activities in one building, while putting all non-customer-facing activities (IT, Operations, Back-Office, Middle-Office) in a whole different building.[5]

Yet most firms that have the separate-door, or separate-building policy, have it in place specifically so that the staff that don't have to dress in a certain way to satisfy the sartorial requirements of outside customers/investors didn't have to. Big Bank B said "not only are we going to shield you from view, we're still going to make you dress uncomfortably."

The Replaceable Engineering Unit

So given that dress codes are a net negative for developers, what could really be behind the correlation between probability of dress codes and size of the firm?

Remember, Big Banks actively fight against entrepreneurial/independent-minded technical staff. And then they make them all dress alike. The two, I think, are inextricably linked by one factor: a desire to smooth out the inherent human differences in employees of a firm grown too big for its management structures to cope with.

The Mythical Man Month was written at a time when almost all developers were working in large institutions, and it speaks to the levels that managers at those institutions will go to try to view individual employees as nothing but entries in a Gantt Chart. It doesn't work, because technologists have different skill levels, different interests, different speeds. But big organizations plan at very senior levels, where the people who are making decisions are unable to even conceptualize the differences between the people underneath them (because they're past the 150-person soft limit on social group sizing).

When faced with the requirement that you manage people beyond your cognitive limits, you can respond in one of two ways:

• Decentralize, and give massive autonomy to the people underneath you whose organizational unit size is within Dunbar's Number; or
• Standardize, and try to force individualism out of the equation.

Let's go back to Lehman Brothers. I think it's actually a telling example, because military units inevitably combine attempts at minimizing individual differences with organizational units that are below Dunbar's Number. So boot camp attempts to minimize individualism at the level of individual troops, but many decisions are made at the level of cognitive limits of social size.

The only way to combine the two though is to have senior executives/military officers making broad-sweep policy decisions ("we're going to start trading Convertible Bonds," "we're going to centralize all treasury activities," "we're going to invade Iraq"), but defer all decisions that rely on the individual skills and proclivities of individuals to the level of managers that actually understand the staff that they have. While many financial technology organizations claim to be applying that principle, ask yourself the following questions:

• Are individual line-level technical leads able to chose the major technologies that go into their software, or is there a central group that specifies the technologies that must be used?
• Are individual line-level managers free to determine the set of methodologies that work best for the team, or is there one over-arching set of methodologies that must be followed?
• Are individual teams able to determine their own deployment strategies based on their technology and the needs of the business, or is there a single policy that determines how software is going to be deployed?
• Do individuals have control over their programming environment (including hardware and software choices, and customization of everything they need), or is there a single image and single standard set of hardware that all developers get?
• Can people choose their attire, when there isn't a valid business reason to dress beyond the comfort level, or is there a dress code that must be adhered to?

My Advice To You

If you ever interview with a firm that has a dress code for techies, don't take the job. And more importantly, tell HR that this is part of the reason you're turning down the job. Because if you are able to find a firm that has reasonable pay, and doesn't have a dress code, that tells you a lot about its relationship with its technical staff. The dress code, on the other hand, is a very clear bright-line test to see an organization that undervalues the individualism that makes strong technologists as valuable as they are.

Plus, working in shorts and a T-shirt is pretty darn cool. Particularly when you get off at Bank or Canary Wharf station and everybody else is dressed up and already sweating at 8:30am.

Footnotes:

[1]: Not the best choice around actually. They're good, but Go Gay just a couple blocks up has a better reputation. However, as I don't drive, the extra couple of blocks I'd have to walk loaded down with lots of dry cleaning trumps an arguably better quality of service.
[2]: I can't/don't iron. When I was growing up, having shirts professionally cleaned and pressed cost so little that there wasn't a point in my parents doing it, and thus they never taught me to. And professionals do a much better job than your house cleaner does (if you're one of the people who has your cleaner do your ironing for you), so the cost really isn't much of an issue.
[3]: Seriously, whoever came up with that whole "dress your best for an interview" advice never had to nip out "for a Doctor's appointment" to interview. I mean, if you're actually gainfully employed, and you don't dress up in your current gig, dressing up is a dead giveaway you're interviewing. So if you do show up extremely well dressed for something that isn't a whole-day interview, it basically says "I'm unemployed."
[4]: I mean, while we're at it, you know who else wears uniforms? Ninjas. And while it's completely Awesome to imagine that we're all ninjas, we're not. If we were, we'd be too busy fighting pirates to get any software written.
[5]: Yep, that means that if you wanted to talk to the people using the software you built, you actually had to go to a whole different building a 5-10 minute walk away. Not that I ever did, mind you.