On the subject of Bamboo upgrade woes

Kirk

Custom shirts make up for Bamboo upgrade woes, right?

Thanks
Atlassian

Dear newly fellow Accel portfolio company Atlassian,

Yes.

Sincerely Yours,
Kirk

Me in my fancy new @atlassian #BestButNotYetPerfect shirt

Fanboi for life, yo.

Bamboo+Clover satisfy requests for test quality

Several times recently I've been asked how much and how well we test our product. Our board wants to make sure that we're building a quality product that won't result in first-customer black-eyes. Professional indemnity insurers want to make sure that action is unlikely to be taken as a result of a bug. Potential partners and customers want to make sure that a first-generation product isn't going to be completely ridden with bugs.

Sure, I can talk about how test-infected we are, and how seriously we take quality development. That goes some distance.

But OpenGamma doesn't have a dedicated QA department. And yet I know that the quality of our source code is exceptionally high. How do I convey this as quickly as possible?

I've found a technique that works pretty darn well.

Bamboo+Clover To The Rescue

All I really have to do is open a web browser and go to our Bamboo server, where I can show a screen that shows the sheer number of build, test, and Clover-coverage build plans we have, and show that they're running all the time. That gets over the "are you guys actually testing" barrier.

For those of you trying to keep track, that's only one laptop-screen-full of many. Don't try to extrapolate the number of projects/plans we have from that. :-)

But how good are those tests? That's where Clover comes in.

Pictures like this go a really long way to putting their minds at ease:

Yes, I know that just because you're executing lines of code it doesn't mean that the tests are actually checking the results and all that jazz. But as a first-cut sign of "we take testing seriously" graphs like that pretty much immediately end the coversation.

Don't Host Crowd and Jira in the same Servlet Container

This took up quite a bit of my Saturday figuring out, so I figured I'd add some pointers for other people to find.

Atlassian doesn't recommend that you host their applications in the same Tomcat instance, rather encouraging you to deploy them in different Tomcat instances and JVMs through the "Standalone" distributions. However, recommendations never stopped me before, so at OpenGamma we have two different sets of Atlassian infrastructure:

• One set of Confluence, Crowd and Jira for the FudgeMsg project running in one Tomcat container on one VM
• One set of Crowd and Jira for our corporate use running in one Tomcat container on one VM
• Bamboo and FishEye for OpenGamma corporate use running behind our firewall in their own standalone implementations in their own VMs.

Yesterday I tried to upgrade Crowd and Jira for the OpenGamma corporate installation. It wasn't pretty.

First of all, I ran into this KnowledgeBase issue, where Confluence, Crowd, and Jira all ship with different versions of the Felix jar for plugin management. This stopped the plugin system from starting for Jira (since Tomcat launches the Crowd application first), so Jira was pretty borked.

Then I ran into something far more pernicious, which other people should be aware of.

Reindexing Requires Crowd

If you're running your applications with Crowd, the application delegates user-related information to Crowd and uses a RESTful approach to loading the data. So far, so good.

However, when Jira attempts to in-situ upgrade an installation (which again they don't recommend), it will do its database wrangling, and then reindex the system with the new functionality (in our case, going from 4.0 to 4.1.1 to allow searches based on votes and watches on issues). All this upgrading it does in the application startup logic, which happens in the Tomcat main thread.

When it gets to reindexing, it then attempts about 10 different RESTful calls to your Crowd instance, as it doesn't have any caches populated on user data. However, while Tomcat has opened up port 80 (or 8080 or whatever) for Crowd, it hasn't enabled application dispatch to the Crowd servlets yet.

This means that all 10 remote calls (which are actually local to the single Tomcat instance since both apps are co-hosted) hang, and the entire server startup process fails.

The only workaround is to launch Crowd in its own servlet container, change your Jira's crowd.properties to refer to the new one, startup, and then undo what you've done.

The Moral Of The Story

When your software vendor recommends that you don't do something, don't do it unless you have an exceptional reason to do so.

If you're a software vendor and you support something but don't recommend it, still test it as there will be customers who ignore your recommendations and do it anyway.

Bamboo 2.2 - This is a Game Changer

Earlier this week, Atlassian announced Bamboo 2.2 had been released. Due to qCon pressures I hadn't had a chance to do a writeup on it until now, but I definitely wanted to mention it. It's a doozy.

The headline feature in 2.2 (and this is so big that I'm shocked that they didn't bump up to 3.0) is Elastic Bamboo. In essence, this allows you to use Amazon EC2 instances as your remote build agents, seamlessly. That's huge. Really huge.

Let's assume you're a developer trying to work out your build and test strategy. Your options right now are to:

• Run all your build agents under the Bamboo process. Not an option if you have to work with a heterogeneous environment, so I'll consider this the Hobbyist approach.
• Stick build agents on your various development machines as-and-when they match the exact configuration that you want to be able to test under. Doing this, though, will unpredictably mess with the machine's performance as Bamboo decides to take away all the resources to run a build-and-test.
• Push VMWare deep into the organization so that you can do your own VMWare instances to match the configurations you want to test against. Doing this, though, means that you have to have these instances up and running pretty much all the time, whether you're testing against them or not.
• Buy hardware just for testing. Uhm, haven't you heard? Capex is SOOO 2007, particularly for development infrastructure. Spend your Capex on 30" displays, yo.

Those are all fine options if you have friendly IT people who love doing things for developers as fast as possible. I seldom run into those. Elastic Bamboo gives you the option of running your build agents in EC2, starting them up to run a build and shutting them down when the build is done. Just think about that. You now have the option to have unlimited build-and-test configurations, completely outside the control of your IT department, and only pay for them when you're actually testing against them! Have a configuration you only supported for an old release that only a few customers are using? No worries, leave the instance in place, and it'll only launch when you change that really old branch. Have a configuration that only one customer is stupid enough to use in production? No worries, just leave the instance configured and run it every week or two. You now have no excuses for not exhaustively testing against your matrix all the time (I'm talking to you, Sonic!).

Along the way, they ended up realizing precisely what I realized when I tried to run build agents in between London, New York and Hong Kong, which is that Bamboo is a hog when it comes to log and artifact shipping over high latency links (and often low latency ones as well). They've fixed that, as if you're developing in Sydney and the closest AWS site you have is US East, you can't rely on Lan speeds anymore. All in all Atlassian resolved 5 issues that I was the original filer for [1]. That's a great track record, and I don't know very many commercial software companies that I could find that out for in the public record. Yet another reason I shill for Atlassian constantly. [2]

In addition to the whole "game-changing" thing, this is also extremely cool from a utility computing play. You probably could have done this yourself already (subject to the artifact/log shipping problems). But they're doing it for you. I think we're going to see that more and more from commercial software vendors, as they add support for major cloud computing platforms into their in-house software networks. Don't be surprised if you start to see anything that could benefit from cloud CPU resources start automatically coming AWS enabled.

I got cornered down t' pub at qCon by The Build Doctor and gave an only slightly inebriated rant about Electric Bamboo. For those of you who don't know what I look or sound like, here I am in all my gesticulating glory:

Major Caveat: My old employers, who are nothing if not early adopters for anything, upgraded an existing infrastructure with a lot of MSBuild, and it's not going particularly well for them. It appears that the officially closing off of BAM-1413 (MSBuild support [3]) actually broke the MSBuild support that was part of the NAnt plugin (which already shipped with Bamboo and supported MSBuild to great success). This is causing massive problems for the dozens of MSBuild-based projects at my old cash money gig. When it gets resolved I'll update.

UPDATE:: If you upgrade to the newest (2.1.9) NAnt plugin it'll fix the problems.

Minor Caveat: The Pre-Post Build Command Plugin doesn't work with 2.2. My old coworkers hacked a local version to get it to work, but there was an API change that broke the plugin.

The worst thing? I don't even get to use Bamboo at my current cash money gig. We're stuck with Electric Commander [4]. Sigh.

Footnotes:
[1]: For those counting: BAM-3177, BAM-1831, BAM-2989, BAM-2852, and BAM-2983
[2]: Fair disclosure: after I published that original mega-shill, they sent me an insanely great laptop bag, which is now my daily turtle shell. Moral: suck-up to cool companies and they send you great swag.
[3]: Oh, look, there I am again in the comments. I'm everywhere in this release.
[4]: Product by company founded by John Ousterhout, inventor of Tcl. I shall assume my gentle readers all have a strongly held opinion about Tcl one way or the other and say no more. But EC is no Bamboo, even though it's been around a lot longer. Or maybe it's just badly run here.

I'm a Relentless Shill for Atlassian

I've been told in the past by work colleagues that I'm a relentless shill for Atlassian products. Currently, we're using most of the suite (JIRA for issue tracking, Clover for Java code coverage, Bamboo for continuous integration, FishEye as a Perforce enhancement). The only piece of shelf-ware we've paid for but haven't rolled out yet is Crowd, because of security issues in our company's IT policies.

Sometimes it seems that I do nothing but try to sell their products, so I wanted to justify quite why I like them so much.

First of all, they produce good products. Each of those products is particularly good in some way, and the fact that they focus on plugin development as well as a single product means that they have an ecosystem which does more than a single product ever could.

Second of all, they integrate well, and so you get network effects by using them together. Also a plus side.

But the main reason why I'm confident in recommending them is that they have an extremely good attitude towards support, and that really matters when you're trying to work with this stuff in a real environment.

Other Companies: Read This. It Matters.

They have an approach to support that I call Extreme Openness. Essentially, as a user (or even prospective user) of their software, you have the level of insight into their internal development processes that you wish you had from even open source projects (where unless you're trolling through the mailing list archives you often can't establish).

For example, the JIRA system that they use to track their own work is publicly accessible. Not only that, you can freely post things in it. For example, I've been quite active in providing bugs and feature enhancement requests. They make all that information available, and I can easily help myself in seeing whether things I've encountered or only dreamed of are on their roadmap, and vote for what I think is important. This gives me an insight into roughly what their priorities are. I may not always agree with them, but at least I'm aware.

It also means that I can see what's forthcoming, up to several releases in advance. This actually meant the difference between our buying a product and not buying it, because we saw that the roadmap involved some features that we wanted. For example, we were evaluating our Continuous Integration technology, and came up with the standard set of contenders (Bamboo, Hudson (which we already used), CruiseControl (which we already used), CruiseControl.NET (which we already used), and TeamCity). We ended up not buying anything, in part because we saw that Bamboo was working on the key feature we needed (remote build agents) and we wanted to wait for its implementation. As soon as they implemented it, we bought the product because it was better than the alternatives. Now we're almost done putting every single project firm-wide into our Bamboo installation.

Contrast this with your general Classic Big Software Company. Maybe you'll drip out bits and pieces of what's forthcoming to key customers, but you definitely won't let all your bugs out into the open like that, and you definitely wouldn't let customers and competitors see what you're working on (and sometimes more importantly, what you're not working on). I like the Atlassian approach quite a bit, because it gives me insight. And if I'm making a commitment to a platform like this, I need to have long-term confidence in it as a platform, and that means insight into development.

To give a contrasting example, SonicMQ comes out with a new release. Did I even know they were working on it? No (and at this point I have a direct line to the head of SonicMQ support for EMEA). Could I easily find out what was in it? No. There're reference notes, but they don't really tell you what's going on. As a customer, did they even email me? No. I found out meeting with the EMEA support manager on a standard meeting that it had come out as he asked me what we thought of it.

Next, you can download all their products and use them in some limited way. No contacting a sales person, no having to hunt for them, you just download them and use them. With SonicMQ, although I'm a customer, in order to get an "official" download of a new release, I have to have my account manager approve our account getting access to a new release, and then it goes through to the worst download manager in the world (including the new Sun JavaSE download web system) and hunt for the downloads that I need. That's rubbish. Atlassian ones are same download no matter what you're going to use, whether eval or paid. I like that a lot.

And they make it really easy to work with their support system, and they're really responsive. They aren't using some archaic system, they aren't making me work with something that makes me rip my hair out. Their system works really well (it's built on JIRA, so they're eating their own dog food), they're fast to respond (although the time zone differences between Sydney and London make for high latency sometimes), and they have proven themselves very intelligent over support channels, and I have no doubt that very often I'm communicating directly with a developer on the product.

Could you scale this up? I think so. Even when I was at BEA working on WebLogic Server, and we had over 50 core developers on the product, we all still did some type of support, we had public downloads of the major products, and while we didn't have public JIRA access, we did announce what features were forthcoming quite a bit and communicated on particular features on public newgroups. Perhaps that's one reason why at the time people liked us.

But you have to support this if you're trying to sell to developers. You have to think of the fact that many of your customers are extremely intelligent (some of them are lesser-skilled, and you have to support them too), and they won't put up with dealing with standard support drones. If your support engineers ever see a script they have to follow, and I sense that, I'm going to do everything in my power to never work with you again.

So if you're planning on starting a software company, let me give you some lessons that you can learn from Atlassian:

• Make it easy (super duper extra easy) for me to download and use your products. Whether I've paid you or not.
• Don't ever stop me from downloading your manuals. Before I even think of installing your software, I'm doing to read your manuals. Cover to cover.
• Open up your processes and let me see what you're working on. I really want to know, and I really want to know in quite depth and detail. Your competitors may find out, but quite frankly, if you're that worried about individual features, you can hide those features themselves. And if your business model requires that nobody knows what you're working on, you're going to fail anyway.
• Have good support. Have excellent support. Make me think that my annual support contract is a bargain. I'm going to pay you, so make me happy I'm paying you.
• Make Good Software.