Bamboo 2.2 - This is a Game Changer

Earlier this week, Atlassian announced Bamboo 2.2 had been released. Due to qCon pressures I hadn't had a chance to do a writeup on it until now, but I definitely wanted to mention it. It's a doozy.

The headline feature in 2.2 (and this is so big that I'm shocked that they didn't bump up to 3.0) is Elastic Bamboo. In essence, this allows you to use Amazon EC2 instances as your remote build agents, seamlessly. That's huge. Really huge.

Let's assume you're a developer trying to work out your build and test strategy. Your options right now are to:

• Run all your build agents under the Bamboo process. Not an option if you have to work with a heterogeneous environment, so I'll consider this the Hobbyist approach.
• Stick build agents on your various development machines as-and-when they match the exact configuration that you want to be able to test under. Doing this, though, will unpredictably mess with the machine's performance as Bamboo decides to take away all the resources to run a build-and-test.
• Push VMWare deep into the organization so that you can do your own VMWare instances to match the configurations you want to test against. Doing this, though, means that you have to have these instances up and running pretty much all the time, whether you're testing against them or not.
• Buy hardware just for testing. Uhm, haven't you heard? Capex is SOOO 2007, particularly for development infrastructure. Spend your Capex on 30" displays, yo.

Those are all fine options if you have friendly IT people who love doing things for developers as fast as possible. I seldom run into those. Elastic Bamboo gives you the option of running your build agents in EC2, starting them up to run a build and shutting them down when the build is done. Just think about that. You now have the option to have unlimited build-and-test configurations, completely outside the control of your IT department, and only pay for them when you're actually testing against them! Have a configuration you only supported for an old release that only a few customers are using? No worries, leave the instance in place, and it'll only launch when you change that really old branch. Have a configuration that only one customer is stupid enough to use in production? No worries, just leave the instance configured and run it every week or two. You now have no excuses for not exhaustively testing against your matrix all the time (I'm talking to you, Sonic!).

Along the way, they ended up realizing precisely what I realized when I tried to run build agents in between London, New York and Hong Kong, which is that Bamboo is a hog when it comes to log and artifact shipping over high latency links (and often low latency ones as well). They've fixed that, as if you're developing in Sydney and the closest AWS site you have is US East, you can't rely on Lan speeds anymore. All in all Atlassian resolved 5 issues that I was the original filer for [1]. That's a great track record, and I don't know very many commercial software companies that I could find that out for in the public record. Yet another reason I shill for Atlassian constantly. [2]

In addition to the whole "game-changing" thing, this is also extremely cool from a utility computing play. You probably could have done this yourself already (subject to the artifact/log shipping problems). But they're doing it for you. I think we're going to see that more and more from commercial software vendors, as they add support for major cloud computing platforms into their in-house software networks. Don't be surprised if you start to see anything that could benefit from cloud CPU resources start automatically coming AWS enabled.

I got cornered down t' pub at qCon by The Build Doctor and gave an only slightly inebriated rant about Electric Bamboo. For those of you who don't know what I look or sound like, here I am in all my gesticulating glory:

Major Caveat: My old employers, who are nothing if not early adopters for anything, upgraded an existing infrastructure with a lot of MSBuild, and it's not going particularly well for them. It appears that the officially closing off of BAM-1413 (MSBuild support [3]) actually broke the MSBuild support that was part of the NAnt plugin (which already shipped with Bamboo and supported MSBuild to great success). This is causing massive problems for the dozens of MSBuild-based projects at my old cash money gig. When it gets resolved I'll update.

UPDATE:: If you upgrade to the newest (2.1.9) NAnt plugin it'll fix the problems.

Minor Caveat: The Pre-Post Build Command Plugin doesn't work with 2.2. My old coworkers hacked a local version to get it to work, but there was an API change that broke the plugin.

The worst thing? I don't even get to use Bamboo at my current cash money gig. We're stuck with Electric Commander [4]. Sigh.

Footnotes:
[1]: For those counting: BAM-3177, BAM-1831, BAM-2989, BAM-2852, and BAM-2983
[2]: Fair disclosure: after I published that original mega-shill, they sent me an insanely great laptop bag, which is now my daily turtle shell. Moral: suck-up to cool companies and they send you great swag.
[3]: Oh, look, there I am again in the comments. I'm everywhere in this release.
[4]: Product by company founded by John Ousterhout, inventor of Tcl. I shall assume my gentle readers all have a strongly held opinion about Tcl one way or the other and say no more. But EC is no Bamboo, even though it's been around a lot longer. Or maybe it's just badly run here.

Perforce, FishEye, CC.Net, Labels, Oh My!

My employer is a big Perforce shop. Although we've got some CVS repositories still lying around for legacy reasons, over the past few years almost all of our source code has managed to make its way into four Perforce instances (separated for geographical and organizational reasons). We use it for pretty much everything, and we've got our entire software development methodology based around its use.

That includes a number of tools that we use that are integrated with it:

• Code review tools (all built in-house)
• Software release and distribution tools (also all built in-house)
  
• Bamboo, as our current continuous integration system
• FishEye, as our SCM web-based visualization system
• Jira, as our issue tracking system

In addition, in the past, before we moved to Bamboo, we had three other CI systems hitting it, CruiseControl, CruiseControl.NET, and Hudson (all but one retired, and that one is being retired shortly).

All of them are hitting it on a regular basis pretty hard to pull metadata (what happened when) and data (to pull the actual SCM data). This is a tale of where it started to go wrong and how we fixed it.

Problem Diagnosis
We noticed that FishEye was behaving pretty badly when it was rescanning a repository. We do this on a regular basis, because FishEye won't automatically rescan changelist descriptions (at least for Perforce), and sometimes we go back and edit changelist descriptions to hook them up to Jira issues or end-user requests, or just to provide more clarity on what someone did in the past. Since FishEye won't pick up those changes (if it's already processed changelist 123456, why would it process it again?), we have to periodically completely rescan our master Perforce server to pull out all changes.

Our master Perforce installation is relatively big for a non-gaming house (I can't give numbers for this in a public forum), and rescanning it was taking several days. In fact, it was taking so long, that we couldn't even plan on doing it over a weekend: if we kicked off the process on Friday night after New York went home, it wouldn't be done by the time Hong Kong and Tokyo came in on Monday. This was a problem.

Also, when Fisheye was doing this, the whole repository was noticably slower. Since the rescans started to take place over normal business hours when people were trying to work, this made the problem doubly bad: not only was Fisheye not available, it was making Perforce slow for users during the rescan.

So I started diagnosing what was going on, and the process that was taking the longest was processing labels. This alone was taking over a day, and because of the way Fisheye does this, and because forking to query Perforce on Solaris 10 is a painful experience, we needed to get this number down. We had way too many labels covering way too many revisions.

Perforce Is Not CVS
The metadata table that holds this data in Perforce (db.label) was absolutely massive: roughly 7GB, or about 60% of our entire Perforce metadata storage. This wasn't good, and it's far from ordinary. When I started investigating, we had over 12000 labels. That's a lot for the number of projects we're hosting and the number of releases we've done, but it turns out that 10000 of them were created by CruiseControl.NET builds.

This was largely done from a misconception of what labels are good for, and is basically a remnant of CVS-style thinking. In CVS, because revisions of files can be interleved together, if you want to reference the state of a particular subsection of the repository as of a particular point in time, you have to add a Tag to every revision of every file involved. This actually goes in and adds metadata for that revision to say it's part of BUILD_5 or some such thing.

A Perforce Label is different. Perforce has atomic, monotonically increasing changelist numbers, where each number uniquely identifies the state of every single revision in every single file in the entire server. And I can use them in all types of contexts. In particular, I can pull down the state of a particular project as of a particular changelist number: "Give me the Fibble project as of changelist 12345." This is how Perforce-integrating CI systems work: they query Perforce to say "tell me all the submitted changelists that I haven't seen", and then sync up particular areas as of those changelist numbers. Therefore, a changelist number is the equivalent of a tag applied to every revision of every file in the whole server.

A Label, on the other hand, is there for cases where you need to refer to revisions of files across multiple changelists. The key use case here is patching. Let's say that you've released version 1.2.0 of your software, and then you start adding changes to support 1.2.1. But in the meantime, you discover a complete showstopper bug that requires you to put out a special release with only that bug fix in, and not any of the other changes you've got prepared for 1.2.1. Since the 1.2.1 features have already started going in, if you try to pull all the source code as of the point where the critical bug fix went in, you'll get the 1.2.1 changes as well. In this case, you create a Label, and you put in the label all the 1.2.0 revisions, as well as the revisions just for the showstopper fix, but none of the rest of the 1.2.1 changes. This gives you a way to refer to a collection of revisions of files across different changelists.

What our CC.Net server was doing (and as I didn't install it, I don't know if this was default or intentional on our part), was for every single build of every single project, creating a new label which contained the state of all the files for that build. But you don't need to do that: since it was pulling everything down as of a particular changelist number, all that accomplished was saying "The revisions for build 30 of the Fibble project are the same as the files as of changelist 12345," which Perforce changelist numbers already give you. So it was unnecessary metadata that was clogging up the server intolerably.

Solution
We deleted all of those 10000 labels (the fact that we had already moved all those projects to Bamboo made this a no-brainer, as we were no longer using the CC.Net provided builds at all). But the size of the db.label table didn't actually shrink. In fact, it actually got slightly bigger during that time.

This is because Perforce as an optimization assumes that you're constantly increasing the amount of metadata that you're putting in as time goes forward, and so doesn't prune the size of the tables. So they were still too big, and sparse, so although we didn't have to do as many queries across them, it was hurting the OS caching of the files.

The solution there is to restore from a checkpoint (a Perforce checkpoint is a gzipped text file containing every binary record in your metadata in plain text; it acts as your primary backup mechanism for the metadata records that it keeps along with your RCS files). Before we did this we went through a pruning process for old workspaces that people hadn't deleted (removing several hundred) to get the db.have file down in size as well.

After this was done, the size of our db.* tables went from 11GB to 3.0GB (our gzipped checkpoints went from 492MB to 221MB). And the FishEye scans went from 3 days to 5 hours. Job done.

Morals
So after all that, what we can draw from this is:

• Don't label unnecessarily. They're relatively expensive in terms of metadata, and you probably don't need to do it.
• Shrink your metadata. Remove anything that you no longer need, including old workspaces.
• When you prune significantly, restore from checkpoint. This is the only way to get Perforce to really shrink down your metadata tables on disk.

Linux Fork Performance Redux: Large Pages

After a comment from Kostas on my last Linux Fork Test post, I worked with my esteemed colleagues to try things out with large page support. Wow, what a difference that made on Linux.

The theory here is that forking involves playing around with TLBs entries quite a bit. Since a monstrous full heap will have a lot of 8KB page TLB entries to contend with, if we shrink the number of TLB entries by a factor of 256 (by working with 2048KB pages rather than 8KB ones), you'll limit the amount of time that the kernel is spending mucking with them.

First of all, make sure you read this: The large memory support page from Sun. Now that we've gotten the formals out of the way, here's some fun we had.

First of all, a stock RHEL 5.2 installation has a HugePages_Total set to 0 (cat /proc/meminfo). No huge pages whatsoever. So you need to bump that up. For my test (maximum 2GB fully populated heap on a 4GB physical RAM system), we decided to set that to 3 GB, which is 1536 2MB pages.

echo 1536 > /proc/sys/vm/nr_hugepages isn't guaranteed to actually do that, and the first time we ran it, we ended up with a whopping 2 HugePages_Total. Second time bumped us up to 4. So we went on a process hunt to eliminate any processes that were stopping us working, and got things down pretty small. Now we were able to get up to 870, which was good enough for my 1GB tests (which indicated the major performance degradation anyway), though not for the 2GB test. (Yes, I know that you're supposed to do this on startup, but I didn't have that option so we did what we could).

And so I kicked things off with the -XX:+UseLargePages flag. Fail.

Every single time I got a Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (ero = 12). And nothing would run. Well, damn!

Turns out those little tiny bits that they say in the support page about not working for non-privileged users are completely accurate. These all went away when I had someone with sudo rights run the process as root, and all my numbers are from running as root. So just assume that even 1.6.0_10 ain't going to allow you to allocate any large pages if you're not root.

So he ran things as root (and I re-ran things as non-root without the UseLargePages flag since I changed the test slightly). Here's some fun comparison:

Heap Size	Large Pages Time	Normal Pages Time	Speedup
128MB	3.589sec	13.217sec	3.68 times faster
256MB	3.62sec	15.314sec	4.23 times faster
512MB	4.638sec	36.692sec	7.91 times faster
1024MB	3.885sec	67.062sec	17.26 times faster

Oh, and that speedup between 512MB and 1024MB? Completely reproducible. Not sure what precisely was going on there, I'm going to assume my test case is flawed somehow.

It's happening so quickly at this point that I'm quite suspicious that all I'm measuring is the /bin/false process startup and teardown performance, as well as the concurrency inside Java. I don't actually think I'm testing anything of any meaningful precision anymore. Maybe at a few million forks or with higher concurrency, but I've achieved essentially a constant amount of time spent forking, so I've gotten out of the heap issue really.

So it turns out that you really can make Java fork like crazy on Linux, as long as you're willing to run as root. And I don't know why and my naive googling didn't help. If someone can let me know, I'd really greatly appreciate it.

Did any of this help Solaris x86? Not one whit. Adding the -XX:UseLargePages flag (even though Solaris 10 doesn't require any type of configuration to make it work) didn't improve performance at all, and Solaris was still twice as slow as Linux without the flag.

Linux: A (Less) Terrible Choice For Java Continuous Integration

Where we last left our intrepid developer, he was floating in a sea of Bamboo+Perforce misery, blithely assuming that moving from Solaris 10 to Linux would solve all of his problems. Oh, what a blissful world he would live in! What joy he would have no longer having to deal with Solaris swap space reservation woes!

Since then, though, he's gotten access to two (almost) the same machines, one running Solaris 10 x86, and one running RHEL 5.2. And while he's vindicated, he's nowhere near vindicated enough for his liking.

Executive Summary: Runtime.exec() performance under Linux is superior to that of Solaris 10 x86, but nowhere near as superior as it should be.

Inspect the following micro-benchmark (and apologies that I don't have the nifty code viewing tools that other bloggers do):

import java.util.concurrent.*;

public class ForkTest
{
    public static void main(String[] args) throws InterruptedException {
        int nThreads = Integer.parseInt(args[0]);
        int nSlabs = Integer.parseInt(args[1]);
        byte[][] bytes = new byte[nSlabs][];
        for(int i = 0; i < nSlabs; i++) {
            bytes[i] = new byte[80 * 1024 * 1024];
        }
        ExecutorService executor = Executors.newFixedThreadPool(Integer.parseInt(args[0]));
        long start = System.currentTimeMillis();
        for(int i = 0; i < 1000; i++) {
            executor.execute(new Runnable() {
                public void run() {
                    try { Runtime.getRuntime().exec("/bin/false").waitFor();}
                    catch (Throwable t) { t.printStackTrace(System.err); }
                }
            });
        }
        executor.shutdown();
        executor.awaitTermination(10L, TimeUnit.MINUTES);
        long end = System.currentTimeMillis();
        double secs = ((double)(end - start)) / 1000.0;
        System.out.println("" + nThreads + " - Forking 1000 times took " + secs + " secs"); 
    }
}

Essentially what the test is doing is:

• Creating a fixed size (-Xms and -Xmx set to the same value) heap
• Allocating some slabs of memory (to fill up the heap) (where I refer to "empty heap" tests, this was set to 0)
• Creating an ExecutorService with a certain number of threads
• Running through 1000 tasks, where each task involved running /bin/false in a sub-process and waiting for it to terminate

I felt that this was probably the best way that I could possibly test whether the behavior that I felt was causing Bamboo to perform badly with Perforce repositories would also affect Linux. Turns out I'm half right; Linux will still suck, but suck 50% less.

General Parameters
Both machines were Sun X4100 (non-M2) servers with two dual-core Opterons (one a pair of 275s and one a pair of 285s), and 4GB physical RAM. All tests were done on 1.6.0_10.

Empty Heap Comparison
Here's the first test: Run through everything with no slabs allocated (empty heap) and see how fast we can go. Results in this graph, but the highlight here is that Solaris was very little affected by the size of the heap, but was always slower than Linux, by roughly a factor of 2.


Full Heap Comparison
Next test was to fill up the heaps and then try. Here you can see that the heap size completely determines performance, but Linux is always better (factor of 2 again).


Full/Empty Comparison
Here are just the Linux values plotted out, and it's pretty clear what's going on.


Interesting Observations
Note that the sweet spot here is two threads. No more, no fewer. On Linux, Solaris, empty, full, doesn't matter. You want to fork as fast as you can? Have two threads doing it. Admittedly, these tests were on 2-socket (4 core total) machines, but when I repeated this on Solaris on one of our 8-socket x4600 machines (16 core total), I ended up with the exact same thing: 2 threads was always ideal.

Uninteresting Observations
"Hey, Kirk, you just proved that forking an empty virtual space is faster than a full virtual space! Big whoop! You're such a Java-specific Moran that you forgot all that from your 31337 days!"

Well, no, not really. What I specifically established is that:

• On neither Linux nor on Solaris are the Sun-provided JVMs using any of the fast-subprocess-spawn operations available to them.
• This is a really clear win for anyone working on CI systems to nag Sun or the OpenJDK crowd to get changed and fully tested.
• Linux is still a factor of 2 ahead of Solaris here. I would have hoped a fast-spawn implementation would be a factor of 10, but I'll take a factor of 2 gladly.
• There is definitely something happening in a fork+exec pattern on Linux which is VM-specific, which means that our suppositions earlier that Linux is going to do optimistic copying aren't panning through to eliminate the costs of a fork with a large amount of allocated memory.

Recommendation
So let's say you have a large, long-running Java server which benefits from having a relatively large heap (like, oh, I dunno, a Continuous Integration server), and you have to shell out constantly because a vendor refuses to support you well (speaking of which, I've actually formally asked Perforce to document the protocol).

If you really want to avoid the whole C++ thing, essentially you should be:

• Forking to a second JVM instance to run a small Java application.
• That small Java application should itself shell out to your command-line application (remember: on Linux with a 128m empty heap you can get up to 175 Runtime.exec()/sec, which is not too shabby)
• Have that small java application just pass stdin/stdout to the parent application.

Yes, this seems completely retarded. I can't believe I'm recommending it. But it would actually work as a consistent approach to the Perforce+Continuous Integration problem.

Culture of Testing and Continuous Integration

Ran across a pretty good post on how defects can lead to never being able to meet any of your three basic criteria (features, quality, time). You should read it. But the Kirk summary here is that while I'm not surprised, there are still a lot of organizations who haven't gotten Test/Continuous-Integration Infected. I've known for years that it's a key component to long-running development project success (hint: if you're using source-code control, it's long-running), but this is a pretty good summary about why if you always skimp on "Quality", you're actually always going to lose on Features (too busy fighting bugs to add them) and Time (end of the cycle is extended dealing with defects in key functionality).

A few quick and recent anecdotes:

• That Java5 vs Java6 ThreadPoolExecutor bug? Caught by our continuous integration server testing against a plethora of JVM and OS combinations.
• In some C++ code, we caught a race condition by running tests over and over under continuous integration (and because our build agents run on ordinary development machines, this exposes them to a variety of configuration and machine load scenarios), and refining the tests until we found the lowest-level part that was failing.
• We have logic that works with some (intentionally) unreliable middleware that people too often assume is reliable. By having tests that spuriously fail under load (the tests assume more reliability than you can actually achieve on machines that are otherwise loaded), it became a good stick to beat the "best-effort is good enough" crowd internally. Tests periodically failing don't lie: I can say "15% of the time, this test fails, and it's using the API as intended."
  

I've had some discussions with people about why they don't move to a CI and Test-Infected model, and I've heard a few arguments. I think most of these come down to two factors.

First, starting is hard. If you're on an existing product which you don't actually like the internal code path on, it can be really difficult to start testing. And if your code isn't written to make it easy to test in isolation, all your tests are system-integration tests, which are pretty poor at providing low-level test cases to track the code itself. The problem seems so daunting that you don't even really start at all.

The easiest solution here is the obvious one: start with some system-integration tests, and as you start working on existing parts of the system (either fixing them or replacing them) start writing lower and lower level tests. Use a code coverage tool (like the shill I am, I'm extraordinarily partial to Clover for Java development), see how you're doing, and make sure you're constantly expanding the code paths that you have under testing. And use it interactively: turn tests on and off and compare your coverage to find out how good your low-level tests and system-wide tests really are at testing your system.

Over time, as your test base grows, you end up exactly where you should be: all non-trivial changes to the code result in changes to the test suite to establish that the code is behaving the way you expect, and you're running them constantly to give you positive feedback that your code is working properly.

But this isn't the rant-filled part of the blog. The following is.

The Test-Infected crowd turns good software engineers against testing. Seriously. If you do anything to an extreme, you turn off software engineers who would otherwise listen to you. So just shut the hell up.

[In particular, if you work for a consulting company and are trying to sell development services, please don't bother trying to tell me how to develop software for long-running maintenance. How in the world would you know? By your very nature, you don't do long-running system development or maintenance. Why would I trust you in any way except as a clearing house for ideas you've seen other people doing? Other people who are probably failing and thus hiring you in the first place? Great. You've seen lots of fail.]

First of all, Test-Driven Development is retarded. There's a reason why every example I've ever seen of it is of extremely trivial code: because it's impossible to do well otherwise. When I start working on something that's non-trivial, do I know enough about the internal implementation details or the external contract that it will present to the rest of my system to be able to write a test that I feel has proper coverage of the internal code paths before I start writing it? No. That's part of the crafting of complicated software: you have to start with a concept and then start iteratively refining it until you end up reaching code that works well. If you actually have so much insight into how you're going to implement something that your Test-First code is going to be a reasonable exercise of your code, then your code is so trivial that there's no point writing it in the first place.

Second of all, I have absolutely no patience whatsoever for people who spend their time agonizing over whether something is a Unit, Functional, Integration, System, Performance, Scalability, blah blah blah test. Seriously. The whole thing is complete engineering navel-gazing. Here's a hint: if you have essentially two completely implementations of some logic, one just to make it a "Mock" implementation (but which has to adhere so strongly to an existing contract that you've written it twice), then you've failed and wasted everybody's time and effort doing it. Performance and scalability tests I can see, because you're probably going to run them manually on a periodic basis under a more controlled environment, but spending days agonizing over your mock framework? That's just crazy.

And unfortunately, I get comments like these from other programmers, who see all the Test-Driven Development and "Mock The World" arguments and it turns them right off. They say "if that's what it means to do all this great testing stuff, I want nothing to do with it," and then they produce crappy software. Fail all over.

You want to write some effective tests? Here are some hints:

• Layer-cake your internal design. Then test up the layer stack. Layer A requires Layer B which requires Layer C? Test all three. Bug only shows up in the tests for Layer A? Then there's a bug in Layer A. It's just that simple.
• Isolate your major components. Write an interface that you know will be easy to mock (something going to a database? write a quick in-memory representation for it) and then put a façade on the low-level code. (Note that if your layers are done as interfaces, it becomes extraordinarily easy to combine this with layer testing to have "proper" unit testing).
• Check your coverage. Unless you're running with something like Clover, you're not going to know how well your tests exercise your code paths. So use a code coverage tool to draw a spotlight on the areas that you're not covering particularly well.
• If you have a big, existing system, write system-wide tests. They're far better than nothing, and they give you a basis for your later refactoring to get to a designed-for-testing scenario.
• Continuous Integration. Make sure you have a single-step "compile the world" target somewhere (even if it's a shell or batch script), grab Bamboo (just don't run it on Solaris, natch), and start. You'll never go back.
• Design for Testing. Make sure as you write code that you think "How am I going to test this?" and then write the tests right away (same SVN/Git/P4/CVS submission), changing your logic as required to make it easy to test.
• Write tests as documentation. Writing some code that someone else is going to use? Create your documentation as a test case. The only documentation. Then if someone says "Hey, you wrote Fibble, how do I use it?" you can point them at the test case, and they can replicate it. If they say "Hey, that's great, but you didn't show me how to use the Fubar functionality," you add a test case that demonstrates that. If your tests start getting so complex that you can't point to any of them as a reasonable real-world demonstration of your code, you need to fix your design. Or add simpler test cases just for demonstration. But once you're doing demos/docs as tests, you can be pretty confident your design "smells" good and if people copy your demo/doc it's always going to work. Because you're testing it constantly.
  

This all becomes second nature over time, but if you ever hear somebody say that you should start by writing your unit test first and pull out your mock-object generator and auto-complete all the methods you haven't even written yet, just poke them in the eye until they go away. They're probably too busy trying to charge your company for a consulting gig to get anything real done anyway.

Solaris 10: A Terrible Choice For Java Continuous Integration

As some of you might know, I'm not permitted to run Linux at work. Rather, I have my choice of Windows or Solaris 10 x86. (And no, I'm not allowed to run OpenSolaris, it's boring old Sol 10). I do a lot of Java. I love Continuous Integration. I started pushing Bamboo here in house. And lo and behold, I converted the masses, and we ended up merging projects from CruiseControl.NET, CruiseControl, and Hudson all to one lovely Bamboo instance. We ended up with 110 build plans in Bamboo, all but about 10 of which (dependant and overnight plans) were hitting Perforce all the time to determine whether they should build.

And then things went truly, truly, horribly wrong, to the point of my almost rescinding my Atlassian Shill status. But it turns out that it's only partially their fault. It's actually Sun's fault. Follow me here on a path down process forking details of joy.

What a Java-based Continuous Integration server does when dealing with Perforce, because Chris won't allow them to open up the protocol or provide a usable Java interface, is:

1. Check to see if there have been any changes in Perforce
2. If there have, sync code and run a build
3. Go to 1

For step #1, you've got two choices. You can either say "give me all changes, and I'll figure out whether the change applies to this build plan" or you can say "give me any changes that apply to each of my build plans." The former involves much fewer Perforce interactions, but the latter is directly supported by Perforce itself, which makes it easy to implement, and thus I would argue more correct, since Perforce has a lot of logic about exactly this type of operation (I want a CI system, not another SCM system).

Here's where the whole thing turns to Fail.

When we started adding more and more projects to our Bamboo installation, it started running more and more slowly, to a point where on a 4-core dual-dual-Opteron box it was averaging a load average of 7, with up to 80% of time spent in the kernel. That type of load was so extreme that all sorts of things started going wrong completely mysteriously.

Because of the whole Java+P4 issue, you have to shell out to run the p4 binary to interact with the Perforce server. Under Unix implementations of the JVM, that involves essentially a fork + exec (under Windows it does not, so this is arguably a superior behavior). Here's where things get sticky.

Working with Atlassian, we figured out that in part, this was because Bamboo was being over aggressive in hitting Perforce, and under aggressive in caching things that seldom/never change. Moreover, it was the cost of actually performing the fork far more than the cost of the subprocess that was killing you. 2.1.2 (to be released on Tuesday) resolves this, and so checking whether there are changes goes from 3 p4 invocations, to 1 p4 invocation. A factor of 3, which in their artificial test suite results in a 76% performance improvement.

Now the one thing that ties everything together here is that the parent process here is a JVM. But not just any JVM, a JVM tuned to running big web applications with a lot of users (the same Tomcat instance hosted our JIRA instance as well). And what's the generic rule of thumb for running a JVM for a Servlet container? More heap. More heap, more heap, more heap.

More heap, more betta, right?

Wrong.

Turns out that JVMs on Solaris 10 blow (funny that, given that they're both from the same company; you might think Sun would have an interest in making sure Java ran best on Solaris, but whatever).

In order to do a traditional fork, although there is a copy-on-write optimization to avoid actually duplication of system memory that's going to be abandoned, in order to comply with the Posix standard for fork, Solaris has to reserve enough swap space for the forked subprocess. Now in top, you won't see it, because that swap space isn't actually in use, just reserved, so you can't have it, but it's not being used. Great.

This is terrible, and is precisely why things like clone() were invented in Linux and used to great success. My best recollection (can someone clue me in here, Lazyweb?) is that this is what Runtime.exec() and ProcessBuilder.start() do on Linux-based JVMs. Solaris traditionally didn't have such a beast, so you're stuck with old Posix fork() behavior. Which sucks for this.

So what Bamboo is doing is running 3 2GB heap forks for every Perforce interaction, which is really really really bad.

When one of my colleagues did a fork test here to see how fast he could do a fork+exec in C with a 2GB memory allocation (and forking to a really small sub-process) on the same hardware and OS configuration, it turns out that doing that you're limited to 3/second, and it consumes about 50% of the machine's total CPU utilization in the kernel. Given that Bamboo was attempting to do the same thing in multiple threads, the fact that it was hitting 80% kernel utilization makes complete sense.

Hence, when I reduced the heap to a mere 512MB, Bamboo actually ran faster. Much faster. Back to being usable factor, even with my 110 plans. Once 2.1.2 comes out we should be really rolling, and we might be able to achieve our ultimate goal, which is to have every single build in our group managed from one metadata server with build agents scattered to the winds. And that would be sweet.

Oh, and don't think Sun doesn't know about this. Turns out in Solaris 10 they added a system call specifically to solve this problem. It's called posix_spawn, and it does precisely what you'd want. And Sun hasn't changed their JVM on Solaris to use it, probably because Sun, like everybody else in Solaris land, targets Solaris 8 for all those people who refuse to upgrade.

And I think that says a lot about the Solaris community.