TBray Response: Sun Should Stop Sucking

(Talking about Tim Bray's opinion on what Sun should do).

As someone who's used a lot of Sun's products, here's my response as to what you can do, but more importantly, what you probably actually will do.

Price Your Hardware Less
Let's look at the Niagara vs. x86

• T5140 base (dual 4-core T2+ processors, 8GB of RAM, some disks) is $15k.
• X4140 (dual 4-core Opteron processors, 8GB of RAM, lots more disks) is $5.6k.
• X4150 (dual 4-core Xeon processors, 8GB of RAM, same disks as the X4140) is $7.3k.

Lemme get this straight, Tim: you think that the web application deployment crowd are willing to spend about 2-3 times the price for your magical CMT platform? Really? Have you met your typical hosting company? Or have you been spending so long at Sun you don't know what people actually care about?

Here's the thing: you actually make good hardware. The X4140? Great server. Your Niagara processors? Probably pretty good (never had a chance to play with one yet). Constellation? Great IB switch. The multithreaded 10GbE NICs? Pretty good hardware if you have an app that is multi-socket based. But people aren't going to run web applications on something that's more than twice the price; web applications are all about horizontal scaleout. Unless your hardware is 2x the performance for 2x the price, you're going to fail.

Quit Confusing Your Branding
We get it. You invented Java. Good on you. I like Java. That's not an excuse for:

• Changing your stock ticker to JAVA.
• Calling every single thing you can, even when it doesn't involve Java at all (Sun Java System Messaging Server, a product which contains precisely 0% Java)
• Grouping completely unrelated products under completely confusing banners (Sun Java Communications Suite, Sun Java System Messaging Server, Sun Java System Application Platform; these are all on your web site as of right now).

When you come up with your next branding exercise, please stop rebranding every single thing you make into one big bag of branding Fail.

Solaris Will Never Beat Linux
Like it or not, Solaris will never beat Linux at this point. You had lots of opportunities to make this not be the case, and you failed. Mostly because of your own stupid decisions, but the simple fact is that at this point, Solaris will never beat Linux for anything other than specialized systems. These are:

• Applications that require it because they were written once 10 years ago and can never change. Milk these guys for as much as you possibly can; it's the Computer Associates business plan and they seem to do okay out of it.
• Storage appliances (CIFS integration and ZFS are good and better than the equivalents in Linux-land).

No matter what Tim says, Solaris will never defeat Linux in the general web application deployment space, and there is absolutely nothing you can ever do to change this. Give up. Give up now. You're way too far behind, you don't get those developers, and you'll never be able to catch up with the state of the world.

The thing is that one of the points that Tim raises, Solaris having such a stable ABI, actually causes them problems in general worldview and software engineering, because it means that they can never actually change anything to make it better. But more than that, it indicates that their core focus is really about all the legacy applications which are tied to their platform, and not about driving new customers to the platform.

What Sun Could Do
Divide yourself logically into the following divisions:

• Legacy Systems. Sparc IV, Solaris, all the old software packages nobody uses, existing StorageTek hardware. Your job is to keep these customers from spending the effort to migrate to something cheaper; no more, no less.
  
• Modern Hardware. Your x86 hardware, IB hardware, networking chipsets, Niagara. Your job is to do advanced development and be technologically advanced, but at least marginally cost competitive.
  
• Open Storage. OpenSolaris, the new Open Storage hardware. Your job here is to provide a new path off all the storage dead ends that you've gone down, and try to eviscerate the big storage vendors who are insanely overpriced at this point.
  
• Goodwill Software. All the stuff you're never really going to make proper money off of, and probably shouldn't have gotten involved in in the first place. MySQL, Java, Glassfish, NetBeans, StarOffice. Your job here is to try to stem the loss that all of these systems are costing you, and keep from allowing their marketing teams from ruining the rest of your branding on profitable products.
  

Note that there are two growth markets in there (Modern Hardware and Open Storage), and the rest is all irrelevant tangents and legacy. The growth markets are where your future lies, and keeping the others around gives you the chance to migrate existing customers to the new platform, keeping your vision of a one-stop-shop IBM killer intact. But you have to be completely honest with yourselves: the existing stuff is legacy and will never go anywhere, and you need to pile resources into the growth areas without confusing your branding or customers.

What Sun Will Do
Here's my predictions:

• Sun will continue to price all their proprietary hardware so absolutely above the costs of generic hardware that only people under serious lockin to their platform even think about buying it, never allowing them to achieve any types of economy of scale.
• Sun will continue to give software products stupid, confusing names. I predict the Sun Java System Enterprise Database Suite being the new name for MySQL.
• Sun will continue to try to drive Solaris to everything through a neverending sequence of initiatives, confusing anybody even considering deploying it, so that you only ever hit the legacy market and Solaris die-hards.
• Sun will continue to invest in stuff that will never ever drive any meaningful revenue to them, but sap massive amounts of engineering resources. To try to justify this to their shareholders, they will come up with confusing branding and marketing initiatives to try to tie everything together.

In short, Sun, I have no fear that you will find some way to drag failure from the claws of oh-so-close. Just like you have for years.

Linux Fork Performance Redux: Large Pages

After a comment from Kostas on my last Linux Fork Test post, I worked with my esteemed colleagues to try things out with large page support. Wow, what a difference that made on Linux.

The theory here is that forking involves playing around with TLBs entries quite a bit. Since a monstrous full heap will have a lot of 8KB page TLB entries to contend with, if we shrink the number of TLB entries by a factor of 256 (by working with 2048KB pages rather than 8KB ones), you'll limit the amount of time that the kernel is spending mucking with them.

First of all, make sure you read this: The large memory support page from Sun. Now that we've gotten the formals out of the way, here's some fun we had.

First of all, a stock RHEL 5.2 installation has a HugePages_Total set to 0 (cat /proc/meminfo). No huge pages whatsoever. So you need to bump that up. For my test (maximum 2GB fully populated heap on a 4GB physical RAM system), we decided to set that to 3 GB, which is 1536 2MB pages.

echo 1536 > /proc/sys/vm/nr_hugepages isn't guaranteed to actually do that, and the first time we ran it, we ended up with a whopping 2 HugePages_Total. Second time bumped us up to 4. So we went on a process hunt to eliminate any processes that were stopping us working, and got things down pretty small. Now we were able to get up to 870, which was good enough for my 1GB tests (which indicated the major performance degradation anyway), though not for the 2GB test. (Yes, I know that you're supposed to do this on startup, but I didn't have that option so we did what we could).

And so I kicked things off with the -XX:+UseLargePages flag. Fail.

Every single time I got a Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (ero = 12). And nothing would run. Well, damn!

Turns out those little tiny bits that they say in the support page about not working for non-privileged users are completely accurate. These all went away when I had someone with sudo rights run the process as root, and all my numbers are from running as root. So just assume that even 1.6.0_10 ain't going to allow you to allocate any large pages if you're not root.

So he ran things as root (and I re-ran things as non-root without the UseLargePages flag since I changed the test slightly). Here's some fun comparison:

Heap Size	Large Pages Time	Normal Pages Time	Speedup
128MB	3.589sec	13.217sec	3.68 times faster
256MB	3.62sec	15.314sec	4.23 times faster
512MB	4.638sec	36.692sec	7.91 times faster
1024MB	3.885sec	67.062sec	17.26 times faster

Oh, and that speedup between 512MB and 1024MB? Completely reproducible. Not sure what precisely was going on there, I'm going to assume my test case is flawed somehow.

It's happening so quickly at this point that I'm quite suspicious that all I'm measuring is the /bin/false process startup and teardown performance, as well as the concurrency inside Java. I don't actually think I'm testing anything of any meaningful precision anymore. Maybe at a few million forks or with higher concurrency, but I've achieved essentially a constant amount of time spent forking, so I've gotten out of the heap issue really.

So it turns out that you really can make Java fork like crazy on Linux, as long as you're willing to run as root. And I don't know why and my naive googling didn't help. If someone can let me know, I'd really greatly appreciate it.

Did any of this help Solaris x86? Not one whit. Adding the -XX:UseLargePages flag (even though Solaris 10 doesn't require any type of configuration to make it work) didn't improve performance at all, and Solaris was still twice as slow as Linux without the flag.

Linux: A (Less) Terrible Choice For Java Continuous Integration

Where we last left our intrepid developer, he was floating in a sea of Bamboo+Perforce misery, blithely assuming that moving from Solaris 10 to Linux would solve all of his problems. Oh, what a blissful world he would live in! What joy he would have no longer having to deal with Solaris swap space reservation woes!

Since then, though, he's gotten access to two (almost) the same machines, one running Solaris 10 x86, and one running RHEL 5.2. And while he's vindicated, he's nowhere near vindicated enough for his liking.

Executive Summary: Runtime.exec() performance under Linux is superior to that of Solaris 10 x86, but nowhere near as superior as it should be.

Inspect the following micro-benchmark (and apologies that I don't have the nifty code viewing tools that other bloggers do):

import java.util.concurrent.*;

public class ForkTest
{
    public static void main(String[] args) throws InterruptedException {
        int nThreads = Integer.parseInt(args[0]);
        int nSlabs = Integer.parseInt(args[1]);
        byte[][] bytes = new byte[nSlabs][];
        for(int i = 0; i < nSlabs; i++) {
            bytes[i] = new byte[80 * 1024 * 1024];
        }
        ExecutorService executor = Executors.newFixedThreadPool(Integer.parseInt(args[0]));
        long start = System.currentTimeMillis();
        for(int i = 0; i < 1000; i++) {
            executor.execute(new Runnable() {
                public void run() {
                    try { Runtime.getRuntime().exec("/bin/false").waitFor();}
                    catch (Throwable t) { t.printStackTrace(System.err); }
                }
            });
        }
        executor.shutdown();
        executor.awaitTermination(10L, TimeUnit.MINUTES);
        long end = System.currentTimeMillis();
        double secs = ((double)(end - start)) / 1000.0;
        System.out.println("" + nThreads + " - Forking 1000 times took " + secs + " secs"); 
    }
}

Essentially what the test is doing is:

• Creating a fixed size (-Xms and -Xmx set to the same value) heap
• Allocating some slabs of memory (to fill up the heap) (where I refer to "empty heap" tests, this was set to 0)
• Creating an ExecutorService with a certain number of threads
• Running through 1000 tasks, where each task involved running /bin/false in a sub-process and waiting for it to terminate

I felt that this was probably the best way that I could possibly test whether the behavior that I felt was causing Bamboo to perform badly with Perforce repositories would also affect Linux. Turns out I'm half right; Linux will still suck, but suck 50% less.

General Parameters
Both machines were Sun X4100 (non-M2) servers with two dual-core Opterons (one a pair of 275s and one a pair of 285s), and 4GB physical RAM. All tests were done on 1.6.0_10.

Empty Heap Comparison
Here's the first test: Run through everything with no slabs allocated (empty heap) and see how fast we can go. Results in this graph, but the highlight here is that Solaris was very little affected by the size of the heap, but was always slower than Linux, by roughly a factor of 2.


Full Heap Comparison
Next test was to fill up the heaps and then try. Here you can see that the heap size completely determines performance, but Linux is always better (factor of 2 again).


Full/Empty Comparison
Here are just the Linux values plotted out, and it's pretty clear what's going on.


Interesting Observations
Note that the sweet spot here is two threads. No more, no fewer. On Linux, Solaris, empty, full, doesn't matter. You want to fork as fast as you can? Have two threads doing it. Admittedly, these tests were on 2-socket (4 core total) machines, but when I repeated this on Solaris on one of our 8-socket x4600 machines (16 core total), I ended up with the exact same thing: 2 threads was always ideal.

Uninteresting Observations
"Hey, Kirk, you just proved that forking an empty virtual space is faster than a full virtual space! Big whoop! You're such a Java-specific Moran that you forgot all that from your 31337 days!"

Well, no, not really. What I specifically established is that:

• On neither Linux nor on Solaris are the Sun-provided JVMs using any of the fast-subprocess-spawn operations available to them.
• This is a really clear win for anyone working on CI systems to nag Sun or the OpenJDK crowd to get changed and fully tested.
• Linux is still a factor of 2 ahead of Solaris here. I would have hoped a fast-spawn implementation would be a factor of 10, but I'll take a factor of 2 gladly.
• There is definitely something happening in a fork+exec pattern on Linux which is VM-specific, which means that our suppositions earlier that Linux is going to do optimistic copying aren't panning through to eliminate the costs of a fork with a large amount of allocated memory.

Recommendation
So let's say you have a large, long-running Java server which benefits from having a relatively large heap (like, oh, I dunno, a Continuous Integration server), and you have to shell out constantly because a vendor refuses to support you well (speaking of which, I've actually formally asked Perforce to document the protocol).

If you really want to avoid the whole C++ thing, essentially you should be:

• Forking to a second JVM instance to run a small Java application.
• That small Java application should itself shell out to your command-line application (remember: on Linux with a 128m empty heap you can get up to 175 Runtime.exec()/sec, which is not too shabby)
• Have that small java application just pass stdin/stdout to the parent application.

Yes, this seems completely retarded. I can't believe I'm recommending it. But it would actually work as a consistent approach to the Perforce+Continuous Integration problem.

Solaris 10: A Terrible Choice For Java Continuous Integration

As some of you might know, I'm not permitted to run Linux at work. Rather, I have my choice of Windows or Solaris 10 x86. (And no, I'm not allowed to run OpenSolaris, it's boring old Sol 10). I do a lot of Java. I love Continuous Integration. I started pushing Bamboo here in house. And lo and behold, I converted the masses, and we ended up merging projects from CruiseControl.NET, CruiseControl, and Hudson all to one lovely Bamboo instance. We ended up with 110 build plans in Bamboo, all but about 10 of which (dependant and overnight plans) were hitting Perforce all the time to determine whether they should build.

And then things went truly, truly, horribly wrong, to the point of my almost rescinding my Atlassian Shill status. But it turns out that it's only partially their fault. It's actually Sun's fault. Follow me here on a path down process forking details of joy.

What a Java-based Continuous Integration server does when dealing with Perforce, because Chris won't allow them to open up the protocol or provide a usable Java interface, is:

1. Check to see if there have been any changes in Perforce
2. If there have, sync code and run a build
3. Go to 1

For step #1, you've got two choices. You can either say "give me all changes, and I'll figure out whether the change applies to this build plan" or you can say "give me any changes that apply to each of my build plans." The former involves much fewer Perforce interactions, but the latter is directly supported by Perforce itself, which makes it easy to implement, and thus I would argue more correct, since Perforce has a lot of logic about exactly this type of operation (I want a CI system, not another SCM system).

Here's where the whole thing turns to Fail.

When we started adding more and more projects to our Bamboo installation, it started running more and more slowly, to a point where on a 4-core dual-dual-Opteron box it was averaging a load average of 7, with up to 80% of time spent in the kernel. That type of load was so extreme that all sorts of things started going wrong completely mysteriously.

Because of the whole Java+P4 issue, you have to shell out to run the p4 binary to interact with the Perforce server. Under Unix implementations of the JVM, that involves essentially a fork + exec (under Windows it does not, so this is arguably a superior behavior). Here's where things get sticky.

Working with Atlassian, we figured out that in part, this was because Bamboo was being over aggressive in hitting Perforce, and under aggressive in caching things that seldom/never change. Moreover, it was the cost of actually performing the fork far more than the cost of the subprocess that was killing you. 2.1.2 (to be released on Tuesday) resolves this, and so checking whether there are changes goes from 3 p4 invocations, to 1 p4 invocation. A factor of 3, which in their artificial test suite results in a 76% performance improvement.

Now the one thing that ties everything together here is that the parent process here is a JVM. But not just any JVM, a JVM tuned to running big web applications with a lot of users (the same Tomcat instance hosted our JIRA instance as well). And what's the generic rule of thumb for running a JVM for a Servlet container? More heap. More heap, more heap, more heap.

More heap, more betta, right?

Wrong.

Turns out that JVMs on Solaris 10 blow (funny that, given that they're both from the same company; you might think Sun would have an interest in making sure Java ran best on Solaris, but whatever).

In order to do a traditional fork, although there is a copy-on-write optimization to avoid actually duplication of system memory that's going to be abandoned, in order to comply with the Posix standard for fork, Solaris has to reserve enough swap space for the forked subprocess. Now in top, you won't see it, because that swap space isn't actually in use, just reserved, so you can't have it, but it's not being used. Great.

This is terrible, and is precisely why things like clone() were invented in Linux and used to great success. My best recollection (can someone clue me in here, Lazyweb?) is that this is what Runtime.exec() and ProcessBuilder.start() do on Linux-based JVMs. Solaris traditionally didn't have such a beast, so you're stuck with old Posix fork() behavior. Which sucks for this.

So what Bamboo is doing is running 3 2GB heap forks for every Perforce interaction, which is really really really bad.

When one of my colleagues did a fork test here to see how fast he could do a fork+exec in C with a 2GB memory allocation (and forking to a really small sub-process) on the same hardware and OS configuration, it turns out that doing that you're limited to 3/second, and it consumes about 50% of the machine's total CPU utilization in the kernel. Given that Bamboo was attempting to do the same thing in multiple threads, the fact that it was hitting 80% kernel utilization makes complete sense.

Hence, when I reduced the heap to a mere 512MB, Bamboo actually ran faster. Much faster. Back to being usable factor, even with my 110 plans. Once 2.1.2 comes out we should be really rolling, and we might be able to achieve our ultimate goal, which is to have every single build in our group managed from one metadata server with build agents scattered to the winds. And that would be sweet.

Oh, and don't think Sun doesn't know about this. Turns out in Solaris 10 they added a system call specifically to solve this problem. It's called posix_spawn, and it does precisely what you'd want. And Sun hasn't changed their JVM on Solaris to use it, probably because Sun, like everybody else in Solaris land, targets Solaris 8 for all those people who refuse to upgrade.

And I think that says a lot about the Solaris community.