Scala Considered Harmful For Large Projects?

Original Tweet

Last week, I was having an extended email conversation with Geir Magnusson Jr. (yep, that Geir, of Gilt Groupe/MongoDB/Apache Software Foundation fame), where we were talking about whether it was a good, bad, or indifferent idea to allow people to start to include Scala into large-scale programming projects that were otherwise based in Java. Geir and I ended up both agreeing that this was actually a bad thing, and should be avoided for the time being.

I tweeted the above missive, and all of a sudden I had more than 10 people contact me saying "please tell me more!" Let no one think I ignore my faithful readers.

What do I mean by a large-scale programming project? I mean one with the following characteristics:

• It has a sufficient size that not every developer is potentially even familiar with every module.
• It has distribution such that in a production use case, it is not a single monolithic process/binary.
• It has enough people working on it that there is some type of specialization in terms of the teams and/or programmers.
• It has flux in the developers (new people joining, people being replaced).
• The same code base has been in constant development for a period of years, or is anticipated to be.

I've written about this in the past, where I referred to my desire for a Journeyman Programming Language. For what it's worth, I'm 100% behind the Backwards Incompatible Java movement spearheaded by one of OpenGamma's developers, Stephen Colebourne. And in that frame, what we're talking about in a large-scale programming project is precisely the context in which a journeyman programming language becomes useful.

So why did Geir and I concur that someone who has a large-scale Java programming project shouldn't start adding Scala into the mix?

Two main reasons:

• The inherent problems in a split-language project;
• The sheer flexbility (by design) of the language;

Do You Really Want Two Programming Languages?

Let's say you've got a single programming language in your large-scale project. In that case, you hire people who know that language; all your tooling is specific to that language; you try to make the best use possible of that language.

Any sufficiently complex large-scale programming project will likely have multiple programming languages inherently (usually because you need to use a particular programming language for a particular function or to integrate with someone else's code). But should you try to intermingle them by design?

Once you've done that, you've started to add unnecessary complexity to the project:

• Do you have to start hiring dual-language programmers, or spend time training them in the language they don't already know?
• How well does the tool chain integrate? Are there key gaps or differences in operation?
• Are you going to inadvertently split the team into people who are comfortable with New Language versus people who aren't? Is that going to reduce your staffing flexibility to assign tasks to developers?
• Is there going to be an impedance mismatch in terms of people or technology in crossing the language boundary? Is that going to cost you?

These are questions that have been faced by programming teams for years, and usually people come down on the side of not allowing programming language proliferation, unless it's part of an effort to upgrade/migrate the whole code base over time from an obsolete language/model to a more modern one. Would you really get that much benefit from Scala to include it at this stage? I doubt it.

Modern Java has a lot of the stuff that's in Scala. It's not as well integrated, much of it is bolted on as an afterthought, and quite a bit of it is based on convention, but you can get a lot of good stuff out of Java written in a very modern style. On balance, I don't think you get that much out of Scala to warrant its introduction into an existing large programming project.

Language Flexibility Bad For Large-Scale Projects

Let's assume that you're working in a programming language like Scala or C++ that lends itself by design to what I would consider to be abuse of the programming facilities (whether it's the ability to write BASIC code inline in the language, the ability to introduce APL-like operator character insanity, or the gratuitous abuse of the compiler that is turing-complete lambda calculus in the templating system). These are, I 100% concur, cool features. It's very very cool to see someone doing something like the BASIC DSL.

The problem is that if you have a language designed to make it easy to do this, you encourage people to do it. And that's when problems start.

Large-scale programming projects need to avoid this type of thing at all costs. In my opinion, there are several key requirements for the codebase for a large-scale programming project:

• The code must be intuitive and fast to comprehend for a developer familiar with the rest of the project, but not the code in any particular module. In other words, anyone on the team can edit/fix/enhance any other part of code (where they understand the business concepts and/or underlying math).
• The code must be intuitive and fast to comprehend at any point in the future. In other words, you can't have serious issues looking at old code (that's still in the code base) or old edits (where you're trying to figure out why a change was made by looking at history).

Does an inherently flexible language with kewl DSLs and custom characters and template abuse satisfy these? No. It doesn't satisfy the temporal understanding clause (because a DSL that doesn't get consistently used over time loses understanding), nor does it satisfy the pan-developer clause (because not everybody on the team, even if they know Scala as well as Java, may know the DSL or be able to effectively type in high-order Unicode glyphs).

Ultimately, a programming language that's useful for large-scale programming projects needs to have clear, unambiguous grammar and syntax so that any developer familiar with the project and the language can instantly figure out what's going on. Any features (operator overloading, terse/different method invocation syntax, DSLs) that add time in trying to figure out what a block of code does slow down the project. Sadly, Scala is chock-full of them, and it appears to be considered good Scala style to use as many of them as possible.

Can Scala Work?

Although you may think I'm all about hatin' on Scala, the answer is actually Yes.

Neither Geir nor I consider Scala inherently harmful to a large-scale programming project. It can be a very good addition, if your tool chain supports it and you're willing to train staff so that everybody can handle it.

But you have to have even more rigorous coding standards and review process to make sure that the features of the language that excite the "insanely cool programming techniques" crowd don't get used. You have to do what every C++ programming group has done for years: pick the features of the language and library ecosystem you're going to use, use them in a consistent way, and never ever deviate.

Note that I actually think that if you're kicking off a new project, you owe it to yourself to take a serious look at Scala as a programming language base for that project. But if you already have a large-scale Java-based programming project, I personally think that I would avoid it as much as possible.

By the way, lest you think that a new component or module comprise a new project in my thoughts, you're sadly wrong. A "new project" is one where, in my opinion, the entire technology team is separate from the previous one. In other words, a new codebase altogether. Think: new startup; new Line-of-Business funded application; heads of technology reporting into different parts of an organization.

Can too good development infrastructure spoil you?

At qCon San Francisco, Asish Kumar, head of development infrastructure at Google, gave a talk titled "Development at the Speed and Scale of Google". While I wasn't at qCon San Francisco, InfoQ has posted the video and slides, which I've just watched. First of all, stop what you're doing and watch the presentation if you're interested in development/build-and-test/SCM infrastructure. Stop now. This post will wait for you.

I originally wanted to write an article about how the talk shows just how much great developer infrastructure technologies can improve performance, and why every firm and team should be devoting significant resources to improving the infrastructure provided to developers. Heck, that's why OpenGamma's systems administrator spends most of his time working on our development infrastructure.

However, then as I kept watching the video, the more I saw a potential downside to this, particularly at Google scale: virtually everything you're using is bespoke. That has a number of potential issues:

• Developers who may be quite familiar with "industry standard" development technologies will take time to come up to speed. While if you're focusing primarily on relatively long duration hires (on the order of years) this doesn't really matter that much, but many industries (for example, Finance, with its strong reliance on short-term contractors) have to factor staff churn in.
• Unlike with most other development tools (which are dominated by Open Source technologies in the particular types of development Google does), once you start aggressively building a bespoke infrastructure you have to keep going down the bespoke route: you've effectively locked yourself into a path where you have to throw more and more resources at building your bespoke infrastructure to keep up with the state of the art.

In general, for Google, I think they broadly made the right choice to go bespoke in their tool chain. They want to leverage their own (highly proprietary) production infrastructure where possible, in probably most cases they're operating at a scale that no other tool chain would be capable of handling, and (Facebook poaching notwithstanding) they focus primarily on longer-term hiring.

But what about the developers themselves? My only worry about this type of thing from a longer term career development perspective is that, quite frankly, you get lazy in managing your own development infrastructure.

Let's say that you join Google, and spend a few years working very productively using their fantastic development infrastructure. And then you want to join a much smaller firm (which may or may not be your own startup). What then?

• Could you even setup a makefile/ant script/maven setup yourself anymore? How long would it take you to relearn it?
• Are you really going to have a good knowledge of what the current state of the art is in generic development infrastructure and build tools? After all, at New Gig you won't have the Google team backing you up.
• Are you going to be perpetually frustrated in a role where you don't have a team working on your infrastructure for you?
• Are you going to start designing systems where the build and test burden would be more than manageable with the infrastructure at Google, but where the lack of that infrastructure means that you might have been better off designing the system in a radically different way?

Note that I'm not actually saying that this is a bad thing, but I think if you find yourself as a developer in an environment where you have a large, well-skilled team taking care of stuff for you (whether that team is systems administration, operations, development infrastructure, whatever), you owe it to yourself to keep up to date with the state of the art in all the things in between writing your code, and your production environment, that other people currently do for you.

Otherwise, when you finally do get to a role where you no longer have those teams to work for you (or you have to figure out if they actually know what they're doing), you're not hamstrung by your background.

Slow Clap for Item Moves in Basecamp

I follow SvN. I'm not entirely sure why at this point, as I very rarely actually see anything of merit on it anymore, but I've not gotten disappointed enough to drop it from my Google Reader.

Yesterday the Basecamp team decided to pat themselves on the back for allowing movement of items in between projects.

I look at the list of super-duper extra hard stuff that's going on, and I see one of two cases that could possibly be true:

• They've sharded their databases to the point that even trivial operations require superhuman effort. Well-done in that case.
• They've designed their database schema in the most completely insane way ever.

My suspicions? A little of column A, a little of column B.

Love this gem:

Is it a to-do list? It might contain to-do items that have associated time tracking entries. Move those time entries to the destination project too.

That seriously doesn't flow for free due to the relational semantics? You don't just have a FK relationship that means you're only updating the list?

Here's the lessons I think everybody should learn from this:

• Sharding MySQL or another low-end relational database isn't a panacea, no matter how much it contributes to the web-scale secret sauce.
• If you are sharding/partitioning data, you should probably think long and hard in advance about how that's going to impact movements across shards. Because they ALWAYS end up happening.

Code in an expert programming language

Provided for your pleasure, anonymously, from an Expert Programming Language. Scala fans, you're about 2 steps removed from this.

redacted: {[str]
   map: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
   pad: #str _ss "="
   var1: 2 _vs/: map?/:str@&64>map?/:str
   var1[-1+#var1]: 6#(*|var1),6#0
   var1: ,/(-6#/:(6#0),/:var1),(pad*6)#0
   : _ci 2 _sv/: -1 8#((#var1) - pad * 8)#var1
}

Bonus prizes if:

• You can recognize the hideous abomination of a language
• You can figure out the common algorithm it's doing. Hint: What should redacted be named?

I Want A New Programming Language

Preface: This has nothing to do with the Oracle/Google Java lawsuit. Read this first.

Dear Lazyweb and Programming Language Inventors:

I want a new programming language. Although I seldom code these days for OpenGamma, I've wanted a new programming language for quite some time. I don't want an extreme language (in syntax or constraints); I don't want a purely experimental language; I don't want a faddish language. What I want is what Stephen Colebourne coins a "journeyman language."

What Is A Journeyman Language

Quite simply, a journeyman language is a programming language designed for journeyman programmers. And those guys are the hundreds of thousands of men and women working on business applications and systems programming every day.

Although there is wide variation in the quality of journeyman programmers, in general, very few of them are in the infamous Top 10% Of All Programmers level. But that's actually okay, and self selecting: the rockstar programmers simply wouldn't do the jobs that the journeyman programmer has to do every day; the job is too boring and lacks enough challenge and intellectual satisfaction/achievement. I have familiarity with firms in the financial services space who explicitly don't even interview high achievers because the employer knows the employee would hate the job and leave.

So what does a Journeyman Programming Language need? It needs to have a few general characteristics:

• It has to be simple enough in syntax and conceptual framework that people not in the top 10% of the profession can feel comfortable working with it.
• It has to be flexible enough to cover the amount of back-end, rich GUI, web app, and systems programming that goes on in the industry.
• It has to make simple, common programming errors (memory allocation, array bounds, etc.) difficult or impossible.
• It has to make it easy for Journeyman Programmers to change projects or jobs on a regular basis.

But it's a mistake to think that a completely dumbed down language can appeal and make a lasting impact in this space. Journeyman Programmers aren't idiots or morons: they often are just as good as rockstars, just not as passionate. That means that they're not investigating programming language features and Open Source libraries in their spare time, they're not going to meetups and blogging and tweeting and everything else. Raw talent often, passion seldom.

The other reason it's a mistake to dumb down the Journeyman Programming Language is that in any sufficiently large firm or project, there is often at least one rockstar programmer. And he needs to be comfortable with the tools that he uses to set the architecture and framework for other developers. Give him BASIC and he'll recoil in horror.

So here's my test: if you could write a reasonably high performance RDBMS system in the language, it has enough features. If you couldn't, it's not good enough. I like this particular test because I've done it several times, and also because there are a bunch of fiddly things having to do with getting primitives in and out of large byte blocks in space that languages like Java are particularly terrible at for no good reason that I can see (which is why LucidDB does query compilation in Java and execution in C++). RDBMS' involve everything you need: functions in the scalar expression system; objects in the query validator, compiler, and optimizer; efficient memory work in the executor. You do all of those in a simple language and system, and you've got my vote.

Positive Features

These are all characteristics that New Programming Language should have.

C-Style Syntax

Like many developers, I was born and raised on C lineage programming languages: C, C++, Java, C#. I've dabbled in many other programming languages (Pascal, Perl, Python, Scheme, Assembler), but nothing to me has the simplicity of syntactical expression that the C lineage of languages has. Let's keep that.

Garbage Collection

Getting rid of forcing memory allocation duties on the developer has been probably the single greatest boon to getting more people writing better quality code than anything else that has come out of programming language and environment development in the last 40 years.

Unambiguous Syntax

This is both a feature and an anti-feature. I want the language to be unambiguous, so that I can look at a chunk of code and, with experience with the language, know what it does. That means that any type of DSL services need to be confined to creating DSLs for other files, not intermingling DSL syntax with New Programming Language syntax.

Objects and Functions

I want an object oriented language, but one which recognizes that I will often have things that are better expressed with functional blocks. The Java approach of hanging static methods on a final class with a private constructor and import static is ridiculous and everybody knows it.

Closures

I want closures with my functions. With a nice syntax. That doesn't look like line noise. I won't want to use them for everything, but I want them to be present.

Useful Concurrency System

I want a number of low-level concurrency systems, as well as convenience operations for actor-style concurrency.

Object Immutability Services

I want some facility to mark that an object instance, or a particular call on it, will not and can not mutate the state. In other words, I want a const style system that actually works. Doesn't have to replicate the exact nature of the const keyword in C++, but just give me something that will lock down an object instance and thus allow it to be shared in a thread-safe environment without limiting me to constructor-only value injection.

Properties

I want C#-style properties. And I want a serialization system that allows me to cleanly map transport-representation objects to hierarchical data representations like Avro and FudgeMsg and JSON and XML (and will bork if I create a DAG or cyclical graph rather than a tree). Not all objects of course, but just ones that I'm going to use for transport. And I want the ability to do fast metadata-style operations on properties (think: for (Property prop : anyObject) { doStuff(prop.getName(), prop.getValue()); }). The way JSON flows for free out of JavaScript is an excellent example. Encapsulation? Pshaw. Journeyman apps are about data as much as objects, and everybody bloody well knows it at this point.

Real Generics

I want Real Generics. With runtime type information. That allow partial specialization.

VM Operation

It better run cleanly on any major OS and hardware combination. And having a VM gives me a lot of runtime management facilities for back-end processes built in (think jconsole or jvisualvm for the JVM). In particular, I'd probably like the JVM if you can shoehorn much of this into the JVM. If not, please look into making the bytecodes register-based like LLVM (even if the binary bytecode format ends up having both stack and register implementations potentially).

Stack Allocation/Memory Packing

If I really know the lifecycle of an object, and I want to bunch several similar objects together to exploit cache line efficiency, don't get in my way. Don't make it the default, but support it.

Fast Compilation and Runtime Linking

In my mind, these are one and the same thing, because each requires and mutually enables the other. The programming language needs to be able to support compile-on-save functionality (a la Eclipse) and complete runtime linkage (i.e. classloading/modules/whatever). These are massive productivity wins for a journeyman programming language.

Convenient Native Code Integration

While the core of the language should be based on bytecodes, sometimes for performance I really want/need to be able to go down to low-level coding in C. Please make that easy (JNI, for example, is horrific). Even better, if I could mark something as "this is in New Programming Language, but if there are native versions of this method/function about, use those instead" that would solve much of the JNI-style problems.

Modules

I like OSGi. I would actually use something like OSGi if it wasn't, well, OSGi. It would have to be baked into the language and compiler stack to be useful.

Partial Classes

I personally love this for combining code generation with hand-written code; C# nailed it again. And Journeyman projects tend to have a lot of places that code generation can/does assist with, particularly given the amount of XML and RDBMS and rich GUI work that goes on in large organizations.

Traits/Shared Code Blocks

A great feature to have a block of code that I can mix in without all the standard multiple inheritance issues coming in. At compile time, of course.

Random Stuff

While I'm at it, a bunch of random stuff I'd like.

• Duck Typing
• switch on anything. Strings, integers, enums, whatever.
• Fixed precision decimals as a first-class type.
• Static typing, with var lvalue determination.
• Matching between module/type/class/whatever name and file name, to aid in automatic refactorings.
• A good standard library, like the Python one, that gives me both an least common denominator and greatest specific feature ability to access the OS' services.

Negative/Indifferent Features

These are all features that I don't care about, or actively want not in the language.

Reuse of existing libraries

If you're targeting an existing VM like the JVM, I really don't care if I can use all the same libraries. Would be nice, but not really necessary. The only ones I'd care about are all integration systems and libraries like JDBC drivers to be honest.

Operator Overloading/Invention

Remember what I said about understandable code? Don't make it easy/possible to create a horrific mess of a language that looks like APL. The only time you ever want operator overloading for the right reasons is for the [] characters. Gosling was right on this one.

Monads

Gack. Yes, I'm very impressed that you found a way to have side effects in your pure functional programming language. No, it's not actually that useful in practice. Imperative programming languages can easily incorporate functional language features, and should do, without incorporating all the baggage that made your Functional Languages Professor at school giggle with delight.

Obsessive Terseness

Verbosity, when applied correctly, makes unfamiliar code easy to understand by someone who wasn't the author. An obsession with achieving the tersest possible language, or smallest possible number of syntactic features, makes code harder to understand. Remember, I'm talking about journeymen programmers here.

Pointers

No, the language can't expose a bloody pointer. There are only a few valid uses for them these days, and I'd hope that they're handled through other facilities. If you really really REALLY need one, just drop down to C and thrash with the system to your heart's content.

Abuse of your compiler

No lambda calculus in the generics system, please. No template metaprogramming, no hiding everything behind private typedefs, let's keep it simple.

Faddish Language Features

XML in my programming language? Yeah, in 20 years that'll totally seem relevant. Keep it in the libraries, thank you very much. Same thing with HTML, CSS, whatever.

Checked Exceptions

File this one under Seemed Like A Good Idea At The Time.

Throw Your Mom

Very clever that C++ allows you to throw anything. Null? A string? Your mom? No, thank you. I'll limit it to proper exceptions, thank you very much.

Why I Don't Have It

So why don't I have this language yet? Well, partially because programming language craftsmanship is hard. I'm pretty sure I'm not good enough to do it, which is usually my default criteria for saying something is Really Hard.

But I think as well the k3wl languages coming out are coming out of language requirements of the Top 10% crowd. They're the ones good enough to actually write the languages, and they're going to write a language that makes them happy. But then you end up with Scala, and then you end up with this monstrosity, and then you make me cry. A language in which that thing is even possible will never be a candidate as a Journeyman Programming Language.

You know who's going to do it? Someone like Gosling, who set about with the needs of the journeyman programmer in Java. But the state of the art has moved on, and Java just isn't suitable anymore.

Who I would really like to do it is Anders Hejlsberg. I am a very big fan of C#-the-Language. It's just that .Net-the-Ecosystem is so Microsoft-specific and horrific it'll never catch on in the wider world, no matter what Miguel de Icaza thinks.

So how's about this:

1. IBM, please hire Anders Hejlsberg away from Microsoft. You know the Oracle/Google suit is scaring the crap out of you right now given how much you've invested in Java. It's not the suit itself, but the sign that Oracle, a major competitor to you, is going to leverage whatever muscle it can around Java against you eventually.
2. IBM, please let Anders build this, which I'll call C-Prime, with smart people from the Java and LLVM communities, who all have a lot to add here.
3. Open Source friendly licensing abounds, and the runtime works on a whole lot of interesting platforms. And if you want to pull a Larry and not support the Solaris port yourself, we'd all totally understand. You support Linux, Windows, and Mac, and 99% of developers are happy.
4. Developers rejoice as they have the New Programming Language.
5. Kirk is happy.

Oh, yeah, and Microsoft? If you could have broken the near pathological obsession with platform lock-in that surrounds all your interesting technology you had a really good shot with the CLR. C# could have been a contender. But at this point, your organization is so broken internally, and your reputation with the types of journeymen who work at large organizations is so tainted, that nothing you produce will get traction. Which is why I want Anders to leave you.

A little revolution every now and then isn't a bad thing. And at this point, I think it's time. Java-the-language will never advance in a standard way going forward; the collapse of the JSR has seen to that. We as a community who has worked on Java needs to move forward and onto the next language designed for the types of people who currently code in Java.

Big-Agile Is The New RUP

Some of you who follow Joel, even if only to comment that he's largely irrelevant these days, might have found this coming a bit out of the blue. Why would he make a blog post just to have a partial transcript of one of the StackOverflow podcasts?

And then I found this beauty. Wow.

Turns out the transcript on Joel was specifically because he believed that the Big Agile guys were (intentionally or deliberately) misquoting him. Some really exceptional bits from the article and comments:

Joel said that the SOLID principle aren’t “agile”. (sigh). Everybody and his uncle thinks he knows what the term “agile” means. But I’m the guy who called the meeting where the name “agile” was picked. I’ve been writing about Agile development since the term Agile development was created. I think I know what is Agile and what isn’t. And I think I have the authority to override Joel on this one. Joel, the SOLID principles are agile.

Wow. Appeal to Authority much? Oh, wait, then someone calls him out on it. Here's Uncle Bob's response:

“But I’m the guy who called the meeting where the name “agile” was picked.” Appeal to Authority is a fallacy.

Not in this case, since I am one of the authors.

Wow. It's as though he actually doesn't understand what "Appeal to Authority" actually means. He seems to quite legitimately believe that since he was one of the original people who coined the term "Agile" he can define precisely what that means for the end of time.

Let's look back at my original purity vs. pragmatism post. Now let's look at the comments on Uncle Bob's post. Remind you of anyone?

Agile Is The Anti-RUP
The original motivation of Agile Methodologies was to get away from the RUP and other similar staged-delivery software engineering/delivery methodologies. The RUP is a big, rigid waterfall model that pretty much guarantees that your project is going to be a late failure. Each step is rigorously enforced, and you can't move from one step to the other until you've completed all the requirements. Furthermore, all the outputs are rigorously defined so, for example, you must have UML Class Diagrams (and no other type of diagrams) before you can commence coding.

However, even though anyone these days could look at the RUP and instantly see that it had to be a great big failure, nobody had a Name for what all the effective techniques that weren't in the RUP (or were completely diametrically opposed to the RUP) were. So when people who are extremely respected started saying, "Hey, you can be more successful by ignoring the RUP and doing it this other way" people loved hearing that. People gave these new Agile Methodologies a try, and even bought into the ones that were packaged as a cohesive, brand-name entity (because the only way to compete with a single-named overarching methodology is with another one: RUP vs. BSDM, XP, Scrum).

The best part was that "agile" was just a set of generic techniques that you could pick and choose from and still get some credibility. You could easily define what you do to someone by saying "yeah, we're big on CI and unit testing, do 4 week spikes, and have 2 people sitting on the business desk" and someone would understand what you're doing, and say, "hey, that's pretty agile."

Big-Agile Is The New RUP
Nowadays, that's not true. Let's look at some of the problems I see with the Big-Agile community:

Arguments From Authority. I'm sickened by someone saying "I alone have the right to decide what is Agile and what is not Agile because I am The Authority." No, no you don't.

Argument from Authority only works with factual statements: "I was in the room when Martin Fowler's head exploded, therefore I am an authority on whether Martin Fowler's head exploded in that room." It doesn't work for an argument of opinion.

You definitely get some kudos for being in the Agile world for that long, being a generically smart guy and by really helping the community grow. But that doesn't give you a trump card you can just unzip and slap on the table.

Simple Becoming Rigorous. Let's take a very simple concept: write tests before you write the code. Sounds pretty simple, right? Apparently not. Apparently saying TDD means a whole lot of very particular things that if you don't like TDD, you apparently aren't doing. Go back to the Uncle Bob post. Look at the comments. They're conflating concepts (Uncle Bob conflating TDD with having a rigorous testing policy), and not responding when someone makes the most valid criticism of all:

If TDD and SOLID is too obscure that regular-intelligent people don’t get it, then the problem is with TDD and SOLID, not with the regular-intelligent people.

You see this as well with all the people who essentially say, "If you're not using BDSM/Scrum/XP, you can't be Agile."

You're Doing It Wrong. This is pretty similar to Simple Becoming Rigorous, but the basic concept comes back to all the old, original XP arguments: If XP Doesn't Work, It's Because You Did It Wrong. XP will always work by definition, therefore if it didn't work, you failed to do it correctly. Tautology. But you see this all the time whenever someone says "Yeah, I tried Agile Technique X, and it didn't work very well for me/my team," and the response is "that's because you didn't do it right." It's condescending, and it assumes that all developers are exactly the same, and all projects are exactly the same as the project that you've just been on.

Agile techniques are supposed to be so simple that you can't do them so wrong that they don't benefit you. TDD? As simple as "write tests before code." Standup meetings? As simple as "ensure meetings are short by not allowing people to sit down." How do you get those wrong?

One Right Answer.  You tried a particular little-agile technique that you don't like or that doesn't work for you? Obviously you're wrong. That technique is perfect and must be applied in all cases. It never doesn't apply.

Comment The First: "maintaining a system that wasn’t written with SOLID in mind is the opposite of enjoyable." Really? You can't handle working with any system that wasn't developed with your personal architectural concept in mind? They're all rubbish? Each and every one of them? I don't know what SOLID is except for what Joel said. Honestly, at this point, I don't care. If it will warp me and my systems so much that I will never be able to look without hostility at anybody who doesn't use it, it's a virus, and I choose not to be infected.

Comment The Second: "I have never met anyone that has honestly given it a fair trail (20 days or so) and not become addicted for life." Note the use of the term "honestly" to conjoin this with You're Doing It Wrong.

Here's someone who really seems to get it:

Some zealots are giving ahering strictly to dogmatic methodologies (not necesarily SOLID) a higher priority than delivering software and satisfying customers. If your code is incredibly clean, but doesn’t meet the user’s needs, it is useless. Of course, ideally, principles and methodolgies are used pragmatically in delivering software that does meet customer’s needs but I have seen developers forget the objective of the system they were working while obsessing on the details of the implementation.

At the end of the day, nobody cares how high your code coverage is, or how many interfaces your system has, or what mocking framework you use, or anything else. They only care about one thing: are you consistently delivering software of value?

Agile vs. Big-Agile: Game On
I fully expect the Big-Agile crowd will probably find this completely insulting, and rely on all of the above arguments:

• He hates Scrum because he only has a passing familiarity with it
• He doesn't purely rely on TDD because he honestly hasn't given it a fair trial
• He doesn't like Pair Programming because he never did it correctly
• His experience writing use cases down doesn't count because he didn't use the correct size of note cards and didn't use the correct pen colors

At this point, I don't care. Let me put a stake in the ground.

Big-Agile Zealots Are Killing The Name Of Agile Methodologies

I've interviewed with a lot of firms recently, and I actually have to completely qualify the fact that I believe in little-agile methodologies, because at this point, they see the word "Agile" on a CV, and they think that you're TI from the Purity vs. Pragmatism article. It turns them off, because you guys are viewed as ranting zealots.

And you know what, when I then explain what I find works well in the real world with the types of people I tend to work with, they do the same thing and like it. They like agile methodologies, they don't like Agile Methodologies. They just find the Big-Agile crowd all completely unbearable to listen to at this point.

So with that in mind, unless we can get the "little-a-agile" terminology spread, we're going to have to part company, so that Big-Agile can own the now tainted "Agile" terminology, and those of us who are far more pragmatic can have our own word to play with that you can't try to authority-away our rights to self define.

Ordinary People, Super-Stars, Compensation and XP

(Background: "We Don't Pay You to Work Here" on Venture Hacks).

I'm going to completely disregard every single point in there about incentivising (yes, I just typed that word; you may hereby assume I've lost all cred) employees without money, and urge everybody to leave their cynicism at the door (because ultimately, if you have to be taught to do that stuff, and you have to make a conscious effort, you will fail at it and should resort to just paying people more; a quality working environment is an intrinsic part of culture and can't be bolted on by paying consultants).

I'm with the article, and at least finding it a useful POV, until this:

Practices like Extreme Programming, that were designed for programmers with ordinary skills, work even better with extraordinary programmers.

No. No, no, no, no. No, no, a thousand times no. Stop right there.

Whoever typed that should be aware that their chances of actually employing extraordinary programmers or working with them are approximately 0. Maybe of the group of programmers with which the author has experience, the extraordinary ones of that group work well with XP. Not in the real world.

I do not know a single extraordinary programmer who likes or would even be willing to tolerate XP.

• Pair programming just doesn't work with extraordinary programmers (and extraordinary ones are usually the ones who literally shove the other part of the pair to the side and pretend it's normal programming).
• 40-hour strict workweek just doesn't work, because many extraordinary programmers can't switch off voluntarily, and go through manic high and low phases of productivity.
• The phrase "YAGNI" makes extraordinary programmers violent with rage, because they are always more skilled than the person uttering it, and always know that Yes, You F**king ARE Gonna Need It, Moran! (YYFAGNIM? [Welsh?]) They understand agile, but at the same time they also understand overall lifecycles and how difficult it is to retrofit Lack Of Design Up-Front f**k-ups later on.
• Perhaps most importantly, extraordinary programmers understand anyone selling any type of Silver Bullet is lying or naive.

And remember, XP is based on an irreducable model of software development methodology: it's the Three Stooges Syndrome of Agile Methodologies. Proponents explicitly state that without doing every single part, the whole edifice crumbles. I do not know of a single programmer I would consider extraordinary who doesn't have at least a few problems with XP, and who would voluntarily join an XP project (and everything these guys do is voluntary: fear of unemployment doesn't do anything for them).

Perhaps the author of the article needs to meet some real extraordinary programmers.

Amazon SDB: "Eventually Consistent" Applies to Credentials

I wanted to get my head around using Amazon SimpleDB for a couple of projects, so I grabbed Typica, the Java library for accessing AWS services. I was having some massive problems with authentication validation errors, but because the Typica distribution doesn't include source code, I realized that the source line numbers don't match up with the binary for 1.4.1.

Even updating to a SVN pull of head didn't solve things. So I waited while I ate my dinner. Some requests went just fine, some failed. Since SDB involves a scatter/gather approach, this meant that my system wasn't working properly as a whole and I couldn't get my aggregate results correctly.

After waiting a while, the problem went away on its own. I think it's a result of the Eventually Consistent principle at work. I had enabled my AWS account so recently it hadn't had a chance to propagate my AuthN/AuthZ credentials to all the SDB nodes. When it finally did, everything was fine. And I can now quantify the time it takes: One Pizza. That's how long it takes. Push data, eat pizza straight out of oven, data consistent.

By the way, now is a perfect time to try out SDB. It's essentially free for the time being, so no reason not to try it out. I'll have more postings on it, because it's pretty cool.

Purity vs. Pragmatism

I was being interviewed for a new job last week (yes, I am actively on the market), and had a very interesting, frustrating run-in with one of the interviewers (disclaimer: by mutual acclaim, the role and I decided we weren't right for each other; you'll figure out why shortly) at a large bank.

We Frustrate Each Other
The frustrating part of the interview came when I realized that The Interviewer (we'll abbreviate it to TI, not to be confused with the hip-hop artist T.I.) and I were disagreeing on virtually every point of philosophical substance. But that was just a manifestation of a broader disagreement that I think went to the core of why the two of us must never work together: we fundamentally disagree with the core principle of software engineering.

Let me explain with a few examples of where we differed to try to explain what was going on (no, this is nowhere near an exhaustive list):

• He believed that checked exceptions in Java are always wrong; I believe that sometimes they're useful and sometimes they're not.
• He believed that you should only accept or return interfaces (and never a concrete class) from a module; I believe that you pick and choose between interfaces and POJOs depending on the context.
• He believed that setter-based Dependency Injection is always wrong and that only constructor-based DI should be used; I believe that you pick the right one for the context.
• He believed that you can only ever use DI with a DI container like Spring or PicoContainer; I believe that it's an architectural principle that can applied with or without DI-specific tools.
• He believed that you cannot consider yourself a practitioner of agile methodology without rigidly adopting a Formal Agile Methodology (Scrum, XP, whatever); I believe that the whole point of agile methodologies is that you pick amongst the parts that help your team develop better software faster.

What's the major differentiation here? Purity.

Purity
TI's approach to every major difference between the two of us fell down on the side of rigid, unbending application of a principle in the interests of purity. My approach is far more fluid and contextual.

Purity in any endeavor (art, design, architecture, music, religion, software engineering) is attractive because it strips away all thought and all decisions, and in doing so, pushes a concept to its ultimate expression. I can understand the sentiment. When I moved into my flat, every single surface was either white (floors, walls, ceilings, some doors) or gray metal (stairs, shelves, other doors). It's a minimalist, pure aesthetic, and it removes all distraction and makes it very simple to make decisions: there's nothing subjective about additions.

Sometimes, purity is exactly what you want. It focuses you, and allows you to fully explore one concept to its extreme (how white can you get the walls and floors? can you get them the same white even though you have to use different paints for different surfaces? can you keep the floor white even though people walk on it and put furniture on it?). Even the exploration of pure silence has begat its own groundbreaking work.

Purity in Software Engineering
Taking a purist approach to a software engineering matter allows you to nail your banner on the church for all to see: I believe in X, therefore X is always correct; by applying X to every situation, I prove the superiority of X and validate my initial conclusion that X is superior. This comes up a lot, particularly in architectural discussions:

• Asynchronous Messaging is great! Let's use it for everything!
• An RDBMS is great! Let's use it for everything!
• REST is great! Let's use it for everything!
• Google Protocol Buffers is great! Let's use it for everything!
• Cubes are great! Let's use them for everything!
  
• Lisp is great! Let's use it for everything!
  

The converse also happens:

• XML is crap! Let's banish it from the world!
• RPC is crap! Let's banish it from the world!
• Solaris is crap! Let's banish it from the world!
• RDBMSes are all crap! Let's banish them from the world!
• TCL is crap! Let's banish it from the world!
  
• Lisp is crap! Let's banish it from the world!
  

Purist decisions are easy. They don't require thought. They don't require constant critical evaluation.

And that's why I view them as intellectual cowardice: by limiting your choices intentionally, by limiting the scope of tools at your disposal, by closing your mind off, you reduce the amount of thought you have to put in. But as engineers, as craftsmen, we live by our minds: we are paid to think, and the more we're paid, in general, the more thinking we're expected to do.

TI could replace himself with someone far less experienced by simply writing his own DSL that works on the JVM that doesn't allow you to do any of the things that he thinks are wrong, and forces you to do all the things that he thinks are right. It wouldn't be that hard. Then he can simply hand that off to someone and say "here you are; a language that obeys every one of my edicts perfectly; you are sure to do an excellent job now that I've handcuffed you to my beliefs."

[Aside: if TI's beliefs realy were so universal as to allow him to view me with revulsion for not sharing them, why isn't there already the TI programming language? Are you to tell me that there isn't a programming language which targets some target environment (JVM, CLR, raw x86 machine code) that forces the practices that he likes and forbids the practices that he doesn't? Really? Does that tell you something perhaps? Because it's not like we have an absence of programming languages these days. And it's not like it's that particularly hard to write a language and target one or more existing VM environments, so all the hard work is taken care of already.]

In Defence Of Pragmatism
I'm impure. I'm about as tainted as you can get. And you know what? I think that makes me more effective, rather than less. Because what I get out of a lack of purity is pragmatism. The two can't coexist particularly effectively: I think you fundamentally agree with one or the other.

Pragmatism allows me to look at a problem, carefully consider the advantages and disadvantages to each potential solution, and then determine the right approach to each particular situation.

Pragmatism allows me the flexibility to do things that I know in other circumstances would be wrong, but in that particular one would be right.

Pragmatism allows me to have a bag of tricks rather than one, and pull them out as I see fit.

Pragmatism allows me to gradually refine my beliefs by using both my favored and unfavored approaches in a variety of circumstances so that I have evidence and backup behind me when I express my opinion.

Pragmatism allows me to work with a variety of code written by a variety of people and engage with it constructively as it is, rather than seeking to rewrite it before I'd be willing to touch it.

Pragmatism allows me to decide where to focus development efforts based on what's most important that day: sometimes rushing a feature into production to make more money, sometimes spending time testing the bejebus out of a small change before I make it.

Pragmatism forces me to build software which is easy to maintain and consistent in its internal architecture and clean in its modularization and consummately tested, because that's the only way to maintain a complex system over time. Pragmatism also tells me that if I know the piece of development is only there to run a single time, focusing on all of that is wasted time better spent elsewhere.

Pragmatism allows me to assess the strengths and weaknesses of my team, and the constraints of my customers, before assessing the correct development practices that will result in that unique group of people producing the best software possible with the fewest resources in the shortest possible time. Pragmatism forces me to understand that no one methodology could possibly work for all engineers, all customers, and all projects.

Pragmatism allows me to focus on one and one thing only: getting the job done well. Do what it takes, but get it done, and make sure it works properly. And that's precisely what we're here to do as software engineers: engineer software. And engineering, unlike art, design, music, or any purely creative endeavor, requires getting something done that works. We're not in this business to craft a purist expression of an idea. We're here to build things that work.

The irony is that many purists get that way out of a misguided belief that they're being pragmatic: buy choosing a single technology or technique or methodology, they've chosen the One True Solution and don't have to consider the same decision with over and over. Furthermore, they've made sure that those people working for/with them (particularly the lesser skilled ones) don't do The Wrong Thing. But that means that they've closed their minds off to the chance that they've chosen incorrectly. Or, even more appropriately, that there is no one right decision.

That's why purity and pragmatism can't coexist: purity requires that you ignore the results in favor of an ideological ideal; pragmatism is all about pursuing results for their own sake.

And that's why I think TI and I couldn't work together: I'm a pragmatist and he's a purist. The two of us were bound to clash, and at least we discovered it quickly.

But in the end, I know my approach is right. I'm proud to be a pragmatist.

Except when it comes to TCL. Man, I wish I could banish it from the world.....

Meaningless, Programming, and Innate Affinity

Clay Shirky guest post on BoingBoing called "Comfort with meaninglessness the key to good programmers". The comments are filled with "my programming language r0x0rZ" and suchlike, but it's another perspective on the often-espounded (not least by yours truly) theory that being an excellent programmer is an innate characteristic. You can be trained to be a software engineer of some skill (and large firms are stocked with hundreds of the type, who are trained to implement business logic in a programming language, or in pixel-pushing GUIs), but there appears to be something innate in making a programmer.

Or, as I was recently told, "so what you're saying is that all of you are naturally freaks."

IEEE 754 Floating Point Binary Representations

Just to gather up a whole bunch of stuff I had to slog through and make this more googleable, allow me to summarize some various trivia having to do with bitwise representation of IEEE 754 floating point values across platforms. This is primarily useful if you need to read and write floating point values from byte arrays or binary network streams across platforms, particularly if you have to interact with Steve Ballmer's Insanity.

First, there is no official standard for endian-ness when transmitting IEEE floating point data over the wire. That means that Java ends up defaulting to in DataInputStream and DataOutputStream to big-endian format (to match the fact that everything is big-endian), C# defaults to host-endian format (always little-endian in practice, as the Mono guys have learned.) for BinaryReader and BinaryWriter. First point of fun.

Secondly, IEEE 754 floating point representation defines an entire range of values to represent NaN, not a single value. Java takes the approach to make things byte compatible in the wire format by always emitting a single constant value for all NaN values (where all the meaningless bits are set to 0), while C# allows whatever cruft happens to be in the value on the CPU to flow through to your binary representation. And don't assume in C# that double.NaN has all those set to 0. It doesn't. In practice, double.NaN in C# is full of cruft.

This is fine if you read in the value and call IsNaN on it, but not so great if you want to check that your serialized/deserialized byte arrays are fine. For that, you need to mask out to ensure that you're always writing a canonical representation of your NaN values.

A useful C# block if you find yourself having to deal with this stuff is the following (using this will ensure that your binary representations are always bit-equivalent with the Java formats):

C# BinaryWriter is Little Endian Because Microsoft Hates The Internet

Let's assume that you're developing the primary runtime class library for a programming language, and you need to write primitive types to a network connection. You've essentially got two options:

• Allow the application to specify the endianness of the data
• Require the application to use a particular endinanness.

The former allows developers more flexibility, but it means they have to think, and thinking is hard.

Let's assume that you've decided to not allow the user to specify the endianness of the data that the user is going to send over the wire easily. What endianness might you then choose for your developers? Might you choose the one that's officially called "Network Byte Order?"

Well, no. You're working on .NET and you work for Microsoft, so you'll use the opposite of that.

Now let's assume that you have a developer who's trying to do the right thing (where the right thing is not to start crying "Waaah, C# sucks, so you have to change the internet to support whatever Ballmer's got cooking"). You might try to make it easy for them to output data in network byte order. How might you do that?

By hiding the .NET equivalent of htonl and ntohl as static methods in System.Net.IPAddress, duh. Because that's obviously where you'd look when thinking of where to find binary data endian conversion routines. What in the world could be more logical or easy to find without StackOverflow?

Note: To be fair, it's entirely possible that Microsoft has not chosen little-endian by conscious decision, but is merely using host byte order. But for compatibility, Mono has to replicate this as little-endian-always. Which is going to be a lot of fun if and when Microsoft actually tries to port this to a new architecture.

Culture of Testing and Continuous Integration

Ran across a pretty good post on how defects can lead to never being able to meet any of your three basic criteria (features, quality, time). You should read it. But the Kirk summary here is that while I'm not surprised, there are still a lot of organizations who haven't gotten Test/Continuous-Integration Infected. I've known for years that it's a key component to long-running development project success (hint: if you're using source-code control, it's long-running), but this is a pretty good summary about why if you always skimp on "Quality", you're actually always going to lose on Features (too busy fighting bugs to add them) and Time (end of the cycle is extended dealing with defects in key functionality).

A few quick and recent anecdotes:

• That Java5 vs Java6 ThreadPoolExecutor bug? Caught by our continuous integration server testing against a plethora of JVM and OS combinations.
• In some C++ code, we caught a race condition by running tests over and over under continuous integration (and because our build agents run on ordinary development machines, this exposes them to a variety of configuration and machine load scenarios), and refining the tests until we found the lowest-level part that was failing.
• We have logic that works with some (intentionally) unreliable middleware that people too often assume is reliable. By having tests that spuriously fail under load (the tests assume more reliability than you can actually achieve on machines that are otherwise loaded), it became a good stick to beat the "best-effort is good enough" crowd internally. Tests periodically failing don't lie: I can say "15% of the time, this test fails, and it's using the API as intended."
  

I've had some discussions with people about why they don't move to a CI and Test-Infected model, and I've heard a few arguments. I think most of these come down to two factors.

First, starting is hard. If you're on an existing product which you don't actually like the internal code path on, it can be really difficult to start testing. And if your code isn't written to make it easy to test in isolation, all your tests are system-integration tests, which are pretty poor at providing low-level test cases to track the code itself. The problem seems so daunting that you don't even really start at all.

The easiest solution here is the obvious one: start with some system-integration tests, and as you start working on existing parts of the system (either fixing them or replacing them) start writing lower and lower level tests. Use a code coverage tool (like the shill I am, I'm extraordinarily partial to Clover for Java development), see how you're doing, and make sure you're constantly expanding the code paths that you have under testing. And use it interactively: turn tests on and off and compare your coverage to find out how good your low-level tests and system-wide tests really are at testing your system.

Over time, as your test base grows, you end up exactly where you should be: all non-trivial changes to the code result in changes to the test suite to establish that the code is behaving the way you expect, and you're running them constantly to give you positive feedback that your code is working properly.

But this isn't the rant-filled part of the blog. The following is.

The Test-Infected crowd turns good software engineers against testing. Seriously. If you do anything to an extreme, you turn off software engineers who would otherwise listen to you. So just shut the hell up.

[In particular, if you work for a consulting company and are trying to sell development services, please don't bother trying to tell me how to develop software for long-running maintenance. How in the world would you know? By your very nature, you don't do long-running system development or maintenance. Why would I trust you in any way except as a clearing house for ideas you've seen other people doing? Other people who are probably failing and thus hiring you in the first place? Great. You've seen lots of fail.]

First of all, Test-Driven Development is retarded. There's a reason why every example I've ever seen of it is of extremely trivial code: because it's impossible to do well otherwise. When I start working on something that's non-trivial, do I know enough about the internal implementation details or the external contract that it will present to the rest of my system to be able to write a test that I feel has proper coverage of the internal code paths before I start writing it? No. That's part of the crafting of complicated software: you have to start with a concept and then start iteratively refining it until you end up reaching code that works well. If you actually have so much insight into how you're going to implement something that your Test-First code is going to be a reasonable exercise of your code, then your code is so trivial that there's no point writing it in the first place.

Second of all, I have absolutely no patience whatsoever for people who spend their time agonizing over whether something is a Unit, Functional, Integration, System, Performance, Scalability, blah blah blah test. Seriously. The whole thing is complete engineering navel-gazing. Here's a hint: if you have essentially two completely implementations of some logic, one just to make it a "Mock" implementation (but which has to adhere so strongly to an existing contract that you've written it twice), then you've failed and wasted everybody's time and effort doing it. Performance and scalability tests I can see, because you're probably going to run them manually on a periodic basis under a more controlled environment, but spending days agonizing over your mock framework? That's just crazy.

And unfortunately, I get comments like these from other programmers, who see all the Test-Driven Development and "Mock The World" arguments and it turns them right off. They say "if that's what it means to do all this great testing stuff, I want nothing to do with it," and then they produce crappy software. Fail all over.

You want to write some effective tests? Here are some hints:

• Layer-cake your internal design. Then test up the layer stack. Layer A requires Layer B which requires Layer C? Test all three. Bug only shows up in the tests for Layer A? Then there's a bug in Layer A. It's just that simple.
• Isolate your major components. Write an interface that you know will be easy to mock (something going to a database? write a quick in-memory representation for it) and then put a façade on the low-level code. (Note that if your layers are done as interfaces, it becomes extraordinarily easy to combine this with layer testing to have "proper" unit testing).
• Check your coverage. Unless you're running with something like Clover, you're not going to know how well your tests exercise your code paths. So use a code coverage tool to draw a spotlight on the areas that you're not covering particularly well.
• If you have a big, existing system, write system-wide tests. They're far better than nothing, and they give you a basis for your later refactoring to get to a designed-for-testing scenario.
• Continuous Integration. Make sure you have a single-step "compile the world" target somewhere (even if it's a shell or batch script), grab Bamboo (just don't run it on Solaris, natch), and start. You'll never go back.
• Design for Testing. Make sure as you write code that you think "How am I going to test this?" and then write the tests right away (same SVN/Git/P4/CVS submission), changing your logic as required to make it easy to test.
• Write tests as documentation. Writing some code that someone else is going to use? Create your documentation as a test case. The only documentation. Then if someone says "Hey, you wrote Fibble, how do I use it?" you can point them at the test case, and they can replicate it. If they say "Hey, that's great, but you didn't show me how to use the Fubar functionality," you add a test case that demonstrates that. If your tests start getting so complex that you can't point to any of them as a reasonable real-world demonstration of your code, you need to fix your design. Or add simpler test cases just for demonstration. But once you're doing demos/docs as tests, you can be pretty confident your design "smells" good and if people copy your demo/doc it's always going to work. Because you're testing it constantly.
  

This all becomes second nature over time, but if you ever hear somebody say that you should start by writing your unit test first and pull out your mock-object generator and auto-complete all the methods you haven't even written yet, just poke them in the eye until they go away. They're probably too busy trying to charge your company for a consulting gig to get anything real done anyway.

POSIX Message Queues: Useful, but Limited

Since my previous post on looking for VMS-style Mailbox semantics for more modern operating systems, I've done some research into POSIX Message Queues based on a comment in the entry. I think they're pretty darn cool, but it seems like they might be targetting a different audience than what I think I was looking for.

It also appears that the reason why I've not really seen much (any?) actual code using POSIX or SysV message queues is that support at least for POSIX Message Queues is relatively recent in the Linux kernel timeframe (2.6.6), and thus hasn't really been around for long enough as a system-primitive to find itself worked into much other software.

Disclaimer: It's entirely possible that at this point I'm largely inventing what I'd love from an internal queuing system, and it's quite probable that VMS Mailboxes don't do any of this. From here on, assume this is Kirk's Dream Land rather than "VMS Rulez".

Message Size
The default Linux implementation has the maximum message size set to 8192 bytes. That's not a lot of data to be honest, although it would be more than enough for an bridge between Erlang processes for most data. Unfortunately, this is a kernel option, and so to increase it you need to muck with your runtime kernel parameters (/proc/sys/fs/mqueue/msgsize_max).

Maximum Queue Size
The maximum number of messages you can put into the queue before it starts to block the sender is 10 messages. Again, kernel configurable (/proc/sys/fs/mqueue/msg_max), but 10 messages is pretty darn small.

I think these two combine to give a pretty clear indication of the type of message semantics they're going for: very fast, small messages. I could see a pretty clear analogue to most tick data in a financial services environment, or audio packets, or Erlang tuples. But this is really clearly designed for super-latency-critical scenarios, and the 10 message maximum is surprisingly small.

In doing some research into this, I came across this comment on a KernelTrap post, which I think gives some pretty clear insight into what's going on here, which is that this is designed for ultra-fast communications between processes (trying to make sure that everything is staying inside the same core and L1 cache if at all possible I would presume).

Persistence
This all makes sense once you realize that POSIX message queues aren't persistent at all: they're like a Priority-queue channel between currently running processes on the same machine, they don't appear to survive machine restarts in any way. This is great as a low-level IPC mechanism, but doesn't really seem to apply to more failsafe long-running messaging scenarios.

When the kernel bounces, all messages and queues disappear.

Initial Reaction
After all this, it appears to me that POSIX Message Queues are really intended to be used for IPC at a very low level of interaction within a particular system, rather than for longer processes with more data. I'd love to see them used for an interaction channel through which a disk-based queue system could be built though.

For that reason, I probably wouldn't attempt to bridge them using AMQP: it would break the semantic model of super-low-latency messaging far too much to try to bridge them over anything slower/higher-latency than Infiniband (though that would be an excellent non-MPI way to use IB, particularly if you have RDMA for the message contents). A persistent queue implementation using disk leveraging POSIX Message Queues for control? That would probably be a much better candidate for AMQP bridging to me.

Solaris 10: A Terrible Choice For Java Continuous Integration

As some of you might know, I'm not permitted to run Linux at work. Rather, I have my choice of Windows or Solaris 10 x86. (And no, I'm not allowed to run OpenSolaris, it's boring old Sol 10). I do a lot of Java. I love Continuous Integration. I started pushing Bamboo here in house. And lo and behold, I converted the masses, and we ended up merging projects from CruiseControl.NET, CruiseControl, and Hudson all to one lovely Bamboo instance. We ended up with 110 build plans in Bamboo, all but about 10 of which (dependant and overnight plans) were hitting Perforce all the time to determine whether they should build.

And then things went truly, truly, horribly wrong, to the point of my almost rescinding my Atlassian Shill status. But it turns out that it's only partially their fault. It's actually Sun's fault. Follow me here on a path down process forking details of joy.

What a Java-based Continuous Integration server does when dealing with Perforce, because Chris won't allow them to open up the protocol or provide a usable Java interface, is:

1. Check to see if there have been any changes in Perforce
2. If there have, sync code and run a build
3. Go to 1

For step #1, you've got two choices. You can either say "give me all changes, and I'll figure out whether the change applies to this build plan" or you can say "give me any changes that apply to each of my build plans." The former involves much fewer Perforce interactions, but the latter is directly supported by Perforce itself, which makes it easy to implement, and thus I would argue more correct, since Perforce has a lot of logic about exactly this type of operation (I want a CI system, not another SCM system).

Here's where the whole thing turns to Fail.

When we started adding more and more projects to our Bamboo installation, it started running more and more slowly, to a point where on a 4-core dual-dual-Opteron box it was averaging a load average of 7, with up to 80% of time spent in the kernel. That type of load was so extreme that all sorts of things started going wrong completely mysteriously.

Because of the whole Java+P4 issue, you have to shell out to run the p4 binary to interact with the Perforce server. Under Unix implementations of the JVM, that involves essentially a fork + exec (under Windows it does not, so this is arguably a superior behavior). Here's where things get sticky.

Working with Atlassian, we figured out that in part, this was because Bamboo was being over aggressive in hitting Perforce, and under aggressive in caching things that seldom/never change. Moreover, it was the cost of actually performing the fork far more than the cost of the subprocess that was killing you. 2.1.2 (to be released on Tuesday) resolves this, and so checking whether there are changes goes from 3 p4 invocations, to 1 p4 invocation. A factor of 3, which in their artificial test suite results in a 76% performance improvement.

Now the one thing that ties everything together here is that the parent process here is a JVM. But not just any JVM, a JVM tuned to running big web applications with a lot of users (the same Tomcat instance hosted our JIRA instance as well). And what's the generic rule of thumb for running a JVM for a Servlet container? More heap. More heap, more heap, more heap.

More heap, more betta, right?

Wrong.

Turns out that JVMs on Solaris 10 blow (funny that, given that they're both from the same company; you might think Sun would have an interest in making sure Java ran best on Solaris, but whatever).

In order to do a traditional fork, although there is a copy-on-write optimization to avoid actually duplication of system memory that's going to be abandoned, in order to comply with the Posix standard for fork, Solaris has to reserve enough swap space for the forked subprocess. Now in top, you won't see it, because that swap space isn't actually in use, just reserved, so you can't have it, but it's not being used. Great.

This is terrible, and is precisely why things like clone() were invented in Linux and used to great success. My best recollection (can someone clue me in here, Lazyweb?) is that this is what Runtime.exec() and ProcessBuilder.start() do on Linux-based JVMs. Solaris traditionally didn't have such a beast, so you're stuck with old Posix fork() behavior. Which sucks for this.

So what Bamboo is doing is running 3 2GB heap forks for every Perforce interaction, which is really really really bad.

When one of my colleagues did a fork test here to see how fast he could do a fork+exec in C with a 2GB memory allocation (and forking to a really small sub-process) on the same hardware and OS configuration, it turns out that doing that you're limited to 3/second, and it consumes about 50% of the machine's total CPU utilization in the kernel. Given that Bamboo was attempting to do the same thing in multiple threads, the fact that it was hitting 80% kernel utilization makes complete sense.

Hence, when I reduced the heap to a mere 512MB, Bamboo actually ran faster. Much faster. Back to being usable factor, even with my 110 plans. Once 2.1.2 comes out we should be really rolling, and we might be able to achieve our ultimate goal, which is to have every single build in our group managed from one metadata server with build agents scattered to the winds. And that would be sweet.

Oh, and don't think Sun doesn't know about this. Turns out in Solaris 10 they added a system call specifically to solve this problem. It's called posix_spawn, and it does precisely what you'd want. And Sun hasn't changed their JVM on Solaris to use it, probably because Sun, like everybody else in Solaris land, targets Solaris 8 for all those people who refuse to upgrade.

And I think that says a lot about the Solaris community.

Tiny Java Containers: None Of Them Do What I Want

Dear Lazyweb,

Although I find it difficult to focus on programming while BBC is live broadcasting Olympic Table Tennis, focus I must. But I have a conundrum: I want a Java code container, but I don't like any of the current methodologies that I'm familiar with. Please help.

Yes, I've looked into Spring and Pico/NanoContainer and all that, but their focus appears to entirely be below the level of what I'm looking at. Their focus appears to be running inside some type of application, rather than launching it (so you configure launching Spring within your Web Application: you don't start Spring at the command line). Some of my problems have to do with packaging and deployment of multi-jar projects, and the IoC containers don't work at that level that I can see.

Plus, while I totally buy IoC as a general principle and use it on a regular basis, I don't need a generic configuration file to do it, particularly not in test cases. If I want a configuration file for a particular context, I bloody write one. It exactly represents my configuration options, is typesafe, has a schema, and can be understood by not-me for later configuration. A configuration file that is capable of representing the entire universe of Java types and construction options is essentially programming in XML. For people who seem to hate J2EE, the IoC-container-crowd seem to have adopted its worst tenet: replacing simple, clear, short snippets of source code with a monstrous amount of generic XML.

But I digress.

Anyway, here's what I want:

• Packages up a set of jar files as a single classloading context. This is pretty important because I think a big problem with modern composed Java applications is assembling all your various jar files and dealing with diamond dependencies. But anyway, I want something that I can pass around and say "this is the code for my application."
  Yes, I know OSGi is a better solution for this, but I'm not there yet, and I won't necessarily be there for quite some time in terms of OSGi-ifying everything I've got. Bundle of JAR files I can handle.
  
• I want to be able to load multiple of those jar bundles at any given time in the same JVM.
  
• I want to be able to run static methods (maybe even main itself) with arguments. Heck, it's my code, I'll even adhere to some type of contract (whether explicit interface or implicit through annotations) to give me better lifecycle support.
  
• I want to be able to run lots of instances of those static methods (think message processors, each instance on a different JMS Topic). I want that to be dynamic (so I have one service kicking off others).
  
• I want to scale those instances on multiple machines. I'd like that to be automatic.
• I want to automatically handle failures of machines by distributing the load on that machine to other machines.
• I want lots and lots of monitoring and administration.
• I don't want to write a single line of code that I cannot adequately unit/functional/integration test outside of the container.
  

What I've been tempted to do in this situation in the past is to just use Tomcat as my generic container, and model everything as a Servlet with no Servlet mapping, so I'm only getting the lifecycle calls that I need and use of web.xml initialization parameters as my configuration. But that just, well, it feels dirty. Plus, it would take longer than I think would be reasonable to launch one of these processor instances, and if I'm running thousands of discrete processor instances, I need control to be quick.

My guess is that there isn't something like this out there.

My guess is that I'm about to write one. I'd rather not, but I don't think that this type of thing exists. If it does, it might be the SpringSource Application Platform, but I can't actually tell what that does to be honest. I mean, it kinda just looks like OSGi++, but I can't tell what the ++ is. (Is it really just Equinox with some extra bits for central repositories and remote configuration? Is that all?)

My guess is that the actual eventual solution here is largely just going to be my leveraging Terracotta for all my distribution and failover requirements, but writing the whole actual lifecycle management work myself.

Oh, and it has to do Java. Telling me Erlang/OTP gives you every single thing here for absolutely free doesn't help me when I'm under a strict Erlang embargo at work.

Help me Lazyweb, Help me!

Google Protocol Buffers and Streaming

Okay, I'm not going to give some huge rundown on RPC or services or network size vs. CPU efficiency. Just a little observation which gave me a little bit of a "huh?" moment the first time I used GPB for something.

Essentially, the way that Google Protocol Buffers are encoded can be seen as a small stack-based state machine that is computed as part of a Builder. The Builder holds the essential state of a particular message representation (such as an Address or AddressBook or something), and runs through the bytes in your wire representation, modifying its current state for the desired fields. When you think you've consumed everything, you then extract your Address or AddressBook or whatever from the Builder.

The commands of this state machine are pretty simple:

• Set the value of Field Number X to value Y (encoded with type Z)
• Push a new context onto the stack for a sub-Message
• Pop the context off the stack to go back to the parent message
  

It's quite clever, but it can lead to some interesting situations. For example, it's entirely valid for a non-repeated field to be set multiple times, so if you're interpreting the network commands, you can get "Set Name to Kirk Wylie" followed by "Set Name to Wylie, Kirk", and (at least in the implementations that I worked with) you get the final value set to "Wylie, Kirk".

That seems like a little piece of trivia, until you realize that Google Protocol Buffers, unlike every other network message representation that I've ever worked with, lack both a sizing prefix and a terminator. Remember, it's a state machine, so it's just going to keep processing.

Again, trivia.

Until you try to store a sequence of discrete messages into a file or over a socket. In which case, what will end up happening is that if you don't explicitly do your own termination or size prefixing, the Builder will just keep processing commands and you'll end up consuming the entire stream and get one message output with only the final values for each field. So if I'm trying to save two Address messages, the first having a name of "Kirk Wylie" and the second having a name of "Wylie, Kirk", I'll only get one output, with "Wylie, Kirk".

This also has the side effect of implicitly, in Java, forcing you to do an unnecessary byte copy. You have to get the prefix number of bytes of the following message (and computing that in the first place before you do the serialization costs you in CPU time), extract the next N bytes from the stream to a byte array, and then have your Builder parse the byte array.

All annoyances more than anything else. But probably useful for other people to know.

AMQP's Semantic Model and Mismatching

My last post with real technical content (on AMQP and standards) attracted some notice, so I wanted to follow-up with some specific issues that I find with AMQP as a programming and application semantic model, as opposed to a low-level description of the protocol itself.

A naive question might be "why in the world is there some type of semantic model in an application protocol? Surely it's just specifying how stuff goes back and forth?" But in almost all protocols there is some type of semantic model embedded: IMAP involves Messages and Folders and connections and IDLE and all that; HTTP has Headers and Cookies and all that. There's no way to provide a completely implementation-neutral protocol without some type of semantic model, because if you strive for it, you're striving for a protocol that doesn't do anything.

So that being said, we'll focus on AMQP as an application semantic model. I'll be comparing and contrasting it with the two systems/semantic models that I've used the most in a production environment: Tibco Rendezvous and JMS (personified by commercial examples from Tibco EMS and Progress SonicMQ). As I haven't used any AMQP system in anger, I won't be discussing anything specific about any particular AMQP system (e.g. Qpid, RabbitMQ). Everything's based on the 0-10 AMQP specification.

First off, let's address Virtual Hosts. Why are these useful at all? The only thing that I see is that they allow you to have the same port open for multiple uses with the same authentication system. But this is really only particularly useful because AMQP recommends a particular listening port. Systems like SonicMQ allow you to have multiple brokers share their administration domain (though on the same machine you'd have to have multiple TCP ports open), which is a much cleaner concept. Virtual Hosts look like a
solution to a problem nobody has. Just listen to different ports, and job's done.

Now there's the matter of the Exchange/Queue model. I had a conversation with the Progress people about this, and I don't think they fully got it, because the use of the term Queue here throws people from JMS-land off: they immediately think JMS Queue, which is a really horrible concept, rather than generic queues.

For clarity for those not familiar with AMQP, an Exchange is an inbound message router, and a Queue is a message delivery vehicle. Messages come into an Exchange, which directs them to Queues. Each Exchange is responsible for routing messages to the consumers that want them, and uses the logic embedded in it to support this. Therefore, I'm going to refer to these as Consumption Queues from here on for clarity.

AMQP defines a number of Exchange types, including

• Direct, where all messages flow through directly to all Consumption Queues which match exactly
  
• Fanout, where all messages flow through directly to all Consumption Queues regardless of any other criteria
• Topic, where all messages flow through based on globbed Topic hierarchies (like foo.bar.*.baz)
• Headers, where header properties determine the Consumption Queues.

So to start out with, let's look for how we'd provide JMS-style semantics. Oh, wait, we kinda can't: JMS Topics are routed via a combination of Topic and Headers, because you can arbitrarily decide what to select based on a combination: there's no Topic And Headers exchange type defined in the spec.

However, the problem goes much deeper, in that the choice of exchange is determined by the publisher. That's just wrong. As a publisher, I shouldn't be deciding on the routing rules for my messages except in the Direct case. I should only be deciding that I'm going to send messages and it should be 100% in the control of the consumers how they're filtered. Those semantics are exactly what JMS requires, and are exactly what all distributed MOM systems require (remember: those rely on publishers just throwing bits onto the wire; publishers don't even necessarily know if there are any consumers listening). Consumers can declare Bindings between their Consumption Queues and the Exchanges, but they can't control the type of exchange which a publisher is publishing.

This Matters.

It may seem like a small semantic issue, but most of the cases where I really care about the semantics of my MOM system are when things are going wrong and I'm live in production and have traders screaming down the phone to the front-line support people. Then, I need the flexibility of controlling things at a purely consumptive level, because when AMQP gets going, I will have no control over many publishers in my enterprise.

Let me elaborate slightly on that point. Assume that AMQP goes gangbusters. Then you'll start to see many systems that currently support things like Tibco Rendezvous (think Reuters data feeds, Sybase Replication Server, stuff like that) supporting publishing out to AMQP. And they'll be commercial systems, and I won't be able to control them in any way. Therefore, if they do something wonky with their Exchanges for their publishers, I won't be able to change them in any way. And I'll be royally screwed if they choose an Exchange type I don't like.

Oh, and I also don't like that clients specify their own temp queue names. Why not have the broker do it? You may say "choose a UUID" but honestly, I've had enough run-ins with lesser skilled developers to actually want to limit their ability to screw things up. Again, solution seeking a problem.

There are a lot of really nice things involved in the spec, but these two things are massive semantic problems to me that should be addressed.

Message Oriented Middleware and Open Protocols

Something that I was recently reminded to post on (due to a comment on my post on Progress and Iona) was the work that I had done into AMQP recently (encompassing RabbitMQ, Qpid, et al).

As some of you might have gathered, I'm a SonicMQ user. I'm a big fan of SonicMQ as a core product. The work that they've done on the core message broker has been phenomenal, and compared to anything else in the market, it's amazing. I'm quite pleased with many parts of the commercial decision that my company made (based on my technical input) to make a very substantial investment in SonicMQ. So don't let anything I'm saying act as SonicMQ bashing: if you are in the market for a commercial, centralized-broker-based MOM system, buy SonicMQ. With the caveats you'll get below.

The one thing that I will pretty openly criticize SonicMQ for (aside from their installers, which are openly toxic and should be approached with nothing other than open hostility) is their non-Java client libraries.

Quite simply put, their C++ client support is precisely what I would expect from a Java-based company: it's sporadic, not well engineered, looks exactly like Java translated to C++ (rather than idiomatic C++ using things like Smart Pointers and the like), and the porting is pretty abysmal (they seem to think that "Solaris" is one unified thing, rather than properly understanding the difference between hardware platforms, OS versions, compilers and versions). Their C# support ain't much better to be fair.

But it shouldn't have to be that way. Quite frankly, I'm shocked at this point that there isn't some type of IETF-supported protocol for MOM in its various forms.

So when I was alerted to AMQP, I was quite excited. The basic premise of AMQP is to provide a vendor neutral wire-level binary protocol for asynchronous, broker-based message oriented middleware. Huzzah! It's all good! We're saved! SonicMQ supports this, C++ domain experts write the client libraries, I can get Python native libraries, and I get the best of both worlds: client libraries in all the languages and environments I have to support, and the best broker-based middleware I can find.

Alas, it's not to be. While it might be time for a vendor-neutral middleware protocol, it doesn't appear that AMQP is it.

To start with, a little birdie who knows many of the LShift people (who are partially responsible for RabbitMQ) indicates that the protocol discussions are degenerating into SQL-99-level vendor arguments, before 1.0 of the protocol has even been ratified. All the vendors of pre-1.0 products are already jockeying for position. And I thought SQL was bad; but at least they all had commercial products out when they started fighting in committee! So the protocol itself seems stillborn by its own vendor quibbles.

But then there's the fact that Merrill LynchJPMorgan (one of the major client drivers of the protocol) pushed for something that doesn't really make sense. It's almost as though the protocol requires an implementation that's half bog-standard JMS, and half Tibco EMS, and all confusing. SonicMQ, for example, has IMHO, the best approach to overcoming the insanity of the JMS specification: Shared Subscriptions. But rather than allowing for vendor differentiation through features like this (and if you're evaluating MOM, you have to fully understand the power this gives you, and is unique to SonicMQ), the AMQP defines a MOM model that nobody currently implements, and is highly unintuitive.

Why, oh, why, couldn't they have just taken something like JMS, taken the edges off it (introspectible queues with online reordering? W-T-F?), and done that as a 1.0? Then at least they'd have a basis for something that would be commonly understood by MOM professionals, and would have had a scope for a vendor community to build implementation support.

Instead we face a situation where we have, simultaneously:

• A protocol nobody supports; based on


• A programming model nobody is used to; based on


• A conceptual model nobody has worked with; with


• No existing functioning reference implementations

This is not the way to build a community! It's classic chicken-and-egg. Give some existing stakeholder some reason to work with it, whether it's a vendor or an MOM professional or just an existing J2EE developer. You can't go from nothing to functioning ecosystem by redefining the world around you.

For example, I grabbed RabbitMQ and the Qpid JMS client (because it was the only one I could find). You'd think I'd be able to point the Qpid JMS client (built just on AMQP) at RabbitMQ and all would be well, right? Nope. Didn't work at all, never figured out why, nobody in the LazyWeb had ever tried such an insane thing. (You use the Qpid clients against Qpid! Not another AMQP product! That would be crazy!)

So my point is that while I think that open protocols for things like this are critical (I'll have a post on open protocols for RDBMS connectivity), this isn't it, for a variety of reasons. I wish it was, and I've pushed for Progress Software to support AMQP as an in-broker connector (entirely possible with their internal architecture), but I doubt it's going to happen.

Do I think SonicMQ is the end-all, be-all of MOM? Heck no. There's a lot of stuff they do badly. But I wish I could evaluate them as a broker rather than having to keep thinking in the back of my mind "no native cPython support; no GCC 4 Intel OpenSolaris x86 64-bit support; ...". Crack THAT, and you've got something.

UPDATED to clarify which other Financial Industry company was originally driving AMQP.