At qCon San Francisco, Asish Kumar, head of development infrastructure at Google, gave a talk titled "Development at the Speed and Scale of Google". While I wasn't at qCon San Francisco, InfoQ has posted the video and slides, which I've just watched. First of all, stop what you're doing and watch the presentation if you're interested in development/build-and-test/SCM infrastructure. Stop now. This post will wait for you.
I originally wanted to write an article about how the talk shows just how much great developer infrastructure technologies can improve performance, and why every firm and team should be devoting significant resources to improving the infrastructure provided to developers. Heck, that's why OpenGamma's systems administrator spends most of his time working on our development infrastructure.
However, then as I kept watching the video, the more I saw a potential downside to this, particularly at Google scale: virtually everything you're using is bespoke. That has a number of potential issues:
- Developers who may be quite familiar with "industry standard" development technologies will take time to come up to speed. While if you're focusing primarily on relatively long duration hires (on the order of years) this doesn't really matter that much, but many industries (for example, Finance, with its strong reliance on short-term contractors) have to factor staff churn in.
- Unlike with most other development tools (which are dominated by Open Source technologies in the particular types of development Google does), once you start aggressively building a bespoke infrastructure you have to keep going down the bespoke route: you've effectively locked yourself into a path where you have to throw more and more resources at building your bespoke infrastructure to keep up with the state of the art.
In general, for Google, I think they broadly made the right choice to go bespoke in their tool chain. They want to leverage their own (highly proprietary) production infrastructure where possible, in probably most cases they're operating at a scale that no other tool chain would be capable of handling, and (Facebook poaching notwithstanding) they focus primarily on longer-term hiring.
But what about the developers themselves? My only worry about this type of thing from a longer term career development perspective is that, quite frankly, you get lazy in managing your own development infrastructure.
Let's say that you join Google, and spend a few years working very productively using their fantastic development infrastructure. And then you want to join a much smaller firm (which may or may not be your own startup). What then?
- Could you even setup a makefile/ant script/maven setup yourself anymore? How long would it take you to relearn it?
- Are you really going to have a good knowledge of what the current state of the art is in generic development infrastructure and build tools? After all, at New Gig you won't have the Google team backing you up.
- Are you going to be perpetually frustrated in a role where you don't have a team working on your infrastructure for you?
- Are you going to start designing systems where the build and test burden would be more than manageable with the infrastructure at Google, but where the lack of that infrastructure means that you might have been better off designing the system in a radically different way?
Note that I'm not actually saying that this is a bad thing, but I think if you find yourself as a developer in an environment where you have a large, well-skilled team taking care of stuff for you (whether that team is systems administration, operations, development infrastructure, whatever), you owe it to yourself to keep up to date with the state of the art in all the things in between writing your code, and your production environment, that other people currently do for you.
Otherwise, when you finally do get to a role where you no longer have those teams to work for you (or you have to figure out if they actually know what they're doing), you're not hamstrung by your background.
Posts
