San Francisco, Here I Come. At least for a visit.
Next week I'm going to be in the San Francisco Bay Area visiting friends and running errands that I've sorely neglected over my unexpected 2-year absence.
I'm already going to be meeting quite a few people down in the Valley, but if you've been lurking around here (or actively participating) and want to meet up, just lemme know (you can email me right through the blog I think, otherwise post a comment and I'll follow up) and I'll try to swing by and see you, particularly if you've got some cool tech going on!
Or cookies.
Monday, November 10, 2008
Sell Me An AMQP Appliance
I've been doing some thinking recently on the bevy of AMQP broker software which is being created, and I think it's a really positive thing that we're definitely seeing a new swathe of innovation in the broker-based middle ware space. In particular, the space seemed to many to be a fossilized space just a year or two ago, but now with AMQP and XMPP PubSub, is getting a lot of new development momentum. But ultimately, I'm starting to think that perhaps much of the innovation will be wasted on a large category of users.
I believe that we're going to see the broker-based MOM space divided into three categories:
The thing I've come to realize is that for the third case, Central Distribution or Dedicated Broker, I don't want to buy or run "software" at all. I want to buy an appliance.
Why An Appliance?
I want to buy an appliance because I don't want to have to deal with the complexity of optimizing my installation (and by that I mean the entire hardware/OS/software stack) for the purposes of running a MOM broker. Let's start anecdotizing.
When my company was planning on rolling out SonicMQ across our entire infrastructure, we spent a lot of time working on our proposed infrastructure. This included, but is not limited to:
If Progress had sold a 1U SonicMQ Appliance, which had 4 external NICs and a pair of HA connectors, we would have bought it. Even at a premium over the software+hardware, because that premium couldn't have cost more than our time did.
Things Are Getting Worse
Now, things are starting to move really quickly on a lot of fronts that would affect an MOM appliance:
Why Hasn't This Happened Yet?
I think the primary reason why this hasn't happened for things like MOM providers as of yet is that we, customers, just plain don't trust vendors. At all.
We don't trust them because:
LeftHand Networks had an extremely interesting approach to this, which is that they had software (SAN/iQ) that really benefitted from close hardware integration, but they partnered with Tier-1 system vendors (Dell, HP) to certify the LeftHand software on the system hardware. Brilliant option. They're now part of HP, but the idea was really sound. But take it a step further, and do what a lot of companies are now doing.
OEM the platform of a Tier-1 system vendor (Dell, HP, IBM, Sun). Sell us that platform as is. Don't bloody pretend it's something magical (for example, don't even bloody think of not allowing me to swap out one of your Special Magic Drives for a stock Dell/HP/IBM/Sun drive). Just sell it as an appliance with your software.
But even more than that, be quite open that what you're selling is:
This covers all the hardware vendor objections (even if you just choose one Tier-One vendor, they all have programs for just this, and we all have experience with all of them), and still allows you to optimize for the OS and hardware like mad.
The Open Source Problem
But what if you want to run things all open-source on your own hardware? Then have the project, rather than distributing an RPM, distribute a full bootable image that you load. Or have your favorite Linux/OpenSolaris distribution come up with an installer that rather than building a general purpose system, builds a system dedicated to just running that particular application (e.g. Anaconda, but custom RPMs and boot images).
I think this is probably going to start being a more common paradigm going forward, and the move to cloud computing is probably going to take things farther as people use something like CohesiveFT or some other mechanism to grab VM instances or instances for their favorite cloud computing provider for infrastructure software.
Why I don't think you're going to want a Cloud Computing solution to this in particular (e.g. an EC2 cluster) is that something like MOM done right in Central Distribution mode is pretty network topology sensitive: it doesn't make sense to virtualize it or run it as a VMWare image. It wants to be close to the hardware. Run it in another type of workload, but don't try to push tens of thousands of persistent messages through it per second; it's never going to work. But the same general approach has a lot of merit for other types of projects.
Conclusion
I think there's a big option for someone to provide an AMQP appliance for the centralized distribution case and push it heavily. There's no reason why I should be running what essentially amounts to a persistent network appliance on generic off-the-shelf hardware and operating systems, and there's no reason at all why I should have to do your performance optimization yourself.
But again, AMQP is the secret ingredient: I'll trust an appliance running custom software to provide a standard network service. I wouldn't like an appliance running custom software for a custom protocol.
I believe that we're going to see the broker-based MOM space divided into three categories:
- Edge Distribution. Imagine that you have 10 processes on the same machine that are all receiving the same data. The most optimal way to distribute the data is for that machine to be running its own broker and connecting with the hub broker to distribute each message once for redelivery to the 10 processes on the edge system.
- Embedded. In this case, an application is being written against the MOM protocols in question, but everything's happening inside the application's configuration, so that there are no external dependencies. Atlassian's Bamboo is a good example of this: they embed ActiveMQ so that their customers don't have to have an existing JMS infrastructure in place, and you don't even necessarily know you're running JMS at all.
- Central Distribution. This is every other case, and generally means a node that is configured just to act as a broker, and isn't really doing other stuff.
The thing I've come to realize is that for the third case, Central Distribution or Dedicated Broker, I don't want to buy or run "software" at all. I want to buy an appliance.
Why An Appliance?
I want to buy an appliance because I don't want to have to deal with the complexity of optimizing my installation (and by that I mean the entire hardware/OS/software stack) for the purposes of running a MOM broker. Let's start anecdotizing.
When my company was planning on rolling out SonicMQ across our entire infrastructure, we spent a lot of time working on our proposed infrastructure. This included, but is not limited to:
- Base Hardware Configuration. What processor? How fast? How many?
- Failover Network Connections. For our High Availability Pairs, what network connection(s) did we want to use for the replication traffic? How precisely were they to be configured?
- The OS and Patch Level. This got particularly hairy with some of the hardware options for the failover network connections, and we had to do a lot of patch wrangling to test out various hardware options.
- Filesystem Layouts. For the various types of filesystem storage (WAL for persistent message storage, message database for persistent topic and queue storage), how did various options of disk partitioning, disk speed, filesystem layout and options affect performance?
- RAM. How much? What GC settings for the JVM? Swap configuration?
- SonicMQ Configuration. How did any of the above affect SonicMQ? Were there any things we had to tweak to get it to perform better?
If Progress had sold a 1U SonicMQ Appliance, which had 4 external NICs and a pair of HA connectors, we would have bought it. Even at a premium over the software+hardware, because that premium couldn't have cost more than our time did.
Things Are Getting Worse
Now, things are starting to move really quickly on a lot of fronts that would affect an MOM appliance:
- Storage is changing rapidly. An ideal platform here would have some flash in it somewhere. But where? Fusion-IO-style PCIe card? Sun-style ZFS log acceleration? I have no idea, and I don't want to have to know precisely how to make your software go faster with Flash added. That's your job, Mr. Vendor.
- Networks are also changing rapidly. What to use for an interconnect? 10 GbE? Infiniband? Something else? You figure it out for me.
- Chips are opening up. Do you work best with Xeon? Opteron? Niagara? Does it matter? How fast do they have to be to saturate all network connections? I don't know. That's your job.
- Can you go faster with TCPoE? Or does it clog you down? What about custom FPGAs for routing tables? Any way to leverage GPGPUs? Again, you're the vendor, figure out the most cost effective way for me to go as fast as possible.
- Can you move into the kernel? If so, what parts? Remember, on a general purpose system my systems team will never allow a vendor to muck with the kernel. Once you're selling an "appliance", you can do what you want.
Why Hasn't This Happened Yet?
I think the primary reason why this hasn't happened for things like MOM providers as of yet is that we, customers, just plain don't trust vendors. At all.
We don't trust them because:
- They don't cycle hardware quickly. Our SonicMQ boxes are running on dual dual-core Opterons. It's monstrously overkill, and they're mostly idle. But generically most customers don't trust vendors, particularly of second-tier appliances, to keep up with industry trends, and that costs you in terms of overall performance. They have to buy in bulk to get the discounts from the chip vendors, and that means that they're going to push their stock as long as they have to.
- They monstrously overcharge. Anyone who buys storage (and a perisistent MOM system is a storage beast as much as a network beast) knows this. Buy a disk drive for $X. Put it in a slide which costs $Y. Sell it for $Z where Z ~= 5(X+Y).
Look, we get it, you need to charge a premium. But don't sell me ordinary equipment and charge a monstrous premium over what I can get myself from my reseller, or by going directly to the OEM. Clue Phone's Ringing: We're Not Idiots.
You can charge a premium where it's warranted. But being an "appliance" vendor is not an excuse to jack prices up to a ridiculous level just because you can. You do that, and we'll start asking for generic software and configure it ourselves again. - They're Another Hardware Vendor. Each hardware vendor we have means another stupid type of disk slide and another "certified" RAM module which is the same as your standard PC RAM. It's all extra stock we have to hold, it's all extra work our purchasing department has to do, it's all extra validation we have to perform, it's all extra overhead.
- They Don't Manage Well. Each appliance vendor seems to think that Their Approach is the Right One. So instead of saying "we're going to support getting machine statistics for Linux boxes" and just deploying that, we customers end up having a myriad of different SNMP MIBs and other little tweaks that means that for each new appliance, we have to change a whole lot of our management infrastructure. Blech.
(Anecdote Time! NetApp provide SNMP MIBs on their network adapters. But for total bits, they don't provide a 64-bit counter. Meaning that if you have a 1Gbps network connection, it'll roll over in like 7 minutes. So if you're not running your stat gathering MRTG/Whatever every 5 minutes, you'll have to factor in rolls of the 32-bit counter. I will assume that you, as an "appliance" vendor, will have similar levels of Stupid).
LeftHand Networks had an extremely interesting approach to this, which is that they had software (SAN/iQ) that really benefitted from close hardware integration, but they partnered with Tier-1 system vendors (Dell, HP) to certify the LeftHand software on the system hardware. Brilliant option. They're now part of HP, but the idea was really sound. But take it a step further, and do what a lot of companies are now doing.
OEM the platform of a Tier-1 system vendor (Dell, HP, IBM, Sun). Sell us that platform as is. Don't bloody pretend it's something magical (for example, don't even bloody think of not allowing me to swap out one of your Special Magic Drives for a stock Dell/HP/IBM/Sun drive). Just sell it as an appliance with your software.
But even more than that, be quite open that what you're selling is:
- Tier-1 Hardware, running
- A customized Linux/OpenSolaris/*BSD operating system, with
- Your MOM software, with
- Perfect configuration, and
- Special Sauce (optional).
This covers all the hardware vendor objections (even if you just choose one Tier-One vendor, they all have programs for just this, and we all have experience with all of them), and still allows you to optimize for the OS and hardware like mad.
The Open Source Problem
But what if you want to run things all open-source on your own hardware? Then have the project, rather than distributing an RPM, distribute a full bootable image that you load. Or have your favorite Linux/OpenSolaris distribution come up with an installer that rather than building a general purpose system, builds a system dedicated to just running that particular application (e.g. Anaconda, but custom RPMs and boot images).
I think this is probably going to start being a more common paradigm going forward, and the move to cloud computing is probably going to take things farther as people use something like CohesiveFT or some other mechanism to grab VM instances or instances for their favorite cloud computing provider for infrastructure software.
Why I don't think you're going to want a Cloud Computing solution to this in particular (e.g. an EC2 cluster) is that something like MOM done right in Central Distribution mode is pretty network topology sensitive: it doesn't make sense to virtualize it or run it as a VMWare image. It wants to be close to the hardware. Run it in another type of workload, but don't try to push tens of thousands of persistent messages through it per second; it's never going to work. But the same general approach has a lot of merit for other types of projects.
Conclusion
I think there's a big option for someone to provide an AMQP appliance for the centralized distribution case and push it heavily. There's no reason why I should be running what essentially amounts to a persistent network appliance on generic off-the-shelf hardware and operating systems, and there's no reason at all why I should have to do your performance optimization yourself.
But again, AMQP is the secret ingredient: I'll trust an appliance running custom software to provide a standard network service. I wouldn't like an appliance running custom software for a custom protocol.
IEEE 754 Floating Point Binary Representations
Just to gather up a whole bunch of stuff I had to slog through and make this more googleable, allow me to summarize some various trivia having to do with bitwise representation of IEEE 754 floating point values across platforms. This is primarily useful if you need to read and write floating point values from byte arrays or binary network streams across platforms, particularly if you have to interact with Steve Ballmer's Insanity.
First, there is no official standard for endian-ness when transmitting IEEE floating point data over the wire. That means that Java ends up defaulting to in
Secondly, IEEE 754 floating point representation defines an entire range of values to represent
This is fine if you read in the value and call
A useful C# block if you find yourself having to deal with this stuff is the following (using this will ensure that your binary representations are always bit-equivalent with the Java formats):
First, there is no official standard for endian-ness when transmitting IEEE floating point data over the wire. That means that Java ends up defaulting to in
DataInputStream and DataOutputStream to big-endian format (to match the fact that everything is big-endian), C# defaults to host-endian format (always little-endian in practice, as the Mono guys have learned.) for BinaryReader and BinaryWriter. First point of fun.Secondly, IEEE 754 floating point representation defines an entire range of values to represent
NaN, not a single value. Java takes the approach to make things byte compatible in the wire format by always emitting a single constant value for all NaN values (where all the meaningless bits are set to 0), while C# allows whatever cruft happens to be in the value on the CPU to flow through to your binary representation. And don't assume in C# that double.NaN has all those set to 0. It doesn't. In practice, double.NaN in C# is full of cruft.This is fine if you read in the value and call
IsNaN on it, but not so great if you want to check that your serialized/deserialized byte arrays are fine. For that, you need to mask out to ensure that you're always writing a canonical representation of your NaN values.A useful C# block if you find yourself having to deal with this stuff is the following (using this will ensure that your binary representations are always bit-equivalent with the Java formats):
Friday, November 07, 2008
Linux Fork Performance Redux: Large Pages
After a comment from Kostas on my last Linux Fork Test post, I worked with my esteemed colleagues to try things out with large page support. Wow, what a difference that made on Linux.
The theory here is that forking involves playing around with TLBs entries quite a bit. Since a monstrous full heap will have a lot of 8KB page TLB entries to contend with, if we shrink the number of TLB entries by a factor of 256 (by working with 2048KB pages rather than 8KB ones), you'll limit the amount of time that the kernel is spending mucking with them.
First of all, make sure you read this: The large memory support page from Sun. Now that we've gotten the formals out of the way, here's some fun we had.
First of all, a stock RHEL 5.2 installation has a
And so I kicked things off with the
Every single time I got a
Turns out those little tiny bits that they say in the support page about not working for non-privileged users are completely accurate. These all went away when I had someone with sudo rights run the process as root, and all my numbers are from running as root. So just assume that even 1.6.0_10 ain't going to allow you to allocate any large pages if you're not root.
So he ran things as root (and I re-ran things as non-root without the UseLargePages flag since I changed the test slightly). Here's some fun comparison:
Oh, and that speedup between 512MB and 1024MB? Completely reproducible. Not sure what precisely was going on there, I'm going to assume my test case is flawed somehow.
It's happening so quickly at this point that I'm quite suspicious that all I'm measuring is the
So it turns out that you really can make Java fork like crazy on Linux, as long as you're willing to run as root. And I don't know why and my naive googling didn't help. If someone can let me know, I'd really greatly appreciate it.
Did any of this help Solaris x86? Not one whit. Adding the
The theory here is that forking involves playing around with TLBs entries quite a bit. Since a monstrous full heap will have a lot of 8KB page TLB entries to contend with, if we shrink the number of TLB entries by a factor of 256 (by working with 2048KB pages rather than 8KB ones), you'll limit the amount of time that the kernel is spending mucking with them.
First of all, make sure you read this: The large memory support page from Sun. Now that we've gotten the formals out of the way, here's some fun we had.
First of all, a stock RHEL 5.2 installation has a
HugePages_Total set to 0 (cat /proc/meminfo). No huge pages whatsoever. So you need to bump that up. For my test (maximum 2GB fully populated heap on a 4GB physical RAM system), we decided to set that to 3 GB, which is 1536 2MB pages.echo 1536 > /proc/sys/vm/nr_hugepages isn't guaranteed to actually do that, and the first time we ran it, we ended up with a whopping 2 HugePages_Total. Second time bumped us up to 4. So we went on a process hunt to eliminate any processes that were stopping us working, and got things down pretty small. Now we were able to get up to 870, which was good enough for my 1GB tests (which indicated the major performance degradation anyway), though not for the 2GB test. (Yes, I know that you're supposed to do this on startup, but I didn't have that option so we did what we could).And so I kicked things off with the
-XX:+UseLargePages flag. Fail.Every single time I got a
Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (ero = 12). And nothing would run. Well, damn!Turns out those little tiny bits that they say in the support page about not working for non-privileged users are completely accurate. These all went away when I had someone with sudo rights run the process as root, and all my numbers are from running as root. So just assume that even 1.6.0_10 ain't going to allow you to allocate any large pages if you're not root.
So he ran things as root (and I re-ran things as non-root without the UseLargePages flag since I changed the test slightly). Here's some fun comparison:
| Heap Size | Large Pages Time | Normal Pages Time | Speedup |
|---|---|---|---|
| 128MB | 3.589sec | 13.217sec | 3.68 times faster |
| 256MB | 3.62sec | 15.314sec | 4.23 times faster |
| 512MB | 4.638sec | 36.692sec | 7.91 times faster |
| 1024MB | 3.885sec | 67.062sec | 17.26 times faster |
Oh, and that speedup between 512MB and 1024MB? Completely reproducible. Not sure what precisely was going on there, I'm going to assume my test case is flawed somehow.
It's happening so quickly at this point that I'm quite suspicious that all I'm measuring is the
/bin/false process startup and teardown performance, as well as the concurrency inside Java. I don't actually think I'm testing anything of any meaningful precision anymore. Maybe at a few million forks or with higher concurrency, but I've achieved essentially a constant amount of time spent forking, so I've gotten out of the heap issue really.So it turns out that you really can make Java fork like crazy on Linux, as long as you're willing to run as root. And I don't know why and my naive googling didn't help. If someone can let me know, I'd really greatly appreciate it.
Did any of this help Solaris x86? Not one whit. Adding the
-XX:UseLargePages flag (even though Solaris 10 doesn't require any type of configuration to make it work) didn't improve performance at all, and Solaris was still twice as slow as Linux without the flag.
Monday, November 03, 2008
C# BinaryWriter is Little Endian Because Microsoft Hates The Internet
Let's assume that you're developing the primary runtime class library for a programming language, and you need to write primitive types to a network connection. You've essentially got two options:
Let's assume that you've decided to not allow the user to specify the endianness of the data that the user is going to send over the wire easily. What endianness might you then choose for your developers? Might you choose the one that's officially called "Network Byte Order?"
Well, no. You're working on
Now let's assume that you have a developer who's trying to do the right thing (where the right thing is not to start crying "Waaah, C# sucks, so you have to change the internet to support whatever Ballmer's got cooking"). You might try to make it easy for them to output data in network byte order. How might you do that?
By hiding the
Note: To be fair, it's entirely possible that Microsoft has not chosen little-endian by conscious decision, but is merely using host byte order. But for compatibility, Mono has to replicate this as little-endian-always. Which is going to be a lot of fun if and when Microsoft actually tries to port this to a new architecture.
- Allow the application to specify the endianness of the data
- Require the application to use a particular endinanness.
Let's assume that you've decided to not allow the user to specify the endianness of the data that the user is going to send over the wire easily. What endianness might you then choose for your developers? Might you choose the one that's officially called "Network Byte Order?"
Well, no. You're working on
.NET and you work for Microsoft, so you'll use the opposite of that.Now let's assume that you have a developer who's trying to do the right thing (where the right thing is not to start crying "Waaah, C# sucks, so you have to change the internet to support whatever Ballmer's got cooking"). You might try to make it easy for them to output data in network byte order. How might you do that?
By hiding the
.NET equivalent of htonl and ntohl as static methods in System.Net.IPAddress, duh. Because that's obviously where you'd look when thinking of where to find binary data endian conversion routines. What in the world could be more logical or easy to find without StackOverflow?Note: To be fair, it's entirely possible that Microsoft has not chosen little-endian by conscious decision, but is merely using host byte order. But for compatibility, Mono has to replicate this as little-endian-always. Which is going to be a lot of fun if and when Microsoft actually tries to port this to a new architecture.
SSL For Self-Signed Bad For The Internet?
In an article (originally posted as a blog entry, then archived as a web page), Nat Tuck complains that Mozilla's behavior on self-signed SSL certificates is bad, because it stops people from using encryption where they don't care about the possibility of a MITM attack. I think he's mostly wrong.
The big reason why I think he's wrong is that I think the default behavior is correct: the vast majority of people don't understand the complexities of a Man-In-The-Middle attack and how that can affect any self-signed certificate. However, I totally understand his desire for more encryption.
I'm pretty annoyed at the fact that at the moment, I'm pretty sure that all non-encrypted traffic on the internet is being logged and scanned at some level by multiple governments. I don't like that at all. Encryption is the only way around it.
But we've had technologies for implementing encryption on-the-fly where we don't care about the possibility of an MITM attack for yonks. Just use DH key exchange as a handshake and then use that as the key for a stream cipher. No need for certificates at all. You get strong encryption between endpoints, and acknowledge that you're potentially subject to a MITM attack. You can even combine it with some advanced DNS checking to minimize the chance that your company/government's proxy is MITM-inspecting every connection. The only problem here is that there isn't a clean URL handler supported by RFCs that is available that I know of.
Why not just start one?
The big reason why I think he's wrong is that I think the default behavior is correct: the vast majority of people don't understand the complexities of a Man-In-The-Middle attack and how that can affect any self-signed certificate. However, I totally understand his desire for more encryption.
I'm pretty annoyed at the fact that at the moment, I'm pretty sure that all non-encrypted traffic on the internet is being logged and scanned at some level by multiple governments. I don't like that at all. Encryption is the only way around it.
But we've had technologies for implementing encryption on-the-fly where we don't care about the possibility of an MITM attack for yonks. Just use DH key exchange as a handshake and then use that as the key for a stream cipher. No need for certificates at all. You get strong encryption between endpoints, and acknowledge that you're potentially subject to a MITM attack. You can even combine it with some advanced DNS checking to minimize the chance that your company/government's proxy is MITM-inspecting every connection. The only problem here is that there isn't a clean URL handler supported by RFCs that is available that I know of.
Why not just start one?
httpe (HTTP Encoded, but not Secure)?
Friday, October 31, 2008
A London TechHub? Don't Be Daft
After Rob Knight gave a shout-out to me in a blog entry, I didn't realize that other people had been talking about trying to set up some type of "TechHub", and now apparently they're talking about doing it in London.
People who have seen me in full-on rant mode down t' pub may recognize this as one of my favorite topics, and one on which I completely agree with Rob Knight in that London is perhaps the stupidest place in the UK to do it, bar none. Don't get me wrong: there are definitely technology firms doing quite interesting work in London, and there are a lot of people that I admire working for technology pure-plays here in London. And that's great. But you're never going to have anything near Silicon Valley style concentration of technology startups or firms here.
Micro-Cluster or Macro-Cluster
Your first question is whether you want a micro-cluster or a macro-cluster. I think the current plans for a Tech Hub are on the micro-cluster form: it's a building or two nearby each other where you can put lots of tech startups in one area. And what does Mike think you need for this?
And when everybody wants them, the cost of them goes up. And that means that startups get priced out of being able to move straight into them. My startup started in the cheapest office space we could get, which was a serviced office in an industrial park in San Mateo, with shared T1 access and no static IP and no public transport and right at the intersection of two freeways which were always massively crowded so you had to commute at odd hours. Because it's what we could afford. And we even had a 7-figure seed round under our belts!
Providing prestige office accomodation for startups doesn't make any sense because they'll be priced right out of moving into them.
So where do startups end up clustering by their very nature? Places which have some of the above, but are not Grade A office space for a variety of reasons. Places like Old Street and Hoxton and Shoreditch and Bermondsey. Oh, and look, those places already have quite a few startups naturally gravitating towards them.
Places like the TechHub already exist, in every way you can imagine, it's just that you can't afford them if you're a tech startup.
The Valley Is Bigger Than London
Okay, so let's reconsider as a macro-hub, where we're trying to figure out how to replicate Silicon Valley rather than Palo Alto. That requires scale. Lots of it.
When you talk about The Valley (and not the San Fernando Valley from whence comes pr0n) [my best recollection was that The Valley essentially spread from San Mateo down to southern San Jose and across to include parts of Fremont; if you include San Francisco and Berkeley you're talking about a huge area] you're talking about an area that that's about 35 miles across. That's pretty much the entire size of Greater London from Heathrow to Upminster.
One of the advantages for many people there is that you can largely pick where you live relatively independently of where you're going to work. When I was there I worked in San Francisco, San Mateo, Burlingame, Menlo Park, and Cupertino. I lived that entire time within a 1 square mile area. You can easily settle down in Cupertino or Sunnyvale or Mountain View and have the entire Silicon Valley employment system available to you. Admittedly, this is part of the whole car/freeway culture, which sucks royally, but can you say the same for London? No.
London is completely impossible to navigate in over any moderate distance whatsoever in a car. So let's stop that route right now.
Let's look at the public transport map then. In fact, the public transportation system of London is irredemably designed to do exactly one thing: get people to and from their jobs in the City, and to and from their playgrounds in the West End. And it can't even do that. You want to go from Non-City-Location-A to Non-City-Location-B via public transportation? Oh, it is to laugh. If you can do it at all, it's going to take you an eternity each way. I had a 45-mile door-to-door commute from San Francisco to Cupertino once. Took me 45 minutes each way. My current commute (from Fulham to Bank) takes me longer than that. For what, 5 miles? And I'm following the direction of the transportation system. For me to go play Table Tennis on Sunday, to go about 2.5 miles, takes me 45 minutes each way in a bus. For 2.5 miles.
In this respect, the outer/inner London system acts as a fully centralized Silicon Valley: you can settle down anywhere with a train into central London and just stay there, as you move from City-job to City-job. More on that next.
Staff Up, Yo.
So let's say that you did want to create a macro-cluster of tech firms. I would say that the #1 hurdle you have to face is this: How do you make it relatively easy to get about 20 A-list software engineers in a startup quickly. Let's look at each of those in turn:
Relatively Easy. By this I mean that your hit rate on qualified candidates isn't less than 1% (which is roughly what my hit rate on CVs to interviews was just after the Dot Com Boom ended as everybody who was pretty marginal in the valley all went looking for jobs at the same time).
20. I pick this because I think if you're looking for a team smaller than this, you could probably do that almost anywhere. And I know that modern programming languages and techniques are making it easier to do more with less, but you've still got a requirement for about 20 technical people overall, whether it's hosting, configuration, testing, the works. To get something that's really good in a commercial context still requires a fair number of people.
A-list. Startups aren't amateur hour. Well, they are in Web 2.0 (oh, snap!), but not in the real world. Bring your A game, yo.
In a startup. This point is critical because you have to get people who are willing and able to work for a startup and in a startup environment.
Quickly. Is it going to take you 12-18 months to ramp a team that quickly? Fail.
What's going to block you in every single one of those criteria? The financial industry.
Follow The Money
Where are all those people that you want working in London? The financial services industry. Why? The money. Why are we such money-grubbing whores? Because London's expensive.
When I moved to London (and it had to be London; I wasn't willing to consider anywhere else in the UK) for personal reasons about 5 years ago, I initially thought I'd go work in a startup or in the tech industry, and I met with quite a few VC-backed firms, and had several meetings with a very prominent London-based VC (who works for a very prominent Sand Hill Road firm's outpost in St. James). His advice stuck with me, and it's never more true. On average, you're always going to earn more in the financial industry with lower variance.
If you play the startup game, over a 10 year period, figure you're going to have one minor pop (acquisition for small money) and one medium pop (acquisition for major money). Assume that you will never be part of a successful IPO (because the chances are so slim that it's laughable). During that period you average £X in annual pay, with those minor and medium pops paying you £Y in total one-off payments.
During that same period in financial services, you'll be earning £A in annual payments, and £B in bonuses every year. In order to make the maths work out, you would have to assume that 10X + Y is more than 10(A + B). It's not. If you're the type of person who's going to be in at the first round of a startup (and thus makes Y meaningful), Y is going to work out at roughly the same as (or even a little more than) 10B. But A is astronomically higher than X.
And the financial industry knows that their competition for top software talent in the UK is the rest of the financial industry, and the technology sector. And they know that by their very nature, most super-strong software engineers would prefer to work in a pure-tech environment where they're not a support structure, but the central pillar of the firm. Therefore, they know that they have to pay them more, and with much lower variance, than they would get in their preferred industry. And they do.
And let's be honest: if someone told you "I'm going to pay you more every year, in fact, I'm even going to pay you enough to own property in Zone 2 in London" versus "I'm going to pay you so little you're going to be renting somewhere 30 minutes from a tube line in Zone 6, but maybe just maybe in 6 years you might do pretty well," it's a pretty easy decision for a lot of people to make.
Which is precisely why Paul Graham likes his serfs to be so young: so that they haven't yet built up any tradition of eating anything other than ramen. When you've been able to afford Nobu, you don't want to go back to nothing better than McDonalds (unless it's on the way home after the pub and you need all the fat and protein you can shove in your face if you're to be functional the next day). You'll do it if you have to, but most people aren't going to choose it.
I guess that's why the vast majority of the people I know here who are of the calibre I worked with in the valley work in finance here in London. And they wouldn't do an early stage startup, because they physically couldn't slash their monthly expenditure to a point that would make the startup pay effective. You might be able to get a few, but 20?
And even if you could get 20, do you think that you could get 400 (20 startups each with 20)? I've been at parties in San Francisco with 400 tech-company software engineers for particular social niches. Do you think you could get 400 A-list software engineers to give up their cushy financial industry salaries and stay living in London on a pittance to work for your startups? Because you'd need at least that to kickstart it.
And this isn't just a London problem. New York isn't a tech industry epicentre for the same reasons. Chicago's interesting in that it has enough financial firms to have spillover effects, but not so big that its gravitational well can't support pure-play technology firms.
Could you just pay startup employees more? Well, how many European VCs do you know who like having their founding team earning into 6 figures sterling before they have a dime of revenue? I didn't think so.
Oh, Noes! The Credit Tsunami!
Oh, wait, the Credit Crisis/Crunch/Quake/Tsunami is going to fix all this by eliminating all the financial industry jobs! My ass.
Bonuses for many are going to temporarily suck. People will be laid off. But I don't know of any strong technologists who have yet been laid off by a financial firm in London (I'm sure there are, but in the transitive closure of people that I know, I don't directly know of any). They're so difficult to hire at the best of times, you want to hold onto them. Don't count on there being such a flood of programming superstars who are given massive severances so they can all start companies at the same time that you can turn back time.
You'll probably see what happened when the Dot-Com bubble burst. You saw all the people who came into the industry (Tech then, Finance now) who really weren't good enough, but who got hired because people needed warm bodies who could do the grunt work (then web coding, now WinForms programming), they'll all go. And a few people who really shouldn't go, but get caught up in a round for some reason or another. And those people will find jobs almost instantly (great Silicon Valley anecdote: I had to delay the CEO of my firm laying me off once because I was on the phone getting a job offer from another firm).
So even if the Credit Implosion results in a different technology employee market in London, you'll just face the problem of identifying the little wheat from the piles of chaff. And that sucks very valuable resources.
If Not Here, Where?
Here's where I (somewhat) agree with Mike and disagree with Rob: Manchester isn't close enough to London. VCs aren't going to want to live in Manchester, and neither are your other financiers and lawyers and all that. They want you to be within a 1 hour drive of their offices on Sand Hill Road in Silicon Valley, which essentially defines where you can have your offices there. And they aren't going to want to get on a plane.
I think you've got an hour-and-a-half to two-hour train journey from central london (which puts you unfortunately in outer-commuter-belt). Places that are interesting:
Where does this leave us? Well, I think essentially the following:
People who have seen me in full-on rant mode down t' pub may recognize this as one of my favorite topics, and one on which I completely agree with Rob Knight in that London is perhaps the stupidest place in the UK to do it, bar none. Don't get me wrong: there are definitely technology firms doing quite interesting work in London, and there are a lot of people that I admire working for technology pure-plays here in London. And that's great. But you're never going to have anything near Silicon Valley style concentration of technology startups or firms here.
Micro-Cluster or Macro-Cluster
Your first question is whether you want a micro-cluster or a macro-cluster. I think the current plans for a Tech Hub are on the micro-cluster form: it's a building or two nearby each other where you can put lots of tech startups in one area. And what does Mike think you need for this?
- Great transport links
- Nearby cheap hotel
- Super-Fast broadband
- Hot-desk facilities
And when everybody wants them, the cost of them goes up. And that means that startups get priced out of being able to move straight into them. My startup started in the cheapest office space we could get, which was a serviced office in an industrial park in San Mateo, with shared T1 access and no static IP and no public transport and right at the intersection of two freeways which were always massively crowded so you had to commute at odd hours. Because it's what we could afford. And we even had a 7-figure seed round under our belts!
Providing prestige office accomodation for startups doesn't make any sense because they'll be priced right out of moving into them.
So where do startups end up clustering by their very nature? Places which have some of the above, but are not Grade A office space for a variety of reasons. Places like Old Street and Hoxton and Shoreditch and Bermondsey. Oh, and look, those places already have quite a few startups naturally gravitating towards them.
Places like the TechHub already exist, in every way you can imagine, it's just that you can't afford them if you're a tech startup.
The Valley Is Bigger Than London
Okay, so let's reconsider as a macro-hub, where we're trying to figure out how to replicate Silicon Valley rather than Palo Alto. That requires scale. Lots of it.
When you talk about The Valley (and not the San Fernando Valley from whence comes pr0n) [my best recollection was that The Valley essentially spread from San Mateo down to southern San Jose and across to include parts of Fremont; if you include San Francisco and Berkeley you're talking about a huge area] you're talking about an area that that's about 35 miles across. That's pretty much the entire size of Greater London from Heathrow to Upminster.
One of the advantages for many people there is that you can largely pick where you live relatively independently of where you're going to work. When I was there I worked in San Francisco, San Mateo, Burlingame, Menlo Park, and Cupertino. I lived that entire time within a 1 square mile area. You can easily settle down in Cupertino or Sunnyvale or Mountain View and have the entire Silicon Valley employment system available to you. Admittedly, this is part of the whole car/freeway culture, which sucks royally, but can you say the same for London? No.
London is completely impossible to navigate in over any moderate distance whatsoever in a car. So let's stop that route right now.
Let's look at the public transport map then. In fact, the public transportation system of London is irredemably designed to do exactly one thing: get people to and from their jobs in the City, and to and from their playgrounds in the West End. And it can't even do that. You want to go from Non-City-Location-A to Non-City-Location-B via public transportation? Oh, it is to laugh. If you can do it at all, it's going to take you an eternity each way. I had a 45-mile door-to-door commute from San Francisco to Cupertino once. Took me 45 minutes each way. My current commute (from Fulham to Bank) takes me longer than that. For what, 5 miles? And I'm following the direction of the transportation system. For me to go play Table Tennis on Sunday, to go about 2.5 miles, takes me 45 minutes each way in a bus. For 2.5 miles.
In this respect, the outer/inner London system acts as a fully centralized Silicon Valley: you can settle down anywhere with a train into central London and just stay there, as you move from City-job to City-job. More on that next.
Staff Up, Yo.
So let's say that you did want to create a macro-cluster of tech firms. I would say that the #1 hurdle you have to face is this: How do you make it relatively easy to get about 20 A-list software engineers in a startup quickly. Let's look at each of those in turn:
Relatively Easy. By this I mean that your hit rate on qualified candidates isn't less than 1% (which is roughly what my hit rate on CVs to interviews was just after the Dot Com Boom ended as everybody who was pretty marginal in the valley all went looking for jobs at the same time).
20. I pick this because I think if you're looking for a team smaller than this, you could probably do that almost anywhere. And I know that modern programming languages and techniques are making it easier to do more with less, but you've still got a requirement for about 20 technical people overall, whether it's hosting, configuration, testing, the works. To get something that's really good in a commercial context still requires a fair number of people.
A-list. Startups aren't amateur hour. Well, they are in Web 2.0 (oh, snap!), but not in the real world. Bring your A game, yo.
In a startup. This point is critical because you have to get people who are willing and able to work for a startup and in a startup environment.
Quickly. Is it going to take you 12-18 months to ramp a team that quickly? Fail.
What's going to block you in every single one of those criteria? The financial industry.
Follow The Money
Where are all those people that you want working in London? The financial services industry. Why? The money. Why are we such money-grubbing whores? Because London's expensive.
When I moved to London (and it had to be London; I wasn't willing to consider anywhere else in the UK) for personal reasons about 5 years ago, I initially thought I'd go work in a startup or in the tech industry, and I met with quite a few VC-backed firms, and had several meetings with a very prominent London-based VC (who works for a very prominent Sand Hill Road firm's outpost in St. James). His advice stuck with me, and it's never more true. On average, you're always going to earn more in the financial industry with lower variance.
If you play the startup game, over a 10 year period, figure you're going to have one minor pop (acquisition for small money) and one medium pop (acquisition for major money). Assume that you will never be part of a successful IPO (because the chances are so slim that it's laughable). During that period you average £X in annual pay, with those minor and medium pops paying you £Y in total one-off payments.
During that same period in financial services, you'll be earning £A in annual payments, and £B in bonuses every year. In order to make the maths work out, you would have to assume that 10X + Y is more than 10(A + B). It's not. If you're the type of person who's going to be in at the first round of a startup (and thus makes Y meaningful), Y is going to work out at roughly the same as (or even a little more than) 10B. But A is astronomically higher than X.
And the financial industry knows that their competition for top software talent in the UK is the rest of the financial industry, and the technology sector. And they know that by their very nature, most super-strong software engineers would prefer to work in a pure-tech environment where they're not a support structure, but the central pillar of the firm. Therefore, they know that they have to pay them more, and with much lower variance, than they would get in their preferred industry. And they do.
And let's be honest: if someone told you "I'm going to pay you more every year, in fact, I'm even going to pay you enough to own property in Zone 2 in London" versus "I'm going to pay you so little you're going to be renting somewhere 30 minutes from a tube line in Zone 6, but maybe just maybe in 6 years you might do pretty well," it's a pretty easy decision for a lot of people to make.
Which is precisely why Paul Graham likes his serfs to be so young: so that they haven't yet built up any tradition of eating anything other than ramen. When you've been able to afford Nobu, you don't want to go back to nothing better than McDonalds (unless it's on the way home after the pub and you need all the fat and protein you can shove in your face if you're to be functional the next day). You'll do it if you have to, but most people aren't going to choose it.
I guess that's why the vast majority of the people I know here who are of the calibre I worked with in the valley work in finance here in London. And they wouldn't do an early stage startup, because they physically couldn't slash their monthly expenditure to a point that would make the startup pay effective. You might be able to get a few, but 20?
And even if you could get 20, do you think that you could get 400 (20 startups each with 20)? I've been at parties in San Francisco with 400 tech-company software engineers for particular social niches. Do you think you could get 400 A-list software engineers to give up their cushy financial industry salaries and stay living in London on a pittance to work for your startups? Because you'd need at least that to kickstart it.
And this isn't just a London problem. New York isn't a tech industry epicentre for the same reasons. Chicago's interesting in that it has enough financial firms to have spillover effects, but not so big that its gravitational well can't support pure-play technology firms.
Could you just pay startup employees more? Well, how many European VCs do you know who like having their founding team earning into 6 figures sterling before they have a dime of revenue? I didn't think so.
Oh, Noes! The Credit Tsunami!
Oh, wait, the Credit Crisis/Crunch/Quake/Tsunami is going to fix all this by eliminating all the financial industry jobs! My ass.
Bonuses for many are going to temporarily suck. People will be laid off. But I don't know of any strong technologists who have yet been laid off by a financial firm in London (I'm sure there are, but in the transitive closure of people that I know, I don't directly know of any). They're so difficult to hire at the best of times, you want to hold onto them. Don't count on there being such a flood of programming superstars who are given massive severances so they can all start companies at the same time that you can turn back time.
You'll probably see what happened when the Dot-Com bubble burst. You saw all the people who came into the industry (Tech then, Finance now) who really weren't good enough, but who got hired because people needed warm bodies who could do the grunt work (then web coding, now WinForms programming), they'll all go. And a few people who really shouldn't go, but get caught up in a round for some reason or another. And those people will find jobs almost instantly (great Silicon Valley anecdote: I had to delay the CEO of my firm laying me off once because I was on the phone getting a job offer from another firm).
So even if the Credit Implosion results in a different technology employee market in London, you'll just face the problem of identifying the little wheat from the piles of chaff. And that sucks very valuable resources.
If Not Here, Where?
Here's where I (somewhat) agree with Mike and disagree with Rob: Manchester isn't close enough to London. VCs aren't going to want to live in Manchester, and neither are your other financiers and lawyers and all that. They want you to be within a 1 hour drive of their offices on Sand Hill Road in Silicon Valley, which essentially defines where you can have your offices there. And they aren't going to want to get on a plane.
I think you've got an hour-and-a-half to two-hour train journey from central london (which puts you unfortunately in outer-commuter-belt). Places that are interesting:
- Brighton. Has a similar vibe to it as San Francisco, and similar numbers of "creative types." But it's so hemmed in by geography it might not be easy to grow it. And it's horrid in the summer as all the day-trippers come down and invade it (disclaimer: I used to have a weekend place there).
- Bristol. Already has a massive HP lab outside of it, good university town, decent weather, fast trains.
- Oxford/Cambridge. Universities (remember, Palo Alto has Stanford and Berkeley has Berkeley, largely defining why Silicon Valley is where it is) bring young talent who haven't been seduced by London yet.
- Reading/Bracknell. The area was starting to get that way (with all the outposts of US tech firms out there), except that it's really soul destroying and foul so nobody young and talented would want to live there (remember: you can work in Silicon Valley [similarly soul destroying and horrible], but live in San Francisco quite easily. I had a friend working in Reading and living in London and it almost killed him). Maybe if Crossrail had gone there, but otherwise too grim.
Where does this leave us? Well, I think essentially the following:
- London has a geography and transportation system that limits you to locating in central London to draw on enough talent;
- The desirability of London for so many reasons makes any office space suitable to drawing on a lot of talent ridiculously expensive; and
- The financial industry will collectively act to ensure that the type of people you need to do a pure-tech startup simply aren't available at the prices that a startup can afford.
Subscribe to:
Posts (Atom)
Posts
