Hardcore Once More

I've mentioned a few times that I've been working on Something Big, and it's time for me to expose myself a little more publicly. It has bearing on the types of things I'll be writing about, so I think it's time to fill in the loyal Kirk's Rants readership.

As most of you know by now, I moved to the UK five and a half years ago after starting my career in Silicon Valley working for pure-technology startups, specializing in the database and web application infrastructure space. When I moved here, I looked at quite a few early-stage technology startups, but ultimately decided to sell out and work in the financial services area. I did this for a few reasons: London at the time wasn't particularly conducive to working in a pure technology startup; financial services were where the Cash Money was at; and I wanted a chance to build a Next Stage for my career. At the time, I felt that working in financial services, given the concentration of financial technology jobs in London, was the right next step for my career.

Over the last few years, I've run the gamut of financial services firms: I started out in charge of the risk management and analytics software for a hedge fund management group; I was in charge of Front Office Technology architecture for a medium-sized derivatives trading group; I worked as a grunt contractor for a Very Big American Bank.

Am I glad I've done it? Most certainly. I've worked with a great group of people, made great friends, and learned a massive amount about how the financial industry actually works. I think it was the right choice for me five and a half years ago. But it's not the right choice for me now.

Since my 6-months at Big Bank B came to an end on Friday, I've taken the Leap of Faith and have co-founded a technology startup. I've joined the circle from Hard Core Startup Geek to Financial Services Sell-Out back to Hard Core Startup Geek.

I can't say precisely what we're doing at this stage, except to say that along the way I've seen a lot of inefficiencies in the way that financial services firms develop software, and I think that we as an industry can do better. I think that financial services are too important to the global economy, and software engineering is too important to modern financial services, to accept the status quo: not only can we do better, but we must do better. Most importantly, I believe that what we as a startup are doing will help.

I'm joined by two exceptional co-founders, and have a network of other technology startups in London that didn't exist a few years ago. As I've written before, London is now a phenomenal location for a technology startup, and what we're doing we couldn't do anywhere else in the world. If you've been on the fence, take the plunge. London is now a fantastic place to do a startup, and this is a fantastic time to do it.

In short, watch this space.

P.S. Maybe I was too entrepreneurial after all.

Big Banks: Embrace Entrepreneurial Techies

After I had been made redundant by Investment Bank A, and before starting at Big Bank B, I interviewed with a Large Teutonic Bank's capital markets division, for a permanent role in London. This was to be a tech lead and architect on a massive greenfield project, where a key part of the role was to bring in "fresh blood" who had worked with some new technologies that the development manager was considering. It required spending quite a bit of time investigating technologies for potential inclusion, doing trial and error, and general research time as the project developed due to the issues of scale involved. Because it involved a lot of real-time updates, massive database work, and distributed systems development, the hiring manager though I'd be an excellent candidate. So did all the other senior architects in the group.

I was turned down for the job by the boss' boss. The reason? "He's too entrepreneurial."

I was recently chatting with another London Techie who was interviewing for a permanent role with a Large Yanqui Bank, and he had a similar experience. The hiring manager liked him and his experience, he got through the gauntlet of developers on the ground who would work with him on a daily basis, and he had to go through the "formality" of the HR interview before they could extend him an offer.

He was turned down for the job by HR. The reason? "He's too independent."

Once is coincidence, twice is a trend.

Entrepreneurial Is An Insult?

So let's consider the two of us. We're both active in the technology community outside of work hours: we blog, we go to meetups, we speak at conferences, we work on open source technologies. We've both established that we're able to apply ourselves at a strong level technically, and are able to pick up new technologies on a regular basis, both in and out of work.

You might think that these would be useful traits. You might think that these are in no small part why the hiring managers wanted to hire us in the first place. You'd be right: they realized that they needed outside skills, and some of them cutting-edge skills that are difficult to get inside the sclerotic community of financial services technology, and that they needed to pull in people who Think For Themselves. But The Organization stopped them.

Apparently both Senior Manager and HR Drone viewed us as unacceptable risks (in fact, it was spelled out precisely why I was a risk: in 5 years, I might not want to be in the same role on the same project). We would think for ourselves, we would seek out new challenges, we would rock the boat.

The fact that these tendencies lead inexorably to us having the skills necessary to perform the jobs has never apparently crossed their minds. They somehow think they can find non-independent, non-entrepreneurial technologists who are familiar with cutting-edge technologies and are able to do proper trial-and-error with design, architecture, and vendor selection. Really? How's that working out for you?

If you're worried that these candidates might eventually want to, gasp, leave the firm, on their own terms (e.g. before you make them redundant), isn't that the price you pay for having a pool of talent you can draw from to get "fresh blood"? If everybody in the world works for the first firm that hires them for the rest of their lives, how would you be able to hire people?

If you're worried, ultimately, that these candidates might lead to uncomfortable situations as they challenge the status quo, are you not implicitly recognizing that your status quo is broken? And that, more importantly, that you want it to stay broken?

Embrace The Uncomfortable

I have a friend who has the theory that the wages techies earn working for Big Banks is compensation for the work and culture they have to endure. I'm not sure I disagree. But you can still find a way in a stultified culture working with boring projects to embrace technically skilled, entrepreneurial techies.

Entrepreneurial techies like facing new challenges, and continually want to apply their skills to making things better. Surely a Large Teutonic/Yanqui Bank has a lot of challenges? And surely hiring someone because they have the skills to design and build a greenfield project implies that they have skills that probably wouldn't be applicable 5 years down the road as the project is in maintenance mode? Why not allow the employee to continually find challenges internally?

Because speaking for myself, I find working on new challenges inside an organization superior to having to find them outside the organization. Your institutional knowledge, built up over extended work periods, is valuable to both sides; your reputation makes it easier to Get Things Done; your domain skills are continually built up. This makes it easier to apply your skills inside the same organization than constantly switching organizations. Why not encourage that by allowing the techie, every couple of years, to switch to a new challenge where their skills and attitude will make a bigger difference?

Independent-thinking techies of all stripes can also be used to help fix the broken parts of that same culture. With enlightened management, someone pointing out that "although this is the way you've always done it, it doesn't have to be like this" can be exactly the type of kick that an organization needs to actually realize that it's fallen into a process/culture-trap, where the wrong bits of the organization are self-perpetuating rather than the good bits. Questioning what's going on, thinking independently, that's the only way that you can ever break out of the bounds of group-think.

Independent, entrepreneurial techies can actually make the biggest impact in the organizations that fight against them the most: they're the ones that need them the most. Use them as agents for change, challenging assumptions, challenging entrenched attitudes, challenging technical group-think. Otherwise, your worst employees (the ones who can't really get a better job elsewhere) win, and you as an organization fail.

Epilogue: Teutonic Bank Is Still Hiring

When the recruiter told me why they had gone dark and ultimately rejected me, I knew that they would never find a candidate that would satisfy both the hiring manager and his boss; they had completely mutually exclusive requirements for this hire. I told said recruiter "in 3 months when they realize they haven't been able to fill the role and come back asking if I'm available, whether I have a job or not, I'm not available."

It actually took 4 months for that call. I wasn't available.

You can has tweets

Now that my tenure at Big Bank B is drawing to a close (more of which anon when I actually have interesting technical bits to discuss), I've joined up with the global hivemind of 160-character inanity that is Twitter.

Although there have been completely unfounded rumors that I've been on Twitter for quite some time, allow me to reassure my devoted audience that these are completely incorrect. Nothing to see there, move along now...

You can find me tweeting as @kirkwy. (why such a strange handle? because every single other short permutation of my name was taken, and a short name gives massive advantage in tweet-for-retweet optimization). And if anybody knows Monty's handle, send it over so that I can stalk him in Twitter as well as the rest of the internet.

Best Multi-OS AuthN Solution?

Dear Lazyweb,

I'm going to be building a heterogenous network consisting of Linux workstations and servers, Mac workstations and laptops, Windows desktops and laptops (maybe the odd server here or there), and maybe a few fringe operating systems. It's completely Legacy Free: nothing there at all. My goals are to make sure that all my various CIFS/NFS/AFS shares are sharing credentials just fine, and that users don't have to maintain multiple passwords.

What's the current vogue for this? Ideally I'd be running something on !Windows, but if I need to stick a Windows Server somewhere I'll do it. I'd rather not, but I will. If not ActiveDirectory, do I roll with LDAP? Which implementation? On which OS? Do I go with kerberos, or the LDAP/NIS for *nix logins? Remote/semi-connected users?

I can't believe I have to ask, but working for Big Companies has addled my Systems Administration skills slightly.

Thanks in advance,
Kirk

P.S. I tried to ask this in ServerFault and before I could add content it warned me that the question was subjective and likely to be closed. So I didn't bother.

What Type of Software Engineer Do You Aspire To Be?

This is the opening part in a series of blog posts I'm writing in helping my nephew, who's in secondary (high to the North Americans reading) school, who wants to go into software as a vocation. I've decided to experiment with a semi-didactic conversation.

If you're planning on getting started in a career in software engineering, the first thing you have to ask yourself is what type of software you're most interested in:

Pure Software Development

I classify this as working with relatively low-level systems and algorithms, and working to such a degree that satisfying business requirements (while extremely important) aren't the primary consideration of the majority of actual work you do. Rather, technical constraints factor in often enough that you are forced to innovate from a technological perspective on a regular basis.

Vocational Software Development

I classify this as working on systems where the primary consideration is the efficient encoding of business rules, where off-the-shelf frameworks/products/systems are able to satisfy virtually all the technical requirements.

While some astute readers may expect me to follow with some pithy comment about the relative quality/skills/intelligence/worthiness of people who work in one of the two areas, I won't. I actually don't fundamentally view the two as mutually exclusive, and I think assigning normative judgement isn't meaningful or helpful.

If you're going to pursue a Pure Software Development career track, you should expect:

• You will need to go to a top-track computer science university [1], particularly one which (historically) taught SICP. [2]
• You will need to work at said university harder than you expect, go beyond your expertise, and generally be humbled at every stage, because you will run into people who are smarter/harder working/better programmers than you.
• You will have to decide to live in one of a few job markets in the US [3], because there's really only jobs that require that type of development in a few places [4]
• You will find the vast majority of software engineering jobs completely boring, and more importantly, if you take a purely vocational job, you will constantly look for technical problems where there are none, to your detriment as a craftsman.
• You will get to advance the state of the art, work with cool technologies, and generally maximize your geek nature on the job.

Perhaps most importantly, however, you will view your career through the eyes of technology.

However, if you're going to pursue a Vocational Software Development career track, you should expect:

• You won't have to go to quite as rarified university, and you might actually properly enjoy your undergraduate years.
• You will be able to live and work pretty much anywhere, and make choices based on you and your family's best interests rather than having to live in an extremely expensive housing market due to a pretty limited set of career opportunities outside those markets.
• You will never find intellectual stimulation in your job, except where you add it to your detriment as a craftsman.
• If you are good, you will never find it particularly difficult to find a job, because people who can efficiently apply technology to solve business problems are always in demand. [5]
• You will never get to advance the state of the art, work with particularly cool technologies, and will require external stimulation to maximize your geek nature.

Perhaps most importantly, however, you will view your career through the eyes of solving business problems.

Note that saying "I want to work in Industry X" doesn't determine which of these tracks you have to go down. You can pursue either track and work in many industries:

• Oil/Gas/Materials Exploration require loads of people who can work with 3d visualization and computational farms and GPGPUs and loads of other super-cool stuff I'm making up because I've never worked in that industry. They also require loads of people who can build systems to handle the business of pumping oil/petrol/gas/coal/gold/coltan out of the ground and into productive use.
• Finance requires loads of people who can work with low-latency market data systems and computational farms and GPGPUs and huge risk data warehouses and loads of other super-cool stuff I can't discuss due to NDAs. It also requires loads of people who can build systems to handle moving data in between mainframes and databases and workflow engines.
• Startups require loads of people who can generically push the state of the art in technology to assist end-users and other technologists. They also require loads of people good at building web applications that don't push the state of the art technologically at all.

Can you combine the two? Of course you can. 37Signals is a great example. They build applications that really don't push any boundaries technologically in any way: they're about more efficient ways of doing things, web-enabled, with excellent design. But they also built Ruby on Rails as a way of achieving those aims better. Financial systems are, at their heart, a way to enable a group of traders to maximize revenue subject to the needs of the business. But most of them require a massive application of technology to be able to satisfy those needs, and so require significant applications of pure technology in order to be successful.

In fact, in my mind, the best software engineers are ones who can handle both the pure technology and business vocational requirements simultaneously. They're rare, though, and most of the ones that I know start from one side and realize that they're still interested in the other, and grow their mentality to be able to simultaneously achieve both technology and business zen. You have to get to grips with what really moves you first, and then work on everything else from that perspective.

So which is it? Pure Technology Track, or Vocational Technology Track?

Footnotes

[1]: Go Bears.
[2]: If you didn't go to one of those schools, and you're getting hot and bothered at this stage, seriously, why are you reading my blog in the first place? Regardless, keep reading, as you'll find I'm not capping on your alma mater in any way.
[3]: In North America: San Francisco Bay Area, Chicago, North Carolina Research Triangle, Salt Lake City, New York, Seattle, Vancouver, Boston, Austin. Yes, there are outlier organizations everywhere, but in general, you better plan on spending the first part of your career one of those places.
[4]: Network effects in play here. You need a certain number of these people before the development team can work, and that means you have to have quite a few to draw on, and such they all flock together.
[5]: Particularly if you and your family are willing to change locations. Hiring for this type of engineer tends to follow cyclical hiring paths, and North America tends to have regional cyclical trends.

Want to know what a really fast MOM system has to handle?

Take a look at what you have to be able to consume if you want to consume live market data from major exchange feeds: April 2009 Capacity Statistics.

These statistics are the aggregate peak message flow on various industry feeds of live market data. In other words, if you want to keep up with the flow, you have to be able to consume at least that many messages per second during peak periods.

The most interesting ones (to me at least) are the ones that are multi-exchange aggregated feeds for options (Siac OPRA and NYSE ArcaBook Options). OPRA hit 869,109 mps peak in April, and NYSE ArcaBook Options hit 565,522 mps peak. If you want to consume all the major feeds listed in one box, you'd need to be able to handle more than 2MM mps.[1] That's a lot of individual messages for any software-only product to handle.

Yet another reason why I believe systems like Tervela and Solace are going to be key to the next generation of market feed aggregation: at levels like that you need something hardware accelerated in order to handle the first round of aggregation. Furthermore, using AMQP (which both currently have, or at least have plans for), you can then integrate your favorite second-tier tick distribution system (RabbitMQ, qPid, 29West [2]). Ingress handling hardware feeds to egress handling software. [3] Who says you have to choose?

Footnotes

[1]: Yes, I know you wouldn't, because they peak at different times of the day, and there is overlap, particularly between the ArcaBook and Siac feeds.
[2]: I know they're a member of the working group, but I can't see anything explicit about AMQP support on their web site as of yet, so I presume it's forthcoming.
[3]: Assuming you don't just want to go hardware-only end-to-end, which is the approach that BarCap is taking for example.

Monty Bites The Hand That Fed Him: Part 2

(Take a look at Part One of my Monty-Watch for some background as to what I think about the situation).

So Monty Widenius and Peter Zaitsev did an interview with Matthew Aslett on the creation of the Open Database Alliance. It's well worth a read.

For the record, I don't want anybody to conflate my opinions on what Monty's done with anybody else associated with the Open Database Alliance. I actually think that having such an organization ex-Oracle to make sure that there's a unified voice for everybody working (and attempting to make money) from the MySQL ecosystem, outside the current corporate owner of the MySQL brand, is a Good Thing (and something I think other projects with a single major corporate sponsor may lead to in the future). As such, having a place for all the various consultancies and technology providers to work together to ensure their interests are looked after is a pretty useful and innovative thing.

However, let's play "Look At The Balls On That Guy"!

Actual Quote Time:

I have, however, offered Oracle a partnership with Monty Program Ab, under which Oracle could get access to some of the critical developer resources Monty Program Ab has available. Monty Program Ab could also help Oracle with their open source strategy and serve as a ‘trust creating’ entity between Oracle and the open source developer community. Oracle has however not yet responded to this.

Kirk's Translation:

I have hired all of the people that ever worked for me when I stormed off from Sun in a huff. Now you may have the MySQL brand name and core IP, but I have all the engineers. Furthermore, I've been sowing as much FUD as I possibly can when, quite frankly, you haven't done anything directly to harm the interests of the MySQL community. If you want me to publicly step down from the FUD-slinging, I have a bank account to which you can send some [more] money.

Anyone who doubts the actual motivations of Monty's recent efforts should read the real quotation (as well as my translation). I think it speaks volumes about what he's attempting to achieve here, which is quite simply to spread enough FUD about Oracle's relationship to MySQL that Oracle feels like it has to engage in some type of action to bring Monty back into the fold in some form, which would involve some type of cash money payment. There really is no other conclusion possible for someone who says, in no uncertain terms: I have all the core developers; I've damaged your relationship with the core community; You could pay me and make the problems go away.