Perpetual Motion Machine Due Diligence Documentation

I'm not a VC, and nor do I play one on TV. Nor do I even pretend on the internet.

However, I've been backed by enough of them (or just had meaningful conversations which didn't lead to funding decisions) to be part of the network of people brought in to do initial due diligence on technology-heavy startups. And I'm getting increasingly annoyed by what I call the Perpetual Motion Machine Due Diligence Document.

How Unseasoned Entrepreneurs View Funders

Let's start with a simple premise: you're seeking funding for a technology-heavy startup. This doesn't apply to a company which has already launched (and thus should have technical credibility already established), and it doesn't apply to consumer-focused startups (where the technology better not be part of the pitch anyway).

You're doing something Really Hard. Often, you're doing something that's against conventional wisdom, to the point where many people in the market might not even attempt it. That's great! That's the type of technology-heavy startup that a lot of VCs might like to back.

But first you have to get past investors. These investors may have come from a technology background, but these days they work in senior management (if they're angels), or are retired, or are VCs and don't play with much outside of Outlook and the iPads. You've given them your slide pitch, and now they've asked for additional documentation on your Really Hard Technology. You know they won't understand it (it's Really Hard! It's against conventional wisdom! Paradigm Shift!), so what do you do?

Easy! You just dumb it down to the most outrageous claims that you possibly can. "Faster-than-light travel impossible? Those guys don't know what we know!" "Infinite compression impossible? We found a way to eliminate data entropy!" "Free energy! Fucking magnets, we know how they work!" The Perpetual Motion Machine Due Diligence Document thus emerges.

What Actually Happens

The naive entrepreneur assumes that his document is going to be read by the investors, and has to be strong enough in its claims to justify an investment, but simple enough to be understood by a general audience. So every one of these documents seems to follow a general pattern:

• This commonly accepted wisdom/theorem is actually wrong.
• We're the only ones who have figured that out.
• Our technology thus solves all problems to all people.

Here's the problem: The Technical Due Diligence Document Is Never Read By The Investor.

Savvy investors know they're out of the state of the art (if they ever were; most skilled VCs were operators more than just raw techies). They know they don't have the background to determine whether your Really Hard Technology is actually good or doable.

So what they do is get a technical due diligence document from the founder, find someone in their network who is skilled with the state-of-the-art in that area, and send them the document for review.

And that's how the document that was only ever going to be read by an investor ends up in my email inbox.

Over-Inflated Claims Destroy Credibility

So now I'm reviewing a document which appears to be written to target a child, and is so laughable in its claims that it amounts to promising perpetual motion. What do I do?

If you think "Even if that's the case, surely the technical reviewer is going to be so blowed away that he'll seek out clarification directly from me, the entrepreneur" you fail at a technical startup. Why would I expose myself to the entrepreneur? What do I get out of that? Nothing good can possibly come of that.

• VC/Angel passes on investing. Entrepreneur blames it on me and smears me in the industry. Whether the entrepreneur ultimately succeeds or fails makes no difference: no upside to me.
• Entrepreneur turns into a complete time waster. I have a limited amount of time that I can devote to doing due diligence, and VCs are aware of that. If they want me to do direct face-to-face due diligence, they'll ask me specifically to do that.
• Link between investor and me gets exposed. Many times both sides don't necessarily want that exposed to the world, particularly if it's a speculative investment possibility.

So what I do is send an email or have a 15-minute phone call explaining that the claims are completely overblown, the document has no technical detail, and there's a very low chance that their claims are actually going to stand up in production.

My word isn't the kiss of death, far from it. A good investor will collect initial opinions from several people he trusts, and determine whether to go farther. If they do, they'll ask for a smaller set of people to do in-person deep-dive due diligence, and use that to further the decision-making process.

But here's the thing: if I'm asked to be one of those people, if I had to say "the initial document is completely overblown in its claims and I doubt it's possible" in the initial review, I almost always decline to be part of the deep-dive due diligence process. Because I no longer trust anything the entrepreneur is going to say.

It's even worse if you're doing the rounds, and I get asked my opinion (by a potential customer, partner, or another investor) having seen that Perpetual Motion Machine Due Diligence Document. I instantly respond that I think it's probably snake-oil and the potential customer/partner/investor should stay away. When they ask me to do another review, again, I decline. I don't have time to do a gratis technology review for someone where I've already lost all trust in the entrepreneur and his claims.

Don't Be That Guy

It's very simple to avoid this.

• Write your technical due diligence documents assuming several experts in your field are going to review it, and the investor is never even going to open the PDF file.
• The more outrageous your claims, the less believable you're going to appear.
• Don't assume a clever engineering workaround, providing a practical solution to a theoretic problem that's good-enough for most use cases, is a bad thing. It's not. It's been the foundation of numerous successful technology ventures.
• Make me, as a reviewer, want to find out more, either out of personal interest, or as a potential customer or partner of yours.

But don't ever make me feel like I've just been handed Yet Another Perpetual Motion Due Diligence Document.

Want to meet me in New York?

While I used to travel to New York City on a regular basis when I was working directly in the financial services industry, since OpenGamma got going I've been doing as little travel as I could get away with. What that's meant in practice is no work trips in a year and a half, and only a few holiday trips as well. Furthermore, when I used to go on business trips, almost all of my time was spent in our local offices, and I didn't really get a chance to meet and greet.

However, that's changing now. I'm going to be in New York City from the 12th through the 18th of August. Yes, I know how hot it is there. Yes, I know it's going to be miserable. Yes, like the locals, I too am hoping that the heat wave will pass. But I've got a few meetings I have to take, so going I am.

If you'd like to meet up for any of the following:

• To learn more about OpenGamma (note: I'll even be packing some demo-ware and the source code if you're that keen to see what we're doing)
• To geek out about databases (of the relational and non-relational type), MOM, Java, data encodings, Atlassian products, or anything else I blog about when I'm not shamelessly promoting myself or my company
• To chat about startups, London, the tech scene here, or general expat stuff
• To publicly berate me about one of the many controversial and offensive statements I've made on my blog or Twitter feed
• To drink (coffee or alcohol) with someone you wouldn't usually drink with
• To offer me a desk to work at when I'm not otherwise engaged (hint hint)

Just hit me up. Comments down below here, kirk at kirkwylie dot com if it's not about OpenGamma, kirk at opengamma dot com if it is!

Open Core, Natural Feature Divisions, and OpenGamma

I've written in the past on Open Core strategies for Open Source technology-based businesses. I've been following this debate for quite some time, and have at least a little bit more to say about it.

Open Core has gotten a bit of a bad name recently, largely due to two major recent events:

• SugarCRM's new version and licensing going so far beyond any other established Open Source technology-based company that I hesitate to even group them into that category;
• Eucalyptus' rumor of a refusal to merge NASA-provided patches to their Open Source licensed core, by a community perception that it would make the Open Source core more competitive with the proprietary version, leading to the creation of the OpenStack initiative.

People who want a lot more backstory on the arguments in the blogosphere should turn to the 451 CAOS Theory writeups (from Matt Aslett - Post The First, and Post The Second).

Personally, I believe that Open Core business strategies can work quite effectively where there are at least two conditions that hold true:

• The core version is useful to a large subset of the target audience, without any requirement to purchase any features or services to achieve its utility; and
• There is a natural split between features/components/modules that are licensed under the Open Source core, and the proprietary extensions.

The key thing to me is that the split has to be natural. An artificial split happens when someone looks at a distinction between an Open Source part of the overall offering and a proprietary part, and can't figure out what rationale might have been used to determine which was which, except for revenue defensibility. If your user base can't look at a feature and instinctively tell you whether it belongs in the Open Source version or the proprietary version it's artificial.

OpenGamma, the company I founded which is building an Open Source Platform for financial analytics and risk management, has from its inception planned on an Open Core strategy for part of its business model. Based on all the controversy, I've clarified our position, and how we naturally divide up features, in a new blog post on our web site. That's the official company statement.

This post is to explain my personal beliefs, and how we divided up the world. Just to make it abundantly clear, let me spell it out: I will not reject a community-generated patch just to maintain the defensibility of our revenue model.

Comments and questions more than welcome, either on the OpenGamma specific story (on the OpenGamma post) or my personal take (on this one).

Update 2010-07-26

Just to be clear, people should be aware of the changes made above about Eucalyptus: they never actually rejected contributions, but there was a rumor of it leading to massive perceptional issues that flowed through into debates about Open Core, regardless of the truth of these allegations. Thanks to some very wise birds who clarified to me back-channel, leading me to make sure that this is very very clear here.

On the subject of Bamboo upgrade woes

Kirk

Custom shirts make up for Bamboo upgrade woes, right?

Thanks
Atlassian

Dear newly fellow Accel portfolio company Atlassian,

Yes.

Sincerely Yours,
Kirk

Me in my fancy new @atlassian #BestButNotYetPerfect shirt

Fanboi for life, yo.

OpenGamma has come out of stealth mode

Ever since I announced here on my blog that OpenGamma was funded, my posting frequency has dropped off precipitously. This was in no small part because I've been spending so much time helping to build OpenGamma, but also because the constraints of being in stealth mode meant that I didn't want to accidentally disclose too much about what we were building.

Today I'm pleased to report that I can finally open up to the world, because OpenGamma has come out of stealth mode.

I'm going to let the OpenGamma blog post and web site speak for itself, except to clarify the distinction between my various internet personas and publishing vehicles:

• My personal blog will remain just that: I'll keep posting on Kirk's Rants about technologies I'm interested in, and startup advice and culture. Nothing will change there, except that my current relatively low rate of updates will likely continue.
• The OpenGamma team (myself included) will be blogging about OpenGamma-related matters, and subjects that are likely to be of particular relevance to the financial technology community on the OpenGamma Blog. Some of this will be technical, some of it will be related to OpenGamma, and some of it will be specific to financial services.
• I'll try to keep the cross-posting from the OpenGamma Blog to my blog to a minimum, and only do so if I think it's a technology-related matter that would be of interest to my readers.
• My personal blog represents nothing other than my personal beliefs, and should not be construed to be the opinion of the OpenGamma group of companies, its board of directors, or executives.

If you've been wondering what I've been doing for the past year, and what OpenGamma is going to keep doing, I encourage you to visit our new web site.

Bamboo+Clover satisfy requests for test quality

Several times recently I've been asked how much and how well we test our product. Our board wants to make sure that we're building a quality product that won't result in first-customer black-eyes. Professional indemnity insurers want to make sure that action is unlikely to be taken as a result of a bug. Potential partners and customers want to make sure that a first-generation product isn't going to be completely ridden with bugs.

Sure, I can talk about how test-infected we are, and how seriously we take quality development. That goes some distance.

But OpenGamma doesn't have a dedicated QA department. And yet I know that the quality of our source code is exceptionally high. How do I convey this as quickly as possible?

I've found a technique that works pretty darn well.

Bamboo+Clover To The Rescue

All I really have to do is open a web browser and go to our Bamboo server, where I can show a screen that shows the sheer number of build, test, and Clover-coverage build plans we have, and show that they're running all the time. That gets over the "are you guys actually testing" barrier.

For those of you trying to keep track, that's only one laptop-screen-full of many. Don't try to extrapolate the number of projects/plans we have from that. :-)

But how good are those tests? That's where Clover comes in.

Pictures like this go a really long way to putting their minds at ease:

Yes, I know that just because you're executing lines of code it doesn't mean that the tests are actually checking the results and all that jazz. But as a first-cut sign of "we take testing seriously" graphs like that pretty much immediately end the coversation.

Performance of Fudge Persistence in MongoDB

Here at OpenGamma we make considerable use of MongoDB, in particular as a persistent store for data which we either don't want to spend the time on normalizing to an RDBMS Schema, or where we positively value the schema-free design approach taken by MongoDB.

We also make extensive use of the Fudge Messaging project for virtually all of our distributed object management needs (whether using Fudge Proto files, or manually doing the Object/Document encoding ourselves). Luckily, the two work extremely well together.

Because of the way that we defined the Fudge encoding specification, and designed the major Fudge Reference Implementation classes and interfaces, it's extremely easy to map the worlds of Fudge, JSON, and XML (in fact, Fudge already supports streaming translations to/from XML and JSON). We've actually had support for converting in between Fudge objects and the BasicDBObject that the MongoDB Java driver uses since Fudge 0.1, and we use it extensively in OpenGamma: anywhere you have a Fudge object, you can seemlessly persist it into a MongoDB database as a document, and load it back directly into Fudge format later on.

So with that in mind, I decided to try some performance tests on some different approaches that you can take to go from a Fudge object to a MongoDB persisted document.

Benchmark Setup

The first dimension of testing is the type of document being persisted. I had two target documents:

Small Document

This document, intended to represent something like a log file entry, consists of 3 primitive field entries, as well as a single list of 5 integers.

Large Document

This document, intended to represent a larger concept more appropriate to the OpenGamma system, consists of several hundred fields in a number of sub-documents (sub-DBObject in MongoDB, sub-FudgeFieldContainer in Fudge), across a number of different types, as well as some large byte array fields.

I considered had three different approaches to doing the conversion between the two types of objects:

MongoDB Native

In this case I just created BasicDBObject instances directly and avoided Fudge entirely as a baseline.

Fudge Converted

Created a Fudge message, and then converted to BasicDBObject using the built-in Fudge translation system

Fudge Wrapped

This one wasn't built in to Fudge yet (and won't be until I can clean it up and test it properly). I kept a Fudge data structure, and just wrapped it in an implementation of the DBObject interface, which delegated all calls to the appropriate call on FudgeFieldContainer.

Additional parameters of interest:

• Used a remote MongoDB server running on Fedora 11 (installed from Yum, mongo-stable-server-20100512-mongodb_1.fc11.x86_64 RPM) running on a VM with reasonably fast underlying disk.
• Local MongoDB server was 1.4.3 x86_64 running on Fedora 13 on a Core i7 with 8GB of RAM and all storage on an Intel SSD
• MongoDB Java Driver 1.4 (pulled from Github)
• JVM was Sun JDK 1.6.0_20 on Fedora 13 x86_64

Benchmark Results

Test Description	MongoDB Native	Fudge Converted	Fudge Wrapped
Creation of 1,000,000 Small MongoDB DBObjects	539ms	1,603ms	839ms
Persistence of 1,000,000 Small MongoDB DBObjects	41,188ms	46,201ms	92,866ms
Creation of 100,000 Large MongoDB DBObjects	15,351ms	23,956ms	15,785ms
Persistence of 100,000 Large MongoDB DBObjects (remote DB)	57,207ms	60,511ms	56,236ms
Persistence of 100,000 Large MongoDB DBObjects (local DB)	66,557ms	74,763ms	58,816ms

Results Explanation

The first thing to point out is that for the small DBObject case, the particular way in which MongoDB encodes data for transmission on the wire matters a lot. In particular, there's one decision that the driver has made that changes everything: it does a whole lot of random lookups.

A BasicDBObject extends from a LinkedHashMap, and so doing object.get(fieldName) is a very fast operation. However, because Fudge is a multi-map, we don't actually do that in Fudge, and by default we store fields as a list of fields (JSON stores lists as a, well, list; Fudge stores them as repeated values with the same field name). Because this makes point lookups slow, we intentionally do whole-message operations as often as we can, and just iterate over all the fields in the message.

The MongoDB driver code does the same thing, but instead of doing a for(Entry entry : entrySet()) style of operation, it iterates over the keys and does a separate get operation for each key. In Fudge, this is potentially a linear search through the whole message.

To work around this, in my wrapper object I built up a map where there was only a single value per field. This works well, but the small document case has 1/6 of the fields be a list, making this test thrash in CPU on doing the document conversion (which explains why the small document persistence test is more than twice as fast with the wrapper as just rebuilding the objects). Yes, I could do this optimization further, but it would be difficult to improve on the combined setup (document construction) and runtime (persistence) performance of just building up a BasicDBObject, which is what the Fudge conversion does anyway.

The wrapped Fudge object wins in every case for the large document test, no matter how many times I run them (and I've done it quite a few times for both local and remote, with all outliers eliminated). Moreover, I actually get faster performance running on a remote DB than on a local DB (which surprised me quite a bit).

The only things that I can conclude from this are:

• FudgeMsg limits the data size on insertion into the message (when you do a msg.add() operation, not on serialization) for small integral values (if you put in a long but it's actually the number 6, Fudge will convert that to a byte). However, the ByteEncoder which converts values in MongoDB to the wire representation will never do this optimization, and will actually upscale small values to at least a 32-bit boundary. This means that if you put data into a FudgeMsg first and then put it into the MongoDB wire encoding, you shrink the size of the message. Given the number of pseudo-random short, int and long values in this message, it's a clean win.
• The object churn for the non-wrapped form (where we construct instances of BasicDBObject from a FudgeFieldContainer) causes CPU effects that the wrapped form doesn't suffer from.

Conclusion

One of the things that was really pleasant for me in running this test is just how nice it is to take a document model that's designed for efficient binary encoding (Fudge), and persist it extremely quickly into a database that's designed for web-style data (MongoDB). The sum total of the actual persistence code is all of about 10 lines; I spend far more lines of code building the messages/documents themselves.

The wrapped object form definitely wins in a number of cases. My current code isn't production-quality by any means, but I think it's a useful thing to add to the Fudge arsenal. That being said, I think the real win is to rethink the way in which we get data into MongoDB in the first place.

Given the way the MongoDB Java driver iterates over fields, it seems to me that a far better solution is to cut out the DBObject system entirely, and write a Fudge persister that speaks the native MongoDB wire protocol directly, and take advantage of the direct streaming capabilities of the Fudge protocol. When we've done that, we should be going just about as fast and efficiently as we can and Fudge will have a seamless combination of rich code-level tools, efficient wire-level protocol for binary serialization, great codecs for working with text encodings like JSON and XML, and a fantastic document/message database facility using MongoDB.