Want to know what a really fast MOM system has to handle?

Take a look at what you have to be able to consume if you want to consume live market data from major exchange feeds: April 2009 Capacity Statistics.

These statistics are the aggregate peak message flow on various industry feeds of live market data. In other words, if you want to keep up with the flow, you have to be able to consume at least that many messages per second during peak periods.

The most interesting ones (to me at least) are the ones that are multi-exchange aggregated feeds for options (Siac OPRA and NYSE ArcaBook Options). OPRA hit 869,109 mps peak in April, and NYSE ArcaBook Options hit 565,522 mps peak. If you want to consume all the major feeds listed in one box, you'd need to be able to handle more than 2MM mps.[1] That's a lot of individual messages for any software-only product to handle.

Yet another reason why I believe systems like Tervela and Solace are going to be key to the next generation of market feed aggregation: at levels like that you need something hardware accelerated in order to handle the first round of aggregation. Furthermore, using AMQP (which both currently have, or at least have plans for), you can then integrate your favorite second-tier tick distribution system (RabbitMQ, qPid, 29West [2]). Ingress handling hardware feeds to egress handling software. [3] Who says you have to choose?

Footnotes

[1]: Yes, I know you wouldn't, because they peak at different times of the day, and there is overlap, particularly between the ArcaBook and Siac feeds.
[2]: I know they're a member of the working group, but I can't see anything explicit about AMQP support on their web site as of yet, so I presume it's forthcoming.
[3]: Assuming you don't just want to go hardware-only end-to-end, which is the approach that BarCap is taking for example.

Monty Bites The Hand That Fed Him: Part 2

(Take a look at Part One of my Monty-Watch for some background as to what I think about the situation).

So Monty Widenius and Peter Zaitsev did an interview with Matthew Aslett on the creation of the Open Database Alliance. It's well worth a read.

For the record, I don't want anybody to conflate my opinions on what Monty's done with anybody else associated with the Open Database Alliance. I actually think that having such an organization ex-Oracle to make sure that there's a unified voice for everybody working (and attempting to make money) from the MySQL ecosystem, outside the current corporate owner of the MySQL brand, is a Good Thing (and something I think other projects with a single major corporate sponsor may lead to in the future). As such, having a place for all the various consultancies and technology providers to work together to ensure their interests are looked after is a pretty useful and innovative thing.

However, let's play "Look At The Balls On That Guy"!

Actual Quote Time:

I have, however, offered Oracle a partnership with Monty Program Ab, under which Oracle could get access to some of the critical developer resources Monty Program Ab has available. Monty Program Ab could also help Oracle with their open source strategy and serve as a ‘trust creating’ entity between Oracle and the open source developer community. Oracle has however not yet responded to this.

Kirk's Translation:

I have hired all of the people that ever worked for me when I stormed off from Sun in a huff. Now you may have the MySQL brand name and core IP, but I have all the engineers. Furthermore, I've been sowing as much FUD as I possibly can when, quite frankly, you haven't done anything directly to harm the interests of the MySQL community. If you want me to publicly step down from the FUD-slinging, I have a bank account to which you can send some [more] money.

Anyone who doubts the actual motivations of Monty's recent efforts should read the real quotation (as well as my translation). I think it speaks volumes about what he's attempting to achieve here, which is quite simply to spread enough FUD about Oracle's relationship to MySQL that Oracle feels like it has to engage in some type of action to bring Monty back into the fold in some form, which would involve some type of cash money payment. There really is no other conclusion possible for someone who says, in no uncertain terms: I have all the core developers; I've damaged your relationship with the core community; You could pay me and make the problems go away.

How Many Times Can Monty Sell MySQL?

UPDATE 2009-05-27: Monty's spoken to Matt Aslett, and I've responded.

COMMENTARY UPDATE 2009-05-27: If you're just reading this for the first time, after posting this it became clear (through back-channel to me) that Sun did have Monty under an Non-Compete, and chose to allow Monty to get out of it. I've commented about that in the comments [which you should really read] and in the Proggit thread. I still think Monty's actions are pretty bad looking even given that, but you should understand that there was a Non-Compete, and Monty was let out of it by Sun, before you read the original article below.

I've been thinking this since it was announced, but Monty's current attempt to monetize MySQL by hamstringing the eventual owner of his original attempt is really quite, ahem, ballsy. For a much less ranty analysis with quotes from M-Dawg himself, see Stephen O'Grady of RedMonk's writeup.

Let me give the Kirk "worked for 3 database companies and founded one expressly to compete with MySQL" Wylie synopsis:

• Monty writes MySQL way back in the day, largely so that he has a database system which doesn't have any of the complex features of an RDBMS that make it work well (you know, referential integrity, transactions, views, proper metadata support).
• People start using it, largely because it's Free-as-in-Beer (this was back in the days of minimum $100K Oracle buys just to run a simple web site), but also because it's easy to setup and administer (which Oracle/Sybase/SQLServer/DB2 were not).
• Monty wants to get rich.
• In an effort to get rich, he takes a boat load of VC funding to push MySQL from being a small open source collective to a Real Company.
• VC funding requires a business model that has real revenue behind it.
• Company adopts a split licensing model (which pissed off a lot of people at the time), and starts being effective in attracting revenue and very, very smart people as executives.
• Monty's dreams of success are realized when Sun pays a king's ransom for MySQL.
• Monty wants to have his cake [1] and eat it too, and gets all pissy and storms off in a strop and founds an attempt to get rich a second time on the same project.
• Oracle buying Sun means people take this attempt even more seriously and he attracts people who never liked the post-VC-funding MySQL business model in the first place to the cause.

So here's the question that everybody should have on their minds: How many times will Monty attempt to get rich off the same project? [2]

Now I wasn't privy to any of the contractual arrangements around MySQL's incorporation, or his common stock stake, or the Sun buyout, or any of his employment agreements [3]. I will, however, postulate that if Monty doesn't have Fuck You money at this point given a $1Bn buyout of the firm he founded, he did something Seriously Wrong, and you probably shouldn't trust his business instincts.

So one of two things is going on here:

• f(Cake + Eating) == Cake
• He fundamentally doesn't agree with a split licensing model and thinks it's doomed to failure. I really hope this isn't the case, because if it is, he was acting disingenuously at best when working for the Original Monty MySQL-Based Get Rich Scheme, by supporting a model that he didn't believe in.

If it's the latter, why did he start down the path of taking VC money in the first place? Seriously, did he honestly believe that he could take a boat-load of risk capital, and not have to provide returns to the limited partners at the core of any risk capital facility? Did he lose some type of boardroom squabble over the direction of MySQL and has been nursing a grudge ever since? [4]

Here's something any founders of Open Source projects need to realize: VC money comes with strings attached; do not take that money if you don't want to take the strings. The strings are entirely financial: VC/risk capital requires a very hefty payout in a relatively short (5-10 years max) timeframe to the limited partners who provided the VC firm with its capital to invest. In order to ramp up revenues in a reasonable timeframe, you will need to have some facility to generate reproducible, cheap-to-deliver revenue in that timeframe. Open Core is one approach, Split Licensing is another, all manner of Services are a third, there are a whole host. But you have to come up with one. Otherwise there's no point in raising risk capital, which must have a hefty payout.

If you just want to have a lifestyle business (and many lifestyle businesses can, over time, still provide you with Fuck You money if you structure them properly), while constantly maintaining an environment where you can do what you want technically in a purely-Libre environment, don't take risk capital. Grow your business organically, and enjoy the life that you've created for yourself.

But the moment you accept that term sheet, you've crossed the barrier beyond a pure hacker coding for fun, and a company executive who must deliver returns to his investors (and that may require doing things that the hacker side of you finds distasteful). If you're not willing to sign up to the transition between pure Open Source techie and Business Executive, don't accept the term sheet. And for the love of the FSM's noodly appendage, don't accept the term sheet thinking that you're going to screw your investors in the long term by going back to your roots once you've got your payout. [5] Doing so screws it for the rest of us.

Here's the #1 problem Monty's move has caused for anyone attempting to make Fuck You money off Open Source: it should make VCs very nervous indeed about Open Source investments. Let's examine what I would consider to be a logical thought process:

• If we invest in an Open Source company, the most likely outcome is an acquisition by another firm.
• If founders of projects make it a habit of storming off to fork their invention because they don't like the monetization model they helped establish, other firms are very unlikely indeed to buy Open Source companies.
• If other companies are unlikely to buy Open Source companies, our return on investment in them will be much lower.
• Therefore, there's no point in looking at them.

None of this impacts Monty: he presumably already has his Fuck You money.

But if I were a VC looking to invest in an Open Source company, I would insist on an enforceable non-compete if I could [6], and I would make sure that my exits prevented the founders from being able to fork. Otherwise, my assets are really only there to make the founders enough money that they can pursue their dream of working on pure Open Source code with enough money that they no longer have to try to get rich. Which is great for them, but not for the rest of us who aren't rich but wish we were.

Remember: going for the brass ring and taking VC money requires that you compromise something. If you don't like it, don't take the money. But once you have, realize that you may need to walk away from your baby once you've got the money for the sake of everybody else.

Just as an aside, bear in mind that nothing should stop you from doing Open Source work, including starting an entirely new project on the same basic idea, once you have your payoff. It happens all the time in other industries (how many networking hardware companies have been founded by the exact same executives?). But resist the urge to fork your original project. It's unseemly at best, and flat-out unethical at worst. If Monty had started MariaDB from scratch, that would be one thing. But he didn't. And that's the thing that makes this all seem, well, just a little bit wrong to me.

Footnotes

[1]: By cake, I mean chedda/dead presidents/papa. Cash money, yo.
[2]: Clearly more than once.
[3]: Hence I am totally unqualified to comment here. I'm doing so anyway, because if you keep reading, this turns less Monty-directed and more general-parable.
[4]: There's a reason nobody ever saw the code for my Compete-with-MySQL Open Source Database startup.
[5]: I'm not actually accusing Monty of this, and nor do I believe it to be the case (believe it or not). I think there's something else going on here. But I could see that some people might think that unethically, and you really shouldn't.
[6]: Yes, there are ways to structure this, usually during the M&A stage, by having deferred payments to the founders which don't trigger if they fork for some period of time, that even comply with California and UK restraint-of-trade law.

Sybase JDBC Drivers Ignore Your Database Selection

A very common strategy in doing database-driven functional tests is for each user to have their own database instance on a shared test machine. This means that they can do whatever they want on that database instance without impacting other users, and that multiple people can run potentially destructive test cases simultaneously.

A common approach to handling that is to append the user name to the database name, and use a system property to choose the correct database instance for that test run (for example, naming your DataSource bean in your Spring context.xml something like db_${user.name}). You then have the JDBC connection declared to have the user name in the connection string (a la jdbc:foo:bar:blahblahblah/myapp_${user.name}). All works great.

Except with Sybase.

In its voyage of annoying developers who have to use their managed language drivers (the C#/ADO.Net ones are particularly noxious), Sybase have decided to screw you by ignoring this parameter when it's not valid.

The Sybase low-level protocol for this scenario largely consists of the following steps (completely simplified):

1. Connect as user Foo
2. Connection is now bound to the default database defined for user Foo
3. Issue a use correct_database statement
4. Connection is now bound to the database specified in step #3

Here's the problem: if step 3 fails because the database desired doesn't exist, Sybase will silently swallow the failure, and you'll end up in the default database for the user, and not tell you (seriously, there's no log or exception or absolutely any sign of what database you're in). Even better, the Sybase driver doesn't even store this in any field that you can introspect in the debugger, so the Sybase client library thinks it's in the database instance you want, when in fact it's not.

If you have a shared test database (with some real data in it for doing manual/gui-directed testing), and surrogate databases for each user's "blow-away-the-world" style testing, this is Very Bad Behavior Indeed.

Solution: Stop using Sybase. Seriously. Yes, it may have been en vogue 10 years ago, but it sucks.

Oracle + Sun : A Java Perspective

Smarter people than I have written about this, but having worked at several of the major players here (a summer at Oracle proper, 12 months at BEA working on WebLogic Server, 2 years working at M7, which got bought by BEA, which got bought by Oracle), I figured I'd pipe in my $0.02.

My #1 concern here is that Larry is going to attempt to use everything in the IP arsenal that he's just acquired to screw IBM. He's done it before, he'll probably continue to do it. Considering the amount of investment that IBM has made into the Java ecosystem (at least as much as BEA, particularly when you consider Eclipse), and the amount of hostility between Oracle and IBM, this wouldn't surprise me one whit. Towards, that end, here's what I'd like to see clarification on:

• Eclipse/SWT. For too long Sun's ridiculous love-fest with NetBeans and Swing has blocked any reasonable approach towards dealing with Eclipse and SWT. The worry that I have here is that since Eclipse == IBM in many people's minds, and both JDeveloper and NetBeans are now under the same company umbrella, Oracle may decide to let commercial considerations (sticking it to IBM) and staff considerations (keeping Sun developers who have a thing for NetBeans, plus everybody internally who's committed to JDeveloper) override ecosystem considerations (we all like Eclipse way more than NetBeans or JDeveloper). My request: please properly support SWT. Not to the exclusion of Swing, but don't fight against it.
• OSGi. OSGi is probably tainted by Sun as being an Eclipse-technology, and therefore hurting NetBeans and helping IBM or some similar ridiculousness. Support it, please, and kill off anything whose sole raison d'etre is to replace it with something lamer. Modularizing the JRE is one thing, providing something completely useless except to replace OSGi is something altogether different.
• Open Java Implementations. Stephen Colebourne has been talking about this at length, and blogged about the handover. Please don't be such jerks on the JCP.
• Open Source Projects. Whither Glassfish and Metro and all the rest of it? They compete with existing BEA/Oracle assets, but are extremely valuable in making the ecosystem valuable enough to allow Oracle to extract value from their proprietary assets.
• API Neutrality. One thing Sun has been good at, because they've never had any world-beating middleware or application infrastructure technologies, is helping to craft APIs that are by and large vendor neutral (such as JDBC and JMS). Oracle, however, doesn't. Is Oracle going to follow a path like Microsoft has with the .Net APIs, where there's enhanced support for whatever Microsoft is shilling and second-class support for everything else (ADO.Net anyone?), or is it going to realize that supporting the ecosystem means vendor neutrality as much as possible? Sun had no choice as all their app infrastructure is second-rate at best, but Oracle has a choice.
• The Whole JCP Itself. What's going to happen to it? How is Oracle going to behave in general? Will we see projects falling out from under the JCP umbrella going forward?
• ZFS [1]. ZFS rocks. Massively. But Oracle's been working on Btrfs in part because Sun refuses to allow ZFS to be licensed in such a way that it can be included in the Linux kernel. Given Oracle's investment in Linux, we can has ZFS in Linux?

Mostly, I think we still have yet to see whether Oracle is going to behave like the BEA side, or the Oracle side: is Oracle going to help build the ecosystem, or is it going to use its new IP assets for proprietary advantage against IBM and Microsoft? I hope it's the former, but only time will tell.

Footnotes

[1]: Yes, it's not Java. But really, I can has? Plz?

My Ideal Social Network

I'm relatively plugged in. I blog (you're soaking in it now!); I have a FriendFeed; I tweet (and those of you who need to know who I am probably already do); I'm on Google Reader; I used to be on Orkut until I started getting befriended by Brazilians I had never met; I'm on LinkedIn (spoiler alert! it discloses who Derivatives Company A and Big Bank B are). I've never gotten onto FaceBook, largely because I've never seen any attraction to it (and if I want to be super-poked, I'd just as soon meet you first, thank you very much).

I don't actually like any of them as a full aggregation basis. I know that FriendFeed is attempting to aggregate everything together, but it's not quite what I need or want.

So all you Web 2.0 guys, here's what I want:

• Subscriber Control: I want to have control over who subscribes to my feeds. In particular, I need to be able to divide my life into at least four categories:
  • Technical Contacts. People who follow me because periodically I write about something someone might find potentially meaningful from a technical perspective.
  • Personal Friends. People who I know personally and might be interested in zoos and what train I got home.
  • Current Coworkers. The main crux here is that there are things that I might want to share with ex-coworkers and personal friends, but not the current ones I have to see every day.
  • Family/Near-Family [1]. These are people who might be interested in stuff going on in my life that I'm not ready to share with other categories.
  Note that nowhere in there is "random gits wot I don't know who want to voyeuristically follow my life." [2] When I do this, I need to be able to either pre-approve a subscriber to a feed, or silently dump a subscriber (see later).
• Publication Buckets: I need to be able to publish any particular item into a bucket, or a whole feed into that bucket.
• Single Republication: I need to be able to take a URL from anywhere (e.g. treat something from Google Reader the same way as something I type directly into a text box) and have it appear to consumers the same way no matter how they choose to subscribe. [3]
• Single Inbox: Sometimes I consume stuff on my phone, sometimes I consume stuff on my laptop, sometimes I consume stuff from Random Web Application. I want "I read this" to mean the same thing on everything.
• Selective Subscribes: I might want to read stuff by Zack Urlocker on MySQL and Sun, but really not care how many miles he ran that day [4]. I should be able to do that, combined with publication buckets.
• API Access: Anything I do with your web app I need to be able to do from any arbitrary app, potentially outside your control.
• Silent Unsubscribes: I need to be able to unsubscribe to someone without them knowing that I've done it. (More on why this is important anon).

Here's the thing about the silent unsubscribe thing. Social Networks largely thrive on number of connections, because that is Very Important to them. That's fine. But I need to be able to structure things in terms of feeds that I follow without feeling like I've given a slight to someone by not re-"friend"ing them (following/friending/whatever), or by dropping them later on. In particular, I may have personal friends who produce drivel I don't have the time or energy to keep up with, but I don't want to slight them by making my personal subscription/publication decisions transparent to them. Fixing this is the heart of the "social" part of networking, and is why eventually every single social network fails as the number of connections grows beyond the desire for humans to have contact to that level.

The core thing here is that this isn't about whether I'm friends with anybody. Many of the people that are my closest friends in the world don't participate in any of this stuff at all. It's about my ability to manage the flow of information in and out of myself that I deem relevant. That's a completely different matter, and the whole focus on "Social" Networks as being about friends and connections is rubbish: it's about my ability to selectively publish and subscribe to categorized feeds that are relevant to my interests at any given point in time.

Someone who can actually do web development should do this [5]. It would rock.

Footnotes

[1]: Note to readers: If I've ever camped out in your house, or you have offspring who refer to me as "Uncle Kirk", you're in here whether I share DNA segments with you or not.
[2]: Seriously, my life isn't actually that interesting. Except that I play with Pumas and Sun Bears and Ocelots and you don't.
[3]: Note that this may mean either compliance on the part of the republishing services (e.g. Google Reader), or it may mean that I have a queue of pending stuff that I have to process before it gets republished; I'm fine with both approaches working together.
[4]: I can say this. M7 Alumni In Da Hizzy!
[5]: Not it. I'm secure enough in my 5k1LLz that I can say that these days, you want me to stay as far the heck away from the browser in a day-to-day coding perspective as possible.

Rise Of The Appliances

I've posted before [1] on my desire for someone to sell me an AMQP Appliance, and while nobody has yet (probably the fact that 1.0 is in the final stages of getting finalized is impacting this), I'm encouraged that somebody will.

Memcached Appliances Ahoy!

I'm particularly encouraged by the last week's frenzy of Appliance Startup Announcements:

• Schooner Information Technologies coming out of stealth with MySQL and Memcached appliances.
• Gear6 launching their own Memcached appliances.
• Virident launching their MySQL and Memcached appliances.

I've read more about the Schooner appliances than the other two [2], but they appear to be roughly what I said my ideal AMQP appliance would be: Tier-1 hardware, linux kernel, custom software, heavily tuned to the particular bits of hardware in the box itself. I'm a little surprised at the $45k price tag, but I suppose it's good to hit the high-end of the market first and work your way down over time. $45k would be justified with the amounts of RAM and SSDs in the Schooner box, but I'm sure they can scale things down to less Flash and less RAM and still have a reasonable margin.

Open Protocols Make It Possible

What makes this possible (particularly that everybody's targetting the currently very-much-in-vogue Memcached) is that there is a standard protocol. While the memcached protocol isn't a de jure standard, it is a de facto standard, and one that at this point is unlikely to change given the amount of software doing interesting things with it as a protocol.

It's only a matter of time before we see the same types of innovation in messaging or anything else for that matter if you can define a stable network interface to it. If you can define an open protocol (whether de facto or de jure), you can build an appliance to serve it. And it would probably be a better quality of service than rolling your own on top of the same hardware and OS combination. Systems which are heavily I/O bound (messaging, databases, network caches) benefit greatly from advanced tuning of the entire stack from hardware through to application code. Thus, they make sense to optimize holistically, whether you're using general purpose hardware or specialized hardware. Appliance vendors can do this. Generic software vendors have to rely on teaching their customers how to do it.

Cloud Hardware Combination

The one problem with this move going forward is that you can't do it with Cloud Computing, because you don't have access to the physical environment and can't put your own hardware in place. And there's a lot of stuff you can't do as a result:

• Appliances like Schooner, much less ultra-high performance messaging brokers like Tervela or Solace
• Direct network connections to the proprietary or specific network of your choice.
• Storage optimization through the use of any type of Flash acceleration
• Anything requiring significant local bandwidth on a single box

That leads me to question whether the current vogue of completely ignoring any type of CapEx whatsoever is actually optimal for your core infrastructure. For example, if you can spend $5k on a machine to do your MOM load perfectly, or your memcached big and fast, or whatever, how much extra are you going to have to spend in overall VM slice hours to equal that performance (for example, are you going to have to run 10 VM slices running memcached and an extra 5 app server slices because you can't hit the performance of a single box)? This is entirely possible for something as memory bandwidth intensive as memcached, or as network bandwidth intensive as an AMQP broker. Where's the crossover point?

The bright spot here, though, is Open Protocols. I know I keep banging on about it, but where this all merges together brilliantly is that you don't have to have your own appliances or hardware in your Cloud Computing vendor: you can just talk open protocols and allow the utility provider to sell it to you. You want memcached? They'll sell you an IP/port that has access to X gigabytes and guaranteed response type of Y for $Z/[request|month|core time]. Open network protocols thus act as the disconnected version of open APIs in the software development world: you can swap out the underlying implementation without substantial changes to the rest of your environment.

Furthermore, working with Open Protocols in this case insulates you from PaaS lock-in. If your utility provider is providing Caching As A Service using their own custom, proprietary protocol, in order to move to a different PaaS vendor you'll have to change your app stack in some pretty substantial ways; if it's both sides providing an equivalent service (meaning protocol and semantics), you can migrate with minimal efforts (particularly if it's a transient service).

Footnotes

[1]: Meta-note: haven't been posting recently, even with the AMQP F2F and Solace announcements, because I've been busy working on some big stuff that I can't talk about at all yet. If it happens, I'll let my adoring public know.
[2]: Because Red Point, which backed Radik in the past, funded them in their Series A. I like to see what they're investing in.