OpenCore/Split Licensing: You Can Do It Wrong

Tarus Balog comes out with another salvo against what he calls Open Core licensing, and I've called Split Licensing. His general statement this time comes down to:

• Hyperic has a feature that the open source community would like;
• They have that feature in the commercial version;
• It would be stupid for them to take that feature and make it part of the Open Source version.

Ergo, Open Core is fundamentally flawed.

Almost at the same time, Zack Urlocker (M7 Alumni In Da Hizzy Yo!) has a new post where he provides the most interesting clue as to why Tarus doesn't fully understand how you can do it right:

Find a way to distinguish between capabilities that businesses want and individual community members may not even care about. For example, Scott Dietzen mentioned e-mail features related to archiving and compliance issues are really only relevant to larger companies. And as he noted, community members will even support the the idea that if you need those features, you should pay.

My Open Source Cookies stream has hit on this before. Just because one firm chose the wrong feature to be part of their cookie doesn't mean that the whole concept is flawed. That's like saying "McDonalds sells disgusting hamburgers; therefore, all hamburgers are rubbish."

Again, to continue my statements on this:

• Pursuing Split Licensing means that you have to be very careful about what you put in the Open Source vs Commercial versions;
• More importantly, it's a case of constant refinement as you discover some features that you thought were commercial only but should be migrated to the Open Source version;
• Targetting features that are only of use to committed commercial users is the key.

It may be that you have a product which is so generic that you can't come up with something that satisfies the Cookie requirement. If it is, you shouldn't be trying Split Licensing at all. Not every business model suits every Open Source project. Choose the right one.

Calypso Acquires Fail

UPDATE 2009-04-23 at end of post

Calypso has acquired Galapagos. Wow. Where to begin here.

First of all, Galapagos was the premier product from CodeFarm, which was forced into administration recently (administration ~= UK Chapter 11). So you could easily rewrite the above headline as "Calypso acquires assets of failing technology vendor for super-cheap." But PR doesn't work on telling the truth.

But the real question is why acquire it at all? Want to know what Galapagos actually does? I know, as I've used it.

It's a genetic algorithm for automagically producing CDOs, whose primary claim to fame is that it has a license to run the Moodys models automatically. What a growth industry that is these days! And so many accretive customers that will come along with the deal!

So let's play a game. We'll call it Rewrite The Press Release

Calypso Technologies today acquired the assets of a bankrupt firm specializing in selling to a failed market relying on a discredited feature. In engaging in this transaction, Calypso found a way to acquire some developers it felt were talented in a market where it currently lacks an R&D facility, while allowing the original developers of the worthless software a face and reputation saving way out of their current predicament.

Man, I should be in PR and not in the actual engineering game!

UPDATE 2009-04-23
Apparently, they haven't realized this dog don't hunt. They're trying to hire sales engineers to sell it in NYC. Uhm, guys? The structured products market is pretty dead, and if it does recover, it's probably not going to want a genetic auto-structuring system.

Let A Million Brokers Bloom

To me, one of the most promising things about AMQP is that it actually has the potential to radically expand the amount of feature-based broker differentiation in the MOM space, by eliminating a lot of the "grunt" stuff that you have to go through in order to define a new broker model.

Right now, if you want to do something interesting (meaning not commercially supported) with asynchronous message-oriented communications, you have to:

• Define your basic messaging model
• Define the wire representation of that model
• Define your protocol
• Write up the server-side handling of the protocol
• Write up the client-side handling of the protocol for each programming language you need to support

Only when you're done with all that do you actually get to do "the fun stuff," which is to write up the unique piece of functionality that meant you wanted to start on the project in the first place.[1]

Consider a particularly germane example. In my current day job I'm working for a firm (and if you're reading this blog you've heard of the firm) that has a proprietary MOM system with (what I can see as) one USP.[2] Aside from that one USP (which I'm not going to identify), it's just a simple pubsub-based MOM broker. But nobody in the market has that feature, so they can't go with a commercial vendor. And there are facets of that USP that means that no commercial vendor is likely to add exactly that feature to their product.

But in order to get that one USP feature, they have to face every single other hurdle above, and constantly deal with refining every single bit over years and years of development. None of which is about that one USP feature, it's all just plumbing. Everything other than implementing that feature is a waste of valuable development effort.

The kicker? That USP is just a custom Exchange type in AMQP 0-X terminology. Nothing more, nothing less.

And that's the beauty of AMQP.

Once we reach the promised land, someone like my current cash money provider could have just grabbed an existing framework (all the AMQP client work and server-side protocol handling in its language of preference), and implemented the custom exchange (or broken existing ones, or got freaky with queue implementations, or whatever it wanted): no need to reinvent the wheel. AMQP acts as the standardization of all the bits that you don't want or need to care about to provide custom message broking behavior, allowing you to focus all your time and attention to the parts that you really need to implement yourself.

This makes the barrier to entry for unique MOM functionality only the cost to implement the unique functionality itself; you don't have to face the hurdle of implementing everything else required for asynchronous message-oriented communications to showcase or provide one single feature.

I consider this to be the messaging version of the story behind the Eigenbase project: a framework for writing custom brokers. A few people I know (big shout out to my homies John Sichi and Julian Hyde[3]) are excellent at writing "interesting" databases (with special sauce for special contexts). However, doing that from scratch each time means that you're constantly reinventing/rewriting the same old tired stuff: a SQL parser, a transactional data store, a wire protocol, blah blah blah. So they started the Eigenbase project as a way of taking care of all that cruft that you just Have To Have so that they could focus on the stuff that's special about Their Database.

Are there equivalently architected Framework Brokers yet for AMQP? Not that I'm aware of (please pipe up in the comments if you think I'm wrong). If nothing else, in the message broker/routing space, there are a lot of things that you might want to do to speed up adhering-to-standards mode that break the general programming model you'd want from a framework-based system. That leads me to believe that just as MySQL didn't go framework-based to become Eigenbase, we'll probably see a set of AMQP brokers focusing on ease of programming custom variants, just like Eigenbase is designed specifically around programming custom variants in the database space.

And at that point, I think we'll start to see a lot of new and interesting implementations, some in-house and some exposed to the world.

Footnotes

[1] Alternatively, you can horribly abuse an open source project, but that's going to do nothing but confuse everybody involved (including yourself) and leave you liable to abandonware if you rip out the insides of a system that you're replacing key components on (so if your open source basis isn't designed as a framework, and you replace some of the key message handling parts, the moment the developers change the internal processes you're effectively stuck).

[2] Unique Selling Proposition. But you knew that of course.

[3] Broadbase In Da Hizzy!

TechCrunch Reply: VCs Shouldn't Only Fly To Emerging Markets...

(Context: TechCrunch saying the US is played out in VC funding.)

First off, allow me to correct one thing, about the confusion between high growth and high tech. Yes, VC funding is about high growth. But more than that, it's about exponential growth triggered by a capital infusion, which is what technology investment is all about: pay money down to write software, and you can sell infinite numbers of copies for only the cost of sale, which is lower (hopefully) than the cost of writing it in the first place. Otherwise, you're in Private Equity land, which is completely different.

But let's take the analogy farther. Why do you think that investment is going into China, India, and Israel? China and India are growing middle classes and have indigenous growth markets, as well as a hinterland (SE Asia for China, Pakistan/Bangladesh for India). What about Israel? That's the one that teaches you something illustrative, because I don't think anybody would really assume that Israel is very well able to effectively do business with its local hinterlands (for a variety of reasons I don't think I need to spell out here).

Israel tells the whole story of why the others are there: lots of skilled engineers who can't/won't migrate to Silicon Valley, and a culture of entrepreneurship.

Hmmmmm.....that sounds familiar..... Oh, wait, now I remember why!

So taking the TechCrunch approach, here's why London is somewhere else you should consider:

• Growth Industries. Remember, London is able to service as a financial and technology centre all of the expanded EU. You think Romania isn't a potential target for growing in transport, consumer goods, all that?
• Immigration Issues. Again, London draws from all over the EU already, and we also can't make it to the US.
• Stable Regulatory/Legal Framework. Okay, so this is a little different than it used to be given regulatory chaos everywhere, but we have an excellent legal framework and strong property rights and laws that understand and can defend against violations of intellectual property.
  
• Bonus: You like London. Okay, let's face it. You're a rich VC. London's very very suited to taking money from super-rich people. Why do you think we have so many Arabs and Russians and other assorted billionaires here? Because we're good at providing avenues for you and your significant others to spend money on a regular basis. Bangalore? Not so much from what I hear. [1]
  

If you're looking at Israel, you should be looking at London.

Footnotes:
[1] Yes, I know I'm incredibly oversimplifying. My point here is that London caters to literally hundreds of dollar billionaires on a daily basis; Bangalore, Tel Aviv and Guangzhou don't. They're getting better, but London's been a billionaire plaything (to a point that makes it impossible for those of us not in that level to live well) for at least a hundred years, and it shows. Trust me, you don't want to have those problems.

Tervela Joins AMQP Working Group

The Press Release. No, it's not on the formal AMQP Working Group web site yet as of my writing this.

I think this is really promising, as it's the first hardware vendor to have formally signed up to join the AMQP Working Group (despite my prodding of Solace to do so). I hope they manage to produce a full hardware implementation of the spec!

It also continues the industry's relentless march to where I believe is its destiny:

• Open Source
• Cheap Appliances
• Custom Hardware
• Cloud Provision
• IBM

If you're not one of those 5, figure out how you can be, because otherwise you're in the downward spiral. If you're paying someone not one of those five, start working on your exit strategy before it's too late and you're paying ground rent until the end of time for a second-rate solution. Don't migrate yet, but have your plan in place.

Update: Here's the Official Tervela site press release.

Designed for Deployment Reliability

I was reading an interesting article on Designed for Reliability on The Daily WTF [1]. And then I started reading the comments. I think a lot of people don't get it.

The real story here (at least to me) is that because the system is so reliable, deployment rules are never done and enforced assuming that it might not be. It's a classic problem, because good deployment rules are hard. If you can keep things up and running, you're better off treating the problem as just maintaining the system, but eventually that just plain old fails. It always will. [2]

Ultimately, any system that hasn't had constant redeployment baked in is going to face this exact problem: the machine exploded, OMFG, how do I get it up and running? For one of your larger distributed systems, could you do that particularly easily? More importantly, have you actually tried?

My contention is this: if you don't design your processes such that you always have a reliable deployment story, your super-expensive hardware/system is worthless. Things always, eventually fail. You have to be ready for it.

Annual Power Down

One practice that I think is useful in a variety of ways is the annual power-down. In some areas of London (and in this I don't know whether it's a requirement all the time, but it's happened to me at two different locations, one in the City and one in Westminster Council, so it appears to be widespread), you have to switch off all electrical power to your office block once a year so that it can be tested and inspected. And that's absolute: no UPS, no diesel generators, you really do have to switch the machines off.

Luckily for all, this is scheduled. Beforehand, everybody who maintains a system that's run in the affected data centre scrambles to make sure that the stuff they support is likely to get up and running again without them doing any work. Mostly this is because the power's gonna come back on early Sunday morning, and if they want to party the previous night, they better make damn sure they're not getting woken up at 9:00 am Sunday morning. Startup scripts are checked, internal consistency of the system is checked, checklists are updated, everybody does their part.

It still won't work of course, which is why there are so many people on call and ready to help nudge their particular part of the world back into operation (if nothing else, hardware that runs full-bore 364 days of the year often picks power-up as a time to fail, so make sure you've got a lot of spare disks and power supplies at hand), but the discipline forces you to, at least once a year, make sure that you can get up and running before Monday morning.

Thus if something catastrophic happens, your scripts are on average 6 months behind (and we all know that's worst case: everybody pays attention to the lessons learned for 3 months after the power down, and everybody starts to get ready 3 months before hand, so worst case is you get caught right in the middle well of apathy), but you should still be able to get up and running with a minimum of fuss.

In this case, the annual power down is part of your methodology to ensure that you have a reproducible deployment system.

Can The System Help You?

So let's say you have a system that is designed to make it extremely easy to make runtime changes that would affect your ability to bring the system up and running again in the case of catastrophic failure. What might you do to help fight against this tendency? I think the only solution here is one of two cases:

• Run every change through an external system that keeps track of what is done and potentially allows replay. Note that for this to work, every change (no matter how minor) must go through The System; this essentially prevents developers or support staff from being able to log into the box at all, because everything has to be tracked (though you could maybe get around this these days by just having some type of logging SSH shell or something).
• Have the system self-maintain. In this case, the system maintains in some stable form (backed up disk for example) the exact configuration that it currently has, as well as perhaps the last N changes, so that it can be brought back to a consistent state quickly in the case of catastrophe.
  

The former has the advantage that the system doesn't have to know what's going on, the latter has the advantage that the system can be written to handle just this situation and not have to log unnecessary trivia.

The latter is actually easier than you might think:

• Your VM (in the case of Erlang/OTP) might write out the state of all directly launched processes (as opposed to software launched ones) so that it knows the location and state of all code used to kick off the system;
• Your container (like OSGi) might silently copy all bundles to a WORM location when loading so that there's an accurate record of what was running at any given point of time;
• Your OS (ZFS FTW) might snapshot the state of the system any time something interesting happens so that you can go back and recover gracefully.
  

The Cloud/Utility Computing Take

So given that I think most systems are ultimately moving to Utility Computing (in-house or out in teh Cloudz), does that help you or hurt you in this case? Help, majorly.

Here's a good test: If You're Ready To Run On EC2, You're Done. Whether or not you're going to deploy on EC2, surviving the semantics of EC2 means you're ready.

The rationale here is simple: the semantics of EC2 require that your system be able to deploy at any time. Sure, you can log into a machine and poke around with it, but you're designing for N-way scalability, remember? If you do that, and you don't fork a new AMI with your changes, you're screwed.

Now that means that you have to spend a lot of time working on your AMIs to make sure that for any given production iteration you've got a reproducibly deployable set that represents the state of the universe for that iteration. This is a problem, and I know that it's caused some people trying to integrate with EC2 no end of woe, because it adds a lot of troublesome, time-consuming grunt-work that they're not used to and can't be made to go faster easily. [3]

But by going through that effort, you know that your system is always ready to handle any given node getting cycled at any time. You know that you're consistently ready to deploy.

Think of it as the deployment equivalent of a test-infected culture:

• Testing and Cloud Deployment both institute costs (development costs and configuration costs accordingly)
• Testing lets you develop faster by making sure you're confident that your system is always working
• Cloud Deployment lets you support faster by making sure you can always bring the system up consistently in a crisis

See the parallels here? Up-front costs (development and configuration) lead to long-term reduction in those same costs by making the process consistent and reproducible.

So even if you're not going to deploy on anything other than your local hardware in a non-virtualized environment, pretending that you are can still reap some benefits. [4]

Conclusion

Design your systems and processes for reproducible deployment. If you don't, I guarantee you that the time you discover that your system can't come up without 3 days of hacking will be less than 24 hours before a major regulatory driven deadline.

Footnotes and Tangents

[1] Yes, this is an old article, but I started writing this a long time ago and am only now finishing my writeup. Yes, I have a whole lot of stuff that I think of a take on and never get around to finishing.
[2] In the Erlang/OTP community, this is definitely seen as an advantage: OTP makes it really easy to replace existing code in a running VM without taking anything down, so you can conceivably have a particular VM that's running for years. Read this post on Vinoski's blog for some classic Erlang insight.
[3] Systems like ElasticServer (from CohesiveFT) can help a lot here, because you can do your round-trip testing on VMWare on your workstation, and then just dump out the resulting images in AMI format rather than VMWare format when you're done.
[4] This isn't something that I've actually run a proper evaluation on, aside from designing distributed systems designed to come up and go down hundreds of times at a go for continuous-integration-driven testing. I'd want to run a formal evaluation of the benefits of virtualization configuration as opposed to test-driven rapid configuration to make this more concrete than just a rant.

Powered By Cloud Rundown

I had the privilege of chairing the Powered By Cloud Privacy, Regulation, and Security in Cloud Computing session last week.

While the snow shut down all of London, and the entire schedule had to be juggled to handle the complete failure of London's transport system, Tim Jackson and the rest of the crew did an amazing job putting together a two-day conference over the two worst weather days of the year.

I had the privilege of chairing the session with:

• Miranda Mowbray of HP Bristol
• Paul Massey of K&L Gates
• Brian Reale of Colosa, and
• Richard Sykes of Intellect

We each had our own take on the subject (Paul talked about regulation and international boundaries, Miranda talked about general concerns, Brian talked about licensing, Richard gave some general comments), and I think it was a pretty great conversation for all involved. I could tell by some of the questions at the end from the audience that this particular subject really is critical to the Cloud adoption community, and hopefully we shed some light on the concerns and the state of the art in the space.

My talk in particular was about how regulations, particularly in financial services, don't explicitly state that you can't use any particular Cloud-Sourced service, but, rather, that interpretations of principles-based regulation cause your internal or external auditor to disallow things that you really should be able to do. Basically, you can use Cloud Computing in IT, if you're willing to stand up to the auditors who are just focused on bureaucratic box-checking.

Slides (short, as we each had 10-15 minutes on our own):

Powered By Cloud 2009-02-03

View more presentations from KirkWylie.