For those of you who have been following my various ramblings on AMQP, specifically on the whole Exchange-Routing semantics (which overflowed into Ben Hood's blog), on Ben and Alexis' request I've started a thread over on rabbitmq-discuss to try to hash things out in a more direct format rather than duelling bloggers. If you think you have something to add, or just want to watch things unfold, I recommend you take a gander over there.
And if nobody takes me up on it, I'll just assume that I've won and the new AMQP spec will have everything I want in it, including a gift certificate to Millie's Cookies. And beer.
Disclaimer: I don't actually use RabbitMQ, my kind hosts just thought it would be as good a place as any for a public MOM-protocol-routing-semantics smackdown.
Monday, October 20, 2008
POSIX Message Queues: Useful, but Limited
Since my previous post on looking for VMS-style Mailbox semantics for more modern operating systems, I've done some research into POSIX Message Queues based on a comment in the entry. I think they're pretty darn cool, but it seems like they might be targetting a different audience than what I think I was looking for.
It also appears that the reason why I've not really seen much (any?) actual code using POSIX or SysV message queues is that support at least for POSIX Message Queues is relatively recent in the Linux kernel timeframe (2.6.6), and thus hasn't really been around for long enough as a system-primitive to find itself worked into much other software.
Disclaimer: It's entirely possible that at this point I'm largely inventing what I'd love from an internal queuing system, and it's quite probable that VMS Mailboxes don't do any of this. From here on, assume this is Kirk's Dream Land rather than "VMS Rulez".
Message Size
The default Linux implementation has the maximum message size set to 8192 bytes. That's not a lot of data to be honest, although it would be more than enough for an bridge between Erlang processes for most data. Unfortunately, this is a kernel option, and so to increase it you need to muck with your runtime kernel parameters (
Maximum Queue Size
The maximum number of messages you can put into the queue before it starts to block the sender is 10 messages. Again, kernel configurable (
I think these two combine to give a pretty clear indication of the type of message semantics they're going for: very fast, small messages. I could see a pretty clear analogue to most tick data in a financial services environment, or audio packets, or Erlang tuples. But this is really clearly designed for super-latency-critical scenarios, and the 10 message maximum is surprisingly small.
In doing some research into this, I came across this comment on a KernelTrap post, which I think gives some pretty clear insight into what's going on here, which is that this is designed for ultra-fast communications between processes (trying to make sure that everything is staying inside the same core and L1 cache if at all possible I would presume).
Persistence
This all makes sense once you realize that POSIX message queues aren't persistent at all: they're like a Priority-queue channel between currently running processes on the same machine, they don't appear to survive machine restarts in any way. This is great as a low-level IPC mechanism, but doesn't really seem to apply to more failsafe long-running messaging scenarios.
When the kernel bounces, all messages and queues disappear.
Initial Reaction
After all this, it appears to me that POSIX Message Queues are really intended to be used for IPC at a very low level of interaction within a particular system, rather than for longer processes with more data. I'd love to see them used for an interaction channel through which a disk-based queue system could be built though.
For that reason, I probably wouldn't attempt to bridge them using AMQP: it would break the semantic model of super-low-latency messaging far too much to try to bridge them over anything slower/higher-latency than Infiniband (though that would be an excellent non-MPI way to use IB, particularly if you have RDMA for the message contents). A persistent queue implementation using disk leveraging POSIX Message Queues for control? That would probably be a much better candidate for AMQP bridging to me.
It also appears that the reason why I've not really seen much (any?) actual code using POSIX or SysV message queues is that support at least for POSIX Message Queues is relatively recent in the Linux kernel timeframe (2.6.6), and thus hasn't really been around for long enough as a system-primitive to find itself worked into much other software.
Disclaimer: It's entirely possible that at this point I'm largely inventing what I'd love from an internal queuing system, and it's quite probable that VMS Mailboxes don't do any of this. From here on, assume this is Kirk's Dream Land rather than "VMS Rulez".
Message Size
The default Linux implementation has the maximum message size set to 8192 bytes. That's not a lot of data to be honest, although it would be more than enough for an bridge between Erlang processes for most data. Unfortunately, this is a kernel option, and so to increase it you need to muck with your runtime kernel parameters (
/proc/sys/fs/mqueue/msgsize_max).Maximum Queue Size
The maximum number of messages you can put into the queue before it starts to block the sender is 10 messages. Again, kernel configurable (
/proc/sys/fs/mqueue/msg_max), but 10 messages is pretty darn small.I think these two combine to give a pretty clear indication of the type of message semantics they're going for: very fast, small messages. I could see a pretty clear analogue to most tick data in a financial services environment, or audio packets, or Erlang tuples. But this is really clearly designed for super-latency-critical scenarios, and the 10 message maximum is surprisingly small.
In doing some research into this, I came across this comment on a KernelTrap post, which I think gives some pretty clear insight into what's going on here, which is that this is designed for ultra-fast communications between processes (trying to make sure that everything is staying inside the same core and L1 cache if at all possible I would presume).
Persistence
This all makes sense once you realize that POSIX message queues aren't persistent at all: they're like a Priority-queue channel between currently running processes on the same machine, they don't appear to survive machine restarts in any way. This is great as a low-level IPC mechanism, but doesn't really seem to apply to more failsafe long-running messaging scenarios.
When the kernel bounces, all messages and queues disappear.
Initial Reaction
After all this, it appears to me that POSIX Message Queues are really intended to be used for IPC at a very low level of interaction within a particular system, rather than for longer processes with more data. I'd love to see them used for an interaction channel through which a disk-based queue system could be built though.
For that reason, I probably wouldn't attempt to bridge them using AMQP: it would break the semantic model of super-low-latency messaging far too much to try to bridge them over anything slower/higher-latency than Infiniband (though that would be an excellent non-MPI way to use IB, particularly if you have RDMA for the message contents). A persistent queue implementation using disk leveraging POSIX Message Queues for control? That would probably be a much better candidate for AMQP bridging to me.
Friday, October 17, 2008
RabbitMQ talk at LShift Last Night
I attended the AMQP+RabbitMQ talk last night at LShift, and it was a pretty good time. Apparently the talk was largely a rehash of the talk that Alexis & company did at Google UK a few weeks ago, and was pretty interesting.
The basic pattern was:
Specifically, in case you haven't been looking at it yet, Erlang gives you a natural model to represent a lot of concepts in building large asynchronous systems. Since there's a lot of processing that goes on in message-oriented systems which is by its very nature asynchronous, it's pretty easy to map that onto Erlang in a pretty trivial way, which drastically reduces the amount of code that you need to write yourself.
Since the major concepts you're working with (Exchanges, Queues, Sockets) map directly to low-level Erlang actors^H^H^H^H^H^Hprocesses, Erlang gives you a really easy way to automatically introspect the state of those things, since they're being mapped to processes. This for a Java guy was pretty cool, because essentially you can introspect into the state of a running Erlang system and see in one fell swoop a snapshot of a process which shows you the equivalent of (for Java):
And then, down t' pub.
One of the most interesting things to me was that quite a few of the people there really weren't part of the converted trying to hear things to make them feel all happy about themselves. Rather, there were a lot of people who aren't currently using MOM systems (or have basically done stuff that's core to MOM themselves, because the basic concepts are so intuitive that if you don't know what's out there, you'll end up writing your own half-baked MOM implementation yourself to do almost any non-trivial asynchronous system) who are eager to look into them now.
That's pretty nice to hear, and I think is one of the major medium-term benefits of AMQP: expansion of the overall MOM space from one dominated by MQSeries which people use because they have to, not because they want to, to one where people actively choose to use MOM because it's seen as a benefit rather than a hurdle. So AMQP really will involve growing the space, rather than cannibalizing it. And that to me is a great thing.
The basic pattern was:
- Talk about MOM (messaging is good) and AMQP (AMQP is also good)
- Talk about RabbitMQ (RabbitMQ is good) and Erlang/OTP (they're also good and ideal for implementation of an AMQP broker)
- Demonstration of some cool stuff you can do tying it all together
Specifically, in case you haven't been looking at it yet, Erlang gives you a natural model to represent a lot of concepts in building large asynchronous systems. Since there's a lot of processing that goes on in message-oriented systems which is by its very nature asynchronous, it's pretty easy to map that onto Erlang in a pretty trivial way, which drastically reduces the amount of code that you need to write yourself.
Since the major concepts you're working with (Exchanges, Queues, Sockets) map directly to low-level Erlang actors^H^H^H^H^H^Hprocesses, Erlang gives you a really easy way to automatically introspect the state of those things, since they're being mapped to processes. This for a Java guy was pretty cool, because essentially you can introspect into the state of a running Erlang system and see in one fell swoop a snapshot of a process which shows you the equivalent of (for Java):
- The stack trace
- Values of all local variables across the stack
- Values of all elements in the heap directly addressible from the bound variables
And then, down t' pub.
One of the most interesting things to me was that quite a few of the people there really weren't part of the converted trying to hear things to make them feel all happy about themselves. Rather, there were a lot of people who aren't currently using MOM systems (or have basically done stuff that's core to MOM themselves, because the basic concepts are so intuitive that if you don't know what's out there, you'll end up writing your own half-baked MOM implementation yourself to do almost any non-trivial asynchronous system) who are eager to look into them now.
That's pretty nice to hear, and I think is one of the major medium-term benefits of AMQP: expansion of the overall MOM space from one dominated by MQSeries which people use because they have to, not because they want to, to one where people actively choose to use MOM because it's seen as a benefit rather than a hurdle. So AMQP really will involve growing the space, rather than cannibalizing it. And that to me is a great thing.
Monday, October 06, 2008
SonicMQ: Constructing New MessagePublishers or not?
Just as a quick follow-on to my previous mini-benchmark, I've now tried the same basic benchmark but testing publishing each message to its own Topic, and tested creating a new
For the lazy and link-click-shy, this is a tiny benchmark of how to publish small messages as fast as possible against SonicMQ 7.5.1 using the 7.5 JMS driver.
There is a difference, but a pretty small one.
So there's a minor performance enhancement (1.5%), but it's pretty small, and judging from network traffic, it's not actually hitting the wire for the new
MessagePublisher per message, versus creating a unbound (Session.createProducer(null)) MessageProducer and using that to send all messages (producer.send(destination, message)).For the lazy and link-click-shy, this is a tiny benchmark of how to publish small messages as fast as possible against SonicMQ 7.5.1 using the 7.5 JMS driver.
There is a difference, but a pretty small one.
- Creating a single
MessageProducerand usingproducer.send(destination, message)gave me 32352mps sent. - Creating a new
MessageProducerfor each new message/topic gave me 31847mps sent. - This was definitely reproducible, and I ran each test many times to make sure I got a pretty consistent result.
So there's a minor performance enhancement (1.5%), but it's pretty small, and judging from network traffic, it's not actually hitting the wire for the new
MessageProducer, so it's entirely a local driver optimization.
SonicMQ New Sessions, Publication, and Performance
I was trying to debug some code today and ended up running a performance benchmark of fast message publishing, and thought it might be useful to the SonicMQ community. Here goes.
Scenario
You have a system that is publishing small messages to the same topic as fast as it possibly can.
Question
What is the impact of creating new
Disclaimer
Please don't consider this a generic SonicMQ benchmark in any form. The code and infrastructure were definitely not setup for a proper benchmark, and I'm sure my employer signed some disclaimer that we wouldn't perform one anyway. This is just an example of how coding practices can affect your performance. Blah Blah Blah don't sue me Blah Blah Blah.
Benchmark Description
Minutae
Results
More interestingly, if you look at the network utilization graph on the Windows machine acting as the test driver, you can see exactly how the performance ramped up:
Conclusion
There appears to be some definite network latency in setting up the
Therefore, you can quite clearly conclude that you should avoid creating unnecessary Sessions and just hold them open for a publishing scenario.
Scenario
You have a system that is publishing small messages to the same topic as fast as it possibly can.
Question
What is the impact of creating new
Sessions on the publishing performance?Disclaimer
Please don't consider this a generic SonicMQ benchmark in any form. The code and infrastructure were definitely not setup for a proper benchmark, and I'm sure my employer signed some disclaimer that we wouldn't perform one anyway. This is just an example of how coding practices can affect your performance. Blah Blah Blah don't sue me Blah Blah Blah.
Benchmark Description
- Create a JMS
Connectionto your SonicMQ broker. - In a tight, single-threaded loop, construct a new
BytesMessage, bung 10 bytes of data into it, and send the message. - Every N messages (where N is varied), shutdown the
Sessionand construct a newSession,Destination, andMessageProducer.
Minutae
- SonicMQ broker was running on a 2-socket dual-core (total of 4 cores) Opteron machine running Solaris 10 x86
- Broker running 7.5.1, 7.5.0 JMS client
- Client was running Windows XP
- JDK 1.5.0_13 used on both broker and client
- 100Mbps available between client and server (broker has 1Gbps available; I'm not allowed to thrash this broker to death during business hours), going through at least 3 switch hops (but no router)
- All messages sent non-persistent, with no consumers whatsoever
Results
New Session Every # Messages | Msg/sec |
|---|---|
| 10 | 4400 |
| 100 | 18653 |
| 1000 | 31113 |
| 10000 | 33074 |
| 100000 | 33507 |
More interestingly, if you look at the network utilization graph on the Windows machine acting as the test driver, you can see exactly how the performance ramped up:
Conclusion
There appears to be some definite network latency in setting up the
Session, MessageProducer, and Destination, but this disappears beyond the every-1000-messages mark.Therefore, you can quite clearly conclude that you should avoid creating unnecessary Sessions and just hold them open for a publishing scenario.
Wednesday, October 01, 2008
IPC: Where My Queues At, Yo?
Updated 2008-10-20: Adrian Found Where My Queues At. Unbeknownst to me, they were there (somewhat all along) in POSIX Message Queues, more in the comments. I've changed this to be part of LazyWeb, because I really didn't know that they existed and haven't actually seen anybody using them. M4D props to Adrian.
Updated 2008-10-20 #2: POSIX Message Queues are close, but not quite it. I'll be doing another post on this, but turns out POSIX Message Queues aren't quite what I think mailbox semantics are, and some differences turn out to be key. Also, they've only been around since Linux kernel 2.6.6, which would explain why I've not seen them in more widespread use.
Not that long ago, my firm was interviewing someone from the Czech Republic who had been working in Frankfurt for the Deutsche Boerse, and some interesting stuff came out of that. Not the least interesting thing was that their entire infrastructure was still based on OpenVMS, and it was working extremely well. The secret? Queues. Or, rather, Mailboxes.
What a lot of people know about VMS, aside from the fact that it's Old and Not UNIX, is that it spawned Windows NT (WNT == VMS+1). What they don't realize is that it's actually quite a reasonable operating system that's kinda gotten a bad name because of the WNT thing, and because it's Not UNIX, but I want to address the thing that it's really gotten right, and that's queuing as an intrinsic form of IPC.
As I understood from the interviews, Deutsche Boerse uses this pretty extensively to distribute work: work is pulled from mailboxes, posted to other mailboxes, in a pretty standard modern MOM pattern. The recipients of messages may be on the same machine, or on another machine. They don't know; they don't care. Sounds like what you'd use an MOM system for, right? Only they don't use an MOM system, they use their operating system.
A Mailbox in VMS essentially forms the basis of a fully asynchronous queuing system. Applications can define mailboxes, to which you can publish messages, and the recipient will be notified at some point in the future that a message is available. And because VMS has a lot of clustering facilities, mailboxes can either be on the local machine or on another machine, and because it's all fully guaranteed delivery (within reasons of course), you have full fire-and-forget semantics.
Contrast all this with your UNIX forms of IPC: Pipes, Files, Signals. What do the former two have in common? They're either synchronous or near-enough in that they involve polling. Signals may be asynchronous, but they can't really carry a payload: they're just a number. These are enough to form the basis for almost all forms of common IPC, except for fully asynchronous ones. What do you do in a UNIX IPC model if you want to send a message and at some point in the future make sure that something else picks it up? Put it in a spool file? That seems pretty clunky, even today.
Sure, you can expand the UNIX model to involve asynchronous IO, but that's really just an asynch vaneer on top of traditional synchronous IPC. It doesn't change the fundamental basis that your basic units of system-system IPC are fundamentally synchronous. This all smells funny.
Given that there are a lot of AMQP people who read my blog apparently, where's the topicality? How about this: why is it necessary that I even have to use something like broker-based MOM when the operating system could realistically do this for me? Why is this all so resolutely in user space when operating systems were doing this decades ago? Why do I need to communicate using sockets (another system-system synchronous IPC system) to another process to do basic message-based IPC?
Why can't I take all the various implementations of this pretty basic concept (fire-and-forget, guaranteed messaging, asynchronous notification, single consumer delivery) and pull them out of their various programming-language (Scala, Erlang) and application-semantic specific models (JMS, WCF), and pull them up a level? Why can't I get this out of my operating system? Why can't we just break out of the UNIX model of decades ago?
Maybe we've accepted that the micro-kernel model of operating systems has really won. And by that, I don't mean the Mach-model of operating systems, but, rather, the model that system-level APIs are pretty much fixed in stone by history, and won't be expanded (yes, they might be expanded in the particulars, but they're not going to be significantly expanded in the general concepts that people are willing to defer to the kernel to do). People do everything they have to in user-space.
But something to me is wanting. I think this really simple form of IPC, which is so insanely beautiful that people are willing to program in actor-based concurrency systems like Erlang just to get it, really should be given a second thought. Maybe not in kernel space, but definitely as a standard that you can assume will be around on any system on which you program.
And the interviewee? A (hopefully) happy employee of my firm. We hire quality when we can find it.
Updated 2008-10-20 #2: POSIX Message Queues are close, but not quite it. I'll be doing another post on this, but turns out POSIX Message Queues aren't quite what I think mailbox semantics are, and some differences turn out to be key. Also, they've only been around since Linux kernel 2.6.6, which would explain why I've not seen them in more widespread use.
Not that long ago, my firm was interviewing someone from the Czech Republic who had been working in Frankfurt for the Deutsche Boerse, and some interesting stuff came out of that. Not the least interesting thing was that their entire infrastructure was still based on OpenVMS, and it was working extremely well. The secret? Queues. Or, rather, Mailboxes.
What a lot of people know about VMS, aside from the fact that it's Old and Not UNIX, is that it spawned Windows NT (WNT == VMS+1). What they don't realize is that it's actually quite a reasonable operating system that's kinda gotten a bad name because of the WNT thing, and because it's Not UNIX, but I want to address the thing that it's really gotten right, and that's queuing as an intrinsic form of IPC.
As I understood from the interviews, Deutsche Boerse uses this pretty extensively to distribute work: work is pulled from mailboxes, posted to other mailboxes, in a pretty standard modern MOM pattern. The recipients of messages may be on the same machine, or on another machine. They don't know; they don't care. Sounds like what you'd use an MOM system for, right? Only they don't use an MOM system, they use their operating system.
A Mailbox in VMS essentially forms the basis of a fully asynchronous queuing system. Applications can define mailboxes, to which you can publish messages, and the recipient will be notified at some point in the future that a message is available. And because VMS has a lot of clustering facilities, mailboxes can either be on the local machine or on another machine, and because it's all fully guaranteed delivery (within reasons of course), you have full fire-and-forget semantics.
Contrast all this with your UNIX forms of IPC: Pipes, Files, Signals. What do the former two have in common? They're either synchronous or near-enough in that they involve polling. Signals may be asynchronous, but they can't really carry a payload: they're just a number. These are enough to form the basis for almost all forms of common IPC, except for fully asynchronous ones. What do you do in a UNIX IPC model if you want to send a message and at some point in the future make sure that something else picks it up? Put it in a spool file? That seems pretty clunky, even today.
Sure, you can expand the UNIX model to involve asynchronous IO, but that's really just an asynch vaneer on top of traditional synchronous IPC. It doesn't change the fundamental basis that your basic units of system-system IPC are fundamentally synchronous. This all smells funny.
Given that there are a lot of AMQP people who read my blog apparently, where's the topicality? How about this: why is it necessary that I even have to use something like broker-based MOM when the operating system could realistically do this for me? Why is this all so resolutely in user space when operating systems were doing this decades ago? Why do I need to communicate using sockets (another system-system synchronous IPC system) to another process to do basic message-based IPC?
Why can't I take all the various implementations of this pretty basic concept (fire-and-forget, guaranteed messaging, asynchronous notification, single consumer delivery) and pull them out of their various programming-language (Scala, Erlang) and application-semantic specific models (JMS, WCF), and pull them up a level? Why can't I get this out of my operating system? Why can't we just break out of the UNIX model of decades ago?
Maybe we've accepted that the micro-kernel model of operating systems has really won. And by that, I don't mean the Mach-model of operating systems, but, rather, the model that system-level APIs are pretty much fixed in stone by history, and won't be expanded (yes, they might be expanded in the particulars, but they're not going to be significantly expanded in the general concepts that people are willing to defer to the kernel to do). People do everything they have to in user-space.
But something to me is wanting. I think this really simple form of IPC, which is so insanely beautiful that people are willing to program in actor-based concurrency systems like Erlang just to get it, really should be given a second thought. Maybe not in kernel space, but definitely as a standard that you can assume will be around on any system on which you program.
And the interviewee? A (hopefully) happy employee of my firm. We hire quality when we can find it.
Tuesday, September 30, 2008
Interop News on AMQP: Huh?
Sorry, I don't usually call out at other writers, but this article makes utterly no sense whatsoever. I mean, m4d pr0p2 for giving a call out to AMQP, but WTF?
Some random stuff in there:
AMQP does a similar thing at the network protocol layer: it defines a semantic layer that assumes a centralized broker-based middleware system and wraps a network protocol around that. It really competes more with the JMS pure-plays than either IBM MQ or Rendezvous. But I don't think it as a protocol provides a compelling reason for people who actually know and want to be on a distributed system like Rendezvous to shift, and people on IBM MQ are there because they have to be.
None of this is to say that the core of his message isn't correct, which is that AMQP is a great concept for a variety of reasons, but his introduction is setting himself up for something he didn't need to.
I still stand by my original point:
Some random stuff in there:
- Right off the bat, the author (Jeff Gould) calls out Rendezvous as a guaranteed message delivery system. But that's actually the opposite of Rendezvous's raison d'etre. Rendezvous is best as a super-low-latency tick distribution system where its default best-effort semantics are good enough, not a guaranteed message delivery system. That's because it's an inherently distributed system. Yes, there are advanced features of Rendezvous that make it nearly guaranteed, but they're pretty ugly to be fair.
- He conflates bridging with heterogenous client connectivity. In general, the JMS API-driven approach that most vendors do with interoperability actually is good enough, because there's going to be something that has to bridge them no matter what, and many different JMS-based systems allow for automatic bridging from a broker to another system quite easily. Just because the code wasn't all written by one vendor (the broker by Vendor A, the client library to interact with the other broker by Vendor B), it isn't a massive problem.
Furthermore, most of the time when you have such an interop problem, someone has to write some bloody code to actually implement all the tiny little differences between the two systems (such as ID issues: if both companies are using the same software, there's probably a conflicting ID namespace somewhere that requires translation). That involves a bridge.
Imagining that there will be a world where two companies running two ever-so-slightly different versions of the same software package will be able to just exchange messages willy-nilly and all will be well and the programmers can all just go home is rubbish. - He assumes that AMQP is a replacement for the two major products that he directly calls out to: IBM MQ (just calling it that since I will always think of it as MQSeries) and Rendezvous. And then refers to Sonic. But here's the thing: none of the three has almost anything to do with each other, and each is used in pretty different spaces.
- IBM MQ is largely used in older systems (think flat-out legacy). It's big, it's slow, it runs on mainframes, and I really don't know of a single person doing work similar to me that's actually deployed it out of choice in ages. And you know what? AMQP will do absolutely 0 to that market. It's very legacy and very entrenched and IBM has absolutely no incentive to change it and never will.
- Rendezvous is a completely decentralized MOM system ideally suited to low-latency best-effort systems (tick distribution, enterprise system monitoring). It does something completely different to IBM MQ, and you would never replace it with MQ. And it has competitors (29 West is my favorite to give a call-out to).
- SonicMQ and other pure-play JMS vendors (Fiorano, Tibco EMS, ActiveMQ, ...) have a broker-based model with guaranteed delivery and best-effort built in. They're both a superset of the IBM MQ and Rendezvous systems, with some special sauce out of having at least a marginally intelligent approach to broker-based middleware worked in.
AMQP does a similar thing at the network protocol layer: it defines a semantic layer that assumes a centralized broker-based middleware system and wraps a network protocol around that. It really competes more with the JMS pure-plays than either IBM MQ or Rendezvous. But I don't think it as a protocol provides a compelling reason for people who actually know and want to be on a distributed system like Rendezvous to shift, and people on IBM MQ are there because they have to be.
None of this is to say that the core of his message isn't correct, which is that AMQP is a great concept for a variety of reasons, but his introduction is setting himself up for something he didn't need to.
I still stand by my original point:
- Vendors cannot support every possible programming language, OS, chip, or compiler with a closed/proprietary client driver.
- Programming language experts are superior at doing that and can provide open source versions.
- This naturally and inevitably leads to possible interoperability gains.
Subscribe to:
Posts (Atom)
Posts
