More Reuters Fail: Does Nobody Use This Stuff?

To follow on from the last Reuters-related voyage of fail, I'm now looking at another one of their products, OpenDACS, and it's got yet more fail. First, some (not particularly) brief background.

When you buy financial data, the stock/commodity/whatever exchanges that are selling the price information want to make sure that you are controlling that data, so that they can get as much money as they possibly can. Therefore, there are all kinds of tight controls on what you can do with it once you get your hands on it, so that the stock exchanges can know exactly how much to charge you for realtime price data. The Reuters/RMDS form of that is called DACS (Data Access Control System), which allows you to locally administer who in your organization is supposed to have access to what types of data, and then every month you run a report and send it to Reuters and they tell you how much you have to pay based on who was permissioned to see what types of data during the course of that month.

In contrast, the last time I worked with Bloomberg data, to change anything like that you had to go through your Bloomberg Representative, which took longer. So the fact that DACS is a local system is actually not a bad part of their design, because it means you don't have to talk to your sales guy for every small change. I mean, in theory you could hack your local database and screw those nasty exchanges, but it doesn't really seem worth it, so the exchanges trust the reports.

Now let's say you're doing a server-side transformation like calculating implied volatilities from listed equity option prices. In general, many transformations like that you want to share amongst multiple end users, because they can be expensive to perform, but you need to make sure that the user who wants the implied volatility has the rights to the underlying instruments. (In non-financial terms, if you have a server-side process calculating a function F(A, B, C), then in order to be able to get the results of F applied to A, B, and C, you need to have the rights to see A, B, and C in their raw form). This is because the end users aren't actually seeing A, B, or C, they're seeing F(A,B,C), so a server is going to listen to A, B, and C on behalf of all users, and then compute F. But this isn't a use case that exchanges are comfortable with, so you have to prove to them that you really are checking your permissions all the way through: you have to prove to them that you're checking against DACS.

DACS works in terms of Permissionable Entities (PEs for short) and Items. An Item is an individual market instrument that you want to get data on. Cisco is an Item. AT&T is an Item. An individual option on Cisco or AT&T is also an Item. There are lots and lots of items. But most of them are paid for in the same bundle (you never call up the London Stock Exchange and say "hey, I'm only interested in real time updates on Vodafone and BT, so just sell me those"). Those bundles are Permissionable Entities, and Reuters sells packages of Permissionable Entities to you.

Yes, that's all just background.

Here's the juicy fail.

OpenDACS allows you to ask "what are the PEs that user X has rights to?"

OpenDACS also allows you to ask "does user X have access to Item Y?"

(stop me if you know where this is going).

Apparently nobody at Reuters thought to hook up the two. That's right, there's no documented way to either:

• Get the PE applicable for a particular Item; or
• Get all the Items that a PE covers.

And in point of fact this data must be available somewhere, or else Reuters couldn't work at all. Some where in Some database they have this data. They just can't/won't give it to you.

Even better, after contacting Reuters, there's no officially supported way to do this at all, documented or undocumented. Nobody ever thought that you might want to do this.

This means that what should be an M+N problem (one call per user to get his PE list; one call per Item to get the PE) has turned into an M*N problem. Given that these are remote calls, that's not great, and traders aren't going to be happy starting up in the morning to wait 15 minutes while all the items that they need are checked against the database. Which will actually happen for all users at once. What joy!

Because apparently I'm the only person in the history of the world to ever actually use any Reuters API and realize that it fundamentally Fails at its most obvious use case.

Reuters, Milliseconds, and Fail

Let's say you work for Reuters and produce something called Reuters Market Data System (RMDS) to send live market data to your customers, who pay you a lot of money for this. Let's say that for historical reasons, in general, RMDS won't give your customers any precision on ticks other than down to the second, and even that's transmitted as a string (no kidding here; you want to know the actual time that tick was for? Better get ready to parse a string. For every single tick. Seriously.).

That's just not good enough given that your tick delivery system isn't guaranteed to give clients ticks in order, and on the US exchanges you get updates on certain rapidly ticking instruments (like SPY and QQQQ) many times a second. So you decide to add an extra millisecond-precision field to the tick to indicate the millisecond for the tick. This seems like a great idea, so you do it.

Only, unlike every rational human being who would give you an absolute time from Unix epoch as an integer, you don't, because you're special. Like an olympian.

Rather, you provide a double precision floating point number of milliseconds since midnight, and don't actually document that in any way.

Let's cover why the double precision bit is so retarded, shall we?

• I need milliseconds becuase I want more precision. That means that the least significant digits are actually the most important to me, not the least, because I already have other representations for the most significant digits. You're putting them in a data structure that strips away that precision.
• It's not an unexact value. It's an exact value. Floating point representation implies a loss of precision by its very nature.
• Nobody expects it. Milliseconds are integers. Always.

So why in the world do you do this? Simple. You're working with 25-year old technology that can't store 64-bit integers. Therefore, if your data type might be a 64-bit integer, you have to store it as a double precision floating point (because your data structure actually can support 64-bit values, just not integer ones).

But here's the kicker: because they're actually milliseconds since midnight, there are only 86,400,000 possible values. Therefore, you can cleanly put them into a 32-bit integer, which your data structure supports.

Meaning, you're doing this because you suck and you want to make my life miserable.

By the way, so that this is better googleable, the actual fields are SALTIM_MS, QUOTIM_MS, TRDTIM_MS, and TIMCOR_MS