Thursday, August 28, 2008

Reuters, Milliseconds, and Fail

Let's say you work for Reuters and produce something called Reuters Market Data System (RMDS) to send live market data to your customers, who pay you a lot of money for this. Let's say that for historical reasons, in general, RMDS won't give your customers any precision on ticks other than down to the second, and even that's transmitted as a string (no kidding here; you want to know the actual time that tick was for? Better get ready to parse a string. For every single tick. Seriously.).

That's just not good enough given that your tick delivery system isn't guaranteed to give clients ticks in order, and on the US exchanges you get updates on certain rapidly ticking instruments (like SPY and QQQQ) many times a second. So you decide to add an extra millisecond-precision field to the tick to indicate the millisecond for the tick. This seems like a great idea, so you do it.

Only, unlike every rational human being who would give you an absolute time from Unix epoch as an integer, you don't, because you're special. Like an olympian.

Rather, you provide a double precision floating point number of milliseconds since midnight, and don't actually document that in any way.

Let's cover why the double precision bit is so retarded, shall we?
  • I need milliseconds becuase I want more precision. That means that the least significant digits are actually the most important to me, not the least, because I already have other representations for the most significant digits. You're putting them in a data structure that strips away that precision.
  • It's not an unexact value. It's an exact value. Floating point representation implies a loss of precision by its very nature.
  • Nobody expects it. Milliseconds are integers. Always.
So why in the world do you do this? Simple. You're working with 25-year old technology that can't store 64-bit integers. Therefore, if your data type might be a 64-bit integer, you have to store it as a double precision floating point (because your data structure actually can support 64-bit values, just not integer ones).

But here's the kicker: because they're actually milliseconds since midnight, there are only 86,400,000 possible values. Therefore, you can cleanly put them into a 32-bit integer, which your data structure supports.

Meaning, you're doing this because you suck and you want to make my life miserable.

By the way, so that this is better googleable, the actual fields are SALTIM_MS, QUOTIM_MS, TRDTIM_MS, and TIMCOR_MS
blog comments powered by Disqus