Dingbat Kabuki and other structural market hacks

When I first started puppetmaster trading, one of my dearest friends, a Yale-educated economist and professor of same, asked me an important question.  He asked:

In the markets, there are always ‘insiders’ who have the ability to trade on knowledge that you can’t know or with an advantage that you can’t have.  How are you going to compete with these players?

I provided a variety of answers, but at the time my conception of the universe of people with both inside knowledge and the ability to trade on it was limited to cases like that of Mr Rajaratnam.  I believed that cases like these were constrained by clear laws that were duly surveilled and prosecuted by the appropriate authorities.  The problem seemed like a very real one, but constrained in size and not essential to my enterprise.  I still hope that my belief of the time was true, but since then I’ve certainly understood that there’s more than one way to hack the market.

For some, a market hack might consist of some kind of simple (or complex) algorithm(s) applied to some set of markets.  But this really isn’t a hack so much as it’s a trading strategy – like many that have long existed – only that it’s now implemented in software where originally it would have been implemented in wetware.  While implementing trading strategies in software does open up new vistas in terms of the kinds of strategies that you can look to implement – computers are faster than people by a noteworthy amount in many tasks – but, for the most part, you’re really still just trading and when you take on positions, you are still bearing risk.  You might be ‘hacking’ but it’s really not a market hack as I’ve come to appreciate.

hack the market structure

Another area where my perspective has changed substantively since those halcyon days of ‘05 is my appreciation for market (micro) structure.  In futures, market structure is pretty plain as most contracts are effectively monopolies run by their listing exchanges.  There are a few cases of instruments which are tradable across markets, but a rich market microstructure is just not a core identifying characteristic for futures like it is in the fragmented and incredibly dynamic and quickly developing world of equities.  So, while there are some dozens of futures exchanges scattered about the globe, as a futures trader, routing algorithms just don’t enter the picture except in limited and relatively simple cases.

If I have a view on or a hedging need for interest rates that I see effectively expressed through futures, then I go to e.g. the cme and I don’t need to worry my pretty head about getting a great execution because the exchange is in one place and it’s all lit up and filled to the brim with liquidity.  Easy peasy.  But if I have a view on, say, General Electric that I want to express with an equity position and I’m trading in sufficient size or sufficiently close to the market that I really want to ensure best execution, then I might find the need to look around some dozens of lit exchanges and maybe even ping about in a dark pool or twelve.  This is decidedly not easy peasy and is one of the reasons wall st has an insatiable hunger for propeller heads with advanced degrees in seemingly unrelated fields.

Last summer’s brouhaha about ‘flash’ orders first illuminated for many a rich ground for genuine market hacks: the incredibly dynamic terrain of equity market micro structure which changes almost daily with the emergence of new exchanges, ATSes, order types, rebates and pricing structures and all of the many other critical minutia that differentiate the many venues.  Some of these undoubtedly provide meaningful and important services; contrary to populist inclination, dark pools are largely a defense against frothy HF trading strategies.

But the concern (if perhaps not all of the attention) is merited.   Equity microstructural ‘rules’ change so quickly that even independent and upright regulators could hardly be expected to keep up.  This is clearly fertile ground for genuine – if perhaps fleeting – market hacks.

steganography and hacking the market

What characteristics make equity market structure a fertile ground for these kinds of genuine market hacks which I’m describing?  I think the main answer is to be found in the ancient art of steganography – hiding in plain site.  Everything about equity market microstructure is public.  All of the rules for each of the venues are available to the people who might make use of them.  Unraveling what they imply about where you should be executing for any given order is the tricky bit which is hardly revealed through a simple reading of all of the various rules.

Where else do we see this phenomenon?  The US Tax code comes to mind (sort of like the iPhone: “there’s a loophole for that”).  The 2,000 page health care bill comes to mind.  Everybody seems to care about health care, but who’s actually read that bill?  Who could?

Totally public yet it may as well be encrypted.  Steganography.

Elizabeth Warren has also made this observation in the context of contracts made between banks and their retail clientele.  Noted scholar Scott Adams dubbed it a ‘confusopoly.’

a group of companies with similar products who intentionally confuse customers instead of competing on price

So, could people hack the tax code?  Hank and I think so.  So do my friends in the lobbying business.  How about all those recently converted bank holding companies – hack much?  In my last post we read one insider’s view that Madoff had effectively hacked the SEC… perhaps regulatory organizations themselves are also hackable instruments!  How about that health care bill?  …

misdirecting the hack

A lot of attention has focused recently on exchange traded markets and people are up in arms about HF traders and proprietary trading.  I’ve argued before that this appears to be a great ploy to take attention away from the real issues at hand; in the worst case, HFT improprieties might account for no more than 1% of the money disappeared in the last few years as part of the so-called ‘credit crisis’.  It’s easy for non-finance-professionals not to understand that the big bad wolves of the credit crisis essentially all happened off market in essentially unregulated multi-trillion dollar otc markets of ingeniously engineered structured products. These are the real market hackers.  Exchange traded instruments had effectively nothing to do with our current circumstances but remain a convenient scapegoat.

Some argue persuasively that there are fundamental, structural market hacks at the very foundations of our financial system.  I wouldn’t know.  But it is the kind of thing one might think about while watching the highly stylized performance of our leaders last night.

dingbat kabuki and the transparent market hack

To my knowledge, Cal’s (go bears!) Professor Brad DeLong coined the term ‘dingbat kabuki’ back in 2005 in shrill response to a Washington Post article.  He reprised the term this week in response to the latest bit of macroeconomic genius out of washington.  What an inspired phrase.  Yesterday at 2:30 and last night at 9pm the markets voiced their applause and cheered on the performance.

Indeed, it is masterful.

–

A technical note about this post.  While writing it, I seem to have accidentally published it at some point and then realized the error and ‘unpublished’ it.  I apologize if this had any untoward effects on you or your RSS reader.

One of the nicest things about the holiday season (Happy New Year, btw) is that it provides a lovely opportunity to spend some quality time with a project that’s a bit more exploratory than might be meaningfully undertaken while trading in lively markets.

A number of months ago, I mentioned using HDF5 to manage tick data as RDBMSes just aren’t up to the task and specialized Tick DBs are absurdly expensive.  While I’d spent some time exploring this idea through the fall, I never had a discrete chunk of time to really explore the technology beyond determing that its Java interfaces weren’t production-worthy.  This meant that we’d have to drop into C to access the functionality we’re interested in and that we’d have to come up with our own bridge out into Java for access by StratBox while StratCloud could access it directly.

Below, I describe what I’ve learned through my holiday geek-spelunking-trek including some timings on various configurable characteristics of HDF5 (e.g., compression and “chunking”).

After spending some time looking at the java interfaces to HDF5, I determined it wasn’t up to snuff.  Why?  Primarily because no-one seems to use it, it lags the main api from a versioning standpoint and it appears to be more-or-less impossible to build from source.  Looking a bit more carefully, it seems to have been written by one (undoubtedly talented and well-meaning) individual who isn’t familiar with java.  (The most egregious example was to use a javax.swing.tree.TreeNode as the base class for a key model object…)

I then spent some time looking at the native api and the underlying object model it exposes.  The model is both powerful and pretty low-level.  They’ve implemented many of the goodies of a file system including groups (“directories”), datasets (“files” or in RDBMS-land, “tables”) and a variety of nice linking mechanisms as well as attributes which might index or otherwise annotate data.  There’s also powerful, extensible I/O options which I didn’t much study beyond compression and “chunking.”

The library is provided with two “first-class” APIs – in C and Fortran – a secondary API in C++ and then the Java interface I mentioned.  Others have written interfaces for other languages, most notably the much-lauded PyTables implementation for Python which is used by many in conjunction with the popular NumPy package.

Given this spread of implementation languages I chose C and determined that I’d steal a page from the talented crew behind QuantLib and use SWIG to expose relevant functionality into Java.  This has proven to be a splendid choice for my needs.

Having gotten this far, I started examining how I’d represent market data with hdf5 and came up with two broadly opposed approaches.  In order to gain some insights from those more experienced, I sent the below problem statement / inquiry to the main HDF5 mailing list:

A description of the data and its use

The data is all timestamped financial streams of “tick” data.  Each record is small (a few hundred bytes at the most), but there are many – in a day you may see many hundred million to a few billion.  Each record is naturally partitioned by instrument (eg, “microsoft”, “ibm”, “dec crude”, etc).  There are less than 30K instruments in the universe I might care about.

I (more or less) don’t care how long it takes to construct the h5 files/structures as it will be performed offline and the only critical query I care about is something like:

“Get ticks for instruments {i1…in} from time t1 to time t2 ordered by time, instr”.

That is, I need to be able to “replay” a subset of the instruments within the data store over some period of time.  But I really care that this be as fast as possible.

Questions

0.  Am I barking up the wrong tree?  Is HDF5 an appropriate technology for the use I’ve described?

1. Given the size/volume of the data, my thought is to partition h5 files by day.  Uncompressed, the files will be on the order of ~25G.  Does this sound reasonable?  What are the key factors impacting this decision from an hdf5 perspective?

Two alternative models come immediately to mind: one big table (OBT) per day ordered by instrument and then time, or one table per instrument (OTPI) ordered by time.  My current inclination is OTPI as it seems more manageable assuming the overhead of so many tables isn’t an issue.

2a.  Are there other, better models you suggest I investigate?

2b.  With the OBT, I’d need to be able “index into” the table to identify the beginning of each instrument’s section (at least).  How would you recommend doing this?  It seems possible to do this with references or perhaps a separate table with numerical indices into the main table.  Any pros/cons/alternatives to these approaches?

2c.  With the OTPI, I’d need to have many tables (at most ~30K) per file.  Is this an issue?

2d. For both models, I’d need to be able to merge sorted sets of h5 data into one sorted set as quickly as possible.  Is there any hdf5 support for doing such a thing or external libraries created for this purpose?

3. What impact on retrieval/querying should I expect to see with varying levels of compression?

4. Any suggestions on chunksizes for this application?

I was fortunate to receive some excellent responses to my query, including from Francesc Alted, the gracious author of the PyTables library, and from a gentleman who’d implemented similar functionality for his own trading environment.  Interestingly, both approaches – OBT and OTPI – were championed.  It seems that OTPI is probably to be preferred if the number of instruments/tables to be stored isn’t excessive (perhaps below 10K though I can’t quantify this) and the frequency of update is significant.  OTPI is easier to implement as it means you can rely more upon the infrastructure provided by HDF5.  OBT instead seems more scalable as you incur less overhead (and goodies) with the one table, though you pay for this by having to implement your own indexing logic.

Given the divergent advice and my own lack of hands-on familiarity with the C library, I decided to try both approaches on a prototype.  Instead of looking at vast amounts of tick data, I’d try both approaches on a smaller store (~1G with ~7K instruments) of OHLCV data.

By far, the easier to implement is the OTPI approach.  However, even with this relatively small amount of data, the difference in write performance and file size was substantial.  Clearly, expanding this to the scale of tick data wasn’t going to yield sufficiently performant results.  I focused on the OBT approach.

—

Given the length of this post and keeping in mind that the holiday season isn’t over just yet (about ten hours remaining as I write this!), I’m going to stop writing now and continue with the remainder of my implementation and findings in a follow-up post later this week..

While Carl Sagan’s famous formulation introduced a generation to the vastness of the cosmos, more recent history suggests that his memorable term might now be more aptly applied to financial extents: our deficits and debts, perhaps, to the economically or politically minded. But for those of us with the markets on our mind, the term has to evoke the enormity of the data we create and must manage every day. We’ve recently been working with the NYSE’s TAQ data in an effort to integrate it into StratBox’s back-testing and optimization capabilities. And the enormity of the data is really just staggering.

Each day, the NYSE publishes all of the day’s quotes and trades as well as some reference data. Compressed, the data will just about fit onto a DVD. For one day. A DVD. Compressed. It’s really mind-boggling. A year of the stuff, uncompressed, will require over a petabyte of storage. Over 1,125,899,906,842,624 bytes. And that’s just the US Equities markets. You want options data, too? I hope your uncle is named EMC, because just managing the data is going to be a challenge…

“Information about money has become almost as important as money itself.” — Walter Wriston, former Chairman of Citicorp

The enormity and profile of market data far exceeds the capacity of traditional RDBMSes. While RDBMSes continue to expand their usable capacity – we have used partitioned tables with nearly a billion rows of market data which have performed astonishingly well – they simply can’t deal with the kinds of quote volumes modern markets are generating daily. This has spawned a host of specialized timeseries database products, like the grandaddy: Sungard’s FAME which I’d used back in the 90’s to write programs to calculate bond indices at JPM, to more recent offerings like Vhayu and Kdb+. These timeseries oriented data products undoubtedly have many distinguishing characteristics and features, but they share one immutable characteristic: they are unbelievably expensive – in some cases a single developer seat costs in the high 6-figures for an annual license.

Thus, while no doubt missing out on some of their high-end features and niceties, we’ve decided to seek solutions from some of the original purveyors of petabyte-scaled data: NASA and the NCSA through their HDF5 system. Designed to support vast scientific data stores and boasting sophisticated capabilities in support of parallel computing environments, it should be possible to get comparable performance to some of the high-end specialized finance products without the sticker shock. Indeed, it’s potentially another example of free software providing a meaningful contribution to finance.

In researching cost-efficient and highly parallel hardware solutions to pair with our emergent data solution, I’ve come to realize that open-source is expanding its reach into the hardware sphere.

This guy shares his experience and “recipe” for building a powerful and unique rendering cluster inside an IKEA filing cabinet. It’s admittedly on the funky side for even a SOHO operation, but it’s no joke – it’s more powerful than a lot of production blade servers used on wall st and it cost him less than $4K. He also includes a (very loosely described) spec for a more powerful next-generation version with some 50-Teraflops of capacity! So, while the data we’re having to deal with is growing at an incredible rate, the tools we have to manage it are growing proportionately for those who know how to leverage the work so many smart people are producing and freely sharing.

As my dad told me years ago:

“Good programmers write good programs. Great programmers steal good programs.”

At this point, we’re still in the “discovery” stage of our development of TAQ+HDF5 for StratBox, but as we progress I’ll periodically post some of our experiences.

—

UPDATE

Speaking of my dad, he saw this posting and pointed me to some class notes he’s been working on which describe the Google approach to massive information processing and fault-tolerance. Interesting and full of great links to both academic and industrial papers/sites on the topic.

I’ve been trading an increasing amount at the open and close of the equity markets using market-on-open (MOO) and market-on-close (MOC) order types and have found that the quality of executions varies enormously between the two types and have spent a bit of time analyzing the differences which I share below.

The quick scoop is that MOC orders almost invariably fill at the exchange’s published closing price, while MOOs vary very substantially from the published open price. Below I quantify my findings in a bit greater depth.

I looked at 846 recent MOO and MOC equity trades made over the past two months. Of all of these trades, only one MOC trade didn’t execute at the published close and the price I got was only off by one penny. Across all of the trades, I received the open or close price 55% of the time.

The remaining 45% of the time I varied from the open by an average of +.04% This means that I actually saw a slight price improvement in the average case. That is, if I was shorting then I executed at a price above the listed open and vice-versa for longs. I’ll take it!

In the below chart I plot the trades against their variance, positive or negative, from the listed open or close.

The biggest difference was a whopping 8.56% but at least it went in my favor. The stdev across all of the trades was .87% so we’re not looking at too disperse a grouping.

This data is a bit skewed as the majority of the MOO orders are going short. This is also a pretty limited universe of trades, so I’ll continue to look at the execution quality I’m getting on these order types and will revisit it if I see any interesting changes.

My interpretation is that my broker is making a best-effort to get a fair open price and they’re doing a creditable job of it. The exchanges are doing a nearly perfect job with MOC orders.

What impact does this have on my strategies? I’m not sure yet, but my first blush impression is that it might be worthwhile to try to get some price improvement over the posted open price as a means of both improving the results and extending the capacity of such strategies. It’s a favorable result as it means that strategies which back-test well on open/close data have a pretty good chance of executing well in reality.

A related issue, which I’m still researching, concerns the capacity of such strategies and may be the topic of a future post…