messages per second across "all" feeds

I came across this compelling site which uses a hardware-based ticker plant (Exegy) in a colo environment to measure peak bandwidth across scads of NA feeds and then, every minute, updates a chart like the above to capture the average messages/sec across all of them.  Pretty swank.

While the uninformed may rail against colocation rather than focus on less intriguing issues like banana-variety corruption, they miss the basic point that colo can be done by anyone with the checkbook and the wish to do so.

unfair advantage?

It’s sort of like that boat in Forrest Gump.  Forrest wanted to be a shrimper.  So he invested in a boat.  With his initial capital, hard work, perseverance and a bit of luck, Forrest made a go of it.  He might easily have not made it. Colo is like that.  You can shrimp without a boat if you have a mask and fins, but it’s likely not a sustainable model… either way, it’s hard to see the harm in Gump’s boat.  Or colocation.

Hat-tip to Rodrick’s Web Log !! for spotting the market data peaks site.

Mmmm... hardware..

I’ve never been a hardware guy. Hardware has gotten so fast throughout my professional life that it has just never been a big issue. Also, on wall st we had a robust and annual budget for h/w so I’d routinely sign-off on hundreds of thousands of dollars on all sorts of machines I’d never lay eyes on and somehow they always did the trick.

Before 9/11, they’d be in server racks in the building or down the street, but since then they might also be in increasingly far-flung places like weehawken or long island, tampa, even texas or beyond. The machines always seemed unbelievably overpriced – I remember over the years pretty consistently paying something like $40K for a low-end db server.  But that’s what it cost and you could only purchase approved products from approved channels, so nobody spent much thought on it.  Now that I don’t have the same kinds of constraints – or budgets! – I increasingly have to think of hardware.

As a software engineer, the hardware itself is also insisting that I pay some uncharacteristic attention to it.  The evolution of processors has reached a point where the programming paradigms many of us have fruitfully employed over many years are no longer suited for getting full performance out of today’s machines.  The recent introduction of remarkably powerful and inexpensive parallel-computing platforms based on GPUs like nvidia’s cuda also outline a future that even current university training doesn’t address in a fashion practically adapted for institutional application.  Cores are multiplying like Tribbles.

The lines between persistent storage and main memory are also blurring as consumer SSDs push up from the ‘low’-end while exotic ioDrives and the like offer a glimpse of a world where the performance gap between the two approaches nil and after their long reign myriad metallic platters will spin no more.

troubling like Tribbles

There have been some steps taken towards taming the core dilemma.  Google’s introduction of the distributed map-reduce paradigm and all of the associated plumbing on top of computers in the ‘cloud’ is probably the boldest and most effective reaction thus far, but it’s not always obvious that you want your stuff running in someone else’s cloud amongst many other natural limitations of this approach.  This is also a solution ‘in the large’ and sometimes you need performant solutions on a different, smaller scale.  Here, the development of functional languages and idioms may be of some help, but there certainly don’t seem to be clear winners yet.

Erlang, Ocaml, Haskell, Scala and others all seem to have very limited impact thus far and all face big challenges before receiving widespread welcome.  Worse, any language can be mangled into expressing things poorly so the languages can only be a meaningful aid to programmers who are able to adjust their mindset for a new world’s computing paradigm.  This likely won’t be easy for many until there is an established set of usable programming idioms and toolsets for dealing with concurrency on a whole new scale.  To me, it seems that functional programming might well be an important part of that, but it’s difficult to imagine it as the complete game changer any time soon.  As it is, many of us have already been using functional languages (in my case, sql and R) on a regular or even daily basis for a long time, so it’s difficult to cast this as the revolution, bottled.

As cores proliferate and the bandwidth amongst them increases, new challenges and opportunities are unveiled.  Feeding all of those hungry cores can be a chore.  If you already had problems with i/o bound processes, then adding cores is sort of like adding liquidity to a debt crisis – not obviously helpful. We faced an issue like this recently while trying to improve the throughput of our backtesting subsystem in particularly poor-performing cases.

In these cases, we use daily data to initialize an intraday strategy.  For example, we might look at all equities over a trailing 3 or 5-year period, and perform various analytics – like calculating correlation matrices, beta against various benchmarks, volatility, etc – to determine which names might be candidates for the strategy’s portfolio.  We found that an inordinate amount of our time was spent just performing these morning analytics and that the cost of a day’s backtest was significantly spent on this oft-repeated morning exercise.  While working through this issue, we noted that we’ve reached a point where a decent, dual nehalem, server (with 16 logical cores) could be built with 48G of RAM for something along the lines of $6K.  Seriously.  So we stuck all of our daily data into memory and the improvement has been essentially infinite.  Maybe not the best example, but hopefully illustrative of the fact that the h/w underneath us is changing qualitatively and that we need to be more active in involving it in our design decisions.

Of course, even a yawning expanse like 48G or even a terabyte (in not too many years, after all, I should be able to buy 1T of RAM for my boxes at a similar price point to what I pay for 48G today), eventually gets consumed and so we can’t hope to employ this solution for all our problems.  We continue to develop our historical TAQ infrastructure (most recently discussed here) and this is certainly an example where buying memory isn’t going to get you very far.  But SSDs now are getting reasonable and so our current approach uses memory as much as possible but when it needs to get volumes of detailed historical data we’ve placed our indices onto SSDs while the actual stores themselves reside on much more affordable RAID arrays.  Reading indices is now much faster and adds minimal i/o overhead to a very i/o-bound problem.

By constantly looking at new languages and concurrency idioms, vigilantly assessing the current and projected costs of ’solid-state’-solutions to our most vexing problems, and just staying active and creative, we hold out some hope that we can transition gracefully to what is looking increasingly like a solid-state future.

One Big Table (and chair)

Last time I described the trajectory of my research into using hdf5 for large amounts of tick data.  This time I describe the basic design of the prototype I implemented and some of its performance characteristics.

Prototype Design

With One Big Table (OBT) holding all of your data, you need some help finding what you need in that data.  To this end, I wrote an index on the big table which basically stored, per instrument, beginning and ending indices into the OBT as well as the timestamp at each extreme.

So, the main table was: { conid, timestamp, open, high, low, close, adjustedClose, volume } and the index table was: { conid, minIndex, maxIndex, minTimestamp, maxTimestamp }.  The index is read into memory and kept with a few other handy bits of data in a structure which represents a “connection” into the OBT.

To identify contracts, I use a long int contract identifier which is already employed within our environments.  For timestamps I used the java convention of using a 64-bit long denoting milliseconds since the “epoch” and, initially, used mktime to support this approach in C.  After my first iteration, I found that an incredible proportion of my time spent writing a HDF5 table from a CSV file was spent in mktime whereupon my Dad suggested the use of gmtime.  Remarkably, this yielded a fully 40% improvement to the process!  It’s nice having a guru in the family.

In order to retrieve data for one contract, I implemented an iterator which would find the appropriate section of the OBT for the particular contract.  Another handy piece of data the connection struct maintained is the entire timestamp column for the main table.  Clearly, this isn’t scalable for really big datasets and for the real implementation I’ll have to read this in on an instrument basis at the time of a query.  But for this data, this seemed the sensible implementation.  Two binary searches are performed on the appropriate, indexed subset of this big array to determine exactly which records will need to be read to satisfy the iterator’s query.  Thus, the iterator is primed with the exact location of the data it will require on initialization.  Then, buffered reads are performed by the iterator as data is requested from it.

My only critical use-case is to retrieve a time-ordered stream of OHLCVs (in this case) across potentially many contracts.  This one query meets my needs both for back-testing purposes as well as statistical calculations.  But it requires an efficient merge operation across a potentially large set of these iterators.  To accomplish this, I’ve got a composite iterator which uses a red-black tree to keep all of the contained iterators sorted in the order of their Next() OHLCV.  Thus, the composite iterator will always return the oldest OHLCV amongst all of its contained iterators.

That’s pretty much it.  Apart from the cached timestamp column, this should all scale well to datasets the size of a day’s worth of tick data for the US equity markets with a few foreign markets and maybe some futures thrown in as well.  Options data is a worse case than the one I’m envisioning, but I imagine a similar approach would still work, though perhaps you may have to distribute a day’s data across multiple files.

Benchmarking the Prototype

The data I tested this implementation on is daily data going back to 1990 on some 7100 us, lse and hk equities.  In uncompressed, CSV format, the data weighed-in at 850M and was made-up of ~16.5M records.  The tests I ran were:

• Read the CSV file and write an HDF5 file varying the compression { TRUE | FALSE } and hdf5 chunk sizes { 2^12, 2^13, 2^15, 2^17, 2^19 }.  (For the chunksizes, I based it roughly on the excellent PyTables documentation and then expressed my preference for base-2 scales…)
• Respond to 100 queries across {1, 2^4, 2^8, 2^10, 2^12) randomized contracts over a randomized 2-year period (within the 19 year range).

The first part is really about hdf5 and how its different options will affect my results.  The second is my actual use-case: “give me an ordered stream for some set of contracts over some (two-year) period”.

The results were interesting as these parameters matter.  Especially compression.  Below you can see the results of the tests.

HDF5 Write Performance

This “write test” is just a C program which reads in an 848M CSV file and writes and indexes an HDF5 file using the OBT approach as described above.  Apart the curious bump in file size for the Compressed+8192 variant, the results aren’t too remarkable except to note that compression wants smaller chunksizes.  Badly.

HDF5 Read Performance

Likewise, the read test is the same C program which then reads from each of the files written with the varying HDF5 write parameters.  Here we really see the effect of compression on performance.  It seems that if you want performance, then compression isn’t for you.  If you need compression, then you need to use small chunksizes.

Apart this, it seems that performance from 1-4096 randomly selected contracts degrades reasonably linearly.  The red-black tree is doing an effective job of merging even a reasonably large number of streams.

From an absolute perspective, the performance strikes me as pretty smoking in the good cases.  In a two year period, you’ll have about 500 trading days.  So, for the {No-compression/32768 chunk size/256 contracts} case we have (500 * 100 * 256) / 10 =  ~1.28M records per second including all of the look-ups.

Java + SWIG + HDF5

One of my needs is to be able to access this functionality in Java so that the StratBox GUI can use this data.  To that end, I made sure that as I was developing the prototype I maintained SWIG interfaces and a parallel test driver in Java.  Apart the initial set-up, this proved pretty easy, though adding SWIG to a project does add some complexity.  In any case, I wanted to see how bad a performance hit I’d get running the same tests in Java.  Again, looking at the case highlighted above { No-compression/32768 chunk size/256 contracts}, the Java/SWIG timings are about twice those of HDF5’s native C.  So, we take a pretty significant hit, but it seems unavoidable and ~600K records a second isn’t exactly slow.

HDF5 Read performance from Java / SWIG

Conclusion

The prototype I’ve implemented has the nice characteristic that its design is very similar to what I expect should work well with much larger quantities of tick data (as opposed to the ohlcv data I’ve used here).  The only significant difference is that seeking within an indexed range would be slower as I’d first have to read in the range instead of keeping it handy in memory.  Apart this, I’d need a layer on top of what I have here to manage HDF5 files as well.  Of course, until I actually implement this for tick data, it’s not certain that it will be adequately performant on that much harder case, but I’m reasonably confident that something like what I’ve done here could be made to work well.

There’s a significant learning curve to using HDF5 and one clearly has to spend some time benchmarking for the specific requirements to make sure that the settings used are appropriate.  If I were a little less obsessed with speed or a little more discerning about how I spend my holidays, I’d likely find using PyTables or something similar to be a much better solution than trying to roll my own in this fashion.

]]> http://www.puppetmastertrading.com/blog/2009/01/06/tick-data-hdf5-part-2/feed/ 6 http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/ http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/#comments Sun, 04 Jan 2009 18:45:36 +0000 tito http://www.puppetmastertrading.com/blog/?p=262

One of the nicest things about the holiday season (Happy New Year, btw) is that it provides a lovely opportunity to spend some quality time with a project that’s a bit more exploratory than might be meaningfully undertaken while trading in lively markets.

A number of months ago, I mentioned using HDF5 to manage tick data as RDBMSes just aren’t up to the task and specialized Tick DBs are absurdly expensive.  While I’d spent some time exploring this idea through the fall, I never had a discrete chunk of time to really explore the technology beyond determing that its Java interfaces weren’t production-worthy.  This meant that we’d have to drop into C to access the functionality we’re interested in and that we’d have to come up with our own bridge out into Java for access by StratBox while StratCloud could access it directly.

Below, I describe what I’ve learned through my holiday geek-spelunking-trek including some timings on various configurable characteristics of HDF5 (e.g., compression and “chunking”).

After spending some time looking at the java interfaces to HDF5, I determined it wasn’t up to snuff.  Why?  Primarily because no-one seems to use it, it lags the main api from a versioning standpoint and it appears to be more-or-less impossible to build from source.  Looking a bit more carefully, it seems to have been written by one (undoubtedly talented and well-meaning) individual who isn’t familiar with java.  (The most egregious example was to use a javax.swing.tree.TreeNode as the base class for a key model object…)

I then spent some time looking at the native api and the underlying object model it exposes.  The model is both powerful and pretty low-level.  They’ve implemented many of the goodies of a file system including groups (“directories”), datasets (“files” or in RDBMS-land, “tables”) and a variety of nice linking mechanisms as well as attributes which might index or otherwise annotate data.  There’s also powerful, extensible I/O options which I didn’t much study beyond compression and “chunking.”

The library is provided with two “first-class” APIs – in C and Fortran – a secondary API in C++ and then the Java interface I mentioned.  Others have written interfaces for other languages, most notably the much-lauded PyTables implementation for Python which is used by many in conjunction with the popular NumPy package.

Given this spread of implementation languages I chose C and determined that I’d steal a page from the talented crew behind QuantLib and use SWIG to expose relevant functionality into Java.  This has proven to be a splendid choice for my needs.

Having gotten this far, I started examining how I’d represent market data with hdf5 and came up with two broadly opposed approaches.  In order to gain some insights from those more experienced, I sent the below problem statement / inquiry to the main HDF5 mailing list:

A description of the data and its use

The data is all timestamped financial streams of “tick” data.  Each record is small (a few hundred bytes at the most), but there are many – in a day you may see many hundred million to a few billion.  Each record is naturally partitioned by instrument (eg, “microsoft”, “ibm”, “dec crude”, etc).  There are less than 30K instruments in the universe I might care about.

I (more or less) don’t care how long it takes to construct the h5 files/structures as it will be performed offline and the only critical query I care about is something like:

“Get ticks for instruments {i1…in} from time t1 to time t2 ordered by time, instr”.

That is, I need to be able to “replay” a subset of the instruments within the data store over some period of time.  But I really care that this be as fast as possible.

Questions

0.  Am I barking up the wrong tree?  Is HDF5 an appropriate technology for the use I’ve described?

1. Given the size/volume of the data, my thought is to partition h5 files by day.  Uncompressed, the files will be on the order of ~25G.  Does this sound reasonable?  What are the key factors impacting this decision from an hdf5 perspective?

Two alternative models come immediately to mind: one big table (OBT) per day ordered by instrument and then time, or one table per instrument (OTPI) ordered by time.  My current inclination is OTPI as it seems more manageable assuming the overhead of so many tables isn’t an issue.

2a.  Are there other, better models you suggest I investigate?

2b.  With the OBT, I’d need to be able “index into” the table to identify the beginning of each instrument’s section (at least).  How would you recommend doing this?  It seems possible to do this with references or perhaps a separate table with numerical indices into the main table.  Any pros/cons/alternatives to these approaches?

2c.  With the OTPI, I’d need to have many tables (at most ~30K) per file.  Is this an issue?

2d. For both models, I’d need to be able to merge sorted sets of h5 data into one sorted set as quickly as possible.  Is there any hdf5 support for doing such a thing or external libraries created for this purpose?

3. What impact on retrieval/querying should I expect to see with varying levels of compression?

4. Any suggestions on chunksizes for this application?

I was fortunate to receive some excellent responses to my query, including from Francesc Alted, the gracious author of the PyTables library, and from a gentleman who’d implemented similar functionality for his own trading environment.  Interestingly, both approaches – OBT and OTPI – were championed.  It seems that OTPI is probably to be preferred if the number of instruments/tables to be stored isn’t excessive (perhaps below 10K though I can’t quantify this) and the frequency of update is significant.  OTPI is easier to implement as it means you can rely more upon the infrastructure provided by HDF5.  OBT instead seems more scalable as you incur less overhead (and goodies) with the one table, though you pay for this by having to implement your own indexing logic.

Given the divergent advice and my own lack of hands-on familiarity with the C library, I decided to try both approaches on a prototype.  Instead of looking at vast amounts of tick data, I’d try both approaches on a smaller store (~1G with ~7K instruments) of OHLCV data.

By far, the easier to implement is the OTPI approach.  However, even with this relatively small amount of data, the difference in write performance and file size was substantial.  Clearly, expanding this to the scale of tick data wasn’t going to yield sufficiently performant results.  I focused on the OBT approach.

—

Given the length of this post and keeping in mind that the holiday season isn’t over just yet (about ten hours remaining as I write this!), I’m going to stop writing now and continue with the remainder of my implementation and findings in a follow-up post later this week..

Just after the dot.com craze, a brilliant friend of mine (who had just sold his web consulting startup) decided to write a book.  The premise was glorious.  A bunch of clever college-age kids formed a startup to predict the stock market.  The method they used was to constantly comb the web with ultra-sophisticated algorithms which would run across giant server farms overnight and ultimately generate tomorrow’s headlines.  Based on the headlines that their system generated, they would place trades that would take advantage of these predicted events.

Sadly, my friend never went on to complete his book, so I don’t know how it all turned out.  (Instead, he went on to start another successful company, this time in the field of robotics.)  While he was writing it, I loved getting new drafts as they were filled with clever ideas.  But the core idea of predicting headlines and then using those headlines to trade always struck me as especially cute.

For those of us without access to news-predicting algos, writing strategies based on the news is rather less straight forward, though there are a growing variety of products and services aiming to fill the gaps.  Today must have been trading-the-news-day as I found a few articles on the topic in my mailbox and even received a cold call from a vendor, Need to Know News, with just such an offering.  Below I’ll look at some of these offerings and consider some of the issues involved in writing trading strategies based on the news.

The idea of trading the news resonates with me as it was one of the first things I tried to automate.  In particular, I spent some time looking to trade crude and natural gas futures based on their respective weekly EIA reports.  This experience led me to a couple conclusions.

The market knows before the news wires do. This is a big problem for two reasons.  The first is just plain-Jane latency, and I know that vendors are now effectively reducing latency to the sub-second level, but the market still knows first.  The second issue is deeper and is captured nicely by a quote I heard (sorry – I don’t remember where) which went something like:

“capital markets are the original social networks”

Which suggests that markets have their own internal languages and understandings.  Thus, translating a market-impacting event, like an EIA report, into a human-readable form to reason about it (even if there’s no human doing the reasoning) and then action it back into the market struck me as a necessarily lossy transformation.

Another way of seeing this is just to consider: what to do based on the news?  Sell a build of inventory?  Maybe with your money.  Compare the number against expectations?  Whose?  The semantic content that exists in the market seems to be intrinsically richer than that one might extract from a news wire.  But this problem of actioning a news item brings up the next issue.

The problem of history.  One of the wonderful and horrible things about market data is that there is a lot of it going back a long ways.  This is an expensive pickle to manage, but it at least means that you have the ability to look back almost arbitrarily far into the past to see how markets responded to various conditions.  This isn’t so true with most news wires.

To some degree, these issues are being addressed by vendors.  And some of them will be addressed by how people utilize the news feeds.  If, instead of trying to write a strategy based wholly on the news, I try to improve an existing strategy by annotating its model with data gleaned from a feed, I might wind up with much better results.

Indeed, one of the papers I came across today is a research report from Securities Industry News (incidentally, my first tip-off that Citi was going down the toilet was when my management told me I could no longer keep my subscription to that fine periodical).  In it, market participants indicate that they expect to improve existing algos more than create brand-new ones.  But they also failed to complain about the lack of back-testable feed histories, so…

The other paper is a much more detailed quantitative exposition on how to build a news reading algo and some statistical analysis on how well it annotated the market.  This one is the work of Andrew Lo’s AlphaSimplex Group and as such is required reading for anyone who really wants to implement such systems.

Ever since my first experiences with trying to trade energy commodities based on the EIA report via news feed, I’ve been pretty skeptical of the utility of trying to algorithmically trade the news.  That said, like everything else in this space, there’s an incredible amount of innovation going on and an incredible number of seriously smart and motivated folks working to ensure that this will be a productive path for those with the ability to tackle the formidable complexities presented.

–

Updated: added link to Need to Know News’ site

While Carl Sagan’s famous formulation introduced a generation to the vastness of the cosmos, more recent history suggests that his memorable term might now be more aptly applied to financial extents: our deficits and debts, perhaps, to the economically or politically minded. But for those of us with the markets on our mind, the term has to evoke the enormity of the data we create and must manage every day. We’ve recently been working with the NYSE’s TAQ data in an effort to integrate it into StratBox’s back-testing and optimization capabilities. And the enormity of the data is really just staggering.

Each day, the NYSE publishes all of the day’s quotes and trades as well as some reference data. Compressed, the data will just about fit onto a DVD. For one day. A DVD. Compressed. It’s really mind-boggling. A year of the stuff, uncompressed, will require over a petabyte of storage. Over 1,125,899,906,842,624 bytes. And that’s just the US Equities markets. You want options data, too? I hope your uncle is named EMC, because just managing the data is going to be a challenge…

“Information about money has become almost as important as money itself.” — Walter Wriston, former Chairman of Citicorp

The enormity and profile of market data far exceeds the capacity of traditional RDBMSes. While RDBMSes continue to expand their usable capacity – we have used partitioned tables with nearly a billion rows of market data which have performed astonishingly well – they simply can’t deal with the kinds of quote volumes modern markets are generating daily. This has spawned a host of specialized timeseries database products, like the grandaddy: Sungard’s FAME which I’d used back in the 90’s to write programs to calculate bond indices at JPM, to more recent offerings like Vhayu and Kdb+. These timeseries oriented data products undoubtedly have many distinguishing characteristics and features, but they share one immutable characteristic: they are unbelievably expensive – in some cases a single developer seat costs in the high 6-figures for an annual license.

Thus, while no doubt missing out on some of their high-end features and niceties, we’ve decided to seek solutions from some of the original purveyors of petabyte-scaled data: NASA and the NCSA through their HDF5 system. Designed to support vast scientific data stores and boasting sophisticated capabilities in support of parallel computing environments, it should be possible to get comparable performance to some of the high-end specialized finance products without the sticker shock. Indeed, it’s potentially another example of free software providing a meaningful contribution to finance.

In researching cost-efficient and highly parallel hardware solutions to pair with our emergent data solution, I’ve come to realize that open-source is expanding its reach into the hardware sphere.

This guy shares his experience and “recipe” for building a powerful and unique rendering cluster inside an IKEA filing cabinet. It’s admittedly on the funky side for even a SOHO operation, but it’s no joke – it’s more powerful than a lot of production blade servers used on wall st and it cost him less than $4K. He also includes a (very loosely described) spec for a more powerful next-generation version with some 50-Teraflops of capacity! So, while the data we’re having to deal with is growing at an incredible rate, the tools we have to manage it are growing proportionately for those who know how to leverage the work so many smart people are producing and freely sharing.

As my dad told me years ago:

“Good programmers write good programs. Great programmers steal good programs.”

At this point, we’re still in the “discovery” stage of our development of TAQ+HDF5 for StratBox, but as we progress I’ll periodically post some of our experiences.

—

UPDATE

Speaking of my dad, he saw this posting and pointed me to some class notes he’s been working on which describe the Google approach to massive information processing and fault-tolerance. Interesting and full of great links to both academic and industrial papers/sites on the topic.