Comments on: managing tick data with hdf5 http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/ Algorithmic trading experiences Wed, 28 Jul 2010 01:46:39 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: tito http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-6625 tito Sat, 31 Oct 2009 12:24:45 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-6625 I've given-up irrevocably on trying to fit this kind of data into a relational db. Round hole, square peg as they say. 100M records is a small fraction of the *daily* US equity activity (top of book only) data delivered by TAQ which has thousands of instruments. If you're interested in ~30 instruments, then an rdbms can be made to work, but not if you want a broader view. That said, I knew a very smart guy who had a scheme for encoding one minute's worth of tick data into a blob which was stored in mysql. Again, up to some size, these can be made to work, but not for larger applications... Just for fun, how fast can you get a merged stream of ticks from your db? With hdf5 in c it was over 1M/sec (including the lookup and merge). I'm guessing that mysql is going to be at least an order of magnitude off that mark... I’ve given-up irrevocably on trying to fit this kind of data into a relational db. Round hole, square peg as they say.

100M records is a small fraction of the *daily* US equity activity (top of book only) data delivered by TAQ which has thousands of instruments. If you’re interested in ~30 instruments, then an rdbms can be made to work, but not if you want a broader view.

That said, I knew a very smart guy who had a scheme for encoding one minute’s worth of tick data into a blob which was stored in mysql. Again, up to some size, these can be made to work, but not for larger applications…

Just for fun, how fast can you get a merged stream of ticks from your db? With hdf5 in c it was over 1M/sec (including the lookup and merge). I’m guessing that mysql is going to be at least an order of magnitude off that mark…

]]> By: Calimera http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-6624 Calimera Sat, 31 Oct 2009 12:04:18 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-6624 Have you tested MySql ? I have One table for ticks partitioned montly... I store 100.000.000 of ticks each month... I collect 30 instruments... Have you tested MySql ?
I have One table for ticks partitioned montly…
I store 100.000.000 of ticks each month…
I collect 30 instruments…

]]> By: tito http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-964 tito Wed, 11 Mar 2009 10:09:57 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-964 David, I agree that it would be nice to see an open source version of a tick db, but I'm not too optimistic. It's a reasonable effort, has a very limited potential audience, and just possibly people have had their fill with handouts for wall st types! The time-synchronous use case you mention seems to me a special case of my one use-case... David, I agree that it would be nice to see an open source version of a tick db, but I’m not too optimistic. It’s a reasonable effort, has a very limited potential audience, and just possibly people have had their fill with handouts for wall st types!

The time-synchronous use case you mention seems to me a special case of my one use-case…

]]> By: David Blentham http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-903 David Blentham Sat, 07 Mar 2009 18:17:20 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-903 A common application for a tick database is being able to create an array of time-synchronous prices across securities (taking into account daylight savings, etc). Someone needs to create a template db in HDF5 for time series. This would fulfill a huge need in the financial market. I have to agree 3rd party vendors are very expensive; the trick they employ is massive indexing. A common application for a tick database is being able to create an array of time-synchronous prices across securities (taking into account daylight savings, etc). Someone needs to create a template db in HDF5 for time series. This would fulfill a huge need in the financial market. I have to agree 3rd party vendors are very expensive; the trick they employ is massive indexing.

]]> By: Hack the market » tick data & hdf5 (part 2) http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-422 Hack the market » tick data & hdf5 (part 2) Tue, 13 Jan 2009 19:30:09 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-422 [...] Last time I described the trajectory of my research into using hdf5 for large amounts of tick data.  This time I describe the basic design of the prototype I implemented and some of its performance characteristics. [...] [...] Last time I described the trajectory of my research into using hdf5 for large amounts of tick data.  This time I describe the basic design of the prototype I implemented and some of its performance characteristics. [...]

]]> By: tito http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-389 tito Sun, 04 Jan 2009 21:04:04 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-389 A buy v build decision is always going to be specific to the organizational and technical contexts for which it's being made and the use-cases required, not to mention cost considerations. This might be especially true for a start-up. It's certainly impossible to economically reproduce the rich functionality of kdb+/Q and the attempt would be foolish. But if what's needed is a sufficiently small subset of their functionality integrated with an existing platform, then it just might be reasonable... Or not. In any case, a non-commercial license is swell for educational purposes, but does nothing to solve the problems I'm looking to address. A buy v build decision is always going to be specific to the organizational and technical contexts for which it’s being made and the use-cases required, not to mention cost considerations. This might be especially true for a start-up. It’s certainly impossible to economically reproduce the rich functionality of kdb+/Q and the attempt would be foolish. But if what’s needed is a sufficiently small subset of their functionality integrated with an existing platform, then it just might be reasonable… Or not. In any case, a non-commercial license is swell for educational purposes, but does nothing to solve the problems I’m looking to address.

]]> By: brainyoga http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/comment-page-1/#comment-387 brainyoga Sun, 04 Jan 2009 19:45:34 +0000 http://www.puppetmastertrading.com/blog/?p=262#comment-387 sounds like you are travelling the well-trodden path of investigating tick databases, which usually end up with the conclusion that it is cheaper to buy one than build one. You mentioned in a previous post that dev-licenses can cost a 6 figure number - Kx Systems has a non-commercial license available for download free of charge. It has some limitations (e.g. 2 hour timeout) but allows you to get your feet wet with it. sounds like you are travelling the well-trodden path of investigating tick databases, which usually end up with the conclusion that it is cheaper to buy one than build one. You mentioned in a previous post that dev-licenses can cost a 6 figure number – Kx Systems has a non-commercial license available for download free of charge. It has some limitations (e.g. 2 hour timeout) but allows you to get your feet wet with it.

]]>