Comments on: tick data & hdf5 (part 2)

By: tito

tito — Thu, 05 Feb 2009 20:10:28 +0000

Nice. There’s some serious cocktail-party-fodder right there ;^>

Sounds like a difficult project – good luck!

By: TJB

TJB — Thu, 05 Feb 2009 19:47:02 +0000

Thanks. I figured it out. I’ll let you know how it goes with PyTables. My dissertation focuses on taking the Huberman & Stanzl type optimal liquidity trading models to data. I use an econometric technique from the empirical IO literature to identify the parameters of the model from transaction data. Then with the estimated parameters I can do forward looking policy simulations. Best regards.

By: tito

tito — Thu, 05 Feb 2009 16:36:45 +0000

I didn’t use PyTables; everything I relate here was done in C and then exposed to java via swig. A similar approach might be taken with python, but I’m pretty certain that you’ll be better off using the excellent pytables package than trying my labor-intensive approach. This should be especially true if your interest is academic rather than for low-latency algo trading!

As for sharing the code, my hope is to solidify and build-upon what I’ve done for eventual integration with our StratBox/StratCloud environments, so it’s not something I’m willing to share at this point. Sorry. The good news is that it really is just a prototype, so likely wouldn’t be of much use to you in its current form!

Not sure what dvd issue you’re referring to, but there were no DVDs involved in my effort…

What’s the subject of your dissertation?

By: TJB

TJB — Thu, 05 Feb 2009 16:11:10 +0000

I am trying to do something similar. I am using PyTables. However, I don’t think I am reading the data from the month DVDs correctly. It should be straight forward, but I am getting junk from the IDX files. Did you use python to read the data from DVD? Care to share your code? I would be much obliged. BTW, this is for work on my dissertation.

By: tito

tito — Tue, 27 Jan 2009 14:00:06 +0000

To my knowledge, there’s no online archive for the hdf5 forum. In order to access past traffic, send an email to ‘hdf-forum-help@hdfgroup.org’ for instructions.

As for using groups & datasets in various ways, this was really the exploration I described in part 1 as OTPI (one table per instrument) vs OBT (one big table). In all cases, I planned to store one day per file. I went with OBT as I felt that it would allow me to avoid the overhead inherent in hdf5’s internal data structures.

By: Larry

Larry — Mon, 26 Jan 2009 22:54:46 +0000

I just joined the HDF5 forum and newsgroup today and was looking forward to reading your thread. It’s likely user error on my part, but I fail to find an archive for either the newsgroup or the forum.

Apparently HDF5 has Groups and Datasets. Did you consider using a Group to represent a day, and Datasets to represent the tickdata for the instrumetns on that day? I wonder if the Group entity is useless or a timehog in storing tickdata.