photo by Simon Rogerson

Some time back, I’d written about NVidia’s CUDA noting that it looked ideal for many asset-pricing and monte-carlo type problems in finance.  At the time, I was hopeful that it would be quickly integrated into existing open source efforts like QuantLib, but adoption has proved slower than I’d hoped, most likely because implementing non-trivial problems on CUDA is, well, even less trivial than doing them without..

LMM on CUDA

Happily, I’ve just seen a promising first step in this direction as Über-quant and C++ artisan Mark Joshi recently announced an open-source project, Kooderive which looks to implement the LIBOR Market Model (LMM)  on top of CUDA.  His announcement on the QuantLib mailing lists reads:

Dear All,

various people have shown interest in the use of CUDA with QuantLib. I
have now made some progress on a CUDA implementation of the LIBOR
market model.

In particular, I now have a path generator for the LMM working which
does 16384 paths for 40 rates, 40 steps, 5 factor model, displaced
diffusion predictor-corrector that takes 0.1 seconds on my Quadro 4600.

The state of the project is code fragments that can be called from
other code. Those who are interested can get the code via
the subversion repository on kooderive.sourceforge.net .  The only
project file is currently for VC9 x64. It also uses thrust and the
CUDA SDK.

The next stage will be writing routines, that use QuantLib for the CPU
stuff and kooderive for the GPU stuff,  to actually price things.

A gentle reminder that I will be giving a course on the LMM and
QuantLib in June in London, and I will include a session on kooderive
if there
is sufficient interest.

I am happy to take code contributions for kooderive. However, I am not
looking for a redesign of the library or contributions which introduce
dependence on other libraries. I am interested in contributions of
separate routines and of optimizations of existing routines that do
not change interfaces.

regards

Mark
–
Pricing exotic interest rate derivatives – The LIBOR Market Model in
QuantLib June 2010, London,
http://www.moneyscience.com/training/index.html

Assoc Prof Mark Joshi
Centre for Actuarial Studies
University of Melbourne
My website is www.markjoshi.com

The above applet re-uses the one I’d written previously in discussing simple stochastic processes.  This time, it illustrates a point & figure chart below the regular line chart.  Point & figure charts expose two characteristics: a “box size” (in ticks) and a “reversal” (in boxes).  The applet allows you to vary both and then generate a day’s worth of random/synthetic data to view it.  One of the nice features of JFreeChart is that you can easily “zoom” into a chart by dragging within the chart.  I’ve disabled this in the line chart but you can try it in the p&f chart.  (Note: you should right-click and “Auto-Range-Both Axes” before you generate new data or you’ll stay in the zoomed segment of the chart.)

Now that I think I understand the basics of point & figure charting, it will be interesting to see what an algo might do with it…

One Big Table (and chair)

Last time I described the trajectory of my research into using hdf5 for large amounts of tick data.  This time I describe the basic design of the prototype I implemented and some of its performance characteristics.

Prototype Design

With One Big Table (OBT) holding all of your data, you need some help finding what you need in that data.  To this end, I wrote an index on the big table which basically stored, per instrument, beginning and ending indices into the OBT as well as the timestamp at each extreme.

So, the main table was: { conid, timestamp, open, high, low, close, adjustedClose, volume } and the index table was: { conid, minIndex, maxIndex, minTimestamp, maxTimestamp }.  The index is read into memory and kept with a few other handy bits of data in a structure which represents a “connection” into the OBT.

To identify contracts, I use a long int contract identifier which is already employed within our environments.  For timestamps I used the java convention of using a 64-bit long denoting milliseconds since the “epoch” and, initially, used mktime to support this approach in C.  After my first iteration, I found that an incredible proportion of my time spent writing a HDF5 table from a CSV file was spent in mktime whereupon my Dad suggested the use of gmtime.  Remarkably, this yielded a fully 40% improvement to the process!  It’s nice having a guru in the family.

In order to retrieve data for one contract, I implemented an iterator which would find the appropriate section of the OBT for the particular contract.  Another handy piece of data the connection struct maintained is the entire timestamp column for the main table.  Clearly, this isn’t scalable for really big datasets and for the real implementation I’ll have to read this in on an instrument basis at the time of a query.  But for this data, this seemed the sensible implementation.  Two binary searches are performed on the appropriate, indexed subset of this big array to determine exactly which records will need to be read to satisfy the iterator’s query.  Thus, the iterator is primed with the exact location of the data it will require on initialization.  Then, buffered reads are performed by the iterator as data is requested from it.

My only critical use-case is to retrieve a time-ordered stream of OHLCVs (in this case) across potentially many contracts.  This one query meets my needs both for back-testing purposes as well as statistical calculations.  But it requires an efficient merge operation across a potentially large set of these iterators.  To accomplish this, I’ve got a composite iterator which uses a red-black tree to keep all of the contained iterators sorted in the order of their Next() OHLCV.  Thus, the composite iterator will always return the oldest OHLCV amongst all of its contained iterators.

That’s pretty much it.  Apart from the cached timestamp column, this should all scale well to datasets the size of a day’s worth of tick data for the US equity markets with a few foreign markets and maybe some futures thrown in as well.  Options data is a worse case than the one I’m envisioning, but I imagine a similar approach would still work, though perhaps you may have to distribute a day’s data across multiple files.

Benchmarking the Prototype

The data I tested this implementation on is daily data going back to 1990 on some 7100 us, lse and hk equities.  In uncompressed, CSV format, the data weighed-in at 850M and was made-up of ~16.5M records.  The tests I ran were:

• Read the CSV file and write an HDF5 file varying the compression { TRUE | FALSE } and hdf5 chunk sizes { 2^12, 2^13, 2^15, 2^17, 2^19 }.  (For the chunksizes, I based it roughly on the excellent PyTables documentation and then expressed my preference for base-2 scales…)
• Respond to 100 queries across {1, 2^4, 2^8, 2^10, 2^12) randomized contracts over a randomized 2-year period (within the 19 year range).

The first part is really about hdf5 and how its different options will affect my results.  The second is my actual use-case: “give me an ordered stream for some set of contracts over some (two-year) period”.

The results were interesting as these parameters matter.  Especially compression.  Below you can see the results of the tests.

HDF5 Write Performance

This “write test” is just a C program which reads in an 848M CSV file and writes and indexes an HDF5 file using the OBT approach as described above.  Apart the curious bump in file size for the Compressed+8192 variant, the results aren’t too remarkable except to note that compression wants smaller chunksizes.  Badly.

HDF5 Read Performance

Likewise, the read test is the same C program which then reads from each of the files written with the varying HDF5 write parameters.  Here we really see the effect of compression on performance.  It seems that if you want performance, then compression isn’t for you.  If you need compression, then you need to use small chunksizes.

Apart this, it seems that performance from 1-4096 randomly selected contracts degrades reasonably linearly.  The red-black tree is doing an effective job of merging even a reasonably large number of streams.

From an absolute perspective, the performance strikes me as pretty smoking in the good cases.  In a two year period, you’ll have about 500 trading days.  So, for the {No-compression/32768 chunk size/256 contracts} case we have (500 * 100 * 256) / 10 =  ~1.28M records per second including all of the look-ups.

Java + SWIG + HDF5

One of my needs is to be able to access this functionality in Java so that the StratBox GUI can use this data.  To that end, I made sure that as I was developing the prototype I maintained SWIG interfaces and a parallel test driver in Java.  Apart the initial set-up, this proved pretty easy, though adding SWIG to a project does add some complexity.  In any case, I wanted to see how bad a performance hit I’d get running the same tests in Java.  Again, looking at the case highlighted above { No-compression/32768 chunk size/256 contracts}, the Java/SWIG timings are about twice those of HDF5′s native C.  So, we take a pretty significant hit, but it seems unavoidable and ~600K records a second isn’t exactly slow.

HDF5 Read performance from Java / SWIG

Conclusion

The prototype I’ve implemented has the nice characteristic that its design is very similar to what I expect should work well with much larger quantities of tick data (as opposed to the ohlcv data I’ve used here).  The only significant difference is that seeking within an indexed range would be slower as I’d first have to read in the range instead of keeping it handy in memory.  Apart this, I’d need a layer on top of what I have here to manage HDF5 files as well.  Of course, until I actually implement this for tick data, it’s not certain that it will be adequately performant on that much harder case, but I’m reasonably confident that something like what I’ve done here could be made to work well.

There’s a significant learning curve to using HDF5 and one clearly has to spend some time benchmarking for the specific requirements to make sure that the settings used are appropriate.  If I were a little less obsessed with speed or a little more discerning about how I spend my holidays, I’d likely find using PyTables or something similar to be a much better solution than trying to roll my own in this fashion.

]]> http://www.puppetmastertrading.com/blog/2009/01/06/tick-data-hdf5-part-2/feed/ 6 http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/ http://www.puppetmastertrading.com/blog/2009/01/04/managing-tick-data-with-hdf5/#comments Sun, 04 Jan 2009 18:45:36 +0000 tito http://www.puppetmastertrading.com/blog/?p=262

One of the nicest things about the holiday season (Happy New Year, btw) is that it provides a lovely opportunity to spend some quality time with a project that’s a bit more exploratory than might be meaningfully undertaken while trading in lively markets.

A number of months ago, I mentioned using HDF5 to manage tick data as RDBMSes just aren’t up to the task and specialized Tick DBs are absurdly expensive.  While I’d spent some time exploring this idea through the fall, I never had a discrete chunk of time to really explore the technology beyond determing that its Java interfaces weren’t production-worthy.  This meant that we’d have to drop into C to access the functionality we’re interested in and that we’d have to come up with our own bridge out into Java for access by StratBox while StratCloud could access it directly.

Below, I describe what I’ve learned through my holiday geek-spelunking-trek including some timings on various configurable characteristics of HDF5 (e.g., compression and “chunking”).

After spending some time looking at the java interfaces to HDF5, I determined it wasn’t up to snuff.  Why?  Primarily because no-one seems to use it, it lags the main api from a versioning standpoint and it appears to be more-or-less impossible to build from source.  Looking a bit more carefully, it seems to have been written by one (undoubtedly talented and well-meaning) individual who isn’t familiar with java.  (The most egregious example was to use a javax.swing.tree.TreeNode as the base class for a key model object…)

I then spent some time looking at the native api and the underlying object model it exposes.  The model is both powerful and pretty low-level.  They’ve implemented many of the goodies of a file system including groups (“directories”), datasets (“files” or in RDBMS-land, “tables”) and a variety of nice linking mechanisms as well as attributes which might index or otherwise annotate data.  There’s also powerful, extensible I/O options which I didn’t much study beyond compression and “chunking.”

The library is provided with two “first-class” APIs – in C and Fortran – a secondary API in C++ and then the Java interface I mentioned.  Others have written interfaces for other languages, most notably the much-lauded PyTables implementation for Python which is used by many in conjunction with the popular NumPy package.

Given this spread of implementation languages I chose C and determined that I’d steal a page from the talented crew behind QuantLib and use SWIG to expose relevant functionality into Java.  This has proven to be a splendid choice for my needs.

Having gotten this far, I started examining how I’d represent market data with hdf5 and came up with two broadly opposed approaches.  In order to gain some insights from those more experienced, I sent the below problem statement / inquiry to the main HDF5 mailing list:

A description of the data and its use

The data is all timestamped financial streams of “tick” data.  Each record is small (a few hundred bytes at the most), but there are many – in a day you may see many hundred million to a few billion.  Each record is naturally partitioned by instrument (eg, “microsoft”, “ibm”, “dec crude”, etc).  There are less than 30K instruments in the universe I might care about.

I (more or less) don’t care how long it takes to construct the h5 files/structures as it will be performed offline and the only critical query I care about is something like:

“Get ticks for instruments {i1…in} from time t1 to time t2 ordered by time, instr”.

That is, I need to be able to “replay” a subset of the instruments within the data store over some period of time.  But I really care that this be as fast as possible.

Questions

0.  Am I barking up the wrong tree?  Is HDF5 an appropriate technology for the use I’ve described?

1. Given the size/volume of the data, my thought is to partition h5 files by day.  Uncompressed, the files will be on the order of ~25G.  Does this sound reasonable?  What are the key factors impacting this decision from an hdf5 perspective?

Two alternative models come immediately to mind: one big table (OBT) per day ordered by instrument and then time, or one table per instrument (OTPI) ordered by time.  My current inclination is OTPI as it seems more manageable assuming the overhead of so many tables isn’t an issue.

2a.  Are there other, better models you suggest I investigate?

2b.  With the OBT, I’d need to be able “index into” the table to identify the beginning of each instrument’s section (at least).  How would you recommend doing this?  It seems possible to do this with references or perhaps a separate table with numerical indices into the main table.  Any pros/cons/alternatives to these approaches?

2c.  With the OTPI, I’d need to have many tables (at most ~30K) per file.  Is this an issue?

2d. For both models, I’d need to be able to merge sorted sets of h5 data into one sorted set as quickly as possible.  Is there any hdf5 support for doing such a thing or external libraries created for this purpose?

3. What impact on retrieval/querying should I expect to see with varying levels of compression?

4. Any suggestions on chunksizes for this application?

I was fortunate to receive some excellent responses to my query, including from Francesc Alted, the gracious author of the PyTables library, and from a gentleman who’d implemented similar functionality for his own trading environment.  Interestingly, both approaches – OBT and OTPI – were championed.  It seems that OTPI is probably to be preferred if the number of instruments/tables to be stored isn’t excessive (perhaps below 10K though I can’t quantify this) and the frequency of update is significant.  OTPI is easier to implement as it means you can rely more upon the infrastructure provided by HDF5.  OBT instead seems more scalable as you incur less overhead (and goodies) with the one table, though you pay for this by having to implement your own indexing logic.

Given the divergent advice and my own lack of hands-on familiarity with the C library, I decided to try both approaches on a prototype.  Instead of looking at vast amounts of tick data, I’d try both approaches on a smaller store (~1G with ~7K instruments) of OHLCV data.

By far, the easier to implement is the OTPI approach.  However, even with this relatively small amount of data, the difference in write performance and file size was substantial.  Clearly, expanding this to the scale of tick data wasn’t going to yield sufficiently performant results.  I focused on the OBT approach.

—

Given the length of this post and keeping in mind that the holiday season isn’t over just yet (about ten hours remaining as I write this!), I’m going to stop writing now and continue with the remainder of my implementation and findings in a follow-up post later this week..

Normally I spend my design-oriented thoughts on object models – when I’m working on StratBox  – or about volatility, latency, executions, &tc – when I’m working on a trading strategy.  But a recent trip abroad has inspired me to consider more fanciful design horizons.

After more than a year of blogging I’ve finally decided to refresh the look of the site and you’re looking at the first iteration of this effort.  Blogging software is pretty remarkable as it allowed me to essentially change the “skin” of the blog without affecting its content.  This is like Kirill Grouchnikov‘s lovely open source “Substance” Look&Feel for Java which does the same trick for swing-based applications: just include his magic code and your system automagically looks a lot better!

More substantively, my recent trip to Israel and the subsequent agreement to open a Tel Aviv office to take advantage of a felicitous new partnership and Israeli algorithmic talent, has led to a broadening of our mission.  This in turn led to the foray into graphic design I describe below.

While we still haven’t gotten around to updating our main site, I have had the opportunity to work again with Erica Green, the talented designer who brought the Puppetmaster Trading bull puppet into existence…

Origins of the bull puppet

The name “Puppetmaster Trading” came about following a discussion I’d had with a friend in spring 2005 shortly before I’d left my job to try my luck with this startup.  After stating that I wanted to become an algorithmic trader and receiving the now familiar blank stare in response, I had explained to him that I explicitly didn’t want to trade myself, but instead wanted to write little autonomous software agents which would themselves trade on my behalf.  He immediately got it and said, “Oh!  So you’d be like a puppetmaster managing your cast of trading marionettes…” and I knew the startup had found its name.

"the bull puppet"

Erica had developed the original logo based on my wife’s loose vision of a trading puppetmaster, but this was long before I knew we’d have to write the algorithmic trading platform itself.  I had hoped to use an off-the-shelf product so I could focus on the algorithms themselves. That didn’t turn out to be feasible…

Between developing the platform, devising algorithms, overseeing the trading “marionettes” that implemented those algorithms and making most of the many mistakes startups make (and possibly a few of our own invention), I never got a chance to go beyond the bull puppet… until now.

Now that StratBox is a reality, and with a new vision for an exchange-colocated, cloud-computing version of the platform, it seemed a good opportunity to renew the collaboration with Erica.  I like her design for StratBox so much that I’ve lassooed it into service on this blog.

At the head of this post, you’ll see my favorite design for our nascent StratCloud offering, though the one below remains in contention.  (I’d love to hear your opinion!)

As it was three years ago, the experience has been very positive.   If you have some challenging graphic design opportunity I encourage you to see if Erica can’t help you realize your vision.

While Carl Sagan’s famous formulation introduced a generation to the vastness of the cosmos, more recent history suggests that his memorable term might now be more aptly applied to financial extents: our deficits and debts, perhaps, to the economically or politically minded. But for those of us with the markets on our mind, the term has to evoke the enormity of the data we create and must manage every day. We’ve recently been working with the NYSE’s TAQ data in an effort to integrate it into StratBox‘s back-testing and optimization capabilities. And the enormity of the data is really just staggering.

Each day, the NYSE publishes all of the day’s quotes and trades as well as some reference data. Compressed, the data will just about fit onto a DVD. For one day. A DVD. Compressed. It’s really mind-boggling. A year of the stuff, uncompressed, will require over a petabyte of storage. Over 1,125,899,906,842,624 bytes. And that’s just the US Equities markets. You want options data, too? I hope your uncle is named EMC, because just managing the data is going to be a challenge…

“Information about money has become almost as important as money itself.” — Walter Wriston, former Chairman of Citicorp

The enormity and profile of market data far exceeds the capacity of traditional RDBMSes. While RDBMSes continue to expand their usable capacity – we have used partitioned tables with nearly a billion rows of market data which have performed astonishingly well – they simply can’t deal with the kinds of quote volumes modern markets are generating daily. This has spawned a host of specialized timeseries database products, like the grandaddy: Sungard’s FAME which I’d used back in the 90′s to write programs to calculate bond indices at JPM, to more recent offerings like Vhayu and Kdb+. These timeseries oriented data products undoubtedly have many distinguishing characteristics and features, but they share one immutable characteristic: they are unbelievably expensive – in some cases a single developer seat costs in the high 6-figures for an annual license.

Thus, while no doubt missing out on some of their high-end features and niceties, we’ve decided to seek solutions from some of the original purveyors of petabyte-scaled data: NASA and the NCSA through their HDF5 system. Designed to support vast scientific data stores and boasting sophisticated capabilities in support of parallel computing environments, it should be possible to get comparable performance to some of the high-end specialized finance products without the sticker shock. Indeed, it’s potentially another example of free software providing a meaningful contribution to finance.

In researching cost-efficient and highly parallel hardware solutions to pair with our emergent data solution, I’ve come to realize that open-source is expanding its reach into the hardware sphere.

This guy shares his experience and “recipe” for building a powerful and unique rendering cluster inside an IKEA filing cabinet. It’s admittedly on the funky side for even a SOHO operation, but it’s no joke – it’s more powerful than a lot of production blade servers used on wall st and it cost him less than $4K. He also includes a (very loosely described) spec for a more powerful next-generation version with some 50-Teraflops of capacity! So, while the data we’re having to deal with is growing at an incredible rate, the tools we have to manage it are growing proportionately for those who know how to leverage the work so many smart people are producing and freely sharing.

As my dad told me years ago:

“Good programmers write good programs. Great programmers steal good programs.”

At this point, we’re still in the “discovery” stage of our development of TAQ+HDF5 for StratBox, but as we progress I’ll periodically post some of our experiences.

—

UPDATE

Speaking of my dad, he saw this posting and pointed me to some class notes he’s been working on which describe the Google approach to massive information processing and fault-tolerance. Interesting and full of great links to both academic and industrial papers/sites on the topic.

One of these days I’m going to give an overview of all the excellent open-source software I use on a daily basis. Until that day comes, I’ll observe that finance remains one of the big areas where open-source software has made relatively limited inroads.

Two production-quality packages fight that unhappy state: QuantLib – a comprehensive framework for quantitative finance – and QuickFix – a full-featured FIX engine. Both are C++ libraries and both provide very nice interfaces to facilitate integration with other languages, including Java. QuantLib is a big and complicated library and integrating it with Java is not totally obvious. Below, I’ll describe how to build and use QuantLib from Java.

These instructions are based on a unix installation. I’m not really a windows developer and don’t have all the shiny tools that windows developers use, so it’s not an area of focus for me. That said, I have managed to build QuantLib under windows by using MinGW+MSYS but it wasn’t terribly easy and I don’t currently have a working installation, so I won’t cover that here. If this is your aim, don’t be dismayed as it is possible and it had all the functionality I enjoy under linux.

Using QuantLib from Java (on linux)

• Build QuantLib
  • Requires a working version of Boost. This may prove to be the hardest step of all and you’ll need to use the ample documentation provided by the Boost team.
  • Once you have a working copy of Boost, building QuantLib should require little more than

  sh autogen.sh
./configure

  make
sudo make install

• Build QuantLib-SWIG
  • Requires a working copy of SWIG. Again, look to the SWIG instructions, but it should be easy.
  • Once SWIG is available, building the QuantLib/SWIG interfaces should only require:

  sh autogen.sh
./configure \

  –with-jdk-include=${JAVA_HOME}/include \
  –with-jdk-system-include=${JAVA_HOME}/include/linux
  make -C Java
sudo make install

• Now you’ll have a Jar file with all of the SWIG/JNI stubs in it available in /usr/local/lib/QuantLib.jar. Add this to your classpath.
• Programs which call QuantLib functionality will need to have the LD_LIBRARY_PATH set. This can be done by invoking the vm with something like:

-Djava.library.path=/usr/local/lib

• Programs which call QuantLib functionality will also need to explicitly load the QuantLib libraries. This can be done with something like the following static block appearing before your main method:

static { // Load QuantLib
try { System.loadLibrary("QuantLibJNI"); }
catch (RuntimeException e) { e.printStackTrace(); }
}

• That’s it. Now test your configuration by running the examples in Quantlib-SWIG/Java/examples.

It’s worth understanding how Quantlib is being used from java. SWIG is creating a JNI interface into those methods within Quantlib which have been exposed through their declaration in the swig *.i files. These files are found in Quantlib-SWIG/SWIG and they determine what functionality from Quantlib will be available to you. You’ll likely need to get familiar with a subset of those files that you care about. If you find that some functionality you care about isn’t exposed in those files, you may need to expose it yourself.

There’s a learning curve, but it’s worth traversing so you can get at all the rich functionality so many smart people have put together.

Among the more challenging questions we face when describing the Puppetmaster environment are those like “how do you create new proprietary trading strategies within the environment?” It’s a difficult question because of expectations – people want to hear about some super simple scripting language that any non-technical person can immediately learn and be up and algorithmically trading in no time. A few platforms intended for retail users offer such things – one is even appropriately named easy language. When researching approaches for our system, we spent some time learning easy language and found that it in fact did make easy things easy!

The problem was that it also made sophisticated things impossible.

This led us to pursue another, more powerful, approach for which we are currently seeking a patent.

That said, once upon a time I’d worked on a system into which some enterprising engineer had embedded a TCL interpreter with which one could monitor and manipulate objects within the runtime environment. We didn’t use it for much, but it was certainly interesting and helped with debugging the live system. Ever since then, I’ve had an interest in embedding a simple language+interpreter into my systems.

Thus, when a friend who builds systems for UBS told me about a talk on the groovy language in the google auditorium. I went along to see what it was about. (Admittedly, I was in significant part motivated to see inside the mysterious googleplex. I found it disappointing unless you fancy an environment that looks like an apple store with free razor scooters for anyone who might fancy a spin…)

The talk itself was also a bit disappointing as the speaker was very permissive with the questions and google had graciously given the crowd access to free beer. Many will argue that java programmers aren’t the sharpest tools in the shed and while I won’t weigh-in on that issue, I’m pretty certain that adding beer to the equation isn’t likely to help.

In any case, I did come away from the talk understanding that 1) if our business takes off in a big way, we will not attempt to motivate people with silly toys and 2) that groovy (or any other such scripting language) isn’t a reasonable mechanism for enabling non-technical people to write trading strategies.

This is no fault of the groovy language which seems fine enough. It’s a problem with the complexity of algorithmic trading. Trading is an intrinsically uncertain, asynchronous and demanding activity. I send orders to exchanges and they respond in their own good time. It might be 20ms if I’m sitting on top of the exchange or it might be seconds if I’m around the world or it might be days or more if it’s some kind of conditional order, but it’s certainly not a fixed time and it’s an uncertain message I’ll receive in response. Messages arrive when they arrive and you have to be ready for them.

If you’re trading a portfolio re-balancing model that trades infrequently and in size then it might suffice to send a few or perhaps a few hundred vwap orders a month and things will mostly take care of themselves. Here, maybe a scripting language is sufficiently expressive. But if you want to implement that vwap algorithm yourself or implement a strategy that trades against your broker’s vwap algorithm or implement a high-frequency strategy that is very “close to the market” requiring extreme low latency and precise order management or write a discovery algorithm that’s employing proprietary understanding of the ubiquitous and ephemeral “dark pools of liquidity,” then no scripting language will suffice.

From a technology perspective, my curiosity for groovy was sated – it’s certainly possible (and even easy!) to embed a powerful scripting language into your application. But from an algorithmic trading perspective, the problem is designing a trading-specific language that allows easy things to be simply done while allowing complex strategies to be efficiently expressed as well. This remains an area of active interest and research as it may someday complement our existing methodology, but for now we think we’ve got the best solution for the difficult “problem with easy.”

The best way to reason about a trading strategy’s performance, that is valuing it, is as an option.

Or perhaps as a collection or portfolio of them.

I have to assume that people reading this have a working idea of what an option is, so I’m not going to provide definitions that can be readily found elsewhere. I will note that my favorite book on the trading of options is by Allen Jan Baird.

Let’s consider the three illustrative trading strategies we’ve looked at up until now. The trend-following strategy suffered many little losses and then enjoyed a big win. Sounds like buying options. The mean-reverting strategy made lots of little profits and then risked getting clobbered with a big loss. Sounds like someone who’s writing options. And the first strategy we looked at, the morning range breakout, had a payoff which looked like a long straddle or strangle where the break-evens were near the observed high and lows for the session (where we set our entry stops).

Now, there’s obvious differences between the trading strategies’ payoff structures as compared to the similar options strategies. There’s no premium, for instance, and that’s clearly significant. The morning range breakout seems to exhibit a sort of knockout effect when a position has been entered but then the market reverses and you’re “knocked-out” of your position. You just take a loss and do not collect even if the market turns back in your direction. With a straddle you don’t have this behavior. There are differences and they are worth keeping in mind. But the reasons for viewing trading strategies as options portfolios are many and compelling.

The superficial reason, as I mentioned, is that the basic payout structures are potentially similar. The deep reason is that ultimately the problems are the same – how to value complex instruments with engineered payouts. And the pragmatic reason is that many many very smart people have applied their considerable brains and diverse skill-sets to advancing options pricing techniques. There’s also a great deal of high quality software available out there which can be used to adapt these time-proven techniques to your own algorithmic trading strategy valuations.

The techniques which we’ve seen up until now, back-testing and parameter optimization, are sort of weak cousins of a family of techniques long used for options pricing: Monte-Carlo (MC) methods. MC simulation can clearly be used to assess a trading strategy’s performance.

In subsequent posts, we’ll talk about some of the details of each of these techniques and about some of their respective trade-offs. That should keep my pump primed for a bit, but in the meanwhile I leave you with a parting inquiry: what other options pricing techniques might we apply to our algorithmic trading practices?