or: scaling with elastic map-reduce

Between a rapidly evolving compute environment in which cores are multiplying like springtime rabbits, and a business domain in which the fecundity of market data is making those same rabbits look downright prudish, we are always looking for ways to scale our efforts.  There are three levels at which this can be typically done:

• “below the processor” with things like CUDA or FPGAs,
• “amongst the cores” with things like TBB, Cilk, &tc, or
• “in the cloud” (or grid) with things likes Amazon’s EC2 and its bewildering and fast-growing coterie of associated technologies and products

Today we’ll look at a simple example which we implement on top of Hadoop and then deploy into Amazon’s cloud to get a back-of-the envelope feel for what kind of scaling we might expect to gain – and at what costs – from using this smorgasboard of technologies.

’smorgasboard’ is being generous

Probably the hardest thing whenever you look at some new set of technologies is just getting your bearings.  Amazon’s stack is complex and very much of a moving target, so I’ll start by providing a bit of a taxonomy of the beasties we’ll need to at least familiarize ourselves with in order to get anything done.

First, there’s the family of Amazon’s Web Services (AWS).  This has been around for a good few years, so some parts of it are quite mature and well documented and understood where others are bleeding edge with an emphasis on bleeding.  All of them can be played with quite easily by simply signing up (this involves providing a credit card, so head’s up).  I can’t think of any more interesting family of technologies, so if you think there’s any possibility these technologies might help you scale, then I encourage you to sign up and give it a spin.  Very interesting stuff.

Anyway, within AWS we’ll play with Elastic Compute Cloud (EC2), Elastic MapReduce (EMR), Simple Storage Service (S3) and Elastic Block Storage (EBS).

EC2 provides you with direct access to Amazon’s racks of humming servers in the cloud.  Thanks to the miracles of modern virtualization technologies, an entire machine can be bundled up as an image file – an Amazon Machine Image (AMI – think of an OS’s DVD but with software that you specify, configured as you please). Once bundled into an AMI, your computer can be ‘installed’ onto any number of machines within the cloud.  Nifty.

Such machines can be bought for long term use (at quite affordable rates), or they can be used on an as-needed basis for an hourly charge.  Hence the ‘Elastic’ in EC2.  They can also be managed programmatically, and until recently, this was pretty much the only way to develop jobs in the cloud.

Each machine that you use in EC2 has some local storage associated with it, but this is not persistent across sessions.  Once your instance goes down, the data is all gone.  To get around this and to provide an all-around storage solution, AWS provides S3 which is, to me, an overly-simplified distributed file system which is something of a pain to use.  It is also limited to file sizes of 5G which I find to be a significant annoyance.  The good news is that S3 seems to be both highly durable and reasonably performant.

A more recent entry in the AWS world of storage is the EBS facility.  Here, you can create arbitrary sized “disks” in the cloud which can be attached directly to your computer nodes just as though they were local disks.  Very handy.  They don’t have the same level of durability as S3 as they’re not distributed, but I haven’t had any difficulties with them and they’re very useful for putting together AMIs within the cloud.

Finally, EMR brings the Hadoop implementation of Google’s marvelous distributed map-reduce model into the AWS cloud.  If you haven’t read about map-reduce, well perhaps you should – it’s really powerful.  Hadoop is an open-source implementation of it done in Java and seems quite nice though it’s under very active development, so the APIs are still very much a work in progress.  Be prepared to change your code.  Even though it’s written in Java, any language can be used to implement your own applications, and it seems as though both ruby and python are used pretty extensively along with java and c/c++.

what map-reduce buys you

The first thing map-reduce buys you is something of a headache as fitting your application design into the map-reduce model isn’t so obvious.  That said, it may well be worth a couple aspirin as it brings a great deal to the game.

Consider an application we might find interesting which we’d like to place in the cloud – strategy back-testing. It seems to be a very natural fit for parallel processing as each strategy is independent of the others, so theoretically we should be able to run each one on it’s own virtual server without interdependencies.  While this is true, it’s rather easier than it sounds.  In order to implement our distributed backtesting platform, we first need to implement one that works on one node.  Fair enough.  We must then introduce a “chunker” which will break the overall set of simulations into bite-sized chunks.  We must then define an intermediate format for sending the strategies (or instructions) to each node.  And we must define an intermediate format for the results.  And then we need to implement an “assembler” which coalesces all of the results back into one result set which can be reported on or displayed etc.  And if one of the nodes fails, we have to notice it.  And if we notice that a node has failed, we need to figure out which job(s) it was responsible for and reallocate them to another node which presumably hasn’t failed…

And on and on.  The point is that this isn’t something that you’re going to get done in an afternoon.  Indeed, it’s unlikely that you could write a correct and complete specification of the system in an afternoon.

What map-reduce buys you is all of the distribution bits that I mention above.  The costs are that you will have to figure out how to model things such that they’re compliant with the map-reduce model and that the end result might not be as efficient as a hand-coded solution.  But when you can scale easily and cheaply, then raw efficiency may not be such an over-arching concern.

scaling in the cloud with elastic map-reduce

In order to get an idea how difficult it is to actually scale a solution, I implemented a simple program using real data and collecting some timings across a variety of  AWS/EMR configurations. To do this, I first had to code against Hadoop v18.3 which is what is supported in AWS.  (Initially I wrote it to v20.2 and was surprised at how much I had to change to back-port it to 18.3.)

The details of my test program aren’t important – basically counting various features from a set of heterogeneous files.  The files are approximately the same size ~1.8G and there were 10 of them, so the overall dataset is a bit over 18G.  I only do one map-reduce; I’m not chaining them together.  On a local machine – a pre-nehalem, dual-chip w/ quad-cores (8 total) xeon server with hardware raid – the process took 1068s or a touch under 18 minutes.

I was impressed to see that with absolutely no code changes, I was able to run the system on EMR without difficulties.  In fact, selecting the number of nodes to run against and the size of those nodes is just a matter of changing launch parameters.  Very nice.  AWS offers a variety of different specs on their servers.  I tested the classic ’small’ server which provides a 32-bit O/S, one core and 1.7G of memory as well as the ‘large’ server which comes with a 64bit O/S and 2X2 cores and 7.5Gs of memory.  There are other configurations available, but I didn’t test them.  In all cases I was running centos5.4.

The results are tabulated and graphed below.

performance locally and in various cloud configurations

time to complete vs. # nodes for large & small nodes

Scaling isn’t exactly linear, but it is good and it is simple and it is cheap & on-demand. Clearly for any given use-case, it will make sense to do tests like this to figure out what makes the most sense.  Another dimension to consider is cost.  Even partial hours count as a full hour, so putting big or many boxes against jobs that don’t take long to run isn’t cost effective.  That said, even though I fecklessly ran jobs which took as little as 10 minutes while paying for 60, the entire costs associated with developing and running the example I present here was less than $4US.

One thing I noticed is that the time to initialize instances seems to grow as you increase the number of nodes, though I didn’t quantify this.  In some cases, it was really quite long before real work started getting done – several minutes at least.  Getting this to work for interactive processes will take some thought and, probably, some dedicated nodes.  Another things to note is that although my local server did a fine job for itself, if we were to increase the dataset by a factor of, say 100, we’d reach a point where it simply couldn’t complete the operation whereas the cloud isn’t so strictly bounded.  It’s also pretty interesting that my local machine with 8-cores and 24G of RAM and pretty decent RAID subsystem got soundly spanked by eight of Amazon’s cheapie 1-core boxes.

Given that it is cheap, massively and simply scalable and not terribly difficult, one could say that if you don’t spend some time with your head in the clouds, then it’s possible you’ve got it stuck in the sand.

The example I’ve chosen is, I think, very nice because it’s a portfolio-oriented strategy, which is pretty much the only kind I care to explore; it’s also based around the concept of pairs trading, which is something which most can easily relate to; and finally, it’s already public domain and yet almost certainly has some juice in it for those who care to understand it and extend it intelligently.

The example comes from the blog of a company, Palantir which does (something like?) analytical / decision support software for both finance and intelligence-gathering services (quants and spooks – spooky quants?).  The specific example is here and is described thusly:

In this study we explore a trading strategy that isolates pairs of instruments within a sector that are highly correlated. We enter a trade if the price paths of these instruments diverge, going long one instrument and short the other, with the assumption that their price paths will converge.

We construct our strategy in four parts: (1) isolate the target set of tradable instruments, (2) choose a rule for finding correlated pairs within that set, (3) pick criteria for entering trades, and (4) test the trading strategy.

…

For this study we want to isolate pairs of instruments whose correlation is above a certain threshold. At the beginning of each quarter, we restrict our target stocks to the top 5 in the sector by 252-day correlation to the S&P 500. Then our Strategy uses a metric that, for a given group of stocks, outputs a list of pairs with a correlation above our target threshold of .60 over the same time period.

…

Across every day and every pair in our trading list, we check the 21-day z-score (number of standard deviations from the mean) of the difference between the two series. We classify the stocks as diverging when the absolute value of the z-score is between 1.5 and 3.0. When this condition is met we short the stock that is rising and long the stock that is falling, sizing our position relative to the z-score and the number of pairs currently trading.

~(sectors) ; reframing the strategy

While I’ll implement a strategy broadly based on their description, I will use some poetic license to modify it in ways that seem to me sensible.  First, I have good experience writing equity strategies which deal with sectors and have pretty uniformly found that sectors, as defined by S&P or whomever, are more trouble than they’re worth.  Instead, looking at correlation matrices and determining on that basis who should be paired, is much more fruitful in my experience.  Tr8der has some very nice posts illustrating (very prettily!) a similar point here and here.  (Actually, his point is a bit broader and represents an area of possibly promising development of this strategy which I won’t pursue here.)

So, we’re going to chunk the idea of explicitly using sectors.  Instead, we’ll simply construct, on a monthly basis (Palantir did this quarterly) a TrailDays correlation matrix from which we’ll extract the set of pairs which have a correlation greater than MinCorrelation.  On a daily basis, we will calculate the ZScoreDays z-scores of each of these pairs and sort them by absolute value, tossing those whose values are less than MinZScore or exceed MaxZScore.  Since I don’t like the idea of having an open-ended number of pairs/positions in my portfolio, I will constrain the portfolio to a maximum of N names and will Allocate $1M to the strategy.   Finally, we will look at the effect of trading an EvenlyWeighted portfolio or a z-score weighted portfolio (as Palantir had done).

The initial ‘universe’ of equities across which we will calculate our correlations will be filtered from my database of daily equity data going back to 2005 where the closing price is over $1, the average daily dollar-volume is over $50M and the data is relatively clean (few if any gaps).  This yields an initial universe of ~600 equities and ~180K pairs.

Assuming that we keep N above 20 or so and we keep our allocation constant at $1M, and assume execution at the close price, we should expect our results to be reasonably believable as it’s unlikely that the size we’re trading in would distort the market.  Scaling any strategy presents its own set of challenges which we’re not going to go into for this example, but an obvious direction for scaling the strategy would be to simply run it over intraday data and scale into and out of positions as conditions permit.

Ok, so that’s what our strategy is going to look like.  Now, how do we “make it so”?

structure of a stratpart

Within Stratbox, strategies are represented as composites made-up of ’stratparts’ each of which has its own metadata descriptor containing parameters.  Stratparts can be composed together within a strategy to enhance or modify the behavior of a strategy.  Thus, one can have a stratpart which implements the above logic paired with another stratpart which will e.g., modify the portfolio to impose beta neutrality, damp volatility or some other form of reporting or risk management.  Given that this example is already complexish, we’ll just stick to the one stratpart.

A stratpart has a few key elements.  One is the metadata descriptor which allows the strategy container (stratbox) to interact with it for back-testing, optimization or forward-walking purposes, or to allow a ‘trader’ to interact with it dynamically during run-time (the so-called ‘gray box’).  Beyond that, it has an essentially event-driven design.  The method quote() is called when relevant (i.e., subscribed) market data arrives.  The method execution() is called when executions for orders that originated with this strategy are received.  And the method strategyEvent() is invoked on a variety of ‘interesting’ events that the strategy implementer might care about.  None of these methods are required, so in its simplest form, a stratpart is a very simple piece of code to write.

services in a stratpart

Once one has complied with the minimal requirements for implementing a stratpart, one is rewarded with a variety of useful services: subscription-based market data feeds as well as a fast snapshot db and an in-memory historical db for quick correlation analyses cover market data needs.  Position management is handled at both the strategy level and at the exchange level (if you’re trading across multiple exchanges).  There is more, including performance analysis (e.g., sharpe, and its various analytic friends) and hierarchical cash allocation within and across strategies, but we won’t look at these in this example.

implementing metadata

So, the first thing we need to do is define our metadata.  This is done through the use of a conventionally-named static method Descriptor().

public static Descriptor Descriptor() {

 ArrayList<Param> params = new ArrayList<Param>();

 String cd = "Unleveraged capital to apply";
 params.add(new Param(InitialBalance, cd, false, 1000000.0));

 String n = "# of instruments to hold in portfolio";
 params.add(new Param(N, n, true, 20));

 String tc = "Trailing days over which to calculate correlations";
 params.add(new Param(TrailDays, tc, true, 252));

 String mc = "the minimum correlation of pairs to consider";
 params.add(new Param(MinCorrelation, mc, true, .75));

 String zd = "trailing days over which to calc z-score";
 params.add(new Param(ZScoreDays, zd, true, 21));

 String mizs = "minimum z-score";
 params.add(new Param(MinZScore, mizs, true, 1.5));

 String mazs = "maximum z-score";
 params.add(new Param(MaxZScore, mazs, true, 3.0));

 String ew = "evenly-weight portfolio? (or weight by z-score)";
 params.add(new Param(EvenlyWeight, ew, true, true));

 Descriptor _desc = new Descriptor
   ("puppetmaster.strats.Pairs", _Desc, params);

 return _desc;
}

public Pairs(Strategy strat, Descriptor d) { super(strat, d); }

Not the most interesting chunk of code, but pretty straightforward and its inclusion opens up a lot of functionality within the environment as each of these parameters can now be systematically or manually modified in-flight.  The third parameter to the Param constructor, a boolean, indicates if we want to allow the system to optimize this parameter.  Thus, the allocation/initialBalance parameter isn’t optimizable as it wouldn’t make any sense to do so.  Even so, we can (ourselves or through an allocation algorithm) modify this value to force the strategy to lighten or extend its financial footprint.  The others are fair game for optimization.

The constructor is also required – with that exact signature.  This is pretty much it for the ‘boilerplate’ code required.

‘installing’ the stratpart

Now that we’ve met the minimum requirements for a stratpart, we can install it into stratbox.  This requires adding one line to your personal stratparts.xml config file.  Like so:

<?xml version='1.0'?>
<stratParts>
 <descriptor class='puppetmaster.model.strategy.FwdWalker'/>
 <descriptor class='puppetmaster.model.strategy.StrategyPortfolio'/>
 <descriptor class='puppetmaster.strats.Pairs'/>
 <descriptor class='puppetmaster.strats.AMBO'/>
 <descriptor class='puppetmaster.strats.meta.AdjMeanReverter'/>
 <descriptor class='puppetmaster.strats.meta.AdjTrendFollower'/>
 <descriptor class='puppetmaster.strats.meta.Binary'/>

Once we’ve informed stratbox of the new stratpart we’ve written, we can view it within our library of stratparts within the strategy wizard in stratbox:

Our Pairs strategy in the Strategy Wizard's stratparts library

And all of the parameters we defined within our descriptor show up on the next page where we can parameterize or optimize the strategy:

Configuring the Pairs stratpart within the strategy wizard

This means that the stratpart can be combined with other stratparts to create new strategies as I’ve described  and that it can partake in the rich set of functionality within the stratbox gui.  Including actually trading.  But first, we must make the strategy actually do something…

implementing event handling

To that end, we must implement the event handling methods I’d described previously.  Since we’re not going to do any detailed execution handling in this strategy, we’ll only implement the quote() method and a subset of the events sent to the strategyEvent() method. In particular, we’ll listen to activation events (so we can initialize cleanly), changes to our descriptor and changes to our positions.

/** listen for select strategyEvents */
public void strategyEvent(StrategyEvent event) {
  super.strategyEvent(event);
  switch (event.type) {
    case Activated:            _init();                 break;
    case DescriptorChanged:    _readDesc();             break;
    case PositionChanged:      _posnChange(event);      break;
  }
}
<pre>

For quotes, we’ll just listen to the Beginning of Day (BOD) event.  If we wanted to trade intraday, we’d have to listen to tick events, but we’ll leave that for another day…

From a high-level, our code is pretty simple.  We determine the universe of pairs we might trade, we select from among them, we determine what our new portfolio should look like and then we trade to transition from our current state to our desired state.  That’s it.

/** mkt data goes here, but we only listen for the BOD
 * ("Beginning of day") event */
public void quote(Quote q) {
  if (q.type() != Quote.Type.BOD) return; // daily strat...

  try {
    List<_Pair> pairs = _getPairs();
    List<_Pair> sel = _selectPairs(pairs);
    List<Position> dp = (List<Position>)_desiredPfolio(pairs,sel);
    _Log.debug("PORTFOLIO: "+dp);
    _trade(dp);
  } catch(Exception e) { _Log.error(e.getMessage(),e); }
}

the messy details

Digging into the details gets more complex, but not horrifically so, as stratbox already has basic analytics like correlation matrices, z-scores, volatility built into it.  Likewise with basic portfolio analytics and transformations. Thus, our code to trade a (pairs-based) long-short portfolio with an arbitrary number of elements is surprisingly simple:

/** given a desired portfolio, trade to make it so */
void _trade(List<? extends Position> desiredFolio) {
  PositionRecord[] currFolio = posns();
  // Given our current portfolio (posns) and our desired state (dfolio),
  //  we generate the set of orders which will transform from the former
  //  to the latter
  //
  List<Order> orders = PortfolioComposer.TransformPortfolio
    (currFolio, desiredFolio, _orderF(), _strat, _strat.account());

  // place the orders for execution
  for (Order order : orders) {
    try {
      order.setType(Order.Type.MOC); // we trade at the close only
      _subscribe(order.contract());
      _execP().placeOrder(order);
    } catch (Exception e) { _Log.error(e.getMessage(), e); }
  }
}

As promised, I provide the full source listing below for those interested, but won’t over burden the discussion with a complete description of every little piece of it.

a telling omission and some results

Early in my career, working as a software engineer in a wall st front office technology department, I was befriended by a manager who told me he had once been a trader.  I asked him why he wasn’t a trader anymore.  He quipped:

I was half of a talented trader.  I knew, in my bones, when to get into a trade.  Sadly, I was never really sure when to get out…

It seems that the author of the Palantir example suffers a similar characteristic to my old friend (or is just being understandably cagey) – she specifies an entry condition, but not an exit!  Since she doesn’t impose any costs for trading, this has no big side effects, but in stratbox we do impose (pretty onerous -> realistic!) costs for trading.

In any case, since this is just a simple example I’ve maintained the ‘no exit’ policy and simply exit a trade once its pair goes out of the acceptable ranges for either correlation or z-score.  This has a bad effect on the strategy as you end up churning a lot of trades at the high end of the z-score range.  That is, if we set our upper boundary for z-score to be 3, then we might enter and exit a position multiple times as the z-score oscillates between, say, 2.9 and 3.1.  A more careful implementation could minimize this effect.

Another difference imposed by our more realistic simulation as opposed to the original example is that I’m restricting the portfolio size explicitly and explicitly limiting myself to trading a given contract within one pair as opposed to allowing a contract to trade across multiple pairs (invoking higher costs but allowing positions to offset and thus providing a neat little optimization – assuming trading costs nothing!).

All the same, the strategy clearly has some positive characteristics and when parameterized to take advantage of its naive exit strategy (by making the z-score low-end quite low, e.g., 0.5) takes on the character of a hedged trend-following strategy.  How so?  Well, while we are getting churned around the high-end (entry) z-scores, our profitable exits only happen once the z-scores have diminished (the prices have converged) considerably.  So we take more, smaller losses while our winners are held a bit longer.

some Pairs results in '10

Adding to my portfolio size improves things generally (and also increases the strategy’s scalability) while increasing the MinCorrelation parameter seems to hurt performance (not sure why, honestly – perhaps because if they’re very highly correlated and already diverging so strongly it may mean they’re actually diverging fundamentally). Returning to the topic of costs, each of these strategies incurred costs of over $5K for the period in question. And returning to the parameter which governs the weighting strategy employed, although I haven’t illustrated it here (as I haven’t looked at it very carefully), my initial finding is that it changed very little. 

There’s a great many areas where this strategy could be improved – that’s the fun, after all! – but hopefully even in its primitive form this example brings to light both an interesting baseline portfolio pairs strategy, while answering the question of how one goes about concretely implementing it within stratbox.

The entire strategy weighs-in at ~400 LOC including comments, boilerplate and individualized imports.  The source is here: Pairs.java.

'optimization' or 'search'?

We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain.  Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode.  I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.

From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good.  Doesn’t mean they’re actually good.  In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively.  I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do.  They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.

They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations.  If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…

Anyway, I felt that had to be said…

Now, I’ll open the floor to a quick illustration of the initialization process described previously and an example of historical forward-walking.

At this point, I’m vested in the inquiry we’d started with the regime-switching post and will apportion time each week towards implementing something like what we’d described there.  I expect the steps to be something like:

1. historical fwd-walking which I’ll illustrate today
2. real-time fwd-walking, that is, dynamically adjusting a live strategy’s parameters based on a continuously repeated ranking of a concurrent ‘live optimization’ of the same strategy
3. real-time allocation to a portfolio of strategies based on regime-switching where regimes are defined by the performance of a heterogeneous set of strategies we run concurrently (again ‘live optimization’)
4. peace on earth

I may have some of the steps wrong, but am hopeful.  We’ll adjust as needed.

If interested, please click on the below image/link which will hopefully take you to a screencast.  This is my first-ever attempt to put a screencast in the blog, so please let me know if you have technical difficulties or suggestions on how to improve the experience.  Or if it’s just a big waste of time!

In the screencast I give a quick illustration of a simple backtest, a simple parameter optimization and finally a fwd-walk.  I’ll appreciate your feedback.

>click<

poor Jorge wasn't ready...

In this post I’m going to revisit some of the topics discussed in the recent ‘containing a strategy‘ and ‘multi-strategy trading with regimes‘ posts, focusing on the process of assembling a strategy and its context in preparation for its launch into any of a variety of modes.

I recently realized that – from the perspective of a strategy container – the process of walk-forward testing is remarkably similar to the regime-switching model we’d discussed previously.  Up until now, I’ve employed walk-forward testing in an ad-hoc manner by taking an existing strategy and then writing a little driver very much like a unit-test scaffolding which would walk the strategy forward, permuting parameters based on previous performance.  Not a general solution, but straight-forward as I employ the strategy parameter optimizer from stratbox in this kind of a toolkit use-case.

I sat down to write one of these walk-forward scaffolds yesterday and started to think about how I could generalize the solution and roll it into stratbox’s GUI and it occurred to me that I could likely kill two birds with one stone…

walk-forward testing

–

I imagine there are different ideas/implementations of it, but for me walk-forward testing is the case where you are using repeated historical parameter ‘optimizations’ to dynamically modify the parameters on a back-tested strategy.  For example, say we have strategy X with parameters p1 and p2.  Let’s say we want to walk-forward test it from the beginning of the year until now.  We’ll need a look-back period for optimizations – say 20 trading days – and a frequency with which to apply the results of our optimizations – say every 5 trading days.  So, on a weekly basis we’re going to look back at the prior (rolling) month performance and adjust our parameters to match the best performer.  How you define ‘best’ is your business but I’ll use something simple like sharpe here.

Continuing with our example, we’ll say that p1 will be permuted 3 ways with values from { 1, 2, 3 } and p2 will be permuted across 4 values { “a”, “b”, “c”, “abcd” }.  So we have a basis of 12 strategies for optimization and we’ll call these O1…O12.

To perform a walk-forward test on X from 2009/01/01-now, we will begin a parallel back-test on O1..O12 starting 20 trading days before Jan1.  On Jan2 we will look at the prior ~month’s results and pick the strategy who had the highest sharpe ratio and clone him and start testing him “walking forward.”  Each week, we’ll repeat the process except that instead of cloning the winner as we did the first time, we’ll just make sure his params are the same as the new winner’s params.

Repeat this process until ‘now’ and you’re done – you’ve performed walk-forward testing on X.

–

Let’s think about what’s happening here.  There are 12 strategies running from ~2008/12/1 and another strategy running from 2009/1/2; all strategies run until now.  There’s also the activity that “we” have done during this process.  What we are doing is the role of a meta-strategy just as we’d seen in the regime-switching case.  The walk-forward meta-strategy needs to watch a set of ‘optimization’ strategies and permute another target (“X marks the spot”) strategy on a regular basis to match parameters with the prior period’s winner.

differences vs. allocating to strategies with regimes

Let’s imagine that we’re going to take the 13 lovingly-named strategies O1…O12 and X, along with the newly revealed meta-strategy we’ll dub “M”, and put them onto a live environment instead of a historical back-test.  Suddenly we’re pretty close to what we’d described previously.  The biggest difference is just the temporality of the strategy context: historical or real-time.  Another difference is that O1…On (let’s generalize just a bit) are no longer necessarily all the same strategy – they may be completely different. X is also no longer a strategy, but a portfolio of strategies:  a replicated subset of O1…On we’ll call X1..Xm.  “Best” is no longer a unary choice but must be rich enough to allow M to support some kind of an allocation algorithm across the X family.

OK, so there are some significant differences.  But from the perspective of the object model on which all of this can be implemented, the differences may be limited enough to warrant tackling together.

ready to launch

Before looking at the workflow of a strategy launch, let’s consider all of the various modes we support with our design:

• Historical: back-test single strategy
• Historical: parameter-based optimization of strategy (decomposes to many back-tests)
• Historical: walk-forward testing of strategy
• Real-time: SIM – run single strategy with real-time market data and an exchange simulator
• Real-time: LIVE – run single strategy with real-time market data and a live connection to OMS/broker/exchange
• Real-time: LIVE+SIM – run two cloned instances using the same market data, but with one routing orders to live venue(s) and the other to a simulator.  This is mostly useful for the development of a simulator or to evaluate the quality of simulation for a given strategy.
• Real-time: optimization – run many strategies, permuted differently across live data but routing to a simulator.  I don’t know what this is useful for, but I find it entertaining… The screenshots from the regimes post were illustrating this mode.
• Real-time: meta-strategy – our mythic regime-switching allocator will be a nice example of this mode.

-

Consider the following sorta-flowchart illustrating the preparation for launch of a strategy.

click for larger version...

The process has 4 or 5 stages depending on whether we’re being “meta” or not – that is, if we’re using optimization of any sort.  The highlighted parallelograms are the concrete end-result(s) for each of their respective steps.

• Assemble Strategy Context – The basic channels of I/O for a trading strategy are its sources for market data and the exchanges where trades can be executed.  The MktDataSrc and ExecutionPlatform abstractions need to be specified.
• Assemble Strategy – Whatever mode we’re looking to employ will require a strategy either to itself go and trade for us or else to serve as the seed or prototype for our activities.  In this step we define that strategy.  For me, this can be a composite process as a strategy is made up of an arbitrary number of stratparts.  The assembly of a strategy can be done manually with a gui or by loading a strategy from a file or other store.  Once the parts are identified, they can be bound together to form a strategy instance.
• Configure Strategy – Once assembled, the strategy needs to be configured.  This may just mean we need to parameterize it appropriately for our use.  If we’re being meta and require optimization then we’ll also need to describe the parameters to modify and the ranges for each.  This gets boiled-down into an OptimizationDescriptor which is composed of a prototype strategy and the parameter/range data required to perform the requisite back-tests.
• Configure Optimization or Meta-strategy – If we’re not running an optimization or utilizing a meta-strategy, this step is skipped.  Otherwise, this is where the optimization and/or meta-strategy should be built and configured.
• Review & Launch – Anytime you might loose a piece of software on an account with real money in it, it behooves you to pause and reflect for a moment.  Now is that moment.  If, after that moment of reflection, both you and whatever risk measures you employ agree that it’s OK, then it’s time to launch.  Happily, this is easy because we now have everything we need.  Again, the meta cases are a bit more involved as the strategies to be optimized or managed must be first generated based on the optimization descriptor we’ve inherited from earlier steps.  Then it’s a matter of assembling the pieces previously selected and configured and away we go.

-

Anytime you might loose a piece of software on an account with real money in it, it behooves you to pause and reflect for a moment.  Now is that moment.

I was taught that writing software is easy if you can express the problem you’re looking to solve and its solution in clear and precise English.  I’m not sure about this being grammatical, clear or precise, but hopefully it’s enough to illustrate a means of uniformly modeling and implementing these beasties across a variety of use-cases.

For me, at least, writing this description is helpful as it forces me to walk through the various twists and turns in some detail before digging into the code.  When I get the integrated walk-forward testing working I’ll try to provide some illustration of its use and will look-ahead to developing a regime-switching example using the models/mechanisms we’ve been discussing.

rebranding opportunity?

A friend of mine pointed out an article he came across on his bloomberg terminal today which reminded him of a strategy I’d described to him sometime back and which we’ve been trading over the past year or so with good results.

To the great chagrin of some of my partners, I even wrote a few posts about the phenomenon underlying our strategy and its evolution as we capitalized on it.  Eventually, they persuaded me to shut up already, but the outline was there for all – including Goldman! – to see.

My first post on the topic, “unsung virtues of a dynamic hedge” published June 4th of last year, was pretty coy and didn’t mention the source of alpha itself but talked about enhancing it with a dynamic hedge.

My next post on the topic, “to dream” was published July 14th of last year and laid out the exploitable discrepancy of the market’s behavior.  Interestingly, the data I provided in that posting went back the same amount of time as in Goldman’s piece.

I explicitly wrote one last time about the strategy in “evolution of a strategy” wherein I detailed the process by which we’d been evolving the strategy.

Now, one of the more entertaining things about having a blog is that you get to see who is viewing your content.  I’m happy to note that all of the major IBs are represented including a variety of distinct IPs within Goldman.

Now, I’m not accusing them of stealing my ideas or anything untoward like that… but I’ll admit that I am wondering how long it’s going to take them to make similar observations across markets beyond US Equities…

Read on for the Bloomberg article…

From the Bloomberg article:

By Alexis Xydias

March 12 (Bloomberg) — Investors should take advantage of others’ “fear” of the night, according to a study by Goldman Sachs Group Inc. that shows holding U.S. stocks overnight since 1993 would have quadrupled an investment.

Buying futures on the Standard & Poor’s 500 Index, or a fund that replicates the benchmark for U.S. equities, just as the trading session ends and selling them when the market opens the next day has yielded 309 percent since 1993, New York-based analyst Peter Berezin wrote in a report sent to clients today. The inverse strategy lost 58 percent.

Investors and traders may have become more reluctant to hold securities overnight, when they’re unable to react to market declines abroad, Berezin wrote. The S&P 500 has plummeted 54 percent since reaching a record in October 2007 as the crisis in credit markets and the collapse of banks including Lehman Brothers Holdings Inc. hammered equities, while the Chicago Board Options Exchange Volatility Index, the so-called gauge of fear, has more than doubled.

“A large number of market participants are averse to holding overnight positions, which causes them to sell at the close (thereby depressing intraday returns) and buy at the open (thereby inflating overnight returns),” Berezin wrote. “Such aversion to overnight risk is likely to be higher during bear markets.”

Overnight Spread

The difference between market-close and market-open prices has widened to 9 basis points since October 2008, the study said, from a “long-term” average of 5 basis points. A basis point is 0.01 percentage point.

The best way to benefit is to hold the S&P 500 at night, while short-selling it during the day, Berezin wrote. The strategy would have returned 507 percent in the period, the analyst said, while acknowledging the increased costs of such a trading-intensive strategy. Short-sellers sell borrowed securities on expectations they will be able to repurchase them at a cheaper price before their loan is due.

The S&P 500 reached 666.79 on March 6, the lowest intraday level in more than 12 years. The benchmark index for American equities has fallen 20 percent this year.

“Even when one is pondering less trading intensive strategies, the analysis above suggests that there is a cost to be paid for avoiding overnight risk,” Berezin wrote.

Just after the dot.com craze, a brilliant friend of mine (who had just sold his web consulting startup) decided to write a book.  The premise was glorious.  A bunch of clever college-age kids formed a startup to predict the stock market.  The method they used was to constantly comb the web with ultra-sophisticated algorithms which would run across giant server farms overnight and ultimately generate tomorrow’s headlines.  Based on the headlines that their system generated, they would place trades that would take advantage of these predicted events.

Sadly, my friend never went on to complete his book, so I don’t know how it all turned out.  (Instead, he went on to start another successful company, this time in the field of robotics.)  While he was writing it, I loved getting new drafts as they were filled with clever ideas.  But the core idea of predicting headlines and then using those headlines to trade always struck me as especially cute.

For those of us without access to news-predicting algos, writing strategies based on the news is rather less straight forward, though there are a growing variety of products and services aiming to fill the gaps.  Today must have been trading-the-news-day as I found a few articles on the topic in my mailbox and even received a cold call from a vendor, Need to Know News, with just such an offering.  Below I’ll look at some of these offerings and consider some of the issues involved in writing trading strategies based on the news.

The idea of trading the news resonates with me as it was one of the first things I tried to automate.  In particular, I spent some time looking to trade crude and natural gas futures based on their respective weekly EIA reports.  This experience led me to a couple conclusions.

The market knows before the news wires do. This is a big problem for two reasons.  The first is just plain-Jane latency, and I know that vendors are now effectively reducing latency to the sub-second level, but the market still knows first.  The second issue is deeper and is captured nicely by a quote I heard (sorry – I don’t remember where) which went something like:

“capital markets are the original social networks”

Which suggests that markets have their own internal languages and understandings.  Thus, translating a market-impacting event, like an EIA report, into a human-readable form to reason about it (even if there’s no human doing the reasoning) and then action it back into the market struck me as a necessarily lossy transformation.

Another way of seeing this is just to consider: what to do based on the news?  Sell a build of inventory?  Maybe with your money.  Compare the number against expectations?  Whose?  The semantic content that exists in the market seems to be intrinsically richer than that one might extract from a news wire.  But this problem of actioning a news item brings up the next issue.

The problem of history.  One of the wonderful and horrible things about market data is that there is a lot of it going back a long ways.  This is an expensive pickle to manage, but it at least means that you have the ability to look back almost arbitrarily far into the past to see how markets responded to various conditions.  This isn’t so true with most news wires.

To some degree, these issues are being addressed by vendors.  And some of them will be addressed by how people utilize the news feeds.  If, instead of trying to write a strategy based wholly on the news, I try to improve an existing strategy by annotating its model with data gleaned from a feed, I might wind up with much better results.

Indeed, one of the papers I came across today is a research report from Securities Industry News (incidentally, my first tip-off that Citi was going down the toilet was when my management told me I could no longer keep my subscription to that fine periodical).  In it, market participants indicate that they expect to improve existing algos more than create brand-new ones.  But they also failed to complain about the lack of back-testable feed histories, so…

The other paper is a much more detailed quantitative exposition on how to build a news reading algo and some statistical analysis on how well it annotated the market.  This one is the work of Andrew Lo’s AlphaSimplex Group and as such is required reading for anyone who really wants to implement such systems.

Ever since my first experiences with trying to trade energy commodities based on the EIA report via news feed, I’ve been pretty skeptical of the utility of trying to algorithmically trade the news.  That said, like everything else in this space, there’s an incredible amount of innovation going on and an incredible number of seriously smart and motivated folks working to ensure that this will be a productive path for those with the ability to tackle the formidable complexities presented.

–

Updated: added link to Need to Know News’ site

While Carl Sagan’s famous formulation introduced a generation to the vastness of the cosmos, more recent history suggests that his memorable term might now be more aptly applied to financial extents: our deficits and debts, perhaps, to the economically or politically minded. But for those of us with the markets on our mind, the term has to evoke the enormity of the data we create and must manage every day. We’ve recently been working with the NYSE’s TAQ data in an effort to integrate it into StratBox’s back-testing and optimization capabilities. And the enormity of the data is really just staggering.

Each day, the NYSE publishes all of the day’s quotes and trades as well as some reference data. Compressed, the data will just about fit onto a DVD. For one day. A DVD. Compressed. It’s really mind-boggling. A year of the stuff, uncompressed, will require over a petabyte of storage. Over 1,125,899,906,842,624 bytes. And that’s just the US Equities markets. You want options data, too? I hope your uncle is named EMC, because just managing the data is going to be a challenge…

“Information about money has become almost as important as money itself.” — Walter Wriston, former Chairman of Citicorp

The enormity and profile of market data far exceeds the capacity of traditional RDBMSes. While RDBMSes continue to expand their usable capacity – we have used partitioned tables with nearly a billion rows of market data which have performed astonishingly well – they simply can’t deal with the kinds of quote volumes modern markets are generating daily. This has spawned a host of specialized timeseries database products, like the grandaddy: Sungard’s FAME which I’d used back in the 90’s to write programs to calculate bond indices at JPM, to more recent offerings like Vhayu and Kdb+. These timeseries oriented data products undoubtedly have many distinguishing characteristics and features, but they share one immutable characteristic: they are unbelievably expensive – in some cases a single developer seat costs in the high 6-figures for an annual license.

Thus, while no doubt missing out on some of their high-end features and niceties, we’ve decided to seek solutions from some of the original purveyors of petabyte-scaled data: NASA and the NCSA through their HDF5 system. Designed to support vast scientific data stores and boasting sophisticated capabilities in support of parallel computing environments, it should be possible to get comparable performance to some of the high-end specialized finance products without the sticker shock. Indeed, it’s potentially another example of free software providing a meaningful contribution to finance.

In researching cost-efficient and highly parallel hardware solutions to pair with our emergent data solution, I’ve come to realize that open-source is expanding its reach into the hardware sphere.

This guy shares his experience and “recipe” for building a powerful and unique rendering cluster inside an IKEA filing cabinet. It’s admittedly on the funky side for even a SOHO operation, but it’s no joke – it’s more powerful than a lot of production blade servers used on wall st and it cost him less than $4K. He also includes a (very loosely described) spec for a more powerful next-generation version with some 50-Teraflops of capacity! So, while the data we’re having to deal with is growing at an incredible rate, the tools we have to manage it are growing proportionately for those who know how to leverage the work so many smart people are producing and freely sharing.

As my dad told me years ago:

“Good programmers write good programs. Great programmers steal good programs.”

At this point, we’re still in the “discovery” stage of our development of TAQ+HDF5 for StratBox, but as we progress I’ll periodically post some of our experiences.

—

UPDATE

Speaking of my dad, he saw this posting and pointed me to some class notes he’s been working on which describe the Google approach to massive information processing and fault-tolerance. Interesting and full of great links to both academic and industrial papers/sites on the topic.

I’ve been trading an increasing amount at the open and close of the equity markets using market-on-open (MOO) and market-on-close (MOC) order types and have found that the quality of executions varies enormously between the two types and have spent a bit of time analyzing the differences which I share below.

The quick scoop is that MOC orders almost invariably fill at the exchange’s published closing price, while MOOs vary very substantially from the published open price. Below I quantify my findings in a bit greater depth.

I looked at 846 recent MOO and MOC equity trades made over the past two months. Of all of these trades, only one MOC trade didn’t execute at the published close and the price I got was only off by one penny. Across all of the trades, I received the open or close price 55% of the time.

The remaining 45% of the time I varied from the open by an average of +.04% This means that I actually saw a slight price improvement in the average case. That is, if I was shorting then I executed at a price above the listed open and vice-versa for longs. I’ll take it!

In the below chart I plot the trades against their variance, positive or negative, from the listed open or close.

The biggest difference was a whopping 8.56% but at least it went in my favor. The stdev across all of the trades was .87% so we’re not looking at too disperse a grouping.

This data is a bit skewed as the majority of the MOO orders are going short. This is also a pretty limited universe of trades, so I’ll continue to look at the execution quality I’m getting on these order types and will revisit it if I see any interesting changes.

My interpretation is that my broker is making a best-effort to get a fair open price and they’re doing a creditable job of it. The exchanges are doing a nearly perfect job with MOC orders.

What impact does this have on my strategies? I’m not sure yet, but my first blush impression is that it might be worthwhile to try to get some price improvement over the posted open price as a means of both improving the results and extending the capacity of such strategies. It’s a favorable result as it means that strategies which back-test well on open/close data have a pretty good chance of executing well in reality.

A related issue, which I’m still researching, concerns the capacity of such strategies and may be the topic of a future post…

In particular, I’ve been looking at various market-breadth ETFs recently as possible fodder for the little dynamic hedger I’ve described before, and I’ve noticed an interesting behavior among several of them…

Like the majority of traders, they do better when they’re not trading!

That is, they actually display better performance at night than they do during the regular trading day; there’s more profit to be had in their gaps between sessions than there is during trading sessions. Below I quantify this observation more thoroughly…

I looked at the performance of a variety of ETFs across three super-simple “strategies”. The first of them trades overnight – buying the close and selling the following morning’s open. The next buys at the open and sells at the close. And the final one is, essentially, buy and hold though it’s implemented as a buy at close, sell and buy at following close. No costs or slippage are included. The table below details the results for the SPY (S&P 500), IWM (Russell 2000) and QQQQ (Nasdaq 100).

In all cases, the ETFs perform better overnight – they make more and have almost precisely half the risk, yielding very substantially better risk-adjusted performance measures (here sharpe is calculated with a constant 5% risk-free rate). These cases are highlighted in yellow in the table. Likewise, the worst performer in all cases was the “day-trader” who bought on the open and sold on the close. One thing to note about the below summary is that all data is taken from inception for each of the ETFs, so their duration varies! (For the SPY, since 1/29/1993; for the IWM, since 5/26/200 and for the QQQQ since 3/10/1999).

I tried the same analysis across a few other ETFs to see if the effect was a global one or a US-specific phenomenon. The results are mixed but likely negative for Europe (EWG), Japan (EWJ) and broad foreign markets (EFA). I’m not going to summarize that data here, but you can grab the excel workbook I used for this exercise here, though I warn you in advance that it’s about 2.3M. (If you find errors – please point them out!)

[UPDATE: .XLS 97-2003 version available here is ~3M]

For the US markets summarized in the table, the conclusion is pretty explicit – the great majority of the market’s profits are attained overnight. Even worse: in spite of the positive returns across all of these markets over the relevant periods, all of them show significantly negative intraday performance.

For long-only day-traders, this is the kind of result that should cause nightmares!

I came across and had to share this excellent vignette by Freeman Dyson on the perils of excess model parameterization…

In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, “How many arbitrary parameters did you use for your calculations?” I thought for a moment about our cut-off procedures and said, “Four.” He said, “I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.”