The example I’ve chosen is, I think, very nice because it’s a portfolio-oriented strategy, which is pretty much the only kind I care to explore; it’s also based around the concept of pairs trading, which is something which most can easily relate to; and finally, it’s already public domain and yet almost certainly has some juice in it for those who care to understand it and extend it intelligently.

The example comes from the blog of a company, Palantir which does (something like?) analytical / decision support software for both finance and intelligence-gathering services (quants and spooks – spooky quants?).  The specific example is here and is described thusly:

In this study we explore a trading strategy that isolates pairs of instruments within a sector that are highly correlated. We enter a trade if the price paths of these instruments diverge, going long one instrument and short the other, with the assumption that their price paths will converge.

We construct our strategy in four parts: (1) isolate the target set of tradable instruments, (2) choose a rule for finding correlated pairs within that set, (3) pick criteria for entering trades, and (4) test the trading strategy.

…

For this study we want to isolate pairs of instruments whose correlation is above a certain threshold. At the beginning of each quarter, we restrict our target stocks to the top 5 in the sector by 252-day correlation to the S&P 500. Then our Strategy uses a metric that, for a given group of stocks, outputs a list of pairs with a correlation above our target threshold of .60 over the same time period.

…

Across every day and every pair in our trading list, we check the 21-day z-score (number of standard deviations from the mean) of the difference between the two series. We classify the stocks as diverging when the absolute value of the z-score is between 1.5 and 3.0. When this condition is met we short the stock that is rising and long the stock that is falling, sizing our position relative to the z-score and the number of pairs currently trading.

~(sectors) ; reframing the strategy

While I’ll implement a strategy broadly based on their description, I will use some poetic license to modify it in ways that seem to me sensible.  First, I have good experience writing equity strategies which deal with sectors and have pretty uniformly found that sectors, as defined by S&P or whomever, are more trouble than they’re worth.  Instead, looking at correlation matrices and determining on that basis who should be paired, is much more fruitful in my experience.  Tr8der has some very nice posts illustrating (very prettily!) a similar point here and here.  (Actually, his point is a bit broader and represents an area of possibly promising development of this strategy which I won’t pursue here.)

So, we’re going to chunk the idea of explicitly using sectors.  Instead, we’ll simply construct, on a monthly basis (Palantir did this quarterly) a TrailDays correlation matrix from which we’ll extract the set of pairs which have a correlation greater than MinCorrelation.  On a daily basis, we will calculate the ZScoreDays z-scores of each of these pairs and sort them by absolute value, tossing those whose values are less than MinZScore or exceed MaxZScore.  Since I don’t like the idea of having an open-ended number of pairs/positions in my portfolio, I will constrain the portfolio to a maximum of N names and will Allocate $1M to the strategy.   Finally, we will look at the effect of trading an EvenlyWeighted portfolio or a z-score weighted portfolio (as Palantir had done).

The initial ‘universe’ of equities across which we will calculate our correlations will be filtered from my database of daily equity data going back to 2005 where the closing price is over $1, the average daily dollar-volume is over $50M and the data is relatively clean (few if any gaps).  This yields an initial universe of ~600 equities and ~180K pairs.

Assuming that we keep N above 20 or so and we keep our allocation constant at $1M, and assume execution at the close price, we should expect our results to be reasonably believable as it’s unlikely that the size we’re trading in would distort the market.  Scaling any strategy presents its own set of challenges which we’re not going to go into for this example, but an obvious direction for scaling the strategy would be to simply run it over intraday data and scale into and out of positions as conditions permit.

Ok, so that’s what our strategy is going to look like.  Now, how do we “make it so”?

structure of a stratpart

Within Stratbox, strategies are represented as composites made-up of ‘stratparts’ each of which has its own metadata descriptor containing parameters.  Stratparts can be composed together within a strategy to enhance or modify the behavior of a strategy.  Thus, one can have a stratpart which implements the above logic paired with another stratpart which will e.g., modify the portfolio to impose beta neutrality, damp volatility or some other form of reporting or risk management.  Given that this example is already complexish, we’ll just stick to the one stratpart.

A stratpart has a few key elements.  One is the metadata descriptor which allows the strategy container (stratbox) to interact with it for back-testing, optimization or forward-walking purposes, or to allow a ‘trader’ to interact with it dynamically during run-time (the so-called ‘gray box’).  Beyond that, it has an essentially event-driven design.  The method quote() is called when relevant (i.e., subscribed) market data arrives.  The method execution() is called when executions for orders that originated with this strategy are received.  And the method strategyEvent() is invoked on a variety of ‘interesting’ events that the strategy implementer might care about.  None of these methods are required, so in its simplest form, a stratpart is a very simple piece of code to write.

services in a stratpart

Once one has complied with the minimal requirements for implementing a stratpart, one is rewarded with a variety of useful services: subscription-based market data feeds as well as a fast snapshot db and an in-memory historical db for quick correlation analyses cover market data needs.  Position management is handled at both the strategy level and at the exchange level (if you’re trading across multiple exchanges).  There is more, including performance analysis (e.g., sharpe, and its various analytic friends) and hierarchical cash allocation within and across strategies, but we won’t look at these in this example.

implementing metadata

So, the first thing we need to do is define our metadata.  This is done through the use of a conventionally-named static method Descriptor().

public static Descriptor Descriptor() {

 ArrayList<Param> params = new ArrayList<Param>();

 String cd = "Unleveraged capital to apply";
 params.add(new Param(InitialBalance, cd, false, 1000000.0));

 String n = "# of instruments to hold in portfolio";
 params.add(new Param(N, n, true, 20));

 String tc = "Trailing days over which to calculate correlations";
 params.add(new Param(TrailDays, tc, true, 252));

 String mc = "the minimum correlation of pairs to consider";
 params.add(new Param(MinCorrelation, mc, true, .75));

 String zd = "trailing days over which to calc z-score";
 params.add(new Param(ZScoreDays, zd, true, 21));

 String mizs = "minimum z-score";
 params.add(new Param(MinZScore, mizs, true, 1.5));

 String mazs = "maximum z-score";
 params.add(new Param(MaxZScore, mazs, true, 3.0));

 String ew = "evenly-weight portfolio? (or weight by z-score)";
 params.add(new Param(EvenlyWeight, ew, true, true));

 Descriptor _desc = new Descriptor
   ("puppetmaster.strats.Pairs", _Desc, params);

 return _desc;
}

public Pairs(Strategy strat, Descriptor d) { super(strat, d); }

Not the most interesting chunk of code, but pretty straightforward and its inclusion opens up a lot of functionality within the environment as each of these parameters can now be systematically or manually modified in-flight.  The third parameter to the Param constructor, a boolean, indicates if we want to allow the system to optimize this parameter.  Thus, the allocation/initialBalance parameter isn’t optimizable as it wouldn’t make any sense to do so.  Even so, we can (ourselves or through an allocation algorithm) modify this value to force the strategy to lighten or extend its financial footprint.  The others are fair game for optimization.

The constructor is also required – with that exact signature.  This is pretty much it for the ‘boilerplate’ code required.

‘installing’ the stratpart

Now that we’ve met the minimum requirements for a stratpart, we can install it into stratbox.  This requires adding one line to your personal stratparts.xml config file.  Like so:

<?xml version='1.0'?>
<stratParts>
 <descriptor class='puppetmaster.model.strategy.FwdWalker'/>
 <descriptor class='puppetmaster.model.strategy.StrategyPortfolio'/>
 <descriptor class='puppetmaster.strats.Pairs'/>
 <descriptor class='puppetmaster.strats.AMBO'/>
 <descriptor class='puppetmaster.strats.meta.AdjMeanReverter'/>
 <descriptor class='puppetmaster.strats.meta.AdjTrendFollower'/>
 <descriptor class='puppetmaster.strats.meta.Binary'/>

Once we’ve informed stratbox of the new stratpart we’ve written, we can view it within our library of stratparts within the strategy wizard in stratbox:

Our Pairs strategy in the Strategy Wizard's stratparts library

And all of the parameters we defined within our descriptor show up on the next page where we can parameterize or optimize the strategy:

Configuring the Pairs stratpart within the strategy wizard

This means that the stratpart can be combined with other stratparts to create new strategies as I’ve described  and that it can partake in the rich set of functionality within the stratbox gui.  Including actually trading.  But first, we must make the strategy actually do something…

implementing event handling

To that end, we must implement the event handling methods I’d described previously.  Since we’re not going to do any detailed execution handling in this strategy, we’ll only implement the quote() method and a subset of the events sent to the strategyEvent() method. In particular, we’ll listen to activation events (so we can initialize cleanly), changes to our descriptor and changes to our positions.

/** listen for select strategyEvents */
public void strategyEvent(StrategyEvent event) {
  super.strategyEvent(event);
  switch (event.type) {
    case Activated:            _init();                 break;
    case DescriptorChanged:    _readDesc();             break;
    case PositionChanged:      _posnChange(event);      break;
  }
}
<pre>

For quotes, we’ll just listen to the Beginning of Day (BOD) event.  If we wanted to trade intraday, we’d have to listen to tick events, but we’ll leave that for another day…

From a high-level, our code is pretty simple.  We determine the universe of pairs we might trade, we select from among them, we determine what our new portfolio should look like and then we trade to transition from our current state to our desired state.  That’s it.

/** mkt data goes here, but we only listen for the BOD
 * ("Beginning of day") event */
public void quote(Quote q) {
  if (q.type() != Quote.Type.BOD) return; // daily strat...

  try {
    List<_Pair> pairs = _getPairs();
    List<_Pair> sel = _selectPairs(pairs);
    List<Position> dp = (List<Position>)_desiredPfolio(pairs,sel);
    _Log.debug("PORTFOLIO: "+dp);
    _trade(dp);
  } catch(Exception e) { _Log.error(e.getMessage(),e); }
}

the messy details

Digging into the details gets more complex, but not horrifically so, as stratbox already has basic analytics like correlation matrices, z-scores, volatility built into it.  Likewise with basic portfolio analytics and transformations. Thus, our code to trade a (pairs-based) long-short portfolio with an arbitrary number of elements is surprisingly simple:

/** given a desired portfolio, trade to make it so */
void _trade(List<? extends Position> desiredFolio) {
  PositionRecord[] currFolio = posns();
  // Given our current portfolio (posns) and our desired state (dfolio),
  //  we generate the set of orders which will transform from the former
  //  to the latter
  //
  List<Order> orders = PortfolioComposer.TransformPortfolio
    (currFolio, desiredFolio, _orderF(), _strat, _strat.account());

  // place the orders for execution
  for (Order order : orders) {
    try {
      order.setType(Order.Type.MOC); // we trade at the close only
      _subscribe(order.contract());
      _execP().placeOrder(order);
    } catch (Exception e) { _Log.error(e.getMessage(), e); }
  }
}

As promised, I provide the full source listing below for those interested, but won’t over burden the discussion with a complete description of every little piece of it.

a telling omission and some results

Early in my career, working as a software engineer in a wall st front office technology department, I was befriended by a manager who told me he had once been a trader.  I asked him why he wasn’t a trader anymore.  He quipped:

I was half of a talented trader.  I knew, in my bones, when to get into a trade.  Sadly, I was never really sure when to get out…

It seems that the author of the Palantir example suffers a similar characteristic to my old friend (or is just being understandably cagey) – she specifies an entry condition, but not an exit!  Since she doesn’t impose any costs for trading, this has no big side effects, but in stratbox we do impose (pretty onerous -> realistic!) costs for trading.

In any case, since this is just a simple example I’ve maintained the ‘no exit’ policy and simply exit a trade once its pair goes out of the acceptable ranges for either correlation or z-score.  This has a bad effect on the strategy as you end up churning a lot of trades at the high end of the z-score range.  That is, if we set our upper boundary for z-score to be 3, then we might enter and exit a position multiple times as the z-score oscillates between, say, 2.9 and 3.1.  A more careful implementation could minimize this effect.

Another difference imposed by our more realistic simulation as opposed to the original example is that I’m restricting the portfolio size explicitly and explicitly limiting myself to trading a given contract within one pair as opposed to allowing a contract to trade across multiple pairs (invoking higher costs but allowing positions to offset and thus providing a neat little optimization – assuming trading costs nothing!).

All the same, the strategy clearly has some positive characteristics and when parameterized to take advantage of its naive exit strategy (by making the z-score low-end quite low, e.g., 0.5) takes on the character of a hedged trend-following strategy.  How so?  Well, while we are getting churned around the high-end (entry) z-scores, our profitable exits only happen once the z-scores have diminished (the prices have converged) considerably.  So we take more, smaller losses while our winners are held a bit longer.

some Pairs results in '10

Adding to my portfolio size improves things generally (and also increases the strategy’s scalability) while increasing the MinCorrelation parameter seems to hurt performance (not sure why, honestly – perhaps because if they’re very highly correlated and already diverging so strongly it may mean they’re actually diverging fundamentally). Returning to the topic of costs, each of these strategies incurred costs of over $5K for the period in question. And returning to the parameter which governs the weighting strategy employed, although I haven’t illustrated it here (as I haven’t looked at it very carefully), my initial finding is that it changed very little. 

There’s a great many areas where this strategy could be improved – that’s the fun, after all! – but hopefully even in its primitive form this example brings to light both an interesting baseline portfolio pairs strategy, while answering the question of how one goes about concretely implementing it within stratbox.

The entire strategy weighs-in at ~400 LOC including comments, boilerplate and individualized imports.  The source is here: Pairs.java.

In the example below, we look at another ‘meta’ strategy named StrategyPortfolio which maintains a dynamic portfolio – P – of strategies which it will select from a set of strategies – S – running concurrently in simulation.  The constituents of P as well as their cash allocations and parameterizations will be rebalanced/adjusted regularly after an initial ‘out-of-sample’ period during which only the S strategies are run.

Apart education, the intention of this strategy, as I’d originally suggested here, is to ‘back-into’ a regime-switching strategy without attempting to directly quantify the regimes explicitly.

This has proved to be even more interesting than I’d expected, not so much because it performs particularly well (though it’s promising), but because of all of the things it has taught us.  In particular, the transitions are a killer and there are properties of strategies which (dis-)qualify them from being effective in such a scheme…

bad news good news

As before, I’m not going to take the time to write-up my results in any formal manner but will again rely upon a quick screencast of the software running.  The good news is that I’m figuring out how to edit these things, so it’s mercifully shorter than earlier screencasts…

Please click here to see the screencast.

I hope you find it interesting and will look forward to any comments or suggestions you might make.

'optimization' or 'search'?

We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain.  Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode.  I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.

From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good.  Doesn’t mean they’re actually good.  In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively.  I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do.  They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.

They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations.  If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…

Anyway, I felt that had to be said…

Now, I’ll open the floor to a quick illustration of the initialization process described previously and an example of historical forward-walking.

At this point, I’m vested in the inquiry we’d started with the regime-switching post and will apportion time each week towards implementing something like what we’d described there.  I expect the steps to be something like:

1. historical fwd-walking which I’ll illustrate today
2. real-time fwd-walking, that is, dynamically adjusting a live strategy’s parameters based on a continuously repeated ranking of a concurrent ‘live optimization’ of the same strategy
3. real-time allocation to a portfolio of strategies based on regime-switching where regimes are defined by the performance of a heterogeneous set of strategies we run concurrently (again ‘live optimization’)
4. peace on earth

I may have some of the steps wrong, but am hopeful.  We’ll adjust as needed.

If interested, please click on the below image/link which will hopefully take you to a screencast.  This is my first-ever attempt to put a screencast in the blog, so please let me know if you have technical difficulties or suggestions on how to improve the experience.  Or if it’s just a big waste of time!

In the screencast I give a quick illustration of a simple backtest, a simple parameter optimization and finally a fwd-walk.  I’ll appreciate your feedback.

>click<

Even if it’s hard to find good stuff, one must still look as there’s always more information that can help you than you can effectively process or retain.  A few weeks ago I was trying to formalize the expected profit function of an algorithm I’m developing and wanted to see what people had written about the topic.  I entered ‘define profit function for trading algo’ into google and was pleasantly surprised to see a paper entitled ‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani.  It doesn’t directly cover the topic I was looking for, but instead addresses a number of related topics I’ve been interested in for some time:

• the treatment of a strategy as an instrument in its own right
• composing portfolios comprised of strategies
• using regime switching techniques to manage portfolios of strategies

In this post, I’ll briefly review their paper, illustrate how one can easily model strategies in relevant ways using the strategy ‘object model’ I’ve described previously through an example, and conclude with some thoughts on how these kinds of strategies might be implemented and further explored.

‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani

Their paper begins:

This paper considers the problem of dynamically allocating capital to a portfolio of trading strategies. The allocation should be robust, and the capital allocated to a trading strategy should reflect the confidence in the expected profit that the strategy will make in current market conditions.

Good trading strategies exploit recurring market dynamics that can be more prevalent in some time periods than in others. Indeed, the concept of regimes is fundamental to financial markets, and much research has focused on the detection of regime shifts. In this paper, we consider a regime as defined by a set of trading strategies that exhibit similar performance in a given time period.

We consider different parameterizations of the same strategy as distinct in our ground set of strategies. The trading problem is to pick a distribution over the ground set that will achieve good performance in the current time period.

They start with some interesting definitions.  A trading strategy is defined as a function which takes a given market State and generates a convenient Answer “denoting all possible trading decisions”.  Strategy functions can also be parameterized and different parameterizations of the same function/strategy are viewed as distinct qualitative strategies.  They assume the existence of  the set of all strategies.  A regime is defined as a set of trading strategies who behave similarly over a given period of time where similarly is measured by the variance of a fitness function.

So they start with a set of strategies, a fitness function and some historical market data or state, and their first task is to identify the regimes.  They do this by running the strategies against randomly selected partitions of historical data, quantifying their respective performances with the fitness function and identifying the clusters deemed statistically significant.  Those sufficiently significant and similar sets of strategies are themselves the reasonable regimes while the set of strategies within all the reasonable regimes are themselves dubbed the reasonable strategies – the candidates for execution.  This might sound strange, but as I’ll illustrate with a concrete example below, this is actually a pretty effective and even intuitive way of defining regimes.

They define function stFuncDist as, for any regime, generating a distribution or weighting of the strategies within the regime. While classifyMarket is a function that generates a distribution across all regimes for any given market state. Both models can be “trained” and they do so by again iterating randomly across chunks of historical data adjusting the weighting constants of these two functions.

Finally, it’s time to trade and they can effectively use classifyMarket to determine a predominant regime and allocate capital as indicated by stFuncDist to the appropriate set of “reasonable strategies.”

They claim to have utilized this technique to generate the results charted in Figure 1, but they give little detail on the initial universe of strategies.  I believe they had one strategy which was parameterized into a set of qualitative strategies and run across one market, the CME’s Nasdaq emini.  I have no reason to doubt that they had exactly this result, but without more details on what they actually did, it’s hard to view it as very meaningful, though it certainly is suggestive.  An interesting thing about how they present their result, which I’ve seen in other academic papers, is that because they’ve precisely defined the set of possible strategies they look at, they can also calculate max and min possible returns over the relevant period.

Figure 1: Out-of-sample profit (middle line) versus possible min (bottom line) and max profit (top line) over all strategies for the period 2006-11-01 to 2008-08-28.

The paper continues with another section, but I’m going to stop my overview here as the last part is less interesting to me and the interested reader should of course read it themselves.

complex strategies or simple strategies combined

One of the broadest design questions I see for the development of effective trading algorithms is whether one should employ complex, customized algorithms or instead utilize simple algos which can be simply combined to produce emergent complexity.  Although the authors of this paper allude to “Good trading strategies”, I wonder if one can even forget the “good” part and instead worry about applying – even bad! – strategies at the right times.

There are a lot of things I like about this paper, but probably most of all it’s their definition and use of a regime.  One of my first posts describes a pair of strategies – a simple mean reverter and a simple trend follower.  These two strategies are about as dumb as you please.  And yet, their behavior is pretty interesting apart the observations I’d published at the time regarding the distribution of their returns.  During the trading day, I’ve taken to running those two, old, dumb strategies across a variety of markets in simulation mode.  Why?  Because it’s easy, costs me nothing and actually is pretty interesting in that the strategies really do cluster in the way Mlnarik et al describe, effectively partitioning the market into visible regimes.

a simple example: identifying regimes with strategies

In order to illustrate this, I created a desktop within stratbox onto which I ran these two qualitative strategies parameterized differently and across six different markets.  In total, I have 48 strategies running with 24 of the mean reverting (MR) variety and 24 of the trend following (TF).  In figure 2 and figure 3, they are sorted by qualitative strategy – all of the MR strategies followed by all of the TF strats.   In figure 4, they’re instead sorted by instrument at the end of the day after all of the strats had been rendered quiescent for the night by their schedulers.  You can click to see a bigger version of these screenshots.  On each, the left side of the screen shows simple charts of the relevant markets while the right contains a table in which each line item is a running strategy.  The first is at the beginning of the day. the second is shortly before going silent at 4pm and the third is shortly after 4pm.

Figure 2: variously parameterized MR & TF strats run in realtime+simulation mode. Sorted with the MR strats first. (click)

Figure 3: later the same day

Figure 4: at EOD, now sorted by instrument

You’ll note that at all times throughout the day, the MR and TF strategies effectively mirrored each other regardless of market.  Although I’m not going to illustrate it, I’m going to just go right ahead and assert that this is almost always true  based on what I’ve seen.  And this is at the heart of the paper’s authors’ insight regarding regimes – they can be effectively identified by examining the performance of even simple but well-defined strategies.

This suggests a simple means of implementing a similar design to that they described right on top of this example I’ve put together.

modeling meta-strategies

We need one more element to manage all of this – a meta-strategy – that is, a strategy for managing other strategies.  We support this idea in stratbox with what we call a controller strat.  The cannonical example of a controller strat is our implementation of the (Don’t) Panic button (the bright red button with an ! in the upper right hand corner) which will get all strategies out of the market with varying degrees of urgency and finality.  This is implemented as a controller strat which injects a stratpart into all running strategies.  The injected stratpart first deactivates its peers and then cancels outstanding orders and then gets out of the market with whatever urgency the user has required.

allocating to portfolios of strategies

In my example above, I have 48 strategies running in simulation across 6 futures markets.  Let’s imagine that at any moment in time, I will always have at least one strategy of each type (MR and TF) running for each market. Thus, I’ll always have 12 live strategies running against real money accounts and 36 candidates running against a simulator.  Effectively, we’ve just described a portfolio of 12 elements where each element is itself a strategy.

It’s not difficult to imagine implementing a controller strat which on a regular schedule will simply adjust the parametrization of the running strategies and their allocation of cash based on the performance of the strategies running in simulation.

It’s not difficult to imagine, but implementing it inevitably poses a variety of challenges not least of which is finding the time!  For now, we’re still finding new and interesting ways to express strategies which deal with portfolios of exchange-traded instruments.  But I think that this is just a point on our technology’s maturation curve which will ultimately lead to the regular development of strategies which operate on portfolios of strategies themselves…

Anyway, it’s nice to see a bit of academic work formalizing what I expect is a natural step in the evolution of trading algorithms.

This isn’t surprising as I wanted the system to support all of the activities associated with our algorithmic trading.  It turns out that that’s a lot to ask of a system.  It also turns out that you learn as you go and so the system continues to evolve.  A few years ago I’d posted about the basics of a strategy container and in this post I’m going to come back to this topic and describe some of the layers of code and thought developed since then.

First, let’s consider the role of a strategy container.  Its job is to intermediate between trading strategies and the external environments with which they interact.  It must also provide services that strategies can use (e.g., position management) and that it wouldn’t make sense for each strategy to re-implement.  In the past I’ve focused on the former responsibility of adapting strategies to external environments.  Why is this necessary and interesting?  Because it allows us to take the same exact strategy and run it live, or in simulation or in backtest, etc.  Interesting and necessary, but not what I want to focus on this time.  Instead, I want to look at the services provided to strategies; the ‘ecosystem’ a strategy container provides in the hope that strategies might flourish within it.

We’ll go from the bottom up.  At the bottom you have one or more pipes of market data coming in.  You might also have news feeds, weather feeds or other such things coming in and we’re just going to conflate them all and say “market data.”  Still at the bottom, you also have one or more two-way pipes with your oms or broker or directly with exchanges or with an exchange simulator.  Within Stratbox we call this abstraction an “ExecutionPlatform” because it’s conceptually where trades get executed.  The lingua franca for execution platforms is FIX, so the baseline object model of a strategy container is likely going to look a lot like messages described by the FIX spec.  Here we have orders, executions and the like.  Within Stratbox we’ve implemented an exchange simulator, a QuickFIX-based FIX interface and a couple broker-specific APIs and they are all of type ExecutionPlatform.  To a strategy, they all look the same.  Likewise with market data.  Within the strategy container, we provide a publish-subscribe model wherein any kind of market data can be subscribed to regardless of the ultimate source of the data.  The intention is always to intermediate between the strategy and its external environment.

the baseline object model

Great.  But now I want to write a trading strategy.  What’s that going to look like?   And this is where things get increasingly interesting as we get to decide what kind of facilities we’re going to provide.  We also need to manage concurrency in some fashion as trading is an intrinsically asynchronous activity.  For this lowest level of strategy, we’ll say that each market feed is handled by a thread which timestamps and enqueues a “Quote” for later consumption and redistribution to relevant subscribers by a MarketDataService within the container.  Likewise, each execution platform will be serviced by a thread who similarly enqueus “Execution” objects from exchanges real and simulated.  The threads handling executions should likely have a higher priority than the threads handling market data.  So, the strategy is naturally exposed to the markets’ asynchronicity right off the bat.  What baseline facilities will our strategy have at its disposal?  Well, it needs to be able to manage orders and positions, so it needs some kind of a “blotter” facility.  With this in place, a strategy can safely assume that it doesn’t need to listen to each execution coming in just to have a correct picture of its book.  It needs access to a MarketDataSvc and it likely also needs access to historical data.  But, staying simple, that’s more or less all we have to provide and we’re going to provide all of these things to the strategy through a single handle: a strategy context.  By swapping out a strategy’s context, we can move the strategy among environments (eg, from simulation to live execution).

With these facilities in place, basic algorithms can be implemented within a strategy and you should be able to test and trade them.

But you’re dealing with very low-level stuff.  You’ll also find that you’re writing the same kind of code over and over.  This inspired the introduction of a layer on top of the baseline object model which I’ve described before: StratParts – a strategy component model.

StratParts – a component model for strategies

With StratParts, we introduced metadata and composition (in the “composite pattern” sense) to our Strategy object model.  A strategy is itself a stratpart which contains other stratparts.  Thus, stratparts introduce a hierarchical structure to a strategy.  Each stratpart publishes a metadata descriptor which is aggregated at the strategy level.  The descriptor contains all of the modifiable characteristics of the strategy and can be changed while a strategy is executing thus giving a sort of grey box capability wherein a trader (or another strategy etc) can modify the behavior of a strategy as it’s running.  Stratparts also create an effective means of providing scoping within your strategy environment.  Stratparts can “see” the activity of other stratparts within the same strategy – they’re all peers in this sense.  This scoping can also be used for allocation of resources across different elements of a strategy.  For example, one stratpart might be allocated 80% of the cash available to the strategy while another stratpart manages the remaining 20%.

Stratparts have proved to be very powerful and we’ve written many of them though we’ll sometimes write strategies as one monolithic stratpart where it makes sense.

Stratparts have a lot of uses but one thing they don’t do is help manage the low-level complexity inherent in trading activities.  This was best described by a trader with whom I’d collaborated.  He’s a reservist in the IDF and uses military metaphors like I use sports metaphors.  He bemoaned the low-level handling of orders required for a very close-to-the-market strategy we were working on and said that:

we need an order like a ‘smart’ missile: fire it and forget it

He was right and that inspired the next level of abstraction/support that we built into the system.

‘smart’ orders and the tradeflow stratpart

The strategy we were developing was meant to look at a universe of futures spreads and generate all of the ‘cycles’ that might result in an arbitrage opportunity.  Since these are rare at best, we were really looking for circumstances where it looked like we had an advantage based on depth-of-market and various heuristics we’d apply.  Among the functionality that he wanted was an ‘order’ which would act as a limit unless some conditions obtained in which case it might pay the spread or otherwise stop being a fixed limit order.  He didn’t want to handle this inside the strategy but instead wanted to fire these things off and forget about them unless they required attention (eg, if they’re rejected by the exchange or his firm’s risk management checks).  Although we ended up calling these smart orders, they’re really a sort of very localized execution strategy themselves.

To support them, we utilized the same workflow (= state machine) framework we’d implemented for our ExchangeSimulator as we wanted to ensure that these smart orders had a very well-defined life-cycle with a clear set of states and guarantees about transitions among them.  We embedded this workflow engine into a stratpart which supported smart orders operating within this “tradeflow”.  Thus, the extra functionality (and weight/expense) of this functionality is only achieved/incurred when actually required or desired.  This has proven to be a powerful addition to the system and raises the level of services enjoyed by prospective strategies substantially, particularly given the variety of orders we’ve implemented which stand ready for use by any new strategy that might want to employ them.

At this point, it should be clear that we’re blurring the lines between the strategies themselves and the container.  Another way of thinking of it is that one container can have a set of pre-made building blocks that can be applied for families of strategies.  Each one provides a particular grammar with which a strategy’s aims can be expressed.  For very close-to-the-market strategies where very low-level handling of orders is required, the tradeflow stratpart provides a set of really useful facilities, but if I’m just trading with limits then it might not be so interesting.

portfolio-oriented strategies and beyond

I’ve mentioned before that I only care to think of strategies that operate on a portfolio.  As such, we’ve written a good deal of facilities for creating, analyzing and manipulating portfolios within strategies.  By using these facilities within a tradeflow stratpart, I can express quite complex strategies relatively simply by using portfolio analytics to determine my current ‘model’ portfolio and then using smart orders to most effectively transition me from my current state to my model state.  For the kind of trading I do, this provides a rich ‘vocabulary’ for the development of strategies.  All the same, my brain is having a tip-of-the-tongue moment as I feel that there’s an entirely different model possible for portfolio-oriented strategies just at the periphery of my imagination…  oh well, there’s always something new to explore.

Different trading styles, perspectives or trading problems are undoubtedly better served by different sorts of facilities.  I’m finding that just as language constrains and shapes thought, the strategy container that you employ shapes and constrains the kinds of strategies that you can implement.

If you have any ideas about such facilities for any kind of trading you do, I’d love to hear them.

–

a note about Stratbox

Although I’m talking about our system in this blog, we’re not marketing the system and happily “just” use it for our own trading activities.  We had looked into marketing the system previously but ultimately feel the same about selling stratbox as selling strategies…

If you have any interest in ETFs, then you should read this paper carefully as it provides a very nice and accessible mathematical treatment of leveraged and inverse ETFs.

I’ve had success using ETFs in portfolio-oriented strategies to conveniently provide specific exposures, eg, to emerging markets.  I’ve also explored strategies that pit ETFs against futures and similar arbs that take advantage of contract rolls or other anomalous behaviors across the markets.  But I’ve never looked at ETFs the way they really should be understood: as structured products that should have well-defined (if not necessarily obvious) properties.

Like many structured products, some of these characteristics are not obvious and may be quite unintuitive but are always important to understand.  For instance, the hedging required to implement these funds is both non-linear and asymmetric.

Specifically, leveraged ETFs must re-balance their exposures on a daily basis to produce the promised leveraged returns. What may seem counterintuitive is that irrespective of whether the ETFs are leveraged, inverse or leveraged inverse, their re-balancing activity is always in the same direction as the underlying index’s daily performance. The hedging flows from equivalent long and short leveraged ETFs thus do not “offset” each other. [...]

The impact is particularly significant for inverse ETFs. For example, a double-inverse ETF promising -2X the index return requires a hedge equal to 6X the day’s change in the fund’s Net Asset Value (NAV), whereas a double-leveraged ETF requires only 2X the day’s change. This daily re-leveraging has profound microstructure effects, exacerbating the volatility of the underlying index and the securities comprising the index.

Hence Mr Zweig’s concern that these ETFs feed the volatility we’ve seen for the last 8 months or so near the market close.  If the day has been up then both “bull” and “bear” levered ETFs will need to buy in order to stay hedged – reinforcing the trend and effectively supporting serial correlation of returns.

Another important property of levered and inverse ETFs is that they are designed to track the daily returns of their underlying index which does not mean that over the long term they will faithfully track the returns of their index.  Cheng and Madhaven illustrate that this can be viewed as an embedded path-dependent option within the product.  This is graphically illustrated with two charts of DIG and DUG over a shorter and longer period.  Over a short period, they track each other admirably, mirror-like, as one would expect.  But over a longer period they diverge quite spectacularly.  In fact, in the example below, one would have lost money betting on either over the same period!

"Heads I win, tails you lose" - ProShares

Another interesting result of Cheng & Madhaven’s research is that the “[hedging] demand as a fraction of closing volume is a non-linear function of return…”.  Thus, the greater the move in the underlying index, the hedging activities of these ETFs will have a (non-linearly!) greater market impact on the close.

There are other gems in here as well including hints at pricing options on these ETFs, and techniques for quantifying the market impact of hedging activities.  But don’t take my word for it.  Go read the original!

For those who spend their workdays in the markets and their weekends pondering derivatives pricing, latency, oceans of market data, portfolio optimization, and how to make every last damn thing faster, a preference for nVidia cards could prove to yield an unexpected benefit.

nVidia has recently unveiled a product line dubbed “TESLA” which leverages their absurdly fast GPUs to provide a supercomputer-like High Performance Computing (HPC) platform at a previously unimaginable price point.  TESLA computers are regular machines that have a set of slightly modified GPUs in them; modified such that they have no video out, but instead become additive processing clusters which the machine can use for compute intensive tasks.  For about $10K you can buy a 1U machine with some 4 teraflops of capacity.  By way of comparison, this is over 20 times faster than the funky Helmer project I’d been drooling over a few months ago in a production-worthy package ready for the server room today.

So, TESLA refers to the machines built with these specialized GPUs.  Making all this power usable is what CUDA is about…

CUDA is nVidia’s model for parallel programming that provides a C-based software environment for the development of applications that take can advantage of nVidia’s GPU architecture.  These advantages center around parallelism, torrential memory bandwidth and obscene floating point performance.   The below graphs illustrate these advantages against high-end Intel hardware.

Although I haven’t yet taken the plunge and purchased one of the trick new TESLA machines, I have been able to dip my toe into the waters as CUDA will work with most newer nVidia cards.  All you need to do is download the CUDA environment and SDK and you can very quickly get up and running.  They have an expansive and well-documented set of examples including many of interest to algorithmic traders – particularly surrounding derivatives pricing and monte-carlo methods.  I haven’t tried yet under linux, but was able to successfully modify their sample programs under vista after less than an hour of fiddling.

I had seen some flashes of interest in CUDA on the Quantlib mailing lists, but I’m not sure if there’s been any real follow-up.  Even if it takes some time for the open source community to get plugged-in, there are already a number of finance-oriented commercial offerings including a plugin to Matlab which allows some operations within Matlab to be offloaded to whatever CUDA subsystems are available.

If you’re comfortable with C and happen to have an nVidia card handy, I encourage you to take a look-see. The price/performance advantage is remarkable and if you have a problem that can profitably make use of it, one of these boxes might pay for itself in a big hurry!

In a scenario of perfect knowledge, this wouldn’t be true. If you somehow *know* with certainty that crude will go up or that Citigroup will go down, then concentrating all of your resources into a position based on that belief might be reasonable. But knowledge seldom comes in such a neat package (and will frequently be illegal to act upon when it does!).

Instead, knowledge will typically come in more conditional and less certain forms: “commodities tend to rise during periods of FUD [Fear-Uncertainty-Doubt]” or “companies who announce stadium naming rights deals tend to under-perform.” In some cases, perhaps the knowledge on which you’ll base your strategy can be quantified probabilistically.

Depending on the nature and quality of the knowledge or hypothesis that forms the basis for a given strategy, one can adapt one’s portfolio construction/optimization based on customized relationships amongst the potential portfolio constituents. But one doesn’t need to be so fancy to see the concrete benefits of our first axiom. Below I detail a simple strategy I’ve put together to explore the forces involved.

The strategy I’ve built constructs a completely random, equally weighted long-short portfolio every day and liquidates the portfolio at the end of the day. Specifically, everyday the strategy:

1. Looks at all equities in the database that are available to trade. There were a bit more than ~4K equities available on any given day for this test.
2. Constructs a portfolio composed of N equities selected at random from the available equities. The portfolio has a fixed number of dollars available to it ($500K) and thus each position within the portfolio will be of size $500K/N or less. Orders are always scaled down so that we’re trading round lots (ie, qty%100=0) and if we can’t buy 100 shares of a particular name (eg, Berkshire Hathaway) then that money goes unused until the following day.
3. Half of the components in the portfolio are bought and half are sold so that we have a dollar-neutral portfolio.
4. Positions are entered at the open and exited at the close. (For these kinds of position sizes, this turns out to be a reasonable assumption, as we’ve illustrated here.)Â No stops are employed.

That’s it. The strategy only exposes ONE parameter for tweaking: N – how many names to hold in the portfolio. I varied this parameter across the set { 2, 4, 6,…, 12 } and ran each value 100 times across one year of data.

By selecting the portfolio’s components randomly, I’m trying to ensure that the strategies are truly non-predictive and thus should have an expected return of $0 without applying friction. By creating a dollar-neutral portfolio, I’m trying to ensure that we’re not accidentally capturing beta. What I’m looking to illustrate is the relationship between the number of elements comprising the portfolio and the risk-adjusted performance of the strategy.

As the following distributions illustrate, the strategy does indeed seem appropriately non-predictive. The distributions are normal and shifted to the left due to the application of realistic fees & commissions. As I’d warned in my very first series of posts, the random set of strategies did produce some nice-looking outliers, but we’re past being fooled by this pyrite of data-mining bias.

Also in terms of showing volatility’s relation to (even trivial) diversification, the experiment did not disappoint. While there’s essentially no relation between the profitability of a strategy and the portfolio size, there is a strong negative relationship between the volatility of the strategy and the diversification employed. Furthermore, the benefits of diversification happen quickly and soon taper off as we can see when we look at the table and chart comparing the number of elements in our portfolio with the average vol across the hundred relevant strategies.

There’s nothing new about this result. Harry Markowitz had explained these phenomena with infinitely greater rigor over half a century ago! But judging from what is published in popular “trading strategies” books and periodicals and even the baseline capabilities of algorithmic trading platforms, it appears that people persist in trying to beat the market without using what is perhaps their most potent weapon.

There’s an almost good reason for this that’s like the parable of the fellow who loses his key on the street but decides to limit his search to the area illuminated by a nearby lamppost. When asked where he’d lost the key, he indicates that he’d lost it somewhere on the street but was searching under the lamppost because the light was so much better there. Writing a single instrument strategy can be difficult enough on its own and adding instruments can make it substantially more difficult. But that’s also not a good reason to persist in limiting the scope of one’s efforts!

While there’s no claim to originality in this example, I hope that it’s nonetheless illuminating to see that even an intentionally non-predictive strategy can be mechanically improved. And we haven’t even applied all of the lessons of portfolio construction that we might have. Thus, while writing a worthwhile strategy remains difficult, one can see that there are some very well understood baseline tools that can be mechanically applied to any strategy that will yield material improvements.

I mentioned several weeks ago that I’ve been developing and trading a strategy that’s proven to be quite interesting and profitable. In that post, I described how I’d tried to improve the strategy through the use of a dynamic hedge. The results of that crude hedge were quite good, but just as no worthwhile software project is ever really complete, trading strategies demand constant iterative development.

Below I describe some of the steps I’ve taken to incrementally improve this strategy, discarding the relatively expensive hedge I’d developed earlier in favor of a complementary strategy. We see that when you combine two positive and uncorrelated results, you end up with a product that is literally better than the sum of its parts.

The core strategy has remained the same, though the universe of instruments it trades has been both expanded – through the inclusion of a broader set of equities & ETFs – and contracted – through the application of some filters which prevent trading of some of the instruments under various conditions. These change have yielded a few more points of annualized return with minimal impact on volatility.

As I’d mentioned, the core strategy builds a portfolio of shorts which is sold at the open and bought back at the close. One of the problems with the original hedge is that it had a cost – literally. In order to hedge my portfolio, I needed to cut-back the size of my portfolio to accommodate the cost of the hedge. I mitigated this issue to some degree by employing futures instead of a broad market ETF, but this still reduced my usable capital by approximately 10-15%. I also had my money sitting idle overnight which seemed a particularly profligate behavior.

This led to the study I described last time in which I observed the relative out-performance of the broad US equity markets overnight. If I could somehow find a way to capture some of this overnight alpha, I’d be able to both hedge my main strategy and better utilize my capital.

My first effort at such a strategy happily achieves both aims by assembling a long portfolio which is held overnight. In the chart below, I capture the returns of the two strategies independently and combined. The core daytime strategy is denoted as “OpenClose” or OC whilst the night-time strategy is “CountingSheep” or CS. Both assume an initial capitalization of $1M, employ no leverage and do not reinvest returns – each day or night they assemble a portfolio with the same $1M and profits are put aside and don’t generate interest.

The chart is nice and certainly a big improvement over the earlier hedged approach, but the real power of combining these two strategies is revealed in the two tables below. The first characterizes their risk-adjusted performance independently and then when combined. Both have a Sharpe ratio of around 2.0, but when combined they yield a new strategy which is about 25% better on a risk-adjusted basis. These cells are highlighted.

The key to their compatibility is their correlation. Or, actually, their absence of correlation. In the below table, you can see the correlations of their returns to one another and broad market ETFs. The short daytime strategy is, not surprisingly, negatively correlated with the broad market while the long night time strategy is positively correlated. The beauty of their combination lies in the lack of correlation between the two of them (highlighted) – they’re essentially uncorrelated.

I hope this post illustrates a couple of different vectors along which strategies can be evolved; in this case, to better manage risk and utilize capital.

One of the key remaining limitations of this particular strategy is its capacity. Increasing the capacity of a relatively short-term strategy like this one requires optimization of the trade executions which is its own black art but one that plenty of smart people are constantly addressing. Perhaps in a future post I’ll review some of the techniques applied to this problem for another perspective on the iterative development/evolution of trading strategies.