The example I’ve chosen is, I think, very nice because it’s a portfolio-oriented strategy, which is pretty much the only kind I care to explore; it’s also based around the concept of pairs trading, which is something which most can easily relate to; and finally, it’s already public domain and yet almost certainly has some juice in it for those who care to understand it and extend it intelligently.

The example comes from the blog of a company, Palantir which does (something like?) analytical / decision support software for both finance and intelligence-gathering services (quants and spooks – spooky quants?).  The specific example is here and is described thusly:

In this study we explore a trading strategy that isolates pairs of instruments within a sector that are highly correlated. We enter a trade if the price paths of these instruments diverge, going long one instrument and short the other, with the assumption that their price paths will converge.

We construct our strategy in four parts: (1) isolate the target set of tradable instruments, (2) choose a rule for finding correlated pairs within that set, (3) pick criteria for entering trades, and (4) test the trading strategy.

…

For this study we want to isolate pairs of instruments whose correlation is above a certain threshold. At the beginning of each quarter, we restrict our target stocks to the top 5 in the sector by 252-day correlation to the S&P 500. Then our Strategy uses a metric that, for a given group of stocks, outputs a list of pairs with a correlation above our target threshold of .60 over the same time period.

…

Across every day and every pair in our trading list, we check the 21-day z-score (number of standard deviations from the mean) of the difference between the two series. We classify the stocks as diverging when the absolute value of the z-score is between 1.5 and 3.0. When this condition is met we short the stock that is rising and long the stock that is falling, sizing our position relative to the z-score and the number of pairs currently trading.

~(sectors) ; reframing the strategy

While I’ll implement a strategy broadly based on their description, I will use some poetic license to modify it in ways that seem to me sensible.  First, I have good experience writing equity strategies which deal with sectors and have pretty uniformly found that sectors, as defined by S&P or whomever, are more trouble than they’re worth.  Instead, looking at correlation matrices and determining on that basis who should be paired, is much more fruitful in my experience.  Tr8der has some very nice posts illustrating (very prettily!) a similar point here and here.  (Actually, his point is a bit broader and represents an area of possibly promising development of this strategy which I won’t pursue here.)

So, we’re going to chunk the idea of explicitly using sectors.  Instead, we’ll simply construct, on a monthly basis (Palantir did this quarterly) a TrailDays correlation matrix from which we’ll extract the set of pairs which have a correlation greater than MinCorrelation.  On a daily basis, we will calculate the ZScoreDays z-scores of each of these pairs and sort them by absolute value, tossing those whose values are less than MinZScore or exceed MaxZScore.  Since I don’t like the idea of having an open-ended number of pairs/positions in my portfolio, I will constrain the portfolio to a maximum of N names and will Allocate $1M to the strategy.   Finally, we will look at the effect of trading an EvenlyWeighted portfolio or a z-score weighted portfolio (as Palantir had done).

The initial ‘universe’ of equities across which we will calculate our correlations will be filtered from my database of daily equity data going back to 2005 where the closing price is over $1, the average daily dollar-volume is over $50M and the data is relatively clean (few if any gaps).  This yields an initial universe of ~600 equities and ~180K pairs.

Assuming that we keep N above 20 or so and we keep our allocation constant at $1M, and assume execution at the close price, we should expect our results to be reasonably believable as it’s unlikely that the size we’re trading in would distort the market.  Scaling any strategy presents its own set of challenges which we’re not going to go into for this example, but an obvious direction for scaling the strategy would be to simply run it over intraday data and scale into and out of positions as conditions permit.

Ok, so that’s what our strategy is going to look like.  Now, how do we “make it so”?

structure of a stratpart

Within Stratbox, strategies are represented as composites made-up of ’stratparts’ each of which has its own metadata descriptor containing parameters.  Stratparts can be composed together within a strategy to enhance or modify the behavior of a strategy.  Thus, one can have a stratpart which implements the above logic paired with another stratpart which will e.g., modify the portfolio to impose beta neutrality, damp volatility or some other form of reporting or risk management.  Given that this example is already complexish, we’ll just stick to the one stratpart.

A stratpart has a few key elements.  One is the metadata descriptor which allows the strategy container (stratbox) to interact with it for back-testing, optimization or forward-walking purposes, or to allow a ‘trader’ to interact with it dynamically during run-time (the so-called ‘gray box’).  Beyond that, it has an essentially event-driven design.  The method quote() is called when relevant (i.e., subscribed) market data arrives.  The method execution() is called when executions for orders that originated with this strategy are received.  And the method strategyEvent() is invoked on a variety of ‘interesting’ events that the strategy implementer might care about.  None of these methods are required, so in its simplest form, a stratpart is a very simple piece of code to write.

services in a stratpart

Once one has complied with the minimal requirements for implementing a stratpart, one is rewarded with a variety of useful services: subscription-based market data feeds as well as a fast snapshot db and an in-memory historical db for quick correlation analyses cover market data needs.  Position management is handled at both the strategy level and at the exchange level (if you’re trading across multiple exchanges).  There is more, including performance analysis (e.g., sharpe, and its various analytic friends) and hierarchical cash allocation within and across strategies, but we won’t look at these in this example.

implementing metadata

So, the first thing we need to do is define our metadata.  This is done through the use of a conventionally-named static method Descriptor().

public static Descriptor Descriptor() {

 ArrayList<Param> params = new ArrayList<Param>();

 String cd = "Unleveraged capital to apply";
 params.add(new Param(InitialBalance, cd, false, 1000000.0));

 String n = "# of instruments to hold in portfolio";
 params.add(new Param(N, n, true, 20));

 String tc = "Trailing days over which to calculate correlations";
 params.add(new Param(TrailDays, tc, true, 252));

 String mc = "the minimum correlation of pairs to consider";
 params.add(new Param(MinCorrelation, mc, true, .75));

 String zd = "trailing days over which to calc z-score";
 params.add(new Param(ZScoreDays, zd, true, 21));

 String mizs = "minimum z-score";
 params.add(new Param(MinZScore, mizs, true, 1.5));

 String mazs = "maximum z-score";
 params.add(new Param(MaxZScore, mazs, true, 3.0));

 String ew = "evenly-weight portfolio? (or weight by z-score)";
 params.add(new Param(EvenlyWeight, ew, true, true));

 Descriptor _desc = new Descriptor
   ("puppetmaster.strats.Pairs", _Desc, params);

 return _desc;
}

public Pairs(Strategy strat, Descriptor d) { super(strat, d); }

Not the most interesting chunk of code, but pretty straightforward and its inclusion opens up a lot of functionality within the environment as each of these parameters can now be systematically or manually modified in-flight.  The third parameter to the Param constructor, a boolean, indicates if we want to allow the system to optimize this parameter.  Thus, the allocation/initialBalance parameter isn’t optimizable as it wouldn’t make any sense to do so.  Even so, we can (ourselves or through an allocation algorithm) modify this value to force the strategy to lighten or extend its financial footprint.  The others are fair game for optimization.

The constructor is also required – with that exact signature.  This is pretty much it for the ‘boilerplate’ code required.

‘installing’ the stratpart

Now that we’ve met the minimum requirements for a stratpart, we can install it into stratbox.  This requires adding one line to your personal stratparts.xml config file.  Like so:

<?xml version='1.0'?>
<stratParts>
 <descriptor class='puppetmaster.model.strategy.FwdWalker'/>
 <descriptor class='puppetmaster.model.strategy.StrategyPortfolio'/>
 <descriptor class='puppetmaster.strats.Pairs'/>
 <descriptor class='puppetmaster.strats.AMBO'/>
 <descriptor class='puppetmaster.strats.meta.AdjMeanReverter'/>
 <descriptor class='puppetmaster.strats.meta.AdjTrendFollower'/>
 <descriptor class='puppetmaster.strats.meta.Binary'/>

Once we’ve informed stratbox of the new stratpart we’ve written, we can view it within our library of stratparts within the strategy wizard in stratbox:

Our Pairs strategy in the Strategy Wizard's stratparts library

And all of the parameters we defined within our descriptor show up on the next page where we can parameterize or optimize the strategy:

Configuring the Pairs stratpart within the strategy wizard

This means that the stratpart can be combined with other stratparts to create new strategies as I’ve described  and that it can partake in the rich set of functionality within the stratbox gui.  Including actually trading.  But first, we must make the strategy actually do something…

implementing event handling

To that end, we must implement the event handling methods I’d described previously.  Since we’re not going to do any detailed execution handling in this strategy, we’ll only implement the quote() method and a subset of the events sent to the strategyEvent() method. In particular, we’ll listen to activation events (so we can initialize cleanly), changes to our descriptor and changes to our positions.

/** listen for select strategyEvents */
public void strategyEvent(StrategyEvent event) {
  super.strategyEvent(event);
  switch (event.type) {
    case Activated:            _init();                 break;
    case DescriptorChanged:    _readDesc();             break;
    case PositionChanged:      _posnChange(event);      break;
  }
}
<pre>

For quotes, we’ll just listen to the Beginning of Day (BOD) event.  If we wanted to trade intraday, we’d have to listen to tick events, but we’ll leave that for another day…

From a high-level, our code is pretty simple.  We determine the universe of pairs we might trade, we select from among them, we determine what our new portfolio should look like and then we trade to transition from our current state to our desired state.  That’s it.

/** mkt data goes here, but we only listen for the BOD
 * ("Beginning of day") event */
public void quote(Quote q) {
  if (q.type() != Quote.Type.BOD) return; // daily strat...

  try {
    List<_Pair> pairs = _getPairs();
    List<_Pair> sel = _selectPairs(pairs);
    List<Position> dp = (List<Position>)_desiredPfolio(pairs,sel);
    _Log.debug("PORTFOLIO: "+dp);
    _trade(dp);
  } catch(Exception e) { _Log.error(e.getMessage(),e); }
}

the messy details

Digging into the details gets more complex, but not horrifically so, as stratbox already has basic analytics like correlation matrices, z-scores, volatility built into it.  Likewise with basic portfolio analytics and transformations. Thus, our code to trade a (pairs-based) long-short portfolio with an arbitrary number of elements is surprisingly simple:

/** given a desired portfolio, trade to make it so */
void _trade(List<? extends Position> desiredFolio) {
  PositionRecord[] currFolio = posns();
  // Given our current portfolio (posns) and our desired state (dfolio),
  //  we generate the set of orders which will transform from the former
  //  to the latter
  //
  List<Order> orders = PortfolioComposer.TransformPortfolio
    (currFolio, desiredFolio, _orderF(), _strat, _strat.account());

  // place the orders for execution
  for (Order order : orders) {
    try {
      order.setType(Order.Type.MOC); // we trade at the close only
      _subscribe(order.contract());
      _execP().placeOrder(order);
    } catch (Exception e) { _Log.error(e.getMessage(), e); }
  }
}

As promised, I provide the full source listing below for those interested, but won’t over burden the discussion with a complete description of every little piece of it.

a telling omission and some results

Early in my career, working as a software engineer in a wall st front office technology department, I was befriended by a manager who told me he had once been a trader.  I asked him why he wasn’t a trader anymore.  He quipped:

I was half of a talented trader.  I knew, in my bones, when to get into a trade.  Sadly, I was never really sure when to get out…

It seems that the author of the Palantir example suffers a similar characteristic to my old friend (or is just being understandably cagey) – she specifies an entry condition, but not an exit!  Since she doesn’t impose any costs for trading, this has no big side effects, but in stratbox we do impose (pretty onerous -> realistic!) costs for trading.

In any case, since this is just a simple example I’ve maintained the ‘no exit’ policy and simply exit a trade once its pair goes out of the acceptable ranges for either correlation or z-score.  This has a bad effect on the strategy as you end up churning a lot of trades at the high end of the z-score range.  That is, if we set our upper boundary for z-score to be 3, then we might enter and exit a position multiple times as the z-score oscillates between, say, 2.9 and 3.1.  A more careful implementation could minimize this effect.

Another difference imposed by our more realistic simulation as opposed to the original example is that I’m restricting the portfolio size explicitly and explicitly limiting myself to trading a given contract within one pair as opposed to allowing a contract to trade across multiple pairs (invoking higher costs but allowing positions to offset and thus providing a neat little optimization – assuming trading costs nothing!).

All the same, the strategy clearly has some positive characteristics and when parameterized to take advantage of its naive exit strategy (by making the z-score low-end quite low, e.g., 0.5) takes on the character of a hedged trend-following strategy.  How so?  Well, while we are getting churned around the high-end (entry) z-scores, our profitable exits only happen once the z-scores have diminished (the prices have converged) considerably.  So we take more, smaller losses while our winners are held a bit longer.

some Pairs results in '10

Adding to my portfolio size improves things generally (and also increases the strategy’s scalability) while increasing the MinCorrelation parameter seems to hurt performance (not sure why, honestly – perhaps because if they’re very highly correlated and already diverging so strongly it may mean they’re actually diverging fundamentally). Returning to the topic of costs, each of these strategies incurred costs of over $5K for the period in question. And returning to the parameter which governs the weighting strategy employed, although I haven’t illustrated it here (as I haven’t looked at it very carefully), my initial finding is that it changed very little. 

There’s a great many areas where this strategy could be improved – that’s the fun, after all! – but hopefully even in its primitive form this example brings to light both an interesting baseline portfolio pairs strategy, while answering the question of how one goes about concretely implementing it within stratbox.

The entire strategy weighs-in at ~400 LOC including comments, boilerplate and individualized imports.  The source is here: Pairs.java.

In the example below, we look at another ‘meta’ strategy named StrategyPortfolio which maintains a dynamic portfolio – P – of strategies which it will select from a set of strategies – S – running concurrently in simulation.  The constituents of P as well as their cash allocations and parameterizations will be rebalanced/adjusted regularly after an initial ‘out-of-sample’ period during which only the S strategies are run.

Apart education, the intention of this strategy, as I’d originally suggested here, is to ‘back-into’ a regime-switching strategy without attempting to directly quantify the regimes explicitly.

This has proved to be even more interesting than I’d expected, not so much because it performs particularly well (though it’s promising), but because of all of the things it has taught us.  In particular, the transitions are a killer and there are properties of strategies which (dis-)qualify them from being effective in such a scheme…

bad news good news

As before, I’m not going to take the time to write-up my results in any formal manner but will again rely upon a quick screencast of the software running.  The good news is that I’m figuring out how to edit these things, so it’s mercifully shorter than earlier screencasts…

Please click here to see the screencast.

I hope you find it interesting and will look forward to any comments or suggestions you might make.

Last time we saw an example of historical forward-walking of a portfolio-oriented day-trading strategy which utilized daily data.  This time we do something a bit more interesting and correspondingly complex.  Today we’ll look at a real-time forward-walk of a moderate-frequency strategy (trades perhaps a few hundred times in  a day) which looks at the top-of-the-book but doesn’t use market-depth.  The strategy is a simple mean-reverter that we’ve described before though we’ve had to make some small changes to get it to behave in the context we’re looking at now…

meta-strategy protocol

careful ...

In particular, we need something of a protocol for sending commands from the meta-strategy to the ‘target’ strategy.  Stratparts generally have a simple mechanism for this as they are notified of any changes to their descriptors.  What they do upon such notifications is their own business.  In order to achieve our goals, we might need to firm this up a bit, but it turns out this quickly becomes tricky terrain.

Consider a single contract strategy, say, a simple mean-reverter like one we’ve seen before (and which I’ll illustrate in the screencast below).  If we want to treat it as an element within a portfolio of strategies, then we need to be able to modify its allocation.  But changing the allocation to a dynamic strategy is very different from changing the allocation to a single name within a portfolio as the allocation effectively becomes a maximum and the strategy may spend the great majority of its time under-allocated.  But this is manageable if we don’t mind being under-allocated at times.

A harder case is if someone comes along and says, “stop trading gold”, and “start trading silver”.  What about my open positions & orders?  If the answer is to go flat, then the character (of the returns) of the strategy is itself going to be effected by the fact that they are being mutated by a meta-strategy; if the answer isn’t to go flat then you’re really talking about a whole different, multi-contract strategy.  So you have to be careful to ensure that the kinds of mutations you are relying upon aren’t excessively disruptive to the strategy or you’re likely to see performance wildly different from normal for the strategy.  As always, if you put garbage in, you shouldn’t expect cotton candy to come out…

For this example, we can control allocation and two simple numeric params governing the strategy.  I’m just going to avoid the please-trade-a-different-contract-case and aim to address that issue by ultimately placing these strategies into a portfolio (some later day).  For now, we avoid the hard cases but I think the results remain interesting all the same…

In particular, just as we’d seen last time (on a totally different time-frame and with a totally different strategy), the forward-walked strategy appears to perform well relative to its peers…

If interested, please click here to see the screencast/example.

-

Happy Thanksgiving!

'optimization' or 'search'?

We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain.  Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode.  I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.

From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good.  Doesn’t mean they’re actually good.  In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively.  I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do.  They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.

They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations.  If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…

Anyway, I felt that had to be said…

Now, I’ll open the floor to a quick illustration of the initialization process described previously and an example of historical forward-walking.

At this point, I’m vested in the inquiry we’d started with the regime-switching post and will apportion time each week towards implementing something like what we’d described there.  I expect the steps to be something like:

1. historical fwd-walking which I’ll illustrate today
2. real-time fwd-walking, that is, dynamically adjusting a live strategy’s parameters based on a continuously repeated ranking of a concurrent ‘live optimization’ of the same strategy
3. real-time allocation to a portfolio of strategies based on regime-switching where regimes are defined by the performance of a heterogeneous set of strategies we run concurrently (again ‘live optimization’)
4. peace on earth

I may have some of the steps wrong, but am hopeful.  We’ll adjust as needed.

If interested, please click on the below image/link which will hopefully take you to a screencast.  This is my first-ever attempt to put a screencast in the blog, so please let me know if you have technical difficulties or suggestions on how to improve the experience.  Or if it’s just a big waste of time!

In the screencast I give a quick illustration of a simple backtest, a simple parameter optimization and finally a fwd-walk.  I’ll appreciate your feedback.

>click<

you, hf-trading

There seems to be a developing meme out there suggesting that algorithmic-, and in particular high-frequency, trading is some kind of gold-rush route to easy money which brings to mind…

…this revision of a paper I’d read previously: “Statistical Arbitrage in the US Equities Market” by Avellaneda and Lee.   It’s a detailed and thoroughly worked (and now re-worked) paper illustrating the development and analysis of a US equity stat-arb strategy based on Principal Component Analysis (PCA) and then revised to use ETFs.

I came across this paper as I have still never used PCA in any of my own strategy development work and read Carol Alexander’s excellent Market Models over my summer vacation with an eye towards giving a PCA hedging model a spin in the near-term. Thus, I wanted another look at this paper as a reference point.  Although it’s an excellent paper, I’m not going to urge you to go out and read it immediately unless you have a reasonably pressing practical interest.  Instead, I find it interesting largely because of one of its authors – Professor Avellaneda – and its conclusions in the form of its strategies’ performance.

I’ve seen Prof Avellaneda speak a number of times at a variety of quant meetups organized by the relevant Columbia/NYU financial engineering depts.  His paper reminds me that at least once during my noisome adolescent years, my father intoned darkly that:

the streets are littered with brilliant minds

Implying that any wits I may believe myself to possess wouldn’t by themselves be worth much in life and that I’d need to bring actual tools to the task of solving problems if I wanted to address interesting ones.  Having seen Mr Avellaneda speak, I’m confident that at my peak, my “processor” was never as fast as his.  Much worse, there is no comparison between the tools he can level at a problem compared to me – he’s on an entirely different playing field so far as concrete mathematical/analytical capabilities go.  That’s why I go see him speak and read his papers.

Thus, the results of Avellaneda and Lee’s work are particularly interesting to me as they’re really pretty dull: something like a Sharpe of .9 and degrading briskly.  Now, you don’t expect people to be providing detailed recipes to wildly profitable strategies, and this result isn’t bad, particularly given that they’re describing strategies which likely have significant capacity.  Still, it illustrates that very smart people working with sophisticated mathematical tools even over extended periods are still operating under noteworthy constraints.  Perhaps also: ideas are relatively easy – examining them in the requisite detail is difficult and time consuming, even for (particularly for?) people with the most finely honed toolsets…

I frequently have friends or colleagues who will observe that if you “just write a strategy that foos when bar but yaddas when baz… you should surely make money.”  Maybe.  But the reality is that just putting together the strategy and working through it takes significant time for anything but the simplest strategies.  Once you add genuine complexity to a strategy, you can spend enormous time tuning it.

This, in turn, poses a dilemma I encounter frequently and honestly don’t have a great answer for:

how to find the balance between continuing development on a known good strategy and initiating the development and analysis of unrelated and novel strategies?

the back of the envelope as canvas

This next (de-)reference isn’t directly pertinent to algo trading, but the lessons learned by building BIG distributed systems can surely be applied elsewhere.  And they’re just plain fascinating.

Google’s Jeff Dean gave a recent talk entitled “Designs, Lessons and Advice from Building Large Distributed Systems” at the Large Scale Distributed Systems and Middleware (somehow “LADIS”) workshop and the slides are here.   Go read them.

If bald exhortation doesn’t convince you maybe slide 24 will:

or perhaps what he does with these baseline numbers in slide 27 will pique your interest:

back of the envelope as art form

One that made me (a serial prototype-builder) cringe:

Important skill: ability to estimate performance of a system design
– without actually having to build it!

Ouch.

maybe it is easy after all

Of course, if you’ve studied Avellaneda & Lee’s paper and it held no challenges or surprises and you’ve reviewed Mr Dean’s presentation and it’s old hat to you, too…

mean-reverting strategies (Picture: wikimedia)

The real world, despite having lost some mind-share to reality tv, Tim Geithner and the internet, remains an interesting place and a potentially fruitful source of inspiration for our own efforts at strategy development.  In this pictorial post, I’ll take a look at some of the physical phenomena which inspire or reflect various trading strategies/domains.

trend-following strategies (Picture: jamiescottimages.com)

trading STIRs

technical analysis

making markets

internalization

writing options

exporting debt

Even if it’s hard to find good stuff, one must still look as there’s always more information that can help you than you can effectively process or retain.  A few weeks ago I was trying to formalize the expected profit function of an algorithm I’m developing and wanted to see what people had written about the topic.  I entered ‘define profit function for trading algo’ into google and was pleasantly surprised to see a paper entitled ‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani.  It doesn’t directly cover the topic I was looking for, but instead addresses a number of related topics I’ve been interested in for some time:

• the treatment of a strategy as an instrument in its own right
• composing portfolios comprised of strategies
• using regime switching techniques to manage portfolios of strategies

In this post, I’ll briefly review their paper, illustrate how one can easily model strategies in relevant ways using the strategy ‘object model’ I’ve described previously through an example, and conclude with some thoughts on how these kinds of strategies might be implemented and further explored.

‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani

Their paper begins:

This paper considers the problem of dynamically allocating capital to a portfolio of trading strategies. The allocation should be robust, and the capital allocated to a trading strategy should reflect the confidence in the expected profit that the strategy will make in current market conditions.

Good trading strategies exploit recurring market dynamics that can be more prevalent in some time periods than in others. Indeed, the concept of regimes is fundamental to financial markets, and much research has focused on the detection of regime shifts. In this paper, we consider a regime as defined by a set of trading strategies that exhibit similar performance in a given time period.

We consider different parameterizations of the same strategy as distinct in our ground set of strategies. The trading problem is to pick a distribution over the ground set that will achieve good performance in the current time period.

They start with some interesting definitions.  A trading strategy is defined as a function which takes a given market State and generates a convenient Answer “denoting all possible trading decisions”.  Strategy functions can also be parameterized and different parameterizations of the same function/strategy are viewed as distinct qualitative strategies.  They assume the existence of  the set of all strategies.  A regime is defined as a set of trading strategies who behave similarly over a given period of time where similarly is measured by the variance of a fitness function.

So they start with a set of strategies, a fitness function and some historical market data or state, and their first task is to identify the regimes.  They do this by running the strategies against randomly selected partitions of historical data, quantifying their respective performances with the fitness function and identifying the clusters deemed statistically significant.  Those sufficiently significant and similar sets of strategies are themselves the reasonable regimes while the set of strategies within all the reasonable regimes are themselves dubbed the reasonable strategies – the candidates for execution.  This might sound strange, but as I’ll illustrate with a concrete example below, this is actually a pretty effective and even intuitive way of defining regimes.

They define function stFuncDist as, for any regime, generating a distribution or weighting of the strategies within the regime. While classifyMarket is a function that generates a distribution across all regimes for any given market state. Both models can be “trained” and they do so by again iterating randomly across chunks of historical data adjusting the weighting constants of these two functions.

Finally, it’s time to trade and they can effectively use classifyMarket to determine a predominant regime and allocate capital as indicated by stFuncDist to the appropriate set of “reasonable strategies.”

They claim to have utilized this technique to generate the results charted in Figure 1, but they give little detail on the initial universe of strategies.  I believe they had one strategy which was parameterized into a set of qualitative strategies and run across one market, the CME’s Nasdaq emini.  I have no reason to doubt that they had exactly this result, but without more details on what they actually did, it’s hard to view it as very meaningful, though it certainly is suggestive.  An interesting thing about how they present their result, which I’ve seen in other academic papers, is that because they’ve precisely defined the set of possible strategies they look at, they can also calculate max and min possible returns over the relevant period.

Figure 1: Out-of-sample profit (middle line) versus possible min (bottom line) and max profit (top line) over all strategies for the period 2006-11-01 to 2008-08-28.

The paper continues with another section, but I’m going to stop my overview here as the last part is less interesting to me and the interested reader should of course read it themselves.

complex strategies or simple strategies combined

One of the broadest design questions I see for the development of effective trading algorithms is whether one should employ complex, customized algorithms or instead utilize simple algos which can be simply combined to produce emergent complexity.  Although the authors of this paper allude to “Good trading strategies”, I wonder if one can even forget the “good” part and instead worry about applying – even bad! – strategies at the right times.

There are a lot of things I like about this paper, but probably most of all it’s their definition and use of a regime.  One of my first posts describes a pair of strategies – a simple mean reverter and a simple trend follower.  These two strategies are about as dumb as you please.  And yet, their behavior is pretty interesting apart the observations I’d published at the time regarding the distribution of their returns.  During the trading day, I’ve taken to running those two, old, dumb strategies across a variety of markets in simulation mode.  Why?  Because it’s easy, costs me nothing and actually is pretty interesting in that the strategies really do cluster in the way Mlnarik et al describe, effectively partitioning the market into visible regimes.

a simple example: identifying regimes with strategies

In order to illustrate this, I created a desktop within stratbox onto which I ran these two qualitative strategies parameterized differently and across six different markets.  In total, I have 48 strategies running with 24 of the mean reverting (MR) variety and 24 of the trend following (TF).  In figure 2 and figure 3, they are sorted by qualitative strategy – all of the MR strategies followed by all of the TF strats.   In figure 4, they’re instead sorted by instrument at the end of the day after all of the strats had been rendered quiescent for the night by their schedulers.  You can click to see a bigger version of these screenshots.  On each, the left side of the screen shows simple charts of the relevant markets while the right contains a table in which each line item is a running strategy.  The first is at the beginning of the day. the second is shortly before going silent at 4pm and the third is shortly after 4pm.

Figure 2: variously parameterized MR & TF strats run in realtime+simulation mode. Sorted with the MR strats first. (click)

Figure 3: later the same day

Figure 4: at EOD, now sorted by instrument

You’ll note that at all times throughout the day, the MR and TF strategies effectively mirrored each other regardless of market.  Although I’m not going to illustrate it, I’m going to just go right ahead and assert that this is almost always true  based on what I’ve seen.  And this is at the heart of the paper’s authors’ insight regarding regimes – they can be effectively identified by examining the performance of even simple but well-defined strategies.

This suggests a simple means of implementing a similar design to that they described right on top of this example I’ve put together.

modeling meta-strategies

We need one more element to manage all of this – a meta-strategy – that is, a strategy for managing other strategies.  We support this idea in stratbox with what we call a controller strat.  The cannonical example of a controller strat is our implementation of the (Don’t) Panic button (the bright red button with an ! in the upper right hand corner) which will get all strategies out of the market with varying degrees of urgency and finality.  This is implemented as a controller strat which injects a stratpart into all running strategies.  The injected stratpart first deactivates its peers and then cancels outstanding orders and then gets out of the market with whatever urgency the user has required.

allocating to portfolios of strategies

In my example above, I have 48 strategies running in simulation across 6 futures markets.  Let’s imagine that at any moment in time, I will always have at least one strategy of each type (MR and TF) running for each market. Thus, I’ll always have 12 live strategies running against real money accounts and 36 candidates running against a simulator.  Effectively, we’ve just described a portfolio of 12 elements where each element is itself a strategy.

It’s not difficult to imagine implementing a controller strat which on a regular schedule will simply adjust the parametrization of the running strategies and their allocation of cash based on the performance of the strategies running in simulation.

It’s not difficult to imagine, but implementing it inevitably poses a variety of challenges not least of which is finding the time!  For now, we’re still finding new and interesting ways to express strategies which deal with portfolios of exchange-traded instruments.  But I think that this is just a point on our technology’s maturation curve which will ultimately lead to the regular development of strategies which operate on portfolios of strategies themselves…

Anyway, it’s nice to see a bit of academic work formalizing what I expect is a natural step in the evolution of trading algorithms.

This isn’t surprising as I wanted the system to support all of the activities associated with our algorithmic trading.  It turns out that that’s a lot to ask of a system.  It also turns out that you learn as you go and so the system continues to evolve.  A few years ago I’d posted about the basics of a strategy container and in this post I’m going to come back to this topic and describe some of the layers of code and thought developed since then.

First, let’s consider the role of a strategy container.  Its job is to intermediate between trading strategies and the external environments with which they interact.  It must also provide services that strategies can use (e.g., position management) and that it wouldn’t make sense for each strategy to re-implement.  In the past I’ve focused on the former responsibility of adapting strategies to external environments.  Why is this necessary and interesting?  Because it allows us to take the same exact strategy and run it live, or in simulation or in backtest, etc.  Interesting and necessary, but not what I want to focus on this time.  Instead, I want to look at the services provided to strategies; the ‘ecosystem’ a strategy container provides in the hope that strategies might flourish within it.

We’ll go from the bottom up.  At the bottom you have one or more pipes of market data coming in.  You might also have news feeds, weather feeds or other such things coming in and we’re just going to conflate them all and say “market data.”  Still at the bottom, you also have one or more two-way pipes with your oms or broker or directly with exchanges or with an exchange simulator.  Within Stratbox we call this abstraction an “ExecutionPlatform” because it’s conceptually where trades get executed.  The lingua franca for execution platforms is FIX, so the baseline object model of a strategy container is likely going to look a lot like messages described by the FIX spec.  Here we have orders, executions and the like.  Within Stratbox we’ve implemented an exchange simulator, a QuickFIX-based FIX interface and a couple broker-specific APIs and they are all of type ExecutionPlatform.  To a strategy, they all look the same.  Likewise with market data.  Within the strategy container, we provide a publish-subscribe model wherein any kind of market data can be subscribed to regardless of the ultimate source of the data.  The intention is always to intermediate between the strategy and its external environment.

the baseline object model

Great.  But now I want to write a trading strategy.  What’s that going to look like?   And this is where things get increasingly interesting as we get to decide what kind of facilities we’re going to provide.  We also need to manage concurrency in some fashion as trading is an intrinsically asynchronous activity.  For this lowest level of strategy, we’ll say that each market feed is handled by a thread which timestamps and enqueues a “Quote” for later consumption and redistribution to relevant subscribers by a MarketDataService within the container.  Likewise, each execution platform will be serviced by a thread who similarly enqueus “Execution” objects from exchanges real and simulated.  The threads handling executions should likely have a higher priority than the threads handling market data.  So, the strategy is naturally exposed to the markets’ asynchronicity right off the bat.  What baseline facilities will our strategy have at its disposal?  Well, it needs to be able to manage orders and positions, so it needs some kind of a “blotter” facility.  With this in place, a strategy can safely assume that it doesn’t need to listen to each execution coming in just to have a correct picture of its book.  It needs access to a MarketDataSvc and it likely also needs access to historical data.  But, staying simple, that’s more or less all we have to provide and we’re going to provide all of these things to the strategy through a single handle: a strategy context.  By swapping out a strategy’s context, we can move the strategy among environments (eg, from simulation to live execution).

With these facilities in place, basic algorithms can be implemented within a strategy and you should be able to test and trade them.

But you’re dealing with very low-level stuff.  You’ll also find that you’re writing the same kind of code over and over.  This inspired the introduction of a layer on top of the baseline object model which I’ve described before: StratParts – a strategy component model.

StratParts – a component model for strategies

With StratParts, we introduced metadata and composition (in the “composite pattern” sense) to our Strategy object model.  A strategy is itself a stratpart which contains other stratparts.  Thus, stratparts introduce a hierarchical structure to a strategy.  Each stratpart publishes a metadata descriptor which is aggregated at the strategy level.  The descriptor contains all of the modifiable characteristics of the strategy and can be changed while a strategy is executing thus giving a sort of grey box capability wherein a trader (or another strategy etc) can modify the behavior of a strategy as it’s running.  Stratparts also create an effective means of providing scoping within your strategy environment.  Stratparts can “see” the activity of other stratparts within the same strategy – they’re all peers in this sense.  This scoping can also be used for allocation of resources across different elements of a strategy.  For example, one stratpart might be allocated 80% of the cash available to the strategy while another stratpart manages the remaining 20%.

Stratparts have proved to be very powerful and we’ve written many of them though we’ll sometimes write strategies as one monolithic stratpart where it makes sense.

Stratparts have a lot of uses but one thing they don’t do is help manage the low-level complexity inherent in trading activities.  This was best described by a trader with whom I’d collaborated.  He’s a reservist in the IDF and uses military metaphors like I use sports metaphors.  He bemoaned the low-level handling of orders required for a very close-to-the-market strategy we were working on and said that:

we need an order like a ’smart’ missile: fire it and forget it

He was right and that inspired the next level of abstraction/support that we built into the system.

’smart’ orders and the tradeflow stratpart

The strategy we were developing was meant to look at a universe of futures spreads and generate all of the ‘cycles’ that might result in an arbitrage opportunity.  Since these are rare at best, we were really looking for circumstances where it looked like we had an advantage based on depth-of-market and various heuristics we’d apply.  Among the functionality that he wanted was an ‘order’ which would act as a limit unless some conditions obtained in which case it might pay the spread or otherwise stop being a fixed limit order.  He didn’t want to handle this inside the strategy but instead wanted to fire these things off and forget about them unless they required attention (eg, if they’re rejected by the exchange or his firm’s risk management checks).  Although we ended up calling these smart orders, they’re really a sort of very localized execution strategy themselves.

To support them, we utilized the same workflow (= state machine) framework we’d implemented for our ExchangeSimulator as we wanted to ensure that these smart orders had a very well-defined life-cycle with a clear set of states and guarantees about transitions among them.  We embedded this workflow engine into a stratpart which supported smart orders operating within this “tradeflow”.  Thus, the extra functionality (and weight/expense) of this functionality is only achieved/incurred when actually required or desired.  This has proven to be a powerful addition to the system and raises the level of services enjoyed by prospective strategies substantially, particularly given the variety of orders we’ve implemented which stand ready for use by any new strategy that might want to employ them.

At this point, it should be clear that we’re blurring the lines between the strategies themselves and the container.  Another way of thinking of it is that one container can have a set of pre-made building blocks that can be applied for families of strategies.  Each one provides a particular grammar with which a strategy’s aims can be expressed.  For very close-to-the-market strategies where very low-level handling of orders is required, the tradeflow stratpart provides a set of really useful facilities, but if I’m just trading with limits then it might not be so interesting.

portfolio-oriented strategies and beyond

I’ve mentioned before that I only care to think of strategies that operate on a portfolio.  As such, we’ve written a good deal of facilities for creating, analyzing and manipulating portfolios within strategies.  By using these facilities within a tradeflow stratpart, I can express quite complex strategies relatively simply by using portfolio analytics to determine my current ‘model’ portfolio and then using smart orders to most effectively transition me from my current state to my model state.  For the kind of trading I do, this provides a rich ‘vocabulary’ for the development of strategies.  All the same, my brain is having a tip-of-the-tongue moment as I feel that there’s an entirely different model possible for portfolio-oriented strategies just at the periphery of my imagination…  oh well, there’s always something new to explore.

Different trading styles, perspectives or trading problems are undoubtedly better served by different sorts of facilities.  I’m finding that just as language constrains and shapes thought, the strategy container that you employ shapes and constrains the kinds of strategies that you can implement.

If you have any ideas about such facilities for any kind of trading you do, I’d love to hear them.

–

a note about Stratbox

Although I’m talking about our system in this blog, we’re not marketing the system and happily “just” use it for our own trading activities.  We had looked into marketing the system previously but ultimately feel the same about selling stratbox as selling strategies…

I’ve been saving the above image in a stubbed-out blog post I’ve wanted to write since a conversation I’d had in Jerusalem last fall.  The recent attention to high frequency trading and all of its attendant evils has reminded me that the topic is relevant and so I relate various thoughts at the risk of jumping on a cacophonous bandwagon of rumbling misinformation.

First of all, the conversation.  It was with a talented guy who acted as the CFO for a variety of companies including a small startup hedge fund which traded US equities at a high frequency.   Although he was a part-time cfo, he seemed pretty plugged-into their trading operations and noted that they use an agency-only brokerage service for automated traders I’m familiar with and that they were “looking at full data for many” hundred stocks concurrently. He remarked that their trading was going well but that their hit rate was something like 4% and dropping.  By hit rate, he meant that they were placing limits frequently and generally pulling the orders if they didn’t get hit immediately.  He didn’t specify, but I imagine that “immediately” might range from milliseconds out to a second or twenty.  If the market is composed of makers and takers, then these guys were definitely makers of liquidity in the strict sense that they were placing limits and making markets.

At the time I thought it was interesting because it seemed that so many people were focused on the very, very short term trade that the frequency was becoming saturated.  It looked like a reminder that trading frequencies populate a spectrum; in this case, this part of the spectrum was becoming so saturated that returns were becoming increasingly difficult to obtain as more players crowded into it.  I’m not sure how this hedge fund has fared, but at the time I remember thinking that they were going to have a tough time competing if they were only geared for high-frequency trading as the space becomes increasingly expensive to play in as the inevitable talent and technology arms race marches on.

Lo and Khandani provide the below image illustrating this phenomenon happening to a class of contrarian strategies Lo & MacKinlay had described in 1990.  The strategies stop working as people squeeze out the alpha.

My conversation in Jerusalem mostly made me think that we were seeing a similar phenomenon amongst HF strategies.

What does it mean for a strategy to be high-frequency?  First of all, it’s a large class of strategies which probably shouldn’t be treated uniformly.  What they have in common is an intention to trade in and out of positions on a frequent basis where frequent will range from sub-seconds out to perhaps several seconds or even minutes in particularly felicitous cases.

Aside from the fact that one can trade at various frequencies, one can mix them and one might even be only peripherally aware of doing so.  A long-only, fundamentally-driven mutual fund (ie, not an algo or high-frequency trader) might call/fax/email/ftp/fix etc their trades into their brokers who might then execute the trades with their in-house or outsourced/white-labeled execution-quality algos.  Those algos might use some very clever close-to-the-market analytics to provide great execution for the client.  Or they might be traded profitably against.  Or both.

In any case, to me it seems clear that there is nothing intrinsically wrong about high-frequency trading itself.  People will always try to react to information as quickly as possible.  Why wouldn’t or shouldn’t they?  They also like to be clever.  Again, why wouldn’t this be expected and good?  I remember Lefevre recounting the use of personal teletypes by big speculators at the turn of the (prior) century.  Not your everyday household item at the time.  I also remember him recounting strategies for moving large positions which involved both buying and selling to hide one’s hand.  Why wouldn’t algos do the same and more?

That the loudest critics have been old-style execution traders “talking their book” to me tells the story here.

One other thing that my conversation evinces is the kinds of biz models being employed by brokers.  Like I said, this little hf hedge fund used an agency-only brokerage that caters to algo/hf traders.  Why is it important to note that it’s “agency only”?  Because, as the Goldman/Aleynikov story illustrates, strategies are organizationally porous in the sense that their value drips away as the human capital behind them moves from organization to organization and understanding of the strat’s internals become understood more broadly outside the organization.  This is likely the dynamic that drove Lo’s example above – more and more traders were employing similar strategies as the knowledge of the strategy leaked further and further from its original source(s).  Likewise, if I see all of your trading activity in sufficient detail, I might be able to reverse-engineer your strategy and work to steal your alpha.

Thus, traders are happy not to advertise their strategies’ behaviors.  So an agency-only broker – a broker who doesn’t engage in prop trading themselves – should inspire trust in a potential client.

Great!  What an honest business model!  But there’s an irony here and it points to the real kinds of problems we should be bugging our regulators to be addressing.  The specific agency-only broker I mention wasn’t self-clearing.  That is, they were using another broker (Goldman in this case) to handle their backoffice duties.  And guess what?  Goldman is anything but agency only.  So, clients who feel warm and fuzzy that they are dealing with an agency-only shop are actually exposing all of their activities with a particularly sophisticated and arguably predatorial prop trader.  It’s like the Guiness ads.

It’s funny to me that while there sits a multi-trillion dollar hole in our fed’s balance sheet which such hippies as fox news and bloomberg (suing to find out) and our US congress (asking politely) can’t seem to tease an explanation for out of the relevant authorities, the blogosphere and regulators seem to focus their invective on short sellers, hedge funds and high-frequency traders.

secret meeting at 85 broad?

as suspected, the seat of evil can be found in NJ

There’s a whole bunch of interesting things about this story of how a programmer has allegedly stolen some of the code at the place he’d worked.  One is the remarkable reverb it’s created amongst bloggers.  The house pictured left is evidently the diabolical mastermind’s home according to the NJ Real Estate Report.  Another is the fact that a programmer stealing some code is news. Funny what becomes news (apparently a fêted pedophile died) and what doesn’t (we are creating millions of refugees in Pakistan).

One angle that I haven’t seen highlighted in all of the commentary is Mr Aleynikov’s choice of weapon.  Seems that he was an erlang guy with an interest in ocaml.  Choosing functional programming for algo trading systems is an interesting but not unique choice.

A few months back I’d seen Jane St’s Yaron Minsky give an interesting talk on OCAML in which he describes the virtues of ocaml as applied to the algo trading domain.  You should watch it as he’s entertaining and clearly a “true believer.”   The key argument he makes is that algo trading requires correctness, agility and performance and that ocaml provides a great set of tools for addressing these needs.  Correctness and agility are supported by the rich type system while performance is just a happy feature of the language.

He describes Jane St’s need for correctness as obsessive – the company’s founders read/review all of the code that goes into production.  Thus, concision and the absence of OO languages’ dynamic dispatching mechanisms are viewed as advantageous.

Mr Minsky admits that there are significant limitations involved in choosing ocaml, including a dearth of external libraries (they write their guis using curses(!)) and a lack of native threading.  The language’s unpopularity turns out to be something of a double-edged sword as it results in the lack of usable libraries but also in a close-knit community of likely above-average developers.

Anyway, it’s an interesting decision for Jane St Capital to have made and it’s compounded by Mr Aleynikov’s use of Erlang/OTP at Goldman.  His linkedin profile describes his recent work as:

• Lead development of a distributed real-time co-located high-frequency trading (HFT) platform. The main objective was to engineer a very low latency (microseconds) event-driven market data processing, strategy, and order submission engine. The system was obtaining multicast market data from Nasdaq, Arca/NYSE, CME and running trading algorithms with low latency requirements responsive to changes in market conditions.
• Implemented a real-time monitoring solution for the distributed trading system using a combination of technologies (SNMP, Erlang/OTP, boost, ACE, TibcoRV, real-time distributed replicated database, etc) to monitor load and health of trading processes in the mother-ship and co-located sites so that trading decisions can be prioritized based on congestion and queuing delays.
• Responsible for development of real-time market feed handlers, order processing engines and trading tools at a Quantitative Equity Trading revenue-making HFT desk.

which makes it sound like they were using Erlang alongside a lot of C++.  I’d also speculate that their use of Erlang was more about concurrency than functional programming.

One of the nice observations made by Mr Minsky is that there’s a magnificent freedom about prop trading: you don’t have clients to please and can thus make genuinely original technology decisions.  It looks to me like the curious case of Serge Aleynikov highlights some of that originality in play in serious algo trading prop shops.

]]> http://www.puppetmastertrading.com/blog/2009/07/08/the-other-interesting-thing-about-the-serge-aleynikov-story/feed/ 4