The example I’ve chosen is, I think, very nice because it’s a portfolio-oriented strategy, which is pretty much the only kind I care to explore; it’s also based around the concept of pairs trading, which is something which most can easily relate to; and finally, it’s already public domain and yet almost certainly has some juice in it for those who care to understand it and extend it intelligently.

The example comes from the blog of a company, Palantir which does (something like?) analytical / decision support software for both finance and intelligence-gathering services (quants and spooks – spooky quants?).  The specific example is here and is described thusly:

In this study we explore a trading strategy that isolates pairs of instruments within a sector that are highly correlated. We enter a trade if the price paths of these instruments diverge, going long one instrument and short the other, with the assumption that their price paths will converge.

We construct our strategy in four parts: (1) isolate the target set of tradable instruments, (2) choose a rule for finding correlated pairs within that set, (3) pick criteria for entering trades, and (4) test the trading strategy.

…

For this study we want to isolate pairs of instruments whose correlation is above a certain threshold. At the beginning of each quarter, we restrict our target stocks to the top 5 in the sector by 252-day correlation to the S&P 500. Then our Strategy uses a metric that, for a given group of stocks, outputs a list of pairs with a correlation above our target threshold of .60 over the same time period.

…

Across every day and every pair in our trading list, we check the 21-day z-score (number of standard deviations from the mean) of the difference between the two series. We classify the stocks as diverging when the absolute value of the z-score is between 1.5 and 3.0. When this condition is met we short the stock that is rising and long the stock that is falling, sizing our position relative to the z-score and the number of pairs currently trading.

~(sectors) ; reframing the strategy

While I’ll implement a strategy broadly based on their description, I will use some poetic license to modify it in ways that seem to me sensible.  First, I have good experience writing equity strategies which deal with sectors and have pretty uniformly found that sectors, as defined by S&P or whomever, are more trouble than they’re worth.  Instead, looking at correlation matrices and determining on that basis who should be paired, is much more fruitful in my experience.  Tr8der has some very nice posts illustrating (very prettily!) a similar point here and here.  (Actually, his point is a bit broader and represents an area of possibly promising development of this strategy which I won’t pursue here.)

So, we’re going to chunk the idea of explicitly using sectors.  Instead, we’ll simply construct, on a monthly basis (Palantir did this quarterly) a TrailDays correlation matrix from which we’ll extract the set of pairs which have a correlation greater than MinCorrelation.  On a daily basis, we will calculate the ZScoreDays z-scores of each of these pairs and sort them by absolute value, tossing those whose values are less than MinZScore or exceed MaxZScore.  Since I don’t like the idea of having an open-ended number of pairs/positions in my portfolio, I will constrain the portfolio to a maximum of N names and will Allocate $1M to the strategy.   Finally, we will look at the effect of trading an EvenlyWeighted portfolio or a z-score weighted portfolio (as Palantir had done).

The initial ‘universe’ of equities across which we will calculate our correlations will be filtered from my database of daily equity data going back to 2005 where the closing price is over $1, the average daily dollar-volume is over $50M and the data is relatively clean (few if any gaps).  This yields an initial universe of ~600 equities and ~180K pairs.

Assuming that we keep N above 20 or so and we keep our allocation constant at $1M, and assume execution at the close price, we should expect our results to be reasonably believable as it’s unlikely that the size we’re trading in would distort the market.  Scaling any strategy presents its own set of challenges which we’re not going to go into for this example, but an obvious direction for scaling the strategy would be to simply run it over intraday data and scale into and out of positions as conditions permit.

Ok, so that’s what our strategy is going to look like.  Now, how do we “make it so”?

structure of a stratpart

Within Stratbox, strategies are represented as composites made-up of ’stratparts’ each of which has its own metadata descriptor containing parameters.  Stratparts can be composed together within a strategy to enhance or modify the behavior of a strategy.  Thus, one can have a stratpart which implements the above logic paired with another stratpart which will e.g., modify the portfolio to impose beta neutrality, damp volatility or some other form of reporting or risk management.  Given that this example is already complexish, we’ll just stick to the one stratpart.

A stratpart has a few key elements.  One is the metadata descriptor which allows the strategy container (stratbox) to interact with it for back-testing, optimization or forward-walking purposes, or to allow a ‘trader’ to interact with it dynamically during run-time (the so-called ‘gray box’).  Beyond that, it has an essentially event-driven design.  The method quote() is called when relevant (i.e., subscribed) market data arrives.  The method execution() is called when executions for orders that originated with this strategy are received.  And the method strategyEvent() is invoked on a variety of ‘interesting’ events that the strategy implementer might care about.  None of these methods are required, so in its simplest form, a stratpart is a very simple piece of code to write.

services in a stratpart

Once one has complied with the minimal requirements for implementing a stratpart, one is rewarded with a variety of useful services: subscription-based market data feeds as well as a fast snapshot db and an in-memory historical db for quick correlation analyses cover market data needs.  Position management is handled at both the strategy level and at the exchange level (if you’re trading across multiple exchanges).  There is more, including performance analysis (e.g., sharpe, and its various analytic friends) and hierarchical cash allocation within and across strategies, but we won’t look at these in this example.

implementing metadata

So, the first thing we need to do is define our metadata.  This is done through the use of a conventionally-named static method Descriptor().

public static Descriptor Descriptor() {

 ArrayList<Param> params = new ArrayList<Param>();

 String cd = "Unleveraged capital to apply";
 params.add(new Param(InitialBalance, cd, false, 1000000.0));

 String n = "# of instruments to hold in portfolio";
 params.add(new Param(N, n, true, 20));

 String tc = "Trailing days over which to calculate correlations";
 params.add(new Param(TrailDays, tc, true, 252));

 String mc = "the minimum correlation of pairs to consider";
 params.add(new Param(MinCorrelation, mc, true, .75));

 String zd = "trailing days over which to calc z-score";
 params.add(new Param(ZScoreDays, zd, true, 21));

 String mizs = "minimum z-score";
 params.add(new Param(MinZScore, mizs, true, 1.5));

 String mazs = "maximum z-score";
 params.add(new Param(MaxZScore, mazs, true, 3.0));

 String ew = "evenly-weight portfolio? (or weight by z-score)";
 params.add(new Param(EvenlyWeight, ew, true, true));

 Descriptor _desc = new Descriptor
   ("puppetmaster.strats.Pairs", _Desc, params);

 return _desc;
}

public Pairs(Strategy strat, Descriptor d) { super(strat, d); }

Not the most interesting chunk of code, but pretty straightforward and its inclusion opens up a lot of functionality within the environment as each of these parameters can now be systematically or manually modified in-flight.  The third parameter to the Param constructor, a boolean, indicates if we want to allow the system to optimize this parameter.  Thus, the allocation/initialBalance parameter isn’t optimizable as it wouldn’t make any sense to do so.  Even so, we can (ourselves or through an allocation algorithm) modify this value to force the strategy to lighten or extend its financial footprint.  The others are fair game for optimization.

The constructor is also required – with that exact signature.  This is pretty much it for the ‘boilerplate’ code required.

‘installing’ the stratpart

Now that we’ve met the minimum requirements for a stratpart, we can install it into stratbox.  This requires adding one line to your personal stratparts.xml config file.  Like so:

<?xml version='1.0'?>
<stratParts>
 <descriptor class='puppetmaster.model.strategy.FwdWalker'/>
 <descriptor class='puppetmaster.model.strategy.StrategyPortfolio'/>
 <descriptor class='puppetmaster.strats.Pairs'/>
 <descriptor class='puppetmaster.strats.AMBO'/>
 <descriptor class='puppetmaster.strats.meta.AdjMeanReverter'/>
 <descriptor class='puppetmaster.strats.meta.AdjTrendFollower'/>
 <descriptor class='puppetmaster.strats.meta.Binary'/>

Once we’ve informed stratbox of the new stratpart we’ve written, we can view it within our library of stratparts within the strategy wizard in stratbox:

Our Pairs strategy in the Strategy Wizard's stratparts library

And all of the parameters we defined within our descriptor show up on the next page where we can parameterize or optimize the strategy:

Configuring the Pairs stratpart within the strategy wizard

This means that the stratpart can be combined with other stratparts to create new strategies as I’ve described  and that it can partake in the rich set of functionality within the stratbox gui.  Including actually trading.  But first, we must make the strategy actually do something…

implementing event handling

To that end, we must implement the event handling methods I’d described previously.  Since we’re not going to do any detailed execution handling in this strategy, we’ll only implement the quote() method and a subset of the events sent to the strategyEvent() method. In particular, we’ll listen to activation events (so we can initialize cleanly), changes to our descriptor and changes to our positions.

/** listen for select strategyEvents */
public void strategyEvent(StrategyEvent event) {
  super.strategyEvent(event);
  switch (event.type) {
    case Activated:            _init();                 break;
    case DescriptorChanged:    _readDesc();             break;
    case PositionChanged:      _posnChange(event);      break;
  }
}
<pre>

For quotes, we’ll just listen to the Beginning of Day (BOD) event.  If we wanted to trade intraday, we’d have to listen to tick events, but we’ll leave that for another day…

From a high-level, our code is pretty simple.  We determine the universe of pairs we might trade, we select from among them, we determine what our new portfolio should look like and then we trade to transition from our current state to our desired state.  That’s it.

/** mkt data goes here, but we only listen for the BOD
 * ("Beginning of day") event */
public void quote(Quote q) {
  if (q.type() != Quote.Type.BOD) return; // daily strat...

  try {
    List<_Pair> pairs = _getPairs();
    List<_Pair> sel = _selectPairs(pairs);
    List<Position> dp = (List<Position>)_desiredPfolio(pairs,sel);
    _Log.debug("PORTFOLIO: "+dp);
    _trade(dp);
  } catch(Exception e) { _Log.error(e.getMessage(),e); }
}

the messy details

Digging into the details gets more complex, but not horrifically so, as stratbox already has basic analytics like correlation matrices, z-scores, volatility built into it.  Likewise with basic portfolio analytics and transformations. Thus, our code to trade a (pairs-based) long-short portfolio with an arbitrary number of elements is surprisingly simple:

/** given a desired portfolio, trade to make it so */
void _trade(List<? extends Position> desiredFolio) {
  PositionRecord[] currFolio = posns();
  // Given our current portfolio (posns) and our desired state (dfolio),
  //  we generate the set of orders which will transform from the former
  //  to the latter
  //
  List<Order> orders = PortfolioComposer.TransformPortfolio
    (currFolio, desiredFolio, _orderF(), _strat, _strat.account());

  // place the orders for execution
  for (Order order : orders) {
    try {
      order.setType(Order.Type.MOC); // we trade at the close only
      _subscribe(order.contract());
      _execP().placeOrder(order);
    } catch (Exception e) { _Log.error(e.getMessage(), e); }
  }
}

As promised, I provide the full source listing below for those interested, but won’t over burden the discussion with a complete description of every little piece of it.

a telling omission and some results

Early in my career, working as a software engineer in a wall st front office technology department, I was befriended by a manager who told me he had once been a trader.  I asked him why he wasn’t a trader anymore.  He quipped:

I was half of a talented trader.  I knew, in my bones, when to get into a trade.  Sadly, I was never really sure when to get out…

It seems that the author of the Palantir example suffers a similar characteristic to my old friend (or is just being understandably cagey) – she specifies an entry condition, but not an exit!  Since she doesn’t impose any costs for trading, this has no big side effects, but in stratbox we do impose (pretty onerous -> realistic!) costs for trading.

In any case, since this is just a simple example I’ve maintained the ‘no exit’ policy and simply exit a trade once its pair goes out of the acceptable ranges for either correlation or z-score.  This has a bad effect on the strategy as you end up churning a lot of trades at the high end of the z-score range.  That is, if we set our upper boundary for z-score to be 3, then we might enter and exit a position multiple times as the z-score oscillates between, say, 2.9 and 3.1.  A more careful implementation could minimize this effect.

Another difference imposed by our more realistic simulation as opposed to the original example is that I’m restricting the portfolio size explicitly and explicitly limiting myself to trading a given contract within one pair as opposed to allowing a contract to trade across multiple pairs (invoking higher costs but allowing positions to offset and thus providing a neat little optimization – assuming trading costs nothing!).

All the same, the strategy clearly has some positive characteristics and when parameterized to take advantage of its naive exit strategy (by making the z-score low-end quite low, e.g., 0.5) takes on the character of a hedged trend-following strategy.  How so?  Well, while we are getting churned around the high-end (entry) z-scores, our profitable exits only happen once the z-scores have diminished (the prices have converged) considerably.  So we take more, smaller losses while our winners are held a bit longer.

some Pairs results in '10

Adding to my portfolio size improves things generally (and also increases the strategy’s scalability) while increasing the MinCorrelation parameter seems to hurt performance (not sure why, honestly – perhaps because if they’re very highly correlated and already diverging so strongly it may mean they’re actually diverging fundamentally). Returning to the topic of costs, each of these strategies incurred costs of over $5K for the period in question. And returning to the parameter which governs the weighting strategy employed, although I haven’t illustrated it here (as I haven’t looked at it very carefully), my initial finding is that it changed very little. 

There’s a great many areas where this strategy could be improved – that’s the fun, after all! – but hopefully even in its primitive form this example brings to light both an interesting baseline portfolio pairs strategy, while answering the question of how one goes about concretely implementing it within stratbox.

The entire strategy weighs-in at ~400 LOC including comments, boilerplate and individualized imports.  The source is here: Pairs.java.

photo by Simon Rogerson

Some time back, I’d written about NVidia’s CUDA noting that it looked ideal for many asset-pricing and monte-carlo type problems in finance.  At the time, I was hopeful that it would be quickly integrated into existing open source efforts like QuantLib, but adoption has proved slower than I’d hoped, most likely because implementing non-trivial problems on CUDA is, well, even less trivial than doing them without..

LMM on CUDA

Happily, I’ve just seen a promising first step in this direction as Über-quant and C++ artisan Mark Joshi recently announced an open-source project, Kooderive which looks to implement the LIBOR Market Model (LMM)  on top of CUDA.  His announcement on the QuantLib mailing lists reads:

Dear All,

various people have shown interest in the use of CUDA with QuantLib. I
have now made some progress on a CUDA implementation of the LIBOR
market model.

In particular, I now have a path generator for the LMM working which
does 16384 paths for 40 rates, 40 steps, 5 factor model, displaced
diffusion predictor-corrector that takes 0.1 seconds on my Quadro 4600.

The state of the project is code fragments that can be called from
other code. Those who are interested can get the code via
the subversion repository on kooderive.sourceforge.net .  The only
project file is currently for VC9 x64. It also uses thrust and the
CUDA SDK.

The next stage will be writing routines, that use QuantLib for the CPU
stuff and kooderive for the GPU stuff,  to actually price things.

A gentle reminder that I will be giving a course on the LMM and
QuantLib in June in London, and I will include a session on kooderive
if there
is sufficient interest.

I am happy to take code contributions for kooderive. However, I am not
looking for a redesign of the library or contributions which introduce
dependence on other libraries. I am interested in contributions of
separate routines and of optimizations of existing routines that do
not change interfaces.

regards

Mark
–
Pricing exotic interest rate derivatives – The LIBOR Market Model in
QuantLib June 2010, London,
http://www.moneyscience.com/training/index.html

Assoc Prof Mark Joshi
Centre for Actuarial Studies
University of Melbourne
My website is www.markjoshi.com

But some are revealing and one in particular is 100% required reading for anyone interested in electronic markets.

The writer introduces himself thusly:

I am Steve Wunsch, the principal inventor of two SEC‐regulated stock exchanges, the Arizona Stock Exchange “AZX” (originally called Wunsch Auction Systems, Inc. “WASI”) and the ISE Stock Exchange, both of which include dark pools. In fact, both of them, like all modern stock exchanges, have both lit and dark components and, thus, have provided me with potentially useful perspective on the dark pool question and on transparency in general. I will focus heavily on the latter, for it is impossible to understand the dark pool issues raised without understanding the value of transparency or, if improperly applied, the lack thereof. The AZX experience was, I believe, particularly instructive in this regard. Its highly transparent call market structure, combined with its unique regulatory status as a “low volume exempt”exchange, enabled me to see transparency and the role of regulation in promoting it from a perspective that I don’t believe anyone else has.

He deftly mixes snark and a historical perspective on regulation with an opinionated and informed view on the forces driving current equity markets’ microstructure arguing that the worst issues are due to regulatory failures.  He concludes, logically enough, that the SEC should be disbanded.  Perhaps his most inflammatory bit is his claim that the “SEC made Madoff.”  For effect, the section is entitled “An American Oligarchy”:

AN AMERICAN OLIGARCHY

It is not in the Commission’s interest to admit failures of policy, such as the ones I have described in this letter, and I have never seen it done. It was not in the Commission’s interest to admit that Bernie Madoff was the SEC’s most trusted and intimate confidante in formulating and selling transparency, electronic trading and
the whole NMS concept to Wall Street, the public and Congress. His legitimate business was the epitome of the kind of transparent electronic competition that NMS’s leveling policies were trying to create, and he occupied the most favored place of all industry advisors on policy and rules as NMS was being created. In a very real and literal sense, Madoff’s legitimate business and NMS were made for each other. NMS cleared a path for the application of continuous transparency by new electronic competitors, very visibly led by Madoff, enabling him to become at one time the third largest market in the United States, even though he wasn’t officially registered as anything but a broker‐dealer.

Had the SEC not emasculated the rules by which the NYSE controlled its members, Madoff would never have happened. In the time before NMS, when the exchange had Rule 390 or the stronger Rule 394 before it, diverting orders away from the floor or selling them to Madoff would have been banned. But on antitrust principles, the SEC wanted to foster NYSE‐busting competition in NMS, and Madoff became its PosterBoy for such competition. In order to make way for him, the SEC opened up a variety of loopholes that allowed orders to be diverted from NYSE to Madoff and printed on regionals like Cincinnati. Rules 19c‐1, 19c‐2 and 19c‐3 were in this vein. There were perennial attempts by the NYSE to plug the loopholes and rein in the membership, but the SEC batted them all away, enabling Madoff to continually grow his business. Eventually, the NMS environment forced the NYSE to abandon Rule 394, then Rule 390 and ultimately its membership organization altogether when it demutualized. This was all very good for Madoff. And Madoff was very good for NMS, giving it industry cred far in excess of what this poorly articulated socialist leveling theory could have had without his support.

In spite of a 457‐page SEC investigation into Madoff and how his Ponzi scheme was missed, the most obvious reasons were not considered, namely, that Madoff played a central role in helping the Commission design and sell NMS, and that NMS made him rich long before the Ponzi scheme. Most importantly, the credibility that theCommission’s collaboration with Madoff on NMS conferred on him was the principal factor enabling him to bring in money for the Ponzi scheme. Although the investigation’s report notes his credibility in the industry, it is mentioned as if itwere just a fact of life and was already there. Not mentioned is that his superior access to the SEC and apparent influence over the Commission, both of which were implicitly proved by his ability to get rich on NMS, are the most important reasons that he had such extraordinary credibility in the industry. The truth is that the SEC made Madoff. He could not have existed as a threat to investors without the Commission’s active and dedicated support over several decades.

Although, in typical blogger fashion, I’ve highlighted his spiciest claim, the rest of the letter is more technical and informative while just as entertaining.  I encourage you to read it and then engage in a thought experiment in which You are the designer of an electronic exchange and must balance the needs of a very heterogeneous set of users and stakeholders while ensuring transparency, liquidity, profitability, “fairness”, performance (he references an exchange targeting 100M executions per second) and utterly fail-safe transactional integrity…

I have embedded the full letter below the break…

FIX interface?

Cloud computing looks to have turned yet another interesting corner.   This time the turn leads towards the development of a liquid, fully electronic new marketplace in “spot instances”.

‘Spot‘ means what you would expect it to in the context of trading: the current pricing for immediate delivery of a commodity.  ‘Instance‘ is the atomic element within Amazon’s cloud environment; an instance is the smallest chunk of computing capability which can be provisioned within the cloud.

Amazon is making markets in cores and they’re exposing functionality just as a regular exchange would: both through user interface ’screens’ as well as programmable APIs.

From their announcement:

Spot Instances enable you to bid for unused Amazon EC2 capacity. Instances are charged the Spot Price set by Amazon EC2, which fluctuates periodically depending on the supply of and demand for Spot Instance capacity. To use Spot Instances, you place a Spot Instance request, specifying the instance type, the region desired, the number of Spot Instances you want to run, and the maximum price you are willing to pay per instance hour. To determine how that maximum price compares to past Spot Prices, the Spot Price history is available via the Amazon EC2 API and the AWS Management Console. If your maximum price bid exceeds the current Spot Price, your request is fulfilled and your instances will run until either you choose to terminate them or the Spot Price increases above your maximum price (whichever is sooner).

embedded optionality

While the inclusion of, effectively, a market data service is neat, probably the most interesting aspect of the initial protocol they’ve designed is that it contains embedded optionality and behaves a bit like barrier options.  That is, when I setup an ‘order’, I need specify a maximum price I’m willing to pay.  When the spot price drops below my max, I get “knocked-into” a contract and instances are allocated to me.  If the spot price rises above my max while I’m running, I get “knocked-out” of the contract and my jobs get terminated.

The intent is to allow for low-priority jobs to be dynamically run whenever pricing drops below a user’s threshold, but the (intended?) consequence is that it adds the delicious and malleable tang of path dependency to these instruments…

secondary markets, FIX, arbitrage..?

Amazon currently controls the market entirely, but it’s not hard to imagine a secondary market evolving.  Given that others are beginning to copy Amazon’s APIs, one can also imagine markets which operate across providers …  perhaps accessed via FIX?…

Who knows?  In the not-too-distant future, we may well be able to implement ‘core arb‘ strategies…or make markets in cores… or find that we can effectively hedge with disciplined exposure to the ‘core market’ or …

messages per second across "all" feeds

I came across this compelling site which uses a hardware-based ticker plant (Exegy) in a colo environment to measure peak bandwidth across scads of NA feeds and then, every minute, updates a chart like the above to capture the average messages/sec across all of them.  Pretty swank.

While the uninformed may rail against colocation rather than focus on less intriguing issues like banana-variety corruption, they miss the basic point that colo can be done by anyone with the checkbook and the wish to do so.

unfair advantage?

It’s sort of like that boat in Forrest Gump.  Forrest wanted to be a shrimper.  So he invested in a boat.  With his initial capital, hard work, perseverance and a bit of luck, Forrest made a go of it.  He might easily have not made it. Colo is like that.  You can shrimp without a boat if you have a mask and fins, but it’s likely not a sustainable model… either way, it’s hard to see the harm in Gump’s boat.  Or colocation.

Hat-tip to Rodrick’s Web Log !! for spotting the market data peaks site.

lemon-law arbitrage?

Derivatives pricing has always been a notoriously complex, computationally expensive and potentially breathtakingly remunerative undertaking.  This is true enough for relatively vanilla, exchange-traded options, but once one goes off-market and starts applying creative financial engineering, it can get much more complicated.  Products like CDOs, CDSs, CDO^2s and their ilk have exploded in recent years creating opaque markets of trillions of notional dollars and accounting complexities we’re still only beginning to understand.

A recent paper, Computational Complexity and Information Asymmetry in Financial Products, by Arora, Barak, Brunnermeier and Ge take things a step or two further as they illustrate using information theory that it may be far worse than imagined as totally undetectable fraud can be engineered into these products.  They show that fraud with these products can be undetectable in the sense that the pricing process is a formally intractable problem when the informational asymmetry inherent in the development of these products is taken into consideration. In this context, “informational asymmetry” is a polite way of saying “fraud.”

The authors, from the Department of Computer Science and Center for Computational Intractability at Princeton (man, I want one of their business cards!), demonstrate that if the designer of, say, a CDO wants to cherry-pick amongst bundled assets to maximize their own return, they can do so in a way such that it would be impossible for a buyer of the derivative to know they were being stiffed.  The problem can be so hard that if you got the NSA’s mythic clusters humming on a pricing model, they might chug away until the sun falls from the sky before they accurately price it…  Co-author Rong Ge provides a FAQ to the paper here and I must hat-tip Andrew Appel for his informative post on the paper.

The “perfect crime” is a puzzle that has occupied the (criminal and otherwise) mind of many a bright and motivated soul from time immemorial.  While some may indulge towards the vulgar or base through violence or vice and others might ponder the perfect crime of passion, the cerebral Queen of Crime is surely some form of regulatory arbitrage: committing the crime for which the law has yet to be written or creatively engineering a legal loophole for a crime one has perpetrated or is about to perpetrate.  The developers of CDOs are to be lauded as it appears they have materially upped the state-of-the-art of the perfect crime.

hmmm… Is there a Nobel for that?

you, hf-trading

There seems to be a developing meme out there suggesting that algorithmic-, and in particular high-frequency, trading is some kind of gold-rush route to easy money which brings to mind…

…this revision of a paper I’d read previously: “Statistical Arbitrage in the US Equities Market” by Avellaneda and Lee.   It’s a detailed and thoroughly worked (and now re-worked) paper illustrating the development and analysis of a US equity stat-arb strategy based on Principal Component Analysis (PCA) and then revised to use ETFs.

I came across this paper as I have still never used PCA in any of my own strategy development work and read Carol Alexander’s excellent Market Models over my summer vacation with an eye towards giving a PCA hedging model a spin in the near-term. Thus, I wanted another look at this paper as a reference point.  Although it’s an excellent paper, I’m not going to urge you to go out and read it immediately unless you have a reasonably pressing practical interest.  Instead, I find it interesting largely because of one of its authors – Professor Avellaneda – and its conclusions in the form of its strategies’ performance.

I’ve seen Prof Avellaneda speak a number of times at a variety of quant meetups organized by the relevant Columbia/NYU financial engineering depts.  His paper reminds me that at least once during my noisome adolescent years, my father intoned darkly that:

the streets are littered with brilliant minds

Implying that any wits I may believe myself to possess wouldn’t by themselves be worth much in life and that I’d need to bring actual tools to the task of solving problems if I wanted to address interesting ones.  Having seen Mr Avellaneda speak, I’m confident that at my peak, my “processor” was never as fast as his.  Much worse, there is no comparison between the tools he can level at a problem compared to me – he’s on an entirely different playing field so far as concrete mathematical/analytical capabilities go.  That’s why I go see him speak and read his papers.

Thus, the results of Avellaneda and Lee’s work are particularly interesting to me as they’re really pretty dull: something like a Sharpe of .9 and degrading briskly.  Now, you don’t expect people to be providing detailed recipes to wildly profitable strategies, and this result isn’t bad, particularly given that they’re describing strategies which likely have significant capacity.  Still, it illustrates that very smart people working with sophisticated mathematical tools even over extended periods are still operating under noteworthy constraints.  Perhaps also: ideas are relatively easy – examining them in the requisite detail is difficult and time consuming, even for (particularly for?) people with the most finely honed toolsets…

I frequently have friends or colleagues who will observe that if you “just write a strategy that foos when bar but yaddas when baz… you should surely make money.”  Maybe.  But the reality is that just putting together the strategy and working through it takes significant time for anything but the simplest strategies.  Once you add genuine complexity to a strategy, you can spend enormous time tuning it.

This, in turn, poses a dilemma I encounter frequently and honestly don’t have a great answer for:

how to find the balance between continuing development on a known good strategy and initiating the development and analysis of unrelated and novel strategies?

the back of the envelope as canvas

This next (de-)reference isn’t directly pertinent to algo trading, but the lessons learned by building BIG distributed systems can surely be applied elsewhere.  And they’re just plain fascinating.

Google’s Jeff Dean gave a recent talk entitled “Designs, Lessons and Advice from Building Large Distributed Systems” at the Large Scale Distributed Systems and Middleware (somehow “LADIS”) workshop and the slides are here.   Go read them.

If bald exhortation doesn’t convince you maybe slide 24 will:

or perhaps what he does with these baseline numbers in slide 27 will pique your interest:

back of the envelope as art form

One that made me (a serial prototype-builder) cringe:

Important skill: ability to estimate performance of a system design
– without actually having to build it!

Ouch.

maybe it is easy after all

Of course, if you’ve studied Avellaneda & Lee’s paper and it held no challenges or surprises and you’ve reviewed Mr Dean’s presentation and it’s old hat to you, too…

Even if it’s hard to find good stuff, one must still look as there’s always more information that can help you than you can effectively process or retain.  A few weeks ago I was trying to formalize the expected profit function of an algorithm I’m developing and wanted to see what people had written about the topic.  I entered ‘define profit function for trading algo’ into google and was pleasantly surprised to see a paper entitled ‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani.  It doesn’t directly cover the topic I was looking for, but instead addresses a number of related topics I’ve been interested in for some time:

• the treatment of a strategy as an instrument in its own right
• composing portfolios comprised of strategies
• using regime switching techniques to manage portfolios of strategies

In this post, I’ll briefly review their paper, illustrate how one can easily model strategies in relevant ways using the strategy ‘object model’ I’ve described previously through an example, and conclude with some thoughts on how these kinds of strategies might be implemented and further explored.

‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani

Their paper begins:

This paper considers the problem of dynamically allocating capital to a portfolio of trading strategies. The allocation should be robust, and the capital allocated to a trading strategy should reflect the confidence in the expected profit that the strategy will make in current market conditions.

Good trading strategies exploit recurring market dynamics that can be more prevalent in some time periods than in others. Indeed, the concept of regimes is fundamental to financial markets, and much research has focused on the detection of regime shifts. In this paper, we consider a regime as defined by a set of trading strategies that exhibit similar performance in a given time period.

We consider different parameterizations of the same strategy as distinct in our ground set of strategies. The trading problem is to pick a distribution over the ground set that will achieve good performance in the current time period.

They start with some interesting definitions.  A trading strategy is defined as a function which takes a given market State and generates a convenient Answer “denoting all possible trading decisions”.  Strategy functions can also be parameterized and different parameterizations of the same function/strategy are viewed as distinct qualitative strategies.  They assume the existence of  the set of all strategies.  A regime is defined as a set of trading strategies who behave similarly over a given period of time where similarly is measured by the variance of a fitness function.

So they start with a set of strategies, a fitness function and some historical market data or state, and their first task is to identify the regimes.  They do this by running the strategies against randomly selected partitions of historical data, quantifying their respective performances with the fitness function and identifying the clusters deemed statistically significant.  Those sufficiently significant and similar sets of strategies are themselves the reasonable regimes while the set of strategies within all the reasonable regimes are themselves dubbed the reasonable strategies – the candidates for execution.  This might sound strange, but as I’ll illustrate with a concrete example below, this is actually a pretty effective and even intuitive way of defining regimes.

They define function stFuncDist as, for any regime, generating a distribution or weighting of the strategies within the regime. While classifyMarket is a function that generates a distribution across all regimes for any given market state. Both models can be “trained” and they do so by again iterating randomly across chunks of historical data adjusting the weighting constants of these two functions.

Finally, it’s time to trade and they can effectively use classifyMarket to determine a predominant regime and allocate capital as indicated by stFuncDist to the appropriate set of “reasonable strategies.”

They claim to have utilized this technique to generate the results charted in Figure 1, but they give little detail on the initial universe of strategies.  I believe they had one strategy which was parameterized into a set of qualitative strategies and run across one market, the CME’s Nasdaq emini.  I have no reason to doubt that they had exactly this result, but without more details on what they actually did, it’s hard to view it as very meaningful, though it certainly is suggestive.  An interesting thing about how they present their result, which I’ve seen in other academic papers, is that because they’ve precisely defined the set of possible strategies they look at, they can also calculate max and min possible returns over the relevant period.

Figure 1: Out-of-sample profit (middle line) versus possible min (bottom line) and max profit (top line) over all strategies for the period 2006-11-01 to 2008-08-28.

The paper continues with another section, but I’m going to stop my overview here as the last part is less interesting to me and the interested reader should of course read it themselves.

complex strategies or simple strategies combined

One of the broadest design questions I see for the development of effective trading algorithms is whether one should employ complex, customized algorithms or instead utilize simple algos which can be simply combined to produce emergent complexity.  Although the authors of this paper allude to “Good trading strategies”, I wonder if one can even forget the “good” part and instead worry about applying – even bad! – strategies at the right times.

There are a lot of things I like about this paper, but probably most of all it’s their definition and use of a regime.  One of my first posts describes a pair of strategies – a simple mean reverter and a simple trend follower.  These two strategies are about as dumb as you please.  And yet, their behavior is pretty interesting apart the observations I’d published at the time regarding the distribution of their returns.  During the trading day, I’ve taken to running those two, old, dumb strategies across a variety of markets in simulation mode.  Why?  Because it’s easy, costs me nothing and actually is pretty interesting in that the strategies really do cluster in the way Mlnarik et al describe, effectively partitioning the market into visible regimes.

a simple example: identifying regimes with strategies

In order to illustrate this, I created a desktop within stratbox onto which I ran these two qualitative strategies parameterized differently and across six different markets.  In total, I have 48 strategies running with 24 of the mean reverting (MR) variety and 24 of the trend following (TF).  In figure 2 and figure 3, they are sorted by qualitative strategy – all of the MR strategies followed by all of the TF strats.   In figure 4, they’re instead sorted by instrument at the end of the day after all of the strats had been rendered quiescent for the night by their schedulers.  You can click to see a bigger version of these screenshots.  On each, the left side of the screen shows simple charts of the relevant markets while the right contains a table in which each line item is a running strategy.  The first is at the beginning of the day. the second is shortly before going silent at 4pm and the third is shortly after 4pm.

Figure 2: variously parameterized MR & TF strats run in realtime+simulation mode. Sorted with the MR strats first. (click)

Figure 3: later the same day

Figure 4: at EOD, now sorted by instrument

You’ll note that at all times throughout the day, the MR and TF strategies effectively mirrored each other regardless of market.  Although I’m not going to illustrate it, I’m going to just go right ahead and assert that this is almost always true  based on what I’ve seen.  And this is at the heart of the paper’s authors’ insight regarding regimes – they can be effectively identified by examining the performance of even simple but well-defined strategies.

This suggests a simple means of implementing a similar design to that they described right on top of this example I’ve put together.

modeling meta-strategies

We need one more element to manage all of this – a meta-strategy – that is, a strategy for managing other strategies.  We support this idea in stratbox with what we call a controller strat.  The cannonical example of a controller strat is our implementation of the (Don’t) Panic button (the bright red button with an ! in the upper right hand corner) which will get all strategies out of the market with varying degrees of urgency and finality.  This is implemented as a controller strat which injects a stratpart into all running strategies.  The injected stratpart first deactivates its peers and then cancels outstanding orders and then gets out of the market with whatever urgency the user has required.

allocating to portfolios of strategies

In my example above, I have 48 strategies running in simulation across 6 futures markets.  Let’s imagine that at any moment in time, I will always have at least one strategy of each type (MR and TF) running for each market. Thus, I’ll always have 12 live strategies running against real money accounts and 36 candidates running against a simulator.  Effectively, we’ve just described a portfolio of 12 elements where each element is itself a strategy.

It’s not difficult to imagine implementing a controller strat which on a regular schedule will simply adjust the parametrization of the running strategies and their allocation of cash based on the performance of the strategies running in simulation.

It’s not difficult to imagine, but implementing it inevitably poses a variety of challenges not least of which is finding the time!  For now, we’re still finding new and interesting ways to express strategies which deal with portfolios of exchange-traded instruments.  But I think that this is just a point on our technology’s maturation curve which will ultimately lead to the regular development of strategies which operate on portfolios of strategies themselves…

Anyway, it’s nice to see a bit of academic work formalizing what I expect is a natural step in the evolution of trading algorithms.

There’s been a lot of attention focused on  trading battlebots recently.  It’s important to keep in mind that this is part of a long-standing, broad and arguably inexorable trend that is now spreading rapidly away from its successful base in industrial manufacturing to every other conceivable field from scheduling and logistics, to CAD and on to more aggressive pursuits like trading and battlefield operations.  Perhaps looking at the state of the art in related fields can inform us about the direction of our algo bots.

This article in Foreign Policy illustrates an area where automation is making great strides into historically human undertakings.  The use of so-called drone aircraft for recon and tactical missile strikes has reached a remarkable milestone: this year, the US Air Force will train more “pilots” for unmanned aircraft than for real fighters or bombers.  Evidently there’s good reason for this change:

By 2013, software and communications improvements will allow the Air Force’s unmanned-aircraft pilots to simultaneously fly three drones at one time, and four in an emergency. Another factor supporting the likely proliferation of drones such as the Predator, Reaper, and Global Hawk is their low cost compared with new manned aircraft such as the F-35 Joint Strike Fighter.

According to the Government Accountability Office, $24.5 million will purchase a set of four MQ-9 Reaper hunter-killer drones plus a ground station and satellite relay. (See page 117 of this report.) The latest guess of the price for a single F-35 fighter-bomber is $100 million. (See page 93.) This gap in cost led Defense Secretary Robert Gates to demand the cancellation of the manned F-22 Raptor program in order to fund the purchase of more drones for service in Afghanistan and Iraq.

So, for the price of one F-35, I can get a 16-strong phalanx of drones and the required ground-based support to deploy them.  For the price of one F-22, I could get over 20.  Sounds like a deal.  But, they’re not really interchangeable parts as the F-35 or Raptor and other fighter-bombers can overcome “air defense threats” while the drones aren’t quite there yet.  Thus, they’re really only good against opponents that don’t have an air force or anti-aircraft capabilities.  In schoolyard parlance: they’re only good for defenseless opposition.  But we’re buying lots of them, so we must have found nails for our hammer.

While developmentally they are likely in different places, there’s probably an analogy to be made between the current state of development between military drones and their algo trading equivalents.

If today’s algos are like today’s drones in that they’re relatively dumb and can’t overcome defensive threats, what might tomorrow’s algos look like?  Who will they be able to target that they can’t currently?

Steal a little and they call you “thief”… Steal a lot and they call you “King” – Bob Dylan

I try to avoid the news during the trading day.  I never trade manually and as I’ve mentioned before, I’ve never yet had much success trading the news and none of our models presently use news feeds for decision-making.  So I really try to avoid keeping excessively abreast of the news as it’s just a distraction from real work.

would-be king

That said, this morning I noted a pretty good jump in our pre-market p&l and wanted to see what splendid news had prompted the spike.  So I scanned some headlines.

On Bloomberg I saw:

•U.S. Payroll Cuts Slow, Jobless Rate Unexpectedly Falls as Recession Eases

•AIG Reports First Profit in Seven Quarters After Investment Losses Shrink

•Dollar Advances as U.S. Employers Cut Fewer Jobs Than Economists Estimated

Unemployment down?  AIG profitable?  Dollar rumbling to strength?  Splendid, splendid and splendid.

I also saw the bit about Hank Greenberg paying the SEC $15M as he “thought it would be good to get it behind us.”  Indeed, good thinking.

And anyone looking at the news this week knows that regulators are likely going to put the kabosh on flash orders and that Goldman trades profitably.  (And they improve!)

my kind of fat tail

Within a few moments, I’d identified the likely causes for our spike in p&l: optimism.  Thank you, optimists!

My immediate question was answered, but the news about Mr Greenberg stuck with me for a moment.  After all, one of the positions we made some money on this morning was AIG, a firm perhaps as central to our current financial crisis as any could possibly be.  A *personal* $15M fine over accounting um irregularities.  Ouch, that’s gotta hurt.  Could you imagine paying a $15M fine?  “Without admitting wrong-doing.”  Remarkable.  I’d own-up to a fair amount of wrong-doing if it meant saving me $15M.  Of course, if the average American got hit by a $15M fine they’d be bankrupt.  What’s it do to Mr Greenberg?

Last I could find him on the Forbes list of “World’s Richest People” was 2005 when he was ranked #79, weighing-in with a fortune of ~$3.2B.  Of course, it’s been a tough few years and he’s probably taken it on the chin.  How much?  No idea, but for the sake of argument, let’s say he’s down to a lightened but still respectable fortune of ~$1B.

The average American instead has a “fortune” of somewhere between $0 and $1M, according to the FED’s most recent Survey of Consumer Finances.

Hank wins

For the sake of our back-of-the-envelope calculation, we’ll say that the average American family is worth $1M, though this is surely fanciful.  This means that Hank is only about 1000 times richer than our fanciful American family who’s equivalent fine would be $15K.

Now, $15K will get you a beer and more, so I’ll never be the one to spit on $15K, but for a $1M family it is something that will look a lot like an ordinary item on their taxes.  Not fun, but a lot of people would rather part with $15K than pass a kidney stone.  Actually, when you pass a kidney stone in the US you also typically have to cough up something like $15K, but that’s another story altogether..

It’s just not the kind of fine you’d think would be very effective at getting people to not ruin the global economy so they can pocket a few quick billions, thank you very much.  If I know I stand to make billions at the cost of being fined in the millions, I just might spin the wheel.  Not much deterrent there.

So, Hank really didn’t do so badly as all that.

Now that that’s out of the way, regulators can address important items like flash orders, short sellers and high-frequency trading.  Meanwhile, an almost manufactured-looking populist backlash suggests darkly that Goldman’s trading gains are ill-gotten.

/pitchfork-mode-on

I don’t personally believe that Goldman is front-running their clients, but I’d prefer to avoid the possibility entirely by disallowing firms with proprietary trading operations from handling client accounts.  In programming, the best way to deal with concurrency issues is to prevent them from happening structurally by employing producer-consumer queues and other such mechanisms.  In just the same way, front-running should be structurally prevented by regulation.  Just make it effectively impossible.  But this eminently sensible model was over-written in the twilight of the Clinton I years by some of the super heroes who brought us all the marvelous financial innovations we’ve enjoyed over the subsequent decade.

more kings, yay!

Back to Goldman.  While I don’t think they’re front-running, everyone ethical/sensible will agree that if they were, it would be bad.  How bad?  Well, 252 trading days times $100M of ill-gotten gains yields something like $25B annually.  Big money by my proletarian standards, but let’s say that Mr Barofsky overcooked his stew by an order of magnitude and this bailout will only cost us ~$2.5T.  That would make the worst case scenario for HFT hijinks something like 1% of our crisis tab.

Worth pursuing, if there’s something there, but probably not the most bang for regulatory buck out there.

What’s an ambitious regulator to do?  Well, faced with symptoms like those evident in today’s US markets, the IMF would recommend breaking-up the finance oligarchy outright:

The crash has laid bare many unpleasant truths about the United States. One of the most alarming, says a former chief economist of the International Monetary Fund, is that the finance industry has effectively captured our government—a state of affairs that more typically describes emerging markets, and is at the center of many emerging-market crises. If the IMF’s staff could speak freely about the U.S., it would tell us what it tells all countries in this situation: recovery will fail unless we break the financial oligarchy that is blocking essential reform.

As Americans we are participants to a historic transfer of wealth from the US taxpayer to [?].  We don’t know how much, but we know it’s measured in Trillions.  And we really can’t say where it has gone.  Remarkable circumstance.

Some are saying that modern finance is so complex that we can’t hope to understand what just happened.  But I don’t think we need people with sophisticated intuitions about volatility smiles or nuanced appreciation for the creative engineering of structured products to solve these mysteries.

We just need garden-variety, honest and empowered investigators.

The “Untouchables” and other real-life gang-busters come to mind.  The techniques they employed would be effective for identifying and bringing down the people who’ve contributed to this epic transfer of wealth.

Just as they identified the “associates” of their prime suspects and then pieced together their activities by following money trails, our own regulators need to do some straightforward investigation of the people involved in this mysterious money hole.  Some will be innocent, some will have committed crimes.

Start with someone central to the fracas.  AIG has a few candidates with both Greenberg and Cassano likely suitable, but there are any number of lists out there implicating people with our “credit crisis.”  Pick one.  Look closely at his (or her) taxes, offshore holdings, business interests.  Identify 50 of their top social & business contacts.  Again, start with taxes.  Do this for a few cycles and you’ll have unwittingly collected the cream and dregs of international finance.  Investigate them.  Look at their taxes.  Follow the money.

identify the bad guys, follow the money, prosecute

People who committed crimes go to jail.  People who didn’t don’t.  Some new laws will need to be made and probably others will need to be taken off the books.  Minimal math required.

But what few honest and competent regulators we have are not empowered to do anything except kvetch pleasantly on the John Stewart show and perhaps muck about with market microstructure.  They are certainly not empowered to take on the beneficiaries of the crisis we’ll spend a generation paying for.  It’s even difficult to imagine how a regulator could be empowered to take down the architects of this most ambitious of capers.  The arch-villains of Mario Puzo or David Chase’s most feverish moments never approached the sophisticated operations unfolding today.

Focusing on market microstructure in an environment like this is truly missing the forest for the leaves.