The example I’ve chosen is, I think, very nice because it’s a portfolio-oriented strategy, which is pretty much the only kind I care to explore; it’s also based around the concept of pairs trading, which is something which most can easily relate to; and finally, it’s already public domain and yet almost certainly has some juice in it for those who care to understand it and extend it intelligently.

The example comes from the blog of a company, Palantir which does (something like?) analytical / decision support software for both finance and intelligence-gathering services (quants and spooks – spooky quants?).  The specific example is here and is described thusly:

In this study we explore a trading strategy that isolates pairs of instruments within a sector that are highly correlated. We enter a trade if the price paths of these instruments diverge, going long one instrument and short the other, with the assumption that their price paths will converge.

We construct our strategy in four parts: (1) isolate the target set of tradable instruments, (2) choose a rule for finding correlated pairs within that set, (3) pick criteria for entering trades, and (4) test the trading strategy.

…

For this study we want to isolate pairs of instruments whose correlation is above a certain threshold. At the beginning of each quarter, we restrict our target stocks to the top 5 in the sector by 252-day correlation to the S&P 500. Then our Strategy uses a metric that, for a given group of stocks, outputs a list of pairs with a correlation above our target threshold of .60 over the same time period.

…

Across every day and every pair in our trading list, we check the 21-day z-score (number of standard deviations from the mean) of the difference between the two series. We classify the stocks as diverging when the absolute value of the z-score is between 1.5 and 3.0. When this condition is met we short the stock that is rising and long the stock that is falling, sizing our position relative to the z-score and the number of pairs currently trading.

~(sectors) ; reframing the strategy

While I’ll implement a strategy broadly based on their description, I will use some poetic license to modify it in ways that seem to me sensible.  First, I have good experience writing equity strategies which deal with sectors and have pretty uniformly found that sectors, as defined by S&P or whomever, are more trouble than they’re worth.  Instead, looking at correlation matrices and determining on that basis who should be paired, is much more fruitful in my experience.  Tr8der has some very nice posts illustrating (very prettily!) a similar point here and here.  (Actually, his point is a bit broader and represents an area of possibly promising development of this strategy which I won’t pursue here.)

So, we’re going to chunk the idea of explicitly using sectors.  Instead, we’ll simply construct, on a monthly basis (Palantir did this quarterly) a TrailDays correlation matrix from which we’ll extract the set of pairs which have a correlation greater than MinCorrelation.  On a daily basis, we will calculate the ZScoreDays z-scores of each of these pairs and sort them by absolute value, tossing those whose values are less than MinZScore or exceed MaxZScore.  Since I don’t like the idea of having an open-ended number of pairs/positions in my portfolio, I will constrain the portfolio to a maximum of N names and will Allocate $1M to the strategy.   Finally, we will look at the effect of trading an EvenlyWeighted portfolio or a z-score weighted portfolio (as Palantir had done).

The initial ‘universe’ of equities across which we will calculate our correlations will be filtered from my database of daily equity data going back to 2005 where the closing price is over $1, the average daily dollar-volume is over $50M and the data is relatively clean (few if any gaps).  This yields an initial universe of ~600 equities and ~180K pairs.

Assuming that we keep N above 20 or so and we keep our allocation constant at $1M, and assume execution at the close price, we should expect our results to be reasonably believable as it’s unlikely that the size we’re trading in would distort the market.  Scaling any strategy presents its own set of challenges which we’re not going to go into for this example, but an obvious direction for scaling the strategy would be to simply run it over intraday data and scale into and out of positions as conditions permit.

Ok, so that’s what our strategy is going to look like.  Now, how do we “make it so”?

structure of a stratpart

Within Stratbox, strategies are represented as composites made-up of ‘stratparts’ each of which has its own metadata descriptor containing parameters.  Stratparts can be composed together within a strategy to enhance or modify the behavior of a strategy.  Thus, one can have a stratpart which implements the above logic paired with another stratpart which will e.g., modify the portfolio to impose beta neutrality, damp volatility or some other form of reporting or risk management.  Given that this example is already complexish, we’ll just stick to the one stratpart.

A stratpart has a few key elements.  One is the metadata descriptor which allows the strategy container (stratbox) to interact with it for back-testing, optimization or forward-walking purposes, or to allow a ‘trader’ to interact with it dynamically during run-time (the so-called ‘gray box’).  Beyond that, it has an essentially event-driven design.  The method quote() is called when relevant (i.e., subscribed) market data arrives.  The method execution() is called when executions for orders that originated with this strategy are received.  And the method strategyEvent() is invoked on a variety of ‘interesting’ events that the strategy implementer might care about.  None of these methods are required, so in its simplest form, a stratpart is a very simple piece of code to write.

services in a stratpart

Once one has complied with the minimal requirements for implementing a stratpart, one is rewarded with a variety of useful services: subscription-based market data feeds as well as a fast snapshot db and an in-memory historical db for quick correlation analyses cover market data needs.  Position management is handled at both the strategy level and at the exchange level (if you’re trading across multiple exchanges).  There is more, including performance analysis (e.g., sharpe, and its various analytic friends) and hierarchical cash allocation within and across strategies, but we won’t look at these in this example.

implementing metadata

So, the first thing we need to do is define our metadata.  This is done through the use of a conventionally-named static method Descriptor().

public static Descriptor Descriptor() {

 ArrayList<Param> params = new ArrayList<Param>();

 String cd = "Unleveraged capital to apply";
 params.add(new Param(InitialBalance, cd, false, 1000000.0));

 String n = "# of instruments to hold in portfolio";
 params.add(new Param(N, n, true, 20));

 String tc = "Trailing days over which to calculate correlations";
 params.add(new Param(TrailDays, tc, true, 252));

 String mc = "the minimum correlation of pairs to consider";
 params.add(new Param(MinCorrelation, mc, true, .75));

 String zd = "trailing days over which to calc z-score";
 params.add(new Param(ZScoreDays, zd, true, 21));

 String mizs = "minimum z-score";
 params.add(new Param(MinZScore, mizs, true, 1.5));

 String mazs = "maximum z-score";
 params.add(new Param(MaxZScore, mazs, true, 3.0));

 String ew = "evenly-weight portfolio? (or weight by z-score)";
 params.add(new Param(EvenlyWeight, ew, true, true));

 Descriptor _desc = new Descriptor
   ("puppetmaster.strats.Pairs", _Desc, params);

 return _desc;
}

public Pairs(Strategy strat, Descriptor d) { super(strat, d); }

Not the most interesting chunk of code, but pretty straightforward and its inclusion opens up a lot of functionality within the environment as each of these parameters can now be systematically or manually modified in-flight.  The third parameter to the Param constructor, a boolean, indicates if we want to allow the system to optimize this parameter.  Thus, the allocation/initialBalance parameter isn’t optimizable as it wouldn’t make any sense to do so.  Even so, we can (ourselves or through an allocation algorithm) modify this value to force the strategy to lighten or extend its financial footprint.  The others are fair game for optimization.

The constructor is also required – with that exact signature.  This is pretty much it for the ‘boilerplate’ code required.

‘installing’ the stratpart

Now that we’ve met the minimum requirements for a stratpart, we can install it into stratbox.  This requires adding one line to your personal stratparts.xml config file.  Like so:

<?xml version='1.0'?>
<stratParts>
 <descriptor class='puppetmaster.model.strategy.FwdWalker'/>
 <descriptor class='puppetmaster.model.strategy.StrategyPortfolio'/>
 <descriptor class='puppetmaster.strats.Pairs'/>
 <descriptor class='puppetmaster.strats.AMBO'/>
 <descriptor class='puppetmaster.strats.meta.AdjMeanReverter'/>
 <descriptor class='puppetmaster.strats.meta.AdjTrendFollower'/>
 <descriptor class='puppetmaster.strats.meta.Binary'/>

Once we’ve informed stratbox of the new stratpart we’ve written, we can view it within our library of stratparts within the strategy wizard in stratbox:

Our Pairs strategy in the Strategy Wizard's stratparts library

And all of the parameters we defined within our descriptor show up on the next page where we can parameterize or optimize the strategy:

Configuring the Pairs stratpart within the strategy wizard

This means that the stratpart can be combined with other stratparts to create new strategies as I’ve described  and that it can partake in the rich set of functionality within the stratbox gui.  Including actually trading.  But first, we must make the strategy actually do something…

implementing event handling

To that end, we must implement the event handling methods I’d described previously.  Since we’re not going to do any detailed execution handling in this strategy, we’ll only implement the quote() method and a subset of the events sent to the strategyEvent() method. In particular, we’ll listen to activation events (so we can initialize cleanly), changes to our descriptor and changes to our positions.

/** listen for select strategyEvents */
public void strategyEvent(StrategyEvent event) {
  super.strategyEvent(event);
  switch (event.type) {
    case Activated:            _init();                 break;
    case DescriptorChanged:    _readDesc();             break;
    case PositionChanged:      _posnChange(event);      break;
  }
}
<pre>

For quotes, we’ll just listen to the Beginning of Day (BOD) event.  If we wanted to trade intraday, we’d have to listen to tick events, but we’ll leave that for another day…

From a high-level, our code is pretty simple.  We determine the universe of pairs we might trade, we select from among them, we determine what our new portfolio should look like and then we trade to transition from our current state to our desired state.  That’s it.

/** mkt data goes here, but we only listen for the BOD
 * ("Beginning of day") event */
public void quote(Quote q) {
  if (q.type() != Quote.Type.BOD) return; // daily strat...

  try {
    List<_Pair> pairs = _getPairs();
    List<_Pair> sel = _selectPairs(pairs);
    List<Position> dp = (List<Position>)_desiredPfolio(pairs,sel);
    _Log.debug("PORTFOLIO: "+dp);
    _trade(dp);
  } catch(Exception e) { _Log.error(e.getMessage(),e); }
}

the messy details

Digging into the details gets more complex, but not horrifically so, as stratbox already has basic analytics like correlation matrices, z-scores, volatility built into it.  Likewise with basic portfolio analytics and transformations. Thus, our code to trade a (pairs-based) long-short portfolio with an arbitrary number of elements is surprisingly simple:

/** given a desired portfolio, trade to make it so */
void _trade(List<? extends Position> desiredFolio) {
  PositionRecord[] currFolio = posns();
  // Given our current portfolio (posns) and our desired state (dfolio),
  //  we generate the set of orders which will transform from the former
  //  to the latter
  //
  List<Order> orders = PortfolioComposer.TransformPortfolio
    (currFolio, desiredFolio, _orderF(), _strat, _strat.account());

  // place the orders for execution
  for (Order order : orders) {
    try {
      order.setType(Order.Type.MOC); // we trade at the close only
      _subscribe(order.contract());
      _execP().placeOrder(order);
    } catch (Exception e) { _Log.error(e.getMessage(), e); }
  }
}

As promised, I provide the full source listing below for those interested, but won’t over burden the discussion with a complete description of every little piece of it.

a telling omission and some results

Early in my career, working as a software engineer in a wall st front office technology department, I was befriended by a manager who told me he had once been a trader.  I asked him why he wasn’t a trader anymore.  He quipped:

I was half of a talented trader.  I knew, in my bones, when to get into a trade.  Sadly, I was never really sure when to get out…

It seems that the author of the Palantir example suffers a similar characteristic to my old friend (or is just being understandably cagey) – she specifies an entry condition, but not an exit!  Since she doesn’t impose any costs for trading, this has no big side effects, but in stratbox we do impose (pretty onerous -> realistic!) costs for trading.

In any case, since this is just a simple example I’ve maintained the ‘no exit’ policy and simply exit a trade once its pair goes out of the acceptable ranges for either correlation or z-score.  This has a bad effect on the strategy as you end up churning a lot of trades at the high end of the z-score range.  That is, if we set our upper boundary for z-score to be 3, then we might enter and exit a position multiple times as the z-score oscillates between, say, 2.9 and 3.1.  A more careful implementation could minimize this effect.

Another difference imposed by our more realistic simulation as opposed to the original example is that I’m restricting the portfolio size explicitly and explicitly limiting myself to trading a given contract within one pair as opposed to allowing a contract to trade across multiple pairs (invoking higher costs but allowing positions to offset and thus providing a neat little optimization – assuming trading costs nothing!).

All the same, the strategy clearly has some positive characteristics and when parameterized to take advantage of its naive exit strategy (by making the z-score low-end quite low, e.g., 0.5) takes on the character of a hedged trend-following strategy.  How so?  Well, while we are getting churned around the high-end (entry) z-scores, our profitable exits only happen once the z-scores have diminished (the prices have converged) considerably.  So we take more, smaller losses while our winners are held a bit longer.

some Pairs results in '10

Adding to my portfolio size improves things generally (and also increases the strategy’s scalability) while increasing the MinCorrelation parameter seems to hurt performance (not sure why, honestly – perhaps because if they’re very highly correlated and already diverging so strongly it may mean they’re actually diverging fundamentally). Returning to the topic of costs, each of these strategies incurred costs of over $5K for the period in question. And returning to the parameter which governs the weighting strategy employed, although I haven’t illustrated it here (as I haven’t looked at it very carefully), my initial finding is that it changed very little. 

There’s a great many areas where this strategy could be improved – that’s the fun, after all! – but hopefully even in its primitive form this example brings to light both an interesting baseline portfolio pairs strategy, while answering the question of how one goes about concretely implementing it within stratbox.

The entire strategy weighs-in at ~400 LOC including comments, boilerplate and individualized imports.  The source is here: Pairs.java.

One of the recurring technology themes in these pages has been the ongoing and dramatic move from single to multi-core systems and the need to seriously increase the parallelism in our software designs. For me, one of the seminal, large-grained design patterns was the SEDA Architecture. For years, this informed my systems’ designs and formed a conceptual backbone for development. That said, I’ve been broadly aware for some time that SEDA’s golden age has (incredibly!) already passed us by, but haven’t identified what might replace it as a reference point for my design efforts.

Before considering tools, languages or patterns that might help, we need to reflect on the problem(s) we’re trying to solve. The problems inside an EMS look to me, after years of development, a lot like network routing problems. Indeed, my current view is that this (not just concurrency as I’d suggested at the time) is why the unfortunate Aleynikov & co. at GS were using erlang.

Why network routing? Think about the load on an EMS. The main issue is that you’re getting many thousands of teeny little messages per second and only a relatively small number of them matter to only a relatively small subset of ‘agents’ within the system. Reducing latency is all about making sure the time you spend on each message is minimized, and that the agents who are interested in a particular message needn’t wait for each other to do whatever they care to do based on the message. So, really you’re trying to route each message through your system with as few ‘hops’ possible and as much parallelism as you can muster under the (radically!) new assumption that you may have hundreds or thousands of cores available to you during the lifetime of the design.

I spent some time thinking (hoping) that languages might help furnish an answer. Perhaps a move to a functional language like erlang, ocaml or scala might help furnish at least a partial answer. But erlang is slow and peculiar, ocaml doesn’t support intra-process concurrency and scala looks like a bloated language on a bloated platform (jvm+java class library). And none of them seem to have achieved anything near the critical mass which is so crucial for the development of usable libraries and the availability of skilled developers with long experience in the technology.  Naturally, reasonable people will disagree about such things, but this is my view (today). Java is ok (and certainly sells servers), but it’s not obvious how it’s going to help me offload my work onto a GPU anytime soon (and jni is both painful and slow) and I’ve never been able to get comfortable with just how damn big VMs get.  Image size isn’t free and if we’re looking to go deep into the sub-millisecond response time, while running thousands of concurrent strategies, it seems we need to disintermediate the VMs and interpreters of the world. If they’re really necessary, they can be happily used for the analysis process (as I currently use R), or they can be lit-up and bridged from some lower-level language for batch-like services.

The good people at Intel have been thinking about this problem for a while as have many other seriously over-educated people. One of the (sensible sounding) conclusions reached as people look for ways to solve problems similar to my own, is that in such systems we should keep messages waiting as little as possible – ideally, not at all(!). This can be a problem in SEDA-like architectures which are basically made-up of (non-blocking, asynchronous) i/o processes linked to (blocking) queues linking pools of workers. Blocking queues can pile up and cause all sorts of problems like priority inversion and other such enigmatically named nasties. Lock-free queues and other data structures, algos and techniques promise some ways around this and I’ve been spending time looking into how they might be employed to address my issues.

Before I’m besieged by throngs of angry erlang/ocaml/scala/java developers, allow me one last observation on the topic.  (Peeved python and ruby users may rant away – vous m’amusez  ;^)

Why might a lock free algorithm be better than an equivalent, hardware-based locking implementation?  The answer isn’t obvious.  If locking is implemented in hardware as is typical (eg, with a compare-and-swap (CAS) instruction), then its explicit cost is measurable in (few) nanoseconds.  Hardware is fast.  The issue isn’t the speed of execution of the underlying primitives so much as it’s a consequence of the side effects of these operations at a very low level.  For real performance, cache coherence is King.  See here for an accessible discussion by IBM’s Paul McKenney and here for some remarkable examples from Igor Ostrovsky.  This indicates that if you want the highest possible performance, you need to be aware of what is happening ‘in the metal.’  So we need to use a system-level language and erlang, java & friends lose their candidacy in spite of any fantastic benefits they might offer.

Given that even the DoD has mostly given up on ADA means that we’re left with C/C++.

Ok, so language doesn’t seem to resolve much for us. (Indeed, it was mostly hopeful thinking on my part – design is mostly language agnostic and hardware is hardware…)

Apart from Intel’s own Threading Building Blocks (TBB) framework, there are a variety of toolkits available for exploiting lock free parallelism. Perhaps the newest and least known is called FastFlow, which is a C++ template library that provides a variety of facilities for writing efficient lock-free network models. It also claims to be faster than TBB, Cilk and OpenMP while holding out the promise of one day becoming CUDA- (or more generally, GPU-) aware which would be an incredible win. Finally, it is very small – the current version (not including tests and examples), weighs in at ~5K lines of (mostly) C++ templates.  Thus, it seems to me particularly well-suited for some experimentation to assess the fit of these techniques in this space and the level of difficulty of doing so.

In the remainder of this post, I’ll briefly describe the FF design and then illustrate a sample C++ program which uses FastFlow to ‘architecturally prototype’ a feed handler interacting with strategies inside an EMS / strategy container.

FastFlow architecture

FastFlow’s (FF) goals seem quite ambitious, but the design is layered to allow users to work at a level that makes sense for the problems they are trying to solve, so they needn’t worry themselves (or even understand necessarily) layers below where they need to work.  At bottom is a thin layer of hardware-specific code supporting, one level up, single-producer-single-consumer (SPSC) lock-free queues and a threading model for their efficient interaction.  Above this are defined objects for generalizing from simple to arbitrary network types.  Here you see the definition of a composable Node which serves as the key abstraction for general-purpose streaming networks.  Above this are defined a variety of skeleton templates for building commonly useful graph types.  Because they are Nodes themselves, they can be nested and coupled to any depth.

Consider a simple pipeline:

a simple FastFlow pipeline

A farm is slightly more complex and acts something like a load balancing threadpool.  The Emitter pushes ‘tasks’ of arbitrary type to an arbitrary number of workers who then coalesce back into an (optional) Collector.  A simple Farm makes no promises about the ordering of tasks, but (I believe) FF provides mechanisms for ensuring their ordering at the collector.

a FastFlow Farm

Finally, because of the composable nature of Nodes, all of these graph types can be combined arbitrarily.  Here you see a Farm of pipelines with an ‘accelerator’ (which, if I understand correctly is just an integrated thread for pushing tasks into the network) and a feedback channel.

composition of FastFlow subnets

For a more complete (and probably accurate!) description of FastFlow, I encourage you to visit their site and tutorial.  The documentation remains limited, but there’s a good and growing selection of examples to examine and all of the source is available for inspection.

a simple example

Given my use-case, my first interest is in seeing how FastFlow might work in the context of asynchronous, non-blocking network handling.  To this end, I’ve built a simple Farm as depicted above in which the Emitter uses select() to read messages off the network which it parses and pushes into the farm for ‘processing’ which is currently NOP-ed.  The messages/tasks are then coalesced into a collector which does nothing more than free memory allocated in the Emitter.  Within the Farm, each of the nodes inhabits its own thread and outside of the farm I’ve implemented a very simple ‘client’ which connects to the Emitter’s socket and sends delimited Integral ‘messages.’  Ordinarily this would appear in its own process, but for the sake of only requiring 1 file, I’ve mashed them all together and the client lives in its own thread.  That’s it.

It turns out that the FastFlow part of this is, by far, the simplest.  Since the FF skeletons are implemented as templates, extending them couldn’t be much easier.  Here’s the definition of the worker and collector components.

// typical worker: does little ;^p
class Worker: public ff_node {
public:
  void * svc(void * task) {
    return task;
  }
};

// collector that just frees the malloc'ed memory
class freeing_collector: public ff_node {
public:
  void * svc(void * task) {
    int * t = (int *)task;
    if (*t == -1) return NULL;
    else free(task);
    return GO_ON;
  }
};

They are both nodes and their only requirement is to implement the svc method which is called by the FF framework when they are meant to service a task.  They can optionally implement svc_init and svc_end methods which provide an opportunity for initialization and cleanup.  The socket-reading emitter makes use of the svc_init method to initially setup the socket.  Later, as it repeatedly reads from the socket into a buffer, it cracks the incoming stream and then calls the ff_send_out() method to push those messages out to the downstream workers.  The whole example is about 350 lines of code and nearly 200 lines are consumed by the Emitter.  Almost all of this bulk is network-related, so I won’t illustrate here.  But highlighted here you can see the main FF-specific part of the Emitter:

 // handle new data coming in
 void new_data(int nbytes, const char* buf) {
   static char remainder[kMaxMsgSz];
   int *t;
   int j = 0;
   int i = 0;
   char msgbuf[kMaxMsgSz];
   for (; i < nbytes; i++) {
     if ( kDelimiter == buf[i] ) {
       int rlen = (j==0) ? strlen(remainder) : 0; // remainder?
       int len = i-j;
       if (rlen>0) strncpy(msgbuf,remainder,rlen);
       int start = (rlen==0) ? 0 : rlen;
       strncpy(&(msgbuf[start]),&(buf[j]),len);
       msgbuf[len+rlen] = '\0';
       t = (int *)malloc(sizeof(int));
       *t = atoi(msgbuf);
       j = i;
       ff_send_out(t);  // push message into farm
     }
   } // for

   int k = j+1;
   if (k < i) { // save 'remainder'
     strncpy(remainder,&(buf[k]),(i-k));
   } else remainder[0] = '\0';
 }

Finally, all of the pieces are assembled and launched in the main().

//  We construct a 'server' which is a fastflow emitter and which reads
//    integral 'msgs' from a client over a non-blocking socket.  Msgs
//    are parsed and fed into a fastflow farm for further handling.
//    The client is placed within its own thread - this is only so the
//    whole example can be placed in one file.
//
int main(int argc, char * argv[]) {
 // use: argv[0] <#msgs=1024> <#port=9999>

 int msgs = (argc > 1) ? atoi(argv[1]) : 1024;
 int port = (argc > 2) ? atoi(argv[2]) : 9999;

 int nworkers = 5;               // how many workers will recv msgs

 printf("main: sending #%d msgs to port <%d>\n",msgs,port);

 select_reader sr(port);         // we create 'server'
 freeing_collector fc;           // and a freeing collector
 ff_farm<> farm;                 // and a farm for it to live in
 farm.add_emitter(&sr);          // add both to the farm
 farm.add_collector(&fc);

 std::vector<ff_node *> workers; // build a collection of workers
 for(int i =0; i < nworkers; i++) { workers.push_back(new Worker); }

 farm.add_workers(workers);      // add all workers to the farm

 farm.run();                     // launch the farm
 printf("main: started farm.\n");

 sr.wait_til_ready();            // don't create client til srvr ready

 // create client which will send msgs via socket
 //
 printf("main: creating client...\n");
 client_thread* client = new client_thread(msgs,port);
 if (client != NULL) client->join();

 farm.wait();                    // wait for farm to be done its workload

 // emit some stats
#if defined(TRACE_FASTFLOW)
 std::cout << "DONE, time= " << farm.ffTime() << " (ms)\n";
 farm.ffStats(std::cout);
#endif

 return 0;
}

The whole example can be found here.

Writing this example was a bit of a ‘tale of two frameworks’ for me as I found the FastFlow part to be really easy and powerful.  On the other hand, I wanted to try out the boost::asio framework for non-blocking networking i/o (primarily in the hope it might cut down the code size a bit) and while implementing the trivial (blocking) client was, well, trivial, implementing a non-blocking server with boost::asio was a serious pain and I ultimately decided that Linus just might be on to something wrt to boost and scrapped that part of the effort (but kept the boost client).

Nothing about this example illuminates the potential performance advantages of a lock free approach, but this exercise has certainly convinced me that the FF authors have managed to combine very low-level performance considerations within a package which can be easily used from a high-level perspective.  Most problems will most likely be better served by higher-level programming languages, but some problems remain intractably bound to the metal and I suspect that writing ultra-low-latency trading systems will remain in the latter class for some time to come.

In the example below, we look at another ‘meta’ strategy named StrategyPortfolio which maintains a dynamic portfolio – P – of strategies which it will select from a set of strategies – S – running concurrently in simulation.  The constituents of P as well as their cash allocations and parameterizations will be rebalanced/adjusted regularly after an initial ‘out-of-sample’ period during which only the S strategies are run.

Apart education, the intention of this strategy, as I’d originally suggested here, is to ‘back-into’ a regime-switching strategy without attempting to directly quantify the regimes explicitly.

This has proved to be even more interesting than I’d expected, not so much because it performs particularly well (though it’s promising), but because of all of the things it has taught us.  In particular, the transitions are a killer and there are properties of strategies which (dis-)qualify them from being effective in such a scheme…

bad news good news

As before, I’m not going to take the time to write-up my results in any formal manner but will again rely upon a quick screencast of the software running.  The good news is that I’m figuring out how to edit these things, so it’s mercifully shorter than earlier screencasts…

Please click here to see the screencast.

I hope you find it interesting and will look forward to any comments or suggestions you might make.

photo by Simon Rogerson

Some time back, I’d written about NVidia’s CUDA noting that it looked ideal for many asset-pricing and monte-carlo type problems in finance.  At the time, I was hopeful that it would be quickly integrated into existing open source efforts like QuantLib, but adoption has proved slower than I’d hoped, most likely because implementing non-trivial problems on CUDA is, well, even less trivial than doing them without..

LMM on CUDA

Happily, I’ve just seen a promising first step in this direction as Über-quant and C++ artisan Mark Joshi recently announced an open-source project, Kooderive which looks to implement the LIBOR Market Model (LMM)  on top of CUDA.  His announcement on the QuantLib mailing lists reads:

Dear All,

various people have shown interest in the use of CUDA with QuantLib. I
have now made some progress on a CUDA implementation of the LIBOR
market model.

In particular, I now have a path generator for the LMM working which
does 16384 paths for 40 rates, 40 steps, 5 factor model, displaced
diffusion predictor-corrector that takes 0.1 seconds on my Quadro 4600.

The state of the project is code fragments that can be called from
other code. Those who are interested can get the code via
the subversion repository on kooderive.sourceforge.net .  The only
project file is currently for VC9 x64. It also uses thrust and the
CUDA SDK.

The next stage will be writing routines, that use QuantLib for the CPU
stuff and kooderive for the GPU stuff,  to actually price things.

A gentle reminder that I will be giving a course on the LMM and
QuantLib in June in London, and I will include a session on kooderive
if there
is sufficient interest.

I am happy to take code contributions for kooderive. However, I am not
looking for a redesign of the library or contributions which introduce
dependence on other libraries. I am interested in contributions of
separate routines and of optimizations of existing routines that do
not change interfaces.

regards

Mark
–
Pricing exotic interest rate derivatives – The LIBOR Market Model in
QuantLib June 2010, London,
http://www.moneyscience.com/training/index.html

Assoc Prof Mark Joshi
Centre for Actuarial Studies
University of Melbourne
My website is www.markjoshi.com

Last time we saw an example of historical forward-walking of a portfolio-oriented day-trading strategy which utilized daily data.  This time we do something a bit more interesting and correspondingly complex.  Today we’ll look at a real-time forward-walk of a moderate-frequency strategy (trades perhaps a few hundred times in  a day) which looks at the top-of-the-book but doesn’t use market-depth.  The strategy is a simple mean-reverter that we’ve described before though we’ve had to make some small changes to get it to behave in the context we’re looking at now…

meta-strategy protocol

careful ...

In particular, we need something of a protocol for sending commands from the meta-strategy to the ‘target’ strategy.  Stratparts generally have a simple mechanism for this as they are notified of any changes to their descriptors.  What they do upon such notifications is their own business.  In order to achieve our goals, we might need to firm this up a bit, but it turns out this quickly becomes tricky terrain.

Consider a single contract strategy, say, a simple mean-reverter like one we’ve seen before (and which I’ll illustrate in the screencast below).  If we want to treat it as an element within a portfolio of strategies, then we need to be able to modify its allocation.  But changing the allocation to a dynamic strategy is very different from changing the allocation to a single name within a portfolio as the allocation effectively becomes a maximum and the strategy may spend the great majority of its time under-allocated.  But this is manageable if we don’t mind being under-allocated at times.

A harder case is if someone comes along and says, “stop trading gold”, and “start trading silver”.  What about my open positions & orders?  If the answer is to go flat, then the character (of the returns) of the strategy is itself going to be effected by the fact that they are being mutated by a meta-strategy; if the answer isn’t to go flat then you’re really talking about a whole different, multi-contract strategy.  So you have to be careful to ensure that the kinds of mutations you are relying upon aren’t excessively disruptive to the strategy or you’re likely to see performance wildly different from normal for the strategy.  As always, if you put garbage in, you shouldn’t expect cotton candy to come out…

For this example, we can control allocation and two simple numeric params governing the strategy.  I’m just going to avoid the please-trade-a-different-contract-case and aim to address that issue by ultimately placing these strategies into a portfolio (some later day).  For now, we avoid the hard cases but I think the results remain interesting all the same…

In particular, just as we’d seen last time (on a totally different time-frame and with a totally different strategy), the forward-walked strategy appears to perform well relative to its peers…

If interested, please click here to see the screencast/example.

-

Happy Thanksgiving!

'optimization' or 'search'?

We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain.  Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode.  I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.

From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good.  Doesn’t mean they’re actually good.  In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively.  I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do.  They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.

They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations.  If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…

Anyway, I felt that had to be said…

Now, I’ll open the floor to a quick illustration of the initialization process described previously and an example of historical forward-walking.

At this point, I’m vested in the inquiry we’d started with the regime-switching post and will apportion time each week towards implementing something like what we’d described there.  I expect the steps to be something like:

1. historical fwd-walking which I’ll illustrate today
2. real-time fwd-walking, that is, dynamically adjusting a live strategy’s parameters based on a continuously repeated ranking of a concurrent ‘live optimization’ of the same strategy
3. real-time allocation to a portfolio of strategies based on regime-switching where regimes are defined by the performance of a heterogeneous set of strategies we run concurrently (again ‘live optimization’)
4. peace on earth

I may have some of the steps wrong, but am hopeful.  We’ll adjust as needed.

If interested, please click on the below image/link which will hopefully take you to a screencast.  This is my first-ever attempt to put a screencast in the blog, so please let me know if you have technical difficulties or suggestions on how to improve the experience.  Or if it’s just a big waste of time!

In the screencast I give a quick illustration of a simple backtest, a simple parameter optimization and finally a fwd-walk.  I’ll appreciate your feedback.

>click<

poor Jorge wasn't ready...

In this post I’m going to revisit some of the topics discussed in the recent ‘containing a strategy‘ and ‘multi-strategy trading with regimes‘ posts, focusing on the process of assembling a strategy and its context in preparation for its launch into any of a variety of modes.

I recently realized that – from the perspective of a strategy container – the process of walk-forward testing is remarkably similar to the regime-switching model we’d discussed previously.  Up until now, I’ve employed walk-forward testing in an ad-hoc manner by taking an existing strategy and then writing a little driver very much like a unit-test scaffolding which would walk the strategy forward, permuting parameters based on previous performance.  Not a general solution, but straight-forward as I employ the strategy parameter optimizer from stratbox in this kind of a toolkit use-case.

I sat down to write one of these walk-forward scaffolds yesterday and started to think about how I could generalize the solution and roll it into stratbox’s GUI and it occurred to me that I could likely kill two birds with one stone…

walk-forward testing

–

I imagine there are different ideas/implementations of it, but for me walk-forward testing is the case where you are using repeated historical parameter ‘optimizations’ to dynamically modify the parameters on a back-tested strategy.  For example, say we have strategy X with parameters p1 and p2.  Let’s say we want to walk-forward test it from the beginning of the year until now.  We’ll need a look-back period for optimizations – say 20 trading days – and a frequency with which to apply the results of our optimizations – say every 5 trading days.  So, on a weekly basis we’re going to look back at the prior (rolling) month performance and adjust our parameters to match the best performer.  How you define ‘best’ is your business but I’ll use something simple like sharpe here.

Continuing with our example, we’ll say that p1 will be permuted 3 ways with values from { 1, 2, 3 } and p2 will be permuted across 4 values { “a”, “b”, “c”, “abcd” }.  So we have a basis of 12 strategies for optimization and we’ll call these O1…O12.

To perform a walk-forward test on X from 2009/01/01-now, we will begin a parallel back-test on O1..O12 starting 20 trading days before Jan1.  On Jan2 we will look at the prior ~month’s results and pick the strategy who had the highest sharpe ratio and clone him and start testing him “walking forward.”  Each week, we’ll repeat the process except that instead of cloning the winner as we did the first time, we’ll just make sure his params are the same as the new winner’s params.

Repeat this process until ‘now’ and you’re done – you’ve performed walk-forward testing on X.

–

Let’s think about what’s happening here.  There are 12 strategies running from ~2008/12/1 and another strategy running from 2009/1/2; all strategies run until now.  There’s also the activity that “we” have done during this process.  What we are doing is the role of a meta-strategy just as we’d seen in the regime-switching case.  The walk-forward meta-strategy needs to watch a set of ‘optimization’ strategies and permute another target (“X marks the spot”) strategy on a regular basis to match parameters with the prior period’s winner.

differences vs. allocating to strategies with regimes

Let’s imagine that we’re going to take the 13 lovingly-named strategies O1…O12 and X, along with the newly revealed meta-strategy we’ll dub “M”, and put them onto a live environment instead of a historical back-test.  Suddenly we’re pretty close to what we’d described previously.  The biggest difference is just the temporality of the strategy context: historical or real-time.  Another difference is that O1…On (let’s generalize just a bit) are no longer necessarily all the same strategy – they may be completely different. X is also no longer a strategy, but a portfolio of strategies:  a replicated subset of O1…On we’ll call X1..Xm.  “Best” is no longer a unary choice but must be rich enough to allow M to support some kind of an allocation algorithm across the X family.

OK, so there are some significant differences.  But from the perspective of the object model on which all of this can be implemented, the differences may be limited enough to warrant tackling together.

ready to launch

Before looking at the workflow of a strategy launch, let’s consider all of the various modes we support with our design:

• Historical: back-test single strategy
• Historical: parameter-based optimization of strategy (decomposes to many back-tests)
• Historical: walk-forward testing of strategy
• Real-time: SIM – run single strategy with real-time market data and an exchange simulator
• Real-time: LIVE – run single strategy with real-time market data and a live connection to OMS/broker/exchange
• Real-time: LIVE+SIM – run two cloned instances using the same market data, but with one routing orders to live venue(s) and the other to a simulator.  This is mostly useful for the development of a simulator or to evaluate the quality of simulation for a given strategy.
• Real-time: optimization – run many strategies, permuted differently across live data but routing to a simulator.  I don’t know what this is useful for, but I find it entertaining… The screenshots from the regimes post were illustrating this mode.
• Real-time: meta-strategy – our mythic regime-switching allocator will be a nice example of this mode.

-

Consider the following sorta-flowchart illustrating the preparation for launch of a strategy.

click for larger version...

The process has 4 or 5 stages depending on whether we’re being “meta” or not – that is, if we’re using optimization of any sort.  The highlighted parallelograms are the concrete end-result(s) for each of their respective steps.

• Assemble Strategy Context – The basic channels of I/O for a trading strategy are its sources for market data and the exchanges where trades can be executed.  The MktDataSrc and ExecutionPlatform abstractions need to be specified.
• Assemble Strategy – Whatever mode we’re looking to employ will require a strategy either to itself go and trade for us or else to serve as the seed or prototype for our activities.  In this step we define that strategy.  For me, this can be a composite process as a strategy is made up of an arbitrary number of stratparts.  The assembly of a strategy can be done manually with a gui or by loading a strategy from a file or other store.  Once the parts are identified, they can be bound together to form a strategy instance.
• Configure Strategy – Once assembled, the strategy needs to be configured.  This may just mean we need to parameterize it appropriately for our use.  If we’re being meta and require optimization then we’ll also need to describe the parameters to modify and the ranges for each.  This gets boiled-down into an OptimizationDescriptor which is composed of a prototype strategy and the parameter/range data required to perform the requisite back-tests.
• Configure Optimization or Meta-strategy – If we’re not running an optimization or utilizing a meta-strategy, this step is skipped.  Otherwise, this is where the optimization and/or meta-strategy should be built and configured.
• Review & Launch – Anytime you might loose a piece of software on an account with real money in it, it behooves you to pause and reflect for a moment.  Now is that moment.  If, after that moment of reflection, both you and whatever risk measures you employ agree that it’s OK, then it’s time to launch.  Happily, this is easy because we now have everything we need.  Again, the meta cases are a bit more involved as the strategies to be optimized or managed must be first generated based on the optimization descriptor we’ve inherited from earlier steps.  Then it’s a matter of assembling the pieces previously selected and configured and away we go.

-

Anytime you might loose a piece of software on an account with real money in it, it behooves you to pause and reflect for a moment.  Now is that moment.

I was taught that writing software is easy if you can express the problem you’re looking to solve and its solution in clear and precise English.  I’m not sure about this being grammatical, clear or precise, but hopefully it’s enough to illustrate a means of uniformly modeling and implementing these beasties across a variety of use-cases.

For me, at least, writing this description is helpful as it forces me to walk through the various twists and turns in some detail before digging into the code.  When I get the integrated walk-forward testing working I’ll try to provide some illustration of its use and will look-ahead to developing a regime-switching example using the models/mechanisms we’ve been discussing.

Even if it’s hard to find good stuff, one must still look as there’s always more information that can help you than you can effectively process or retain.  A few weeks ago I was trying to formalize the expected profit function of an algorithm I’m developing and wanted to see what people had written about the topic.  I entered ‘define profit function for trading algo’ into google and was pleasantly surprised to see a paper entitled ‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani.  It doesn’t directly cover the topic I was looking for, but instead addresses a number of related topics I’ve been interested in for some time:

• the treatment of a strategy as an instrument in its own right
• composing portfolios comprised of strategies
• using regime switching techniques to manage portfolios of strategies

In this post, I’ll briefly review their paper, illustrate how one can easily model strategies in relevant ways using the strategy ‘object model’ I’ve described previously through an example, and conclude with some thoughts on how these kinds of strategies might be implemented and further explored.

‘Multi-strategy trading utilizing market regimes’ by Mlnarik, Ramamoorthy and Savani

Their paper begins:

This paper considers the problem of dynamically allocating capital to a portfolio of trading strategies. The allocation should be robust, and the capital allocated to a trading strategy should reflect the confidence in the expected profit that the strategy will make in current market conditions.

Good trading strategies exploit recurring market dynamics that can be more prevalent in some time periods than in others. Indeed, the concept of regimes is fundamental to financial markets, and much research has focused on the detection of regime shifts. In this paper, we consider a regime as defined by a set of trading strategies that exhibit similar performance in a given time period.

We consider different parameterizations of the same strategy as distinct in our ground set of strategies. The trading problem is to pick a distribution over the ground set that will achieve good performance in the current time period.

They start with some interesting definitions.  A trading strategy is defined as a function which takes a given market State and generates a convenient Answer “denoting all possible trading decisions”.  Strategy functions can also be parameterized and different parameterizations of the same function/strategy are viewed as distinct qualitative strategies.  They assume the existence of  the set of all strategies.  A regime is defined as a set of trading strategies who behave similarly over a given period of time where similarly is measured by the variance of a fitness function.

So they start with a set of strategies, a fitness function and some historical market data or state, and their first task is to identify the regimes.  They do this by running the strategies against randomly selected partitions of historical data, quantifying their respective performances with the fitness function and identifying the clusters deemed statistically significant.  Those sufficiently significant and similar sets of strategies are themselves the reasonable regimes while the set of strategies within all the reasonable regimes are themselves dubbed the reasonable strategies – the candidates for execution.  This might sound strange, but as I’ll illustrate with a concrete example below, this is actually a pretty effective and even intuitive way of defining regimes.

They define function stFuncDist as, for any regime, generating a distribution or weighting of the strategies within the regime. While classifyMarket is a function that generates a distribution across all regimes for any given market state. Both models can be “trained” and they do so by again iterating randomly across chunks of historical data adjusting the weighting constants of these two functions.

Finally, it’s time to trade and they can effectively use classifyMarket to determine a predominant regime and allocate capital as indicated by stFuncDist to the appropriate set of “reasonable strategies.”

They claim to have utilized this technique to generate the results charted in Figure 1, but they give little detail on the initial universe of strategies.  I believe they had one strategy which was parameterized into a set of qualitative strategies and run across one market, the CME’s Nasdaq emini.  I have no reason to doubt that they had exactly this result, but without more details on what they actually did, it’s hard to view it as very meaningful, though it certainly is suggestive.  An interesting thing about how they present their result, which I’ve seen in other academic papers, is that because they’ve precisely defined the set of possible strategies they look at, they can also calculate max and min possible returns over the relevant period.

Figure 1: Out-of-sample profit (middle line) versus possible min (bottom line) and max profit (top line) over all strategies for the period 2006-11-01 to 2008-08-28.

The paper continues with another section, but I’m going to stop my overview here as the last part is less interesting to me and the interested reader should of course read it themselves.

complex strategies or simple strategies combined

One of the broadest design questions I see for the development of effective trading algorithms is whether one should employ complex, customized algorithms or instead utilize simple algos which can be simply combined to produce emergent complexity.  Although the authors of this paper allude to “Good trading strategies”, I wonder if one can even forget the “good” part and instead worry about applying – even bad! – strategies at the right times.

There are a lot of things I like about this paper, but probably most of all it’s their definition and use of a regime.  One of my first posts describes a pair of strategies – a simple mean reverter and a simple trend follower.  These two strategies are about as dumb as you please.  And yet, their behavior is pretty interesting apart the observations I’d published at the time regarding the distribution of their returns.  During the trading day, I’ve taken to running those two, old, dumb strategies across a variety of markets in simulation mode.  Why?  Because it’s easy, costs me nothing and actually is pretty interesting in that the strategies really do cluster in the way Mlnarik et al describe, effectively partitioning the market into visible regimes.

a simple example: identifying regimes with strategies

In order to illustrate this, I created a desktop within stratbox onto which I ran these two qualitative strategies parameterized differently and across six different markets.  In total, I have 48 strategies running with 24 of the mean reverting (MR) variety and 24 of the trend following (TF).  In figure 2 and figure 3, they are sorted by qualitative strategy – all of the MR strategies followed by all of the TF strats.   In figure 4, they’re instead sorted by instrument at the end of the day after all of the strats had been rendered quiescent for the night by their schedulers.  You can click to see a bigger version of these screenshots.  On each, the left side of the screen shows simple charts of the relevant markets while the right contains a table in which each line item is a running strategy.  The first is at the beginning of the day. the second is shortly before going silent at 4pm and the third is shortly after 4pm.

Figure 2: variously parameterized MR & TF strats run in realtime+simulation mode. Sorted with the MR strats first. (click)

Figure 3: later the same day

Figure 4: at EOD, now sorted by instrument

You’ll note that at all times throughout the day, the MR and TF strategies effectively mirrored each other regardless of market.  Although I’m not going to illustrate it, I’m going to just go right ahead and assert that this is almost always true  based on what I’ve seen.  And this is at the heart of the paper’s authors’ insight regarding regimes – they can be effectively identified by examining the performance of even simple but well-defined strategies.

This suggests a simple means of implementing a similar design to that they described right on top of this example I’ve put together.

modeling meta-strategies

We need one more element to manage all of this – a meta-strategy – that is, a strategy for managing other strategies.  We support this idea in stratbox with what we call a controller strat.  The cannonical example of a controller strat is our implementation of the (Don’t) Panic button (the bright red button with an ! in the upper right hand corner) which will get all strategies out of the market with varying degrees of urgency and finality.  This is implemented as a controller strat which injects a stratpart into all running strategies.  The injected stratpart first deactivates its peers and then cancels outstanding orders and then gets out of the market with whatever urgency the user has required.

allocating to portfolios of strategies

In my example above, I have 48 strategies running in simulation across 6 futures markets.  Let’s imagine that at any moment in time, I will always have at least one strategy of each type (MR and TF) running for each market. Thus, I’ll always have 12 live strategies running against real money accounts and 36 candidates running against a simulator.  Effectively, we’ve just described a portfolio of 12 elements where each element is itself a strategy.

It’s not difficult to imagine implementing a controller strat which on a regular schedule will simply adjust the parametrization of the running strategies and their allocation of cash based on the performance of the strategies running in simulation.

It’s not difficult to imagine, but implementing it inevitably poses a variety of challenges not least of which is finding the time!  For now, we’re still finding new and interesting ways to express strategies which deal with portfolios of exchange-traded instruments.  But I think that this is just a point on our technology’s maturation curve which will ultimately lead to the regular development of strategies which operate on portfolios of strategies themselves…

Anyway, it’s nice to see a bit of academic work formalizing what I expect is a natural step in the evolution of trading algorithms.

Mmmm... hardware..

I’ve never been a hardware guy. Hardware has gotten so fast throughout my professional life that it has just never been a big issue. Also, on wall st we had a robust and annual budget for h/w so I’d routinely sign-off on hundreds of thousands of dollars on all sorts of machines I’d never lay eyes on and somehow they always did the trick.

Before 9/11, they’d be in server racks in the building or down the street, but since then they might also be in increasingly far-flung places like weehawken or long island, tampa, even texas or beyond. The machines always seemed unbelievably overpriced – I remember over the years pretty consistently paying something like $40K for a low-end db server.  But that’s what it cost and you could only purchase approved products from approved channels, so nobody spent much thought on it.  Now that I don’t have the same kinds of constraints – or budgets! – I increasingly have to think of hardware.

As a software engineer, the hardware itself is also insisting that I pay some uncharacteristic attention to it.  The evolution of processors has reached a point where the programming paradigms many of us have fruitfully employed over many years are no longer suited for getting full performance out of today’s machines.  The recent introduction of remarkably powerful and inexpensive parallel-computing platforms based on GPUs like nvidia’s cuda also outline a future that even current university training doesn’t address in a fashion practically adapted for institutional application.  Cores are multiplying like Tribbles.

The lines between persistent storage and main memory are also blurring as consumer SSDs push up from the ‘low’-end while exotic ioDrives and the like offer a glimpse of a world where the performance gap between the two approaches nil and after their long reign myriad metallic platters will spin no more.

troubling like Tribbles

There have been some steps taken towards taming the core dilemma.  Google’s introduction of the distributed map-reduce paradigm and all of the associated plumbing on top of computers in the ‘cloud’ is probably the boldest and most effective reaction thus far, but it’s not always obvious that you want your stuff running in someone else’s cloud amongst many other natural limitations of this approach.  This is also a solution ‘in the large’ and sometimes you need performant solutions on a different, smaller scale.  Here, the development of functional languages and idioms may be of some help, but there certainly don’t seem to be clear winners yet.

Erlang, Ocaml, Haskell, Scala and others all seem to have very limited impact thus far and all face big challenges before receiving widespread welcome.  Worse, any language can be mangled into expressing things poorly so the languages can only be a meaningful aid to programmers who are able to adjust their mindset for a new world’s computing paradigm.  This likely won’t be easy for many until there is an established set of usable programming idioms and toolsets for dealing with concurrency on a whole new scale.  To me, it seems that functional programming might well be an important part of that, but it’s difficult to imagine it as the complete game changer any time soon.  As it is, many of us have already been using functional languages (in my case, sql and R) on a regular or even daily basis for a long time, so it’s difficult to cast this as the revolution, bottled.

As cores proliferate and the bandwidth amongst them increases, new challenges and opportunities are unveiled.  Feeding all of those hungry cores can be a chore.  If you already had problems with i/o bound processes, then adding cores is sort of like adding liquidity to a debt crisis – not obviously helpful. We faced an issue like this recently while trying to improve the throughput of our backtesting subsystem in particularly poor-performing cases.

In these cases, we use daily data to initialize an intraday strategy.  For example, we might look at all equities over a trailing 3 or 5-year period, and perform various analytics – like calculating correlation matrices, beta against various benchmarks, volatility, etc – to determine which names might be candidates for the strategy’s portfolio.  We found that an inordinate amount of our time was spent just performing these morning analytics and that the cost of a day’s backtest was significantly spent on this oft-repeated morning exercise.  While working through this issue, we noted that we’ve reached a point where a decent, dual nehalem, server (with 16 logical cores) could be built with 48G of RAM for something along the lines of $6K.  Seriously.  So we stuck all of our daily data into memory and the improvement has been essentially infinite.  Maybe not the best example, but hopefully illustrative of the fact that the h/w underneath us is changing qualitatively and that we need to be more active in involving it in our design decisions.

Of course, even a yawning expanse like 48G or even a terabyte (in not too many years, after all, I should be able to buy 1T of RAM for my boxes at a similar price point to what I pay for 48G today), eventually gets consumed and so we can’t hope to employ this solution for all our problems.  We continue to develop our historical TAQ infrastructure (most recently discussed here) and this is certainly an example where buying memory isn’t going to get you very far.  But SSDs now are getting reasonable and so our current approach uses memory as much as possible but when it needs to get volumes of detailed historical data we’ve placed our indices onto SSDs while the actual stores themselves reside on much more affordable RAID arrays.  Reading indices is now much faster and adds minimal i/o overhead to a very i/o-bound problem.

By constantly looking at new languages and concurrency idioms, vigilantly assessing the current and projected costs of ’solid-state’-solutions to our most vexing problems, and just staying active and creative, we hold out some hope that we can transition gracefully to what is looking increasingly like a solid-state future.

This isn’t surprising as I wanted the system to support all of the activities associated with our algorithmic trading.  It turns out that that’s a lot to ask of a system.  It also turns out that you learn as you go and so the system continues to evolve.  A few years ago I’d posted about the basics of a strategy container and in this post I’m going to come back to this topic and describe some of the layers of code and thought developed since then.

First, let’s consider the role of a strategy container.  Its job is to intermediate between trading strategies and the external environments with which they interact.  It must also provide services that strategies can use (e.g., position management) and that it wouldn’t make sense for each strategy to re-implement.  In the past I’ve focused on the former responsibility of adapting strategies to external environments.  Why is this necessary and interesting?  Because it allows us to take the same exact strategy and run it live, or in simulation or in backtest, etc.  Interesting and necessary, but not what I want to focus on this time.  Instead, I want to look at the services provided to strategies; the ‘ecosystem’ a strategy container provides in the hope that strategies might flourish within it.

We’ll go from the bottom up.  At the bottom you have one or more pipes of market data coming in.  You might also have news feeds, weather feeds or other such things coming in and we’re just going to conflate them all and say “market data.”  Still at the bottom, you also have one or more two-way pipes with your oms or broker or directly with exchanges or with an exchange simulator.  Within Stratbox we call this abstraction an “ExecutionPlatform” because it’s conceptually where trades get executed.  The lingua franca for execution platforms is FIX, so the baseline object model of a strategy container is likely going to look a lot like messages described by the FIX spec.  Here we have orders, executions and the like.  Within Stratbox we’ve implemented an exchange simulator, a QuickFIX-based FIX interface and a couple broker-specific APIs and they are all of type ExecutionPlatform.  To a strategy, they all look the same.  Likewise with market data.  Within the strategy container, we provide a publish-subscribe model wherein any kind of market data can be subscribed to regardless of the ultimate source of the data.  The intention is always to intermediate between the strategy and its external environment.

the baseline object model

Great.  But now I want to write a trading strategy.  What’s that going to look like?   And this is where things get increasingly interesting as we get to decide what kind of facilities we’re going to provide.  We also need to manage concurrency in some fashion as trading is an intrinsically asynchronous activity.  For this lowest level of strategy, we’ll say that each market feed is handled by a thread which timestamps and enqueues a “Quote” for later consumption and redistribution to relevant subscribers by a MarketDataService within the container.  Likewise, each execution platform will be serviced by a thread who similarly enqueus “Execution” objects from exchanges real and simulated.  The threads handling executions should likely have a higher priority than the threads handling market data.  So, the strategy is naturally exposed to the markets’ asynchronicity right off the bat.  What baseline facilities will our strategy have at its disposal?  Well, it needs to be able to manage orders and positions, so it needs some kind of a “blotter” facility.  With this in place, a strategy can safely assume that it doesn’t need to listen to each execution coming in just to have a correct picture of its book.  It needs access to a MarketDataSvc and it likely also needs access to historical data.  But, staying simple, that’s more or less all we have to provide and we’re going to provide all of these things to the strategy through a single handle: a strategy context.  By swapping out a strategy’s context, we can move the strategy among environments (eg, from simulation to live execution).

With these facilities in place, basic algorithms can be implemented within a strategy and you should be able to test and trade them.

But you’re dealing with very low-level stuff.  You’ll also find that you’re writing the same kind of code over and over.  This inspired the introduction of a layer on top of the baseline object model which I’ve described before: StratParts – a strategy component model.

StratParts – a component model for strategies

With StratParts, we introduced metadata and composition (in the “composite pattern” sense) to our Strategy object model.  A strategy is itself a stratpart which contains other stratparts.  Thus, stratparts introduce a hierarchical structure to a strategy.  Each stratpart publishes a metadata descriptor which is aggregated at the strategy level.  The descriptor contains all of the modifiable characteristics of the strategy and can be changed while a strategy is executing thus giving a sort of grey box capability wherein a trader (or another strategy etc) can modify the behavior of a strategy as it’s running.  Stratparts also create an effective means of providing scoping within your strategy environment.  Stratparts can “see” the activity of other stratparts within the same strategy – they’re all peers in this sense.  This scoping can also be used for allocation of resources across different elements of a strategy.  For example, one stratpart might be allocated 80% of the cash available to the strategy while another stratpart manages the remaining 20%.

Stratparts have proved to be very powerful and we’ve written many of them though we’ll sometimes write strategies as one monolithic stratpart where it makes sense.

Stratparts have a lot of uses but one thing they don’t do is help manage the low-level complexity inherent in trading activities.  This was best described by a trader with whom I’d collaborated.  He’s a reservist in the IDF and uses military metaphors like I use sports metaphors.  He bemoaned the low-level handling of orders required for a very close-to-the-market strategy we were working on and said that:

we need an order like a ‘smart’ missile: fire it and forget it

He was right and that inspired the next level of abstraction/support that we built into the system.

‘smart’ orders and the tradeflow stratpart

The strategy we were developing was meant to look at a universe of futures spreads and generate all of the ‘cycles’ that might result in an arbitrage opportunity.  Since these are rare at best, we were really looking for circumstances where it looked like we had an advantage based on depth-of-market and various heuristics we’d apply.  Among the functionality that he wanted was an ‘order’ which would act as a limit unless some conditions obtained in which case it might pay the spread or otherwise stop being a fixed limit order.  He didn’t want to handle this inside the strategy but instead wanted to fire these things off and forget about them unless they required attention (eg, if they’re rejected by the exchange or his firm’s risk management checks).  Although we ended up calling these smart orders, they’re really a sort of very localized execution strategy themselves.

To support them, we utilized the same workflow (= state machine) framework we’d implemented for our ExchangeSimulator as we wanted to ensure that these smart orders had a very well-defined life-cycle with a clear set of states and guarantees about transitions among them.  We embedded this workflow engine into a stratpart which supported smart orders operating within this “tradeflow”.  Thus, the extra functionality (and weight/expense) of this functionality is only achieved/incurred when actually required or desired.  This has proven to be a powerful addition to the system and raises the level of services enjoyed by prospective strategies substantially, particularly given the variety of orders we’ve implemented which stand ready for use by any new strategy that might want to employ them.

At this point, it should be clear that we’re blurring the lines between the strategies themselves and the container.  Another way of thinking of it is that one container can have a set of pre-made building blocks that can be applied for families of strategies.  Each one provides a particular grammar with which a strategy’s aims can be expressed.  For very close-to-the-market strategies where very low-level handling of orders is required, the tradeflow stratpart provides a set of really useful facilities, but if I’m just trading with limits then it might not be so interesting.

portfolio-oriented strategies and beyond

I’ve mentioned before that I only care to think of strategies that operate on a portfolio.  As such, we’ve written a good deal of facilities for creating, analyzing and manipulating portfolios within strategies.  By using these facilities within a tradeflow stratpart, I can express quite complex strategies relatively simply by using portfolio analytics to determine my current ‘model’ portfolio and then using smart orders to most effectively transition me from my current state to my model state.  For the kind of trading I do, this provides a rich ‘vocabulary’ for the development of strategies.  All the same, my brain is having a tip-of-the-tongue moment as I feel that there’s an entirely different model possible for portfolio-oriented strategies just at the periphery of my imagination…  oh well, there’s always something new to explore.

Different trading styles, perspectives or trading problems are undoubtedly better served by different sorts of facilities.  I’m finding that just as language constrains and shapes thought, the strategy container that you employ shapes and constrains the kinds of strategies that you can implement.

If you have any ideas about such facilities for any kind of trading you do, I’d love to hear them.

–

a note about Stratbox

Although I’m talking about our system in this blog, we’re not marketing the system and happily “just” use it for our own trading activities.  We had looked into marketing the system previously but ultimately feel the same about selling stratbox as selling strategies…