or: scaling with elastic map-reduce

Between a rapidly evolving compute environment in which cores are multiplying like springtime rabbits, and a business domain in which the fecundity of market data is making those same rabbits look downright prudish, we are always looking for ways to scale our efforts.  There are three levels at which this can be typically done:

• “below the processor” with things like CUDA or FPGAs,
• “amongst the cores” with things like TBB, Cilk, &tc, or
• “in the cloud” (or grid) with things likes Amazon’s EC2 and its bewildering and fast-growing coterie of associated technologies and products

Today we’ll look at a simple example which we implement on top of Hadoop and then deploy into Amazon’s cloud to get a back-of-the envelope feel for what kind of scaling we might expect to gain – and at what costs – from using this smorgasboard of technologies.

’smorgasboard’ is being generous

Probably the hardest thing whenever you look at some new set of technologies is just getting your bearings.  Amazon’s stack is complex and very much of a moving target, so I’ll start by providing a bit of a taxonomy of the beasties we’ll need to at least familiarize ourselves with in order to get anything done.

First, there’s the family of Amazon’s Web Services (AWS).  This has been around for a good few years, so some parts of it are quite mature and well documented and understood where others are bleeding edge with an emphasis on bleeding.  All of them can be played with quite easily by simply signing up (this involves providing a credit card, so head’s up).  I can’t think of any more interesting family of technologies, so if you think there’s any possibility these technologies might help you scale, then I encourage you to sign up and give it a spin.  Very interesting stuff.

Anyway, within AWS we’ll play with Elastic Compute Cloud (EC2), Elastic MapReduce (EMR), Simple Storage Service (S3) and Elastic Block Storage (EBS).

EC2 provides you with direct access to Amazon’s racks of humming servers in the cloud.  Thanks to the miracles of modern virtualization technologies, an entire machine can be bundled up as an image file – an Amazon Machine Image (AMI – think of an OS’s DVD but with software that you specify, configured as you please). Once bundled into an AMI, your computer can be ‘installed’ onto any number of machines within the cloud.  Nifty.

Such machines can be bought for long term use (at quite affordable rates), or they can be used on an as-needed basis for an hourly charge.  Hence the ‘Elastic’ in EC2.  They can also be managed programmatically, and until recently, this was pretty much the only way to develop jobs in the cloud.

Each machine that you use in EC2 has some local storage associated with it, but this is not persistent across sessions.  Once your instance goes down, the data is all gone.  To get around this and to provide an all-around storage solution, AWS provides S3 which is, to me, an overly-simplified distributed file system which is something of a pain to use.  It is also limited to file sizes of 5G which I find to be a significant annoyance.  The good news is that S3 seems to be both highly durable and reasonably performant.

A more recent entry in the AWS world of storage is the EBS facility.  Here, you can create arbitrary sized “disks” in the cloud which can be attached directly to your computer nodes just as though they were local disks.  Very handy.  They don’t have the same level of durability as S3 as they’re not distributed, but I haven’t had any difficulties with them and they’re very useful for putting together AMIs within the cloud.

Finally, EMR brings the Hadoop implementation of Google’s marvelous distributed map-reduce model into the AWS cloud.  If you haven’t read about map-reduce, well perhaps you should – it’s really powerful.  Hadoop is an open-source implementation of it done in Java and seems quite nice though it’s under very active development, so the APIs are still very much a work in progress.  Be prepared to change your code.  Even though it’s written in Java, any language can be used to implement your own applications, and it seems as though both ruby and python are used pretty extensively along with java and c/c++.

what map-reduce buys you

The first thing map-reduce buys you is something of a headache as fitting your application design into the map-reduce model isn’t so obvious.  That said, it may well be worth a couple aspirin as it brings a great deal to the game.

Consider an application we might find interesting which we’d like to place in the cloud – strategy back-testing. It seems to be a very natural fit for parallel processing as each strategy is independent of the others, so theoretically we should be able to run each one on it’s own virtual server without interdependencies.  While this is true, it’s rather easier than it sounds.  In order to implement our distributed backtesting platform, we first need to implement one that works on one node.  Fair enough.  We must then introduce a “chunker” which will break the overall set of simulations into bite-sized chunks.  We must then define an intermediate format for sending the strategies (or instructions) to each node.  And we must define an intermediate format for the results.  And then we need to implement an “assembler” which coalesces all of the results back into one result set which can be reported on or displayed etc.  And if one of the nodes fails, we have to notice it.  And if we notice that a node has failed, we need to figure out which job(s) it was responsible for and reallocate them to another node which presumably hasn’t failed…

And on and on.  The point is that this isn’t something that you’re going to get done in an afternoon.  Indeed, it’s unlikely that you could write a correct and complete specification of the system in an afternoon.

What map-reduce buys you is all of the distribution bits that I mention above.  The costs are that you will have to figure out how to model things such that they’re compliant with the map-reduce model and that the end result might not be as efficient as a hand-coded solution.  But when you can scale easily and cheaply, then raw efficiency may not be such an over-arching concern.

scaling in the cloud with elastic map-reduce

In order to get an idea how difficult it is to actually scale a solution, I implemented a simple program using real data and collecting some timings across a variety of  AWS/EMR configurations. To do this, I first had to code against Hadoop v18.3 which is what is supported in AWS.  (Initially I wrote it to v20.2 and was surprised at how much I had to change to back-port it to 18.3.)

The details of my test program aren’t important – basically counting various features from a set of heterogeneous files.  The files are approximately the same size ~1.8G and there were 10 of them, so the overall dataset is a bit over 18G.  I only do one map-reduce; I’m not chaining them together.  On a local machine – a pre-nehalem, dual-chip w/ quad-cores (8 total) xeon server with hardware raid – the process took 1068s or a touch under 18 minutes.

I was impressed to see that with absolutely no code changes, I was able to run the system on EMR without difficulties.  In fact, selecting the number of nodes to run against and the size of those nodes is just a matter of changing launch parameters.  Very nice.  AWS offers a variety of different specs on their servers.  I tested the classic ’small’ server which provides a 32-bit O/S, one core and 1.7G of memory as well as the ‘large’ server which comes with a 64bit O/S and 2X2 cores and 7.5Gs of memory.  There are other configurations available, but I didn’t test them.  In all cases I was running centos5.4.

The results are tabulated and graphed below.

performance locally and in various cloud configurations

time to complete vs. # nodes for large & small nodes

Scaling isn’t exactly linear, but it is good and it is simple and it is cheap & on-demand. Clearly for any given use-case, it will make sense to do tests like this to figure out what makes the most sense.  Another dimension to consider is cost.  Even partial hours count as a full hour, so putting big or many boxes against jobs that don’t take long to run isn’t cost effective.  That said, even though I fecklessly ran jobs which took as little as 10 minutes while paying for 60, the entire costs associated with developing and running the example I present here was less than $4US.

One thing I noticed is that the time to initialize instances seems to grow as you increase the number of nodes, though I didn’t quantify this.  In some cases, it was really quite long before real work started getting done – several minutes at least.  Getting this to work for interactive processes will take some thought and, probably, some dedicated nodes.  Another things to note is that although my local server did a fine job for itself, if we were to increase the dataset by a factor of, say 100, we’d reach a point where it simply couldn’t complete the operation whereas the cloud isn’t so strictly bounded.  It’s also pretty interesting that my local machine with 8-cores and 24G of RAM and pretty decent RAID subsystem got soundly spanked by eight of Amazon’s cheapie 1-core boxes.

Given that it is cheap, massively and simply scalable and not terribly difficult, one could say that if you don’t spend some time with your head in the clouds, then it’s possible you’ve got it stuck in the sand.

One of the recurring technology themes in these pages has been the ongoing and dramatic move from single to multi-core systems and the need to seriously increase the parallelism in our software designs. For me, one of the seminal, large-grained design patterns was the SEDA Architecture. For years, this informed my systems’ designs and formed a conceptual backbone for development. That said, I’ve been broadly aware for some time that SEDA’s golden age has (incredibly!) already passed us by, but haven’t identified what might replace it as a reference point for my design efforts.

Before considering tools, languages or patterns that might help, we need to reflect on the problem(s) we’re trying to solve. The problems inside an EMS look to me, after years of development, a lot like network routing problems. Indeed, my current view is that this (not just concurrency as I’d suggested at the time) is why the unfortunate Aleynikov & co. at GS were using erlang.

Why network routing? Think about the load on an EMS. The main issue is that you’re getting many thousands of teeny little messages per second and only a relatively small number of them matter to only a relatively small subset of ‘agents’ within the system. Reducing latency is all about making sure the time you spend on each message is minimized, and that the agents who are interested in a particular message needn’t wait for each other to do whatever they care to do based on the message. So, really you’re trying to route each message through your system with as few ‘hops’ possible and as much parallelism as you can muster under the (radically!) new assumption that you may have hundreds or thousands of cores available to you during the lifetime of the design.

I spent some time thinking (hoping) that languages might help furnish an answer. Perhaps a move to a functional language like erlang, ocaml or scala might help furnish at least a partial answer. But erlang is slow and peculiar, ocaml doesn’t support intra-process concurrency and scala looks like a bloated language on a bloated platform (jvm+java class library). And none of them seem to have achieved anything near the critical mass which is so crucial for the development of usable libraries and the availability of skilled developers with long experience in the technology.  Naturally, reasonable people will disagree about such things, but this is my view (today). Java is ok (and certainly sells servers), but it’s not obvious how it’s going to help me offload my work onto a GPU anytime soon (and jni is both painful and slow) and I’ve never been able to get comfortable with just how damn big VMs get.  Image size isn’t free and if we’re looking to go deep into the sub-millisecond response time, while running thousands of concurrent strategies, it seems we need to disintermediate the VMs and interpreters of the world. If they’re really necessary, they can be happily used for the analysis process (as I currently use R), or they can be lit-up and bridged from some lower-level language for batch-like services.

The good people at Intel have been thinking about this problem for a while as have many other seriously over-educated people. One of the (sensible sounding) conclusions reached as people look for ways to solve problems similar to my own, is that in such systems we should keep messages waiting as little as possible – ideally, not at all(!). This can be a problem in SEDA-like architectures which are basically made-up of (non-blocking, asynchronous) i/o processes linked to (blocking) queues linking pools of workers. Blocking queues can pile up and cause all sorts of problems like priority inversion and other such enigmatically named nasties. Lock-free queues and other data structures, algos and techniques promise some ways around this and I’ve been spending time looking into how they might be employed to address my issues.

Before I’m besieged by throngs of angry erlang/ocaml/scala/java developers, allow me one last observation on the topic.  (Peeved python and ruby users may rant away – vous m’amusez  ;^)

Why might a lock free algorithm be better than an equivalent, hardware-based locking implementation?  The answer isn’t obvious.  If locking is implemented in hardware as is typical (eg, with a compare-and-swap (CAS) instruction), then its explicit cost is measurable in (few) nanoseconds.  Hardware is fast.  The issue isn’t the speed of execution of the underlying primitives so much as it’s a consequence of the side effects of these operations at a very low level.  For real performance, cache coherence is King.  See here for an accessible discussion by IBM’s Paul McKenney and here for some remarkable examples from Igor Ostrovsky.  This indicates that if you want the highest possible performance, you need to be aware of what is happening ‘in the metal.’  So we need to use a system-level language and erlang, java & friends lose their candidacy in spite of any fantastic benefits they might offer.

Given that even the DoD has mostly given up on ADA means that we’re left with C/C++.

Ok, so language doesn’t seem to resolve much for us. (Indeed, it was mostly hopeful thinking on my part – design is mostly language agnostic and hardware is hardware…)

Apart from Intel’s own Threading Building Blocks (TBB) framework, there are a variety of toolkits available for exploiting lock free parallelism. Perhaps the newest and least known is called FastFlow, which is a C++ template library that provides a variety of facilities for writing efficient lock-free network models. It also claims to be faster than TBB, Cilk and OpenMP while holding out the promise of one day becoming CUDA- (or more generally, GPU-) aware which would be an incredible win. Finally, it is very small – the current version (not including tests and examples), weighs in at ~5K lines of (mostly) C++ templates.  Thus, it seems to me particularly well-suited for some experimentation to assess the fit of these techniques in this space and the level of difficulty of doing so.

In the remainder of this post, I’ll briefly describe the FF design and then illustrate a sample C++ program which uses FastFlow to ‘architecturally prototype’ a feed handler interacting with strategies inside an EMS / strategy container.

FastFlow architecture

FastFlow’s (FF) goals seem quite ambitious, but the design is layered to allow users to work at a level that makes sense for the problems they are trying to solve, so they needn’t worry themselves (or even understand necessarily) layers below where they need to work.  At bottom is a thin layer of hardware-specific code supporting, one level up, single-producer-single-consumer (SPSC) lock-free queues and a threading model for their efficient interaction.  Above this are defined objects for generalizing from simple to arbitrary network types.  Here you see the definition of a composable Node which serves as the key abstraction for general-purpose streaming networks.  Above this are defined a variety of skeleton templates for building commonly useful graph types.  Because they are Nodes themselves, they can be nested and coupled to any depth.

Consider a simple pipeline:

a simple FastFlow pipeline

A farm is slightly more complex and acts something like a load balancing threadpool.  The Emitter pushes ‘tasks’ of arbitrary type to an arbitrary number of workers who then coalesce back into an (optional) Collector.  A simple Farm makes no promises about the ordering of tasks, but (I believe) FF provides mechanisms for ensuring their ordering at the collector.

a FastFlow Farm

Finally, because of the composable nature of Nodes, all of these graph types can be combined arbitrarily.  Here you see a Farm of pipelines with an ‘accelerator’ (which, if I understand correctly is just an integrated thread for pushing tasks into the network) and a feedback channel.

composition of FastFlow subnets

For a more complete (and probably accurate!) description of FastFlow, I encourage you to visit their site and tutorial.  The documentation remains limited, but there’s a good and growing selection of examples to examine and all of the source is available for inspection.

a simple example

Given my use-case, my first interest is in seeing how FastFlow might work in the context of asynchronous, non-blocking network handling.  To this end, I’ve built a simple Farm as depicted above in which the Emitter uses select() to read messages off the network which it parses and pushes into the farm for ‘processing’ which is currently NOP-ed.  The messages/tasks are then coalesced into a collector which does nothing more than free memory allocated in the Emitter.  Within the Farm, each of the nodes inhabits its own thread and outside of the farm I’ve implemented a very simple ‘client’ which connects to the Emitter’s socket and sends delimited Integral ‘messages.’  Ordinarily this would appear in its own process, but for the sake of only requiring 1 file, I’ve mashed them all together and the client lives in its own thread.  That’s it.

It turns out that the FastFlow part of this is, by far, the simplest.  Since the FF skeletons are implemented as templates, extending them couldn’t be much easier.  Here’s the definition of the worker and collector components.

// typical worker: does little ;^p
class Worker: public ff_node {
public:
  void * svc(void * task) {
    return task;
  }
};

// collector that just frees the malloc'ed memory
class freeing_collector: public ff_node {
public:
  void * svc(void * task) {
    int * t = (int *)task;
    if (*t == -1) return NULL;
    else free(task);
    return GO_ON;
  }
};

They are both nodes and their only requirement is to implement the svc method which is called by the FF framework when they are meant to service a task.  They can optionally implement svc_init and svc_end methods which provide an opportunity for initialization and cleanup.  The socket-reading emitter makes use of the svc_init method to initially setup the socket.  Later, as it repeatedly reads from the socket into a buffer, it cracks the incoming stream and then calls the ff_send_out() method to push those messages out to the downstream workers.  The whole example is about 350 lines of code and nearly 200 lines are consumed by the Emitter.  Almost all of this bulk is network-related, so I won’t illustrate here.  But highlighted here you can see the main FF-specific part of the Emitter:

 // handle new data coming in
 void new_data(int nbytes, const char* buf) {
   static char remainder[kMaxMsgSz];
   int *t;
   int j = 0;
   int i = 0;
   char msgbuf[kMaxMsgSz];
   for (; i < nbytes; i++) {
     if ( kDelimiter == buf[i] ) {
       int rlen = (j==0) ? strlen(remainder) : 0; // remainder?
       int len = i-j;
       if (rlen>0) strncpy(msgbuf,remainder,rlen);
       int start = (rlen==0) ? 0 : rlen;
       strncpy(&(msgbuf[start]),&(buf[j]),len);
       msgbuf[len+rlen] = '\0';
       t = (int *)malloc(sizeof(int));
       *t = atoi(msgbuf);
       j = i;
       ff_send_out(t);  // push message into farm
     }
   } // for

   int k = j+1;
   if (k < i) { // save 'remainder'
     strncpy(remainder,&(buf[k]),(i-k));
   } else remainder[0] = '\0';
 }

Finally, all of the pieces are assembled and launched in the main().

//  We construct a 'server' which is a fastflow emitter and which reads
//    integral 'msgs' from a client over a non-blocking socket.  Msgs
//    are parsed and fed into a fastflow farm for further handling.
//    The client is placed within its own thread - this is only so the
//    whole example can be placed in one file.
//
int main(int argc, char * argv[]) {
 // use: argv[0] <#msgs=1024> <#port=9999>

 int msgs = (argc > 1) ? atoi(argv[1]) : 1024;
 int port = (argc > 2) ? atoi(argv[2]) : 9999;

 int nworkers = 5;               // how many workers will recv msgs

 printf("main: sending #%d msgs to port <%d>\n",msgs,port);

 select_reader sr(port);         // we create 'server'
 freeing_collector fc;           // and a freeing collector
 ff_farm<> farm;                 // and a farm for it to live in
 farm.add_emitter(&sr);          // add both to the farm
 farm.add_collector(&fc);

 std::vector<ff_node *> workers; // build a collection of workers
 for(int i =0; i < nworkers; i++) { workers.push_back(new Worker); }

 farm.add_workers(workers);      // add all workers to the farm

 farm.run();                     // launch the farm
 printf("main: started farm.\n");

 sr.wait_til_ready();            // don't create client til srvr ready

 // create client which will send msgs via socket
 //
 printf("main: creating client...\n");
 client_thread* client = new client_thread(msgs,port);
 if (client != NULL) client->join();

 farm.wait();                    // wait for farm to be done its workload

 // emit some stats
#if defined(TRACE_FASTFLOW)
 std::cout << "DONE, time= " << farm.ffTime() << " (ms)\n";
 farm.ffStats(std::cout);
#endif

 return 0;
}

The whole example can be found here.

Writing this example was a bit of a ‘tale of two frameworks’ for me as I found the FastFlow part to be really easy and powerful.  On the other hand, I wanted to try out the boost::asio framework for non-blocking networking i/o (primarily in the hope it might cut down the code size a bit) and while implementing the trivial (blocking) client was, well, trivial, implementing a non-blocking server with boost::asio was a serious pain and I ultimately decided that Linus just might be on to something wrt to boost and scrapped that part of the effort (but kept the boost client).

Nothing about this example illuminates the potential performance advantages of a lock free approach, but this exercise has certainly convinced me that the FF authors have managed to combine very low-level performance considerations within a package which can be easily used from a high-level perspective.  Most problems will most likely be better served by higher-level programming languages, but some problems remain intractably bound to the metal and I suspect that writing ultra-low-latency trading systems will remain in the latter class for some time to come.

photo by Simon Rogerson

Some time back, I’d written about NVidia’s CUDA noting that it looked ideal for many asset-pricing and monte-carlo type problems in finance.  At the time, I was hopeful that it would be quickly integrated into existing open source efforts like QuantLib, but adoption has proved slower than I’d hoped, most likely because implementing non-trivial problems on CUDA is, well, even less trivial than doing them without..

LMM on CUDA

Happily, I’ve just seen a promising first step in this direction as Über-quant and C++ artisan Mark Joshi recently announced an open-source project, Kooderive which looks to implement the LIBOR Market Model (LMM)  on top of CUDA.  His announcement on the QuantLib mailing lists reads:

Dear All,

various people have shown interest in the use of CUDA with QuantLib. I
have now made some progress on a CUDA implementation of the LIBOR
market model.

In particular, I now have a path generator for the LMM working which
does 16384 paths for 40 rates, 40 steps, 5 factor model, displaced
diffusion predictor-corrector that takes 0.1 seconds on my Quadro 4600.

The state of the project is code fragments that can be called from
other code. Those who are interested can get the code via
the subversion repository on kooderive.sourceforge.net .  The only
project file is currently for VC9 x64. It also uses thrust and the
CUDA SDK.

The next stage will be writing routines, that use QuantLib for the CPU
stuff and kooderive for the GPU stuff,  to actually price things.

A gentle reminder that I will be giving a course on the LMM and
QuantLib in June in London, and I will include a session on kooderive
if there
is sufficient interest.

I am happy to take code contributions for kooderive. However, I am not
looking for a redesign of the library or contributions which introduce
dependence on other libraries. I am interested in contributions of
separate routines and of optimizations of existing routines that do
not change interfaces.

regards

Mark
–
Pricing exotic interest rate derivatives – The LIBOR Market Model in
QuantLib June 2010, London,
http://www.moneyscience.com/training/index.html

Assoc Prof Mark Joshi
Centre for Actuarial Studies
University of Melbourne
My website is www.markjoshi.com

FIX interface?

Cloud computing looks to have turned yet another interesting corner.   This time the turn leads towards the development of a liquid, fully electronic new marketplace in “spot instances”.

‘Spot‘ means what you would expect it to in the context of trading: the current pricing for immediate delivery of a commodity.  ‘Instance‘ is the atomic element within Amazon’s cloud environment; an instance is the smallest chunk of computing capability which can be provisioned within the cloud.

Amazon is making markets in cores and they’re exposing functionality just as a regular exchange would: both through user interface ’screens’ as well as programmable APIs.

From their announcement:

Spot Instances enable you to bid for unused Amazon EC2 capacity. Instances are charged the Spot Price set by Amazon EC2, which fluctuates periodically depending on the supply of and demand for Spot Instance capacity. To use Spot Instances, you place a Spot Instance request, specifying the instance type, the region desired, the number of Spot Instances you want to run, and the maximum price you are willing to pay per instance hour. To determine how that maximum price compares to past Spot Prices, the Spot Price history is available via the Amazon EC2 API and the AWS Management Console. If your maximum price bid exceeds the current Spot Price, your request is fulfilled and your instances will run until either you choose to terminate them or the Spot Price increases above your maximum price (whichever is sooner).

embedded optionality

While the inclusion of, effectively, a market data service is neat, probably the most interesting aspect of the initial protocol they’ve designed is that it contains embedded optionality and behaves a bit like barrier options.  That is, when I setup an ‘order’, I need specify a maximum price I’m willing to pay.  When the spot price drops below my max, I get “knocked-into” a contract and instances are allocated to me.  If the spot price rises above my max while I’m running, I get “knocked-out” of the contract and my jobs get terminated.

The intent is to allow for low-priority jobs to be dynamically run whenever pricing drops below a user’s threshold, but the (intended?) consequence is that it adds the delicious and malleable tang of path dependency to these instruments…

secondary markets, FIX, arbitrage..?

Amazon currently controls the market entirely, but it’s not hard to imagine a secondary market evolving.  Given that others are beginning to copy Amazon’s APIs, one can also imagine markets which operate across providers …  perhaps accessed via FIX?…

Who knows?  In the not-too-distant future, we may well be able to implement ‘core arb‘ strategies…or make markets in cores… or find that we can effectively hedge with disciplined exposure to the ‘core market’ or …

poor Jorge wasn't ready...

In this post I’m going to revisit some of the topics discussed in the recent ‘containing a strategy‘ and ‘multi-strategy trading with regimes‘ posts, focusing on the process of assembling a strategy and its context in preparation for its launch into any of a variety of modes.

I recently realized that – from the perspective of a strategy container – the process of walk-forward testing is remarkably similar to the regime-switching model we’d discussed previously.  Up until now, I’ve employed walk-forward testing in an ad-hoc manner by taking an existing strategy and then writing a little driver very much like a unit-test scaffolding which would walk the strategy forward, permuting parameters based on previous performance.  Not a general solution, but straight-forward as I employ the strategy parameter optimizer from stratbox in this kind of a toolkit use-case.

I sat down to write one of these walk-forward scaffolds yesterday and started to think about how I could generalize the solution and roll it into stratbox’s GUI and it occurred to me that I could likely kill two birds with one stone…

walk-forward testing

–

I imagine there are different ideas/implementations of it, but for me walk-forward testing is the case where you are using repeated historical parameter ‘optimizations’ to dynamically modify the parameters on a back-tested strategy.  For example, say we have strategy X with parameters p1 and p2.  Let’s say we want to walk-forward test it from the beginning of the year until now.  We’ll need a look-back period for optimizations – say 20 trading days – and a frequency with which to apply the results of our optimizations – say every 5 trading days.  So, on a weekly basis we’re going to look back at the prior (rolling) month performance and adjust our parameters to match the best performer.  How you define ‘best’ is your business but I’ll use something simple like sharpe here.

Continuing with our example, we’ll say that p1 will be permuted 3 ways with values from { 1, 2, 3 } and p2 will be permuted across 4 values { “a”, “b”, “c”, “abcd” }.  So we have a basis of 12 strategies for optimization and we’ll call these O1…O12.

To perform a walk-forward test on X from 2009/01/01-now, we will begin a parallel back-test on O1..O12 starting 20 trading days before Jan1.  On Jan2 we will look at the prior ~month’s results and pick the strategy who had the highest sharpe ratio and clone him and start testing him “walking forward.”  Each week, we’ll repeat the process except that instead of cloning the winner as we did the first time, we’ll just make sure his params are the same as the new winner’s params.

Repeat this process until ‘now’ and you’re done – you’ve performed walk-forward testing on X.

–

Let’s think about what’s happening here.  There are 12 strategies running from ~2008/12/1 and another strategy running from 2009/1/2; all strategies run until now.  There’s also the activity that “we” have done during this process.  What we are doing is the role of a meta-strategy just as we’d seen in the regime-switching case.  The walk-forward meta-strategy needs to watch a set of ‘optimization’ strategies and permute another target (“X marks the spot”) strategy on a regular basis to match parameters with the prior period’s winner.

differences vs. allocating to strategies with regimes

Let’s imagine that we’re going to take the 13 lovingly-named strategies O1…O12 and X, along with the newly revealed meta-strategy we’ll dub “M”, and put them onto a live environment instead of a historical back-test.  Suddenly we’re pretty close to what we’d described previously.  The biggest difference is just the temporality of the strategy context: historical or real-time.  Another difference is that O1…On (let’s generalize just a bit) are no longer necessarily all the same strategy – they may be completely different. X is also no longer a strategy, but a portfolio of strategies:  a replicated subset of O1…On we’ll call X1..Xm.  “Best” is no longer a unary choice but must be rich enough to allow M to support some kind of an allocation algorithm across the X family.

OK, so there are some significant differences.  But from the perspective of the object model on which all of this can be implemented, the differences may be limited enough to warrant tackling together.

ready to launch

Before looking at the workflow of a strategy launch, let’s consider all of the various modes we support with our design:

• Historical: back-test single strategy
• Historical: parameter-based optimization of strategy (decomposes to many back-tests)
• Historical: walk-forward testing of strategy
• Real-time: SIM – run single strategy with real-time market data and an exchange simulator
• Real-time: LIVE – run single strategy with real-time market data and a live connection to OMS/broker/exchange
• Real-time: LIVE+SIM – run two cloned instances using the same market data, but with one routing orders to live venue(s) and the other to a simulator.  This is mostly useful for the development of a simulator or to evaluate the quality of simulation for a given strategy.
• Real-time: optimization – run many strategies, permuted differently across live data but routing to a simulator.  I don’t know what this is useful for, but I find it entertaining… The screenshots from the regimes post were illustrating this mode.
• Real-time: meta-strategy – our mythic regime-switching allocator will be a nice example of this mode.

-

Consider the following sorta-flowchart illustrating the preparation for launch of a strategy.

click for larger version...

The process has 4 or 5 stages depending on whether we’re being “meta” or not – that is, if we’re using optimization of any sort.  The highlighted parallelograms are the concrete end-result(s) for each of their respective steps.

• Assemble Strategy Context – The basic channels of I/O for a trading strategy are its sources for market data and the exchanges where trades can be executed.  The MktDataSrc and ExecutionPlatform abstractions need to be specified.
• Assemble Strategy – Whatever mode we’re looking to employ will require a strategy either to itself go and trade for us or else to serve as the seed or prototype for our activities.  In this step we define that strategy.  For me, this can be a composite process as a strategy is made up of an arbitrary number of stratparts.  The assembly of a strategy can be done manually with a gui or by loading a strategy from a file or other store.  Once the parts are identified, they can be bound together to form a strategy instance.
• Configure Strategy – Once assembled, the strategy needs to be configured.  This may just mean we need to parameterize it appropriately for our use.  If we’re being meta and require optimization then we’ll also need to describe the parameters to modify and the ranges for each.  This gets boiled-down into an OptimizationDescriptor which is composed of a prototype strategy and the parameter/range data required to perform the requisite back-tests.
• Configure Optimization or Meta-strategy – If we’re not running an optimization or utilizing a meta-strategy, this step is skipped.  Otherwise, this is where the optimization and/or meta-strategy should be built and configured.
• Review & Launch – Anytime you might loose a piece of software on an account with real money in it, it behooves you to pause and reflect for a moment.  Now is that moment.  If, after that moment of reflection, both you and whatever risk measures you employ agree that it’s OK, then it’s time to launch.  Happily, this is easy because we now have everything we need.  Again, the meta cases are a bit more involved as the strategies to be optimized or managed must be first generated based on the optimization descriptor we’ve inherited from earlier steps.  Then it’s a matter of assembling the pieces previously selected and configured and away we go.

-

Anytime you might loose a piece of software on an account with real money in it, it behooves you to pause and reflect for a moment.  Now is that moment.

I was taught that writing software is easy if you can express the problem you’re looking to solve and its solution in clear and precise English.  I’m not sure about this being grammatical, clear or precise, but hopefully it’s enough to illustrate a means of uniformly modeling and implementing these beasties across a variety of use-cases.

For me, at least, writing this description is helpful as it forces me to walk through the various twists and turns in some detail before digging into the code.  When I get the integrated walk-forward testing working I’ll try to provide some illustration of its use and will look-ahead to developing a regime-switching example using the models/mechanisms we’ve been discussing.

you, hf-trading

There seems to be a developing meme out there suggesting that algorithmic-, and in particular high-frequency, trading is some kind of gold-rush route to easy money which brings to mind…

…this revision of a paper I’d read previously: “Statistical Arbitrage in the US Equities Market” by Avellaneda and Lee.   It’s a detailed and thoroughly worked (and now re-worked) paper illustrating the development and analysis of a US equity stat-arb strategy based on Principal Component Analysis (PCA) and then revised to use ETFs.

I came across this paper as I have still never used PCA in any of my own strategy development work and read Carol Alexander’s excellent Market Models over my summer vacation with an eye towards giving a PCA hedging model a spin in the near-term. Thus, I wanted another look at this paper as a reference point.  Although it’s an excellent paper, I’m not going to urge you to go out and read it immediately unless you have a reasonably pressing practical interest.  Instead, I find it interesting largely because of one of its authors – Professor Avellaneda – and its conclusions in the form of its strategies’ performance.

I’ve seen Prof Avellaneda speak a number of times at a variety of quant meetups organized by the relevant Columbia/NYU financial engineering depts.  His paper reminds me that at least once during my noisome adolescent years, my father intoned darkly that:

the streets are littered with brilliant minds

Implying that any wits I may believe myself to possess wouldn’t by themselves be worth much in life and that I’d need to bring actual tools to the task of solving problems if I wanted to address interesting ones.  Having seen Mr Avellaneda speak, I’m confident that at my peak, my “processor” was never as fast as his.  Much worse, there is no comparison between the tools he can level at a problem compared to me – he’s on an entirely different playing field so far as concrete mathematical/analytical capabilities go.  That’s why I go see him speak and read his papers.

Thus, the results of Avellaneda and Lee’s work are particularly interesting to me as they’re really pretty dull: something like a Sharpe of .9 and degrading briskly.  Now, you don’t expect people to be providing detailed recipes to wildly profitable strategies, and this result isn’t bad, particularly given that they’re describing strategies which likely have significant capacity.  Still, it illustrates that very smart people working with sophisticated mathematical tools even over extended periods are still operating under noteworthy constraints.  Perhaps also: ideas are relatively easy – examining them in the requisite detail is difficult and time consuming, even for (particularly for?) people with the most finely honed toolsets…

I frequently have friends or colleagues who will observe that if you “just write a strategy that foos when bar but yaddas when baz… you should surely make money.”  Maybe.  But the reality is that just putting together the strategy and working through it takes significant time for anything but the simplest strategies.  Once you add genuine complexity to a strategy, you can spend enormous time tuning it.

This, in turn, poses a dilemma I encounter frequently and honestly don’t have a great answer for:

how to find the balance between continuing development on a known good strategy and initiating the development and analysis of unrelated and novel strategies?

the back of the envelope as canvas

This next (de-)reference isn’t directly pertinent to algo trading, but the lessons learned by building BIG distributed systems can surely be applied elsewhere.  And they’re just plain fascinating.

Google’s Jeff Dean gave a recent talk entitled “Designs, Lessons and Advice from Building Large Distributed Systems” at the Large Scale Distributed Systems and Middleware (somehow “LADIS”) workshop and the slides are here.   Go read them.

If bald exhortation doesn’t convince you maybe slide 24 will:

or perhaps what he does with these baseline numbers in slide 27 will pique your interest:

back of the envelope as art form

One that made me (a serial prototype-builder) cringe:

Important skill: ability to estimate performance of a system design
– without actually having to build it!

Ouch.

maybe it is easy after all

Of course, if you’ve studied Avellaneda & Lee’s paper and it held no challenges or surprises and you’ve reviewed Mr Dean’s presentation and it’s old hat to you, too…

Mmmm... hardware..

I’ve never been a hardware guy. Hardware has gotten so fast throughout my professional life that it has just never been a big issue. Also, on wall st we had a robust and annual budget for h/w so I’d routinely sign-off on hundreds of thousands of dollars on all sorts of machines I’d never lay eyes on and somehow they always did the trick.

Before 9/11, they’d be in server racks in the building or down the street, but since then they might also be in increasingly far-flung places like weehawken or long island, tampa, even texas or beyond. The machines always seemed unbelievably overpriced – I remember over the years pretty consistently paying something like $40K for a low-end db server.  But that’s what it cost and you could only purchase approved products from approved channels, so nobody spent much thought on it.  Now that I don’t have the same kinds of constraints – or budgets! – I increasingly have to think of hardware.

As a software engineer, the hardware itself is also insisting that I pay some uncharacteristic attention to it.  The evolution of processors has reached a point where the programming paradigms many of us have fruitfully employed over many years are no longer suited for getting full performance out of today’s machines.  The recent introduction of remarkably powerful and inexpensive parallel-computing platforms based on GPUs like nvidia’s cuda also outline a future that even current university training doesn’t address in a fashion practically adapted for institutional application.  Cores are multiplying like Tribbles.

The lines between persistent storage and main memory are also blurring as consumer SSDs push up from the ‘low’-end while exotic ioDrives and the like offer a glimpse of a world where the performance gap between the two approaches nil and after their long reign myriad metallic platters will spin no more.

troubling like Tribbles

There have been some steps taken towards taming the core dilemma.  Google’s introduction of the distributed map-reduce paradigm and all of the associated plumbing on top of computers in the ‘cloud’ is probably the boldest and most effective reaction thus far, but it’s not always obvious that you want your stuff running in someone else’s cloud amongst many other natural limitations of this approach.  This is also a solution ‘in the large’ and sometimes you need performant solutions on a different, smaller scale.  Here, the development of functional languages and idioms may be of some help, but there certainly don’t seem to be clear winners yet.

Erlang, Ocaml, Haskell, Scala and others all seem to have very limited impact thus far and all face big challenges before receiving widespread welcome.  Worse, any language can be mangled into expressing things poorly so the languages can only be a meaningful aid to programmers who are able to adjust their mindset for a new world’s computing paradigm.  This likely won’t be easy for many until there is an established set of usable programming idioms and toolsets for dealing with concurrency on a whole new scale.  To me, it seems that functional programming might well be an important part of that, but it’s difficult to imagine it as the complete game changer any time soon.  As it is, many of us have already been using functional languages (in my case, sql and R) on a regular or even daily basis for a long time, so it’s difficult to cast this as the revolution, bottled.

As cores proliferate and the bandwidth amongst them increases, new challenges and opportunities are unveiled.  Feeding all of those hungry cores can be a chore.  If you already had problems with i/o bound processes, then adding cores is sort of like adding liquidity to a debt crisis – not obviously helpful. We faced an issue like this recently while trying to improve the throughput of our backtesting subsystem in particularly poor-performing cases.

In these cases, we use daily data to initialize an intraday strategy.  For example, we might look at all equities over a trailing 3 or 5-year period, and perform various analytics – like calculating correlation matrices, beta against various benchmarks, volatility, etc – to determine which names might be candidates for the strategy’s portfolio.  We found that an inordinate amount of our time was spent just performing these morning analytics and that the cost of a day’s backtest was significantly spent on this oft-repeated morning exercise.  While working through this issue, we noted that we’ve reached a point where a decent, dual nehalem, server (with 16 logical cores) could be built with 48G of RAM for something along the lines of $6K.  Seriously.  So we stuck all of our daily data into memory and the improvement has been essentially infinite.  Maybe not the best example, but hopefully illustrative of the fact that the h/w underneath us is changing qualitatively and that we need to be more active in involving it in our design decisions.

Of course, even a yawning expanse like 48G or even a terabyte (in not too many years, after all, I should be able to buy 1T of RAM for my boxes at a similar price point to what I pay for 48G today), eventually gets consumed and so we can’t hope to employ this solution for all our problems.  We continue to develop our historical TAQ infrastructure (most recently discussed here) and this is certainly an example where buying memory isn’t going to get you very far.  But SSDs now are getting reasonable and so our current approach uses memory as much as possible but when it needs to get volumes of detailed historical data we’ve placed our indices onto SSDs while the actual stores themselves reside on much more affordable RAID arrays.  Reading indices is now much faster and adds minimal i/o overhead to a very i/o-bound problem.

By constantly looking at new languages and concurrency idioms, vigilantly assessing the current and projected costs of ’solid-state’-solutions to our most vexing problems, and just staying active and creative, we hold out some hope that we can transition gracefully to what is looking increasingly like a solid-state future.

This isn’t surprising as I wanted the system to support all of the activities associated with our algorithmic trading.  It turns out that that’s a lot to ask of a system.  It also turns out that you learn as you go and so the system continues to evolve.  A few years ago I’d posted about the basics of a strategy container and in this post I’m going to come back to this topic and describe some of the layers of code and thought developed since then.

First, let’s consider the role of a strategy container.  Its job is to intermediate between trading strategies and the external environments with which they interact.  It must also provide services that strategies can use (e.g., position management) and that it wouldn’t make sense for each strategy to re-implement.  In the past I’ve focused on the former responsibility of adapting strategies to external environments.  Why is this necessary and interesting?  Because it allows us to take the same exact strategy and run it live, or in simulation or in backtest, etc.  Interesting and necessary, but not what I want to focus on this time.  Instead, I want to look at the services provided to strategies; the ‘ecosystem’ a strategy container provides in the hope that strategies might flourish within it.

We’ll go from the bottom up.  At the bottom you have one or more pipes of market data coming in.  You might also have news feeds, weather feeds or other such things coming in and we’re just going to conflate them all and say “market data.”  Still at the bottom, you also have one or more two-way pipes with your oms or broker or directly with exchanges or with an exchange simulator.  Within Stratbox we call this abstraction an “ExecutionPlatform” because it’s conceptually where trades get executed.  The lingua franca for execution platforms is FIX, so the baseline object model of a strategy container is likely going to look a lot like messages described by the FIX spec.  Here we have orders, executions and the like.  Within Stratbox we’ve implemented an exchange simulator, a QuickFIX-based FIX interface and a couple broker-specific APIs and they are all of type ExecutionPlatform.  To a strategy, they all look the same.  Likewise with market data.  Within the strategy container, we provide a publish-subscribe model wherein any kind of market data can be subscribed to regardless of the ultimate source of the data.  The intention is always to intermediate between the strategy and its external environment.

the baseline object model

Great.  But now I want to write a trading strategy.  What’s that going to look like?   And this is where things get increasingly interesting as we get to decide what kind of facilities we’re going to provide.  We also need to manage concurrency in some fashion as trading is an intrinsically asynchronous activity.  For this lowest level of strategy, we’ll say that each market feed is handled by a thread which timestamps and enqueues a “Quote” for later consumption and redistribution to relevant subscribers by a MarketDataService within the container.  Likewise, each execution platform will be serviced by a thread who similarly enqueus “Execution” objects from exchanges real and simulated.  The threads handling executions should likely have a higher priority than the threads handling market data.  So, the strategy is naturally exposed to the markets’ asynchronicity right off the bat.  What baseline facilities will our strategy have at its disposal?  Well, it needs to be able to manage orders and positions, so it needs some kind of a “blotter” facility.  With this in place, a strategy can safely assume that it doesn’t need to listen to each execution coming in just to have a correct picture of its book.  It needs access to a MarketDataSvc and it likely also needs access to historical data.  But, staying simple, that’s more or less all we have to provide and we’re going to provide all of these things to the strategy through a single handle: a strategy context.  By swapping out a strategy’s context, we can move the strategy among environments (eg, from simulation to live execution).

With these facilities in place, basic algorithms can be implemented within a strategy and you should be able to test and trade them.

But you’re dealing with very low-level stuff.  You’ll also find that you’re writing the same kind of code over and over.  This inspired the introduction of a layer on top of the baseline object model which I’ve described before: StratParts – a strategy component model.

StratParts – a component model for strategies

With StratParts, we introduced metadata and composition (in the “composite pattern” sense) to our Strategy object model.  A strategy is itself a stratpart which contains other stratparts.  Thus, stratparts introduce a hierarchical structure to a strategy.  Each stratpart publishes a metadata descriptor which is aggregated at the strategy level.  The descriptor contains all of the modifiable characteristics of the strategy and can be changed while a strategy is executing thus giving a sort of grey box capability wherein a trader (or another strategy etc) can modify the behavior of a strategy as it’s running.  Stratparts also create an effective means of providing scoping within your strategy environment.  Stratparts can “see” the activity of other stratparts within the same strategy – they’re all peers in this sense.  This scoping can also be used for allocation of resources across different elements of a strategy.  For example, one stratpart might be allocated 80% of the cash available to the strategy while another stratpart manages the remaining 20%.

Stratparts have proved to be very powerful and we’ve written many of them though we’ll sometimes write strategies as one monolithic stratpart where it makes sense.

Stratparts have a lot of uses but one thing they don’t do is help manage the low-level complexity inherent in trading activities.  This was best described by a trader with whom I’d collaborated.  He’s a reservist in the IDF and uses military metaphors like I use sports metaphors.  He bemoaned the low-level handling of orders required for a very close-to-the-market strategy we were working on and said that:

we need an order like a ’smart’ missile: fire it and forget it

He was right and that inspired the next level of abstraction/support that we built into the system.

’smart’ orders and the tradeflow stratpart

The strategy we were developing was meant to look at a universe of futures spreads and generate all of the ‘cycles’ that might result in an arbitrage opportunity.  Since these are rare at best, we were really looking for circumstances where it looked like we had an advantage based on depth-of-market and various heuristics we’d apply.  Among the functionality that he wanted was an ‘order’ which would act as a limit unless some conditions obtained in which case it might pay the spread or otherwise stop being a fixed limit order.  He didn’t want to handle this inside the strategy but instead wanted to fire these things off and forget about them unless they required attention (eg, if they’re rejected by the exchange or his firm’s risk management checks).  Although we ended up calling these smart orders, they’re really a sort of very localized execution strategy themselves.

To support them, we utilized the same workflow (= state machine) framework we’d implemented for our ExchangeSimulator as we wanted to ensure that these smart orders had a very well-defined life-cycle with a clear set of states and guarantees about transitions among them.  We embedded this workflow engine into a stratpart which supported smart orders operating within this “tradeflow”.  Thus, the extra functionality (and weight/expense) of this functionality is only achieved/incurred when actually required or desired.  This has proven to be a powerful addition to the system and raises the level of services enjoyed by prospective strategies substantially, particularly given the variety of orders we’ve implemented which stand ready for use by any new strategy that might want to employ them.

At this point, it should be clear that we’re blurring the lines between the strategies themselves and the container.  Another way of thinking of it is that one container can have a set of pre-made building blocks that can be applied for families of strategies.  Each one provides a particular grammar with which a strategy’s aims can be expressed.  For very close-to-the-market strategies where very low-level handling of orders is required, the tradeflow stratpart provides a set of really useful facilities, but if I’m just trading with limits then it might not be so interesting.

portfolio-oriented strategies and beyond

I’ve mentioned before that I only care to think of strategies that operate on a portfolio.  As such, we’ve written a good deal of facilities for creating, analyzing and manipulating portfolios within strategies.  By using these facilities within a tradeflow stratpart, I can express quite complex strategies relatively simply by using portfolio analytics to determine my current ‘model’ portfolio and then using smart orders to most effectively transition me from my current state to my model state.  For the kind of trading I do, this provides a rich ‘vocabulary’ for the development of strategies.  All the same, my brain is having a tip-of-the-tongue moment as I feel that there’s an entirely different model possible for portfolio-oriented strategies just at the periphery of my imagination…  oh well, there’s always something new to explore.

Different trading styles, perspectives or trading problems are undoubtedly better served by different sorts of facilities.  I’m finding that just as language constrains and shapes thought, the strategy container that you employ shapes and constrains the kinds of strategies that you can implement.

If you have any ideas about such facilities for any kind of trading you do, I’d love to hear them.

–

a note about Stratbox

Although I’m talking about our system in this blog, we’re not marketing the system and happily “just” use it for our own trading activities.  We had looked into marketing the system previously but ultimately feel the same about selling stratbox as selling strategies…

There’s been a lot of attention focused on  trading battlebots recently.  It’s important to keep in mind that this is part of a long-standing, broad and arguably inexorable trend that is now spreading rapidly away from its successful base in industrial manufacturing to every other conceivable field from scheduling and logistics, to CAD and on to more aggressive pursuits like trading and battlefield operations.  Perhaps looking at the state of the art in related fields can inform us about the direction of our algo bots.

This article in Foreign Policy illustrates an area where automation is making great strides into historically human undertakings.  The use of so-called drone aircraft for recon and tactical missile strikes has reached a remarkable milestone: this year, the US Air Force will train more “pilots” for unmanned aircraft than for real fighters or bombers.  Evidently there’s good reason for this change:

By 2013, software and communications improvements will allow the Air Force’s unmanned-aircraft pilots to simultaneously fly three drones at one time, and four in an emergency. Another factor supporting the likely proliferation of drones such as the Predator, Reaper, and Global Hawk is their low cost compared with new manned aircraft such as the F-35 Joint Strike Fighter.

According to the Government Accountability Office, $24.5 million will purchase a set of four MQ-9 Reaper hunter-killer drones plus a ground station and satellite relay. (See page 117 of this report.) The latest guess of the price for a single F-35 fighter-bomber is $100 million. (See page 93.) This gap in cost led Defense Secretary Robert Gates to demand the cancellation of the manned F-22 Raptor program in order to fund the purchase of more drones for service in Afghanistan and Iraq.

So, for the price of one F-35, I can get a 16-strong phalanx of drones and the required ground-based support to deploy them.  For the price of one F-22, I could get over 20.  Sounds like a deal.  But, they’re not really interchangeable parts as the F-35 or Raptor and other fighter-bombers can overcome “air defense threats” while the drones aren’t quite there yet.  Thus, they’re really only good against opponents that don’t have an air force or anti-aircraft capabilities.  In schoolyard parlance: they’re only good for defenseless opposition.  But we’re buying lots of them, so we must have found nails for our hammer.

While developmentally they are likely in different places, there’s probably an analogy to be made between the current state of development between military drones and their algo trading equivalents.

If today’s algos are like today’s drones in that they’re relatively dumb and can’t overcome defensive threats, what might tomorrow’s algos look like?  Who will they be able to target that they can’t currently?

I’ve been saving the above image in a stubbed-out blog post I’ve wanted to write since a conversation I’d had in Jerusalem last fall.  The recent attention to high frequency trading and all of its attendant evils has reminded me that the topic is relevant and so I relate various thoughts at the risk of jumping on a cacophonous bandwagon of rumbling misinformation.

First of all, the conversation.  It was with a talented guy who acted as the CFO for a variety of companies including a small startup hedge fund which traded US equities at a high frequency.   Although he was a part-time cfo, he seemed pretty plugged-into their trading operations and noted that they use an agency-only brokerage service for automated traders I’m familiar with and that they were “looking at full data for many” hundred stocks concurrently. He remarked that their trading was going well but that their hit rate was something like 4% and dropping.  By hit rate, he meant that they were placing limits frequently and generally pulling the orders if they didn’t get hit immediately.  He didn’t specify, but I imagine that “immediately” might range from milliseconds out to a second or twenty.  If the market is composed of makers and takers, then these guys were definitely makers of liquidity in the strict sense that they were placing limits and making markets.

At the time I thought it was interesting because it seemed that so many people were focused on the very, very short term trade that the frequency was becoming saturated.  It looked like a reminder that trading frequencies populate a spectrum; in this case, this part of the spectrum was becoming so saturated that returns were becoming increasingly difficult to obtain as more players crowded into it.  I’m not sure how this hedge fund has fared, but at the time I remember thinking that they were going to have a tough time competing if they were only geared for high-frequency trading as the space becomes increasingly expensive to play in as the inevitable talent and technology arms race marches on.

Lo and Khandani provide the below image illustrating this phenomenon happening to a class of contrarian strategies Lo & MacKinlay had described in 1990.  The strategies stop working as people squeeze out the alpha.

My conversation in Jerusalem mostly made me think that we were seeing a similar phenomenon amongst HF strategies.

What does it mean for a strategy to be high-frequency?  First of all, it’s a large class of strategies which probably shouldn’t be treated uniformly.  What they have in common is an intention to trade in and out of positions on a frequent basis where frequent will range from sub-seconds out to perhaps several seconds or even minutes in particularly felicitous cases.

Aside from the fact that one can trade at various frequencies, one can mix them and one might even be only peripherally aware of doing so.  A long-only, fundamentally-driven mutual fund (ie, not an algo or high-frequency trader) might call/fax/email/ftp/fix etc their trades into their brokers who might then execute the trades with their in-house or outsourced/white-labeled execution-quality algos.  Those algos might use some very clever close-to-the-market analytics to provide great execution for the client.  Or they might be traded profitably against.  Or both.

In any case, to me it seems clear that there is nothing intrinsically wrong about high-frequency trading itself.  People will always try to react to information as quickly as possible.  Why wouldn’t or shouldn’t they?  They also like to be clever.  Again, why wouldn’t this be expected and good?  I remember Lefevre recounting the use of personal teletypes by big speculators at the turn of the (prior) century.  Not your everyday household item at the time.  I also remember him recounting strategies for moving large positions which involved both buying and selling to hide one’s hand.  Why wouldn’t algos do the same and more?

That the loudest critics have been old-style execution traders “talking their book” to me tells the story here.

One other thing that my conversation evinces is the kinds of biz models being employed by brokers.  Like I said, this little hf hedge fund used an agency-only brokerage that caters to algo/hf traders.  Why is it important to note that it’s “agency only”?  Because, as the Goldman/Aleynikov story illustrates, strategies are organizationally porous in the sense that their value drips away as the human capital behind them moves from organization to organization and understanding of the strat’s internals become understood more broadly outside the organization.  This is likely the dynamic that drove Lo’s example above – more and more traders were employing similar strategies as the knowledge of the strategy leaked further and further from its original source(s).  Likewise, if I see all of your trading activity in sufficient detail, I might be able to reverse-engineer your strategy and work to steal your alpha.

Thus, traders are happy not to advertise their strategies’ behaviors.  So an agency-only broker – a broker who doesn’t engage in prop trading themselves – should inspire trust in a potential client.

Great!  What an honest business model!  But there’s an irony here and it points to the real kinds of problems we should be bugging our regulators to be addressing.  The specific agency-only broker I mention wasn’t self-clearing.  That is, they were using another broker (Goldman in this case) to handle their backoffice duties.  And guess what?  Goldman is anything but agency only.  So, clients who feel warm and fuzzy that they are dealing with an agency-only shop are actually exposing all of their activities with a particularly sophisticated and arguably predatorial prop trader.  It’s like the Guiness ads.

It’s funny to me that while there sits a multi-trillion dollar hole in our fed’s balance sheet which such hippies as fox news and bloomberg (suing to find out) and our US congress (asking politely) can’t seem to tease an explanation for out of the relevant authorities, the blogosphere and regulators seem to focus their invective on short sellers, hedge funds and high-frequency traders.

secret meeting at 85 broad?