'optimization' or 'search'?

We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain.  Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode.  I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.

From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good.  Doesn’t mean they’re actually good.  In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively.  I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do.  They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.

They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations.  If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…

Anyway, I felt that had to be said…

Now, I’ll open the floor to a quick illustration of the initialization process described previously and an example of historical forward-walking.

At this point, I’m vested in the inquiry we’d started with the regime-switching post and will apportion time each week towards implementing something like what we’d described there.  I expect the steps to be something like:

1. historical fwd-walking which I’ll illustrate today
2. real-time fwd-walking, that is, dynamically adjusting a live strategy’s parameters based on a continuously repeated ranking of a concurrent ‘live optimization’ of the same strategy
3. real-time allocation to a portfolio of strategies based on regime-switching where regimes are defined by the performance of a heterogeneous set of strategies we run concurrently (again ‘live optimization’)
4. peace on earth

I may have some of the steps wrong, but am hopeful.  We’ll adjust as needed.

If interested, please click on the below image/link which will hopefully take you to a screencast.  This is my first-ever attempt to put a screencast in the blog, so please let me know if you have technical difficulties or suggestions on how to improve the experience.  Or if it’s just a big waste of time!

In the screencast I give a quick illustration of a simple backtest, a simple parameter optimization and finally a fwd-walk.  I’ll appreciate your feedback.

>click<

Given the enduring value of return distributions, I found this morning’s post in ft.com/alphaville especially interesting.  They cite a Dresdner study examining the distribution of returns for Goldman Sachs’ prop trading in 2003 and 2008.  Eye opening stuff.

normal

not so much

In a scenario of perfect knowledge, this wouldn’t be true. If you somehow *know* with certainty that crude will go up or that Citigroup will go down, then concentrating all of your resources into a position based on that belief might be reasonable. But knowledge seldom comes in such a neat package (and will frequently be illegal to act upon when it does!).

Instead, knowledge will typically come in more conditional and less certain forms: “commodities tend to rise during periods of FUD [Fear-Uncertainty-Doubt]” or “companies who announce stadium naming rights deals tend to under-perform.” In some cases, perhaps the knowledge on which you’ll base your strategy can be quantified probabilistically.

Depending on the nature and quality of the knowledge or hypothesis that forms the basis for a given strategy, one can adapt one’s portfolio construction/optimization based on customized relationships amongst the potential portfolio constituents. But one doesn’t need to be so fancy to see the concrete benefits of our first axiom. Below I detail a simple strategy I’ve put together to explore the forces involved.

The strategy I’ve built constructs a completely random, equally weighted long-short portfolio every day and liquidates the portfolio at the end of the day. Specifically, everyday the strategy:

1. Looks at all equities in the database that are available to trade. There were a bit more than ~4K equities available on any given day for this test.
2. Constructs a portfolio composed of N equities selected at random from the available equities. The portfolio has a fixed number of dollars available to it ($500K) and thus each position within the portfolio will be of size $500K/N or less. Orders are always scaled down so that we’re trading round lots (ie, qty%100=0) and if we can’t buy 100 shares of a particular name (eg, Berkshire Hathaway) then that money goes unused until the following day.
3. Half of the components in the portfolio are bought and half are sold so that we have a dollar-neutral portfolio.
4. Positions are entered at the open and exited at the close. (For these kinds of position sizes, this turns out to be a reasonable assumption, as we’ve illustrated here.)Â No stops are employed.

That’s it. The strategy only exposes ONE parameter for tweaking: N – how many names to hold in the portfolio. I varied this parameter across the set { 2, 4, 6,…, 12 } and ran each value 100 times across one year of data.

By selecting the portfolio’s components randomly, I’m trying to ensure that the strategies are truly non-predictive and thus should have an expected return of $0 without applying friction. By creating a dollar-neutral portfolio, I’m trying to ensure that we’re not accidentally capturing beta. What I’m looking to illustrate is the relationship between the number of elements comprising the portfolio and the risk-adjusted performance of the strategy.

As the following distributions illustrate, the strategy does indeed seem appropriately non-predictive. The distributions are normal and shifted to the left due to the application of realistic fees & commissions. As I’d warned in my very first series of posts, the random set of strategies did produce some nice-looking outliers, but we’re past being fooled by this pyrite of data-mining bias.

Also in terms of showing volatility’s relation to (even trivial) diversification, the experiment did not disappoint. While there’s essentially no relation between the profitability of a strategy and the portfolio size, there is a strong negative relationship between the volatility of the strategy and the diversification employed. Furthermore, the benefits of diversification happen quickly and soon taper off as we can see when we look at the table and chart comparing the number of elements in our portfolio with the average vol across the hundred relevant strategies.

There’s nothing new about this result. Harry Markowitz had explained these phenomena with infinitely greater rigor over half a century ago! But judging from what is published in popular “trading strategies” books and periodicals and even the baseline capabilities of algorithmic trading platforms, it appears that people persist in trying to beat the market without using what is perhaps their most potent weapon.

There’s an almost good reason for this that’s like the parable of the fellow who loses his key on the street but decides to limit his search to the area illuminated by a nearby lamppost. When asked where he’d lost the key, he indicates that he’d lost it somewhere on the street but was searching under the lamppost because the light was so much better there. Writing a single instrument strategy can be difficult enough on its own and adding instruments can make it substantially more difficult. But that’s also not a good reason to persist in limiting the scope of one’s efforts!

While there’s no claim to originality in this example, I hope that it’s nonetheless illuminating to see that even an intentionally non-predictive strategy can be mechanically improved. And we haven’t even applied all of the lessons of portfolio construction that we might have. Thus, while writing a worthwhile strategy remains difficult, one can see that there are some very well understood baseline tools that can be mechanically applied to any strategy that will yield material improvements.

I’ve been trading an increasing amount at the open and close of the equity markets using market-on-open (MOO) and market-on-close (MOC) order types and have found that the quality of executions varies enormously between the two types and have spent a bit of time analyzing the differences which I share below.

The quick scoop is that MOC orders almost invariably fill at the exchange’s published closing price, while MOOs vary very substantially from the published open price. Below I quantify my findings in a bit greater depth.

I looked at 846 recent MOO and MOC equity trades made over the past two months. Of all of these trades, only one MOC trade didn’t execute at the published close and the price I got was only off by one penny. Across all of the trades, I received the open or close price 55% of the time.

The remaining 45% of the time I varied from the open by an average of +.04% This means that I actually saw a slight price improvement in the average case. That is, if I was shorting then I executed at a price above the listed open and vice-versa for longs. I’ll take it!

In the below chart I plot the trades against their variance, positive or negative, from the listed open or close.

The biggest difference was a whopping 8.56% but at least it went in my favor. The stdev across all of the trades was .87% so we’re not looking at too disperse a grouping.

This data is a bit skewed as the majority of the MOO orders are going short. This is also a pretty limited universe of trades, so I’ll continue to look at the execution quality I’m getting on these order types and will revisit it if I see any interesting changes.

My interpretation is that my broker is making a best-effort to get a fair open price and they’re doing a creditable job of it. The exchanges are doing a nearly perfect job with MOC orders.

What impact does this have on my strategies? I’m not sure yet, but my first blush impression is that it might be worthwhile to try to get some price improvement over the posted open price as a means of both improving the results and extending the capacity of such strategies. It’s a favorable result as it means that strategies which back-test well on open/close data have a pretty good chance of executing well in reality.

A related issue, which I’m still researching, concerns the capacity of such strategies and may be the topic of a future post…

While doing some research on the quality and volume of executions at the open and close of US equity markets, I came across two topical research reports by Celent, a finance consultancy. The first report is a detailed look at execution quality on nasdaq issues while the second addresses the same topic for the nyse. An abstract of the first report can be found here and of the second here. Both are interesting enough on their own, though I’ve yet to acquire the full reports.

Apart the story told by these two graphics, the articles highlight a few further trends. One is that execution quality is dramatically improving across the board in terms of both speed and price; since 2006, NYSE execution speeds have dropped a remarkable 92%. At the same time, the difference in price quality across venues has diverged, such that everyone will get you filled faster and better than before, but some do so with much more consistently good prices.

If I can get a hold of the original reports, I’ll write a more detailed review of their contents, but for now I just include the links, the catchy speed-charts and these few high-points. For shops dealing with smart order routing algorithms, the reports might be well worth purchasing and studying directly.

I mentioned several weeks ago that I’ve been developing and trading a strategy that’s proven to be quite interesting and profitable. In that post, I described how I’d tried to improve the strategy through the use of a dynamic hedge. The results of that crude hedge were quite good, but just as no worthwhile software project is ever really complete, trading strategies demand constant iterative development.

Below I describe some of the steps I’ve taken to incrementally improve this strategy, discarding the relatively expensive hedge I’d developed earlier in favor of a complementary strategy. We see that when you combine two positive and uncorrelated results, you end up with a product that is literally better than the sum of its parts.

The core strategy has remained the same, though the universe of instruments it trades has been both expanded – through the inclusion of a broader set of equities & ETFs – and contracted – through the application of some filters which prevent trading of some of the instruments under various conditions. These change have yielded a few more points of annualized return with minimal impact on volatility.

As I’d mentioned, the core strategy builds a portfolio of shorts which is sold at the open and bought back at the close. One of the problems with the original hedge is that it had a cost – literally. In order to hedge my portfolio, I needed to cut-back the size of my portfolio to accommodate the cost of the hedge. I mitigated this issue to some degree by employing futures instead of a broad market ETF, but this still reduced my usable capital by approximately 10-15%. I also had my money sitting idle overnight which seemed a particularly profligate behavior.

This led to the study I described last time in which I observed the relative out-performance of the broad US equity markets overnight. If I could somehow find a way to capture some of this overnight alpha, I’d be able to both hedge my main strategy and better utilize my capital.

My first effort at such a strategy happily achieves both aims by assembling a long portfolio which is held overnight. In the chart below, I capture the returns of the two strategies independently and combined. The core daytime strategy is denoted as “OpenClose” or OC whilst the night-time strategy is “CountingSheep” or CS. Both assume an initial capitalization of $1M, employ no leverage and do not reinvest returns – each day or night they assemble a portfolio with the same $1M and profits are put aside and don’t generate interest.

The chart is nice and certainly a big improvement over the earlier hedged approach, but the real power of combining these two strategies is revealed in the two tables below. The first characterizes their risk-adjusted performance independently and then when combined. Both have a Sharpe ratio of around 2.0, but when combined they yield a new strategy which is about 25% better on a risk-adjusted basis. These cells are highlighted.

The key to their compatibility is their correlation. Or, actually, their absence of correlation. In the below table, you can see the correlations of their returns to one another and broad market ETFs. The short daytime strategy is, not surprisingly, negatively correlated with the broad market while the long night time strategy is positively correlated. The beauty of their combination lies in the lack of correlation between the two of them (highlighted) – they’re essentially uncorrelated.

I hope this post illustrates a couple of different vectors along which strategies can be evolved; in this case, to better manage risk and utilize capital.

One of the key remaining limitations of this particular strategy is its capacity. Increasing the capacity of a relatively short-term strategy like this one requires optimization of the trade executions which is its own black art but one that plenty of smart people are constantly addressing. Perhaps in a future post I’ll review some of the techniques applied to this problem for another perspective on the iterative development/evolution of trading strategies.

I’ve recently been working-on and trading an equity strategy that has some great characteristics and some interesting challenges. The great characteristics revolve around its profitability, volatility and simplicity. The challenges start with the fact that the strategy generates alpha on the short side – thus, you are intrinsically swimming against the tide and can conceivably be ruined in a hurry. Your broker might also be unable to find inventory to short. Other challenges include the native capacity of the strategy – it’s not fundamentally scalable as a strategy and only a relatively small amount of money could be put against it without incurring increasingly onerous costs and risks. In any case, it’s been a fun strategy to develop as it’s an interesting puzzle and it makes money.

Discussing the strategy recently with a potential client, they observed that such a strategy wouldn’t be acceptable within their environment (apart the capacity issues) as their risk management practices required all strategies to maintain dollar neutrality – for any dollar of x that they used to buy something, they needed to sell a dollar of y. This led to an interesting experiment for me, the results of which I share with you below.

I wrote a simple, dynamic dollar-neutral hedging StratPart which would do as this trader had described: for any amount of money the strategy spent, the dynamic hedger would employ the same notional amount of money on the other side of the market on a specified instrument. Thus, configured to hedge with the SPY, it would buy $100K of SPY for any $100K my strategy sold. I say “notional” because if the StratPart is configured with a future, then it would buy the appropriate number of contracts based on the notional value of the futures position.

This was about the simplest imaginable StratPart and it didn’t take more than an hour to get it working. In spite of the simplicity of implementation, the results were surprising and, honestly, delightful. The below table characterizes the strategy’s performance in a few different configurations: the raw/unhedged strategy, hedged with the SPY, and hedged against ES (s&p emini future).

As I’d mentioned, the unhedged strategy is already quite nice. Adding the SPY hedge has a very nice effect in that you dampen both volatility and profitability, but in a positive ratio such that the resulting risk-adjusted behavior is better than the raw strategy. That said, the cost of the hedge is quite high and this limits the overall profitability quite substantially. The futures hedge addresses this with its lower costs. It maintains the profitability of the raw strategy while mixing-in the excellent volatility-adjusted performance of the hedge. These returns are calculated against historical data from 2003. The returns are *not* being reinvested. The below graph illustrates the NAV for each of these configurations assuming an initial cash position of $1M.

Clearly, this is a nice and cheap improvement to the base strategy. And there’s still clear room for improvement. One of the problems with this hedge is that the strategy itself is essentially uncorrelated with the S&P500. Thus, the hedge does a very suboptimal job of damping the strategy’s native volatility. A more correlated instrument or set of instruments would likely yield better results.

All the same, the results were very easy to implement within StratBox and pretty dramatically illustrate the unsung virtues of a dynamic hedge.

Recently, I went to play with it and to my shock and horror it had become a wretched loser. In fact, an incredible loser. What had happened? I looked throughout my code and couldn’t find changes; this was corroborated by my CVS repository – no changes had been made to the strategy in a long while. Any coder is familiar with the natural entropy of software systems known pejoratively as “code rot” but this seemed an especially extreme case.

To make a long story short(er), I eventually realized that the code hadn’t changed and the data hadn’t changed, but that my configuration defaults had changed – specifically those governing trading commissions and fees for equities. We’d recently done a client project in which the client had a pretty inefficient/expensive trading platform that made up for its cost by virtue of its facility for conveniently handling large numbers of managed accounts. In any case, in developing that client’s trading strategy I had set costs for trading equities out to a penny a share a trade to ensure that we had a system robust enough to withstand the costs inherent in their trading platform.

The difference this made on Dr. Lo’s strategy is pretty remarkable and so I’ll share it with you.

Without friction, my modified version of Dr Lo’s strategy yielded the following (back-tested) results since last May 1st:

Now, it’s a bumpy ride, no doubt, but a potentially rewarding one for those who can handle the volatility. With returns like that, one could conceivably seek to dampen the vol and then lever the strategy. Of course, this is assuming that one can actually execute the strategy profitably.

Assuming that every trade costs $.01 / share totally reverses the results of the strategy:

That looks pretty bad, right? This is what I returned to upon revisiting this strategy. In my effort to determine what had happened, I inverted the strategy to see how it’s inverse behaved. Corroborating observations I’ve made here previously, this didn’t help and so we had the degenerate case where a strategy and its inverse were both terrible losers. It’s a lot less bizarre than intuition might suggest!

Assuming somewhat more realistic friction for a sophisticated hedge fund, we get results more in line with what my recollection had suggested. It’s very possible for a fund to bear costs of somewhat less than 1/10 of a penny per share traded, but in keeping with a conservative approach, I illustrate the returns with costs pegged at 1/8 of a penny per share traded:

It’s a pretty remarkable difference and it illustrates the knife’s edge upon which one must dance to profitably execute an entire class of potentially fruitful quantitative strategies!

I’ve written a good deal about losers and ideas that might not yield the results one’s looking (hoping) for, but I haven’t written too much about life’s winners. This, of course, is absolutely par for the course amongst traders. People aren’t in the habit of giving away trade secrets, leaving sums of money on the sidewalk or revealing their trading strategies. When they do (or claim to) is likely a good time to keep an especially watchful eye on your possessions…

Thus, while I’m certainly not going to reveal any trade secrets or the internals of profitable trading strategies, it makes sense to address the issue of how to know when you’ve got something that’s good as opposed to just lucky or excessively curve-fitted.

We’ve recently been fortunate to partner with an experienced portfolio manager. Like all of these kinds of organizations, they keep their cards close to their chest, so I won’t reveal who they are just yet, but at some point they might be open to exposing their part in the relationship. In any case, they came to us about six months ago with an US equity sector rotation model based on a sweet kernel of an idea for creatively ranking sectors and the names within it. The essential characteristic of the strategy that made it a winner is simple. What it does really well is one of the hardest things in trading: figuring out when to let a trade go. This portfolio-management trend-following strategy has the lucrative ability to let its winners run while dropping its losers.

How can we characterize that a bit less loosely? Consider the below sets of distributions. The top one captures the overall strategy returns for 300 permutations of the strategy while the bottom one captures the returns on individual trades across all 300 instances of the strategy.

The top distribution looks pretty normal, albeit shifted rather smartly rightward of a random strategy (which should be centered somewhere around $0). The more interesting distribution, though, is that at the trade level where we can see that the strategy is letting its winners run while sharply curtailing the losers. It’s worth noting that the strategy doesn’t use stops or targets. If it had stops, say set at -20%, then we would inevitably see a clustering around that value. Instead, we see a natural sloping – but short – tail extending off to the left while on the right, we appreciate a felicitous long tail of profits.

The solidity of the strategy is underscored by the excellent performance it exhibits across all permutations, that is, it is not excessively sensitive to how it’s configured. Clearly, this is not a case where a data-mining exercise has yielded a normal distribution around $0 and we’ve slyly picked a convenient outlier. Although I don’t illustrate it here, we’ve also seen that the strategy behaves well across both different time periods and across different markets.

In its original format, the strategy was a brute which exhibited tremendous historical profitability but was perturbed by gut-wrenching volatility which left it with a sharpe in the .5-.8 range in spite of returns far above the broad market. Not bad, but not for the faint of heart and certainly not applicable in most institutional contexts.

The distributions above are from a more refined model which does away with the notion of sectors, replacing them with categories of global ETFs. The average sharpe amongst this family of strategies is above 1 and the average trade yields a profit of over 2.5% across many time frames and sets of markets. Our continued development on the model has shown it amenable to treatment with some form of 130/30, 120/20, &tc. formulation.

The keys elements which identify this strategy as a winner and not just a pretender are:

• it’s not excessively sensitive to particular parameters
• even the worst permutations of the strategy are winners (they beat relevant benchmarks by volatility-adjusted measures)
• it’s successful across a variety of time periods
• it’s successful for long time periods
• it’s successful across a wide set of markets

In a later post I hope to discuss the detailed process we went through to develop incremental improvements across iterations of the model (and possibly some of the marketing materials we’ve developed for the product), but it’s important to note that data-mining was no part of its development. Instead, we used the analytical and simulation capabilities of our platform to quantitatively identify the underlying nature of the strategy through each iteration. Between iterations, we used old fashioned creativity and – yes – (educated) guesswork to determine further enhancements and refinements we’d try.

Winners come in all different shapes and sizes, but they share important and identifiable core characteristics; I hope this post provides some insights into how you can quantitatively identify those characteristics in your own strategy development efforts.

This past week I had the opportunity to see MIT’s Professor Andrew Lo present his paper “What Happened to the Quants In August 2007?” as part of the seminar series on quantitative finance presented by NYU and Columbia and sponsored by BlackRock and other relevant institutions. If you’re in the NYC area and interested in such things, I recommend attending any lectures which might capture your fancy.

I had read his paper some time back and implemented, within the Puppetmaster environment, the mean-reversion trading strategy he used as a microscope into what transpired last August. I was interested to see him speak as he’s a seminal thinker on hedge funds and quantitative finance, but also because the strategy he described works pretty well and I thought he might hint at various improvements.

I’ve stolen a line from his paper to serve as the title of this post as it captures one of the central dilemmas faced by algorithmic traders.

The quote is:

In the face of the large losses of August 7-8, most of the affected funds – which includes market-neutral, long/short equity, 130/30, and certain long-only funds – would likely have cut their risk prior to Thursday’s open by reducing their exposures or “de-leveraging”, either voluntarily or because they exceeded borrowing and risk limits set by their prime brokers and other creditors. This was both prudent and, unfortunately, disastrous.

My business partner likes to speak of the art and science of algorithmic trading and, while it might make me cringe as a bit touchy-feely, this is a perfect example of where she’s precisely right. In the case Professor Lo describes, these funds may have been compelled to unwind their strategies as they were excessively levered. (Ironically, the funds that were able to hold fast were almost immediately rewarded with record gains which nearly offset the record losses they’d incurred.) But there’s a general problem here for quantitative strategies, namely, what to do when a profitable strategy incurs unusual losses?

While there’s considerable science which can be applied to this question, at the end of the day the decisions one makes or, better – the policies one establishes – rely as much on the “art” the practitioner applies as any quantitative measure which can be objectively administered. Since a significant advantage of programmatic execution is that it takes such discretionary decision-making out of the trading equation, this is an issue of some consequence. While there’s no magical formula that I know of, there are at least two reasonable approaches to the question.

The first is to have a coherent understanding as to why your strategy is profitable, so that when it seems to stop working you can develop and test explanatory hypotheses with an eye towards developing a well-reasoned approach to the issue. Of course, if your strategy is the result of some sort of data-mining effort, than this can be difficult or impossible – yet another argument against a purely data-mining approach to strategy development! In the example Dr. Lo provides, an astute practitioner might be able to deduce that similar strategies had over-committed and been forced to liquidate; patience might be maintained while the world reverted to a (hopefully!) profitable state of equilibrium. While this approach may be workable in some cases, it seems to ask a lot from the practitioner as a day-to-day policy and might cause more problems than it resolves in the heat of the moment.

The second and probably more workable approach is to structurally minimize the problem by treating strategies like financial instruments in their own right and then apply traditional portfolio management techniques to their allocation. With this approach, we might only allow some fraction of our (levered) portfolio to be managed by any particular algorithm and partner it with other strategies which exhibit negative or complementary correlation characteristics. While it’s well-understood that in periods of serious financial dislocation “correlations go to 1″ and thus this approach won’t cure all ills, it provides a sound decision-making foundation for addressing the issue. Dynamic portfolio re-balancing schemes can thus be used to address these cases perhaps in concert with an effort to understand the root causes of the under-performance.

It’s certainly possible that even a sophisticated and automated application of portfolio management to this problem would yield similarly prudent yet disastrous results, but this seems to me to provide the best framework for reasoning about such issues and implementing appropriate preventive policies.