or: scaling with elastic map-reduce
Between a rapidly evolving compute environment in which cores are multiplying like springtime rabbits, and a business domain in which the fecundity of market data is making those same rabbits look downright prudish, we are always looking for ways to scale our efforts. There are three levels at which this can be typically done:
- “below the processor” with things like CUDA or FPGAs,
- “amongst the cores” with things like TBB, Cilk, &tc, or
- “in the cloud” (or grid) with things likes Amazon’s EC2 and its bewildering and fast-growing coterie of associated technologies and products
Today we’ll look at a simple example which we implement on top of Hadoop and then deploy into Amazon’s cloud to get a back-of-the envelope feel for what kind of scaling we might expect to gain – and at what costs – from using this smorgasboard of technologies.
Read more…
People have asked me how I go about implementing a strategy in Stratbox. While I’ve illustrated a good number of strategies running in Stratbox in these pages, I’ve never walked through a non-trivial example from conception, through design, implementation and iteration. Today we’ll go through a reasonably complex example in total (I’ll provide source) detail.
The example I’ve chosen is, I think, very nice because it’s a portfolio-oriented strategy, which is pretty much the only kind I care to explore; it’s also based around the concept of pairs trading, which is something which most can easily relate to; and finally, it’s already public domain and yet almost certainly has some juice in it for those who care to understand it and extend it intelligently.
The example comes from the blog of a company, Palantir which does (something like?) analytical / decision support software for both finance and intelligence-gathering services (quants and spooks – spooky quants?). The specific example is here and is described thusly:
Read more…
'optimization' or 'search'?
We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain. Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode. I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.
From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good. Doesn’t mean they’re actually good. In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively. I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do. They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.
They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations. If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…
Anyway, I felt that had to be said…
Read more…
poor Jorge wasn't ready...
In this post I’m going to revisit some of the topics discussed in the recent ‘containing a strategy‘ and ‘multi-strategy trading with regimes‘ posts, focusing on the process of assembling a strategy and its context in preparation for its launch into any of a variety of modes.
I recently realized that – from the perspective of a strategy container – the process of walk-forward testing is remarkably similar to the regime-switching model we’d discussed previously. Up until now, I’ve employed walk-forward testing in an ad-hoc manner by taking an existing strategy and then writing a little driver very much like a unit-test scaffolding which would walk the strategy forward, permuting parameters based on previous performance. Not a general solution, but straight-forward as I employ the strategy parameter optimizer from stratbox in this kind of a toolkit use-case.
I sat down to write one of these walk-forward scaffolds yesterday and started to think about how I could generalize the solution and roll it into stratbox’s GUI and it occurred to me that I could likely kill two birds with one stone…
rebranding opportunity?
A friend of mine pointed out an article he came across on his bloomberg terminal today which reminded him of a strategy I’d described to him sometime back and which we’ve been trading over the past year or so with good results.
To the great chagrin of some of my partners, I even wrote a few posts about the phenomenon underlying our strategy and its evolution as we capitalized on it. Eventually, they persuaded me to shut up already, but the outline was there for all – including Goldman! – to see.
My first post on the topic, “unsung virtues of a dynamic hedge” published June 4th of last year, was pretty coy and didn’t mention the source of alpha itself but talked about enhancing it with a dynamic hedge.
My next post on the topic, “to dream” was published July 14th of last year and laid out the exploitable discrepancy of the market’s behavior. Interestingly, the data I provided in that posting went back the same amount of time as in Goldman’s piece.
I explicitly wrote one last time about the strategy in “evolution of a strategy” wherein I detailed the process by which we’d been evolving the strategy.
Now, one of the more entertaining things about having a blog is that you get to see who is viewing your content. I’m happy to note that all of the major IBs are represented including a variety of distinct IPs within Goldman.
Now, I’m not accusing them of stealing my ideas or anything untoward like that… but I’ll admit that I am wondering how long it’s going to take them to make similar observations across markets beyond US Equities…
Read on for the Bloomberg article…
Read more…
Inevitably one of the first ideas people have when they start thinking about how to write a trading algorithm turns out to be among the hardest: trading the news. The problems are many and in some cases not so obvious…but the natural appeal of the idea seems universally compelling.
Just after the dot.com craze, a brilliant friend of mine (who had just sold his web consulting startup) decided to write a book. The premise was glorious. A bunch of clever college-age kids formed a startup to predict the stock market. The method they used was to constantly comb the web with ultra-sophisticated algorithms which would run across giant server farms overnight and ultimately generate tomorrow’s headlines. Based on the headlines that their system generated, they would place trades that would take advantage of these predicted events.
Sadly, my friend never went on to complete his book, so I don’t know how it all turned out. (Instead, he went on to start another successful company, this time in the field of robotics.) While he was writing it, I loved getting new drafts as they were filled with clever ideas. But the core idea of predicting headlines and then using those headlines to trade always struck me as especially cute.
For those of us without access to news-predicting algos, writing strategies based on the news is rather less straight forward, though there are a growing variety of products and services aiming to fill the gaps. Today must have been trading-the-news-day as I found a few articles on the topic in my mailbox and even received a cold call from a vendor, Need to Know News, with just such an offering. Below I’ll look at some of these offerings and consider some of the issues involved in writing trading strategies based on the news. Read more…
While Carl Sagan’s famous formulation introduced a generation to the vastness of the cosmos, more recent history suggests that his memorable term might now be more aptly applied to financial extents: our deficits and debts, perhaps, to the economically or politically minded. But for those of us with the markets on our mind, the term has to evoke the enormity of the data we create and must manage every day. We’ve recently been working with the NYSE’s TAQ data in an effort to integrate it into StratBox‘s back-testing and optimization capabilities. And the enormity of the data is really just staggering.
Each day, the NYSE publishes all of the day’s quotes and trades as well as some reference data. Compressed, the data will just about fit onto a DVD. For one day. A DVD. Compressed. It’s really mind-boggling. A year of the stuff, uncompressed, will require over a petabyte of storage. Over 1,125,899,906,842,624 bytes. And that’s just the US Equities markets. You want options data, too? I hope your uncle is named EMC, because just managing the data is going to be a challenge…
Read more…
I’ve been trading an increasing amount at the open and close of the equity markets using market-on-open (MOO) and market-on-close (MOC) order types and have found that the quality of executions varies enormously between the two types and have spent a bit of time analyzing the differences which I share below.
The quick scoop is that MOC orders almost invariably fill at the exchange’s published closing price, while MOOs vary very substantially from the published open price. Below I quantify my findings in a bit greater depth.
Read more…
People have long imagined ways to make money while they slept. Happily, it’s not a pursuit I’m particularly bothered by, but as I develop trading strategies, I do make note of different market behaviors that correlate to the time of day. Or night.
In particular, I’ve been looking at various market-breadth ETFs recently as possible fodder for the little dynamic hedger I’ve described before, and I’ve noticed an interesting behavior among several of them…
Like the majority of traders, they do better when they’re not trading!
That is, they actually display better performance at night than they do during the regular trading day; there’s more profit to be had in their gaps between sessions than there is during trading sessions. Below I quantify this observation more thoroughly…
Read more…
I came across and had to share this excellent vignette by Freeman Dyson on the perils of excess model parameterization…
In desperation I asked Fermi whether he was not impressed by the agreement between our calculated numbers and his measured numbers. He replied, “How many arbitrary parameters did you use for your calculations?†I thought for a moment about our cut-off procedures and said, “Four.†He said, “I remember my friend Johnny von Neumann used to say, with four parameters I can fit an elephant, and with five I can make him wiggle his trunk.â€
