sensitivity testing
'optimization' or 'search'?
We’ve been looking at how a strategy container might view and implement a variety of modes for strategies it will launch and contain. Last time I documented a uniform initialization process for many of them, including a posited walk-forward parameter optimization mode. I’ve implemented an initial version of this that I’ll illustrate through a screencast (first ever – be gentle) below, but before continuing want to raise a couple of cautionary notes about the slope we’re traversing here.
From the very first post on this blog I’ve tried to underline the danger that over ‘optimization’ poses in view of the simple unalterable fact that if you look at enough random junk, you are bound to see things that look impossibly good. Doesn’t mean they’re actually good. In the context of trading strategy development, this is a particular danger as strategy parameter optimizers are easy to come by and can be very misleading if employed naively. I think this is in part due to the term ‘optimization’ which is really a stretch for what these tools do. They’re better described as search tools as they are really searching through a tuple-space of possible parameter combinations that you’ve specified, and then ranking them by some criteria you specify.
They’re still useful, but less as ‘optimizers’ and more as tools for judging the sensitivity of the strategy to different parameterizations. If the strategy demonstrates good performance and stability over a variety of market conditions and parameterizations, you may just have found yourself a winner…
Anyway, I felt that had to be said…
Now, I’ll open the floor to a quick illustration of the initialization process described previously and an example of historical forward-walking.
At this point, I’m vested in the inquiry we’d started with the regime-switching post and will apportion time each week towards implementing something like what we’d described there. I expect the steps to be something like:
- historical fwd-walking which I’ll illustrate today
- real-time fwd-walking, that is, dynamically adjusting a live strategy’s parameters based on a continuously repeated ranking of a concurrent ‘live optimization’ of the same strategy
- real-time allocation to a portfolio of strategies based on regime-switching where regimes are defined by the performance of a heterogeneous set of strategies we run concurrently (again ‘live optimization’)
- peace on earth
I may have some of the steps wrong, but am hopeful. We’ll adjust as needed.
If interested, please click on the below image/link which will hopefully take you to a screencast. This is my first-ever attempt to put a screencast in the blog, so please let me know if you have technical difficulties or suggestions on how to improve the experience. Or if it’s just a big waste of time!
In the screencast I give a quick illustration of a simple backtest, a simple parameter optimization and finally a fwd-walk. I’ll appreciate your feedback.
>click<
EMS Internals, back-testing, performance analysis, portfolio management, regime-switching, strategy development
Hey Puppetmaster-
Thanks for the demo, I enjoyed seeing your StratBox go through its paces!
You asked for feedback so I’ll give a little. Please keep in mind that this comes from the perspective of
a) I really admire your work and am educated and entertained by your blog
b) I spent the better part of a decade coaching engineers on giving power demos of algorithmically intense software
c) I respectfully offer a few comments in the spirit of giving back
As far as mechanics go, it worked fine for me with no technical difficulties.
What I think would make it a better experience is hearing more about the meaning and impact of some of the features you showed. For example, in the intro, you said this was about examples of initialization. What jumped to mind was “count = 0;” Happily, the content is really very interesting and even groundbreaking so next video please give us in the audience more early clues to why we care.
I wish we’d lingered a little more on the Fwd-Walk set-up. So you just drop that in with any other strat? Can it FWD-Walk with risk management strats and twiddle their parameters too? Since Fwd-Walk itself is parameterized, can you do Fwd-Walk of FWD-Walk? That is different from Regime Switch right?
Digging insight out of the black window (debug trace?) seemed out of character of the rest of the user experience. Can each strat have the ability to create a custom tab in the results box to write any info it has to share?
Remember, I think your software is great, and you’re really very articulate. There’s not many people that can send me to a dictionary but you are one of them! Your video makes for a fine blog post – you’re getting this feedback and more if you want it because you asked and I happen to take a keen interest in how software is presented.
Cheers,
Jackie
Jackie, thank you very much for your helpful, thorough and gracious comments.
> So you just drop that in with any other strat?
Yes.
> Can it FWD-Walk with risk management strats and twiddle their parameters too?
Yes, exactly. The fwdwalker will permute parameters for any number of stratparts within a strategy.
> Since Fwd-Walk itself is parameterized, can you do Fwd-Walk of FWD-Walk?
I suppose one could but recoil at the thought! ;^>
> That is different from Regime Switch right?
Yes – we’re still a few steps away. Next I’ll try to show a live fwd-walker… hopefully thereafter I’ll be able to implement some kind of a live regime-switching/allocation example.
My choice to show the debugging output was made on-the-fly and surely you’re right that it wasn’t elegant. Logging is very configurable, but I don’t generally present much of it in the gui; exactly as you suggest, strats can indeed write their own gui(s) to tabs in their own stratview but generally when they do this it will be a graph or chart of internal variables it might care about rather than text…
Anyway, thank you for sharing your professionally-honed insights!