Sunday, September 27, 2009

The Limitations of Mechanical Trading Systems

The first two posts in this series described some lessons I was learning while developing a trading system and broader reflections on theory-building in market investigation. In this last post, I'll tell you where an intensive weekend of research has left me and where I plan to go from here.

Please note that my observations are for only the markets and market data that I am observing; I cannot make with confidence judgments about other markets and data. Please also take my observations with many grains of salt, as system development is hardly my area of expertise.

What I can tell you with respect to the market variables I am investigating, which include price change, momentum, and market breadth/strength, is that a proper model for the markets would have to be one that incorporates the notion of state dependence. Over the course of years of daily data, it appears to me that the stock market exists in a finite number of states and that the predictive value of any traditional market indicator is contingent upon the market state at that time.

I suspect this is why market indicators as a whole are not good predictors of future price action, as Aronson's research suggests. What is an accurate predictive indicator in one market state--say, a narrow trading range--is not predictive in another, such as a vigorous trend. When we average performance of trading systems/indicators across a lookback period, we inevitably are averaging days from different market conditions/states. The result is that performance may look smashingly good for a period of time and then inexplicably degrade.

To make this more concrete, suppose I have a patient who has a bipolar (manic-depressive) mood disorder with episodes of psychosis (loss of contact with reality due to delusions or hallucinations). My patient has at least five different states: manic/non-psychotic; manic/psychotic; depressed/non-psychotic; depressed/psychotic; and relatively normal mood/cognition. If I try to predict how my patient will respond to something I say, I have to take into account the state that he is in. A backpat of encouragement would be read one way in a depressed state; another way in a manic one. It would have one meaning in a psychotic state; another meaning in a normal cognitive mode. Any global prediction across all states is apt to be weak. Indeed, a clinician that treated the patient the same across all conditions--a mechanical counseling system--would be hamhanded at best.

And that's the problem with mechanical trading systems with respect to the variables I'm studying. They are hamhanded. They work well when markets behave "normally" and can work surprisingly poorly when market states shift.

That isn't to say that successful systems can't be developed; I just think that such systems would have to factor in state-dependence and market context. In a sense, such a system would really embrace a set of subsystems, at least one for each distinctive market state. Their logic and validation would be non-linear: to test each subsystem, you'd have to go back to only those periods from the past that fit the state definition.

Procedurally, that means not just testing systems on the last X number of years of data. It is probably more akin to conducting nearest-neighbor modeling, where the neighbors are selected from the last X years of data based upon an objective definition of state. A set of tools for conducting such investigations can be found here; a set of common mistakes in these investigations is here.

Before using those tools, it would be helpful to develop an explanatory model that incorporates the notion of states and state dependence, per my recent post on theories and science. Such a model, if useful, would suggest candidate variables/predictors for those modeling tools. That's where I hope to be taking all of this.



MikeH said...

Yes, most ideas in mechanical systems are of the linear thinking type - works great until the environment changes. I think that backtesting a linear idea over short time frames may be preferable, as long as you acknowledge that the system will not live forever. Hey, flipping houses was a profitable idea for about 5 years, even though historical returns in the residential market where somewhere around 2%.

Radek said...

Dr. B, I am a mechanical system trader (Forex: $EURUSD). I tried to include my qualitative ideas into my trading system (which I have fully programmed and tested in Java), but with little success in terms of enhancing performance. Qualitative filters reduce my number of trades (no doubt eliminate some bad ones), but they also take out a large number of good trades.... I will think more deeply of this, do some more testing and let you know.

Thank you for keeping me (us) in the research mode.

BipolarPorch said...

As someone with bipolar disorder, I agree with your analogy.

As far as trading goes, predictive or not, it is bound to fail over the long term due to the raqndomness of the markets. For details check out All the academic research on the subject of performance prediction can be found there.


Visit my Bipolar News Site...

Curtis said...

Well, you might also want to look at very basic processes such as driving a car, image perception, or speech generation. There are a lot of things that are easy for people that are really difficult for computers: especially pattern recognition.

I think mechanical studies are useful though to show all the things that don't work. The problem is that we, as people, are often wrong. Highly trained scientist make mistakes in their research on a constant basis. So, it's not good for the average person.

Let's take a common trader slang, that is that good traders don't predict the market or that it's not required. Why should anyone who can't predict the market expect to make a trade and get better then average (poor) returns?

At some point, you have to make a prediction X is more likely then Y.

Steve said...

IMO it is not nearly as difficult as you make it sound to take state into account. What you are really talking about is market timing, or aligning capital with prevailing market conditions.

Something as simple as the Thrust Trend Model can keep you on the right side of the market trend the vast majority of the time, which is enough to give you a huge edge.

Then when you combine the simple but effective market timing system(TTM) with a predictive model to select outperforming stocks, you compound the effect of two tradeable edges.

Then if you want to reduce volatility or MDD, you can incorporate a simple hedge to harvest your alpha. Done!

BTW, I recently completed a 20 month live test of such a system at my blog.

beeba said...


I totally agree. I just went through a rather crappy summer where a trading system I had used profitably from february to may was basically useless from early june until a couple of weeks ago. i knew that something had changed, and my suspicion was that it was the ever-decreasing volume in the markets over the summer.

speaking as a scientist / software engineer, your idea of combining a given state description to a particular trading strategy and seeing in which states the strategy excels and in which states it fails is good advice.