Saturday, September 26, 2009

Reflections on Qualitative Research, Science, and Trading System Development

I recently mentioned lessons that I was learning in developing a trading system. Since 3 AM this morning, I've been taking apart every signal from the past several years and investigating the factors differentiating the successful signals from the unsuccessful ones.

My sense, for what it's worth, is that far too little attention is paid to the value of qualitative research in markets. Qualitative research refers to systematic observation that is designed to *generate* hypotheses, not test or validate them. Qualitative research does not replace quantitative work; its purpose is different.

We can think of qualitative research as theory building research. It is the observation that we perform to develop an understanding of phenomena.

When Darwin collected his notebooks of observations from nature, he organized the information in a way that enabled him to generate an explanatory framework: evolution. That framework not only explained existing observations; it suggested new ones. It is the testing of those fresh observations that forms the backbone of quantitative research.

"There is nothing more practical than a good theory," psychologist Kurt Lewin once observed. His point was that science aims at more than prediction: it seeks explanation. Indeed, it is through understanding that we are able to generate predictions.

A technical presentation of the structure of scientific theories can be found here. One theme that has emerged over the course of philosophy of science is the role of models and analogies in generating theories and explanations. We explain something we don't know well by casting it in terms of what is better known. This process of analogy helps us think about complex, partially understood phenomena in novel ways: the good theory is practical to the extent that it leads us to observations we otherwise would have never made.

As I pour through data, I increasingly realize that equating a "scientific" approach to markets with a predictive one is a limited and limiting perspective. There are many predictive statements generated regarding markets daily; few of them are backed by formal efforts at explanation and understanding. Before we enter the laboratory--like Darwin--we need to fill in our notebooks and view our investigations as ways of generating hypotheses, not conclusions.


Manny said...

I've had some good results doing factor analysis on trading systems. One direction you can go is (in Tradestation), have TS write various factor readouts to a file, each time a signal is generated. Imagine a spreadsheet with the following columns: signal, Date, Time, Factor 1, Factor 2, ...Profit Loss $. Then you could sort the spreadsheet based on profit. You could also run a correlation to see the correlation strength of each factor to profit. You could change the facotrs without changing the signal.

Rudy said...

In other words, grade your set-ups. For example a simple system might be: A for no negatives, B for 1 neg, C - for 2 neg, etc. And better yet, trade only the highest grade set-ups. My 2 cents. Rudy

JimRI said...

Dr. Brett,

Very interesting challenge you have undertaken. My take is that markets are a manifestation of human herd behavior and while patterns can be observed, predicting them in real time comes close to just pure luck. As an engineer, I have never been able to model human behavior mathematically with any reliability. Perhaps computer programs that are playing arbitrage schemes or even running pump & dump schemes if done fast with very sophisticated machines may (and does as GS has shown) yield consistent results.

Toby Crabel in his Opening Range Breakout book, is clear that even when that method was working, it worked better when monitored and prompted by humans.

Any system that works, and is available to more than a few people will likely stop working after a time. As more use it, the people who are loosing to it will change what they are doing and then the patterns in the market will be different from what the program recognizes.

I am thinking about a system that comes up with a strength and direction based on Tick, Tick rate of change, Volume, volume rate of change and price action. That is a challenge with the software I have. This is basically to mimic what I do when I watch the charts and then alert me to upcoming situations. I have not thought about programming it to enter trades automatically.

MikeH said...

Interesting link here on the apprentice like growth of NFL coaches.

Thanks for the tip Manny, I have a signal I'm trying to relate to intermarket movement.

Radek said...

Is there a way to systematize one's qualitative ideas??

Brett Steenbarger, Ph.D. said...

Thanks for the comments and observations. I can't speak for systems overall, and please take my observations with multiple grains of salt, as I'm new to system development.

What I can say with respect to my own system work is that it does appear to be possible to systematize one's qualitative observations, but not necessarily in a linear scheme. If you view certain market conditions as qualitatively different from other ones, backtesting across all days from a lookback period stops making sense. I'll try to post more on this topic--


abel said...

My own work in this area in various market constructs takes the form of quantifying observations that repeat with frequent regularity. Doing this, along with quantifying the specifics, and establishing a baseline, also establishes probable expectancy percentages.

I then take those baseline attributes, and focus on the instances that exhibit the most 'robustness', which I define as possessing both the highest percentage ratios and the highest frequencies of occurences, and then introduce a variable which is designed to improve the overall baseline probabilities.

The variable is directly associated with the model inputs, and is designed to denote whether the market is presently indicating less-than normal strength (market is weaker than it may appear and therefore a sell) or less-than normal weakness (market is stronger than it may appear and therefore a buy).

Modeling price-behavior with these processes has produced frequently-occuring trading setups with very high ratios of profitable outcomes.