Wednesday, February 14, 2007

Testing Your Trading Ideas: A Best Practice in Trading

Note: I'm doing the writeup, but credit this idea to professional trader and system developer Henry Carstens. His article linked below is one of the best trading articles I have ever read. If you recall, Henry is the developer who worked with one of the blog readers to develop his trading edge. From that interaction, we learned quite a bit about what it takes to succeed at trading. Most important, we saw that the partnership of programmers and traders is of benefit to both: traders learn which of their ideas work best (and how they can be improved), and developers gain new market insights from experienced traders. For that reason, today's Best Practice is to test your trading ideas, either by learning one of the testing platforms yourself or by partnering with a programmer already familiar with the testing process. At the end of this post, you'll see how Henry's article can help you take steps in these directions.


Traders often think of trades in terms of "setups". These are conditions that have to be met in order to enter the market. A setup typically defines a situation in which demand and supply become sufficiently imbalanced that they lead to directional market movement. A setup can be defined as a configuration in an oscillator, a chart pattern, or an indicator reading.

Of course, much more goes into a trade idea than the initial setup. There is also the issue of money management: how much capital to allocate to the trade, and there is the issue of exiting the setup once entered. Traders also typically address the issue of stop loss points--where to get out when a trade turns out poorly--and they also make assumptions about which trading instruments to utilize in pursuing the setup.

As you can see, there are many different components to a trade. When we review our P/L, we tend to attribute our results to our setups. But that isn't necessarily accurate. Our results are also a function of our money management, our stops, and our exit criteria. Results also vary as a function of *what* we trade, not just how we trade it.

This is a major challenge for the discretionary trader. We might be making money--or just approaching profitability--but we don't really know which parts of our trading are working for us and which aren't. Because we don't know if our setups, exits, stops, etc. are holding us back or contributing to our success, it is difficult to maximize our performance.

That's where testing our trading ideas comes into play. When we test ideas, we go back into recent history and see, first hand, how each component of our trading contributes to P/L. Very often, as was the case with the recent trader, we find that our ideas work for us, but still can be improved. Sometimes we find that what we thought was working really isn't adding to our success. A while back, for instance, I tested my morning setups and found they were working better in the Russell futures than in the S&P 500. As a result, I commonly place trades in both markets, which has improved my profitability.

The prospect of testing trading ideas is daunting for many traders, who might be intimidated by the programming. Current platforms, however, make such testing easier than ever. This article by Henry Carstens is the clearest summary of how to test ideas that I've yet encountered. I strongly recommend it as reading for any trader who wants to better understand what goes into good trading ideas. There are also sections on how to test the significance of trading ideas and how to optimally allocate capital to those ideas. An additional, fantastic feature of the article is the template at the end, written in TradeStation code, than enables you to drop in your ideas and test them out for yourself!

If you've been successful with your ideas but don't want to tackle the programming yourself, consider teaming up with an experienced programmer. To help traders with this challenge, I will collect names and contact information from all experienced traders and experienced programmers who might want to collaborate. Simply email me at the address given at the right side of the TraderFeed homepage (under the section "About Me") and I will do my best to get you connected.

My goal is not just to help traders learn, but to help traders learn from their own experience and from each other. Henry's idea of using testing to sharpen our trading is a great way to move ourselves to the next level of performance. That's why I consider it a "best practice".


NO DooDahs said...

The paper covers very well how to develop and test the idea and the associated money management, but "inventory turnover" has to be accounted for as well. A tradable idea with good expectancy per dollar risked needs to generate enough trades to compound equity at a significantly higher rate than given by an alternative benchmark. A good example might be comparing a market maker with very small expectancy per dollar risked versus a long-term trend-follower with high expectancy per dollar risked, often the market maker will compound equity more quickly because they turn over their equity faster, i.e. they make many, many more trades. Some very good systems generate relatively few trades over time, such as the unilateral pairs trade with QQQQ and SPY.

Another key question left unanswered is how to benchmark the performance of the system once it's been tested and optimized. The inventory turnover is necessary to calculate the statistics needed. Once we have turnover, we can develop some other stats, namely the compounding rate, maximum drawdown, length of maximum drawdown, and expected maximum number of losing trades.

The compounding rate has to be compared to a benchmark, either another system or perhaps "buy and hold" on the issue in question. If a trading system generates 13% annual compounding with a lot of effort, and "buy and hold" gets 10% annually over the testing period with no effort, the trader needs to decide if the effort is worthwhile.

We also need to keep in mind the psychological toughness necessary to trade a system with large drawdowns and/or low winning percentages. A 35% win rate can be dramatically profitable if the $win to $loss ratio is high, but will anyone trade it consistently, or will they give up after the tenth losing trade in a row (a highly possible outcome)?

Trading & Investing Essentials said...

This blog has comments, articles, and other information on trading by John Forman, who wrote The Essentials of Trading. John’s been in the markets for almost 20 years. He publishes his thoughts about trading (mostly educational), provides reviews of books and other trading related stuff, and posts information on his trading education work.

for more details click here:

Brett Steenbarger, Ph.D. said...

Hi NO DooDahs,

Great points; thanks for the feedback and observations. Many of the summary reports produced by the system testing platforms help answer these questions. You're absolutely right that one needs to have a structured, disciplined personality to trade mechanical systems. I think, however, it is possible for discretionary traders to test their ideas, factor the results into their discretion, and improve what they do without necessarily shifting to a mechanical trade. Thanks!


Brett Steenbarger, Ph.D. said...

Hi John,

Thanks for the link. I've enjoyed your book The Essentials of Trading and look forward to the live chat we'll be doing together on the 22nd.


D TradeIdeas said...

Excellent article by Carstens. I'm even in more agreement with 'no doodahs'. I offer the 2x2 decision matrix for evaluating trade ideas that summarizes much of what's been said with one picture.

Brett Steenbarger, Ph.D. said...

Thanks for that link, David--


brywend said...

Hi Henry,
I just wanted to say that I really enjoyed your article and found it very useful. It was especially helpful to have the formulas and the Tradestation code template. Thanks!
Bryan Wendon

bwilhite said...


I also think that some of the metrics you mentioned for discretionary traders would be very applicable to system development. Many of these are already included in reports provided by TradeStation, but I think "Enhancing Trader Performance" provides some good context on interpreting them. There might be other metrics that aren't really provided by TradeStation, but it's been so long since I've used that software that I can't remember.

Not as a counterpoint, but as an addition to No_Doodahs' post. Another thing to consider is time in the market. I would take a 13% return from a system that is only in the market 1/2 the time over a 10% 'system' (buy and hold) that is in the market all the time. I would make that decision all day, provided everything else is ok.


Brett Steenbarger, Ph.D. said...

Hi BW,

Great point about time in the market. It's risk-adjusted returns that matter most!