His final example is particularly valuable: it's possible to run so many tests and so reduce your sample size that you wind up with false certainty.
It's also possible to find results specific to the historical period that you are testing--dangerous if regimes are in the process of shifting.
Used properly, however, the combination of careful observation and testing can yield very fertile hypotheses for traders. Sometimes knowing that there is no significant edge is just as valuable as identifying one. That was helpful in not assuming a bearish edge on Friday, despite seeming overbought conditions.
RELEVANT POST:
Exploring Historical Market Patterns in Excel
.
Exploring Historical Market Patterns in Excel