Researchers who study backtest overfitting argue that in-sample performance is nearly worthless for judging a strategy — walk-forward and out-of-sample testing are the standard defences.[1]
- In-sample — the data you tuned on; it almost always looks great.
- Out-of-sample — data held back and never touched; the honest scoreboard.
- Walk-forward — repeat the split across rolling windows so an edge must hold in period after period, not just once.
- Significance test — shuffle the returns to see how often pure luck would match your result.
It’s the difference between a student who has already seen the exam answers and one sitting a fresh test. Only the fresh test tells you whether they actually learned anything.
Even a strong out-of-sample result is not a promise — the real future is a bigger, tougher hold-out than any backtest. But a strategy that fails out-of-sample almost certainly won’t work live, which is most of the value: it rules things out.
Finisdom’s Strategy Lab runs every rule through an out-of-sample split, a six-filter robustness gauntlet, and a permutation “skill vs luck” test — and shows plainly how few survive.