Backtest Reality Check: Is Your Edge Real or Curve-Fit?

Three mechanical problems inflate almost every backtest before curve fitting even enters the picture. Lookahead bias is the first: the test uses information that was not available at the moment of the trade — a closing price to trigger an entry at the close, a restated earnings figure, an index membership that was announced later. The result is a strategy that trades on knowledge the live version will never have.

Survivorship bias is the second. Screen today's index constituents across ten years of history and you have quietly excluded every company that went to zero, got delisted, or was acquired at a discount. The universe you tested outperformed by construction.

Optimistic fills are the third and the most common. Backtest engines fill at the price you asked for, in full, instantly. Real markets fill you at the touch, partially, after the move you were reacting to has already happened. The tighter the timeframe and the thinner the instrument, the larger that gap becomes — which is why a strategy that dies on slippage in this calculator usually dies in live trading too.

Curve fitting is tuning a strategy until it describes one specific stretch of history rather than a repeatable market behaviour. It rarely feels like cheating while it happens. A rule underperforms, so you add a volatility filter. Results improve. A drawdown appears in one quarter, so you skip the first thirty minutes of the session. Results improve again. Each individual change is defensible, and the equity curve gets smoother with every one.

The rule of thumb is trades per optimized parameter. Every threshold, lookback length, time window, and filter you tuned counts as one parameter, and each one gives the strategy another degree of freedom to bend around the data. Under about 30 trades per parameter, the fit is doing most of the work. Between 30 and 100 you are in a grey zone. Over 100, the result starts to survive on structure rather than tuning.

The check that actually settles it is out-of-sample data: parameters chosen on one period, results measured on a period the tuning never touched.

More than most backtests contain. A win rate measured over 40 trades carries a 95% confidence interval roughly fifteen percentage points wide in each direction — a 60% result is consistent with a true rate in the mid-forties. That interval is often wider than the edge being claimed, which is why small-sample backtests can be simultaneously accurate and useless.

Narrowing the interval to about five percentage points typically takes several hundred trades, and the requirement grows as the win rate approaches 50%, where variance is highest. This is also why intraday strategies reach statistical significance faster than position strategies: not because they work better, but because they generate the sample far more quickly.