Why backtests are so easy to make "look great"
A backtest is fundamentally "running the model on historical data and pretending." Any small engineering error can produce absurdly good results — and humans are naturally biased toward believing favorable outcomes (confirmation bias).
A common beginner aha moment: after fixing the bugs, the equity curve drops from "+200% annualized" to "-10% annualized."
Seven most common traps
1. Look-ahead bias
Using information that didn't yet exist at the modeled point in time.
E.g., using "split-adjusted" prices retroactively when those adjustments were only known later.
2. Survivorship bias
The backtest only includes companies still alive today — delisted / bankrupt ones are absent.
Result: returns overstated by several percentage points.
3. Overfitting
The model performs well in training only because it captured noise as signal. Validation collapses.
4. Data dredging / data snooping
Repeated parameter tuning on the same validation set — the validation set loses its "unseen data" property and silently becomes another training set.
5. Slippage / fee omission
Backtest fills at "closing price" — real execution has slippage, commissions, taxes. Low-frequency strategies may lose ~1%; high-frequency strategies can flip from profitable to losing.
6. Liquidity assumption is too generous
Your strategy buys 1,000 shares of a small-cap daily in backtest — but real order book volume is only 2,000 shares; you literally can't get filled, or your impact dominates.
7. Publication bias
The model you actually built and shipped is the "best of 100 tries" — multiple hypothesis testing makes luck look like skill.
Engineer's-style pre-deployment checklist
- Does the dataset include delisted companies?
- Do all features use only information available strictly before time t?
- Are train / validation / test split strictly chronologically?
- Did you simulate realistic slippage and fees?
- Is each order under 1% of daily volume?
- Did you use walk-forward cross-validation?
- Did you report the entire research path, not just the best model?
- Does the model behave reasonably across at least 3 different market regimes (bull, bear, sideways)?
- Did you validate once and only once on a fully independent final test set?
Missing any one of these — the backtest is not credible.
Why "walk-forward" validation is required
Classical cross-validation uses random sampling — which severely leaks future in financial time series.
Correct setup:
T1 T2 T3 T4 T5
[train][test]
[train][test]
[train][test]
[train][test]
Each fold uses only past data to train, future data to test — mirrors real-world execution.
The sim-to-real gap
Even with every trap above avoided, production performance still differs from simulation:
- Your orders themselves affect future prices (market impact)
- HFT latency, network jitter, exchange rate-limits aren't in any backtest
- Real psychology — can you actually hold the algorithm through a -20% drawdown?
Industry rule of thumb: production results are typically 30%–50% lower than backtests. A 20% annualized backtest may produce 10–14% live — which is already very good.
Important questions
Is an 80% win rate a good backtest result?
Depends on payoff ratio. 80% wins of +1% with losses of -10% have negative expected value. Win rate alone is meaningless.
Backtest looks great but live loses money — where's the problem?
Probability-ranked: (1) overfitting (2) data leakage (3) optimistic slippage (4) regime change (5) execution psychology. First two are most common.
Does paper trading solve the problem?
Partially — eliminates real capital risk, but doesn't simulate your own orders' market impact. At least 3 months of paper trading is the minimum pre-deployment requirement.
Quiz
Q1. What is survivorship bias?
A. Your chance of "surviving" B. Backtest sample only includes companies that still exist today, missing delisted/bankrupt ones, inflating returns
C. Related to model survival D. User retention
Q2. For financial time series, you should use:
A. Random cross-validation B. Walk-forward — chronological, always past for training, future for testing
C. K-fold cross-validation D. Leave-one-out
Q3. Industry rule of thumb: production performance is typically lower than backtest by approximately:
A. 0% B. 30%–50% C. The same D. Production is always higher
Reference Answers
Q1: B Q2: B Q3: B
Further reading: Wikipedia: Backtesting · Wikipedia: Overfitting · Wikipedia: Survivorship Bias · Investopedia: Look-Ahead Bias
Educational content only — not investment advice. Backtest results don't guarantee future performance. Validate thoroughly before going live.
Get the pre-trade checklist.
We are turning these guides into a searchable checklist for checking terms, rules and risk before you trade.