The graveyard of systematic investing is full of strategies with brilliant backtests. This final module is about designing research so your strategies have a real chance of working when you deploy them live.
TK
Tanmay Kurtkoti
Founder & CEO, RupeeCase
⏱ 16 min read⟳ Updated 6 May 2026◆ Expert
Every strategy that has ever been backtested looks good in the backtest. That's the first thing to understand. The backtest is the minimum bar | not the evidence that a strategy works. The real evidence is out-of-sample performance: how the strategy does on data it wasn't designed or tuned on.
This module covers the specific practices that separate strategies likely to survive out-of-sample from those that won't. These are not abstract principles | they're actionable design decisions you make when building a strategy.
Why backtests lie: the three sources of bias
Most strategies fail out-of-sample for three reasons that all look like the same problem (overfitting) but have different roots:
In-sample overfitting: Parameter tuning. If you test 50 lookback windows for a momentum signal and pick the one that performs best in-sample, you've captured noise, not signal. The optimal parameter in-sample is unlikely to remain optimal out-of-sample.
Backtest snooping: Using the same historical data to develop and validate a strategy. If you build a strategy, check how it performs on 2010 to 2020 data, tweak it until it looks good, then "validate" on the same 2010 to 2020 data | that's not validation. That's a second in-sample test with a memory.
Survivorship bias: Only analysing stocks that survived the full backtest period. Indian equity databases often contain only current Nifty 500 constituents | stocks that were delisted, merged, or crashed out of the index are missing. This inflates returns by systematically excluding the worst outcomes.
The survivorship bias problem in India: Building a 15-year backtest on the "current" Nifty 500 universe means you're backtesting on 500 companies that have already succeeded. Companies like Unitech, DHFL, JP Associates, and Yes Bank (at their highs) were all Nifty 500 constituents and would have been selected by value or momentum screens | but they're not in any current constituent list. Proper backtests use point-in-time index membership data.
The walk-forward validation framework
The only valid way to test a strategy on historical data is to simulate the actual research and deployment process | develop on a training window, test on a genuinely held-out window, then advance forward in time and repeat.
1
Anchor date split
Define a hard split: everything before the anchor date is training data (for strategy design and parameter selection); everything after is test data (touched only once, for final validation). For Indian markets with 20 years of history, a reasonable split is 2003 to 2016 training, 2017 to present test. Never look at the test data until the strategy is completely specified.
2
Use canonical parameters
For factor signals with academic backing (12-1 momentum, P/B value, ROE quality), use the parameter specifications from published research | not whatever looks best in your training data. If the strategy is sound, the canonical parameters will work. If you need to tune parameters to make it work in-sample, the strategy probably doesn't have genuine edge.
3
Walk-forward test
Advance through the test period month by month. At each step: (a) use only data available up to that date, (b) generate the strategy's signal and portfolio, (c) record the following month's return. Never look ahead. This simulates live deployment with full fidelity.
4
Stress test across sub-periods
Does the strategy work in every 3-year sub-period of the test window, or only in aggregate? A strategy that loses badly in 2020 but wins big in 2021 to 2022 may have aggregate alpha driven by one unusual period. Rolling window analysis (module 4.4) applied to the test period reveals this.
5
Robustness check across parameter variants
Test the strategy with parameters slightly different from your canonical choice. Does a momentum strategy with 11-month lookback work similarly to 12-month? Does 13-month? If the strategy only works for a narrow band of parameters and falls apart outside it, the in-sample result is likely noise.
Economic intuition as the first filter
Before any backtest, there should be an answer to: Why should this signal predict returns? What is the economic mechanism?
Signals with clear economic mechanisms are far more likely to persist out-of-sample than signals discovered purely through data mining:
Momentum works because investors underreact to fundamental information | news takes time to be fully priced in. This mechanism operates as long as markets have imperfect information processing, which is a structural feature, not a temporary anomaly.
Quality works because high-quality businesses compound capital at above-average rates, and markets systematically undervalue the persistence of quality. This mechanism is tied to the economics of competitive advantage.
A signal discovered by testing 500 financial ratios and picking the one with the highest in-sample Sharpe ratio has no named mechanism. It found noise. It will not work out-of-sample.
~50%
Fraction of strategies with strong in-sample Sharpe ratios (above 1.0) that show statistically significant out-of-sample alpha. The rest are overfitted noise.
3.0+
Minimum t-statistic to treat a new factor as genuine, per Harvey et al. (2016) | accounting for the multiple testing bias in academic factor research.
The practical decision rule: A strategy earns the right to live deployment when it satisfies all three of: (1) clear economic mechanism documented before backtesting, (2) satisfactory walk-forward test on genuinely held-out data, and (3) robustness to parameter variation within a reasonable range. Satisfying one or two out of three is not enough.
How RupeeCase is built around these principles
Every strategy on RupeeCase uses published academic parameters | not parameters selected from in-sample optimization. The backtester uses point-in-time Nifty 500 membership data to avoid survivorship bias. Walk-forward testing is the default validation mode, not full-period in-sample backtest. The philosophy: show you fewer, more honest results | not more impressive-looking ones. Available at invest.rupeecase.com.
✓ Path 5 Complete
Advanced Quant Methods
5 modules: statistical foundations, time series analysis, ML for alpha, alternative data, and out-of-sample validation. You now have the full toolkit.
A validation method where the strategy is tested sequentially on held-out data, simulating actual live deployment to avoid backtest snooping bias.
Survivorship bias
The error of only including stocks that survived the full backtest period | systematically excluding companies that were delisted, went bankrupt, or fell out of the index.
Backtest snooping
Using the same historical data to both develop and validate a strategy. Creates inflated performance estimates because the strategy has been implicitly tuned to that specific data.
Canonical parameters
Parameter values for factor signals specified in published academic research | used instead of in-sample optimized values to avoid overfitting and improve out-of-sample performance.
Point-in-time data
Historical data that reflects exactly what was known at each historical date | including index membership, financial ratios, and corporate actions as they were at the time, not as they appear today.
TK
A note from the author
Why this matters
The graveyard of Indian quant strategies is filled with backtests that looked spectacular in-sample and collapsed on day one of live trading. Learning to build strategies that survive out-of-sample is the single most important skill I can pass on. This module distils seventeen years of painful lessons into a repeatable process.
Want to put this into practice? RupeeCase is the systematic investing terminal built around everything you're learning here, factor scores, strategy backtests, portfolio construction for Indian markets.
This assessment covers everything in Path 5: Advanced Quant Methods, statistical foundations, time series analysis, machine learning for alpha, alternative data in India, and building strategies that survive out-of-sample.
Questions are drawn from all five modules. You need 21 correct answers out of 30 to pass. You can retry as many times as you like.
🎉
Path Test Passed!
You scored 24/30. Excellent work.
↓ Enter your name and email below to unlock your downloadable certificate.
📅
Not quite, try again
You scored 18/30. You need 21 to pass.
🎓 Almost there, claim your certificate
Enter your name and email to generate your Path 5: Advanced Quant Methods certificate. We’ll also send you a confirmation with a link to the certificate image.
✓ Details recorded. Your certificate is ready below, click Download!
Something went wrong recording your details. Your certificate is still available to download below.
🔒 Your email is only used to send your certificate. We will never share it with third parties or use it for marketing without consent.
🎓 Path 5 Certificate
Your Path 5: Advanced Quant Methods certificate is ready. Download it as a PNG to share on LinkedIn or X.
Research Lab Qualifier
Path 5, Module 5 of 5 done, complete all 5 + path test to unlock
Walk-forward validation refits the model periodically on a rolling training window and tests on the next slice. Stronger than a single train-test split.
Path 5 complete, what’s next
Indian Financial Products
Equity shares, derivatives, mutual funds, fixed income, and everything you need to know about Indian financial products.