Walk-Forward Backtesting: How to Test a Trading Strategy Without Fooling Yourself

Repository reference: cutebacktests

Abstract

Walk-forward backtesting is the simplest answer to one of the oldest problems in trading research: a strategy can look excellent when it is allowed to learn from the same period that is used to judge it. In this repository, the move toward harsher out-of-sample discipline was not an academic side note. It was one of the main reasons the project stopped describing the opportunity set as broad and started describing it as narrow, selective, and portfolio-oriented.

The clearest positive artifact is c66_opening_compression_option_native_short_balance_dte35_v1, summarized in baseline_summary.json. Its base out-of-sample return was 19.18%, stress-medium was 16.70%, stress-harsh was 15.56%, and all three scenarios held 76 out-of-sample trades. Those numbers matter because they come after the framework became less flattering. Walk-forward backtesting is valuable for exactly that reason. It makes many ideas look worse, but the few that survive become much more interesting.

Question

The practical question is not "should I do some out-of-sample testing?" Every serious researcher will answer yes to that. The real question is what kind of evidence walk-forward backtesting should produce before a strategy is treated as credible.

In this repo, the answer became stricter over time. A strategy was no longer interesting merely because one parameter set had attractive PnL. It had to remain believable when evaluated across folds, across stress scenarios, and under combined diagnostics that were closer to the portfolio object that would eventually be traded. That is a much more demanding standard than a single train-test split.

Method: How Walk-Forward Backtesting Works

Walk-forward backtesting means evaluating a strategy on repeated out-of-sample windows instead of asking one long sample to do all the work. In practice, that usually implies a cycle: fit or select on one historical segment, test on the next unseen segment, roll forward, and then combine the out-of-sample results only after the whole sequence has finished.

The March 8 framework audit in Backtesting Framework Issue Summary makes clear why that sequence has to be paired with correct aggregation. Before the audit, top-level PBO and DSR used the wrong fold granularity, and combined Sharpe and Sortino were built from flattened per-symbol return rows. After the patch, robustness diagnostics were split properly into dashboard and selection scenarios, and combined risk metrics were computed from realized calendar-day PnL rather than from a flattering pseudo-daily series.

That distinction matters because walk-forward backtesting is only as honest as the object being evaluated. If folds are mis-aggregated or the portfolio series is constructed incorrectly, the test will still carry an out-of-sample label while flattering the strategy. A serious walk-forward regime needs two forms of discipline at once. It needs temporal separation between selection and evaluation, and it needs the right statistical object at evaluation time.

Evidence / Results

This repository now offers both a positive and a negative example of why walk-forward discipline matters.

The positive example is c66. In Toward The One Piece Of Sharpe, the repository's strongest deployable candidate is described with:

• base out-of-sample return 19.18%
• stress-medium out-of-sample return 16.70%
• stress-harsh out-of-sample return 15.56%
• 76 out-of-sample trades in all three scenarios

Those figures do not prove perfection, but they do show something rare. The branch stayed positive under harsher assumptions without changing its out-of-sample trade count. That is a much better sign than a single backtest with a higher headline return and unstable sample size.

The negative example is broad ORB. The repo's audit in ORB Framework Audit concluded that the framework itself was becoming sounder while the broad ORB search space remained weak or too sparse. The surviving pocket was narrow: directional ORB, 5-7DTE, 5 minute opening range, and range-stop geometry. The broad 0DTE, 1DTE, and 2-3DTE lanes did not survive as general claims. That is exactly what walk-forward thinking is supposed to do. It should narrow the set of strategies that still deserve attention.

What Worked

What worked was the repo's shift from broad frontier search to narrower out-of-sample credibility. Once the project started treating realism and fold quality as first-class concerns, it became much easier to separate "interesting in-sample behavior" from "a strategy that might deserve a portfolio slot."

This is one reason c66 matters so much in the public narrative. It is not merely the branch with the best surviving number. It is the branch that looked stable enough across out-of-sample stress variants to become the current lead_paper_bot in PAPER_BOTS.md. Walk-forward backtesting did not create that edge. It made it legible.

What Failed

What failed was the hope that one good-looking family could be saved by more sweeps inside the same broad search space. The ORB audit and later roadmap effectively rejected that path. The repo's own summary uses the phrase framework_sound_strategy_mismatch to describe the problem, then pivots toward portfolio assembly rather than toward more broad ORB search. That is a negative result, and it is one of the most valuable ones in the whole project.

Walk-forward discipline also exposed a subtler failure mode: adjacent strategies can share a story without sharing robustness. The repo's compression family is a good example. c66 survived well enough to become the lead paper bot, but the related c52_opening_compression_option_native_balance_v1 remained infeasible and failed pbo_ok plus a local dsr_ok check, as discussed in Episode 6. That is exactly why walk-forward backtesting should be applied to concrete descendants, not to vague family-level narratives.

Takeaway

Walk-forward backtesting is how you force a strategy to keep earning your attention as time moves forward. In this repository, it helped reveal that most broad intraday options ideas weakened under honest pressure, while a very small set of narrower sleeves remained credible enough to justify more work.

If you want the diagnostics layer beneath this topic, How to Avoid Overfitting in Trading Backtests With Walk-Forward Validation and Strategy Robustness Testing: PBO, Deflated Sharpe, and Overlap Filters Explained go deeper into the selection metrics and gates. For the broader simulator question, What Is Realistic Options Backtesting? A Practical Guide for Serious Traders is the right starting point. Join the research log to get the next backtest and failure report.