Different tests answer different questions
Backtesting applies specified rules to historical data under a declared execution model. Forward testing applies frozen rules to observations arriving after the test starts, often in simulation. Neither is a guarantee, and neither becomes useful merely because a spreadsheet ends in profit.
| Stage | Input | Useful question | Main limitation |
|---|---|---|---|
| Historical development | Data used to design rules. | Can the idea be defined and modeled? | Selection can fit noise and hindsight. |
| Historical validation | A separate interval used to compare candidates. | Do candidate choices survive another period? | Repeated tuning turns it into development data. |
| Untouched historical test | Data not consulted during selection. | How does the frozen choice behave on reserved observations? | Still a model of fills, liquidity and behavior. |
| Forward simulation | Incoming observations and simulated orders. | Can the rules be recognized and executed as events occur? | Simulation can differ from real queues, fills and pressure. |
| Any later live evaluation | Real orders and account records. | What actually executed, cost and changed? | Real financial risk; not a mandatory step for every reader. |
“Out of sample” describes the relationship between data and model selection, not a magic label. A historical holdout can supply out-of-sample evidence. So can incoming observations, until repeated adjustment starts using them to select the next favorite version.
Write the simulation's contract before running it
Specify the product, universe, feed, session and time zone. Say which information produces a decision and the earliest price at which the resulting order could execute. If a rule depends on a completed candle, an earlier intrabar entry cannot be awarded retrospectively.
Record how missing data, corporate actions or contract rolls are handled where relevant. A bar containing both stop and target does not show which was reached first. Use suitable event data or classify the outcome as unresolved under a predeclared policy rather than always choosing the favorable sequence.
State costs and price construction together. If fills already use executable bid and ask prices, adding the same spread again can double-count it. A mid-price model that omits spread has the opposite problem. Commission, financing and slippage also need explicit treatment appropriate to the product.
The backtesting template should hold those assumptions alongside the result. Your strategy specification should let another reader apply the same rule without guessing.
New market hours or an automated assistant can change those assumptions. The future-of-trading outlook explains how to assess an added session, order authority and a different instrument wrapper before treating them as improvements.
A positive gross example becomes negative after costs
Consider 20 invented trades, each with USD 100 initial planned risk. Twelve gain +1R and eight lose −1R. Assume USD 25 in total round-trip friction per trade. The counts and charges are fictional arithmetic inputs, not market observations or a fee estimate for any broker.
| Item | Calculation | USD | R |
|---|---|---|---|
| Gross winners | 12 × 100 | 1,200.00 | +12.00 |
| Gross losers | 8 × −100 | −800.00 | −8.00 |
| Gross total | 1,200 − 800 | 400.00 | +4.00 |
| Assumed costs | 20 × 25 | 500.00 | 5.00 |
| Net total | 400 − 500 | -100.00 | -1.00 |
The gross average is +0.20R per trade. The cost allowance is 0.25R per trade, leaving −0.05R net. The recorded win frequency is still 60%; it has not rescued the result. In a real model, cost and size may vary, so reconcile each trade rather than automatically applying this fixed amount.
Keep the selection history, including the failures
Use chronological boundaries appropriate to the research question and document them before comparing results. Keep development, candidate validation and final evaluation roles distinct. Declare what happens to indicators and positions spanning a boundary so future observations do not leak into an earlier decision.
Track every version tried: parameters, filters, markets, periods and rejected candidates. A final holdout helps, but does not erase repeated selection elsewhere or automatically solve overfitting. Bailey and colleagues examine why testing many alternatives on finite data can create apparent winners.
After evaluating a final sample, changing the rule means starting a new version. Do not rename the consulted sample “untouched” and use it again as independent proof. Preserve the original record, then design the next test with a clear account of what is already known.
Compare costs, opportunity frequency, exposure, drawdown and duration as well as average return. A rule selected because of one exceptionally favorable trade or period needs that dependence made visible. The drawdown guide explains why one past maximum loss is not a future ceiling.
Forward-test the disagreement between intention and execution
Freeze the active rules and start a timestamped opportunity log. Record not only trades taken, but also eligible trades skipped, ineligible trades taken, missed fills, cancellations and unavailable data. Otherwise the recorded sample quietly changes from the tested process to whatever the operator happened to do.
| Observation | Check before blaming the strategy |
|---|---|
| Different signal | Feed, session, time zone, indicator inputs and version. |
| Worse fill | Order type, trigger, spread, delay and the simulator's fill assumptions. |
| Different result in R | Initial risk, actual size, partial exits and net costs. |
| Skipped opportunity | Eligibility known at the time, operator action and any documented exclusion. |
| Unexpected account loss | Open positions, financing, gaps, order rejection and risk-control behavior. |
Simulation cannot fully reproduce the financial and emotional consequences of live risk. NFA's treatment of hypothetical performance explains important hindsight and risk limitations for its regulated members. It is a useful warning to understand, not a claim that the site or every reader is subject to the same membership rule.
Use the journal workflow and the process-review example to separate execution differences from eligible trades that simply lost. Do not assume a mismatch is always a discipline problem.
Finish with a decision and a record someone can reproduce
STRATEGY VALIDATION — VERSIONED RECORD Strategy / version / date frozen: Universe, instrument, venue and contract: Data source, price fields, adjustments and missing-data policy: Session, timezone and decision frequency: Signal timestamp and earliest executable order: Entry, cancellation, stop, exit and position-size rules: Cost model and whether recorded outcomes are gross or net: Intrabar uncertainty and stop/target event-order policy: Development interval: Validation interval and number of trials: Final untouched test interval: Open positions at interval boundaries: Results and failures of every version: Forward-test start / frozen rules / planned review: Eligible / taken / rejected / missed / unknown opportunities: Intended prices versus actual fills and costs: Net expectancy, exposure, drawdown and duration: Discrepancies requiring investigation: Decision and next test:
A useful conclusion states whether the idea is rejected, still uncertain or ready for another defined evaluation. Include what would change that conclusion and which evidence is missing. A profitable chart is not enough, and moving to live trading is not an obligation created by finishing a worksheet.
If position-size assumptions are inconsistent, check the calculator's formulas against the execution venue before rerunning the test. If the rules cannot be reproduced, return to the specification. More historical bars cannot repair an ambiguous instruction.
Questions worth answering
Is forward testing the same as out-of-sample testing?
Not necessarily. A historical holdout is out-of-sample when it did not influence selection. Forward testing collects incoming observations after the rule is frozen. If you repeatedly tune on either sample, it no longer serves as untouched evidence for the selected rule.
How long should I forward test?
Define the review process before starting. Required evidence depends on frequency, variation, market conditions, costs and execution uncertainty. A fixed number of days or trades does not make every strategy reliable.
Why can a winning backtest lose in simulation?
Possible explanations include selection bias, missing costs, unattainable fills, changed conditions, data differences and execution errors. Reconcile the records instead of assuming either the trader or the strategy must be the only cause.
Can a good test guarantee future returns?
No. Testing can expose weaknesses and constrain a hypothesis; it cannot remove market uncertainty, changing conditions or execution risk. Historical and simulated outcomes are not promises of live results.
Sources and scope
- Bailey et al.: The Probability of Backtest Overfitting
Primary research on repeated model selection and the limitations of simple evaluation approaches.
- NFA Compliance Rule 2-29: Hypothetical Results
Member-specific promotional rules explaining hindsight and the absence of actual financial risk in hypothetical results.
- SEC Investor.gov: Stop, Stop-Limit and Trailing Stop Orders
Order-type and execution-price limitations relevant when specifying fills.
The cost example is reproducible invented arithmetic. No personal account history, tested win rate or future return is claimed. The source material does not endorse the site's example protocols.