Backtesting AI trading systems should answer a practical question: could this strategy have made these decisions using information available at the time, under plausible trading conditions? A rising equity curve is not enough. The test must reproduce the information boundary, model training process and execution constraints the strategy would face.
Two problems deserve particular attention. Data leakage lets unavailable information enter the research process. Overfitting turns historical noise into an apparently profitable rule. A system can suffer from either problem, or both, without producing an obvious error message.
Data Leakage and Overfitting Are Different Problems
Leakage compromises the information used to build or evaluate a model. Fitting a normalizer on the entire dataset, including the test period, lets future observations influence earlier predictions. Feature selection and missing value replacement can create the same problem. Learn these transformations from training data only, then apply them unchanged to the corresponding validation or test data. The scikit-learn guidance on data leakage covers these preprocessing failures and the use of pipelines to reduce them.
Overfitting does not require future information. A researcher might test hundreds of feature combinations, holding periods and entry thresholds, then select whichever performed best. The winner may reflect repeated searching rather than a durable relationship. AI makes that search faster; it does not make the statistical bill disappear.
A clean dataset therefore does not guarantee a trustworthy backtest. Information controls and model selection controls need separate checks.
Reconstruct What Was Knowable at Each Decision
Use point-in-time data: records that preserve what was available at each historical decision timestamp. A financial period ending in March does not make the associated earnings figures available in March. Publication, delivery and processing must occur before the strategy can act.
For a hypothetical earnings strategy, suppose a release appears at 4:05 p.m. A system that reads the release cannot also buy at the regular session’s 4:00 p.m. closing price. That backtest has acquired a time machine, not an execution advantage.
| Leakage route | Practical control |
|---|---|
| Revised financial or economic figures replace earlier releases | Retrieve the version available at the decision timestamp |
| Today’s index members define the historical stock universe | Reconstruct historical membership, including exits and delistings |
| Features use a completed price bar before it closes | Make the feature available only after the bar is complete |
| News carries an original timestamp but contains later edits | Preserve document versions and their availability times |
When combining news, price data and company filings with AI, keep publication time, vendor receipt time and system processing time separate. Define which timestamp governs eligibility before running the test.
Split by Time, Then Check Label Overlap
For a strategy intended to predict future observations, use chronological evaluation. Train on an earlier window and evaluate on a later one, repeating the process through successive periods. The TimeSeriesSplit documentation describes ordered splits and an optional gap between training and testing observations.
A chronological boundary alone is not sufficient. Suppose Monday’s training example has a target calculated from returns through Friday, but validation begins Wednesday. That training label contains outcomes from the validation period.
Remove training examples whose outcome windows cross the evaluation boundary. Set any gap using actual label horizons and information availability, rather than an arbitrary number of rows. For irregular event data, inspect timestamps directly.
Keep model selection inside the training process. Use earlier internal validation windows to choose parameters, then assess the selected procedure on the next untouched window. Fit preprocessing again within each training window. Otherwise, a carefully ordered split can still contain a carelessly fitted transformation.
Count the Experiments, Not Just the Winning Model
Maintain an experiment register covering rejected models, feature sets, prompts, thresholds and trading rules. A final strategy selected from 500 attempts should not be assessed as though it were the only idea tested.
The research paper The Probability of Backtest Overfitting develops a framework for assessing selection risk across strategy configurations and explains why a conventional holdout can be unreliable in investment backtesting. Its practical warning is straightforward: selecting the historical winner changes how convincing that winner’s results are.
Reserve a final chronological holdout for the frozen research procedure. If its results trigger new features or rule changes, treat it as development data from that point onward. Renaming a folder “final_test_v7” does not restore independence.
Before opening that holdout, define acceptance criteria. Compare against a simple baseline, examine performance across separate periods and test nearby parameter values. Prefer a broad region of acceptable results to one spectacular setting surrounded by losses.
Language Models Need an Additional Information Audit
An LLM can receive only historical documents yet still carry later information in its trained parameters. Restricting a retrieval database does not remove knowledge acquired during pretraining. Research on look-ahead bias in GPT sentiment strategies examines this contamination risk when model training and backtest periods overlap.
Record the model version, documented training cutoff, prompts, retrieved documents and generated outputs. Treat historical tests inside an uncertain training window cautiously. Asking a model to “ignore future events” is not an information barrier.
Use prospective evaluation with a frozen model and timestamped inputs as a separate test. Also distinguish an LLM that generates historical signals from one that only helps write backtest software: the former needs a model knowledge audit; both need a code and data audit.
Test Tradable Returns Before Considering Deployment
Evaluate the complete strategy, not prediction accuracy alone. Specify entry timing, position sizing, spreads, commissions, slippage, financing and applicable borrowing costs. Avoid assuming every limit order fills or every trade executes at the displayed midpoint.
Report net returns alongside drawdown, turnover, exposure, trade count and performance by period. Stress execution assumptions and inspect whether a few trades account for most profits. These are research checks, not proof that future conditions will cooperate.
Finish with a reproducible record: dataset versions, availability rules, split boundaries, experiment history, model settings and execution assumptions. Move a frozen version into prospective paper trading, comparing expected decisions with recorded outcomes. Any later deployment should retain human oversight of automated trading systems and predefined review thresholds.
A useful backtest is not the prettiest reconstruction of the past. It is a documented attempt to find out where a trading idea breaks before real capital supplies the answer.