How many backtests? Overfitting explained

EA Trade Tester Demo lab How we test For AI agents

Published 11 November 2026 · methods v2.5 · by TrueTest

Overfitting means the settings fit the past, not the market. The usual cause is simple: many backtests were run and the best one was kept. Each extra try raises the bar a real edge has to clear. The trouble is that almost nobody counts their tries.

Test my EA with my count See a full example first

Drop your .mq5/.mq4 code, a Python bot or an MT5 report on the home page. The first read takes seconds and needs no sign-up; the lab verdict is free too. Only the detailed technical report is paid (the full PDF report is GBP 49). The price shown is the total price: no VAT is added. TrueTest is not VAT-registered in the UK. We read your code as text and never run it as given; an outside AI service drafts the rule card only if you tick the box (see Security). Not financial advice.

What counts as one try?

Any version you looked at and could have kept: each setting of each input, each symbol, each timeframe, each EA, each filter added after seeing the results, and choosing long-only or short-only after seeing them. Tries multiply: 3 EAs × 4 symbols × 3 timeframes × 50 settings = 1,800 tries.

How does the bar rise with the tries?

Tries	Luck bar (t-value)	What it means
1	1.96	the usual 95% bar of a single, planned test
10	1.96	the bar never goes below 1.96
100	2.53	a typical small optimisation
1,000	3.26	a full MT5 optimisation on one symbol
5,000	3.69	several symbols and timeframes

The bar is t0(N), the t-value the best of N no-edge tries is expected to reach, and never less than 1.96. These are the same numbers our lab uses (from our own gate statistics).

How do MT5 optimisation passes count?

Every pass of an MT5 optimisation is a try, whether you looked at it or not, because the optimiser showed you the best. If you upload an MT5 optimisation result (.xml), we count its passes for you. The "genetic" optimiser tests fewer passes than the full grid, but it searches for the best one on purpose, so count at least the passes it ran.

But my settings are close to each other. Is that really 1,000 tries?

Settings next to each other behave alike, so the number of truly independent tries is lower than the count. Counting every try errs on the safe side. We also run a neighbour map: each numeric setting one step either side. A result that only works at one exact value is fragile, whatever its t-value.

What do we count for you?

The higher of what you tell us and what we count for your email: every version you test with us adds to it, for up to 24 months after your last activity (a coded count with no email address in it).

More guides: MT5 Strategy Tester report checker · Free forex EA tester online · Is my EA martingale or grid? How to tell · How to test a gold (XAUUSD) EA properly · EA costs: spread, commission and swap · MQL5 / MQL4 EA code checker · EA trading at the wrong hour? Server time · Why live results are worse than the backtest · Does my EA repaint or look ahead? · Can a scalping EA make money after costs? · Does my EA make money every year? · Will my EA pass a prop-firm challenge? · What to check before you buy an EA

Questions people ask

Does walk-forward testing remove overfitting?

It helps, but if you run many walk-forward tests and keep the best, the same problem returns one level up. Count those runs too.

Is it cheating to optimise?

No. Optimising is normal; forgetting how many versions you tried is the problem. The luck bar lets you optimise honestly.

What if I do not know how many tries I made?

Make an honest high guess. A higher count only makes the test stricter; it never makes a real edge disappear if the edge is strong.

Where can I try the numbers?

The luck bar calculator works in your browser and sends nothing.