Is my EA backtest real or luck?
EA Trade TesterDemo labHow we testFor AI agents
Almost every EA that gets sold or shared has a good-looking backtest. That is not because most EAs work: it is because the backtest that gets shown is the best one out of many. If you try enough settings, symbols or EAs, one of them looks good by chance alone. The question is not "is the curve going up?" but "is it going up by more than chance would give the best of my tries?".
Why can a strategy with no edge still show a great backtest?
Take 100 strategies that have no edge at all: on average each one makes nothing after costs. Backtest all 100 and keep the best. That best one will not show nothing: it will show the luckiest result of 100, and its statistics will look like a real edge. The more you try, the better the best one looks. MT5's optimiser makes this easy: one optimisation run can test thousands of passes in an afternoon.
How does the luck bar decide it?
We measure a backtest by its t-value: the average trade divided by how much that average could move by chance (with trades on the same day grouped together, because they are not independent). With one try, a t-value above 1.96 is the usual bar. With N tries, the best of N strategies with no edge is expected to reach a t-value called t0(N), so a result has to clear max(1.96, t0(N)):
| Versions you tried | The luck bar |
|---|---|
| 1 to 20 | 1.96 |
| 50 | 2.28 |
| 100 | 2.53 |
| 500 | 3.05 |
| 1,000 | 3.26 |
| 5,000 | 3.69 |
The bar grows slowly, but it never stops growing: a t-value that is convincing for one try is ordinary for a thousand.
A worked example
An EA was optimised over 2 symbols, 4 timeframes and 60 settings: 2 × 4 × 60 = 480 tries. The best pass shows 520 trades with an average of +0.09 R a trade and a standard deviation of 1.0 R, so t = 0.09 / (1.0 / √520) = 2.05. That clears 1.96, so it looks "significant". But the luck bar for 480 tries is about 3.04: the best of 480 tries with no edge is expected to look this good. This backtest is not evidence of an edge. It may still be real, but only new data can show it.
What else makes a backtest look better than it is?
- Costs left out or too low: in our own research, 86% of the failed tests whose reasons we listed had costs among the reasons (What our tests taught us).
- Drift: a long-only EA in a rising market makes money without any timing skill. We compare the trades with the same trades entered one day later.
- One good year: a profit that comes from one year is a story about that year. We show every calendar year.
- Hidden risk: a grid or martingale shows a smooth curve until it does not (how to spot one).
What does TrueTest do with your EA?
We read your code for faults, write its rules out as a plain-English rule card for you to check, then backtest our own rebuilt copy on up to 16 years of prices for 8 symbols with recorded spreads, commission and swaps, and again with extra costs. The luck bar counts every version you tell us about and every version you test with us. A backtest can at best be PROMISING: the verdict REAL needs new trades after the rules were frozen (the free forward watch). Details: How we test.
Related pages
- How many backtests? Overfitting explained
- MT5 Strategy Tester report checker
- EA costs: spread, commission and swap
- Luck bar calculator
- EA Trade Tester home: test your EA, bot, indicator or report
Questions people ask
How many tries should I count?
Every setting, symbol, timeframe and EA you tested and threw away, plus the one you kept. If you picked long-only or short-only after seeing the results, count that as a try too. When in doubt, count more: a higher count only makes the test stricter.
Is a t-value of 2 good?
For a single, pre-planned test it is borderline. For the best of 100 tries the bar is about 2.53, so 2 is what luck alone is expected to give.
Can I check the maths without uploading anything?
Yes: the luck bar calculator runs in your browser and sends nothing.
Does clearing the luck bar mean the EA will make money?
No. It means the backtest is unlikely to be pure chance after counting your tries. Costs, risk and the future can still go against it. Not financial advice.