How Watchdog and Allocate work.
Plain language, but exact. This page describes what the code does today, including the parts that are imperfect and the false alarm rates we measured on our own rule.
The expected range comes from your own backtest
Watchdog does not compare your strategy with a market benchmark or with other traders. It takes the closed trades of the backtest you linked to the strategy, replays them in random blocks many times, and asks one question: given this backtest, how much should the last stretch of trades have made?
- Each simulated path draws 100 trades from the backtest, in blocks of consecutive trades (next section), starting from zero. From each path Watchdog reads how much it made after 1 trade, after 2 trades, and so on up to 100.
- Watchdog builds 1,000 paths per check.
- For every number of trades it reads off percentiles of what the paths made: p0.5, p1, p5, p10, p25, p50, p75, p90 and p95. p50 is the middle outcome. p5 means 5 % of the simulated paths made less after that many trades; p0.5 means only 0.5 % did.
- The random generator uses a fixed seed, so the same backtest and the same live trades always give the same band. Because every path is a fixed 100 trades long, the band does not shift when more live trades arrive.
- Only closed trades count. An open position has no final result yet, and the backtest only contains closed trades, so floating P&L is left out of this comparison on purpose.
- Partial closes count as separate exits on both sides: the EA reports one row per exit deal, and an MT5 Strategy Tester report lists one exit deal per partial close too.
The band follows your live curve
For live trade t, let m be the smaller of t and 100. The band at trade t is the live equity m trades earlier, plus those percentiles for m trades:
Until 100 trades have closed, m equals t, so the band starts at your starting balance and widens as the account trades. From then on it keeps the same width and moves with the curve: each point asks whether the last 100 trades look like what the backtest produces over 100 trades.
Why not a band that fans out from trade 0? Its width grows with the square root of the trade count and it never forgets: a long winning run builds a cushion, and after a few hundred trades an account could give back a large part of its profit and still sit inside the band. The rolling band has the same width at trade 400 as at trade 100, so a profitable past does not hide a losing present.
Resampling in blocks, not single trades
Shuffling single trades breaks up losing streaks, which makes the band too narrow: a live losing streak that the backtest itself contains would then look like a problem. So Watchdog draws blocks.
- Per strategy (one EA magic number): the band is built from runs of consecutive backtest trades. The block length is the square root of the backtest's trade count, rounded, and capped at 25 trades and at the 100 trade window. A block that reaches the end of the backtest wraps around to the start. Losing streaks up to the block length survive intact.
- Portfolio (the whole account): the band is built from whole calendar days of the backtest, by UTC date. All trades of a drawn day stay together, in order. Strategies on one account often lose on the same day, and drawing whole days keeps that shared bad day together instead of treating it as independent bad luck.
- Uncertainty about the edge itself: a backtest of a few hundred trades only estimates the average trade. The standard error of that estimate is the standard deviation of trade P&L divided by the square root of the number of trades. Each simulated path gets one random shift, drawn from a normal distribution with that standard error, added to every trade on the path. For a per strategy band the error is taken from block averages when that is larger (it needs at least five blocks); for the portfolio band, from day totals when that is larger.
Without that shift, a 200 trade backtest's per check false alarm rate at p5 is about 10 % instead of 5 %. A healthy strategy whose real edge is slightly below the backtest's estimate would otherwise drift out of the band purely from that estimation error.
Lot size scaling between backtest and live
If you backtested at 0.10 lots and trade live at 0.05, the backtest's dollar results are twice as large as anything the live account can produce. When both the live trades and the backtest trades carry a lot size, Watchdog multiplies every backtest trade by:
- Per strategy, this uses that strategy's own trades on both sides.
- For the portfolio band, each backtest trade uses the ratio of its own magic number. A trade whose strategy has no ratio keeps a scale of 1.0.
- If either side reports no lot size, nothing is rescaled. The median is used rather than the average, so a few unusual trades barely move it.
The status rule
After every closed trade, the live closed trade equity is compared with the band at that trade, which is built from the last 100 trades (all of them until there are 100). There are four outcomes:
- Not enough trades yet: fewer than 20 closed live trades. No verdict and no drift alert. Where a band can already be drawn, the dashboard shows an early view of where the curve sits so far.
- Inside: at or above the p25 line at the latest trade.
- Watch: below the p25 line at the latest trade. This includes a dip below the alarm line that has not lasted long enough to count.
- Outside: below the p0.5 line, the 0.5th percentile, at each of the last 12 consecutive trade checks, the latest one included, from 20 closed trades on. In plain terms: the last 100 trades made less than all but 1 in 200 of the 100 trade stretches the backtest simulates, and stayed there for 12 trades in a row.
A healthy strategy crosses a band far more often than the band's level suggests when it is looked at after every trade. Requiring 12 checks in a row below the 0.5th percentile is what keeps the false alarm rate low. The measured numbers are in the next section.
- A drift email goes out only when a strategy changes into Outside, at most once every 6 hours. Staying Outside sends nothing more. Recovering resets it, so a later relapse can alert again.
- A scheduled check runs every 10 minutes, so alerts do not need a browser tab open. Emails are only sent if alert emails are switched on in your notification settings.
How often it raises a false alarm, and how often it catches a real one
We measured the rule on strategies that are healthy by construction, and on strategies that lose their edge, with the simulator in the Watchdog backend (scripts/false_alarm_sim.py, which replays the exact production rule). The setup:
- Synthetic mode: a known strategy generates both a fresh backtest and the live run, so the backtest's own estimation error is part of the test.
- Four strategy shapes (balanced, trend, mean reverting, streaky) and backtests of 200 and 1,000 trades.
- 300 live runs per cell, 500 simulated paths, and the production setup: trade block bootstrap with edge uncertainty.
- Each range runs from the best case to the worst case across those shapes and backtest sizes. "Any" is the chance of at least one false Outside on a healthy strategy over that many live trades.
| Rule | Per check | Any, 100 trades | Any, 200 trades | Any, 400 trades |
|---|---|---|---|---|
| Band from trade 0, p1, 8 in a row (earlier rule) | 0.8 to 2.6 % | 1.7 to 7.3 % | 3.0 to 9.0 % | 3.7 to 10.3 % |
| Rolling 100 trade band, p0.5, 12 in a row (in use) | 0.2 to 1.4 % | 0.7 to 4.7 % | 2.3 to 6.0 % | 3.0 to 10.7 % |
The worst case over the first 100 live trades is 4.7 % (a mean reverting strategy with a short backtest), which is why the drift email quotes about 5 %. The rate per check stays flat over time, so a long running healthy strategy can still be flagged once: up to about 11 % over 400 trades. That is why the email calls it possible drift, not proof.
The other side of the trade off is how often a real decline is caught. We took the same strategies, let them trade normally for 150 trades, then took away part of their average profit per trade and counted how many were flagged in the next 250 trades:
| What happens at trade 150 | Earlier rule | Rolling band (in use) |
|---|---|---|
| The edge is gone (each trade makes 0 on average) | 10.8 % | 17.2 % |
| The edge reverses (each trade loses what it used to make) | 26.0 % | 41.5 % |
Averages over the same eight cells. A band that follows the curve catches about 1.6 times as many declines at a similar false alarm rate, but a strategy that merely loses its edge is still caught in a minority of cases within 250 trades. A backtest of a few hundred trades only estimates a strategy's edge, and the band allows for that, so a modest decline is hard to tell from ordinary estimation error. Do not read the absence of an alert as proof that a strategy is healthy. The drawdown limits cover the cases where it is not.
Drawdown limits are a separate, hard rule
Limit alerts have nothing to do with the backtest band. They compare the account with numbers you set, the way a broker or prop firm does.
- They use live equity, floating P&L included, because that is what a broker or prop firm enforces.
- Max drawdown is measured from the starting balance, or from the highest equity ever reported (never lower than the starting balance), depending on the mode you picked.
- Daily drawdown is measured from the account's equity at the start of the day. The day starts at midnight in the reset time zone you chose, or UTC if none is set.
- Alerts fire at 70 %, 90 % and 100 % of each limit. If the drawdown jumps past several tiers at once you get one email, for the highest. Dropping back below a tier re-arms it.
What Watchdog does not do
- It does not trade. The EA is read only: it reports equity, balance, closed trade history and open positions, and never places, changes or closes an order. It never sees your trading password.
- It does not tell you what to do. Outside means the live results are unusual for this backtest, not that you must switch the strategy off.
- It does not guarantee anything. Inside does not mean the strategy will keep working, and Outside does not prove it is broken.
- It does not replace your broker's or prop firm's numbers. Their own calculation of a limit is the one that counts.
How Allocate grades a strategy
Each uploaded backtest gets a letter from A+ to F. The score is a weighted mix of three parts, each on a 0 to 100 scale:
- Return factor score: return factor is net profit divided by maximum drawdown. A return factor of 3 or more scores 100; below that the score is proportional, so 1.5 scores 50.
- Survival score: Allocate removes the best 5 % of winning trades (at least one) and recomputes the return factor. If it is still positive, the score is the trimmed return factor as a share of the original, capped at 100. If the strategy is no longer profitable without them, the score is 0.
- Concentration score: 100 minus the percentage of gross profit that comes from the top 5 % of winning trades.
- A strategy that lost money over the backtest is graded F, and so is one with no measurable drawdown, because no return factor can be computed.
- Cutoffs: A+ from 95, A from 90, A- from 85, B+ from 80, B from 75, B- from 70, C+ from 65, C from 60, C- from 55, D+ from 50, D from 40, F below 40.
- Shown next to the grade but not part of it: top 1 % and top 10 % profit concentration, the Gini coefficient of winning trades, recovery factor, the longest drawdown in days and the largest single day's share of profit.
The ingredients follow the common profit concentration approach. The weights and the letter cutoffs are our own design, not a published standard.
Correlation between strategies
- Pearson correlation of daily dollar P&L. Scaling a strategy up or down does not change it.
- For each pair, only days that both strategies cover (first to last trade) and on which at least one of them traded. A day on which only one traded counts as 0 for the other. Days on which neither traded are skipped, so adding a third strategy never changes an existing pair's number.
- A pair needs at least 30 such shared days. With fewer, Allocate shows n/a (insufficient overlap) instead of a number that would mostly be noise.
- Labels: 0.8 or higher is high overlap (they tend to draw down together); 0.3 up to 0.8 is moderate overlap; from -0.3 up to 0.3 is independent; below -0.3 is a natural hedge.
The optimizer and your drawdown limit
The optimizer looks for the risk per strategy that gives the combined portfolio the highest return factor (net profit divided by max drawdown), while the combined max drawdown stays within the limit you set. It runs entirely in your browser.
- It searches multiples of each strategy's backtest risk. With up to 4 strategies it tries every combination of 0.1, 0.2, 0.3, 0.5, 0.7, 1, 1.5, 2 and 3 times, then refines the best one. With more strategies it runs a line search per strategy between 0.02 and 3 times, up to 6 passes, from 3 fixed and 8 random starting points.
- Walk forward: the search runs on the first 60 %, 70 % and 80 % of the history and is tested on the rest. The recommendation comes from the split that was strong and consistent in both windows: its score is the lower of the two return factors, reduced by the relative gap between them. Only if no split fits the limit does it fall back to a search over the full history.
- For speed, the search itself works on daily totals. The final recommendation is then replayed trade by trade over the full history, and if that replay breaches your limit, every strategy's risk is scaled down by the same factor until it fits.
- Fits the limit means fits it on the backtest history. A future drawdown can be deeper than anything the backtest contains.
A five trade backtest, step by step
Made up numbers, and far too few trades for a real verdict. This only shows the mechanics. The percentiles in step 4 come from running Watchdog's own envelope function on these five trades.
- 1. Lot size. Backtest at 0.10 lots, trade results +40, -20, +30, -10 and +10 dollars. Live at 0.05 lots, so the scale is 0.05 ÷ 0.10 = 0.5 and the backtest becomes +20, -10, +15, -5, +5.
- 2. Blocks. The square root of 5 is about 2.24, rounded to 2. The first 4 trades of a path are two blocks of 2. If the random starts land on trade 4 and trade 5, the blocks are (-5, +5) and (+5, +20), the second one wrapping around to the start. From $1,000 the path reads $995, $1,000, $1,005, $1,025.
- 3. Edge uncertainty. Mean trade $5, standard deviation $12.75, standard error 12.75 ÷ √5 ≈ $5.70. Five trades make only two blocks, too few for the block based error, so this one is used. If the path's random shift is -3, its trades become -8, +2, +2, +17 and the path reads $992, $994, $996, $1,013.
- 4. Percentiles. Repeated 1,000 times (seed 42), the spread of what the first 4 trades make, added to the $1,000 start, is shown in the table below. With fewer than 100 live trades the band is anchored at the starting balance, so this is the band for trade 4.
- 5. Status. Say the live account is at $990 after 4 trades: below p25, above p1. With 20 or more closed trades that would read Watch. With 4 it reads Not enough trades yet: an early view only, no alert.
- 6. Outside. Only if the curve is below the p0.5 line at 12 checks in a row, for example after trades 21 through 32, would the status become Outside. From trade 101 on, each point of the band is anchored on the live equity 100 trades earlier instead of the starting balance.
| p1 | p5 | p25 | p50 | p75 | p95 |
|---|---|---|---|---|---|
| $966 | $978 | $1,004 | $1,021 | $1,037 | $1,067 |
Known limitations
- Backtests are not the future. Every band and every optimizer result is only as good as the backtest behind it. Markets change, and a strategy can stop working in ways no backtest contains.
- Early view under 20 trades. Below 20 closed live trades Watchdog gives no verdict and sends no drift alert. A strategy can lose a lot in 19 trades; the drawdown limits are what cover that period.
- Heavily optimised backtests flag healthy strategies more often. A backtest tuned hard on past data shows a better edge than the strategy really has. The edge uncertainty shift covers ordinary sampling error, not curve fitting, so a strategy trading at its true, smaller edge can drift below the band.
- False alarms never reach zero. Measured at 2.8 % to 6.8 % over the first 100 live trades and up to 10 % over 200. Short backtests of streaky strategies are the worst case.
- Closed trades only. A large open loss shows up in the drawdown limits straight away, but in the band only once it closes.
- Separate deal fees are not counted. If your broker reports fees as their own deal fee, live P&L reads slightly better than it is.
Analysis of your own data, not financial advice, no guarantee of future results.