Methodology

How Watchdog and Allocate work.

Plain language, but exact. This page describes what the code does today, including the parts that are imperfect and the false alarm rates we measured on our own rule.

Cephic TeamUpdated Nov 8, 202614 min read

Expected range Block resampling Lot size scaling Status rule False alarm rates Drawdown limits What Watchdog does not do Allocate grading Correlation Optimizer Worked example Known limitations
WATCHDOG 1

The expected range comes from your own backtest

Watchdog does not compare your strategy with a market benchmark or with other traders. It takes the closed trades of the backtest you linked to the strategy, replays them in random blocks many times, and asks one question: given this backtest, how much should the last stretch of trades have made?

The band follows your live curve

For live trade t, let m be the smaller of t and 100. The band at trade t is the live equity m trades earlier, plus those percentiles for m trades:

band at trade t = equity m trades ago + percentile of what m trades make in the backtest

Until 100 trades have closed, m equals t, so the band starts at your starting balance and widens as the account trades. From then on it keeps the same width and moves with the curve: each point asks whether the last 100 trades look like what the backtest produces over 100 trades.

Why not a band that fans out from trade 0? Its width grows with the square root of the trade count and it never forgets: a long winning run builds a cushion, and after a few hundred trades an account could give back a large part of its profit and still sit inside the band. The rolling band has the same width at trade 400 as at trade 100, so a profitable past does not hide a losing present.

WATCHDOG 2

Resampling in blocks, not single trades

Shuffling single trades breaks up losing streaks, which makes the band too narrow: a live losing streak that the backtest itself contains would then look like a problem. So Watchdog draws blocks.

Without that shift, a 200 trade backtest's per check false alarm rate at p5 is about 10 % instead of 5 %. A healthy strategy whose real edge is slightly below the backtest's estimate would otherwise drift out of the band purely from that estimation error.

WATCHDOG 3

Lot size scaling between backtest and live

If you backtested at 0.10 lots and trade live at 0.05, the backtest's dollar results are twice as large as anything the live account can produce. When both the live trades and the backtest trades carry a lot size, Watchdog multiplies every backtest trade by:

scale = median live lot size ÷ median backtest lot size
WATCHDOG 4

The status rule

After every closed trade, the live closed trade equity is compared with the band at that trade, which is built from the last 100 trades (all of them until there are 100). There are four outcomes:

A healthy strategy crosses a band far more often than the band's level suggests when it is looked at after every trade. Requiring 12 checks in a row below the 0.5th percentile is what keeps the false alarm rate low. The measured numbers are in the next section.

WATCHDOG 5

How often it raises a false alarm, and how often it catches a real one

We measured the rule on strategies that are healthy by construction, and on strategies that lose their edge, with the simulator in the Watchdog backend (scripts/false_alarm_sim.py, which replays the exact production rule). The setup:

RulePer checkAny, 100 tradesAny, 200 tradesAny, 400 trades
Band from trade 0, p1, 8 in a row (earlier rule)0.8 to 2.6 %1.7 to 7.3 %3.0 to 9.0 %3.7 to 10.3 %
Rolling 100 trade band, p0.5, 12 in a row (in use)0.2 to 1.4 %0.7 to 4.7 %2.3 to 6.0 %3.0 to 10.7 %

The worst case over the first 100 live trades is 4.7 % (a mean reverting strategy with a short backtest), which is why the drift email quotes about 5 %. The rate per check stays flat over time, so a long running healthy strategy can still be flagged once: up to about 11 % over 400 trades. That is why the email calls it possible drift, not proof.

The other side of the trade off is how often a real decline is caught. We took the same strategies, let them trade normally for 150 trades, then took away part of their average profit per trade and counted how many were flagged in the next 250 trades:

What happens at trade 150Earlier ruleRolling band (in use)
The edge is gone (each trade makes 0 on average)10.8 %17.2 %
The edge reverses (each trade loses what it used to make)26.0 %41.5 %

Averages over the same eight cells. A band that follows the curve catches about 1.6 times as many declines at a similar false alarm rate, but a strategy that merely loses its edge is still caught in a minority of cases within 250 trades. A backtest of a few hundred trades only estimates a strategy's edge, and the band allows for that, so a modest decline is hard to tell from ordinary estimation error. Do not read the absence of an alert as proof that a strategy is healthy. The drawdown limits cover the cases where it is not.

WATCHDOG 6

Drawdown limits are a separate, hard rule

Limit alerts have nothing to do with the backtest band. They compare the account with numbers you set, the way a broker or prop firm does.

WATCHDOG 7

What Watchdog does not do

ALLOCATE 1

How Allocate grades a strategy

Each uploaded backtest gets a letter from A+ to F. The score is a weighted mix of three parts, each on a 0 to 100 scale:

score = 0.40 × return factor score + 0.35 × survival score + 0.25 × concentration score

The ingredients follow the common profit concentration approach. The weights and the letter cutoffs are our own design, not a published standard.

ALLOCATE 2

Correlation between strategies

ALLOCATE 3

The optimizer and your drawdown limit

The optimizer looks for the risk per strategy that gives the combined portfolio the highest return factor (net profit divided by max drawdown), while the combined max drawdown stays within the limit you set. It runs entirely in your browser.

WORKED EXAMPLE (ILLUSTRATIVE)

A five trade backtest, step by step

Made up numbers, and far too few trades for a real verdict. This only shows the mechanics. The percentiles in step 4 come from running Watchdog's own envelope function on these five trades.

Illustrative: equity after 4 trades, from a $1,000 start
p1p5p25p50p75p95
$966$978$1,004$1,021$1,037$1,067
LIMITS

Known limitations

Analysis of your own data, not financial advice, no guarantee of future results.