Skip to main content
Back to home

Including the steps that failed

Validation journey

Each milestone has a plain one-line summary. To see the actual test numbers, click "Show validation results" on that milestone.

Where things stand — updated 2026-10-10

In short: no strategy has been shown to have a real edge.

  • The research process's general robustness test: no strategy has passed.
  • The original 5 strategies went through formal statistical validation — and all failed. The 6th, Every Signal Max 3 (formerly research candidate FWD-01), passed when declared alone and passed the holdout, but neither is clean evidence. It was published in EA v1.1.0 on 10 Oct 2026.
  • Holm-adjusted p-value: 0.128 for the 4 Pullback/Every Signal strategies (the released 24/7 version), 0.064 for London Open 7AM. The pre-registered threshold is p < 0.05.
  • The new EA matches the old EA trade-for-trade on MT5 (129/129 and 180/180 trades, 2025) — so no new coding bugs. This does NOT prove the bot is profitable.
  • The original version and Every Signal Max 3 are running on a demo account (play money) to measure real costs.
Building the system (20 Aug → 17 Sep 2026) (9)

Detail from day one, for anyone who wants to read closely. Including the directions we dropped.

  1. 2026-08-20

    Building the research lab

    Starting from zero: writing software to test strategies on past data (backtesting), for gold XAUUSD only.

    Show validation results
    • First commit of the research repo: “Initial commit: xau-strategy-lab quant research/backtest lab”.

    Source: trading-research-lab git log (first commit, 2026-08-20)

  2. 2026-08-21

    Collecting tick data from 3 sources and auditing it

    Downloaded gold tick data from Dukascopy, HistData and Exness, then checked it for gaps and errors before use.

    Show validation results
    • Audited the Dukascopy/HistData pipeline, normalised Exness 2020–2026, documented mid-year source cut-offs.
    • HistData was later excluded from the numbers because it didn't meet completeness requirements.
  3. 2026-08-21

    Building the stress-test suite (L5) — and the first strategy fails

    Wrote 6 robustness tests (higher costs, delayed orders, shuffling, random simulation). Ran them on the original Sonic R strategy right away: it failed.

    Show validation results
    • 6/6 L5 tests done, 107/107 software tests pass.
    • Then applied L5 to ema_trend, breakout, mean_reversion and tuned sonic_r: none passed.

    Source: trading-research-lab git log (2026-08-21) · trading-research-lab/docs/roadmap-quant-trading.md (L5)(not published)

  4. 2026-08-21

    Many ideas in one day: pyramiding, trailing, time filters

    Tried adding to winners (pyramiding), trailing stops, combining strategies… Many looked good on the full data but were uneven when split over time.

    Show validation results
    • The default trailing stop turned out to hurt results; switched off.
    • Walk-forward for pyramiding: uneven across periods; full-range numbers shouldn't be trusted.
    • Found that _run_sim() silently dropped SELL trades and trades after 2 daily losses — a hidden filter that prettified numbers.

    Source: trading-research-lab/docs/roadmap-quant-trading.md (2026-08-21 updates)(not published)

  5. 2026-08-22

    Split into 2 systems: research and live trading

    Research and order execution became two separate codebases that don't use each other's code. Risk rules are written once and shared, so backtests and live trading compute the same way.

    Show validation results
    • Two repos: trading-research-lab (research) and panda-trading-brain (live). They only exchange approved result files and the shared trading-contracts library.

    Source: trading-quant/CLAUDE.md (QĐ-F, QĐ-H)(not published)

  6. 2026-08-27 → 2026-09-06

    One way to build candles, Dukascopy admitted as second source

    All M15/H1/H4 candles are rebuilt from ticks by a single piece of code. Dukascopy CSV data was checked before being allowed as a cross-check source.

    Show validation results
    • RL-6: Canonical Bar Builder + MarketContext.
    • RL-7: dukascopy_csv provider admission, done 2026-09-06.

    Source: trading-research-lab/docs/roadmap-quant-trading.md (RL-6 / RL-7)(not published)

  7. 2026-09-06

    A pre-registered family of 7 hypotheses: 0/7 supported

    Wrote down 7 ideas to test in advance, ran exactly as written. None passed. One filter had a real effect, but only turned a losing strategy into a smaller loser.

    Show validation results
    • Family XAU7_20260905: 0/7 SUPPORTED, all Holm p = 1.0.
    • H05 (squeeze filter): +0.14R/+0.18R improvement but still negative in absolute terms (−0.03R / −0.02R).

    Source: trading-research-lab/docs/roadmap-quant-trading.md (XAU7)(not published)

  8. 2026-09-16

    Test plans must be written down before running

    From here on, every validation must lock its pass/fail criteria before running — so criteria can't be adjusted after seeing results.

    Show validation results
    • Schema study-prereg/1, deterministic candidate derivation, frozen from a committed spec (T3-3).

    Source: trading-research-lab git log (2026-09-16)

  9. 2026-09-17

    Re-measuring the old set and trying R-step exits

    Re-measured the CHAMPION set, then tried moving the stop-loss up in steps as profit grows (2R, 3R…). This is where the Pullback and Every Signal strategies on this page came from.

    Show validation results
    • CHAMPION C1 re-measure + step-R exit candidates A–D; robustness gate added to the research process.

    Source: trading-research-lab git log (2026-09-17)

  1. 2026-10-10

    Candidate FWD-01 published: Every Signal Max 3 (D_V1_CAP3) in EA v1.1.0

    The author chose to publish the candidate that had been kept private, as the EA's 6th strategy. Status unchanged: not validated, still under prospective validation. Every caveat about RL-14 and the holdout still applies.

    Show validation results
    • Rules: exactly Every Signal, capped at 3 open trades instead of 20. InpMode = 8, preset file CLMCA_D_V1_CAP3.set.
    • Regression on Vantage XAUUSD M15, 2025: the 5 existing strategies match v1.0.2 trade for trade (Every Signal: 594 identical deals); the new strategy's public build matches the earlier private build (372 identical deals). Self-test 80/80.

    Source: clmca-ea CHANGELOG.md v1.1.0(not published) · panda-trading-brain/mql5/forward/clmca-cap3-private/README.md(not published)

  2. 2026-09-21

    Every Signal Max 3 goes into an EA: checked against research, then onto demo

    The candidate (then called FWD-01) was added to a separate EA build to run on demo. Before running, the EA's MT5 results were compared with research's trade ledger: under 5% apart, a limit set in advance.

    Show validation results
    • MT5 Strategy Tester (Exness, 24/7): 2018–2023 gave 826 trades vs 815 in research (+1.3%); 2024 → 31 Aug 2026 gave 454 vs 441 (+2.9%). Open trades never exceeded 3. Self-test and independent review both PASS.
    • Continuous MT5 run 2018 → Aug 2026, after swap and commission: 1,282 trades, net +$10,273, max drawdown $3,146, longest losing streak 55 trades. This is an in-sample backtest, not evidence.
    • Prospective validation on demo starts here; not enough data to report yet.

    Source: trading-research-lab/.ai/evidence/champion-rerun-2026-09-17/forward-demo/PROMPT-brain-ea-D_V1_CAP3.md §1–§3(not published) · panda-trading-brain/mql5/forward/clmca-cap3-private/README.md(not published) · data/backtests.json(not published)

  3. 2026-09-21

    Holdout for research candidate FWD-01: SURVIVES — but NOT clean evidence

    The 2024 → Aug 2026 data was opened exactly once. Result: it passed the test. But this period had been seen before the strategy was chosen, so it is not clean out-of-sample evidence. Read the 5 points on the Holdout page before the numbers.

    Show validation results
    • 441 trades, average +0.462R per trade, one-sided p 0.0054. The first run was rejected for a technical bug and not counted; Panda granted one exception and the retry was machine-proven to be the same test.
    • Before the run, the chance of SURVIVES was only about 14–21% if the strategy were exactly as good as in training.
    • FWD-01 is now Every Signal Max 3 (D_V1_CAP3), public since 10 Oct 2026.

    Source: trading-research-lab/artifacts/research/rl16-holdout-confirm/rl16-hc-c541b2cac9868d47/outcome.json(not published)

  4. 2026-09-20

    RL-14: same trades — declare 1 candidate and it passes, declare 5 and it fails

    Research candidate FWD-01 went through formal testing twice, on the same 1,046 trades. Declared as the only candidate it PASSED; declared with the other 4 it FAILED. The declared number of candidates is the only thing that decides the verdict.

    Show validation results
    • K = 1: p = 0.0338 (passes the 0.05 threshold). K = 5: p = 0.1605 (all 5 fail).
    • K = 1 is not an honest count of what was tried: the strategy was picked after reading RL-13's results, against about 130 exploratory configurations.
    • The candidate is Every Signal capped at 3 open trades, split out of the exploration after RL-13. Now Every Signal Max 3 (D_V1_CAP3).

    Source: trading-research-lab/artifacts/research/rl14-d-cap3-k1/rl10-d-2a8db1fca18c3e1a · rl10-d-dbc75b3959f0a8bf · trading-research-lab/.ai/evidence/champion-rerun-2026-09-17/forward-demo/PROMPT-brain-ea-D_V1_CAP3.md §1–§3(not published)

  5. 2026-09-20

    Dropping the session filter (24/7) — also FAILED

    Removing the session filter gives more trades and higher total profit, but formal validation still failed — just as the filtered version did before it. Panda chose to release this 24/7 version anyway, and the figures on this site are for it. Neither version has a basis for trading real money.

    Show validation results
    • Holm-adjusted p-value: 0.128 (threshold p < 0.05). Every Signal also failed a second condition: median quarterly profit factor below 1 (0.997).
    • Average profit per trade barely moved: +0.262R vs +0.271R for the filtered version. The higher total comes mostly from more trades (909 vs 718, +27%).
    • Research notes explicitly: the 24/7 variant was chosen AFTER seeing the numbers on the same data, so the Holm correction doesn't account for that selection.
  6. 2026-09-19

    Formal validation of the other 4 strategies — all 4 FAILED

    Classic Pullback, Near-Touch Pullback, 3-Bar Pullback and Every Signal were formally validated against criteria written down in advance. All 4 failed. Together with London Open 7AM, all 5 strategies have now failed. There's no basis for trading real money; the demo run continues as the only source of new data.

    Show validation results
    • Data: Exness 2016–2023. RL-10 criteria: mean R > 0.05R, weekly block bootstrap, Holm correction, α = 0.05.
    • Holm p = 0.238 for all 4 (needs < 0.05). Trades: 718 · 976 · 860 · 1,136. Mean R (before swap): +0.271 · +0.246 · +0.252 · +0.235.
    • Also found: the earlier exploratory expectations computed ATR the wrong way (EMA instead of Wilder) due to a column mix-up in a script. The EA uses Wilder as designed and needs no change.
  7. 2026-08-21

    Robustness test (L5): no strategy passed

    Strategies in the research library were stress-tested many ways: higher costs, delayed orders, shuffled trade order. None held up, even after parameter tuning.

    Show validation results
    • The gate includes Monte Carlo, bootstrap, spread/slippage/latency stress and price noise.
    • Tested: original sonic_r plus 12 tuned combos, ema_trend, breakout, mean_reversion. Result: none passed.

    Source: trading-research-lab/roadmap-quant-trading.md (lines 36, 137–170)(not published)

  8. 2026-09-08 → 2026-09-12

    Older parameter set (CHAMPION) failed formal validation

    An older strategy family (not the 6 on this page) was formally validated in 3 configurations. All 3 failed.

    Show validation results
    • Pre-registered survival criterion: mean R > 0.05R (one-sided), 10,000 weekly block bootstraps, Holm correction.
    • 3 configurations (pyramid ≤ 8 layers, no pyramid, approx. sonic_default_v1): all DOES_NOT_SURVIVE, Holm p = 0.7835.

    Source: trading-research-lab/roadmap-quant-trading.md ('RL-10 ĐÓNG' section)(not published)

  9. 2026-09-12

    Found that old numbers had been 'prettified' — fixed

    Splitting the data into quarters, the typical result was below break-even. We also found a hidden filter that made numbers look better than they were. It has been removed from every script.

    Show validation results
    • 18-fold quarterly walk-forward: median profit factor 0.93 (below 1.0 means losing); only 8/18 folds net positive.
    • Hidden filter: dropped SHORT trades and trades after 2 daily losses, applied after the run. 17 scripts fixed, 6 deleted.

    Source: trading-research-lab/roadmap-quant-trading.md (line 921)(not published) · trading-quant/CLAUDE.md ('Rủi ro đang mở')(not published)

  10. 2026-09-18

    London Open 7AM: formal validation — FAILED

    Results look profitable, but the Holm-adjusted p-value was 0.064, above the pre-registered 0.05 threshold. It missed the threshold narrowly, but a fail is a fail.

    Show validation results
    • Data: Exness 2016–2023 (32 quarters), 337 trades.
    • Pre-registered: mean R > 0.05R, 10,000 weekly block bootstraps, Holm correction, α = 0.05.
    • Result: DOES_NOT_SURVIVE, Holm p = 0.063994 (needs < 0.05). An independent verifier re-ran it byte-for-byte identical.
  11. 2026-09-18

    Thresholds now scale with gold's volatility

    Gold in 2024–2026 trades far higher than before, so fixed-dollar thresholds no longer made sense. They now scale with recent volatility (ATR). Preliminary checks passed; Every Signal passed only narrowly.

    Show validation results
    • Three thresholds moved to ATR14 multiples: price band width, EMA34 slope, stop-loss buffer.
    • Preliminary checks: cost stress, parameter sensitivity, two half-periods, prop-account Monte Carlo.

    Source: trading-research-lab/.ai/evidence/champion-rerun-2026-09-17/forward-demo/BRIEF-brain-ea-4-modes-v0-v1.md(not published) · robustness-atr/REPORT.md (old numbers: EMA-style ATR, session-filtered version — history only)

  12. 2026-09-18

    Looser 'pullback' rule — Near-Touch and 3-Bar Pullback are born

    Both variants passed preliminary checks. But they were picked after trying about 140 options, so there's a risk we picked a lucky one.

    Show validation results
    • Parameter sensitivity fairly flat for both.
    • Formal validation later (RL-12, 2026-09-19): both FAILED.
  13. 2026-09-19

    New EA matches old EA on MT5

    Re-ran 2025 with the 5-mode EA: every trade matches the earlier 4-mode EA. This only checks for coding bugs; it doesn't prove profitability.

    Show validation results
    • Classic Pullback: 129/129 trades match. Every Signal: 180/180 trades match. MT5 Strategy Tester, 2025.
  14. ongoing

    Demo run to measure real costs

    The original version runs on a demo account (play money) to measure slippage and spread and compare with the backtest. A few months of data can't prove profitability.