# trading-research-lab — Data Pipeline Audit (HistData + Exness + Normalization)

**Date:** 2026-08-24
**Scope:** `data/raw/histdata`, `data/raw/exness`, `data/normalized/source={histdata,exness}`, `src/xau_lab/data/{histdata,exness,normalize,dukascopy,validation}.py`, `scripts/{resumable_histdata_download,ingest_exness_archives,normalize_histdata}.py`, `src/xau_lab/types.py`, `config/default.yaml`.
**Not in scope:** backtest engine correctness, strategy logic, prop-firm risk simulation (see `docs/audit/2026-08-21-source-concordance-and-code-audit.md` for backtest-engine coverage).
**Method:** direct code inspection (every claim below cites a file/line or an on-disk artifact) + a fresh aggregation script reading every `metadata.json` currently on disk across all three sources (5,009 partitions total: 181 Dukascopy + 2,404 HistData + 2,424 Exness). No polars/pyarrow query engine was available in this sandbox (see §8 note); tick-level re-derivation was not independently re-run — findings that rely on the source's own recorded metadata are marked as such, and findings inherited from the prior 2026-08-21 audit are marked "inherited, re-verified against current code" or "inherited, not re-run" as appropriate.

A prior, deeper audit already exists at `docs/audit/2026-08-21-source-concordance-and-code-audit.md` (plus two Exness addenda). This report does not repeat that work; it independently re-verifies the parts still relevant to ingestion/schema/storage/validation/metadata, checks whether anything has changed since 2026-08-21, and adds several findings that prior audit did not cover (notably §6-C).

---

## 1. Executive Summary

The three-source ingestion pipeline (Dukascopy as ground truth, HistData and Exness as candidate secondary sources) is carefully engineered at the mechanical level: atomic writes with fsync, SHA-256 checksums on every raw archive, immutable-raw-data discipline (a checksum mismatch never overwrites, it stops), and a real 6-state interval taxonomy for Dukascopy. That part has not regressed since the 2026-08-21 audit.

However, two of the three sources are explicitly self-flagged as not execution-eligible, and the reasons are still live, unresolved defects — not just conservative caution:

- HistData's timestamp-basis bug (finding **C1** in the prior audit — an unconditional "+5h EST, no DST" conversion that is empirically wrong for roughly 9 months of every year) is **still present in the code today**, verified fresh against the current file (`histdata.py:29`, `:192-197`; only one commit has ever touched this file, and it is the audit commit itself).
- A new finding from this pass (**not** in the prior audit): `DataValidator.validate_range()` — the one function `ExperimentRunner` calls to gate every backtest — hardcodes `DukascopyDownloader` for its interval-completeness check regardless of the `source` argument passed in. Calling it with `source="exness"` (which `configs/champions/champion_v1.yaml` implies is a live use case) checks Exness tick data against **Dukascopy's** raw hourly archives, which is a source/logic mismatch, not a validation of the actual dataset under test.

Data quality at the row level is solid for Dukascopy (33.4M ticks, 0 duplicates, 0 reordering, 0 boundary issues) and workable-but-imperfect for HistData/Exness (2.11% and 0.50% duplicate-timestamp ticks respectively, both explicitly informational-severity by design, not blocking).

---

## 2. Files Inspected

Code (full read): `src/xau_lab/data/histdata.py`, `src/xau_lab/data/exness.py`, `src/xau_lab/data/dukascopy.py`, `src/xau_lab/data/normalize.py`, `src/xau_lab/data/validation.py`, `src/xau_lab/types.py`, `scripts/resumable_histdata_download.py`, `scripts/ingest_exness_archives.py`, `scripts/normalize_histdata.py`, `config/default.yaml`, `configs/champions/champion_v1.yaml` (partial), `tests/test_histdata.py`.
Code (grepped/spot-read): `src/xau_lab/experiments/runner.py`, `src/xau_lab/experiments/sweep.py`, `src/xau_lab/cli.py`, `tests/test_normalize.py`, `pyproject.toml`.
Docs (read in full): `docs/audit/2026-08-21-source-concordance-and-code-audit.md`, `docs/audit/2026-08-21-exness-source-verification.md`, `docs/audit/2026-08-21-exness-normalization-extended-2020-2026.md`.
On-disk artifacts: every `metadata.json` under `data/normalized/source={dukascopy,histdata,exness}/symbol=XAUUSD/**` (5,009 files, read programmatically), raw archive counts under `data/raw/{histdata,exness,dukascopy}`, `logs/histdata_copy_local*.log`, `logs/histdata_normalize_local.log`.
Git: `git log -- src/xau_lab/data/histdata.py src/xau_lab/data/exness.py` (checked whether anything changed post-audit).

---

## 3. Existing Components

| Component | Location | Status |
| :--- | :--- | :--- |
| Ingestion — Dukascopy (network download, reference/ground-truth source) | `src/xau_lab/data/dukascopy.py` (`DukascopyDownloader`) | Working. 6-state `IntervalStatus`, atomic writes, checksum-verified cache, retry/backoff, 429-aware. Local coverage in this sandbox: `2025-01-06` → `2025-07-04 16:00 UTC` only (per `docs/audit/2026-08-21-exness-normalization-extended-2020-2026.md`, confirmed by my own metadata scan: `max_timestamp_utc = 2025-07-04T16:58:59Z`). |
| Ingestion — HistData (network download, monthly ZIP) | `scripts/resumable_histdata_download.py` | Working. Atomic write + fsync + SHA-256, token-scrape + POST flow against histdata.com, ZIP-structure validation (exactly one CSV + report). Raw coverage: 12/12 months for 2020–2025, 7/12 for 2026 (through July). |
| Ingestion — Exness (local copy-in, not a network fetch) | `scripts/ingest_exness_archives.py`, `src/xau_lab/data/exness.py::ingest_raw_archive` | Working. One yearly ZIP per year, checksum + provenance manifest, refuses to silently overwrite a mismatched existing archive. Raw coverage: 2020–2026, one archive per year (7 files), all present on disk. |
| Normalization — HistData → daily UTC Parquet | `src/xau_lab/data/histdata.py::HistDataTickNormalizer` (schema v3) | Working mechanically; **timestamp conversion is known-wrong for ~9 months/year (§5, §6-A)**. 2,404 daily partitions on disk, `2020-01-01`→`2026-07-31`. |
| Normalization — Exness → daily UTC Parquet | `src/xau_lab/data/exness.py::ExnessTickNormalizer` (schema v1) | Working; streaming single-pass CSV reader (no full-file load), day-bucketed with a hard-stop safety valve for unsafe reordering. 2,424 daily partitions on disk, `2020-01-02`→`2026-05-31`. Timestamp basis independently verified correct for H1-2025 only (§5). |
| Normalization — Dukascopy → daily UTC Parquet | `src/xau_lab/data/normalize.py::TickNormalizer` (schema v2) | Working; per-hour interval-completeness gate (`normalize_day`) refuses to write a day unless every hour is `AVAILABLE` or `EXPECTED_CLOSED`. This is the only source with a real completeness contract. 181 daily partitions on disk. |
| Schema (shared column contract) | `src/xau_lab/types.py::TICK_COLUMNS`, `BAR_COLUMNS` | Defined once, imported by `dukascopy.py`/`normalize.py`/`validation.py`. **Not** imported/used by `histdata.py` or `exness.py` — those two hand-roll an equivalent-but-not-identical column list (§5-Schema). |
| Storage layout | `data/raw/{source}/{symbol}/...`, `data/normalized/source={source}/symbol={symbol}/year=/month=/day=/ticks.parquet` | Working. Consistent Hive-style partitioning across all three sources; each partition has a sibling `metadata.json`. Parquet written zstd level-3 with column statistics (`histdata.py:148`, `exness.py:118`, `normalize.py:203-208`). |
| Validation | `src/xau_lab/data/validation.py::DataValidator` | Working for tick-level rules (ask≥bid, positive price, price/spread range, non-finite, duplicate/regression detection). **Interval-completeness half is hardcoded to Dukascopy regardless of the `source` argument (§6-C) — a real defect, not a limitation.** Never called from any ingestion/normalization script; only called from the backtest-gating path (`runner.py:220`, `sweep.py:304`, `cli.py:229`). |
| Metadata / provenance | `manifest.json` per raw archive, `metadata.json` per normalized partition | Working per-source, but **not unified**: three different `normalization_schema_version` counters (histdata=3, exness=1, dukascopy=2) with no cross-source schema registry (§5-Schema, §8). |

---

## 4. Data Quality Findings

All counts below come from a fresh script I ran against every `metadata.json` file currently on disk (5,009 files: 181 Dukascopy + 2,404 HistData + 2,424 Exness). Source: this session's own aggregation, not copied from the prior audit.

**Timestamp basis**
- Dukascopy: true UTC by construction (CDN path is hour-named, decoder does no timezone arithmetic — `dukascopy.py:75-87`, `:219`). Treated as ground truth by the codebase itself.
- HistData: `TIMESTAMP_BASIS = "EST_FIXED_UTC_MINUS_05_NO_DST"` (`histdata.py:29`), applied unconditionally as `.dt.offset_by("5h")` (`histdata.py:192-197`) for every month, every year, no seasonal branch. The prior audit (§C1, 2026-08-21) proved this false for the EU/UK-DST season using real Dukascopy-vs-HistData minute alignment (perfect 0.0 delta pre-2025-03-30, ~1-hour/$4-$75 delta post-2025-03-30). I confirmed the underlying code is byte-identical to what that audit inspected (single git commit touching the file, dated the same day as the audit) and spot-checked a live post-boundary partition (`data/normalized/source=histdata/.../day=07/metadata.json` for 2025-04-07) still records `"utc_conversion": "timestamp_utc = source_EST_fixed + 05:00"` with the same unconditional formula. **This is not a one-off historical finding — it is still shipping in every HistData partition normalized after this bug was found**, including partitions created after the audit (histdata data was renormalized 2026-08-21 per `logs/histdata_normalize_local.log`, i.e. after the bug's discovery, still with the old formula).
- Exness: `TIMESTAMP_BASIS = "CLAIMED_UTC_Z_SUFFIX_UNVERIFIED"` (`exness.py:45`). Independently verified correct against Dukascopy for the Jan–Jul 2025 overlap window only (`docs/audit/2026-08-21-exness-source-verification.md`). **2020–2024, H2 2025, and 2026 are unverified** — no Dukascopy ground truth exists locally for those windows (confirmed: local Dukascopy raw data stops 2025-07-04) — yet all of those years are fully normalized and sitting on disk today (2,424 partitions spanning 2020-01-02 → 2026-05-31, per my own scan). `execution_eligible=false` is correctly still set for all of them, but nothing prevents a researcher from reading and trusting that unverified data directly from the parquet files (§6-B).

**Ordering**
- All three normalizers sort each day's frame by `timestamp_utc` before writing (`histdata.py:237`, `exness.py:376`, `normalize.py:199`) — verified in code, not just asserted.
- Exness's cross-chunk assembly relies on the source file being *approximately* ascending; a hard-stop guard exists for the one unsafe case (a day's rows arriving after that day's partition was already written — `exness.py:501-508`). On current disk data, 9 partitions carry `source_stream_reordering_detected: true` (all classified "safe" per the code's own logic — I did not independently re-derive this from raw CSV bytes, only confirmed the flag and the guard logic exist and are exercised).

**Duplicates** (exact-millisecond-timestamp collisions, computed by each normalizer itself at write time — `is_duplicated().sum()`):

| Source | Total ticks | Duplicate-timestamp ticks | Rate |
| :--- | ---: | ---: | ---: |
| Dukascopy | 33,367,266 | 0 | 0.00% |
| HistData | 378,156,576 | 7,968,490 | 2.11% |
| Exness | 202,551,262 | 1,013,002 | 0.50% |

`DataValidator.validate_ticks_df()` classifies same-millisecond duplicates as `issue_type=SUB_MILLISECOND_TICKS`, `severity="info"` (`validation.py:233-243`) — informational only, never blocks. This is a reasonable design choice for genuine multi-tick-per-ms feeds, but it means the 2.11%/0.50% rates above are silently accepted wherever this validator is used; nothing surfaces them except a human reading the report.

**Missing data**
- Dukascopy: 27/181 daily partitions (14.9%) have zero ticks — consistent with weekend closures given a ~181-day window; each constituent hour is individually classified (`AVAILABLE`/`EXPECTED_CLOSED`/`EMPTY`/`MISSING_UNEXPECTED`/`DOWNLOAD_FAILED`/`CORRUPT`), so a genuine gap during an expected-open hour would surface as `MISSING_UNEXPECTED`, not silently look like a closure.
- HistData: 372/2,404 (15.5%) zero-tick days. HistData has **no per-hour interval status** at all — `"interval_semantics": "monthly_archive_with_gap_report_no_hour_status"` is recorded in every partition's own metadata (`histdata.py:257`), i.e. the code itself documents that it cannot distinguish "day genuinely had no ticks" from "an hour inside this day is silently missing from the source's monthly archive." The source's own gap report (present in the ZIP alongside the CSV) is downloaded but **never parsed or cross-checked** — confirmed by absence of any reference to the `.txt` report content beyond `zip_members()` recording its name/size (`resumable_histdata_download.py:95-121`). **UNKNOWN**: whether any of the 372 zero-tick HistData days are genuine gaps rather than closures — not verifiable without parsing that report, which this pipeline does not do.
- Exness: 671/2,424 (27.7%) zero-tick days — the highest rate of the three, but a known, disclosed cause accounts for a large share of it: both the 2025 and 2026 source yearly exports stop mid-year (2025 CSV's last row is `2025-07-16`, 2026's is `2026-05-31` — `docs/audit/2026-08-21-exness-normalization-extended-2020-2026.md`, and my own scan's `max_timestamp_utc` for Exness = `2026-05-31T23:59:59Z` matches exactly). Every day after that cutoff correctly normalizes to an empty (not missing/error) partition. This is a source-completeness limitation, not a pipeline defect.

**Invalid values**
- `DataValidator.validate_ticks_df()` implements a real rule set: null/non-finite check, ask≥bid (inverted quote), positive price, price-range, spread-range, derived-column-consistency (spread/mid must algebraically match bid/ask), negative-size checks, and timestamp-regression detection (`validation.py:59-248`). It explicitly never drops rows — a report only (`validation.py:13-16`, `:245-247`).
- However, **this validator is not wired into the ingestion or normalization scripts at all** — `grep` confirms `DataValidator`/`validate_ticks_df`/`validate_range` are referenced only from `experiments/runner.py`, `experiments/sweep.py`, `cli.py`, and the validator's own module/tests. `histdata.py`, `exness.py`, `normalize.py`, and their respective CLI scripts never call it. So a HistData or Exness partition can contain an inverted quote, a non-finite price, or an out-of-range spread and be written to disk and used for any purpose that doesn't go through `ExperimentRunner`/`ExperimentSweep` (e.g. ad hoc research notebooks/scripts reading the parquet directly) with zero validation ever applied. I did not independently re-run `validate_ticks_df` against the raw parquet content in this pass (blocked — see §8), so I cannot report whether such rows actually exist; I can only confirm the check exists but is not enforced at write time.

---

## 5. Schema Assessment

**Current schema.** Two layers exist in the codebase, and they diverge:

1. `src/xau_lab/types.py::TICK_COLUMNS` — the "official" shared contract: `timestamp_utc, bid, ask, bid_size, ask_size, spread, mid, source, symbol` (9 columns, no `source_volume`). Imported and used by `dukascopy.py` and `normalize.py`/`validation.py`.
2. `histdata.py::HistDataTickNormalizer._empty_ticks()` and `exness.py::ExnessTickNormalizer._empty_ticks()` each **hand-declare their own schema dict** rather than importing `TICK_COLUMNS`. HistData's adds a 10th column, `source_volume` (its one unsided volume field — `histdata.py:110-124`); Exness's matches the 9-column `TICK_COLUMNS` set exactly but was typed independently rather than imported (`exness.py:252-265`).

Net effect: a HistData parquet partition has one extra column (`source_volume`) that Dukascopy/Exness partitions don't have, and none of the three sources' Python schema declarations are actually the same object — they happen to agree by convention today, but nothing enforces that they stay in sync if `types.py::TICK_COLUMNS` changes. `bid_size`/`ask_size` are `null` for both HistData and Exness (neither source provides sided size — documented honestly in each partition's `bid_ask_size_policy` field), so any downstream code assuming those columns are populated for non-Dukascopy sources would be silently wrong, not erroring.

**Recommended quant-grade schema** (recommendation only, no code written):

- Make `TICK_COLUMNS` (or an explicit superset) the single source of truth all three normalizers import, including HistData's `source_volume` as an optional/nullable 10th column declared once in `types.py` rather than locally in `histdata.py`.
- Add a machine-readable `timestamp_basis_verified: bool` + `timestamp_basis_verified_window: [date, date] | null` field to every partition's metadata, separate from the free-text `timestamp_basis` label — today "verified" status lives only in prose (audit markdown files), not in the artifact itself, so a script reading `data/normalized/source=exness/.../2022/.../metadata.json` cannot programmatically tell that year is unverified without cross-referencing a doc that isn't co-located with the data.
- Unify `normalization_schema_version` into one monotonically-increasing, cross-source integer (or a `{source}:{version}` compound key registered somewhere), rather than three independent per-source counters that happen to currently read 3/1/2 with no shared meaning.
- Have `DataValidator.validate_range()` accept a source-specific "completeness backend" (a small interface Dukascopy/HistData/Exness can each implement) instead of hardcoding `DukascopyDownloader` — this is both a schema and a correctness fix (see §6-C).
- Consider parsing HistData's bundled gap-report `.txt` (currently downloaded and hashed but never read) into structured per-day/per-hour coverage metadata, closing the "UNKNOWN whether zero-tick days are genuine" gap noted in §4.

---

## 6. Risks

### Critical

**6-A. HistData timestamps are demonstrably wrong for ~9 months of every year, and the bug is still live.** Every HistData partition for late-March through the following DST transition is time-shifted by up to an hour versus true UTC (proven in the 2026-08-21 audit against real Dukascopy ground truth; code unchanged since). `execution_eligible=false` correctly prevents this from reaching a live/paper backtest through the sanctioned path, but nothing prevents it from being read directly (e.g. any ad hoc script, notebook, or future feature that loads `data/normalized/source=histdata/...` parquet files without going through the `execution_eligible` gate). 2,404 partitions are affected in aggregate proportion to the DST-season fraction of each year.

### High

**6-B. Exness data outside the Jan–Jul-2025 window (2020–2024, H2 2025, 2026 — the large majority of the 2,424 normalized partitions) has never been checked against any independent ground truth**, yet it is fully normalized and on disk, and `configs/champions/champion_v1.yaml` names `source: exness` as the dataset backing the frozen `CHAMPION_BE1` research config. `execution_eligible=false` is set correctly for all of it, but — same caveat as 6-A — that flag only protects code paths that check it.

**6-C. (New finding, not in the prior audit.) `DataValidator.validate_range()` — the function `ExperimentRunner.run_experiment()` calls to gate every backtest (`runner.py:220`, also `sweep.py:304`) — ignores its own `source` parameter for the interval-completeness half of the check.** It always constructs `DukascopyDownloader` and always resolves raw-file paths under `data/raw/dukascopy/...` (`validation.py:270`, `:285`), regardless of what `source` string was passed in. `BacktestConfig.source` is a free-form field (`types.py:259`) that downstream configs do set to `"exness"` (per `champion_v1.yaml`). Concretely, calling `validate_range(symbol, start, end, source="exness")` loads real Exness ticks for the tick-level rule checks, but then checks *completeness* against whatever Dukascopy raw archives happen to exist locally for those same calendar dates — a source that (a) may not have any local data at all for most of the Exness date range (Dukascopy raw only covers 2025-01-06→2025-07-04 in this sandbox) and (b) has nothing to do with whether the Exness data actually being tested is complete. This either spuriously fails a legitimate Exness-sourced backtest with `MISSING_UNEXPECTED_INTERVAL` errors, or — if Dukascopy raw data happens to exist for the same window — gives a false sense that Exness completeness was checked when it was not. I did not trace far enough into the champion strategy's actual data-loading path to confirm whether it calls `run_experiment()`/`validate_range()` at all (it may read the pre-built `m15_ctx_full.parquet` cache directly, bypassing this entirely) — flagged as **UNKNOWN**, worth a follow-up trace before relying on this pathway.

### Medium

**6-D. Duplicate-timestamp ticks (2.11% HistData, 0.50% Exness) are permanently informational-severity** (`validation.py:238`, `issue_type=SUB_MILLISECOND_TICKS`) with no path to escalate or even count them at the dataset level outside of what each normalizer happens to record in its own `metadata.json`. Not necessarily wrong (real feeds do have multiple ticks per millisecond), but there is no independent confirmation in this codebase that these are genuine same-millisecond prints rather than a parsing/dedup artifact — the prior audit did not investigate duplicate rates at all (it reported 0 duplicates in its narrower Dukascopy/HistData 181-day overlap sample, which is consistent with what I found for that same window but doesn't cover the full 2020-2026 range now on disk).

**6-E. Validation is entirely opt-in and disconnected from ingestion.** `DataValidator` is never called by any of the three ingestion/normalization scripts or their underlying modules — only by the backtest-gating path. A partition can be written, sit on disk, and be read by any future script for years without ever being checked for inverted quotes, non-finite values, or out-of-range prices unless that future code happens to explicitly call `validate_ticks_df()`.

### Low

**6-F. HistData's gap-report file is fetched and checksummed but never parsed** (§4/§5) — a genuine source-provided quality signal is being discarded.

**6-G. Three independent `normalization_schema_version` counters with no shared registry** (§5) — low risk today (each source's normalizer only reads/writes its own version), but a latent trap if any future code assumes a single global schema version.

---

## 7. Recommended Roadmap

*(Presented in the order requested by the review template; this is not a re-statement of the prior audit's "Required Fixes" list in §7 of that report, though it overlaps with it.)*

**Phase 1 — Data correctness**
Re-derive HistData's true timestamp basis (the prior audit already specifies the method — extend `scripts/build_source_concordance_report.py`'s `dst_lag_scan` to the autumn 2025 DST boundary) and fix `histdata.py:192-197` to apply it, rather than continuing to ship the falsified "+5h always" conversion. Fix `DataValidator.validate_range()` to resolve completeness against the actual `source` argument instead of hardcoding Dukascopy (6-C) — this is a correctness bug in the one function whose entire job is to prevent a backtest from silently running on bad data.

**Phase 2 — Metadata/reproducibility**
Add a structured `timestamp_basis_verified` field co-located with the data (not only in prose audit docs) so a script can programmatically refuse to trust an unverified year. Unify the three per-source schema-version counters. Wire `DataValidator` into the normalization scripts themselves (even just logging a warning) so quality issues surface at ingestion time, not only if/when a backtest happens to run over that range later.

**Phase 3 — Performance optimization**
Not assessed in this pass (out of stated scope; the Exness streaming reader and Dukascopy per-hour caching already look reasonable from a design read, but no profiling was done).

**Phase 4 — Feature generation**
Not assessed in this pass (out of stated scope — this is a data-layer audit, not a feature/bars/strategy audit).

---

## 8. Open Questions

1. Is HistData's true timestamp basis known/derivable now (autumn-2025 DST boundary re-scan), or does it require new Dukascopy raw data past 2025-07-04 that isn't available in this sandbox yet?
2. Does the `CHAMPION_BE1` backtest/research pipeline (`configs/champions/champion_v1.yaml`, `source: exness`) actually route through `ExperimentRunner.run_experiment()`/`DataValidator.validate_range()`, or does it read `data/results/_cache/m15_ctx_full.parquet` directly and bypass validation entirely? (Needed to know whether 6-C is a live, exercised bug or a latent one.)
3. Is there a plan/timeline to acquire independent ground truth (Dukascopy or another source) for Exness 2020-2024, H2 2025, and 2026, or is `execution_eligible=false` intended to be permanent for those years?
4. Should HistData's bundled gap-report `.txt` be parsed into structured per-hour coverage, or is monthly-archive-only granularity considered acceptable long-term for a source that's explicitly secondary/comparison-only?
5. Is the 2.11% HistData / 0.50% Exness duplicate-timestamp rate expected/normal for these vendors, or worth investigating as a parsing artifact? (No baseline exists in this codebase to compare against.)
6. Should `DataValidator` be invoked automatically at normalization time (fail-fast at ingestion) rather than only at backtest time, given it currently only runs when `ExperimentRunner`/`ExperimentSweep` happen to be used?
7. What is the retention/versioning policy for raw archives if a source (e.g. Exness) re-issues a corrected export — the current `ingest_raw_archive()` refuses to overwrite a mismatched existing archive and requires manual investigation (`exness.py:186-190`); is that the intended permanent policy?
8. Was `python3.12` ever actually usable via `.venv/bin/python3` in this sandbox, or has the venv been broken since the repo move referenced in `CLAUDE.md` ("Dời repo = chết venv")? I could not run polars/pyarrow in this session (`.venv/bin/python3` is a broken symlink to a missing `python3.12`; the sandbox filesystem itself is also at 100% disk usage, blocking a fresh `pip install`) — this genuinely blocked independent tick-level re-derivation in this pass and should be fixed before the next data audit needs to run real queries.

---

## Final Output

### 1. Files inspected
See §2 above (full list): 6 core `src/xau_lab/data/*.py` modules + `types.py`, 3 ingestion/normalization scripts, `config/default.yaml`, `configs/champions/champion_v1.yaml` (partial), 3 prior audit docs (read in full), 1 test file read in full plus 4 more grepped, `runner.py`/`sweep.py`/`cli.py` (grepped/spot-read), every `metadata.json` on disk (5,009 files, read programmatically), raw archive listings, 3 log files.

### 2. Major findings
- HistData's "fixed EST, no DST" timestamp conversion is still live and still wrong for ~9 months/year (§4, §6-A) — confirmed unchanged since the 2026-08-21 discovery.
- `DataValidator.validate_range()` hardcodes Dukascopy for interval-completeness regardless of the `source` argument — a new finding this pass, exercised by at least one real config (`champion_v1.yaml`) (§6-C).
- HistData duplicate-timestamp rate 2.11% (7.97M/378M ticks), Exness 0.50% (1.01M/202M ticks), Dukascopy 0.00% — measured fresh from on-disk metadata across the full current dataset, not just the prior audit's 181-day sample (§4).
- Exness data for 2020-2024/H2-2025/2026 (the large majority of its 2,424 normalized partitions) has never been checked against independent ground truth; only Jan-Jul 2025 was verified (§6-B).
- `DataValidator` exists with a real, non-trivial rule set but is never invoked by any ingestion/normalization script — only by the backtest-gating path (§4, §6-E).
- Schema is not actually unified across sources: `histdata.py`/`exness.py` hand-declare their own column schemas instead of importing `types.py::TICK_COLUMNS`; HistData carries an extra `source_volume` column the others don't have (§5).

### 3. Risk ranking
Critical: 1 (6-A). High: 2 (6-B, 6-C). Medium: 2 (6-D, 6-E). Low: 2 (6-F, 6-G).

### 4. Recommended next action
Fix 6-A (HistData DST) and 6-C (validate_range source-mismatch) before any further research decision leans on either HistData seasonality/cross-source comparisons or an Exness-sourced backtest's "validated" status — both are Phase 1 (data correctness) items with a clear, already-scoped fix path (the prior audit's `dst_lag_scan` methodology for 6-A; making `validate_range` source-aware for 6-C). Everything else in §7 can follow once those two are closed.
