Qualify UK model and data candidates automatically before promotion by MaxGhenis · Pull Request #574 · PolicyEngine/policyengine.py

MaxGhenis · 2026-10-11T15:36:42Z

Merge hold: cos d1613 (security). Until 16 Oct 2026 this branch's history carried a pre-suppression aggregate-impact report (tests/fixtures/uk_metrics/budget_metrics_report.json, blobs c79f1a06 from 3ea0b19 and 6b599fb5 from b424ffb). That report published Disabled Students' Allowance, which comes from fewer than 10 records, unsuppressed. The tip now carries W3's suppressed 1.58.1 report at tool commit 228a05a3e. Max is deciding whether to rewrite the history. Don't merge, force-push or delete anything for it until d1613 is ruled.

Fixes #573

Draft, stacked on #571 (receipt schema v1). The diff includes #571's commits until that merges; review origin/w4/receipt-schema-v1...HEAD.

Part of the UK Autumn Budget 2026 CI plan (workstream W4). The Budget is on 28 October and there is no model freeze, so the latest qualified model/data pair is what runs that day. This PR builds the workflow that qualifies a candidate pair and promotes it.

What it adds

Workflow (.github/workflows/certify-uk-release.yaml, scripts/check_uk_candidate.py)

Triggered by a policyengine-uk or UK data release dispatch, by hand, and on a schedule that catches up anything missed.
resolve finds the candidate pair and deduplicates: a pair that is already certified, stale or in flight is not run twice.
prepare certifies the candidate on a branch: manifest, pins, lockfile and TRO are regenerated and tested together, and a draft PR is opened.
gates runs the local gates; request-replay and poll-replay run the scorecard replay; finalize writes the qualification receipt to the ledger.
promote is serialized (concurrency promote-uk) and picks the dominating promotable pair, whatever order the candidates finished in. accept records a human acceptance of a needs_review receipt.

Gates (provenance/uk_candidate.py). All eleven are mandatory; a gate that cannot run reports unavailable, which makes the verdict incomplete, never qualified.

Gate	What it checks
`installed_versions`, `core_compatibility`	The installed model and core are the candidate's, and the model's own requirement admits that core.
`wheel_hashes`	The downloaded wheels match PyPI and the pin, their members match their RECORDs, and the installed files match the wheels.
`data_access`, `dataset_bytes`, `supported_years`	The release manifest and dataset are reachable; the simulated file has the pinned bytes and the UK layout; the required years are supported.
`budget_inputs`	52 inputs the Budget measures depend on are present and usable, compared with the previous pair.
`reform_contract`	W1's contract runner passes on every route. The gate reads the whole report: cases, sampled properties and population checks. It rejects a report that is not a full run of this candidate (other versions, dataset, seed or sample size; a property that did not run; a `holds` or `summary` that disagrees with the rows).
`metrics_review`	W3's metrics tool, vendored from a pinned policyengine-uk commit, on the candidate and the previous pair, including W3's standard reform costings. Hard checks fail; moves outside the review bounds need review.
`reform_responsiveness`	Every one of W3's ten standard reforms ran and moved its target variable in the stated direction by at least its yearly minimum, in every year from 2025-26 to 2031-32. The set, targets and minimums are the metrics tool's own (`standard_reforms.yaml`); the tool checks them as `responsive:<id>`. This gate requires those checks to be present and passing, and recomputes each verdict from the tool's per-year numbers. A reform that returns £0 or is nearly inert fails the candidate; there is no review path.
`replay`	W6's replay corpus, through an immutable request whose id is the digest of its content. The receipt must name that request, match the status file's digest, and name the candidate's wheels, data and bundle. Structural, coverage, mapping, inertness and construction flags block; numerical and support flags need review.

Certification (certification.py, manifest.py)

A new basis, qualified_exact_pair, records the pair id and its ledger URL and certifies exactly one model version, one core version and one dataset digest. It is never broadened to a range.
The runtime accepts that basis only for the exact model version and build id.

Ledger (provenance/qualification_ledger.py, .github/qualification-ledger.sh): an orphan branch, uk-qualification-ledger, with one directory per pair and one per attempt.

Consumers (push.yaml, scripts/release_consumers.py): on release, a policyengine-release dispatch carries the UK bundle identity (model, core, data package version, dataset SHA-256, certificate digest) and the qualification verdict, promotable flag and receipt digest. It goes to sim-api and api as before, and now to the dashboard and the scorecard, which also gets a released-bundle replay request.

Invariants (each tested)

The verdict is a pure, monotone function of the gates: rejected > incomplete > needs_review > qualified. Making any gate worse never improves it (Hypothesis).
Promotion is independent of arrival order and never moves to a dominated pair (Hypothesis over permutations).
A request's id is the RFC 8785 digest of its document, and the scorecard's golden request and receipt reproduce byte for byte.
The mirrors and the scorecard's schemas agree (differential): both accept the golden documents and reject the same mutated ones.
An exact-pair certificate needs evidence: validation refuses one without a promotable ledger receipt for that pair.
The contract gate never passes a partial or foreign report. Tests use W1's real reports: the full all-pass run passes, the pinned 2.125.1 run fails on its 101 defects, and each of ten tamperings is an error.
A reform that does nothing never qualifies. The responsiveness gate is monotone in each reform's response: shrinking a value toward zero or flipping its sign never turns a failure into a pass (Hypothesis). Tests cover exactly £0, nearly inert, one inert year, wrong sign, a missing year, a reform that did not run and a report with no reform checks at all, on W3's real reform list. A differential test makes the gate error when the tool's verdict disagrees with its own numbers.
expected_release_bundle is computed as the scorecard computes it: no tokens and no network for the manifest request, so the packaged certification is used. A test runs the manifest fetch in that environment and sees it fail over.

Open items

Decision d1344: decided yes (Max, 15 Oct 2026). Max delegated the methodology calls the CI plan raises to the workstream sessions. Each call needs independent review agreement and an exception recorded in the receipt (cos decisions: "delegate the methodology calls the CI plan raises to the workstream sessions, each with independent review agreement and receipt-recorded exceptions"). This PR lands as built: promotion.automatic is false and the bounds in uk_review_bounds.json are proposals, so promotion stays a reviewed PR merge. A follow-up PR turns on automatic promotion and adopts the review bounds under d1344, with an independent review and the basis recorded in each qualification receipt, once this PR has landed on its gates (coordinator, 15 Oct). The responsiveness minimums are W3's proposals in its pinned metrics tool, and adopting them is W3's call under d1344.
Metrics tool pin. The tool is pinned to policyengine-uk#2335's head when I read it (fd944159), an unmerged commit. It moves to the merged commit when that PR lands.
A real run is queued. One end-to-end run of the runner and this gate on the certified pair is waiting in the host's heavy-run queue; I'll add its result here.
Admin, with the coordinating session: the GitHub App on policyengine-scorecard and uk-autumn-budget-dashboard-2026 with Contents: write; sender tokens that cover policyengine.py; the scorecard's Hugging Face secret.
2031. The Budget horizon includes 2031, but policyengine-uk projects to 2030. The supported_years gate reports that as needs_review.
Dependencies. The reform_contract gate reports unavailable until W1's runner (Add the UK Budget reform contract runner #577, after Route UK dated reforms through policyengine-uk's statutory pipeline #575) is importable from the installed wrapper. The replay schemas are pinned to an unmerged scorecard commit (dc040de).

Checks run locally

166 tests pass in the qualification, gate, pipeline and certification suites.
ruff is clean; actionlint is clean on the workflow.
Independent review (GPT-6.1 Sol) is running at def8b8d. The responsiveness gate (28c90ea, 4da37e4) landed after it started and will get its own review pass.

axiom: n/a: release/qualification infrastructure, no policy change.

🤖 Generated with Claude Code

QualificationReceipt records a candidate pair, the policyengine.py commit that qualified it, every mandatory gate and a verdict that is a pure function of the gates. select_promotion_target picks the newest promotable pair that strictly advances the certified pair, independent of arrival order. The replay request, status and receipt mirror the scorecard's v1 JSON Schemas (vendored unchanged) and reproduce its golden digests. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ispatch Adds the certify-uk-release workflow and scripts/check_uk_candidate.py: resolve and deduplicate a candidate pair, build a candidate commit with the manifest, pins, lock, TRO and test constants regenerated together, run the mandatory gates on it (no skips), send the scorecard an immutable replay request, record the qualification receipt in the ledger branch, and promote the newest promotable pair in order. Certification gains an exact qualified_exact_pair basis tied to a ledger receipt and a core check; the runtime honours it only for exactly its model and data build. The release dispatch carries the certified bundle identity and reaches the dashboard and the scorecard without risking the existing consumers. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The receipt builder now verifies installed files against wheel RECORDs itself, so the qualification gates use the same implementation and hand the receipt builder wheel paths rather than asserted hashes. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The scorecard relaxed two rules: requested_by.workflow_run_id is optional, and a failed receipt may carry no variant or execution when it failed before the candidate was identified. The mirrors follow, and such a receipt fails the replay gate. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The reform contract gate now fails on any failing or erroring population check as well as cases and properties, and treats a report as an error unless it is a full run of this candidate: sampled properties ran, the population checks ran when a dataset was given, the reported versions, dataset digest, seed and sample size are the ones requested, and the runner's own holds and summary agree with its rows. Tests use W1's real reports (pinned policyengine-uk 2.125.1 fails on its 101 defects). The wheel gate's summary reports the archive-member check too. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Replaces the synthetic all-pass report with W1's full run (every route, 20 examples, the certified dataset): 267 pass, 6 pending d1344, 67 unsupported with reasons, both properties and 13/13 population checks. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Re-vendor the replay schemas and golden fixtures at scorecard dc040de: the golden receipt gains results.manifest_sha256. - ReplayReceipt mirror: a rejected receipt has no variant, identity or execution and integrity not_run; a complete receipt has its variant identity and a sealed results manifest; execution.python is anchored. The request mirror refuses '..' path segments. Differential tests check the mirror and the schemas reject the same documents. - expected_release_bundle runs in the scorecard's certification environment (no tokens, no proxy exceptions, proxies to 127.0.0.1:9), so packaged certification is used for a private or public data repo; a test executes the manifest fetch under it and sees it fail over. - The release dispatch carries the receipt's promotable flag. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Plan invariant "Reform responsiveness" (11 Oct 2026). A new mandatory gate, reform_responsiveness, reads the candidate's metrics report and fails unless every standard reform ran and, in every horizon year, changed the government balance in the direction of its expected impact by at least the tool's reform threshold (GBP 0.1bn), and in the reference year by at least a share of the expected impact (25%, a proposal pending d1344). A reform that returns exactly zero or is nearly inert fails; it never passes as a small number. The standard set is the metrics tool's own reform list, vendored at the pinned commit with the tool, plus additional_reforms. The metrics runner now passes those reforms to the tool, which it did not before, so the reform costings also enter the review-bounds comparison. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

W3 moved the standard reform set and the responsiveness rule into the metrics tool (policyengine-uk#2335, pinned here at fd944159): ten reforms, each with the model variable it must move, the direction and a yearly minimum, checked by the tool as responsive:<id>. The gate now holds the candidate's report to those checks instead of keeping a second definition: reform_runs:<id> and responsive:<id> must be present and passing for every reform in the tool's standard set, the report must cover the horizon, and the tool's verdict is recomputed from its reform_targets.<id> values (a disagreement is an error). The runner no longer passes a reform list, so the tool costs its own set. Both check families are hard in the metrics review as well. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…checks - Pin the metrics tool to policyengine-uk#2335 at c7a6c9639. - overlap_guard: joins the known-defect checks (it fails on main until policyengine-uk#2235), so it fails a candidate only when it worsens. - A known-defect check has worsened when the previous pair did not fail it (passed, skipped or absent), or failed it in fewer scopes, when the tool reports failing_scopes on both rows. - skipped is neither a pass nor a failure; a check the previous pair passed that is now skipped needs review. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Pin the metrics tool at policyengine-uk#2335 bd8bff74c, which adds failing_scopes to each check row (the only change since c7a6c9639). - Add W3's real compute_budget_metrics report on uk-data 1.58.0 as a fixture: seven years, the overlap guard and the ten standard reforms. On it the responsiveness gate passes, the UC standard allowance and State Pension reforms clear their minimums in every year, and the four checks that fail on main are all no-worsening checks, so a pair compared with itself passes. - The backtest's two failure modes, applied to the real report, fail the gate (and are an error while the tool's own check still says pass). Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Swap the real metrics report for the one produced at the pinned tool commit (bd8bff74c), whose check rows carry failing_scopes. Its 1,470 metric rows are identical to the earlier run's. On it, sign:vat and the overlap guard fail in all seven years and the two counted-once failures have no year. A candidate that fails sign:vat in a year the previous pair did not has worsened and fails; one that fails it in fewer years passes. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ing receipts An independent review of #574 was cut off by a transport failure, but its reproductions showed ten ways a gate passed without evidence. All ten are fixed here and tested. Reform contract: - A full run must cover what the runner's own module declares (ROUTES, each case in CASES on its routes, PROPERTIES). A recounted partial report, a dropped route, case or property, or a runner that declares no coverage is an error. - A sampled property that passed must have run the examples requested. Metrics review: - A report whose status is not ok, or with no metrics or no checks, fails (two empty reports used to compare as a pass). - A check with a status outside pass, warn, fail and skipped fails. - A hard check the previous pair ran that the candidate lacks fails; other vanished checks need review. Replay: - Flags count wherever the receipt states them, never from the summary counters alone, and a receipt that contradicts itself fails: flagged rows of a kind exceeding its count, failures under a complete integrity, supported years with no artifacts, nothing replayed. - The comparison base must be the one the request asked for. - Inert measure-years with no inertness flag need review. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- A core the publisher never claimed is admitted only by the qualified pair, so the certificate's basis is then qualified_exact_pair whatever the model's own claim. Before, a claimed model with an unclaimed core was certified on the model's basis, the certificate named no pair, and no ledger evidence was ever required. - The evidence check now covers the default dataset's reference (the repository, path, revision and bytes the runtime resolves) and the certified dataset name, and a record with no model wheel hash is refused instead of assumed to match. - At run time an exact-pair certificate holds only for the core it was tested with: the installed core is read and compared. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- A candidate branch names its pair id, so two pairs that share model and data versions (another core, another wheel) can never share a branch. Before, a qualified pair could ready an incomplete pair's branch. - Promotion readies a branch only while its pull request is at exactly the commit the qualification ran on, and an automatic merge passes --match-head-commit, so the replayed and tested bytes are the promoted bytes. - A forced re-qualification stages a new attempt key when one already exists for that base, instead of being stopped by the duplicate guard. - The ledger publish retry replays its commit with cherry-pick on the fetched tip, so two runs that both create the ledger no longer fail on HEAD~1; a test runs that race against a real bare repository. Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

policyengine-uk#2335 moved one commit: the tool digest is now taken when the module loads, so a result is stamped with the code that produced it. standard_reforms.yaml is byte-identical and the report format unchanged, so the fixtures stay the bd8bff74c run; the README says why. Also sorts one import block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…and immutable Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…e tested pair Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…onsiveness

…unner's own declarations The fixture is the real contract report (673 pass, 6 pending d1344, 79 unsupported by design, 19 population rows); the test reads the coverage from the merged runner module, so a stricter coverage rule cannot reject a real full run unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ady PR before replacing its head The request-replay child inherited POLICYENGINE_SKIP_COUNTRY_IMPORTS, so pe.uk was None and every replay request failed; it now imports the UK model directly. A forced re-qualification returns an already-ready candidate PR to draft before pushing an unqualified head. Round-3 review H1 and M6. Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Online acceptance could admit another runtime model or build through the build-model, fingerprint and publisher-claim paths. Round-3 review H2. Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s across bases Round-5 review findings on #574: - A dominant candidate qualified against an older certificate could be readied after another candidate became main, though baseline-dependent gates (supported years, metrics, responsiveness) would fail against the new main: C dropped 2032 that B had added. Promotion now requires the target's full previous certificate to equal current main's before it readies the target or lets it close other candidates. A stale target is reported with an instruction to requalify with --force against main. - An authorized forced replacement that expired stayed draft forever once main advanced, because recovery looked only at the current base's attempts. Recovery and key allocation now follow the pair's newest staged attempt across bases, keeping the exact superseded-receipt binding, owner and expiry checks, and the protection of newer work. Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Round-6 review of #574 (first attempt) found that two attempts of one pair both write the pair's published files, and the publish script's retry cherry-picked one onto the other. Git merged the indented JSON receipts line by line into a valid-looking receipt that was neither attempt's: the newer attempt's harness with the older attempt's passing replay. Promotion then readied a commit whose replay never ran. - The publish script replays a pair's published files (qualification_receipt.json, latest_attempt.json, pair.json) as binary, so a concurrent change to them conflicts. If those are the only conflicts and LEDGER_REDERIVE is set, it keeps the tip's copies and recomputes every pair's published files from its immutable attempts; otherwise the publish fails. The workflow sets LEDGER_REDERIVE to the new `rederive-ledger` command. - Readers trust a published receipt only if it is the current receipt of the pair's latest-staged attempt (a review acceptance aside): latest_receipts leaves other pairs out and reports why, promotion prints them, resolve, recoverable and accept refuse them, and fetch_ledger_receipt checks the attempt latest_attempt.json names. - Tests reproduce the race with the real script and git in both orders, with and without the hook, and refuse a hand-made merged receipt in every reader. Each mutant of the fix fails them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The repository's .python-version asks for 3.14, while each qualification job syncs a 3.13 environment with the uk and dev extras. A syncing `uv run` would replace that environment with base dependencies, so the snapshot, bundle, gate and replay steps could lose pytest and the country model. Set UV_PYTHON to 3.13 and run each step with `uv run --no-sync` after the explicit sync. The ledger re-derivation hook now runs the interpreter each job provisioned: the system Python where the job installs with `uv pip install --system`, and the synced environment's interpreter in prepare and qualify. Add the round-7 review's missing tests: a large move in the wrong direction is never a response for an up or a down reform, honest or with a lying tool verdict; and rederivation drops a review that accepts another receipt or is not a valid acceptance. Each kills its mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…onsiveness

Main reworded the oracle of extended_freeze's 2034 personal allowance check (and changed two CGT oracles with the realisation response). The W1 full-run fixture comes from the 2.126.12-era runner, and two checks share that 2034 label, told apart by oracle, so the test's declared coverage, built from today's runner, no longer matched the report and the gate found no unique identity. In production a report and its declarations come from one installed runner. The test helper now declares such a label as the report's runner did: one reworded declaration takes the one recorded oracle no current declaration matches. Also pin round 8's Low: preparation syncs the candidate lock after regenerating it and before snapshots, the bundle check and staging. Deleting or moving that sync fails the new test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cos d1344 (15 October 2026): automatic promotion is on once #574 lands on its gates; this PR merges after #574. promotion.automatic is now true, with that basis in uk.json. The automatic step used `gh pr merge --auto`, but main is not branch-protected and the repository has allow_auto_merge off, so it would have failed silently (check=False) and nothing would ever be promoted. Promotion now merges the readied candidate at its qualified commit only once every check on its pull request has passed (a skipped job counts as passed; pending, failed, cancelled, none reported, or an unreadable list waits for the next promote run). --match-head-commit still refuses a branch that moved. Four mutants of the check gate are killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Round 9 of the review found a High CI could not see: the merge with main brought contract 1.1.0, whose run-level row names the shared parameter check "every parameter's dated values and metadata, and every node's metadata and band order, in ...". The gate required the 1.0.0 wording ("... metadata in "), so every real run would have gated as an error and every UK candidate been rejected. The gate now fixes only the stem both contracts share (RUN_LEVEL_CHECK_PREFIX), and a test builds the row from the installed runner itself; it fails on the old prefix. CI stayed green because the 1.0.0 fixture no longer matched the runner's contract version, so the 52 tests that gate a report through the real runner's declarations skipped. tests/fixtures/uk_contract now holds a real full run of contract 1.1.0 (the runner at this head, the certified policyengine-uk 2.130.1 and core 3.33.0, uk-data 1.58.1's certified dataset, seed 0, 20 examples): 792 pass, 25 pending a methodology call, 101 unsupported, both properties and all 19 population rows passing. Those tests run again, and a fixture whose contract version the runner has moved past now fails rather than skips. The 1.0.0 fixture and the test helper's oracle pairing that bridged it are gone; the freeze test selects the mandatory 2034 check by structure, not by its wording. Twelve-file suite on the certified pair: 1502 passed, 3 skipped (was 1442 passed, 61 skipped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The 1.58.0 report predates the tool's small-cell rule and published Disabled Students' Allowance's total and caseload, which come from fewer than 10 records, unsuppressed; totals containing it gave it back. The 1.58.1 report at tool commit 228a05a3e publishes those figures as suppressed and rebuilds every total without them. The gate and tests that read the report move to the suppression format in the next commit. History is left as is pending a security decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

W3's aggregate-impact tool (policyengine-uk#2335, approved head 228a05a3e) publishes a figure from fewer than 10 records as value null, suppressed true, and rebuilds every total that holds one without it, listing it under excludes. The metrics review now: - never compares a suppressed figure, and does not count it as non-finite; suppressed on both sides there is nothing to compare; - withholds, for review, a figure suppressed on one side and any total whose sides exclude different figures, computing no change (it would give the small cell back). The report lists excluded ids without the sign each enters its total with, so differing exclusions are never lined up; - fails a report with a suppressed value, a non-boolean flag, malformed excludes, or an excludes entry the same report does not suppress; - treats a skipped check as not run: a skipped hard check, or one the previous pair ran, needs review. The responsiveness gate cannot recompute the tool's verdict in a withheld reform year, so a passing responsive check there needs review; a failing one still fails. The pin moves to 228a05a3e, whose standard reforms carry the minimums adopted under cos d1344, and the fixtures follow. Rules agreed with W3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin the metrics tool at 1191ac5a4 (policyengine-uk#2335's approved head), whose report gives each excluded figure's sign in exclude_signs, and refresh the fixture (values, suppression and checks unchanged). - An additive total (model_totals, balance, household net income: the tool's compare._additive) whose sides exclude different figures is lined up as compare._aligned does: each side takes out, at its own published value and the excluding side's sign, every figure only the other side excludes. The relative bound stays on the previous published total, as the tool's flag threshold is. Other totals, or ones that cannot be lined up (a needed figure not published, a shared figure with different signs, invalid signs), are withheld for review. - The format check now requires the small-cell fields (a report from a tool before suppression fails), integer years, exclude_signs of +1 or -1 for exactly the excluded figures, and a caseload suppressed together with its spending head. - supp-r1 P1: a suppressed reform target is never copied into evidence; one carrying a value is an error. Only reports in the small-cell format are written to the public ledger; any other is kept as its digest and problems. - supp-r1 P2s: a report that repeats a check fails, an unknown status in the previous report fails, repeated reform rows are an error, and any warning the previous report did not give is new (the tool's new_warning). - Tests: lining up against an independent reference (Hypothesis), the bound denominator, the real report with a synthetic previous head, the boolean-target mutant, and a responsiveness property whose generator varies one reform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Round 10 of #574 found that the reform contract fixture committed in a416230 publishes two counts from fewer than 10 records of the survey data (people whose 2027 tax fell: 4; UC payments that fell: 1), and that the ledger would publish the same report shape. - The reform contract gate now hashes and publishes publishable_contract_report: population rows, the only ones computed on the dataset, have every count from 1 to 9 in their detail replaced by "fewer than 10" (the metrics tool's privacy.py rule). The details' report_sha256 is of that published report too; a digest of the full report would let anyone recover a suppressed count by trying nine values. The gate still judges the report as returned; no rule reads detail. - The fixture's two counts are suppressed the same way (three rows each), noted in its README; the unused 1.0.0 run on 2.126.8 is removed. The history is covered by cos d1613. - Round 10 lows: the released run is also gated with its assertion ids removed (kills the three surviving resolution mutants); the run-level row test substitutes the runner's whole row; README nits fixed. - Tests: the published evidence on the real run, and a Hypothesis property that no count from 1 to 9 survives publication while everything else is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…clusions supp-r2 findings on the metrics gate: - P1: a report that breaks the small-cell format was withheld from the ledger, but its values still reached the comparison's moves and the responsiveness table, which the ledger also publishes. Now neither gate compares or records any value when a report breaks the format (the comparison's values_withheld), the responsiveness gate checks the format itself (an error), and the tool's detail text is dropped too. Duplicate metric rows are part of the format check, so the ledger never publishes a public copy beside a suppressed one. - P1: equal excluded ids with different signs took the equal-exclusion fast path; the tool compares the signed maps. Exclusions are now equal only when their signs are, so such a total is withheld in every section. - P2: a published head's caseload counts only its positive records, so it can be suppressed on its own; only a published caseload beside a suppressed head fails now. - Test gaps: shared exclusions skipped when lining up (M03), negative heads (M06), a zero sign (M09), a boolean year (M15), and an unfinished report still checked before publication (M28); the lining-up property now generates opposite signs and asserts that a move is recorded exactly when expected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

supp-r3 on #574: - Medium: the April rate-change row's 2027_total_change, summed over the people whose tax moved, was published even when only 1 to 9 of them moved, beside their suppressed counts. A total named in POPULATION_TOTALS is now published only when the counts of the records it sums over are whole counts adding up to 0 or 10 or more; any other number that is not a whole count (a total the runner adds later, a count written as a float) is withheld. - Low: NumPy scalars became their text through json.dumps(default=str) and escaped suppression; _json_safe now turns them into the Python numbers they hold. - Test gaps: the ledger evidence on pass, fail and error, for counts 0 to 11 with NumPy and float counts; totals over 5, 9, 10 and 1,097 records; the property test now hits 1, 9 and 10 often. - Nits: equal exclusions need equal signs (docstring); one figure with two signs gets that reason in every section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

supp-r4 on #574 found that the rows of one population check on different routes share a subsample, so the difference between two routes' published totals or counts is what a handful of records did, and the same holds between two candidates (cos d1601). No per-value rule can bound every combination, and no gate rule reads `detail`, so the ledger now publishes each population row's check, route and status and, from its detail, only its shape and true/false findings: every count, total, amount and text is withheld. This replaces the per-value suppression and the list of totals (and with them supp-r4's Low 1, which came from trusting value types). Also: NumPy unwrapping in evidence covers only bool, integer and floating scalars, as the receipt does (supp-r4 nit), with a test that a float32 NaN becomes "NaN" (J03); the sign-conflict reason is asserted in every section (N01). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

supp-r5 on #574 (approved; lows): - Population rows now publish only the runner's fixed fields (POPULATION_ROW_FIELDS: check, route, status, reason, seed, subsample size and the identity fields) as reported; any other field is withheld. From `detail`, only top-level True/False/None findings are kept, under the runner's own keys: a nested list or mapping is withheld whole, since its length or keys could be a count or name records. - docs/engineering/runbooks/uk-qualification.md says what the ledger withholds and how to rerun a failed population check locally with access to the dataset, keeping its figures off the record. - Tests: every non-detail field of every row published as reported (C04), an error row's status and withheld text (C07), a NaN total (P12), nested and list values withheld, and the property test on the top-level rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Confirmation review of #574 (approved; Low 2): materialize_dataset fetches the dataset the checked-out bundle manifest names, so following the runbook from an older checkout would fetch older data even with the candidate's wheels installed. The steps now check out harness.json's git_commit first and check the dataset's SHA-256 against pair.json's data.dataset_sha256 before running the contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Confirmation review of #574 (approved; Low 1): an unknown row field that held true, false or None was published, while the docstrings and the changelog say any field outside POPULATION_ROW_FIELDS is withheld. Its name is not the runner's, so it is now withheld whatever its value. The detail keys stay the runner's own, as documented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis · 2026-10-16T13:13:48Z

Merged at reviewed head 5736f99 with a merge commit. Gates: gh pr checks exit 0 (13/13 pass), mergeable, no change requests. Independent review: supp-r5 APPROVE at d2aa8d4, confirmation APPROVE at 574e262, and #635 round 7 APPROVE of the two follow-up commits (574e262..5736f99). The merge hold for cos d1613 was lifted when d1613 was ruled "No action" (Max's standing rule d932; applied by the UK hub 2026-10-16 07:12Z).

🤖 Generated with Claude Code

…s core Follows #574 on main. Its qualified-pair runtime check compared the bundle's data_build_id, which certification now takes from the cut tag, with the release manifest's build_id, the line name for every Microcosm cut: a qualified cut would be refused at import, and another cut of the same line would have matched. The runtime check, the UK candidate pair's data build id and the Zenodo description now use release_build_id. With --core-version, a regional release must also cover the pinned core by its own build or claims; a qualified pair names the primary release only. Refs #553 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis force-pushed the w4/uk-candidate-qualification branch from cff5709 to a769331 Compare October 11, 2026 19:48

MaxGhenis mentioned this pull request Oct 11, 2026

Add the UK Budget reform contract runner #577

Merged

MaxGhenis and others added 6 commits October 11, 2026 16:57

Use the reform contract runner's default sample size

d3bde21

Refs #573 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis force-pushed the w4/uk-candidate-qualification branch from 414cc55 to ca3840d Compare October 11, 2026 21:06

MaxGhenis and others added 11 commits October 11, 2026 17:31

MaxGhenis mentioned this pull request Oct 12, 2026

Ask policyengine.py to qualify each published data release PolicyEngine/policyengine-uk-data#577

Open

MaxGhenis and others added 9 commits October 12, 2026 08:29

Enforce the qualified core before online compatibility acceptance

23698f4

Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Require exact replay wheel evidence and record baseline wheel identity

537c0c1

Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Prepare exact UK candidate sidecars and keep staged receipts ordered …

3d19e3c

…and immutable Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Require full contract coverage and bind finalized gate evidence to th…

b87a776

…e tested pair Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Attach the deferred-import lint suppression to the import statement

f58b11a

Co-Authored-By: GPT-6.1 Sol via Subfleet <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into w4/qualification-resp…

ca93076

…onsiveness

juaristi22 mentioned this pull request Oct 15, 2026

Make a retention-rate elasticity of 1 the default CGT realisation response PolicyEngine/policyengine-uk#2419

Merged

Max Ghenis and others added 4 commits October 15, 2026 09:42

Merge remote-tracking branch 'origin/main' into w4/qualification-resp…

09910b1

…onsiveness

This was referenced Oct 15, 2026

Turn on automatic UK promotion under d1344 (replay policy, known gaps, review bounds, liveness) #634

Closed

Turn on automatic UK promotion under d1344 #635

Merged

MaxGhenis mentioned this pull request Oct 16, 2026

Read scorecard#183's base-side grid record and apply known gaps #637

Merged

MaxGhenis and others added 11 commits October 16, 2026 00:27

MaxGhenis marked this pull request as ready for review October 16, 2026 13:13

MaxGhenis merged commit 535df76 into main Oct 16, 2026
13 checks passed

MaxGhenis added a commit that referenced this pull request Oct 16, 2026

Merge main (with #574, #632 and #635) into w4/replay-known-gaps

00627f3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Oct 16, 2026

Push-to-main Test job times out at 60 minutes, so no release has gone out since 16 Oct #640

Closed

Let the push-to-main Test job run for 90 minutes #641

Merged

Apply the small-cell rule and flat findings to the Budget contract's population rows #643

Open

MaxGhenis deleted the w4/uk-candidate-qualification branch October 16, 2026 19:25

Navigation Menu

Repository navigation

Qualify UK model and data candidates automatically before promotion - #574

Qualify UK model and data candidates automatically before promotion#574
MaxGhenis merged 50 commits into
mainfrom
w4/uk-candidate-qualification

MaxGhenis commented Oct 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

MaxGhenis commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MaxGhenis commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it adds

Invariants (each tested)

Open items

Checks run locally

Uh oh!

Uh oh!

MaxGhenis commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MaxGhenis commented Oct 11, 2026 •

edited

Loading