Repository navigation
Certify the Microcosm UK national and local-area pair with policyengine-uk 2.132.6 - #685
Conversation
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed at 8bd01dd. There are 15 findings below, most severe first. Six are on lines in the diff and are inline comments. The other nine are in code this PR doesn't touch but whose behaviour the new bundle changes, so their detail is here.
Before merging: the branch conflicts with main since the 6.9.0 release (5f97905), which rewrote manifest.json, both TROs, pyproject.toml and uv.lock. After the rebase the 6.8.0 stamps become 6.9.0 and the TROs need regenerating.
-
Constituency and local-authority outputs now raise on the default dataset. Inline on
manifest.json:106. -
With an HF token set,
import policyenginefails when the installed policyengine-uk isn't 2.132.6. model_version.py#L151- The national release manifest claims
policyengine-uk ==2.132.6only.PolicyEngineUKLatest()is built at import (uk/model.py:489) and callscertify_data_release_compatibilitywith no guard. With a token it fetches that manifest and raisesData release manifest is not certified for the runtime model version …. Soimport policyenginefails in that environment, US use included. Without a token it falls back tounverified_data_release_manifest_unavailable. - Triggers: policyengine-uk 2.134.9 (the planned next pair, or the #602 work) or 2.130.1 installed next to this release. On main the eFRS publisher claim covered versions newer than its build, which is how 2.130.1 was certified, so drift only warned.
tests/test_manifest_version_mismatch.pysays version drift "must not raise", but it stubs this call.
- The national release manifest claims
-
The post-release scorecard request can't be built for this bundle, and it fails silently. release_consumers.py#L55-L63, uk_qualification.py#L1768-L1814
_hf_filebuildshttps://huggingface.co/{repo_id}/resolve/…with nodatasets/. The UK data package is now a dataset repo, so the fetch gets a 401 or 404.- Even with the URL fixed,
released_bundle_requestreadsartifact.size_bytes. Microcosm release manifests don't carry it, and the request model requires an integer ≥ 1 (qualification.py:792). - The step is
continue-on-error(push.yaml:313-318) and the next steps are gated on its outcome. So no released-bundle replay is sent, and the job stays green. test_released_bundle_request_is_valid_for_the_scorecardnow runs on the frozen model-repo bundle, with a stubhf_fileand a synthetic size, so it can't catch either problem.
-
The new eFRS overlays make a rollback bundle unimportable. Inline on
manifest.json:271. -
A core-only bump would silently decertify every UK receipt. Inline on
manifest.json:24. -
A forced requalification always fails at prepare. uk_qualification.py#L774-L794
prepare_candidate_commitrunscertify-datawithout--regional-manifest-urior--drop-regional-release, and the routed bundle now trips the guard incertification.py:1348-1358. So the forced cross-line run in the runbook (uk-qualification.md:87-88) takes thepromote-uklock and then exits 2. No workflow input can pass either flag. It fails before staging, so nothing is left half-done.- The prepare test now seeds from the unrouted frozen fixture (
test_uk_qualification_pipeline.py:605), so it can't see this.
-
Receipts for runs on the local-area file name the national build. receipt.py#L3788
_data_identitytreats everydatasetsentry as part of the certified release. A receipt formicrocosm_uk_2024_25_densetherefore carries the national cut asbuild_idandrelease_manifest_revision, next to the densedataset_revision. The national release manifest doesn't list the dense file.- These receipts are uncertified, so nothing checks the fields. But they misattribute every constituency and local-authority result, and they enter the execution identity.
- Fix: count an entry as in the release only when its revision is the record's own.
-
The ledger concurrency guard now passes vacuously. test_qualification_ledger_concurrency.py#L64-L69
- This file wasn't moved to the frozen bundle.
BASEis the packaged manifest andPAIRis a uk-datamake_pair, soplan_promotiondrops the pair as other-line, andassert plan.ready_commit is None(L284-285) can't fail. - A promotable receipt naming B's commit gives
ready_commit=Noneagainst the packaged bundle andbbbb…against the frozen one. - All three uses of
BASE(L68, L170, L284) need to move together. Choosing the bundle in one place would have covered this file: a shared CLI loader intests/qualification_fixtures.py, instead of the ten inlinecli.BUNDLE_PATHoverrides, five of which are never read.
- This file wasn't moved to the frozen bundle.
-
The catch-up keeps re-dispatching dead uk-data attempts. check_uk_candidate.py#L529-L550
cmd_recoverablehas no data-line filter. An attempt abandoned around the merge stays recoverable forever after its lease. Each two-hourly re-dispatch (force=false) resolvesother_linebefore the recovery branch (uk_qualification.py:423-463), so nothing clears it: about 12 green no-op runs a day per pair.- The ledger currently has one at risk:
d14ea200…/attempts/f40946222a8f, whose lease expired at 18:02Z with no replay request. cmd_refresh_pr_checkslikewise keeps re-running CI on the 17 uk-data candidate PRs, though that is capped.- Fix: skip pairs that are off main's data line.
-
UK and US
calibration_diagnostics.jsonshare a cache path. dataset_materialization.py#L119- Materialized files are cached by basename, so in a shared
data_dirthe two countries overwrite each other. The first result'spaththen holds the other country's bytes, and every alternate call re-downloads. - Nothing materializes both today.
- Fix: namespace the destination by country or revision.
- Materialized files are cached by basename, so in a shared
-
The constant rewrite collides on this pair's values. Inline on
test_uk_qualification_pipeline.py:535. -
The changelog and docs miss two consequences. Inline on the changelog fragment.
-
next()without a default in the receipt test. Inline ontest_execution_receipt.py:308. -
Each area run loads the whole local-area file, and the simulation cache keeps it. datasets.py#L365-L370
load_datasetsbuilds a newPolicyEngineUKDataseton every call. It reads every table of the ~720 MB file (the eFRS is 128 MB) plus the data-year tables, only to keep about 1/650 or 1/361 of households (model.py:193-198). sim-api's executor callsensure_datasetson each request.Simulation.ensure()caches a shallow copy that keeps the full input dataset (simulation.py:228-236). The LRU holds 100 entries and evicts by count only, though a cache hit reads only outputs. When each request loads its own dataset, warm workers can hold about 1–2 GB per cached area request.- The US local-area path already works this way; this PR turns it on for 1,011 UK areas.
- Fix: a per-process dataset cache, and caching the simulation without its input dataset.
-
The UC-floor exception isn't recorded where agents read the rule. data-certification.md#L173-L176
- The skill, which
CLAUDE.mdmakes canonical, says a UK default switch carries the #491 floor within 10%, anduk_uc_floor.jsonsays "must". - After merge the certified default fails the floor (0.897), and the acceptance lives only in this PR body. The next agent that applies the skill will flag it again or treat the floor as optional.
- Fix: record the exception in the skill or
uk_uc_floor.json, or at least in the changelog.
- The skill, which
| }, | ||
| "default_dataset": "enhanced_frs_2024_25", | ||
| "default_dataset_uri": "hf://policyengine/policyengine-uk-data-private/enhanced_frs_2024_25.h5@1.58.1", | ||
| "default_dataset": "microcosm_uk_2024_25", |
There was a problem hiding this comment.
1. Constituency and local-authority outputs now raise on the default dataset.
The Microcosm national file has no constituency_code_oa or la_code_oa; only the local-area file does. The eFRS it replaces had both. So on simulations built from the default dataset, these calls raise:
compute_uk_geography_disclosure(..., levels=("region", "constituency")), the example indocs/outputs.md:279-284anddocs/disclosure.md:14-19. Error: "gives no household an area in any of ['constituency_code_oa']".compute_uk_constituency_impactsandcompute_uk_local_authority_impacts. Error: "require baseline household column 'constituency_code_oa'".
All of these run on main. Checked against the certified files: the national file raises, while the eFRS and the local-area file don't.
Only the existing caveat at docs/regions.md:169-171 says to use the routed dataset. The changelog and those two examples don't.
Suggest a changelog and docs note to run these on microcosm_uk_2024_25_dense, and an error message that names that dataset.
There was a problem hiding this comment.
Fixed in 5f3df72. Both errors now name the dataset the bundle routes the area regions to, through a new uk_area_dataset_hint that gives nothing when no area is routed. docs/outputs.md and docs/disclosure.md now say to run constituency and local-authority levels on microcosm_uk_2024_25_dense. The changelog says so too (02ec73e).
| "revision": "655dd07e4bb9c777b00dac044949611f1feb824f", | ||
| "sha256": "584ae33d80ca0431254610a3f8254d132da73477d31966d6446282861ecae50d" | ||
| }, | ||
| "enhanced_frs_2024_25": { |
There was a problem hiding this comment.
4. The new eFRS overlays make a rollback bundle unimportable.
These four overlay names are the ones a uk-data release certifies. enhanced_frs_2024_25 is also what the documented area_coded_national flow carries (data-certification.md:63-80). certify_data_release rewrites data_releases without looking at dataset_overlays, so two paths break:
- A rollback (
certify-dataon uk-data 1.58.x with--drop-regional-release): the overlay shadows the certified default. - The national-first flow: the overlay collides with a certified entry.
Either way certify-data exits 0 and plain bundle.py check passes. Then _apply_dataset_overlays raises on every get_release_manifest("uk"), so import policyengine fails. That includes installs without policyengine-uk, because the countries package builds the UK registry. Reproduced with the real certify-data.
CI would catch it, but the rollback path is the one used under time pressure.
Suggest certify-data apply or prune the overlays before writing, plus a line in the rollback docs.
There was a problem hiding this comment.
Fixed in 22a2c49.
certify_data_releasenow removes anydataset_overlays.<country>entry whose name the new release certifies, and lists the removed names in its warnings.bundle.py checknow refuses an overlay that shadows a certified name, throughvalidate_dataset_overlaysin the stdlib validator.- A test runs the rollback: it certifies the enhanced FRS with
--drop-regional-releaseover a routed bundle that carries the eFRS overlays. - The re-certification section of the skill now describes this.
| "compatibility_basis": "legacy_compatible_model_package", | ||
| "data_build_fingerprint": "sha256:c094d7006fdd81df220090284212f16bbffb15624643bc442c4bb3e1b1e5dac8", | ||
| "data_build_id": "policyengine-uk-data-1.58.1" | ||
| "certified_for_core_version": "3.33.3", |
There was a problem hiding this comment.
5. A core-only bump would silently decertify every UK receipt.
This is the first built_with_model_package certificate on main to name a core. From here on, receipts certify this value, not the packages pin (receipt.py:3737-3738). Nothing keeps the two in step:
bundle.py update-packages --core …changes onlypackages(prepare_package_bundle_update.py:38).certify-data --country us --core-version …also moves the shared pin.check --strict-troscompares the core only forqualified_exact_pair(generate_trace_tros.py:141).
So a core-only bump makes every UK receipt uncertified while every check passes. Such bumps have happened before (390cb72). The model has a guard against the same drift (test_runtime_pin_update_requires_matching_certified_model), but the core has none.
The test changes hide this:
_bundle_for(test_execution_receipt.py:135-137) now overwrites the certified core with the installed one.- The differential case uses
_certify_core, which edits both fields. - No test now reaches the
or corefallback that receipts from 6.5.0–6.8.0 verify through.
Suggest:
- a check that compares the pin with any certified core the certificate states;
- one test that edits only
packages.policyengine-core; - one test with the key absent.
There was a problem hiding this comment.
Fixed in 7e8df2d. The strict TRO check now refuses a core pin that differs from any core a certificate names, not only for a qualified exact pair. So update-packages --core, or a US run with --core-version, now fails check --strict-tros until UK is re-certified.
New tests:
- the strict check with a pin-only change;
- a receipt after a pin-only edit, where the core the certificate names decides;
- a certificate that names no core, where the pin is the certified core, in both directions.
| SUCCESSOR_CUT = "microcosm-uk-2024-25-national-20261101T000000Z-aaaaaaaa" | ||
|
|
||
|
|
||
| def _successor_record(old): |
There was a problem hiding this comment.
11. The constant rewrite collides on this pair's values.
certified_constant_replacements rewrites by value, and this certificate makes several values coincide:
- model and built-with are both 2.132.6;
- build id and revision are both the cut tag;
- the UK data version "0.1.0" equals the US one.
Simulated on copies of the constant files:
- A model-only bump also rewrites
UK_BUILT_WITH_MODEL_VERSION. - A data-version bump rewrites
US_DATA_RELEASE_VERSIONand the US assertion intest_models.py:140. - A uk-data record writes its build id into
UK_DATA_RELEASE_REVISION. - The dense pins (
UK_LOCAL_AREA_RELEASE_ID,UK_CERTIFIED_LOCAL_AREA_URI) are never rewritten.
_successor_record moves the colliding values together and keeps the data version, so this test can't see any of it. It's unreachable today, because resolve only builds uk-data pairs, and wrong pins would fail CI rather than ship. But the Microcosm automation will rely on this test.
Keying the replacements by constant would fix it.
There was a problem hiding this comment.
Partly fixed. The rest is deferred.
- Fixed (8ea62c1): a candidate from another data line no longer rewrites the constants at all, so a uk-data build id can no longer land in
UK_DATA_RELEASE_REVISION. - Deferred: the other collisions need a promotion on the Microcosm line, and resolve cannot build one yet. I would key the replacements by constant as part of that automation. It also has to rewrite the local-area pins, and a rewrite by value cannot find those.
| @@ -0,0 +1 @@ | |||
| Switch the UK default dataset from the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1) to Microcosm UK 2024-25 (`microcosm_uk_2024_25`, cut `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), certified with policyengine-uk 2.132.6 and policyengine-core 3.33.3, the versions it was built with. Constituency and local-authority regions now load the Microcosm local-area release (`microcosm_uk_2024_25_dense`, cut `microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe`) and are in the default UK region registry. The Enhanced FRS 2024-25 datasets stay loadable by name. UK results change with the data. | |||
There was a problem hiding this comment.
12. The changelog and docs miss two consequences.
The fragment doesn't mention:
parliamentary_constituency_weights,local_authority_weightsandlocal_geography_weightsno longer resolve by name.resolve_dataset_referencefalls through to the Microcosm release manifest, andmaterialize_datasetraises "Unknown managed dataset". Downstream bump PRs get the changelog as their notice, and sim-api's region fallback is one caller.- Area outputs on the default dataset need the local-area file (see the comment on
manifest.json:106).
Separately, two docs are stale:
docs/households.md:108-110still says the bundle certifies policyengine-core 3.33.0.README.md:73-74still calls the UK data repo a model repo.
There was a problem hiding this comment.
Fixed in 02ec73e.
- Changelog: it now says that the three weight matrices no longer resolve by name, that area outputs need
microcosm_uk_2024_25_dense, and which policyengine-uk version import accepts when a token is set. docs/households.md: it now names core 3.33.3.- README: it now says the Microcosm default is in a dataset repository.
| # Same bytes registered under another certified dataset name. | ||
| record = bundle["data_releases"]["uk"] | ||
| record["datasets"]["frs_2024_25"]["sha256"] = record["certified_data_artifact"][ | ||
| other = next( |
There was a problem hiding this comment.
13. next() without a default.
A national-only re-certification (--drop-regional-release) leaves one H5 plus the JSON and NPZ files. With no second H5, this next() fails the case with StopIteration.
Either fix keeps the case independent of the routing:
- register the default entry under a fresh name (not an overlay name);
- or drop the
.h5filter.
There was a problem hiding this comment.
Fixed in 7e8df2d. The case now registers the default entry under a fresh name, so it no longer depends on the routing.
…ne-uk 2.132.6 scripts/bundle.py certify-data on the 19 October national cut (microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71), with its local-area cut (microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe) as the regional release, at policyengine-uk 2.132.6 and policyengine-core 3.33.3, the versions both cuts were built with. Constituency and local-authority regions route to the local-area file and join the default UK registry. The enhanced FRS 2024-25 datasets stay loadable by name through dataset_overlays at 1.58.1. uv.lock was re-resolved with uv 0.12.14 (only policyengine-core and policyengine-uk move), and both TRO sidecars were regenerated. The tests that pinned the enhanced FRS, and the UK model-surface snapshot, now pin this pair; the receipt test helper also points a certificate's named core at the installed core. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The qualification tests build their candidates on the policyengine-uk-data line and read the packaged bundle, which is now certified on the Microcosm line, so every such candidate became other_line. They now read a frozen copy of the last bundle main certified on the uk-data line (2591af3), including through the candidate CLI's BUNDLE_PATH. The constant-rewrite test and the real-CLI replay test still read the packaged bundle: the constant files pin it, and the rewrite now takes it to a successor cut on its own line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The UK data package is now a Hugging Face dataset repository, and Microcosm release manifests record no artifact sizes. The release consumer addressed every repository as a model repository, and the request needs the dataset's size, so the post-release replay request could not be built, silently, as its step continues on error. released_bundle_request now passes the repository type to hf_file and reads a missing size through hf_size; release_consumers.py builds the datasets/ URL and reads X-Linked-Size from a HEAD request. The release job warns when the request cannot be built. The request test runs on both the uk-data model repository and the packaged dataset repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlays may not shadow a certified dataset name, but certify-data rewrote data_releases without looking at dataset_overlays. A rollback to policyengine-uk-data, or a national-first route carrying the enhanced FRS, would certify names the bundle keeps as overlays, and then every release-manifest read, and so import policyengine, would fail. Certification now removes the overlays the new release certifies and says so in its warnings, and bundle.py check refuses an overlay that shadows a certified name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This is the first certificate on main to name its core. Receipts certify the core a certificate names, falling back to the core pin only when it names none, but the strict TRO check compared the pin with the certificate only for a qualified exact pair. So a core-only pin change (update-packages --core, or certifying the other country with --core-version) would pass every check and leave every UK receipt uncertified. The strict check now refuses a core pin that differs from any core a certificate names. New tests cover the check, a pin-only edit against a certificate that names its core, and a certificate that names none. The receipt test for another dataset name no longer depends on the routing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prepare_candidate_commit ran certify-data without a regional flag, so on a main routed to a local-area release every forced requalification failed at preparation: certification refuses to drop the routing silently, and no workflow input could pass a flag. A pair on main's data line now keeps main's regional release, which must serve the pair by its own claims. A pair from another line drops the routing, which belongs to the line it would replace, and leaves the test constants alone, since rewriting them by value from another line's certificate writes into unrelated constants. The preparation test now also runs on the packaged, routed bundle, and the runbook says what a forced cross-line run does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmd_recoverable had no data-line filter, so an attempt for a uk-data pair abandoned around the switch stayed recoverable: every two-hourly re-dispatch resolved as other_line before recovery and cleared nothing. It now skips an unforced attempt whose pair is off main's data line, and cmd_refresh_pr_checks no longer re-runs CI for such pairs, which promotion never takes. The qualification tests now choose main's bundle in one place: load_candidate_cli in tests/qualification_fixtures.py replaces ten inline loaders, and the ledger concurrency test, which still read the packaged bundle, now uses the uk-data-line bundle its pair belongs to, so its promotion assertion is no longer vacuous. That frozen bundle is now main's 6.9.0 release (5f97905). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_data_identity treated every certified dataset entry as part of the primary release, so a receipt for a run on the local-area file named the national cut as its build and manifest revision, beside the local file's own revision. Only an entry at the primary release's revision now takes the release's identity; any other entry is named, like an overlay, by its own revision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Materialized bundle artifacts were cached by basename, so in a shared data directory the UK and US calibration_diagnostics.json overwrote each other. A nested artifact now keeps its release path under the data directory; top-level files, every H5 among them, keep their cache path. A path that is absolute or leaves the repository is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Microcosm national default has no constituency_code_oa or la_code_oa; only the local-area file does, while the enhanced FRS it replaces had both. So constituency and local-authority outputs on the default dataset now raise. Their errors now name the dataset the bundle routes those regions to, and the disclosure and outputs docs say to run those levels on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The changelog now says that area outputs need the local-area dataset, that the enhanced FRS weight matrices no longer resolve by name, and which policyengine-uk version import accepts with a token. The data certification skill records the accepted Universal Credit floor exception, so the next certification neither flags it again nor treats the floor as optional. households.md and the README no longer name the old core pin or call the UK data repository a model repository. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8bd01dd to
02ec73e
Compare
|
Addressed at 02ec73e. The six inline findings have replies in their threads; the other nine are below. Rebase. The branch is rebased on main at 5f97905 (6.9.0). I re-ran the certification there, so the 6.9.0 stamps and both TROs are regenerated.
|
juaristi22
left a comment
There was a problem hiding this comment.
Re-reviewed at 02ec73e.
First review:
- Fixed (12 of 15): findings 1, 3–10, 12, 13 and 15. I checked each fix.
- 2: agreed. Main is just as strict, since the eFRS claims are exact, so no change is needed.
- 11 and 14: deferred, as you said.
New: 15 findings, most severe first. 14 are inline on lines in the diff. The rest are below, because they sit in code the diff doesn't touch.
-
The new core check freezes the shared core pin while it binds, and stops binding after a re-certification done the skill's way. Inline on
generate_trace_tros.py:151. -
Constituency and local-authority results can no longer carry a certified receipt. receipt.py#L3882
- A receipt is
certifiedonly when the run usedcertified_data_artifact.datasetwith its bytes (receipt.py:3878-3888;docs/receipts.md:13), and the certificate names one dataset, the national file. certify-datadoes vet the local-area release. It checks the role, the single H5, and that the release's own claims cover the model and core, and it pins the file's bytes indatasets. But none of that becomes part of the certificate, and the TRO doesn't cover the file either.- It never showed before:
- receipts are UK-only, so US state and district runs never had them;
- until this PR the UK bundle routed no area regions, so area outputs ran on the certified national eFRS.
- Now
compute_uk_geography_disclosure(levels=("constituency",))has to run onmicrocosm_uk_2024_25_dense, and its receipt isuncertified. The new test attests/test_execution_receipt.py:413-427pins that. docs/regions.md:107calls the file "a certified local-area release", and nothing else says so.
Either:
- document that area results carry uncertified receipts (changelog,
regions.md,receipts.md); - or extend the certificate and the TRO to cover the routed dataset: certify-data already has its sha256, build and manifest, and the receipt would then accept a run on that dataset.
- A receipt is
-
Forced cross-line attempts still loop in the catch-up. Inline on
check_uk_candidate.py:553. -
The scorecard's own catch-up still needs
size_bytes, which Microcosm manifests don't carry. Inline onuk_qualification.py:1829. -
The released-bundle request tests don't pin the Hugging Face request details. Inline on
test_uk_qualification_pipeline.py:3379. -
The reason given for not running reform responsiveness doesn't hold. Inline on
data-certification.md:193. -
Pruned overlays are gone for good, but three texts still say overlays survive re-certification. Inline on
certification.py:1384. -
Cross-line candidate PRs never close, and their body says merging promotes the pair. Inline on
uk_qualification.py:855. -
On Windows, the cache-path guard misses backslash and drive-letter paths. Inline on
dataset_materialization.py:96. -
Five new tests assume the packaged bundle is a Microcosm main. Inline on
test_uk_qualification_pipeline.py:2946. -
The fragment type and a few texts are inaccurate. Inline on
certify-uk-microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71.changed.md:1. -
Same-line preparation can't pass a model or core change. Inline on
uk_qualification.py:774. -
release_consumers.pyre-implements the shared Hugging Face helpers, with drift. Inline onrelease_consumers.py:55. -
The overlay-shadowing rule is written three times. Inline on
bundle_validation.py:27. -
Core drift is checked in two branches with two messages. Inline on
generate_trace_tros.py:142.
| raise ValueError( | ||
| f"{prefix}: runtime core differs from the qualified exact pair" | ||
| ) | ||
| elif ( |
There was a problem hiding this comment.
1. The new core check freezes the shared core pin while it binds, and stops binding after a re-certification done the skill's way.
- Freeze. The certificate names core 3.33.3, and both Microcosm releases claim exactly
policyengine-core ==3.33.3.update-packages --core X(still described as a bundle-only PR indocs/bundles.md:119-128) now failscheck --strict-trosin PR CI.- So does a US
certify-data --core-version X, which moves the shared pin (certification.py:1397). - Re-certifying UK with
--core-version Xis refused for both the national and the routed dense release; the regional release gets no qualified-pair escape (certification.py:1171-1181). - So no core bump can land until Microcosm publishes cuts that claim the new core, including one a US release needs. The pin has moved 8 times since June, twice core-only.
- Opt-in.
certify-datarecords a core only when given--core-version(certification.py:1002-1003), and the skill's UK commands don't pass it. The next re-certification done the skill's way drops the guard, and receipts fall back to the pin. Three of the new receipt tests would trip, but only as bare assertion failures.
Suggest:
- say this in
docs/bundles.md, the skill and the changelog; - default
--core-versionto the pin, or add it to the skill's commands; - decide whether UK's exact claims should block a US core bump.
There was a problem hiding this comment.
Addressed in e691df0, together with finding 15.
- Documented: the skill now says what a named core binds. Because Microcosm claims one exact core, a core bump needs cuts that claim the new core. The other way is to re-certify UK without
--core-version: receipts then certify the pin, as they did before October.docs/bundles.mdnotes the binding next toupdate-packages --core, and the changelog says it too (2993625). - Guard kept: the skill's UK commands now pass
--core-version, so a re-certification done the skill's way keeps the guard. - Tests: the receipt tests now set the named core they test, and no longer assume the packaged certificate has one.
- Still open: whether UK's exact claims should block a US core bump. Until the next Microcosm pair claims its own core, re-certifying UK without
--core-versionis the way to move the core.
| candidate = read_json(attempt / "candidate.json") | ||
| # An unforced run of a pair from another data line resolves as | ||
| # other_line before any recovery, so dispatching it clears nothing. | ||
| if not candidate.get("force") and ( |
There was a problem hiding this comment.
3. Forced cross-line attempts still loop in the catch-up.
recoverable skips only unforced off-line attempts, but the catch-up re-dispatches what it prints with -f force=false (certify-uk-release.yaml:464). candidate_action then returns other_line before reaching the recovery branch (uk_qualification.py:423-434).
- After the switch every new uk-data attempt is forced, because resolve only builds uk-data pairs.
- A forced run "for evidence" that is cancelled, times out or finalizes
incompletetherefore stays recoverable for good once its lease passes. It is re-dispatched every 2 hours. - Simulated through the CLI: 11 dispatches over 2 days, with the ledger unchanged. Once uk-data's main moves, each re-dispatch also fails resolve with a mismatch.
- The new test asserts the forced attempt is printed (
test_uk_qualification_pipeline.py:2972-2973), while the runbook says recovery skips these pairs. - The notice also fires before
attempt_is_recoverable, so attempts that are still live log "not recovering" every 2 hours.
Suggest skipping every off-line pair, as refresh-pr-checks does (or passing force through), with the check after attempt_is_recoverable.
There was a problem hiding this comment.
Fixed in 741b269.
- Recovery now skips every pair from another data line, forced or not.
- The skip comes after the recoverability check, so live attempts log nothing.
- The test covers both cases.
- The runbook says to re-run a forced cross-line run by hand.
| ) | ||
| data_repo_id = reference.get("repo_id") or data_package["repo_id"] | ||
| data_repo_type = reference.get("repo_type") or repo_type | ||
| size_bytes = artifact.size_bytes |
There was a problem hiding this comment.
4. The scorecard's own catch-up still needs size_bytes, which Microcosm manifests don't carry.
The X-Linked-Size fallback covers only this push path.
- When that request fails, which is now only a warning, the scorecard's scheduled discovery reads
released["size_bytes"]from the release manifest (pipeline/uk_replay_discover.py:471). - Microcosm's assemblers never write it (
tools/assemble_uk_release_dir.py:462-469and the dense one), and the pinned national manifest has none. - So discovery raises KeyError, logs "not requested" and retries on every run.
A one-line size_bytes in each assembler fixes both paths for future cuts. For this pinned cut, only a scorecard-side fallback works.
Worth checking: the scorecard's docs describe its Hugging Face token as reading policyengine-uk-data-private only. If so, it can't fetch populace-uk-private data even through the fixed push path.
There was a problem hiding this comment.
Agreed. Both fixes belong outside this repository, and the PR body now lists them as follow-ups.
- Microcosm's release assemblers should write
size_bytes. That fixes both paths for future cuts. - For the pinned cut, the scorecard needs a fallback of its own. It also needs read access to
policyengine/populace-uk-privatefor its Hugging Face token. I have not checked that access yet.
| repo_type = record["data_package"]["repo_type"] | ||
| fetched = [] | ||
|
|
||
| def hf_file(repo, rev, path, kind): |
There was a problem hiding this comment.
5. The released-bundle request tests don't pin the Hugging Face request details.
The stubs record only repo and type, the size call's arguments are never asserted, and the release_consumers HEAD stub ignores the URL.
In a mutation run, all four cases still passed after each of these changes:
- hard-coding the model namespace in
_hf_fileor_hf_size, which is item 3 of the first review; - sending the size HEAD to the wrong repo, revision, path or type;
- dropping the Authorization header;
- not passing
hf_size.
Suggest asserting the full (repo, revision, path, repo_type) for both calls, and the URL and headers the script builds.
There was a problem hiding this comment.
Fixed in e40d3ba.
- Request test: it now asserts the full repository, revision, path and repository type of both Hugging Face calls.
- Release-consumer test: it asserts:
- the URLs, quoted;
- the Authorization header, which now also comes from
HUGGINGFACE_HUB_TOKEN; - each HEAD answer the size reader handles: a redirect with
X-Linked-Size, a direct 200, and a redirect without a size that it follows.
| weighted national totals only. Before quoting any other figure from UK data | ||
| weighted national totals only. | ||
|
|
||
| **Recorded exception (policyengine.py#685, October 2026).** The Microcosm UK |
There was a problem hiding this comment.
6. The reason given for not running reform responsiveness doesn't hold.
The PR body says metrics review, reform responsiveness and replay "measure a candidate against the previous certified pair". That's true for metrics review and replay. It isn't for reform responsiveness:
reform_responsiveness(report, standard_reforms, years)(uk_candidate.py:2062-2068) takes no previous report;gate_responsivenessnever reads the previous certificate. The gate checks the candidate's own reform responses against fixed minima.
So it could run on the national cut, and it matters here: the PR's known gaps include a rate schedule this cut has no carriers for (the SDLT higher rates).
Suggest running it and reporting pass or fail. Otherwise, record an acceptance for the comparison gates next to the UC-floor exception here.
There was a problem hiding this comment.
Agreed: reform responsiveness checks the candidate against fixed minima, not against the previous pair. I ran it on the national cut at 2.132.6, and the result is needs review.
- Nine of the ten standard reforms pass in every year.
- For the tenth, the tool's own check passes. But the tool's small-cell rule withholds that reform's figures in every year, so the gate cannot recompute them.
- A disclosure-control item on that reform went to the author separately.
The skill now records the acceptance of the two comparison gates, the metrics review and the replay, next to the UC-floor exception (6bed939). The PR body is corrected.
| @@ -0,0 +1 @@ | |||
| Switch the UK default dataset from the Enhanced FRS 2024-25 (policyengine-uk-data 1.58.1) to Microcosm UK 2024-25 (`microcosm_uk_2024_25`, cut `microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71`), certified with policyengine-uk 2.132.6 and policyengine-core 3.33.3, the versions it was built with. Constituency and local-authority regions now load the Microcosm local-area release (`microcosm_uk_2024_25_dense`, cut `microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe`) and are in the default UK region registry. UK results change with the data. The national file has no constituency or local-authority codes, so constituency and local-authority outputs (`compute_uk_geography_disclosure` at those levels, `compute_uk_constituency_impacts`, `compute_uk_local_authority_impacts`) need simulations on `microcosm_uk_2024_25_dense`. The Enhanced FRS 2024-25 datasets stay loadable by name, but its weight matrices (`parliamentary_constituency_weights`, `local_authority_weights`, `local_geography_weights`) no longer resolve. With a Hugging Face token set, importing policyengine with any policyengine-uk other than 2.132.6 raises, as it did before for versions the Enhanced FRS release did not claim. The certified default is below the Universal Credit caseload floor (0.897 of DWP's benchmark), an exception accepted and recorded in the data certification skill. | |||
There was a problem hiding this comment.
11. The fragment type and a few texts are inaccurate.
- Fragment type. It says the weight matrices "no longer resolve", which is a removal.
bump_version.py:86-88makesremoveda minor bump, so as typed this ships as 6.9.1. Earlier default switches shipped as patches, so this is a judgment call. - "0.897 of DWP's benchmark". The PR body calls the 6.7m benchmark policyengine-uk-data's derivation. DWP's Stat-Xplore figure, which Microcosm targets, is 6.2m.
- Stale texts.
- The
uk_qualificationmodule docstring still says prepare always regenerates the test constants. test_execution_receipt.py:335-336says receipts from 6.5.0 to 6.8.0 name no core. 6.9.0's don't either.
- The
There was a problem hiding this comment.
- Fragment type: the weight matrices moved to a
removedfragment, so this ships as a minor release. To ship it as a patch instead, fold that fragment back into thechangedone. - The 0.897: it now reads "of the floor's 6.7m benchmark, policyengine-uk-data's derivation", with Stat-Xplore's 6.2m beside it.
- Stale texts: both are corrected. The module docstring is fixed in 741b269, and the receipt-test comment in e691df0.
| return [] | ||
| if data_line(pair) != certified_data_line(bundle): | ||
| return ["--drop-regional-release"] | ||
| return [ |
There was a problem hiding this comment.
12. Same-line preparation can't pass a model or core change.
It keeps main's dense release, which claims exactly ==2.132.6 / ==3.33.3, and the regional release gets no qualified-pair escape.
A probe with a same-line pair:
- 2.132.6 / 3.33.3 prepares;
- 2.140.0, or core 3.34.0, is refused.
It's unreachable today, but the first automated Microcosm qualification would fail here, and only the docstring says so.
Suggest carrying a regional cut in the candidate, or refusing same-line routed preparation up front with a clear message.
There was a problem hiding this comment.
Fixed in 3b2db90. Preparation now refuses a same-line pair up front when its model or core differs from what main's local-area cut was certified for. The message says that a local-area cut for those versions is needed.
|
|
||
|
|
||
| def _hf_file(repo_id: str, revision: str, path: str) -> bytes: | ||
| HF_REPO_PREFIXES = {"model": "", "dataset": "datasets/", "space": "spaces/"} |
There was a problem hiding this comment.
13. release_consumers.py re-implements the shared Hugging Face helpers, with drift.
HF_REPO_PREFIXES, _hf_url and _hf_headers duplicate https_dataset_uri and hugging_face_auth_headers in manifest.py. The copies differ in three ways:
- Token. With only
HUGGINGFACE_HUB_TOKENset (listed in this script's ownTOKEN_VARIABLES), both requests go out unauthenticated. CI setsHUGGING_FACE_TOKEN, so only manual runs are affected. - Quoting. The copy skips the quoting the shared helper applies.
- Small-file branch. The
_hf_sizedocstring says a small file "answers directly with its length", but Hugging Face answers HEAD for a non-LFS file with a relative 307 and noX-Linked-Size. So theContent-Lengthbranch never runs. That is harmless for the LFS default H5.
There was a problem hiding this comment.
Fixed in e40d3ba. The script now uses https_dataset_uri and hugging_face_auth_headers.
On the small-file branch: at a tag revision, Hugging Face answered HEAD for the 4 KB release manifest with a 200 and its Content-Length, which I checked, so that branch does run. _hf_size now also follows a redirect that carries no size.
| ) | ||
|
|
||
|
|
||
| def validate_dataset_overlays(bundle: Mapping[str, Any]) -> None: |
There was a problem hiding this comment.
14. The overlay-shadowing rule is written three times.
The three copies:
- the runtime
_apply_dataset_overlays; - this validator;
- the pruning in
certify_data_release.
They share no code. The certify copy's or name == data_release["default_dataset"] can never fire, because the default is always in datasets.
One helper here, which runtime code and scripts already import, would keep "certify output passes check" true by construction.
There was a problem hiding this comment.
Fixed in 90172bd. shadowed_dataset_overlays states the rule once. The runtime, the validator and certification all use it, and the default check that could never fire is gone.
| prefix = f"{country.country_id}: strict TRACE release" | ||
| certification = country.certification | ||
| core = bundle.get("packages", {}).get("policyengine-core", {}) | ||
| if certification and certification.compatibility_basis == "qualified_exact_pair": |
There was a problem hiding this comment.
15. Core drift is checked in two branches with two messages.
The exact-pair branch and the new elif compare the same pin and named core. The exact-pair message has no versions and no remedy.
One unconditional comparison, plus the exact-pair check that it names a pair and a core, raises on the same inputs (checked over 72 combinations). It would give every basis one message, and one test instead of two.
There was a problem hiding this comment.
Fixed in e691df0. One comparison now covers every basis and gives the versions and the remedy. The exact-pair branch only checks that the pair names its pair and its core. The exact-pair test now matches the shared message.
The strict check compared the core pin with the certificate's core in two branches, with two messages. One comparison now covers every basis and names the versions and the remedy; the exact-pair branch only checks that the pair names its pair and core. A named core binds the core pin, and Microcosm UK releases claim one exact core, so a core bump now needs Microcosm cuts that claim the new core, or a UK re-certification without --core-version. The skill and docs/bundles.md say so, and the skill's UK commands pass --core-version so the binding survives a re-certification done its way. The receipt tests set the named core they test instead of assuming the packaged certificate has one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rule that an overlay may not shadow a certified name was written three times: at runtime, in the maintenance validator, and in the pruning certification does. shadowed_dataset_overlays in the stdlib validator now states it, and all three use it. A pruned overlay does not come back, so a later certification that stops certifying its name (rolling forward after a rollback, or a ship-together run after a national-first one) dropped the name silently. Certification now warns about every name that stops resolving. The texts that said overlays always survive re-certification now say which ones do, and the skill has a re-add step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catch-up re-dispatches recoverable attempts without force, so even a forced run of a uk-data pair resolved as other_line and cleared nothing: it was re-dispatched every two hours for good. Recovery now skips every pair from another line, after the recoverability check so that live attempts log nothing. A forced cross-line run opens a candidate pull request that can never be promoted, and its body said merging promotes the pair. Promotion now closes the open pull requests of pairs from another line (the ledger keeps their evidence), and such a PR says it is never promoted. The runbook no longer says an unforced run is recorded. Tests that need a routed Microcosm main now read a frozen copy of the bundle this PR certifies, so a rollback of the packaged bundle does not break them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
release_consumers.py had its own URL and header helpers, which missed HUGGINGFACE_HUB_TOKEN and the shared helper's quoting. It now uses https_dataset_uri and hugging_face_auth_headers. _hf_size reads a small file's Content-Length directly, and follows a redirect that carries no size. The released-bundle request test now asserts the full repository, revision, path and repository type of both Hugging Face calls, and the release-consumer test asserts the URLs, the Authorization header and each HEAD answer the size reader handles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache-path guard validated with PurePosixPath but joined with the platform Path, so on Windows a backslash, drive-letter or UNC path passed the check and landed outside the data directory. Repository paths use forward slashes, so the guard now also refuses backslashes, drives and roots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preparation keeps main's regional release for a pair on main's data line, but that release was certified for main's model and core and gets no qualified-pair escape, so certification refused any other model or core deep inside certify-data. Preparation now refuses such a pair up front and says a local-area cut for its versions is needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weight matrices that no longer resolve by name are a removal, so they move to their own removed fragment, which makes the release a minor one; the switch itself stays a change. The change fragment now says that receipts for constituency and local-authority runs are uncertified, since the certificate covers the national dataset only, that the named core binds the core pin, and that 0.897 is against the floor's 6.7m benchmark, not DWP's 6.2m. regions.md and receipts.md say what certification does and does not cover for the local-area release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The metrics review and the scorecard replay measure a candidate against the previous certified pair, which a change of data line makes meaningless, so they were not run for the Microcosm switch. The skill now records that acceptance beside the UC-floor exception. Reform responsiveness checks the candidate against fixed minima, so it was run on the national cut. The UKDS output rule is its own paragraph again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed at 6bed939. All 14 inline findings have replies in their threads. This comment covers the rest.
The branch is still on main 5f97905. |
juaristi22
left a comment
There was a problem hiding this comment.
Re-reviewed at 6bed939.
Round 2:
- 13 of the 15 findings are fixed. I checked each fix.
- Finding 1 stays as designed: the escape is kept so that UK and US cores need not move together. The escape is re-certifying without
--core-version, after which receipts certify the pin. - Finding 4 is the cross-repo follow-up listed in the PR body.
New: 14 findings, most severe first. 14 are inline on lines in the diff.
-
Core binding: three text fixes. Inline on
generate_trace_tros.py:163. -
Promotion closes an in-flight off-line PR, and never closes one without a receipt. Inline on
uk_qualification.py:1634. -
Promote returns PRs it just closed to draft. Inline on
check_uk_candidate.py:992. -
No recorded acceptance for the two needs-review gates. Inline on
data-certification.md:227. -
The path guard misses a drive letter in a later segment. Inline on
dataset_materialization.py:98. -
The redirect branch can't get a size from Hugging Face. Inline on
release_consumers.py:86. -
The same-line check compares with main's certificate, not with the local-area release's claims. Inline on
uk_qualification.py:779. -
The request test still doesn't pin the per-dataset override. Inline on
test_uk_qualification_pipeline.py:3452. -
Nothing records that promotion now closes other-line PRs. Inline on
certify-uk-microcosm-uk-2024-25-national-20261019T083323Z-6cba5d71.changed.md:1. -
Stale text. Inline on
uk_qualification.py:1752. -
The new closing paths are barely tested. Inline on
uk_qualification.py:1644. -
The off-line close loop copies the superseded one. Inline on
check_uk_candidate.py:977. -
The overlay refusal has three wordings. Inline on
manifest.py:334. -
on_main_linedefaults toTrue. Inline onuk_qualification.py:1722.
| raise ValueError( | ||
| f"{prefix}: runtime core {core.get('version')} differs from the " | ||
| f"certified core {certification.certified_for_core_version}; " | ||
| "re-certify the data release for the new core, or without " |
There was a problem hiding this comment.
1. Core binding: three text fixes.
The escape stays as designed. Three texts around it are off:
- Exact pairs. This message offers "or without --core-version" for every basis. For a
qualified_exact_paircertificate, certify-data refuses that ("A qualified exact pair names its core version",certification.py:327-332). Dropping the pair id as well is also refused, unless a publisher claim covers the model. Suggest offering it only for the other bases. - Skill commands. The skill's main UK command (
data-certification.md:25-26) and the UK example indocs/bundles.md:138-141omit--core-version, while the other skill commands pass it. Say which is the default, and when to bind the core. - Wording.
regions.md:109-110says certification checks a local-area release's own core claim. It does that only with--core-version.bundles.md:131citescheck --strict-tros, which also needs--include-tros.
There was a problem hiding this comment.
Fixed in 6ee9157. The rule itself also changed, per María's ruling: the UK certificate now follows the pin, and says so.
- Default: without
--core-versiona certificate recordscore_binding: bundle_pinand names no core. Certification and the strict check warn, but do not refuse, when the pin is outside a release's claims. The TRO records the binding. - Binding:
--core-version(core_binding: certified) names and binds the core, and the strict check refuses drift. Its remedy now offers "without--core-version" only for bases other than an exact pair. An exact pair is told to requalify. - Docs:
- The skill's UK commands drop
--core-version. - A new "The core" section says which mode is the default and when to bind.
regions.mdandbundles.mdsay the same, with--include-tros.
- The skill's UK commands drop
- UK: re-certified this way, so it warns about nothing, because both releases claim the pin.
- US: the check now shows that the US release claims only 3.32.5. The pin was already past that on main.
| # ledger keeps its evidence. | ||
| receipts = list(receipts) | ||
| line = certified_data_line(bundle) | ||
| off_line = sorted( |
There was a problem hiding this comment.
2. Promotion closes an in-flight off-line PR, and never closes one without a receipt.
close_off_linetakes every open branch whose off-line pair has a published receipt. It never checks that the PR head is that receipt's commit; the target path does check (line 1696).- In-flight close. The runbook and the recovery notice both say to re-run an off-line pair with force. When that pair already has a receipt:
- The re-run pushes a new commit and opens a new draft PR.
- Its gates take hours. Any promote job in that time closes the PR: the cron runs every 2 hours, promote runs
if: always(), and it shares no lock with qualify. - The verdict step then finds no PR and exits 1.
- Never closed. An off-line attempt whose replay job dies before publishing keeps its PR open for good: recovery skips it, and promotion only sees receipts.
Suggest any of:
- close only when the open head equals
receipt.harness.git_commit, or when no newer attempt is live; - close receipt-less off-line PRs once their attempt is no longer live;
- let the verdict step tolerate a closed PR.
There was a problem hiding this comment.
- Closing: an other-line PR now closes only when its open head is the commit its receipt qualified. A forced re-run in flight at a newer head is left to its own receipt.
- Dead attempts: the PR of an attempt that dies before publishing a receipt stays open, and the runbook now says to close it by hand.
| ], | ||
| check=False, | ||
| ) | ||
| # Only the plan's target may be ready, and only while _promote finds it |
There was a problem hiding this comment.
3. Promote returns PRs it just closed to draft.
- The draft sweep below works from the PR listing taken before the closes.
- So a non-draft PR that was just closed gets
gh pr ready N --undo. gh refuses that for a closed PR, before any API call. The run then prints "could not return N to draft" and exits 1. - It needs a PR left ready, by a person or by an interrupted run. So it is rare with automatic promotion, and the superseded case already had it.
gh pr closefailures are also ignored (check=False).
Suggest skipping closed branches in the sweep (or un-drafting before closing), and warning when a close fails.
There was a problem hiding this comment.
Fixed in b25cb78. One _close_pull_requests helper closes the PRs, warns when a close fails, and returns the PRs it closed. The draft sweep skips those.
| - **Comparison gates.** The metrics review and the scorecard replay measure a | ||
| candidate against the previous certified pair, and a change of data line | ||
| moves the totals by design, so they were not run for that switch; its pull | ||
| request compares the two lines' weighted totals instead. The other gates, |
There was a problem hiding this comment.
4. No recorded acceptance for the two needs-review gates.
uk.json:38requires a named acceptance for a needs-review receipt, and the UC-floor entry here records one.- Reform responsiveness and budget-critical inputs both came back needs review. Neither this list nor the PR body records an acceptance, and "were run" reads as a pass.
Suggest one line for each gate. For example: "Reform responsiveness: needs review. The metrics tool's small-cell rule withholds a standard reform's figures, so the gate cannot recompute its verdict. Accepted by in #685."
There was a problem hiding this comment.
Recorded in 0f5e05b. Both acceptances are in the skill, next to the UC-floor exception, and in the PR body.
- Inputs: the line gives the reason. The enhanced FRS column carries no values, because Incapacity Benefit is closed, so both datasets give zero.
- Responsiveness: the line names no reform and gives no counts.
| relative = PurePosixPath(path) | ||
| # Repository paths use forward slashes. A backslash, drive or root would | ||
| # make the platform path leave data_dir on Windows. | ||
| windows = PureWindowsPath(path) |
There was a problem hiding this comment.
5. The path guard misses a drive letter in a later segment.
PureWindowsPath(path).driveand.rootlook only at the first segment, and the join then restarts at a later drive.- So on Windows,
releases/D:x.h5ora/b/E:/x.h5lands outsidedata_dir. With the default./data, any drive letter does. - A drive in a middle segment stayed inside under main's basename code, so that part is new.
Suggest checking the joined path (destination.is_relative_to(data_dir) and destination != data_dir, as verify._resolve_local does), or refusing any segment that has a drive. Add these shapes to the test.
There was a problem hiding this comment.
Fixed in 1fda069.
- The guard now refuses a drive or root in any segment.
- It also checks that the joined path stays inside the data directory.
- The test has your two shapes, plus an empty path.
| verdict = receipt.verdict if receipt else "pending" | ||
| if on_main_line: | ||
| outcome = ( | ||
| "Merging this PR promotes the pair. Until cos d1344 is ruled, " |
There was a problem hiding this comment.
10. Stale text.
"Until cos d1344 is ruled, promotion is a reviewed merge": uk.json records d1344 as ruled on 15 October, with automatic promotion on. The sentence moved unchanged into the new branch. Suggest dropping the clause.
| ) | ||
| receipts = _same_line_receipts(receipts, bundle) | ||
|
|
||
| def plan(**fields: Any) -> PromotionPlan: |
There was a problem hiding this comment.
11. The new closing paths are barely tested.
- Only the no-target plan asserts
close_off_line. - No test mixes a same-line target with an off-line open PR, runs
promote --applyon a Microcosm main, or runsfinalize --pr-bodyfor an off-line pair. - So deleting the close loop, or forcing
on_main_line=True, still passes the suite. - The
plan(**fields)closure also means a future directreturn PromotionPlan(...)would drop the closes. Computing the off-line set once, shared with_same_line_receipts, would avoid that.
There was a problem hiding this comment.
Fixed in b25cb78.
- New tests:
- a same-line target beside an other-line PR, which is closed only at its receipt commit;
promote --applyon a Microcosm main, where the PR is closed and not returned to draft;finalize --pr-bodyfor an other-line pair.
- Refactor: the plan now splits receipts by data line once, and the same-line plan returns plain
PromotionPlans, so no return can drop the closes.
| ], | ||
| check=False, | ||
| ) | ||
| for branch in plan.close_off_line: |
There was a problem hiding this comment.
12. The off-line close loop copies the superseded one.
It copies the loop above it, including the ignored exit code and a numbers.get(branch, branch) fallback that can't fire. One helper, or one loop over (branch, comment) pairs, would also be the place for finding 3's fixes.
There was a problem hiding this comment.
Fixed in b25cb78. There is now one helper over (number, comment) pairs, and it uses numbers[branch].
| "additive, not overrides." | ||
| ) | ||
| certified_datasets[overlay_name] = overlay_reference | ||
| if default_dataset in shadowed: |
There was a problem hiding this comment.
13. The overlay refusal has three wordings.
There is one helper now, but three messages. The runtime keeps separate default and collision messages and names only the first shadowed overlay; the validator names them all. One message naming every shadowed overlay would do.
There was a problem hiding this comment.
Fixed in 1fda069. shadowing_overlays_message is now the one refusal. It names every shadowed overlay, both at runtime and in the check.
| *, | ||
| ledger_url: str, | ||
| prepared: Optional[Mapping[str, Any]] = None, | ||
| on_main_line: bool = True, |
There was a problem hiding this comment.
14. on_main_line defaults to True.
No caller uses the default, and it is the misleading value for a caller that forgets the flag. Suggest making it keyword-required with no default. cmd_finalize also builds the body when --pr-body is absent.
There was a problem hiding this comment.
Fixed in b25cb78. on_main_line is now keyword-required with no default, and finalize builds the body only when --pr-body is given.
The bundle has one core pin, shared by both countries. A certificate that names no core follows it, so another country can move the pin, but until now that showed only as a missing field, and nothing compared the pin with the data release's own core claims. A certificate now says how it covers the core: core_binding bundle_pin (the default, no --core-version) or certified (--core-version names and binds it). Certification checks the bundle's pin against each release's build core and claims and warns when it is outside them; the strict TRO check warns the same way later, and refuses only a pin that differs from a named core. The TRO records the binding. UK is re-certified to follow the pin: both Microcosm releases claim 3.33.3, the pin, so it warns about nothing. The check now also shows that the US release claims only 3.32.5, which the pin has been past since before this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… routing - Promotion closed an other-line PR whenever its pair had a receipt, even while a forced re-run was in flight at a newer head, whose verdict step then found no PR. It now closes such a PR only at the commit its receipt qualified. The plan splits receipts by data line once, and the same-line plan returns plain PromotionPlans, so no return can drop the closes. - Promote closed PRs and then tried to return them to draft, which gh refuses. One helper now closes PRs, warns when a close fails, and the draft sweep skips what it closed. - candidate_pr_body's on_main_line has no default, finalize builds the body only when asked for it, and the stale cos d1344 clause is gone. - Preparation now refuses a same-line pair on a routed main outright: the routing would need a local-area cut for the pair's versions, which a candidate does not carry, and the claims check it replaced compared with main's certificate rather than the cut. New tests cover a same-line target beside an other-line PR, promote --apply on a Microcosm main, and finalize for an other-line pair. The frozen routed bundle is refreshed to this PR's certificate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The overlay rule had one helper but three messages, one naming only the first shadowed overlay. shadowing_overlays_message is now the one refusal, naming every shadowed overlay, at runtime and in the check. - The cache-path guard looked for a drive only in the first segment, and the join restarts at a later one on Windows. It now refuses a drive or root in any segment and checks that the joined path stays inside the data directory. - requests asks for compressed answers, which carry no Content-Length, so the size HEADs now ask for the identity encoding, as huggingface_hub does. - The request test now has a dataset entry with its own repository, revision and type, and asserts both calls and the whole data block literally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The data certification skill records María Juaristi's acceptance of the two needs-review gates: the missing incapacity_benefit_reported, whose enhanced FRS column carries no values, and the reform the small-cell rule withholds from reform responsiveness. The changelog says how certificates now cover the core, that promotion closes other-line PRs, and that recovery no longer re-dispatches them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed at 12cfbc0. All 14 findings have replies in their threads.
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes #553
This PR makes Microcosm UK 2024-25 the UK default dataset, in place of the Enhanced FRS 2024-25. Constituency and local-authority regions use the Microcosm local-area release. It uses the tooling from #615.
This is an interim certification. Both Microcosm cuts were built with policyengine-uk 2.132.6, so this PR pins that version instead of a newer one. A Microcosm pair built on policyengine-uk 2.134.9 or later is planned within about a week, and it will replace this pair. "Known gaps until the next pair" below lists what that pair is expected to change.
What changes
enhanced_frs_2024_25from policyengine-uk-data 1.58.1.microcosm_uk_2024_25, national cutmicrocosm-uk-2024-25-national-20261019T083323Z-6cba5d71.region_datasets.constituencyandregion_datasets.local_authorityroute tomicrocosm_uk_2024_25_dense, local-area cutmicrocosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe(regional_release_kind: local_area).compatibility_basis: built_with_model_package).core_binding: bundle_pin), so the pin can move with the US line. Both Microcosm releases claim 3.33.3. If the pin moves outside a release's claims, certification and the strict check warn rather than refuse. The skill's new "The core" section has the rule.uv.lockwas re-resolved with uv 0.12.14, the version CI uses. Only these two packages move.enhanced_frs_2024_25,frs_2024_25and their_tinyvariants stay loadable by name, throughdataset_overlays.ukentries at 1.58.1.constituency_code_oaandla_code_oaon the local-area file, which carries both columns.How it was made
mainat 5f97905 (6.9.0). The certification was re-run there, so the 6.9.0 stamps and both TROs are regenerated.uv lock(uv 0.12.14), thenrelease_lock.py.dataset_overlays.ukentries were added.bundle.py generate --include-tros.bundle.py check --published-spm --include-tros --strict-trospasses.test_models,test_release_manifests,test_uk_regions,test_bundle,test_bundle_metadata,test_dataset_runtime).other_line, so they now run against a frozen copy of the last bundle certified on that line (tests/fixtures/uk_qualification/uk_data_line_bundle.json, main's 6.9.0 release). One sharedload_candidate_cligives the candidate CLI that bundle. The constant-rewrite test still uses the packaged bundle, and takes it to a successor Microcosm cut.countries.md,microsim.mdandregions.mddescribe the new default.Review fixes
These commits address the review at 8bd01dd, in this order:
bundle.py checkrefuses an overlay that shadows a certified name. This keeps a rollback importable.These commits address the second review, at 02ec73e:
docs/bundles.mdsay what a named core binds, and the skill's UK commands pass--core-version.bundle.py checkand by certification. Certification also warns about every name that stops resolving.These commits address the third review, at 6bed939:
core_binding: bundle_pinis the default and follows the pin;certifiednames and binds the core. Certification and the strict check warn when the pin is outside a release's claims. UK is re-certified to follow the pin.Acceptance evidence
Budget reform contract 2.1.0 (#574's qualification gate)
This was run on the national cut with policyengine-uk 2.132.6 and policyengine-core 3.33.3.
extended_freeze, in 2034-35. They check that indexation resumes after the freeze, which needs policyengine-uk 2.134.9 or later. The 2.130.1 pin onmainhas the same gap. We accept it for this interim pair.Other #574 gates
These were also run on the national cut with policyengine-uk 2.132.6.
incapacity_benefit_reported. The enhanced FRS column carries no values, because Incapacity Benefit is closed, so both datasets give zero.Universal Credit coverage floor (#491)
The floor compares 2025 values for Great Britain with two benchmarks.
We accept this result. The floor's benchmarks are not the figures that Microcosm calibrates to:
This is not the failure that #491 describes, where a dataset carried about half of the caseload. The benchmark definitions will be settled when the UK data checks are consolidated with the data pipelines, which is a separate piece of work.
Weighted national totals, 2025, UK (£bn)
Moving policyengine-uk from 2.130.1 to 2.132.6 on the enhanced FRS changes all ten totals by 0.0%. So this table shows only the data change.
All figures are weighted totals over the whole population.
Known gaps until the next pair
On the same national cut, policyengine-uk 2.134.9 changes most totals by less than 1%. The exceptions:
Also missing until then:
extended_freezeindexation after 2033;After this merges
microcosm-data,policyengine/populace-uk-private). Automated policyengine-uk-data candidates are thenother_line: an unforced run stops there, and none is ever promoted.mainthe pin was already 3.33.0, so this was already the case; the warning makes it visible.removedfragment makes this a minor release.uk_area_regions_routed()holds.size_bytes.size_bytesfor the pinned cut, so it needs a fallback of its own.policyengine/populace-uk-private. That access is not yet checked.policyengine[models]==6.7.5. Itsuk_constituency_breakdownanduk_local_authority_breakdownapply the enhanced FRS weight matrices frompolicyengine/policyengine-uk-datato the default dataset's households. They must change before policyengine-api takes a release with this bundle.Checks
bundle.py check --published-spm --include-tros --strict-trosandrelease_lock.pypass.🤖 Generated with Claude Code