Validate implicit dataset role capacities and role array shapes

[MaxGhenis]

Summary

Relative to current master, this PR rejects recognized implicit dataset defaults that exceed a role's finite per-group capacity and rejects non-vector or wrong-length role arrays during joining. Master already assigns one literal default key per person. Accepted defaults keep their role meaning, cardinality, query results, and serialization.

An earlier round integrated the overlapping change from merged #573 through a plain merge of gh/master at 8fdf59e95a2faa990ed179f8554b722978024836. That merge preserved master's person-count default sizing and detection of period-suffixed entity role columns, and added regression coverage for their precedence. The caller comparison now uses this master baseline. Capacity enforcement newly rejects some datasets whose queries previously succeeded; it does more than move an existing failure earlier. The fallback that selected a first role or subrole was removed in the previous round and remains removed.

Caller semantics

Unless stated otherwise, these examples use the country template's household roles and no role column.

Caller input and operation	Current master	This PR
Persons [1,2,3], household [10], memberships [10,10,10], default "child"; nb_persons(role=CHILD)	Three CHILD roles; [3]	Unchanged: [3]
Same memberships, default "member" or compound key "parent"; parent count and parent sum of [1,2,3]	Unrecognized roles [0,0,0]; count [0], sum [0]	Unchanged: no literal flattened-role key matches; 0 + 0 + 0 = 0
Persons [1,2], households [10,20], memberships [10,10], default "first_parent"; unique-role lookup of [100,200]	Construction accepts two unique-role holders; lookup raises ValueError	Construction raises ValueError because 2 > 1
Noncompound member role with max=2; persons [1,2,3], group [10], memberships [10,10,10]; member count and filtered sum of [1,2,3]	Construction succeeds; count [3], sum [6], since 1 + 2 + 3 = 6	Construction raises ValueError because 3 > 2; no query runs
Direct join with three memberships and roles ["child"]	Stores a one-entry role array; count raises, while project(np.array([7]), role=CHILD) broadcasts to [7,7,7]	Join raises ValueError; non-vector arrays are also rejected

Entity-specific role columns, including period-suffixed flat-file columns, still override generic role, which overrides the implicit default. Explicit integer indices and literal strings retain their interpretation. default_role=None still requires a role column; one global default key still applies to every group entity. Well-shaped explicit dataset assignments, direct-join capacity handling, situation defaults, and situation capacity validation remain unchanged. Only implicit defaults receive the new capacity check; excessive explicit capacities are not newly rejected.

The role-shape guard checks role-array dimensionality and equality with the supplied membership array's length before group-ID mapping. The join also retains master's declared-ID mapping and membership validation from #592; the role-shape guard does not independently validate membership dimensionality or agreement with the declared person count. Preserving master's person-count default sizing means a malformed implicit dataset with a different membership count is rejected by this role-length check.

subsample() can now raise the default-role capacity error when the system has no person_<entity>_role variable, even if explicit roles were supplied initially, because the rebuild loses those roles and falls back to the default; the older role loss is separate work.

Methodology

• Literal defaults: retain master's literal flattened-role-key matching, including unrecognized unmatched and compound keys. Selecting a first role or subrole is the alternative. Accepted assignments and their cardinality do not change relative to current master.
• Bounded implicit defaults: reject uniform defaults exceeding finite per-group capacity. Selecting holders, assigning different subroles, or changing their roles are alternatives. This changes which datasets construction accepts, including datasets with previously successful queries; it does not choose which person receives a role.
• Validation scope: reject non-vector or membership-length-mismatched role arrays and retain existing explicit-capacity handling. Rejecting excessive explicit capacities too is the alternative. Malformed role arrays fail during joining relative to master; well-shaped explicit assignments retain their existing interpretation.

These are Core simulation contracts. No statute, regulation, form, or tax-policy instruction applies. Primary sources are master's defaults and role-column precedence, literal flattened-role matching, the capacity descriptor, subrole uniqueness, situation capacity enforcement, and the unique-role lookup contract.

Invariants and coverage

For well-formed datasets without role columns, with defined group roles, string default keys, and one-dimensional membership arrays containing exactly one declared group ID per person, across ARRAYS, TIME_PERIOD_ARRAYS, and FLAT_FILE:

• Accepted defaults produce len(members_role) == len(members_entity_id) == number of persons, including zero persons and empty declared groups.
• A literal flattened key assigns that role uniformly; unmatched and compound keys retain the unrecognized representation. No first-role or first-subrole substitution occurs.
• Recognized bounded implicit defaults are accepted only when every group's holder count is at most Role.max. Capacities zero, one, and greater than one have coverage.
• Within the group-ID limits below, masks, counts, sums, projections, and unique/partner operations match independent role-table and raw-membership oracles. For nonempty person arrays, maxima also match the independent oracle; zero-person maxima are outside this default-role property oracle. Dump/restore example tests preserve recognized and unrecognized role arrays.

For datasets with explicit roles, entity roles take precedence over generic roles and defaults, including period-suffixed flat-file columns. The new Hypothesis property verifies this with integer and string encodings, conflicting generic roles, and repeated unique explicit roles whose capacity handling remains unchanged.

Hypothesis generated 200 default-role/capacity/query cases, 50 malformed-role-shape cases, and 100 suffix-precedence cases, plus eight fixed examples. Its default-query generator places empty groups after occupied groups in sorted ID order, and its flat-file cases use consecutive occupied IDs. These default-role properties retain that coverage scope; the join-identity and ID-renumbering behavior fixed by #592 has separate example and property coverage, both rerun after this merge.

Merge with master after #592

Merged gh/master at 8e9725c8e6192bafa75dee005da13b27df41cf33 into the approved head ef0f9d5e2c59c791c33ed24adacbb0971a42824b with a plain merge, without rebasing. Master's #592 and this PR both changed join_with_persons in policyengine_core/simulations/simulation_builder.py. The merge keeps #592's value-based group_positions lookup with exact cross-dtype matching, rejection of repeated declared IDs, undeclared memberships, unmatchable kinds and NaN memberships, and flat-file ID renumbering. It also keeps this PR's role-array dimensionality and membership-length validation before ID mapping, so malformed roles still receive the role-shape error, and the finite-capacity check for recognized implicit default roles in build_from_dataset.

All six requested files passed separately in the foreground with uv run pytest -q <file>, Python 3.14.4, and the required four-thread caps:

File	Result
tests/core/test_join_with_persons_identity.py	23 passed
tests/core/test_join_with_persons_identity_property.py	6 passed
tests/core/test_dataset_default_role.py	41 passed
tests/core/test_dataset_default_role_property.py	3 passed
tests/core/test_simulation_builder.py	35 passed
tests/core/test_subsample_structural_ids.py	27 passed

Total: 135 passed. The default-role example file includes six added cases proving that malformed-role errors take precedence over repeated or undeclared group IDs. ruff format --check and ruff check passed on the resolved builder and strengthened test file; both staged and unstaged git diff --check passed, and no tracked conflict markers or unmerged index entries remain. The dependency setup's incidental uv.lock version refresh is excluded from the merge.

Earlier validation (before this merge)

Every file below ran separately in the foreground with Python 3.13.9 through uv run --no-sync. A workspace-local runner reused preinstalled hub dependencies read-only while the workspace dependency install completed; tested Core source came from this checkout, or the pinned master export for baseline witnesses.

• tests/core/test_dataset_default_role.py: 35 passed.
• tests/core/test_dataset_default_role_property.py: 3 passed, with 350 passing generated cases and eight fixed examples.
• tests/core/test_simulation_builder.py: 35 passed.
• tests/core/test_bincount_minlength.py: 2 passed.
• tests/core/test_entities.py: 23 passed.
• tests/core/test_projectors.py: 6 passed.
• tests/core/test_dump_restore.py: 1 passed.
• tests/core/test_dataset_unknown_column_warning.py: 2 passed.
• Total: 107 passed across eight files.
• Pinned-master witnesses: 3 passed. These reproduce the three-child count, successful excessive-capacity count/sum, and successful broadcasting projection from a short role array. The initial projection fixture lacked simulation links; it was corrected before the successful baseline run.
• At the requested 9ba5d19 head, both new suffix regressions failed: absent generic roles caused the capacity error; conflicting generic roles incorrectly won. Both pass after the merge in the complete example file. The merge preview and actual merge both reproduced the default-branch conflict.
• Ruff formatted the four changed Python files without further edits; ruff format --check .: 397 files already formatted; ruff check .: passed.
• git diff --check: passed.

No full suite, microsimulation, or documentation build was run locally for this focused round. API documentation uses autodoc; the loader and join source docstrings describe the durable contract. The towncrier fragment now describes the residual capacity and shape changes relative to current master. Documentation review: impact medium, confidence high from source inspection, baseline witnesses, and focused coverage; zero-person maxima remain outside the default-role property oracle, while master's #592 resolves the earlier join-identity limitation.

[MaxGhenis]

US + core hub merge audit, core#595 at 8f58c5d9be83bfa97bec0eb4f7c6c52b779d7a86 (squash): validate implicit dataset role capacities and role array shapes.

• Review:
  • A GPT-6.1 Sol shell review (20261018-121200-hub-core595-review-sh) APPROVED ef0f9d5e2c with no P1 or P2. Every new rejection test failed at the merge base with DID NOT RAISE ValueError and passed at the head.
  • Join each person to the group their ID names #592 then merged and rewrote the same join_with_persons. A hub build merged master and resolved the conflict, running Validate implicit dataset role capacities and role array shapes #595's role-shape check before Join each person to the group their ID names #592's value-based group_positions.
  • A Sol merge review (20261018-195516-hub-core595-review-merge) APPROVED this exact head. Both PRs' contracts survive. Six added cases pin the validation order: each would fail if the mapping ran first. Both PRs' files pass at the merge (135 cases).
  • The change: implicit default roles over Role.max, and malformed role arrays, now raise at build instead of producing corrupt simulations. This is a bug fix with one defensible reading.
• PE-US gate: PE-US main 83e03a92f6 against this head, which contains master 8e9725c8e6: 25 core test files, 494 passed, rc 0.
• Gates checked live at merge:
  • every check passes (19 of 19);
  • MERGEABLE;
  • not a draft;
  • no CHANGES_REQUESTED review;
  • head pinned.
• Impact: zero for shipped country packages, by the review's verification of their code:
  • PE-US and UK default to the member role, which has no max, so the capacity check can't fire.
  • The CA, IL and NG parent default matches no role.
  • PE-US and policyengine.py pass person-length role arrays.
  • The full-sample control run is queued on the hub's heavy lock, to confirm.
• Follow-ups: the older subsample role loss (task_9ea1b19d), and a clearer error message for 2-D role arrays (P3).

[MaxGhenis]

Hub control after merge (confirmation). A full-sample A/B ran on PE-US's default dataset for 2026, with core base=8fdf59e95a2faa990ed179f8554b722978024836 pr=ef0f9d5e2c59c791c33ed24adacbb0971a42824b us=83e03a92f6e7934b0a4af86d6425c5d272af409a. It found 0 records changed and 0 change in household_net_income, income_tax, state_income_tax and snap, matching the merge audit's zero-impact finding. The files are in fixes/core595-impact/ on the hub host.