Repository navigation
Run the truth report in CI as an artifact; drop the test that could never run - #169
Merged
Merged
Conversation
…ever run ## Summary - **What:** CI runs `validation/truth_report.py` on every push and uploads its output as the `truth-report` artifact; `tests/test_bishop_reference.py` is removed. - **Why:** the review's action A3 asked for the truth report to run in CI as an artifact, and to resolve the skipped residual test; #162 committed the script but nothing ran it. The test needed a workbook of Dr. Bishop's (`cn-tab comparison sds 1 (3-5-26).xlsx`) that is not in the repository, so it skipped on every run. - **Scope:** ci.yml (two steps), the validation page generator and the regenerated `validation.md`, CONTRIBUTING.md, CHANGELOG. Stacked on #167. ## Contract / Invariants (must remain true) - [x] **Default behavior unchanged:** nothing under `processbehavior/` changes; Bishop gate 1547/0/4 - [x] **Residuals unaffected:** no library change - [x] **Row/index alignment preserved:** no library change - [x] **Lane boundaries preserved:** no library change - [x] **Output schema compatible:** no library change ## Behavior Changes (explicit) - **New API:** none - **New semantics (CI and docs):** - every CI run on ubuntu/3.13 keeps `truth_report.md` as the `truth-report` artifact (the step takes under a second) - validation.md: residual values one by one are "not compared"; the truth report section says CI keeps its output - CONTRIBUTING's validation section points to the end-to-end check only ## Key Design Decisions - **A report, not a gate:** the step never fails on the report's numbers; it fails only if the script errors - **Same pinned action** as the package job (`actions/upload-artifact@043fb46… # v7`) - **Drop rather than keep skipping:** a test that can never run in CI only adds a skip to every run; residual charts stay compared with VAS through their limits ## Risks & Mitigations - **P0 risks:** none to numbers - **Mitigations:** the regenerated validation.md is committed, so the drift step passes ## Tests ### Added / Updated - [x] removed `tests/test_bishop_reference.py` (one parametrized test, always skipped) ### Regression Focus (must fail if broken) - [x] validation.md drift step still guards the page ## Manual Verification - [x] `pytest tests/`: 2688 passed, 6 skipped (one fewer skip) - [x] `python validation/e2e_bishop_report.py`: 1547 passed, 0 failed; validation.md regenerated - [x] `python validation/truth_report.py --out truth_report.md` runs in under a second - [x] ci.yml parses; `ruff check .` clean ## Notes - **Docs:** validation.md, CONTRIBUTING.md; CHANGELOG [Unreleased] Removed and CI / Infrastructure - **Follow-ups (not in this PR):** none
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
validation/truth_report.pyon every push and uploads its output as thetruth-reportartifact;tests/test_bishop_reference.pyis removed.cn-tab comparison sds 1 (3-5-26).xlsx) that is not in the repository, so it skipped on every run.validation.md, CONTRIBUTING.md, CHANGELOG. Stacked on Retire the examples/ demos and the effects notebook that no longer run #167.Contract / Invariants (must remain true)
processbehavior/changes; Bishop gate 1547/0/4Behavior Changes (explicit)
truth_report.mdas thetruth-reportartifact (the step takes under a second)Key Design Decisions
actions/upload-artifact@043fb46… # v7)Risks & Mitigations
Tests
Added / Updated
tests/test_bishop_reference.py(one parametrized test, always skipped)Regression Focus (must fail if broken)
Manual Verification
pytest tests/: 2688 passed, 6 skipped (one fewer skip)python validation/e2e_bishop_report.py: 1547 passed, 0 failed; validation.md regeneratedpython validation/truth_report.py --out truth_report.mdruns in under a secondruff check .cleanNotes