Skip to content

Run the truth report in CI as an artifact; drop the test that could never run - #169

Merged
cnicholas merged 2 commits into
mainfrom
ci/truth-report-artifact
Nov 3, 2026
Merged

cnicholas merged 2 commits into
mainfrom
ci/truth-report-artifact

Conversation

@cnicholas

Copy link
Copy Markdown
Owner

Summary

Contract / Invariants (must remain true)

  • Default behavior unchanged: nothing under processbehavior/ changes; Bishop gate 1547/0/4
  • Residuals unaffected: no library change
  • Row/index alignment preserved: no library change
  • Lane boundaries preserved: no library change
  • Output schema compatible: no library change

Behavior Changes (explicit)

  • New API: none
  • New semantics (CI and docs):
    • every CI run on ubuntu/3.13 keeps truth_report.md as the truth-report artifact (the step takes under a second)
    • validation.md: residual values one by one are "not compared"; the truth report section says CI keeps its output
    • CONTRIBUTING's validation section points to the end-to-end check only

Key Design Decisions

  • A report, not a gate: the step never fails on the report's numbers; it fails only if the script errors
  • Same pinned action as the package job (actions/upload-artifact@043fb46… # v7)
  • Drop rather than keep skipping: a test that can never run in CI only adds a skip to every run; residual charts stay compared with VAS through their limits

Risks & Mitigations

  • P0 risks: none to numbers
  • Mitigations: the regenerated validation.md is committed, so the drift step passes

Tests

Added / Updated

  • removed tests/test_bishop_reference.py (one parametrized test, always skipped)

Regression Focus (must fail if broken)

  • validation.md drift step still guards the page

Manual Verification

  • pytest tests/: 2688 passed, 6 skipped (one fewer skip)
  • python validation/e2e_bishop_report.py: 1547 passed, 0 failed; validation.md regenerated
  • python validation/truth_report.py --out truth_report.md runs in under a second
  • ci.yml parses; ruff check . clean

Notes

  • Docs: validation.md, CONTRIBUTING.md; CHANGELOG [Unreleased] Removed and CI / Infrastructure
  • Follow-ups (not in this PR): none

…ever run

## Summary
- **What:** CI runs `validation/truth_report.py` on every push and uploads its output as the `truth-report` artifact; `tests/test_bishop_reference.py` is removed.
- **Why:** the review's action A3 asked for the truth report to run in CI as an artifact, and to resolve the skipped residual test; #162 committed the script but nothing ran it. The test needed a workbook of Dr. Bishop's (`cn-tab comparison sds 1 (3-5-26).xlsx`) that is not in the repository, so it skipped on every run.
- **Scope:** ci.yml (two steps), the validation page generator and the regenerated `validation.md`, CONTRIBUTING.md, CHANGELOG. Stacked on #167.

## Contract / Invariants (must remain true)
- [x] **Default behavior unchanged:** nothing under `processbehavior/` changes; Bishop gate 1547/0/4
- [x] **Residuals unaffected:** no library change
- [x] **Row/index alignment preserved:** no library change
- [x] **Lane boundaries preserved:** no library change
- [x] **Output schema compatible:** no library change

## Behavior Changes (explicit)
- **New API:** none
- **New semantics (CI and docs):**
  - every CI run on ubuntu/3.13 keeps `truth_report.md` as the `truth-report` artifact (the step takes under a second)
  - validation.md: residual values one by one are "not compared"; the truth report section says CI keeps its output
  - CONTRIBUTING's validation section points to the end-to-end check only

## Key Design Decisions
- **A report, not a gate:** the step never fails on the report's numbers; it fails only if the script errors
- **Same pinned action** as the package job (`actions/upload-artifact@043fb46… # v7`)
- **Drop rather than keep skipping:** a test that can never run in CI only adds a skip to every run; residual charts stay compared with VAS through their limits

## Risks & Mitigations
- **P0 risks:** none to numbers
- **Mitigations:** the regenerated validation.md is committed, so the drift step passes

## Tests
### Added / Updated
- [x] removed `tests/test_bishop_reference.py` (one parametrized test, always skipped)

### Regression Focus (must fail if broken)
- [x] validation.md drift step still guards the page

## Manual Verification
- [x] `pytest tests/`: 2688 passed, 6 skipped (one fewer skip)
- [x] `python validation/e2e_bishop_report.py`: 1547 passed, 0 failed; validation.md regenerated
- [x] `python validation/truth_report.py --out truth_report.md` runs in under a second
- [x] ci.yml parses; `ruff check .` clean

## Notes
- **Docs:** validation.md, CONTRIBUTING.md; CHANGELOG [Unreleased] Removed and CI / Infrastructure
- **Follow-ups (not in this PR):** none
@cnicholas
cnicholas changed the base branch from chore/retire-examples to main November 3, 2026 15:04
@cnicholas
cnicholas merged commit 7813a4d into main Nov 3, 2026
19 checks passed
@cnicholas
cnicholas deleted the ci/truth-report-artifact branch November 3, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant