Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,19 @@ jobs:
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13'
run: git diff --exit-code -- docs/reference/validation.md

- name: Truth report (ubuntu / 3.13 only; a report, not a gate)
# How closely the methods recover answers known by construction. Kept as an
# artifact on every run so the figures behind validation.md's last section are at hand.
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13'
run: python validation/truth_report.py --out truth_report.md

- name: Upload the truth report
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.13'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: truth-report
path: truth_report.md

- name: Upload coverage to Codecov
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **`examples/` and `validation/effects_visualization.ipynb`.** These demos predated the 0.2.0
API (`ProcessDataFrame`, `study.sds`, moved data files) and no longer ran; nothing linked to
them. The maintained, CI-executed tutorials are in `docs/tutorials/`.
- **`tests/test_bishop_reference.py`.** Its one test compared residual values with a workbook
of Dr. Bishop's that is not in the repository, so it skipped on every run. Residual charts are
still compared with VAS through their limits.

### CI / Infrastructure
- **Python 3.14 is tested** on Ubuntu and Windows, in CI and in the release gate, and listed in
the package classifiers.
- **The truth report runs in CI.** `validation/truth_report.py` runs on every push (Ubuntu,
Python 3.13) and its output is kept as the `truth-report` artifact.

## [0.5.0] - 2026-10-29

Expand Down
8 changes: 1 addition & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,13 +74,7 @@ The full test suite is the gate: if it fails, the change is not ready.
The file `validation/PBTESTDATABASE_T100.csv` is ground-truth Minitab output
for DS 1–6 from Bishop's reference materials. Numerical changes that touch
residuals (R1–R5), DS detection, limit calculations, or variance
decomposition must keep the relevant Bishop-reference tests green:

```bash
pytest tests/test_bishop_reference.py -v # skips unless Bishop's residual workbooks are present
```

There is also a full end-to-end check that compares every chart center, control
decomposition must keep the end-to-end check green. It compares every chart center, control
limit, capability index and loss component against Bishop's VAS runs for the
10-12-2026 manual (PM SDS 1–6, PM INERT, his known-effects data and the Medicare data):

Expand Down
6 changes: 3 additions & 3 deletions docs/reference/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,7 @@ These outputs have no VAS reference in this run. They are covered by unit tests,
other references, but no VAS value is checked:

- **Residual values one by one.** Charts of R1-R6 are compared through their limits above;
`tests/test_bishop_reference.py` compares the values themselves when the reference
spreadsheets are present, which they are not in this repository, so it skips.
the values themselves are not compared.
- **Signals** (run rules and points beyond the limits).
- **Staged (phased) Xbar and S charts.** These are checked against Wheeler's cavity data and
Minitab's per-stage limits in `tests/test_phased_xbar_s.py`.
Expand All @@ -64,7 +63,8 @@ other references, but no VAS value is checked:

`validation/truth_report.py` asks a different question: how closely the methods, as
implemented, recover answers known by construction (pure noise, and data built from known
effects). It is a report, not a gate.
effects). It is a report, not a gate. CI runs it on every push and keeps its output as the
`truth-report` artifact.

## Values with no reference yet

Expand Down
82 changes: 0 additions & 82 deletions tests/test_bishop_reference.py

This file was deleted.

6 changes: 3 additions & 3 deletions validation/e2e_bishop_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -870,8 +870,7 @@ def generate_myst_summary(all_results, aux_by_run, runs):
'other references, but no VAS value is checked:',
'',
'- **Residual values one by one.** Charts of R1-R6 are compared through their limits above;',
' `tests/test_bishop_reference.py` compares the values themselves when the reference',
' spreadsheets are present, which they are not in this repository, so it skips.',
' the values themselves are not compared.',
'- **Signals** (run rules and points beyond the limits).',
"- **Staged (phased) Xbar and S charts.** These are checked against Wheeler's cavity data and",
" Minitab's per-stage limits in `tests/test_phased_xbar_s.py`.",
Expand All @@ -882,7 +881,8 @@ def generate_myst_summary(all_results, aux_by_run, runs):
'',
'`validation/truth_report.py` asks a different question: how closely the methods, as',
'implemented, recover answers known by construction (pure noise, and data built from known',
'effects). It is a report, not a gate.',
'effects). It is a report, not a gate. CI runs it on every push and keeps its output as the',
'`truth-report` artifact.',
'',
'## Values with no reference yet',
'',
Expand Down
Loading