[cnicholas]

Summary

• What: a script, scripts/refresh_notebooks.py, that re-executes the docs notebooks; CI runs it with --check in place of nbmake; the notebooks are refreshed.
• Why: the docs site shows each notebook's saved outputs without running them, and nbmake only checked that they execute. Eight tutorials had drifted (limits from before 0.4.0 in the Design-State-1 analysis; flat Xbar limits where they now vary with n in the reference validation; older process means, lane charts and signal counts), and Derived Variables had never been executed, so the site showed its code with no output.
• Scope: the script and its tests; ci.yml (one step); the [test] extra; nine refreshed notebooks with three prose fixes; one plotting fix the check exposed (below); CONTRIBUTING.md; CHANGELOG. Stacked on Make signal rules 2, 5 and 6 match their definitions; correct the WECO rates #168, whose signal-rule fixes the refreshed outputs show.

Contract / Invariants (must remain true)

• Default behavior unchanged: no computed number changes; Bishop gate 1547/0/4. One plotting fix: the factor x time interaction chart's lines (legend and colours) are now in factor order on every machine
• Residuals unaffected: no library change
• Row/index alignment preserved: no library change
• Lane boundaries preserved: no library change
• Output schema compatible: no library change; notebook code cells and metadata unchanged

Behavior Changes (explicit)

• New API: python scripts/refresh_notebooks.py [PATH ...] [--check]
• New semantics:
  • CI executes every docs notebook (globbed, no hand-kept list) and fails if what a reader sees differs from what is saved: printed text, errors, tables, and each figure's data, title, axis titles, annotations and shape positions
  • Ignored as noise: the temporary path in a warning's location, a derived variable's random id, how the kernel chunks printed text, plotly templates, typed-array encoding, NaN vs null; figure numbers are compared to a relative 1e-9
  • A refresh keeps the saved bytes of every cell whose visible content is unchanged, so float noise does not churn the diff

Key Design Decisions

• Compare what a reader sees, not bytes: CI resolves plotly 7.x while local venvs may have 6.x; figures are compared on decoded data and labels

• Refresh the whole set from one command; CONTRIBUTING says when to run it and to reread the prose around changed outputs

• Prose: only text a refreshed output contradicted was changed

  • signal-detection: Rules 4 and 5 read "Suggests a sustained shift" and "Suggests drift"
  • sds-validation: the Slide 19 and 32 reference values now quote the VAS rerun of 13 October 2026 used by the gate (PP 1.58, PPL 1.84 on Slide 32); the current process mean is explained as the mean of the cell means (237.83) beside the plain average (237.81)
  • stratified-analysis: Lane D's one signal is a chance six-point trend in a lane with no special cause

• Interaction chart order (found by this check on its first CI run): create_time_interaction_chart sorted by time with an unstable sort, so the order of factor combinations within a time depended on the numpy version (Linux CI 2.4.6/2.5.4 vs macOS 2.4.0). It now uses kind='stable', keeping the groupby's factor order; test_lines_are_in_factor_order pins it

Risks & Mitigations

• P0 risks: a false drift report in CI from an environment difference
• Mitigations: the check passed three runs in a row locally; normalisation covers the differences found (stream chunking, NaN, random ids, temp paths); if CI finds another, normalise that field rather than fall back to text only

Tests

Added / Updated

• tests/test_refresh_notebooks.py (10 tests): docs notebooks found; warning paths, random ids, stream chunking, plotly template, bdata-vs-list and last-digit float noise ignored; a changed number in text, a changed figure value, a changed figure label and a reordered legend all count as drift
• test_lines_are_in_factor_order (interaction chart legend order)

Regression Focus (must fail if broken)

• a stale saved output passes CI (negative check: one saved digit changed in a scratch copy fails with the cell named and the difference shown)
• noise fails CI (three consecutive clean runs)

Manual Verification

• python scripts/refresh_notebooks.py --check: all 14 current
• pytest tests/: 2707 passed, 7 skipped
• python validation/e2e_bishop_report.py: 1547 passed, 0 failed
• ruff check . clean; mypy shows the same 15 advisory errors
• mystmd@1.10.1 build --html succeeds; only the three existing changelog citation warnings

Notes

• Docs: CONTRIBUTING.md "Docs notebooks"; CHANGELOG [Unreleased] Fixed and CI / Infrastructure
• sds-validation diff is large (about 100k lines) because two R2/X charts now plot 1599 points where 1565 were saved, which shifts whole arrays; the encoding is unchanged and the file grew by 12 KB
• Follow-ups (not in this PR): none