Skip to content

Make signal rules 2, 5 and 6 match their definitions; correct the WECO rates - #168

Merged
cnicholas merged 2 commits into
mainfrom
fix/signal-rule-lengths
Nov 3, 2026
Merged

cnicholas merged 2 commits into
mainfrom
fix/signal-rule-lengths

Conversation

@cnicholas

Copy link
Copy Markdown
Owner

Summary

  • What: bugfix in three signal rules, and corrected chance rates and wording on the WECO rules page.
  • Why: an external review found the page's Rule 7 rate wrong (0.003% where it is 0.33%). Checking every rule found six of eight rates wrong, and three rules in the code that did not do what the docs, the rule descriptions, RULE_MIN_OBSERVATIONS, Nelson and Wheeler say: Rule 5 fired only on 7 points (6 rises), Rule 6 only on 15 alternating points, and Rule 2 ignored points beyond 3σ. The existing tests used 16-point sequences, so the boundaries were never pinned.
  • Scope: signals/detectors.py (three rules), the Rule 2 description, boundary tests, validation/chance_signals.py (two more columns), weco-rules.md, the signal-detection notebook's prose, key-concepts.md, CHANGELOG. No new parameter.

Contract / Invariants (must remain true)

  • Default behavior unchanged: chart, capability and loss numbers unchanged; Bishop gate 1547/0/4. Signal output on X charts changes by design (below)
  • Residuals unaffected: no residual code touched
  • Row/index alignment preserved: violation flags still mark the last point of each window
  • Lane boundaries preserved: no plotting change
  • Output schema compatible: SignalResult unchanged; the Rule 2 description text adds "or beyond"

Behavior Changes (explicit)

  • New API: none
  • New semantics:
    • Rule 5 fires on 6 consecutive rising (or falling) points; it needed 7
    • Rule 6 fires on 14 consecutive alternating points; it needed 15
    • Rule 2 counts any point more than 2σ out on the same side, including beyond 3σ
    • mR, Xbar and S use Rule 1 by default, so they are unaffected unless more rules are requested

Key Design Decisions

  • Fix the code, not the docs: every written definition (docs, RULE_DESCRIPTIONS, RULE_MIN_OBSERVATIONS = 6 and 14, Nelson, Wheeler, Minitab test 5) agreed; the code was the outlier
  • Exact rates: per-window probabilities for independent normal points, computed exactly (Euler zigzag numbers for Rule 6); per-chart rates measured by chance_signals.py, now with X under Rule 1 and Rules 1-4 so the rule-set table has a source
  • Wording: a signal is a reason to look for a cause; "almost certainly", "process has shifted" and "is drifting" become "suggests"

Risks & Mitigations

  • P0 risks: users comparing signal counts across versions see more X-chart signals
  • Mitigations: CHANGELOG states the change and the size (70% to 79% of 100-point pure-noise X charts with a signal); rules= still selects any subset

Tests

Added / Updated

  • TestRule5Trend: 6 rising points fire (once, at the 6th), 5 do not, 6 falling fire, a tie breaks the run
  • TestRule6Oscillation: 14 alternating points fire (once), 13 do not
  • TestRule2ZoneA: points at 2.5σ and 3.5σ fire, both sides, and with per-row limits

Regression Focus (must fail if broken)

  • off-by-one in trend or oscillation length
  • Rule 2 dropping points beyond the limit

Manual Verification

  • pytest tests/: 2696 passed, 7 skipped
  • python validation/e2e_bishop_report.py: 1547 passed, 0 failed; validation.md unchanged
  • python validation/chance_signals.py rerun; weco-rules.md and the notebook table use its output
  • pytest --nbmake docs/tutorials/signal-detection.ipynb; ruff check . clean; mypy shows the same 15 advisory errors
  • Per-window rates: 0.27%, 0.31%, 0.55%, 0.78%, 0.28%, 0.46%, 0.33%, 0.010%

Notes

  • Docs: weco-rules.md (rates, wording, measured rule-set table, an explicit target for the per-chart section), signal-detection notebook prose (cells 7, 22, 28), key-concepts.md Rule 2; CHANGELOG [Unreleased] Fixed
  • Follow-ups (not in this PR): the notebooks' saved outputs, including signal-detection's, are refreshed in the notebook-refresh PR that follows this one

…O rates

## Summary
- **What:** bugfix in three signal rules, and corrected chance rates and wording on the WECO rules page.
- **Why:** an external review found the page's Rule 7 rate wrong (0.003% where it is 0.33%). Checking every rule found six of eight rates wrong, and three rules in the code that did not do what the docs, the rule descriptions, `RULE_MIN_OBSERVATIONS`, Nelson and Wheeler say: Rule 5 fired only on 7 points (6 rises), Rule 6 only on 15 alternating points, and Rule 2 ignored points beyond 3σ. The existing tests used 16-point sequences, so the boundaries were never pinned.
- **Scope:** `signals/detectors.py` (three rules), the Rule 2 description, boundary tests, `validation/chance_signals.py` (two more columns), weco-rules.md, the signal-detection notebook's prose, key-concepts.md, CHANGELOG. No new parameter.

## Contract / Invariants (must remain true)
- [x] **Default behavior unchanged:** chart, capability and loss numbers unchanged; Bishop gate 1547/0/4. Signal output on X charts changes by design (below)
- [x] **Residuals unaffected:** no residual code touched
- [x] **Row/index alignment preserved:** violation flags still mark the last point of each window
- [x] **Lane boundaries preserved:** no plotting change
- [x] **Output schema compatible:** `SignalResult` unchanged; the Rule 2 description text adds "or beyond"

## Behavior Changes (explicit)
- **New API:** none
- **New semantics:**
  - Rule 5 fires on 6 consecutive rising (or falling) points; it needed 7
  - Rule 6 fires on 14 consecutive alternating points; it needed 15
  - Rule 2 counts any point more than 2σ out on the same side, including beyond 3σ
  - mR, Xbar and S use Rule 1 by default, so they are unaffected unless more rules are requested

## Key Design Decisions
- **Fix the code, not the docs:** every written definition (docs, `RULE_DESCRIPTIONS`, `RULE_MIN_OBSERVATIONS` = 6 and 14, Nelson, Wheeler, Minitab test 5) agreed; the code was the outlier
- **Exact rates:** per-window probabilities for independent normal points, computed exactly (Euler zigzag numbers for Rule 6); per-chart rates measured by `chance_signals.py`, now with X under Rule 1 and Rules 1-4 so the rule-set table has a source
- **Wording:** a signal is a reason to look for a cause; "almost certainly", "process has shifted" and "is drifting" become "suggests"

## Risks & Mitigations
- **P0 risks:** users comparing signal counts across versions see more X-chart signals
- **Mitigations:** CHANGELOG states the change and the size (70% to 79% of 100-point pure-noise X charts with a signal); `rules=` still selects any subset

## Tests
### Added / Updated
- [x] `TestRule5Trend`: 6 rising points fire (once, at the 6th), 5 do not, 6 falling fire, a tie breaks the run
- [x] `TestRule6Oscillation`: 14 alternating points fire (once), 13 do not
- [x] `TestRule2ZoneA`: points at 2.5σ and 3.5σ fire, both sides, and with per-row limits

### Regression Focus (must fail if broken)
- [x] off-by-one in trend or oscillation length
- [x] Rule 2 dropping points beyond the limit

## Manual Verification
- [x] `pytest tests/`: 2696 passed, 7 skipped
- [x] `python validation/e2e_bishop_report.py`: 1547 passed, 0 failed; `validation.md` unchanged
- [x] `python validation/chance_signals.py` rerun; weco-rules.md and the notebook table use its output
- [x] `pytest --nbmake docs/tutorials/signal-detection.ipynb`; `ruff check .` clean; mypy shows the same 15 advisory errors
- [x] Per-window rates: 0.27%, 0.31%, 0.55%, 0.78%, 0.28%, 0.46%, 0.33%, 0.010%

## Notes
- **Docs:** weco-rules.md (rates, wording, measured rule-set table, an explicit target for the per-chart section), signal-detection notebook prose (cells 7, 22, 28), key-concepts.md Rule 2; CHANGELOG [Unreleased] Fixed
- **Follow-ups (not in this PR):** the notebooks' saved outputs, including signal-detection's, are refreshed in the notebook-refresh PR that follows this one
@cnicholas
cnicholas merged commit 85bb0db into main Nov 3, 2026
19 checks passed
@cnicholas
cnicholas deleted the fix/signal-rule-lengths branch November 3, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant