Repository navigation
Make signal rules 2, 5 and 6 match their definitions; correct the WECO rates - #168
Merged
Merged
Conversation
…O rates ## Summary - **What:** bugfix in three signal rules, and corrected chance rates and wording on the WECO rules page. - **Why:** an external review found the page's Rule 7 rate wrong (0.003% where it is 0.33%). Checking every rule found six of eight rates wrong, and three rules in the code that did not do what the docs, the rule descriptions, `RULE_MIN_OBSERVATIONS`, Nelson and Wheeler say: Rule 5 fired only on 7 points (6 rises), Rule 6 only on 15 alternating points, and Rule 2 ignored points beyond 3σ. The existing tests used 16-point sequences, so the boundaries were never pinned. - **Scope:** `signals/detectors.py` (three rules), the Rule 2 description, boundary tests, `validation/chance_signals.py` (two more columns), weco-rules.md, the signal-detection notebook's prose, key-concepts.md, CHANGELOG. No new parameter. ## Contract / Invariants (must remain true) - [x] **Default behavior unchanged:** chart, capability and loss numbers unchanged; Bishop gate 1547/0/4. Signal output on X charts changes by design (below) - [x] **Residuals unaffected:** no residual code touched - [x] **Row/index alignment preserved:** violation flags still mark the last point of each window - [x] **Lane boundaries preserved:** no plotting change - [x] **Output schema compatible:** `SignalResult` unchanged; the Rule 2 description text adds "or beyond" ## Behavior Changes (explicit) - **New API:** none - **New semantics:** - Rule 5 fires on 6 consecutive rising (or falling) points; it needed 7 - Rule 6 fires on 14 consecutive alternating points; it needed 15 - Rule 2 counts any point more than 2σ out on the same side, including beyond 3σ - mR, Xbar and S use Rule 1 by default, so they are unaffected unless more rules are requested ## Key Design Decisions - **Fix the code, not the docs:** every written definition (docs, `RULE_DESCRIPTIONS`, `RULE_MIN_OBSERVATIONS` = 6 and 14, Nelson, Wheeler, Minitab test 5) agreed; the code was the outlier - **Exact rates:** per-window probabilities for independent normal points, computed exactly (Euler zigzag numbers for Rule 6); per-chart rates measured by `chance_signals.py`, now with X under Rule 1 and Rules 1-4 so the rule-set table has a source - **Wording:** a signal is a reason to look for a cause; "almost certainly", "process has shifted" and "is drifting" become "suggests" ## Risks & Mitigations - **P0 risks:** users comparing signal counts across versions see more X-chart signals - **Mitigations:** CHANGELOG states the change and the size (70% to 79% of 100-point pure-noise X charts with a signal); `rules=` still selects any subset ## Tests ### Added / Updated - [x] `TestRule5Trend`: 6 rising points fire (once, at the 6th), 5 do not, 6 falling fire, a tie breaks the run - [x] `TestRule6Oscillation`: 14 alternating points fire (once), 13 do not - [x] `TestRule2ZoneA`: points at 2.5σ and 3.5σ fire, both sides, and with per-row limits ### Regression Focus (must fail if broken) - [x] off-by-one in trend or oscillation length - [x] Rule 2 dropping points beyond the limit ## Manual Verification - [x] `pytest tests/`: 2696 passed, 7 skipped - [x] `python validation/e2e_bishop_report.py`: 1547 passed, 0 failed; `validation.md` unchanged - [x] `python validation/chance_signals.py` rerun; weco-rules.md and the notebook table use its output - [x] `pytest --nbmake docs/tutorials/signal-detection.ipynb`; `ruff check .` clean; mypy shows the same 15 advisory errors - [x] Per-window rates: 0.27%, 0.31%, 0.55%, 0.78%, 0.28%, 0.46%, 0.33%, 0.010% ## Notes - **Docs:** weco-rules.md (rates, wording, measured rule-set table, an explicit target for the per-chart section), signal-detection notebook prose (cells 7, 22, 28), key-concepts.md Rule 2; CHANGELOG [Unreleased] Fixed - **Follow-ups (not in this PR):** the notebooks' saved outputs, including signal-detection's, are refreshed in the notebook-refresh PR that follows this one
14 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RULE_MIN_OBSERVATIONS, Nelson and Wheeler say: Rule 5 fired only on 7 points (6 rises), Rule 6 only on 15 alternating points, and Rule 2 ignored points beyond 3σ. The existing tests used 16-point sequences, so the boundaries were never pinned.signals/detectors.py(three rules), the Rule 2 description, boundary tests,validation/chance_signals.py(two more columns), weco-rules.md, the signal-detection notebook's prose, key-concepts.md, CHANGELOG. No new parameter.Contract / Invariants (must remain true)
SignalResultunchanged; the Rule 2 description text adds "or beyond"Behavior Changes (explicit)
Key Design Decisions
RULE_DESCRIPTIONS,RULE_MIN_OBSERVATIONS= 6 and 14, Nelson, Wheeler, Minitab test 5) agreed; the code was the outlierchance_signals.py, now with X under Rule 1 and Rules 1-4 so the rule-set table has a sourceRisks & Mitigations
rules=still selects any subsetTests
Added / Updated
TestRule5Trend: 6 rising points fire (once, at the 6th), 5 do not, 6 falling fire, a tie breaks the runTestRule6Oscillation: 14 alternating points fire (once), 13 do notTestRule2ZoneA: points at 2.5σ and 3.5σ fire, both sides, and with per-row limitsRegression Focus (must fail if broken)
Manual Verification
pytest tests/: 2696 passed, 7 skippedpython validation/e2e_bishop_report.py: 1547 passed, 0 failed;validation.mdunchangedpython validation/chance_signals.pyrerun; weco-rules.md and the notebook table use its outputpytest --nbmake docs/tutorials/signal-detection.ipynb;ruff check .clean; mypy shows the same 15 advisory errorsNotes