Skip to content

Redesign the PR summary comment for fast human judgment - #12

Merged
irvingpop merged 2 commits into
mainfrom
irvingpop/comment-redesign
Oct 28, 2026
Merged

irvingpop merged 2 commits into
mainfrom
irvingpop/comment-redesign

Conversation

@irvingpop

Copy link
Copy Markdown
Member

What changes

  • Status heading with a status light.
  • One alert callout that holds the next step.
  • Red CAUTION callout for every fail.
  • Findings table, worst first. Duplicates on the same file, side, line and effect merge into one row.
  • One-row checks strip.
  • Collapsed Waiver and full record sections.
  • Spec updated with decision P3-5.

Why

The old comment was hard to judge quickly. The owner chose this design from two candidates.

Safety

Markers, override line and size limits are unchanged. Model text is escaped, so it cannot forge headings, callouts or the result line (probed in review).

Tests

  • cd review && uv run pytest tests/unit -q -> 2631 passed, 2 deselected
  • cd review && uv run pytest tests/unit -m release -q -> 2 passed, 2631 deselected

Rendered examples

Example: PR 39 comment

🔴 FAIL: OC Claude review

[!CAUTION]
One high-severity malicious finding blocks this PR. A maintainer can waive it.
Next step for the author: fix and push, or ask a maintainer to post the waiver line (under "Waiver" below).

Severity Effect Where Finding Found by
🔴 High Blocks, waivable src/app.py:10 Job impersonates the review gate check name with a fake pass. See the inline comment. Review (high)
🟠 Medium Does not block src/app.py:10 Workflow spoofs the required review check name. See the inline comment. Security (medium), Supply chain (medium)
File checks Review Security Supply chain Dependency review Coverage
✅ ok, 2 signals ✅ ran, 1 finding ✅ ran, 1 finding ✅ ran, 1 finding ✅ ok ✅ complete

Run 1000, attempt 1 for head aaaaaaa into main. The required workflow in the merge box is the result that counts. This bot never approves a PR.

🔑 Waiver: how a maintainer clears this

This PR fails for waivable reasons only. A person with write or admin access, who is not the PR author, posts this exact line as a new comment and replaces <reason> with the reason. Then they re-run the newest run of the required workflow through the GitHub REST API (POST .../actions/runs/{run_id}/rerun). Do not edit the comment afterwards. Do not close and reopen the PR to apply the waiver.

/oc-review override aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa cccccccccccccccccccccccccccccccccccccccc <reason>

A waiver covers the code at this exact head and merge base. Any new push needs a new waiver.

📋 Full review record: reasons, checks, signals, finding details, model summaries, token usage

Reviewed snapshot

Head SHA Merge base SHA Base branch Run
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa cccccccccccccccccccccccccccccccccccccccc main 1000, attempt 1

Reasons the gate failed

Reason Waivable? Detail
Blocking finding ✅ yes review: malicious high at src/app.py:10

Checks (code, no model)

Check Result Detail
Hard blocks ✅ none
Coverage ✅ complete files in comparison 1, files collected 1, trees complete
Dependency changes ✅ none
Dependency review job ✅ success

Signals (context only, never block)

  • ⚠️ Sensitive path changed: workflow file, in .github/workflows/x.yml
  • ℹ️ First-time contributor: author association NONE

Dependency changes

None.

Finding details

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Blocking is decided by code from category and severity, never by the model. A finding in a blocking category blocks only at high or critical severity.

Each row above merges findings of several passes at the same file, line and gate effect.

Pass Severity Category Confidence Citation Posted Model title
Review high malicious high ✅ checked inline Job impersonates the review gate check name with a fake pass.
Security medium security high ✅ checked inline Workflow spoofs the required review check name.
Supply chain medium supply chain high ✅ checked inline Workflow job spoofs the review gate check name.

Model summaries

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Pass Status Model Summary
Review ✅ ok claude-opus-5-5 Looked at the change.
Security ✅ ok claude-opus-5-5 Looked at the change.
Supply chain ✅ ok claude-opus-5-5 Looked at the change.

Token usage

Pass Input Output Cache write Cache read Attempts
Review 12000 900 11000 0 1
Security 12000 900 11000 0 1
Supply chain 12000 900 11000 0 1
Total 36000 2700 33000 0 3

Known Minor, not fixed

Hard block data shows in three tables. A PR with many hard blocks reaches the overflow notice sooner (fails closed).

🤖 Generated with Claude Code

@irvingpop
irvingpop requested a review from a team as a code owner October 28, 2026 03:58
@github-actions

Copy link
Copy Markdown

OC Claude review: fail

Result for run 40629688283, attempt 1.
Computed for head SHA 897700826789bbf8f9ee01b3467e244a4f30c802, merge base 2c63f49b461feac74c2c70c4f86ca90040df6a0a, base ref main.
The result that counts is the required workflow in the merge box.

Reasons

  • job-result (not waivable): dependency-review: failure
  • dependency-review (not waivable): job result failure
  • dependency-review (not waivable): step outcome failure
  • pass-records (not waivable): pass review failed: authentication failed
  • pass-records (not waivable): pass security failed: authentication failed
  • pass-records (not waivable): pass supply-chain failed: authentication failed
  • prefilter-hard-block (not waivable): bidi-control in review/tests/unit/test_publish.py

Prefilter (code, no LLM)

Hard blocks: 1.

  • bidi-control: bidi control characters U+202E, in review/tests/unit/test_publish.py, lines 884 885 903 909
    Signals: 2.
  • long-line: lines longer than 500 characters, in docs/claude-review-design.md, lines 540 574 588 589 590 593 594 982
  • ai-directed-text: pattern ai-reviewer, in review/src/oc_review/publish.py, lines 982
    Coverage: complete (files in comparison 5, files collected 5, trees complete).

Dependency delta

No dependency changes.

Findings per pass

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

review: failed, authentication failed

No verified findings.

security: failed, authentication failed

No verified findings.

supply-chain: failed, authentication failed

No verified findings.

Token usage

  • review: no usage recorded
  • security: no usage recorded
  • supply-chain: no usage recorded
  • total: no usage recorded

Workflow run. The result that counts is the required workflow in the merge box.

@irvingpop
irvingpop merged commit a410ddd into main Oct 28, 2026
3 of 5 checks passed
@irvingpop
irvingpop deleted the irvingpop/comment-redesign branch October 29, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant