You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Test markers name the source an expected value comes from, but only reviewers checked them, so a typo, a stale section, an unpinned link into another repository, or a forgotten scaffold marker could reach master unnoticed.
The preflight job now runs ci/check-oracle-markers.ps1 over the tracked Rust, C#, and PowerShell files, where a Pester test is an It block and its marker a # oracle: comment. It enforces the marker grammar of test-oracle-reviewer: known kinds, property laws by name, no scaffold, and regressions as an issue or a tracked fuzz crash file. It resolves each contract, intent and docs/ spec section by anchor, heading text or heading number in a tracked file, and rejects an anchor id repeated within such a file. A citation into another repository must pin a commit of at least 12 hex digits; the file is fetched at that commit through the GitHub API and resolved the same way. When the token cannot read that repository, the reference gets only a syntax check, while a bad token or a rate limit fails the job rather than skip every pin.
On a pull request it also requires a marker on each new test, a test being new when its function name, or its It name in Pester, is declared more often than at the base, so moving a test or editing its attributes does not count.
A weekly workflow runs the script with -DriftReport, which compares each pinned section with its repository's default branch and lists the pins to bump and the tests to re-check, without gating anything.
ci/check-oracle-markers.intent.md records these rules beside the script, so a later change cannot quietly undo one, and the script's Pester tests, run in their own job, cite its sections in their markers.
Since every citation names its file with the anchor, the file is the anchor's namespace: intent-files now names an anchor after its section alone and asks only that it be unique within its own document, and intent-review and contract-review report an id repeated within a file.
test-oracle-reviewer now states the marker as a line comment of the test's language, since a single // form would lead a Pester test to a comment PowerShell never reads.
Test markers name the source an expected value comes from, but only
reviewers checked them, so a typo, a stale section, an unpinned link
into another repository, or a forgotten scaffold marker could reach
master unnoticed.
The preflight job now runs ci/check-oracle-markers.ps1 over the
tracked Rust and C# files. It enforces the marker grammar of
test-oracle-reviewer: known kinds, property laws by name, no scaffold,
and regressions as an issue or a tracked fuzz crash file. It resolves
each contract, intent and docs/ spec section by anchor, heading text
or heading number in a tracked file. A citation into another
repository must pin a commit of at least 12 hex digits; the file is
fetched at that commit through the GitHub API and resolved the same
way. When the token cannot read that repository, the reference gets
only a syntax check, while a bad token or a rate limit fails the job
rather than skip every pin.
On a pull request it also requires a marker on each new test, a test
being new when its function name is declared more often than at the
base, so moving a test or editing its attributes does not count.
A weekly workflow runs the script with -DriftReport, which compares
each pinned section with its repository's default branch and lists
the pins to bump and the tests to re-check, without gating anything.
Pester tests for the script run in their own job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test-oracle-reviewer now says what CI enforces beyond the grammar: a
cited file must be committed, a fuzz crash file force-added past
fuzz/.gitignore, and a pin the token cannot read gets only a syntax
check. It also says when a test counts as new and that a weekly
report lists pinned sections that changed upstream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
$GitHubApi default block (ci/check-oracle-markers.ps1, the param block): the Pester tests replace it with a fake API, so nothing checks that it sends the token as a bearer header and returns an error status, such as 401 or 404, instead of throwing. After changing it, run a pinned marker against a public repository with a valid token, an invalid one, and none. That was done by hand for this change: a valid token and no token both resolve the section, and an invalid token stops the check with returned 401: Bad credentials.
Remote file and outline caches now compare paths ordinally, so a
citation with the wrong case no longer reuses another's result. A code
fence closes only on its own delimiter repeated at least as often, so
headings inside nested examples stay code. An anchor names a section
only when a heading follows it, so a stale anchor no longer resolves.
New-test detection now reads whole attribute blocks, including ones
spanning lines, and bounds each test at the end of the item it
decorates, so a multiline case no longer splits a test and a marked
helper after an unmarked test no longer counts as its marker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every citation names its file with the anchor, as in
`intent <path> §<anchor>`, so an id repeated in another file never makes
one ambiguous; only a repeat within the same file does. intent-files
now asks anchor ids to be unique within their own document only, and
intent-review and contract-review report an id repeated within a file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skill teaches the marker grammar; how CI enforces it belongs to the
check. The one fact a contributor needs at the moment it bites, that a
fuzz crash file must be force-added past fuzz/.gitignore, now sits in
the check's error message instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An anchor id is unique within its own file and every citation names
that file, so the file is already the namespace and a scope prefix
only repeats it, as in `CONTRACT.agent-identity.md §agent-identity-channel`.
intent-files now defines the anchor as a short kebab-case name of the
section, and the intent-drift-audit example follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check's rules were decided in review and lived only in its code and
commit messages, so a later change could quietly undo one, such as
skipping a pin on any API error or making the drift report a gate.
The intent file records them beside the script, each section anchored
so the script's own tests can cite the rule they check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PowerShell tests can copy their expected values from the code under
test as easily as Rust and C# tests, so the check covers them from the
start. A PowerShell marker is a `# oracle:` comment, and a Pester test
is an `It` block, found with the PowerShell parser because the
hashtables of `-ForEach` cases defeat brace counting. New Pester tests
are detected by `It` name, as Rust and C# tests are by function name.
The check also fails on an anchor id repeated within a file a marker
may cite, since a citation of it could resolve to either section.
The check's own tests now carry markers citing the intent file, or
CommonMark where Markdown decides the expected value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read at fd4a745c on cbenoit-oracle-marker-check (PR #2141).
Drafted on delegation ("bootstrap based on all I said so far") and approved as a whole ("I also reviewed the intent file, it's good, I approve it"); rows 32 to 35 were added after the approval and adopted separately ("The four additions are fine").
Provenance
#
Commitment as written
Human's words (if tidied)
Level
1
The check enforces the markers test-oracle-reviewer teaches; the rules and the skill are one rule, and a difference is a defect.
—
3
2
Whether the cited source decides the value is not checked; that stays with review.
—
3
3
The check covers tests in Rust, C#, and PowerShell (Pester).
"I would suggest we support Pester from day one."
2
4
Marker is a line comment of the language: // in Rust and C#, # in PowerShell.
—
3
5
A marker sits on a test, one case, or one assertion.
"on a test, a case row, or a single assertion"
2
6
A Pester test is an It block; a case is one -ForEach or -TestCases entry.
—
3
7
Only tracked files are read, because ignored or uncommitted files exist locally but not in CI.
"apply the review fixes (tracked files only, …)"
3
8
The kinds are exactly the eight.
"Kinds: spec … mutant, scaffold (weak)"
2
9
A spec names a document; a spec under docs/ takes its path instead.
"spec <name> §<section>: a named spec (RFC 9421, X.690…). A spec under docs/ takes its path instead."
2
10
A spec name that looks like a URL, an @ ref, or a file name outside docs/ is rejected.
—
3
11
A regression names a Jira key, #123, owner/repo#123, or a committed crash file under fuzz/.
SHA of at least 12 hex digits; branches and tags rejected, because they move.
"Reject branches and tags, because they move."
1
21
Unpinned cross-repository references fail.
"Unpinned cross-repository forms fail."
1
22
Fetched at that commit, resolved by the same rules; deterministic.
"A commit never changes, so this is deterministic."
2
23
Cached by <owner>/<repo>@<sha>:<path>.
"Cache by owner/repo@sha:path."
1
24
Without the privilege to read the repository, only a syntax check.
"If it's not possible to fetch the file at that commit because the privilege is missing, I'm good with simply skipping the check, at which point it's just a syntax check"
2
25
A bad token or a rate limit fails rather than skip.
—
3
26
On a pull request, every new test carries a marker.
"new tests carry a marker"
2
27
An existing unmarked test blocks nothing; moving or editing a test does not make it new.
"apply the review fixes (… new tests by function name …)"
3
28
Drift report: scheduled, not a pull-request gate; compares the pinned section with the default branch.
"Drift report (scheduled, not a pull-request gate)"
2
29
Unchanged → proposed mechanical pin bump; changed, moved, or removed → listed for a person to re-check.
"Unchanged → propose a mechanical pin bump. Changed, moved or removed → list the test for a person to re-check against the new text."
2
30
One table per repository: test, marker, status.
"Output one table per repository: test, marker, status."
1
31
A PowerShell script, not an xtask, because it runs with no build step ahead of every other CI job.
chose "Keep PowerShell" over "Move to xtask"; the reason is agent wording
3
| 32 | contract takes a path to a CONTRACT*.md file, intent to an INTENT*.md or *.intent*.md file. | "The four additions are fine" | 3 |
| 33 | spec, contract, and intent cite a section as <document> §<section>. | "The four additions are fine" | 3 |
| 34 | Every kind but scaffold takes a reference. | "The four additions are fine" | 3 |
| 35 | The repeated-anchor rule covers files a marker may cite, under docs/ or a contract or intent file. | "The four additions are fine" | 3 |
Left out on purpose
Exact drift statuses beyond the three (current, skipped, unresolved) and what distinguishes moved from changed (same body, new heading): implementation of row 29.
Running in preflight, the token secret and its fallback, the weekly schedule, -Base HEAD^1: CI wiring, not a property of the check.
New-test detection by function or It name count: the mechanism behind row 27.
The repository-wide anchor-uniqueness rule: dropped from intent-files in the same pull request, since ids need only be unique within their own document; anchors are now named after their section alone.
Markdown rules the check follows, such as fenced code and line endings: CommonMark decides them, and the tests cite it directly.
Anything naming a particular private repository: kept out at your request.
Placement
One file, ci/check-oracle-markers.intent.md, beside the script it governs; every section carries an anchor so the script's tests can cite it.
Pester evaluates `-ForEach` data during discovery, before `BeforeAll`
runs, so fixture rows that spelled the marker through a variable set
there carried none. They now write it as "`# oracle:", which builds the
marker without the tests file itself holding one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A marker is a line comment of the test's language, so one sentence
covers Rust, C#, and PowerShell alike, and naming a single comment
syntax would lead a Pester test to a `//` that PowerShell never reads as
a comment. The exact forms stay in the oracle marker check's intent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test's end is now found on code with comments and string literals
blanked across lines, and a semicolon ends an item only outside
parentheses and brackets, so a brace in a comment or raw string, or a
fixed-size array parameter, no longer moves the end of a test.
A name's new tests in a file are counted as the lesser of how much more
often the file and the tree declare it than at the base, so an existing
unmarked test stays exempt when a namesake joins its file; when markers
fall short, the unmarked tests the diff adds are reported first.
Pester names are counted regardless of the case of `It`, which
PowerShell ignores, and a bare Markdown file name is rejected as a spec
name, since it is a path that would escape resolution.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extract markers from comments instead of raw source lines
ci/check-oracle-markers.ps1:49
[P2] Extract markers from actual comments, not raw source lines. A Rust multiline raw string or PowerShell here-string containing a standalone // oracle: property round-trip or # oracle: property round-trip makes an otherwise unmarked new test pass. Literal fixture text containing a scaffold marker instead fails CI. C# multiline strings have the same problem. Both Test-Markers and $Marked inspect the original text; literal blanking only protects test-boundary detection. Share language-aware comment extraction between validation and marker ownership, preserve the supported Rust doctest form, and add fixtures for marker-looking string data.
Enforce anchor uniqueness across all fetched files
ci/check-oracle-markers.ps1:281
[P2] Apply document-wide anchor uniqueness to fetched files too. This condition rejects duplicate IDs only when the citation names that ID. In the pinned repeated-anchor fixture, changing §x to the heading §A makes the same document pass. That conflicts with ci/check-oracle-markers.intent.md §paths and the local Test-Anchors check. Reject any nonempty duplicate list, report the offending ID, and add a pinned-file case cited by heading or number.
Make marker casing consistent with validation scans
ci/check-oracle-markers.ps1:574
[P2] Use the same marker casing rules as the validation scan. PowerShell's -match is case-insensitive, but Test-Markers uses case-sensitive git grep. A new test containing only // ORACLE: scaffold therefore counts as marked here while its marker is never validated, so the test passes CI. Use -cmatch to agree with the lowercase validation scan, and add a fixture showing that an uppercase label does not satisfy the marker requirement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test markers name the source an expected value comes from, but only reviewers checked them, so a typo, a stale section, an unpinned link into another repository, or a forgotten scaffold marker could reach master unnoticed.
The preflight job now runs
ci/check-oracle-markers.ps1over the tracked Rust, C#, and PowerShell files, where a Pester test is anItblock and its marker a# oracle:comment. It enforces the marker grammar oftest-oracle-reviewer: known kinds, property laws by name, noscaffold, and regressions as an issue or a tracked fuzz crash file. It resolves each contract, intent anddocs/spec section by anchor, heading text or heading number in a tracked file, and rejects an anchor id repeated within such a file. A citation into another repository must pin a commit of at least 12 hex digits; the file is fetched at that commit through the GitHub API and resolved the same way. When the token cannot read that repository, the reference gets only a syntax check, while a bad token or a rate limit fails the job rather than skip every pin.On a pull request it also requires a marker on each new test, a test being new when its function name, or its
Itname in Pester, is declared more often than at the base, so moving a test or editing its attributes does not count.A weekly workflow runs the script with
-DriftReport, which compares each pinned section with its repository's default branch and lists the pins to bump and the tests to re-check, without gating anything.ci/check-oracle-markers.intent.mdrecords these rules beside the script, so a later change cannot quietly undo one, and the script's Pester tests, run in their own job, cite its sections in their markers.Since every citation names its file with the anchor, the file is the anchor's namespace:
intent-filesnow names an anchor after its section alone and asks only that it be unique within its own document, andintent-reviewandcontract-reviewreport an id repeated within a file.test-oracle-reviewernow states the marker as a line comment of the test's language, since a single//form would lead a Pester test to a comment PowerShell never reads.🤖 Generated with Claude Code