Skip to content

docs(evidence): N1 competitor gate — GO-CANDIDATE-N1 (3/4 missed by CA2000 AllPaths) #417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PhysShell merged 2 commits into from
Oct 20, 2026
Merged

docs(evidence): N1 competitor gate — GO-CANDIDATE-N1 (3/4 missed by CA2000 AllPaths) #417

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d06caf7
docs(evidence): N1 competitor gate pre-registration and reproductions
PhysShell Oct 20, 2026
b5d6f18
docs(evidence): N1 competitor gate — GO-CANDIDATE-N1 (3/4 missed by C…
PhysShell Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions docs/evidence/n1-competitor-gate-2026-10/00-preregistration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# N1 competitor gate — pre-registration

Written 2026-10-20, 21:11 +05. The four fix diffs have been read in order to build the reproductions. No analyzer
has been run on any reproduction.

Authority: owner decision `GO-N1-COMPETITOR-GATE`, given after `KILL-N3-PILOT` was accepted. This gate is not a GO
on an N1 PoC.

- **Budget.** 0.25 agent-day.
- **Rules of the work.** No third-party project is built and no package is downloaded. The analyzers come from the
.NET SDK and from the local NuGet cache, the same versions as direction-research Phase 5.

## Question

N1: a disposable is acquired, a step that can throw runs before ownership is handed over (return, field or
collection store, wrapping), and no dispose runs on the throw path.

Do existing analyzers miss the four confirmed N1 defects of Phase 0 once they are configured to look? The question
matters most for CA2000 with exception paths on (`dispose_analysis_kind = AllPaths`), because that is the
documented setting for exactly this case.

## Why the earlier "not detected" does not answer it

Phase 5 ran the analyzers over loose source trees. Those runs had about 35,000 compile errors (whizbang: 34,859 on
base), so the disposable types were mostly unresolved, and CA2000 ran with its default kind. Those misses therefore
say little about what CA2000 can do.

## Pairs (fixed now)

| Id | Fix | Buggy method | Leaked resource and acquisition | Throwing step before hand-over | Hand-over on success |
|---|---|---|---|---|---|
| W | whizbang-lib/whizbang#1204 | `AzureServiceBusConnectionRetry.CreateClientWithRetryAsync` | `new ServiceBusClient(cs)` (`IAsyncDisposable`), inside a retry loop's `try` | `await VerifyNamespaceReachableAsync(…)` | `return client;` |
| R | MarimerLLC/rockbot#638 (record 638-2) | `McpBridgeService.ConnectServerAsync` | `await McpClient.CreateAsync(transport, …)`, an async factory (`IAsyncDisposable`) | `await newClient.ListToolsAsync(…)`, prompt listing | `_clients[name] = newClient;` (dictionary field) |
| X | NexusFFXIV/NexusKit#34 (record 34-2) | `PluginHostBuilder.BuildAsync` | `services.BuildServiceProvider(validateScopes: false)`, a factory (`ServiceProvider`, `IDisposable` + `IAsyncDisposable`) | `await DbInitializer.InitializeAsync(…)`, service resolution | `return new PluginHost(provider);` (wrapping) |
| B | Wizard-Software/BareWire#73 | `RabbitMqTransportAdapter.ConsumeAsync` (an `async` iterator) | `await _connection.CreateChannelAsync(…)`, an async factory (`IChannel`, `IAsyncDisposable` + `IDisposable`) | `await consumerChannel.BasicQosAsync(…)` | `_activeConsumerChannels[id] = consumerChannel;` (dictionary field) |

The four fixes are by different owners.

## Reproductions

For each pair: one file `repro/<Id>.cs` with two self-contained classes, `<Id>Buggy` and `<Id>Fixed`. The
acquisition line carries the marker comment `// ACQ:<Id>`.

- **Kept from the real code.**
- the acquisition expression form: `new`, factory call, or awaited async factory;
- the throwing step, as an awaited call that can throw;
- the success hand-over (return, wrap, or dictionary store);
- the enclosing control flow: the retry loop with `try`/`catch` in W, the `try` around the attempt in R, and the
`async` iterator in B;
- each fix's exact cleanup shape: W `try`/`catch` + `DisposeAsync` + rethrow, with acquisition through a
`Func<…>` seam; R a nullable local, a `published` flag and cleanup on the failure path; X `try`/`catch` + wrap +
`DisposeAsync` + rethrow; B early registration into the dictionary, then `try`/`catch` + release.
- **Third-party types are stubs** with the real disposable interfaces and the members used: `ServiceBusClient`,
`McpClient`, `IChannel`/`IConnection`. `ServiceCollection`/`ServiceProvider` are the real types from the local
ASP.NET Core reference pack.
- **Lines not on the resource's path are dropped**: logging, metrics, unrelated locals.
- The reproductions must compile with **0 errors**. An analyzer that sees a compile error proves nothing.

## Analyzers and configurations

One SDK-style project, `net10.0`, with no `PackageReference`. Analyzers are referenced by path:

- the .NET SDK `Microsoft.CodeAnalysis.NetAnalyzers` (CA2000);
- `IDisposableAnalyzers` 4.0.8 (IDISP001–IDISP026);
- `SonarAnalyzer.CSharp` 10.35.0.4138 (S2930 and others).

Every rule of every analyzer is set to warning (`dotnet_analyzer_diagnostic.severity = warning`). Configurations:

| K | `dispose_analysis_kind` | ownership transfer at constructor | ownership transfer at method call |
|---|---|---|---|
| K1 | default (`NonExceptionPaths`) | default (false) | default (false) |
| K2 | `AllPaths` | false | false |
| K3 | `AllPaths` | **true** | false |
| K4 | `AllPaths` | false | **true** |
| K5 | `AllPathsOnlyNotDisposed` | false | false |

**Harness control.** A two-method file, `Control.cs`:

- `new StreamReader(path)`, then a call that can throw, then `Dispose()`. CA2000 must report it under K2 and must
not report it under K1.
- the same with `using`: silent in every configuration.

If this control fails, the harness is invalid and the gate stops as `INVALID`, not as KILL.

## Scoring (fixed now)

- **Relevant diagnostic.** A diagnostic whose message is about disposal or ownership, located on the leaked
resource's acquisition statement or its declared variable, in the buggy method.
- **Pair verdict.**
- **`CAUGHT`:** at least one configuration gives a relevant diagnostic on the buggy reproduction. This holds even
when the fixed reproduction also gets one: a flag on the buggy site counts against N1's novelty. Whether the
flag also distinguishes the fix is recorded separately.
- **`MISSED`:** no relevant diagnostic on the buggy reproduction in any of K1–K5, from any of the three analyzers.

**Decision.**
- **`GO-CANDIDATE-N1`** (for an owner decision, not a PoC GO) if at least **2** pairs are `MISSED`.
- **`KILL-N1`** otherwise.

Fidelity risk: a stub or a dropped line can change an analyzer's result. Every `MISSED` is therefore re-checked by
listing the analyzer's documented limits for that shape: a factory, an `async` iterator or a dictionary store. A
`MISSED` that is explained only by a reproduction deviation from the real code becomes `UNRESOLVED` and does not
count toward GO.
100 changes: 100 additions & 0 deletions docs/evidence/n1-competitor-gate-2026-10/REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# N1 competitor gate — report

**Result: `GO-CANDIDATE-N1`.** This is a decision for the owner, not a GO on an N1 PoC.

| Pair | Verdict | What CA2000 (`AllPaths`) does |
|---|---|---|
| W, whizbang#1204 | CAUGHT | reports the buggy `new ServiceBusClient(…)` in K2–K4; silent on the fix |
| R, rockbot#638 | MISSED | does not track `await McpClient.CreateAsync(…)` |
| X, NexusKit#34 | MISSED | does not track `services.BuildServiceProvider(…)` |
| B, BareWire#73 | MISSED | does not track `await _connection.CreateChannelAsync(…)` |

So 3 of 4 pairs are missed by every configuration (K1–K5) of all three analyzers: CA2000, IDisposableAnalyzers
4.0.8 and Sonar 10.35. In run 2, none of the three analyzers reports anything on the R, X or B acquisition lines.

Rules: [`00-preregistration.md`](00-preregistration.md), committed in `d06caf75` before any analyzer run. Time used:
21:11–21:20 +05 on 2026-10-20 for the scored work, inside the 0.25 agent-day budget.

No third-party project was built and no package was downloaded. The analyzers came from the SDK and the local NuGet
cache. Every build ran from a copy in the session scratch directory, with a nuget.org-only config and no
`PackageReference`.

## 1. Runs, in order (runs 1 and 2 kept in `data/`; run 0's output was overwritten and is not kept)

| Run | What changed | Result |
|---|---|---|
| 0 | first build | 0 diagnostics matched, because of a bug in my SARIF reader: csc wrote SARIF 1.0 and the reader expected 2.1. Not a result; the reader was fixed. No rule changed |
| 1 (`run1_stubs_in_compilation.tsv`) | reader fixed | control valid; W CAUGHT; R, X, B silent |
| 2 (`run2_scored.tsv`) | **scored run**: the stubs moved into a separate referenced assembly (§2) | control valid; W CAUGHT; R, X, B MISSED |

The harness control was valid in both runs 1 and 2:
- `new StreamReader` followed by an opaque throwing call and `Dispose` is reported under K2–K5 and silent under K1;
- the same with `using` is silent everywhere.

## 2. Fidelity correction before scoring (disclosed)

The pre-registration requires every MISSED to be checked for a reproduction deviation. The first check found one.

In run 1, the stubs were in the analysed compilation. `McpClient.CreateAsync` is a stub whose body is `throw`, and
CA2000's interprocedural analysis can read that body.

The real types (`ServiceBusClient`, `McpClient`, `IChannel`) live in external packages whose bodies the analyzer
does not see. Run 2 therefore compiles the stubs as their own assembly (`stubs/`), referenced by the reproductions.
The reproductions and the rules did not change. The verdicts were the same in both runs.

## 3. Why R, X and B are missed: analyzer limits, not reproduction artifacts

The probes are not scored (`probes/`, `data/probes_opaque_step.tsv`, `data/ablation_and_probes.tsv`). Each one has
the same leak: acquire a disposable, make an **opaque** throwing call through an interface, then dispose or hand it
over. Only the acquisition form changes.

A first probe set used empty local `MayThrow` methods. CA2000 proved that those cannot throw and stayed silent, so
the set was invalid; its output was overwritten and is not kept. It was redone with interface calls, which is how
the reproductions throw.

| Acquisition form (K2, `AllPaths`) | Tracked? |
|---|---|
| `new T(…)`; an in-source sync or async factory; `File.OpenText`; an external static sync factory (P10, P12) | yes |
| `new T(…)` then store in a dictionary, wrap and return, return, or inside an `async` iterator (P5–P8) | yes |
| **await of an external async factory** (P11), the shape of R and B | **no** |
| **result of an interface method** (P13), the shape of B (`IConnection.CreateChannelAsync`) | **no** |
| **`ServiceCollection.BuildServiceProvider()`** (P9), the shape of X; the real M.E.DI type | **no**, even in a three-line method; the reason was not established |

The ablation of each MISSED reproduction (`probes/Ablation.cs`) found no other feature that silences the analyzer:
- removing the catch-all retry loop (R) leaves it silent;
- removing the resolve calls (X) leaves it silent;
- removing the iterator, or swapping the interface for a class (B), leaves it silent.

The acquisition form alone decides. In all three pairs that form is the real code's own API, so the MISSEDs stand
and none is `UNRESOLVED`.

## 4. What this does and does not show

Confirmed:
- CA2000 with `AllPaths` is the documented tool for this class, and it does catch the `new`-acquired case (W).
- For the acquisition forms that are common in modern async .NET, it reports nothing on these reproductions:
awaited async factories, interface factory methods and `BuildServiceProvider`.
- IDisposableAnalyzers and Sonar report nothing on any of the four acquisition lines. Inference, not measured: they
accept a return or store as a hand-over and do not model the exception path before it.

Not shown:
- **No real build was run.** The verdicts come from faithful reproductions of the methods with stub third-party
types, not from building the four repositories. CA2000 on the real repositories could differ if a real package
type carries attributes or annotations that the stubs lack.
- **Only the options in the pre-registration were tested.** `interprocedural_analysis_kind` and
`excluded_symbol_names` were not varied.
- W's fixed version is silent partly because the fix acquires through a `Func<…>` seam that CA2000 does not
track. A `new` + `try`/`catch` fix was not tested.
- That Own.NET can do better is not shown. Phase 0 recorded that Own.NET tracks only non-escaping `new` locals and
missed 3/3 here. A detector for factory-acquired and awaited acquisitions would be new work.
- Mature-code yield and precision of such a detector are unknown. This gate measured only competitor coverage.

## 5. Files

- [`00-preregistration.md`](00-preregistration.md)
- [`repro/`](repro/): W, R, X, B buggy/fixed reproductions, `Control.cs` and `N1Repro.csproj`.
- [`stubs/`](stubs/): the third-party stand-ins, built as a separate assembly.
- [`probes/`](probes/): explanatory probes and ablation. `ProbeExternals.cs` goes into the stub assembly for the
ablation run only.
- [`scripts/run_gate.py`](scripts/run_gate.py): builds K1–K5 and collects the diagnostics on each acquisition line.
- `data/`: outputs of run 1, run 2 and the probes.
21 changes: 21 additions & 0 deletions docs/evidence/n1-competitor-gate-2026-10/data/ablation_and_probes.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
config id class file line rule message
K1 _build rc=0 0 errors=0 results=27
K2 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K2 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K2 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
K2 _build rc=0 0 errors=0 results=32
K2 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K3 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K3 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K3 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
K3 _build rc=0 0 errors=0 results=32
K3 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K4 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K4 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K4 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
K4 _build rc=0 0 errors=0 results=32
K4 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K5 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K5 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
K5 _build rc=0 0 errors=0 results=28
K5 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
Loading
Loading