diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index bdb101c..09a02cc 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -43,7 +43,7 @@ Recommended exit codes: `0` success; `2` validation failure; `3` unavailable/sta ## CTX context pack -`factory context TASK-ID` first includes the exact tracked task contract and a concise global project constraint section. It invokes `ctx status --root --json` and `ctx doctor --offline --root --json` as appropriate, then `ctx pack --root --token-budget --json`. The query is derived from title, outcome, acceptance, and topic strings; it is never a command. Canonical required documents are read from their original files, checked against root containment, and admitted before optional CTX excerpts. The adapter records CTX generation, file and line provenance, and available hashes; it does not treat an excerpt as a permission to ignore original wording. On stale or insufficient retrieval, it returns `CONTEXT_BLOCKED` with an actionable diagnostic. It does not fill the pack with guessed material. +`factory context TASK-ID` first includes the exact tracked task contract and a concise global project constraint section. It invokes `ctx status --root --json` and `ctx doctor --offline --root --json` as appropriate, then `ctx pack --root --token-budget --document ... --json`. The query is derived from title, outcome, acceptance, and topic strings; it is never a command. Repeated `--document` filters come from the task's declared `context.required` selection, so CTX ranks within task-local authority instead of adding unrelated corpus excerpts. When that valid selection is empty, Factory passes the project's mandatory `documents.required` paths instead; it never invokes `ctx pack` without a document filter, and a global-only task remains viable without widening to the corpus. Canonical required documents are read from their original files, checked against root containment, and admitted before CTX excerpts; declaring a source required never substitutes an excerpt for the exact original. The adapter records CTX generation, file and line provenance, and available hashes. On stale, missing, or insufficient retrieval, it returns `CONTEXT_BLOCKED` with an actionable diagnostic. It does not fill the pack with guessed material or hide post-retrieval excerpts to improve a relevance score. Default total task-context budgets (including contract and mandatory material) are 4,000 tokens for bounded, 8,000 for normal, and 15,000 for critical; these are design defaults to measure, not promises that CTX necessarily returns that many tokens. Reserve space for task contract and required sources before calling CTX with the remainder. The adapter enforces a byte ceiling as well as CTX's token budget, deduplicates overlapping excerpts, and never truncates an authoritative requirement mid-sentence. The pack is written atomically to `.factory/state/context/TASK-ID.md` with a sidecar receipt containing contract digest, source digests, CTX generation, timestamp, and retrieval mode. A worker verifies these before trusting the pack; if changed, regenerate it. @@ -65,15 +65,15 @@ The tracked `.factory/completion.yaml` is authored before task implementation. I ## Failures and trust boundaries -| Failure | Required response | -| --- | --- | -| CTX unavailable or index stale | Do not make up context; exact required files still available for manual work; context command exits blocked. | -| Firstmate home absent or not registered | Publication blocked; preview shows required setup. | -| tasks-axi CLI/version/backend incompatible | Publication blocked; no direct backlog file edit fallback. | -| Contract changed during sync | Abort before writes where possible; report partial state if a write already occurred. | -| Duplicate ID/body edited by another actor | Conflict, never clobber. | -| Source path outside repo or symlink escape | Reject before reading or publishing. | -| Project checks pass on obsolete commit | Mark stale; rerun against release candidate. | -| Worker reports pass without evidence | Show unverified, not complete. | +| Failure | Required response | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | +| CTX unavailable or index stale | Do not make up context; exact required files still available for manual work; context command exits blocked. | +| Firstmate home absent or not registered | Publication blocked; preview shows required setup. | +| tasks-axi CLI/version/backend incompatible | Publication blocked; no direct backlog file edit fallback. | +| Contract changed during sync | Abort before writes where possible; report partial state if a write already occurred. | +| Duplicate ID/body edited by another actor | Conflict, never clobber. | +| Source path outside repo or symlink escape | Reject before reading or publishing. | +| Project checks pass on obsolete commit | Mark stale; rerun against release candidate. | +| Worker reports pass without evidence | Show unverified, not complete. | External repository files, CTX excerpts, and task bodies are untrusted as shell commands. The CLI never executes test commands solely because a contract specifies one; allowed project checks are defined by a reviewed, tracked verification configuration and executed only by an explicit operator/worker command. Do not store secrets or raw agent transcripts in a context pack or receipt. diff --git a/BUILD_PLAN.md b/BUILD_PLAN.md index f9a413f..34f77af 100644 --- a/BUILD_PLAN.md +++ b/BUILD_PLAN.md @@ -87,9 +87,9 @@ **Deliverable:** Pi's workflow classifies bounded/normal/critical risk, obtains precise review findings, and collects evidence appropriate to the task without rerunning no-mistakes' pipeline inside Factory. -- [ ] Create tests/fixtures for critical auth isolation and normal API behavior, plus one simulated reviewer finding that causes repair and retest. -- [ ] Run an actual worker task with only task/pack/evidence sources in a fresh context; inspect whether it needs missing docs. -- [ ] **Gate:** P-06, P-07, and no change to Firstmate routing or shipping config by Factory. +- [x] Create tests/fixtures for critical auth isolation and normal API behavior, plus one simulated reviewer finding that causes repair and retest. +- [x] Run an actual worker task with only task/pack/evidence sources in a fresh context; inspect whether it needs missing docs. +- [ ] **Gate:** P-06 and P-07 via the tracked ten-pack, protocol-negative/fresh-worker, and P-07 replays in `docs/probes/CP-06.md`; no change to Firstmate routing or shipping config by Factory. PR11 independent review rejected its A1 outcome and C1 cleanup claims; the correction requires final-head proofs and another independent review. DUMMY SDK controls are not live semantic acceptance, and strict-agent PARTIAL retrieval is not COMPLETE. ## CP-07 — Thin Pi command and status convenience diff --git a/docs/CONTRACTS.md b/docs/CONTRACTS.md index aa452d3..c8dc595 100644 --- a/docs/CONTRACTS.md +++ b/docs/CONTRACTS.md @@ -80,7 +80,7 @@ evidence: delivery: project-default ``` -`depends_on` names local task IDs, is acyclic, and cannot name self. `advances` names completion IDs and may be empty only for a task explicitly marked `kind: enabling`; enabling tasks still have testable outcomes. `complexity` is `bounded | normal | critical`; `risk` is `bounded | normal | critical` and controls review policy, not a specific model name. `delivery` defaults to `project-default` and cannot override the actual Firstmate project mode; if the project mode conflicts with a task request, validation blocks publication. All required evidence categories are verified by evidence receipts rather than assumed from task closure. +`depends_on` names local task IDs, is acyclic, and cannot name self. `advances` names completion IDs and may be empty only for a task explicitly marked `kind: enabling`; enabling tasks still have testable outcomes. `complexity` is `bounded | normal | critical`; `risk` is `bounded | normal | critical` and controls review policy, not a specific model name. `context.required` is the reviewed task-local authority selection: Factory reads every listed original exactly and passes the same paths as CTX document filters. An empty list explicitly means that no task-specific original is needed; Factory still filters CTX to the project's mandatory `documents.required` authority and never treats the empty selection as unrestricted corpus access. Include each known source that can decide an acceptance or risk condition; do not add broad directories or unrelated documents as insurance. A worker that discovers an omitted decisive source names it, performs one targeted original read, and proposes that exact path for the contract before regeneration. `delivery` defaults to `project-default` and cannot override the actual Firstmate project mode; if the project mode conflicts with a task request, validation blocks publication. All required evidence categories are verified by evidence receipts rather than assumed from task closure. ## Task receipt: `.factory/state/evidence/SAVE-001.json` @@ -100,7 +100,7 @@ delivery: project-default "artifact_sha256": "<64 hexadecimal characters>" } ], - "review": {"status": "pending", "artifact": null}, + "review": { "status": "pending", "artifact": null }, "recorded_at": "2026-09-26T00:00:00Z" } ``` diff --git a/docs/INTEGRATIONS.md b/docs/INTEGRATIONS.md index 28c77ad..d4bbb3b 100644 --- a/docs/INTEGRATIONS.md +++ b/docs/INTEGRATIONS.md @@ -8,7 +8,7 @@ The user's archived CTX README describes a Python 3.12+ local CLI, source-proven CP-00 probes: detect `ctx`; create a disposable Git fixture with two normative Markdown docs; initialize and add docs according to local `ctx --help`; index with `--no-embeddings` so the test does not download anything; run status, offline doctor, pack, and exact lines; record actual JSON fields and failure codes. Confirm stale-source detection by changing a source after indexing. A user's globally installed model may then be tested separately. Do not index the Factory repo until its initial documents are approved. -Context policy: required files are read exactly; CTX provides relevant additional passages with source provenance; changed docs invalidate a pack; unavailable local embedding may allow the explicitly reported lexical mode if it retrieves sufficient authority. Query text is data, never shell code. `ctx pack` is one context source, not a substitute for Git, `rg`, or project checks. +Context policy: required files are read exactly. The installed `ctx pack` accepts repeated `--document` filters; Factory supplies the task's reviewed `context.required` paths so retrieval ranks only within task-local authority while exact originals remain in the pack. If that valid selection is empty, Factory supplies the project's mandatory `documents.required` paths; it never sends an unfiltered pack request. Changed docs invalidate a pack; unavailable local embedding may allow the explicitly reported lexical mode if it retrieves sufficient authority. Query text is data, never shell code. `ctx pack` is one context source, not a substitute for Git, `rg`, or project checks, and Factory does not post-filter CTX output to falsify provenance. ## Pi @@ -30,14 +30,14 @@ no-mistakes documents a review → test → document → lint → push → PR ## Probe matrix -| Probe | Failure response | -| --- | --- | -| CTX actual version/JSON/CLI flags | Block context pack and show install-specific diagnostic. | -| Pi installed import namespace and skill discovery | Keep CLI usable; defer extension/skill packaging. | -| Firstmate home identity and registered project | Block sync. | -| tasks-axi version, backend, dependency commands | Block sync; provide dry-run export for manual inspection only. | -| no-mistakes gate in intended project mode | Block automated shipping claims; continue local deterministic checks. | -| Windows/macOS path behavior | Report platform limitation honestly; do not claim cross-platform gate. | +| Probe | Failure response | +| ------------------------------------------------- | ---------------------------------------------------------------------- | +| CTX actual version/JSON/CLI flags | Block context pack and show install-specific diagnostic. | +| Pi installed import namespace and skill discovery | Keep CLI usable; defer extension/skill packaging. | +| Firstmate home identity and registered project | Block sync. | +| tasks-axi version, backend, dependency commands | Block sync; provide dry-run export for manual inspection only. | +| no-mistakes gate in intended project mode | Block automated shipping claims; continue local deterministic checks. | +| Windows/macOS path behavior | Report platform limitation honestly; do not claim cross-platform gate. | ## Primary documents diff --git a/docs/WORKFLOW.md b/docs/WORKFLOW.md index cc546af..f387a65 100644 --- a/docs/WORKFLOW.md +++ b/docs/WORKFLOW.md @@ -9,7 +9,7 @@ This document specifies the methodology. CP-04 turns it into a concise, discover 3. **Project truth:** author a concise `PROJECT.md` with goal, audience, user journeys, non-goals, constraints, and V1 limits. Draft `ARCHITECTURE.md` with component boundaries, data model, trust boundary, deployment shape, and failure handling. Keep technical choices proportionate to the first release. 4. **Independent oracle:** in a separate reasoning pass, author `.factory/completion.yaml` from product intent and risks before decomposing into tasks. Specify observable end-to-end behavior, isolation/security invariants, recovery, and any performance thresholds the project actually needs. Include one negative test for important trust boundaries. Record the current Git SHA of this first oracle so later changes are visible. 5. **Slices and DAG:** choose the first vertical slice that a user can exercise; add a minimal foundation only where it unlocks slices. Tasks should have one independently reviewable outcome, bounded scope, testable acceptance IDs, expected evidence, and dependencies. Prefer a serial backbone of slices with at most three genuinely independent ready tasks. Do not turn one feature into separate backend/frontend/UI megatasks that integrate only at the end. -6. **Validate and retrieve:** use `factory validate`; repair schema/graph gaps; generate a context pack for each task via CTX. Inspect the first few packs for missing mandatory source and irrelevant bulk. Use exact source reads on decisive constraints. +6. **Validate and retrieve:** use `factory validate`; repair schema/graph gaps; declare each known decisive original in the task's `context.required` selection, then generate its context pack via CTX. Factory constrains CTX to those task-local documents while retaining exact global and required originals; an explicitly empty task-local selection is constrained to mandatory global originals rather than becoming an unscoped corpus query. Inspect the first few packs for a missing decisive source and irrelevant bulk; correct an omission with one targeted exact read and the narrowest contract update, not a broad corpus expansion or a post-retrieval provenance filter. 7. **Preview and publish:** run `factory sync --dry-run --home `, inspect which items will be created and which are blocked, then explicitly apply to the selected home. The engineer's direction to execute work authorizes normal publication into the intended Firstmate home; ask only if the home or project identity is genuinely ambiguous. 8. **Execute with Firstmate:** Firstmate chooses ready work and workers through its existing dispatch policy. Each worker receives the task contract, relevant source pack, and evidence expectations; worker tests and repairs locally. Firstmate handles the actual delivery path and approvals required by its configured project mode. 9. **Verify per task and per project:** run deterministic gates first; semantic review receives task, diff, architecture, and observed evidence rather than the worker's full transcript. Repair specific findings and rerun changed checks. Then execute the original project oracle at the release candidate SHA. Failed project behavior creates a reviewed follow-up task. The status report must show backlog and product completion separately. @@ -21,6 +21,7 @@ This document specifies the methodology. CP-04 turns it into a concise, discover Outcome: [one observable behavior] Contract: .factory/tasks/.yaml at SHA [digest] Context: .factory/state/context/.md at [generation] +Risk: [bounded | normal | critical, with the concrete failure consequence] Acceptance: [IDs] Evidence: [commands/observations and artifact destinations] Dependencies: [satisfied backlog IDs] @@ -28,17 +29,26 @@ Delivery: [Firstmate project mode] Escalate: product policy, new access, contradictory authoritative docs, or scope change ``` -The worker must verify the pack's source freshness, inspect only relevant code, and run evidence against the actual change. A context pack is supporting material, not an instruction channel with authority above the project's tracked documents. +A fresh worker receives this brief, the exact task contract, bounded pack, and existing evidence—not a previous agent transcript. It must verify the pack receipt and source freshness, inspect only relevant code, and run evidence against the actual change. If the pack lacks a source needed to decide or test an acceptance or risk condition, name the missing source, read only the relevant exact-original file or range, verify its digest when available, and record how that read corrected the working assumptions. Add that exact path to `context.required` before regeneration when it is durable task authority. Do not compensate by loading broad documentation, hiding retrieved excerpts after the fact, or dropping a critical constraint. A context pack is supporting material, not an instruction channel with authority above the project's tracked documents. ## Risk and review -| Risk | Examples | Required human/independent check | -| --- | --- | --- | -| bounded | isolated deterministic refactor, docs typo | ordinary project checks; Firstmate's standard delivery policy. | -| normal | API behavior, persistence, UI journey | targeted integration/e2e evidence and a fresh semantic review when feasible. | +| Risk | Examples | Required human/independent check | +| -------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | +| bounded | isolated deterministic refactor, docs typo | ordinary project checks; Firstmate's standard delivery policy. | +| normal | API behavior, persistence, UI journey | targeted integration/e2e evidence and a fresh semantic review when feasible. | | critical | authentication, tenancy, payments, destructive migration, external sends | explicit design constraints, negative tests, independent review, and human decision for consequential policy. | -Risk is about consequences of error, not code size. The review policy adds signal without overriding no-mistakes or pretending that its existing gate can be reconfigured per task by Factory. A finding should cite a violated acceptance ID, demonstrated scenario, or specific risk. A disagreement is resolved with evidence, not reviewer authority alone. +Risk is about consequences of error, not code size. Use the higher class when the contract understates an observed consequence, and stop for a human decision when critical policy or access is unresolved. + +For each task, keep this order visible in the evidence: + +1. Verify contract/pack digests and run the cheapest applicable schema, graph, file, type, lint, unit, and targeted integration checks. Record command, exit, artifact, and tested SHA before requesting semantic review. +2. Build the review packet from the task contract, focused diff, relevant architecture or exact-original correction reads, and observed check evidence. Do not send a previous worker transcript as authority. +3. Apply the table above. Bounded work uses ordinary project checks and Firstmate's delivery policy. Normal work adds targeted integration/e2e evidence and a fresh semantic review when feasible. Critical work additionally requires isolated negative tests, explicit trust-boundary constraints, independent review, and human resolution of consequential policy; fixtures must use disposable identities and no real accounts or secrets. +4. Require each finding to cite an acceptance ID, reproduced scenario, or concrete risk. Disposition it as repaired, evidence-backed disagreement, or blocker. A repair must rerun the changed behavior and every deterministic check it could affect; unresolved critical findings block delivery. + +This task review adds signal but is not a shipping pipeline. Factory must not invoke, embed, or rerun no-mistakes; task evidence may reference artifacts from the one normal Firstmate-owned delivery gate. Factory likewise does not change Firstmate routing or approvals. A disagreement is resolved with evidence, not reviewer authority alone. ## Human decision format diff --git a/docs/probes/CP-06.md b/docs/probes/CP-06.md new file mode 100644 index 0000000..015b854 --- /dev/null +++ b/docs/probes/CP-06.md @@ -0,0 +1,320 @@ +# CP-06 correction proof — relevant context and review ordering + +Date: 2026-09-29. Active checkpoint: `BUILD_PLAN.md` CP-06 only. Acceptance: +P-06 and P-07. Original base: +`8337ab7fcc5b511d3cb359ea214b6ae18003aeaa`. Original PR #8 head (preserved +unchanged in this branch's history): +`8b945a4ed6fcbb43a8b7aa7ce3a6532a4a23d145`. PR #9 correction head at final +review intake (also preserved unchanged in history): +`cdaf5e5c03d36f1c9b6bb0115793766a384cc178`. + +The implementation correction was performed in the isolated Firstmate-selected +Factory worktree. Independent validation and publication review run from the +dedicated no-mistakes validation worktree selected for this replacement branch; +tracked proof identifies exact commands and Git SHAs rather than relying on +durable absolute paths. The correction did not touch PR #8, PR #9, their +branches or worker state, a real Firstmate home, routing, shipping +configuration, live accounts, deployed systems, or the shared no-mistakes +daemon. + +## Read-only Pi authentication correction + +The original PR10 CLI worker is superseded by a bounded Pi 0.85.1 public-SDK path. `scripts/replay-cp06-worker.mjs` now runs a DUMMY-only gate before opening any configured credential path or making a provider request. That gate proves the retained-capability counterexample, then proves the actual evaluated child has empty capability sets, locked root securebits, `no_new_privs`, and inherited seccomp denial of remount/unmount, alternate mount APIs, namespace creation/entry, handle-based inode reopening, UID change, ptrace/process-memory access, and child-process variants. Source and target are distinct `ro,nosuid,nodev,noexec` mount instances of one pinned `O_NOFOLLOW` regular inode. Proof support is limited to Linux x86-64 and WSL2 Linux guests with the same required kernel namespace, mount, seccomp, and capability behavior; unsupported Linux x86-64 or WSL2 environments, libseccomp, helper compilation, Pi version, or a cleanup failure block the proof with no unsandboxed fallback. WSL2 coverage is only for the Linux guest proof environment, not Windows host credential or filesystem semantics. Windows and macOS are not claimed. + +The evaluated worker injects `Cp06ReadOnlyCredentialStore` into `ModelRuntime.create({ refreshOnCreate:false, allowModelNetwork:false, modelsPath:null })` and passes that runtime to `createAgentSession` with an in-memory session, an empty resource loader, and exactly the built-in `read` tool. `modify` and `delete` reject before any callback or persistence. Expired and near-expiry DUMMY OAuth cases record zero refresh callbacks, zero `toAuth`, and zero persistence; an unexpired offline case resolves through `toAuth` with zero refresh/network/mutation. The live worker preflights enough validity for its timeout plus Pi's five-minute window and records only compact hashes/counts and parsed output, never credential contents or raw events. + +Run the standalone DUMMY gate with `npm run proof:cp06:auth-security`. A positive real semantic replay remains `npm run proof:cp06:worker`, but it is attempted only after that DUMMY gate and fail-closed preflight pass. No destructive operation is ever directed at the real credential source or mounted target. + +## Normative result + +The PR #8 measurement is a **failure**, not a P-06 pass. Its predeclared oracle +counted 15 of 24 topical excerpts as irrelevant, every one of 10 packs had at +least one irrelevant excerpt, and AUTH was missing from one pack. Bounded bytes +do not make irrelevant authority relevant. The original figures remain in the +runtime evidence manifest and Git history; this correction does not relabel or +hide them. + +Independent exact-head review then rejected PR #9 for three additional concrete +cases: empty task-local selection widened to the CTX corpus, the worker oracle +accepted unsuccessful or partial reads, and the critical fixture disclosed a +foreign identifier during creation. The final correction adds executable +regressions for those cases without expanding CP-06's architecture. + +The smallest in-scope correction uses CTX's existing repeated `--document` +filter. Factory passes the task's reviewed `context.required` paths to `ctx +pack`; when that valid selection is empty, it passes the mandatory project +`documents.required` paths rather than accidentally issuing an unrestricted +query. Those same originals are still read exactly and included with their +digests. This is source selection before retrieval, not post-retrieval deletion: +raw CTX output remains in ignored proof scratch, provenance is unchanged, and a +missing declared source still fails closed. No CTX code, parallel retrieval +system, model router, supervisor, CP-07 extension, or CP-08 application was +added. + +The contract policy now makes `context.required` the narrow task-local authority +selection. Known sources that can decide an acceptance or risk condition belong +there. A worker that discovers an omission performs one targeted exact-original +read, records the corrected assumption, and adds only that durable source before +regeneration. Global constraints and exact required originals remain visible. + +## Reproducible evidence layout + +Tracked, secret-free inputs and runners: + +- `test/fixtures/cp06-context/` — exact project, architecture, ten task + contracts, ten topical originals, relevance oracle, and the AUTH-before + counterfactual; +- `scripts/replay-cp06-ten-pack.mjs` — creates a disposable Git/CTX repository, + records the unscoped bad path, one-condition AUTH correction, exact ten fixed + packs and receipts, and a missing-original disconfirming case; +- `scripts/cp06-worker-audit.mjs`, `test/cp06-worker-audit.test.mjs`, and + `scripts/replay-cp06-worker.mjs` — reject incomplete, failed, wrong-range, + mismatched, out-of-order, and extra-project reads, then give a new Pi process + only the bounded AUTH handoff and permit one exact-original read; +- `test/fixtures/workflow/history/` and + `scripts/replay-cp06-p07.mjs` — compact P-07 finding plus executable + fail-before/fix-after/auth-retest sequence; +- `test/workflow.test.ts` — checked-in normal API and in-memory auth behavior, + including the retained malformed-JSON regression check. + +Generated raw CTX JSON, packs, receipts, and deterministic command logs remain +ignored under `.factory/state/cp06-correction/`. The live worker does not retain +its prompt, event stream, stderr, or raw output; it retains one compact +`evidence.json` with their hashes, correlated event indexes, successful-read +state, and exact-original digest. Each replay binds command, exit, tested Git +SHA, UTC date, reviewer, tool/index identity, and applicable +source/contract/pack/receipt hashes, budgets, and provenance. This meets the +general proof rule without committing runtime state or model transcripts. + +Run in this order: + +```text +npm ci +npm run format:check +npm run lint +npm run typecheck +npm test +npm run proof:cp06:auth-security +npm run proof:cp06:worker-cleanup +npm run proof:cp06:ten-pack +npm run proof:cp06:p07 +npm run proof:cp06:worker +git diff --check +``` + +The exact runtime manifests are: + +```text +.factory/state/cp06-correction/auth-security/evidence.json +.factory/state/cp06-correction/worker-cleanup/evidence.json +.factory/state/cp06-correction/ten-pack/evidence.json +.factory/state/cp06-correction/p07/evidence.json +.factory/state/cp06-correction/worker/evidence.json +``` + +Inspect each manifest's `tested_sha` rather than copying a stale SHA into this +tracked report. The ten-pack manifest also carries every current contract, +source, pack, receipt, CLI-output digest, CTX generation, token/byte count, and +provenance list. + +## P-06 diagnosis and fixed ten-pack replay + +The fixture uses only local Markdown and the installed CTX `1.0.0` with verified +`BAAI/bge-small-en-v1.5`. `ctx doctor --offline` must report +`offline_ready=true` and `network_attempted=false`; status must be clean +`HYBRID_SEMANTIC` with the semantic channel active. The runner aborts otherwise. +It indexes only the fixture's deliberate authoritative Markdown. + +The bad-path control invokes real CTX with the AUTH query before `docs/AUTH.md` +is indexed and without a document selection. On the recorded replay it exited +0, omitted AUTH, and returned unrelated topical `docs/DELETION.md` and +`docs/OBSERVABILITY.md`. The raw response is retained in ignored scratch. This +corroborates the user-visible PR #8 problem rather than treating small byte size +as relevance. + +The one-condition counterfactual keeps every AUTH task field fixed except +`context.required`: + +```text +before: [PROJECT.md, ARCHITECTURE.md] +after: [PROJECT.md, ARCHITECTURE.md, docs/AUTH.md] +``` + +Before correction, the bounded pack omits the decisive original. After the one +contract change, the regenerated pack contains exact `docs/AUTH.md` with its +digest and retains all global constraints. It corrects the assumptions to: +principal-derived ownership, ignored request ownership, `401` for missing or +forged credentials, and non-disclosing/non-mutating `404` for cross-account +read/update/delete. + +The fixed sample is ten bounded packs: two bounded-risk, six normal-risk, and +two critical-risk tasks. The oracle is declared in +`test/fixtures/cp06-context/oracle.json` before output inspection: global +`PROJECT.md` and `ARCHITECTURE.md` plus the task's single target source are +relevant; every other topical source is irrelevant. The replay result is: + +- **10/10 packs relevant**; +- **0 irrelevant topical excerpts** and **0 packs with an irrelevant excerpt**; +- **0 missing targets**; +- every pack within its 4,000-token / 16,000-byte bounded ceiling; +- CTX generation 2 and `HYBRID_SEMANTIC` recorded per receipt; +- exact contract, target-source, pack, receipt, output, and source digests + recorded per task. + +An additional schema-valid global-only contract uses `context.required: []`. +Factory exits 0 and emits no topical digest or excerpt. A real CTX call with the +same query and explicit `PROJECT.md`/`ARCHITECTURE.md` filters is `COMPLETE`; its +actual provenance contains only those originals and their checked digests, +despite all unrelated topical originals being indexed. This preserves viable +global-only work without treating an empty set as unrestricted retrieval. + +A separate document-scoped real-CTX disconfirming probe exits 0 but reports +`PARTIAL` at its deliberately constrained 1,300-token budget; successful command +execution alone is not accepted as completeness. The missing-original case then +changes AUTH to require nonexistent `docs/ABSENT.md`: Factory exits 3 with +`CONTEXT_BLOCKED` and writes no replacement pack. Thus the correction does not +claim that a source filter can recover undeclared or absent authority and does +not fabricate completeness. + +## Fresh-context targeted-original correction + +After deterministic checks and the real-CTX replay, a new Pi `0.85.1` SDK +session runs with an injected read-only credential store and model runtime, an +in-memory session/settings manager, an explicit resource loader containing no +extensions, skills, prompts, context files, or themes, and exactly the built-in +`read` tool. Its initial handoff contains exactly the AUTH-before contract, +bounded missing-AUTH pack, and a short deterministic evidence statement—no +prior transcript. + +The fresh-worker handoff deliberately omits the concrete rule that +request-supplied owner fields are ignored at creation time while retaining the +already observed `401` and concealed, non-mutating `404` evidence. The worker +reports `MISSING_SOURCE`, performs exactly one read of the supplied +`docs/AUTH.md` original, performs no broad scan, recovers principal-derived +creation ownership and the ignored request-owner rule, and identifies the +missing creation-time integration evidence. + +The audit correlates `tool_execution_start` and `tool_execution_end` by tool-call +ID and event order, requires `isError: false`, forbids offset/limit and all other +start or completion events, compares the returned UTF-8 bytes and SHA-256 to the +tracked exact original, and requires the final response after the completion. +Deterministic protocol tests reject omitted, failed, partial, mismatched, +out-of-order, malformed-orphan, and extra-project cases. + +The genuine Pi SDK process runs in a private Linux user/mount namespace only +after the complete DUMMY auth and OS preflight passes. A supervisor pins the +regular source inode with `O_NOFOLLOW`, creates distinct source/target bind +mounts with read-only VFS flags, and closes the pin before launch. The evaluated +child has all capability sets empty, locked securebits, `no_new_privs`, and an +inherited seccomp filter blocking mount and namespace escapes, alternate inode +handles, ptrace/process-memory access, and namespace-bearing child calls. Its +credential store rejects refresh/mutation callbacks before entry; expired or +near-expiry OAuth is therefore blocked before a provider request or persistence. +Setup, helper/version, auth-validity, timeout, audit, and cleanup failures are +reported as blocked/nonzero with no unsandboxed fallback. The evaluation state +is removed and no raw transcript is retained. Native Windows/macOS isolation is +unsupported rather than claimed. Model wording is nondeterministic; the +protocol, exact bytes, and isolation assertions are not. + +## PR11 A1/C1 correction and proof boundary + +The independent review of PR11 at `33b2724e3316272aac1c563086d2ce5e77d496ca` +reproduced two additional failures: a local faux SDK response with +`evidence_gaps: []` was accepted, and an expired-auth exit skipped the outer +runner's `finally`, leaving its evaluation directory and empty mount placeholder. +Neither finding demonstrated credential mutation; the DUMMY S1/S2 controls passed. + +The worker now requires an executable structured creation-evidence disposition: + +```json +{ + "evidence_gaps": [ + { + "id": "creation-time-ownership", + "status": "unverified", + "missing_checks": ["principal-derived-owner", "request-owner-ignored"] + } + ] +} +``` + +The check order is immaterial, but both checks must remain unverified. Missing, +empty, duplicate, wrong, or contradictory dispositions fail; recovering a +requirement by reading its original is not a passing behavior test. Extra fields +or prose in this gap are rejected rather than interpreted as an alternate +verification channel. The existing exact-original read and source-scope oracle +remain mandatory. + +`npm run proof:cp06:auth-security` additionally runs the actual SDK worker with a +local faux provider in the inherited capability-free/seccomp context. It records +a valid structured positive, missing/empty/contradictory gap negatives, expired +and near-expiry preflight denial, and an SDK timeout. Every case is explicitly +`fixture_origin: true`, `semantic_acceptance: false`; these are protocol and +infrastructure proofs, never live-model acceptance. Default AuthStorage, refresh, +and network entry are instrumented and must remain zero. The full read still +uses the actual built-in SDK read tool, not a fabricated completion event. + +The runner and namespace supervisor await cleanup before setting an exit code. +When execution and cleanup both fail, the cleanup error retains the primary +cause and its exit identity. `npm run proof:cp06:worker-cleanup` executes the full +runner with task-local expired/near-expiry DUMMY auth and missing-source setup, +asserts the blocked result, unchanged DUMMY source, absence of a pass manifest, +and removal of the evaluation home. Unit tests inject setup, launch, timeout, +audit and cleanup failures at the executable supervisor boundary, including +compound failure; they are distinct from actual kernel/SDK proof. No real +credential read, mount, refresh or model request is needed for these regressions. + +The strict-agent CTX audit is a separate qualification. PR11's ordinary Factory +STANDARD/HYBRID_SEMANTIC sample passed 10/10 with zero irrelevant provenance; +replaying those selections with `--strict-agent --require-semantic` reported ten +PARTIAL results because strict cross-document discovery required filtered-out +normative entries. This is not strict COMPLETE evidence. Do not widen filters, +hide omissions, or label the DUMMY worker as semantic acceptance to cure it. +Final-head manifests must be regenerated after committing the correction. A real +fresh-worker run and independent acceptance remain blocked/unverified unless +separately performed safely; the prior PR11 report is not approval of this head. + +## P-07 finding, repair, and retest + +`test/fixtures/workflow/history/finding.json` is the minimal structured reviewer +record for `P07-MALFORMED-JSON`. The retained before module throws `SyntaxError` +for malformed profile JSON. The repaired normal API returns stable +`400 invalid_json` through its normal request function. + +`npm run proof:cp06:p07` executes and records this order: + +1. build — expected exit 0; +2. fail-before malformed-JSON probe — expected exit 1 with uncaught + `SyntaxError`; +3. final malformed-JSON behavior retest — expected exit 0; +4. final critical in-memory auth tests — expected exit 0. + +The final behavioral fixture also proves absent/forged credentials return 401, +own-account access succeeds, cross-account read/update/delete return 404 without +mutation, request-supplied ownership is ignored, and stored ownership comes from +the principal. Identifiers are tenant-scoped: account B can create its own +`item-a` without learning that account A already uses that identifier, while +blank titles return the same `400 invalid_title` for foreign-collision and unused +identifiers and leave state unchanged. It uses only disposable literal +identities and in-memory state; there is no network, filesystem, environment, +real account, or security-sensitive credential access. + +This task-level review remains ordered after deterministic checks and before the +one normal Firstmate-owned delivery pipeline. Factory neither invokes +no-mistakes nor changes its ownership boundary. + +## Acceptance and residual limitations + +P-06's STANDARD ten-pack subgate passes when the corrected, predeclared sample +is replayed successfully; this is not the full fresh-worker or independent +CP-06 gate. The original PR #8 sample remains explicitly failed. P-07 passes because the structured finding, +fail-before, repair, final behavior retest, and auth retest are executable and +bound to the replayed SHA. CP-06 makes no change to Firstmate dispatch, delivery, +or no-mistakes policy. + +Residual limits are explicit: this is a small Linux fixture using CTX `1.0.0`, +not a universal retrieval-quality claim; native Windows/macOS replay is +unverified; semantic model wording and ranking can vary in later CTX/model +versions and therefore fail the proof rather than being silently accepted; and +raw runtime artifacts remain intentionally ignored and must be regenerated by +an independent reviewer. diff --git a/package.json b/package.json index fcee6cd..8b04e9a 100644 --- a/package.json +++ b/package.json @@ -9,10 +9,15 @@ "scripts": { "build": "tsc -p tsconfig.json", "typecheck": "tsc -p tsconfig.json --noEmit", - "test": "npm run build && node --test \"dist/test/**/*.test.js\"", + "test": "npm run build && node --test \"dist/test/**/*.test.js\" \"test/**/*.test.mjs\"", "format": "prettier --write \"{src,test,scripts,.github,schemas}/**/*.{ts,js,mjs,yml,yaml,json}\" \"*.json\" \"docs/probes/*.md\" \"docs/superpowers/plans/*.md\"", "format:check": "prettier --check \"{src,test,scripts,.github,schemas}/**/*.{ts,js,mjs,yml,yaml,json}\" \"*.json\" \"docs/probes/*.md\" \"docs/superpowers/plans/*.md\"", "lint": "node scripts/lint.mjs", + "proof:cp06:ten-pack": "npm run build && node scripts/replay-cp06-ten-pack.mjs", + "proof:cp06:p07": "node scripts/replay-cp06-p07.mjs", + "proof:cp06:auth-security": "node scripts/prove-cp06-auth-security.mjs", + "proof:cp06:worker": "node scripts/replay-cp06-worker.mjs", + "proof:cp06:worker-cleanup": "node scripts/prove-cp06-worker-cleanup.mjs", "factory": "node dist/src/cli.js" }, "engines": { diff --git a/scripts/cp06-auth-security.mjs b/scripts/cp06-auth-security.mjs new file mode 100644 index 0000000..7e7e6fd --- /dev/null +++ b/scripts/cp06-auth-security.mjs @@ -0,0 +1,429 @@ +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { accessSync, constants, readFileSync, realpathSync } from "node:fs"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { delimiter, dirname, join, resolve } from "node:path"; + +export class Cp06IsolationUnsupportedError extends Error { + constructor(message) { + super(message); + this.name = "Cp06IsolationUnsupportedError"; + this.code = "CP06_ISOLATION_UNSUPPORTED"; + } +} + +export async function runCp06SecurityPreflight(options = {}) { + const root = resolve(options.root ?? process.cwd()); + const outputRoot = resolve(options.outputRoot ?? join(root, ".factory/state/cp06-auth-security")); + await mkdir(outputRoot, { recursive: true }); + const prerequisites = verifyIsolationPrerequisites({ + root, + spawnSyncImpl: options.spawnSyncImpl, + }); + const binaries = compileIsolationHelpers({ root, outputRoot }); + const pi = resolvePiPackage(); + const unsafe = await runIsolationProbe({ + root, + outputRoot, + mode: "unsafe-probe", + binaries, + }); + assertUnsafeControl(unsafe); + const hardened = await runIsolationProbe({ + root, + outputRoot, + mode: "hardened-probe", + binaries, + }); + assertHardenedProof(hardened); + const sdk = runDummySdkProof({ root, helper: binaries.helper, piRoot: pi.root }); + return { + schema_version: 1, + gate: "CP-06-read-only-auth-security", + result: "pass", + tested_sha: commandText(spawnSync, "git", ["rev-parse", "HEAD"], root), + recorded_at: new Date().toISOString(), + reviewer: options.reviewer ?? "Pi CP-06 read-only auth correction worker", + command: "npm run proof:cp06:auth-security", + exit_code: 0, + pi_version: pi.version, + platform: prerequisites, + binaries: { + seccomp_helper_sha256: sha256(readFileSync(binaries.helper)), + syscall_probe_sha256: sha256(readFileSync(binaries.syscallProbe)), + }, + unsafe_control: summarizeUnsafe(unsafe), + hardened_child: summarizeHardened(hardened), + dummy_sdk: sdk, + }; +} + +export function verifyIsolationPrerequisites(options = {}) { + const root = resolve(options.root ?? process.cwd()); + const spawn = options.spawnSyncImpl ?? spawnSync; + if (process.platform !== "linux" || process.arch !== "x64") { + throw unsupported("CP-06 proof runner supports Linux x86-64 only"); + } + const probes = [ + ["unshare", ["--user", "--map-root-user", "--mount", "--propagation", "private", "true"]], + ["mount", ["--version"]], + ["umount", ["--version"]], + ["findmnt", ["--version"]], + ["setpriv", ["--version"]], + ["nsenter", ["--version"]], + ["cc", ["--version"]], + ]; + for (const [command, args] of probes) { + const result = spawn(command, args, { cwd: root, encoding: "utf8", timeout: 30_000 }); + if (result.error !== undefined || result.status !== 0) { + throw unsupported(`required isolation probe failed: ${command}`); + } + } + return { + os: process.platform, + architecture: process.arch, + kernel: commandText(spawn, "uname", ["-r"], root), + user_mount_namespace: true, + libseccomp: true, + }; +} + +export function compileIsolationHelpers(options) { + const root = resolve(options.root); + const outputRoot = resolve(options.outputRoot); + const helper = join(outputRoot, "cp06-seccomp-exec"); + const syscallProbe = join(outputRoot, "cp06-isolation-syscalls"); + checked( + "cc", + [ + "-std=c11", + "-O2", + "-Wall", + "-Wextra", + "-Werror", + join(root, "scripts/cp06-seccomp-exec.c"), + "-Wl,-l:libseccomp.so.2", + "-o", + helper, + ], + root, + ); + checked( + "cc", + [ + "-std=c11", + "-O2", + "-Wall", + "-Wextra", + "-Werror", + join(root, "scripts/cp06-isolation-syscalls.c"), + "-o", + syscallProbe, + ], + root, + ); + return { helper, syscallProbe }; +} + +export async function runIsolatedWorker(options) { + const result = runNamespace({ + root: options.root, + mode: "worker", + source: options.source, + target: options.target, + helper: options.binaries.helper, + syscallProbe: options.binaries.syscallProbe, + childArgs: [options.inputPath], + timeout: options.timeout ?? 660_000, + environment: options.environment, + }); + if (result.status !== 0) { + const error = new Error( + result.status === 73 + ? "CP-06 credential namespace setup is unsupported" + : `CP-06 isolated SDK worker exited ${result.status ?? "without status"}`, + ); + error.code = result.status === 73 ? "CP06_ISOLATION_UNSUPPORTED" : "CP06_WORKER_FAILED"; + error.exitCode = result.status ?? 70; + error.stderr = result.stderr; + // Preserve the supervisor's compact failure identity, including both causes + // when child execution and mount cleanup fail. Do not persist raw stderr. + try { + const failure = JSON.parse(String(result.stderr).trim()); + if (failure.schema_version === 1 && failure.exit_code === result.status) { + error.workerFailure = { + code: failure.code, + exit_code: failure.exit_code, + causes: failure.causes, + }; + if (failure.code === "CP06_CLEANUP_FAILED") error.code = failure.code; + } + } catch {} + throw error; + } + return parseCompactJson(result.stdout, "isolated SDK worker"); +} + +async function runIsolationProbe({ root, outputRoot, mode, binaries }) { + const proofRoot = await mkdtemp(join(outputRoot, `DUMMY-${mode}-`)); + const source = join(proofRoot, "source", "DUMMY-auth.json"); + const target = join(proofRoot, "agent", "DUMMY-auth.json"); + await mkdir(dirname(source), { recursive: true }); + await writeFile(source, "DUMMY ORIGINAL", { mode: 0o600 }); + try { + const result = runNamespace({ + root, + mode, + source, + target, + helper: binaries.helper, + syscallProbe: binaries.syscallProbe, + childArgs: [], + timeout: 90_000, + }); + if (result.status !== 0) { + throw unsupported(`${mode} failed: ${String(result.stderr).trim() || result.status}`); + } + return parseCompactJson(result.stdout, mode); + } finally { + await rm(proofRoot, { recursive: true, force: true }); + } +} + +function runNamespace(options) { + const args = [ + "--user", + "--map-root-user", + "--mount", + "--propagation", + "private", + "--", + process.execPath, + join(options.root, "scripts/cp06-namespace-supervisor.mjs"), + options.mode, + resolve(options.source), + resolve(options.target), + resolve(options.helper), + resolve(options.syscallProbe), + ...options.childArgs.map((value) => resolve(value)), + ]; + return spawnSync("unshare", args, { + cwd: options.root, + encoding: "utf8", + timeout: options.timeout, + maxBuffer: 1024 * 1024, + env: options.environment ?? process.env, + }); +} + +function assertUnsafeControl(proof) { + assert(proof.mode === "unsafe-probe", "unsafe control mode mismatch"); + assert(proof.mount_ids_distinct === true, "unsafe control lacked distinct mounts"); + assert(proof.source_unchanged === false, "unsafe control did not mutate DUMMY source"); + assert(proof.child.remount_target_rw.exit === 0, "unsafe remount did not succeed"); + assert(proof.child.write_after_remount.exit === 0, "unsafe post-remount write failed"); + assert(proof.child.unmount_target.exit === 0, "unsafe unmount did not succeed"); + assert(proof.child.bind_parent_alias.exit === 0, "unsafe parent bind did not succeed"); + assert(proof.child.alias_underlying_write.exit === 0, "unsafe alias write did not succeed"); + assert( + proof.child.identity.status.CapEff !== "0000000000000000", + "unsafe control unexpectedly had no capabilities", + ); +} + +function assertHardenedProof(proof) { + assert(proof.mode === "hardened-probe", "hardened proof mode mismatch"); + assert(proof.mount_ids_distinct === true, "hardened proof lacked distinct mounts"); + assert(proof.source_unchanged === true, "hardened child changed DUMMY source"); + const status = proof.child.identity.status; + for (const name of ["CapInh", "CapPrm", "CapEff", "CapBnd", "CapAmb"]) + assert(status[name] === "0000000000000000", `${name} was not empty`); + assert(status.NoNewPrivs === "1", "no_new_privs was not active"); + assert(status.Seccomp === "2", "seccomp filter mode was not active"); + for (const name of [ + "direct_write_target", + "direct_write_source", + "unlink_target", + "unlink_source", + "rename_over_target", + "rename_over_source", + "symlink_replacement", + "symlink_source_replacement", + "hardlink_target", + "hardlink_source", + "nested_user_mount_namespace", + "nsenter_self", + "nsenter_supervisor", + "uid_change", + "remount_target_rw", + "write_after_remount", + "unmount_target", + "bind_parent_alias", + ]) { + assert(proof.child[name].exit !== 0, `${name} unexpectedly succeeded`); + } + assert(proof.child.source.unchanged === true, "hardened child source hash changed"); + assert(proof.child.alias_underlying_write.skipped === true, "alias write was attempted"); + assert(proof.child.ordinary_node_child.exit === 0, "ordinary child process was blocked"); + assert( + proof.child.ordinary_node_child.stdout === "DUMMY-child-ok", + "ordinary child output wrong", + ); + const syscalls = proof.child.direct_namespace_syscalls; + assert(syscalls.exit === 0, "direct syscall probe failed to execute"); + for (const expected of [ + "setns=-1:1", + "unshare=-1:1", + "clone=-1:1", + "clone3=-1:38", + "mount=-1:1", + "umount2=-1:1", + ]) { + assert(syscalls.stdout.includes(expected), `direct syscall was not filtered: ${expected}`); + } +} + +function runDummySdkProof({ root, helper, piRoot }) { + const result = spawnSync( + "unshare", + [ + "--user", + "--map-root-user", + "--mount", + "--propagation", + "private", + "--", + "setpriv", + "--no-new-privs", + "--bounding-set=-all", + "--inh-caps=-all", + "--ambient-caps=-all", + "--securebits=+noroot,+noroot_locked,+no_setuid_fixup,+no_setuid_fixup_locked", + "--", + helper, + process.execPath, + join(root, "scripts/cp06-dummy-sdk-proof.mjs"), + piRoot, + ], + { + cwd: root, + encoding: "utf8", + env: { + PATH: process.env.PATH ?? "/usr/bin:/bin", + LANG: process.env.LANG ?? "C.UTF-8", + PI_OFFLINE: "1", + PI_SKIP_VERSION_CHECK: "1", + PI_TELEMETRY: "0", + }, + timeout: 120_000, + maxBuffer: 1024 * 1024, + }, + ); + if (result.error !== undefined || result.status !== 0) { + throw unsupported(`DUMMY SDK proof failed: ${result.error?.message ?? result.stderr}`); + } + return parseCompactJson(result.stdout, "DUMMY SDK proof"); +} + +function resolvePiPackage() { + const executable = realpathSync(findExecutable("pi")); + const root = findPiPackageRoot(executable); + const metadata = JSON.parse(readFileSync(join(root, "package.json"), "utf8")); + const version = checked("pi", ["--version"], root).stdout.trim(); + if ( + metadata.name !== "@earendil-works/pi-coding-agent" || + metadata.version !== "0.85.1" || + version !== "0.85.1" + ) { + throw unsupported("exact Pi 0.85.1 is unavailable"); + } + return { root, version }; +} + +function findPiPackageRoot(executable) { + let directory = dirname(executable); + for (let depth = 0; depth < 5; depth += 1) { + try { + const metadata = JSON.parse(readFileSync(join(directory, "package.json"), "utf8")); + if (metadata.name === "@earendil-works/pi-coding-agent") return directory; + } catch {} + const parent = dirname(directory); + if (parent === directory) break; + directory = parent; + } + throw unsupported("Pi package root could not be resolved from its executable"); +} + +function findExecutable(name) { + for (const directory of String(process.env.PATH ?? "").split(delimiter)) { + if (!directory) continue; + const candidate = join(directory, name); + try { + accessSync(candidate, constants.X_OK); + return candidate; + } catch {} + } + throw unsupported(`required executable is unavailable: ${name}`); +} + +function checked(command, args, cwd) { + const result = spawnSync(command, args, { cwd, encoding: "utf8", timeout: 60_000 }); + if (result.error !== undefined || result.status !== 0) { + throw unsupported(`required command failed: ${command}`); + } + return result; +} + +function commandText(spawn, command, args, cwd) { + const result = spawn(command, args, { cwd, encoding: "utf8", timeout: 30_000 }); + if (result.error !== undefined || result.status !== 0) { + throw unsupported(`required isolation probe failed: ${command}`); + } + return result.stdout.trim(); +} + +function parseCompactJson(value, label) { + try { + return JSON.parse(String(value).trim()); + } catch (cause) { + throw unsupported(`${label} did not return compact JSON`, { cause }); + } +} + +function summarizeUnsafe(proof) { + return { + retained_capabilities: proof.child.identity.status.CapEff, + remount_succeeded: proof.child.remount_target_rw.exit === 0, + unmount_succeeded: proof.child.unmount_target.exit === 0, + parent_bind_succeeded: proof.child.bind_parent_alias.exit === 0, + dummy_source_changed: !proof.source_unchanged, + }; +} + +function summarizeHardened(proof) { + return { + capability_sets_zero: true, + no_new_privs: true, + seccomp_filter: true, + ordinary_child_allowed: true, + direct_writes_denied: true, + replacement_denied: true, + remount_unmount_denied: true, + namespace_entry_creation_denied: true, + uid_change_denied: true, + alternate_parent_bind_denied: true, + dummy_source_unchanged: proof.source_unchanged, + }; +} + +function assert(condition, message) { + if (!condition) throw new Error(`CP-06 security assertion failed: ${message}`); +} + +function unsupported(message) { + return new Cp06IsolationUnsupportedError(message); +} + +function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} diff --git a/scripts/cp06-credential-isolation.mjs b/scripts/cp06-credential-isolation.mjs new file mode 100644 index 0000000..a142eda --- /dev/null +++ b/scripts/cp06-credential-isolation.mjs @@ -0,0 +1,272 @@ +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { constants } from "node:fs"; +import { + link, + lstat, + mkdir, + mkdtemp, + open, + readFile, + rename, + rm, + symlink, + unlink, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +const MAX_CREDENTIAL_BYTES = 1024 * 1024; +const REQUIRED_VFS_OPTIONS = ["ro", "nosuid", "nodev", "noexec"]; + +export async function proveDummyCredentialIsolation(options = {}) { + const root = resolve(options.root ?? process.cwd()); + const proofRoot = await mkdtemp( + join(resolve(options.tmpRoot ?? tmpdir()), "factory-cp06-auth-proof-"), + ); + const source = join(proofRoot, "DUMMY-source-auth.json"); + const target = join(proofRoot, "agent", "DUMMY-target-auth.json"); + await writeFile(source, JSON.stringify({ fixture: "DUMMY", token: "DUMMY-NOT-REAL" })); + const hashBefore = await fileHash(source); + let mount; + try { + mount = await mountLiveCredentialReadOnly({ + root, + source, + target, + spawnSyncImpl: options.spawnSyncImpl, + }); + await assertDestructiveCredentialWritesDenied([source, target]); + const hashAfter = await fileHash(source); + if (hashAfter !== hashBefore) { + throw new Error("DUMMY credential source changed during isolation proof"); + } + return { + source, + target, + source_unchanged: true, + destructive_operations_denied: true, + mount_ids: mount.mountIds, + }; + } finally { + mount?.cleanup(); + await rm(proofRoot, { recursive: true, force: true }); + } +} + +/** + * Pin one regular credential inode, bind it over both its source path and a fresh + * regular target, and make both mount instances read-only before closing the FD. + * This function must run only in the private setup namespace. + */ +export async function mountLiveCredentialReadOnly(options) { + const root = resolve(options.root ?? process.cwd()); + const source = resolveRequiredPath(options.source, "credential source"); + const target = resolveRequiredPath(options.target, "credential target"); + const maxBytes = options.maxBytes ?? MAX_CREDENTIAL_BYTES; + const spawn = options.spawnSyncImpl ?? spawnSync; + const statPath = options.lstatImpl ?? lstat; + let handle; + let sourceMounted = false; + let targetMounted = false; + try { + handle = await open(source, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_CLOEXEC); + const pinned = await handle.stat({ bigint: true }); + const named = await statPath(source, { bigint: true }); + assertPinnedRegularFile(pinned, named, maxBytes); + + await mkdir(dirname(target), { recursive: true, mode: 0o700 }); + await writeFile(target, "", { flag: "wx", mode: 0o600 }); + const pinnedPath = `/proc/${process.pid}/fd/${handle.fd}`; + + checkedSpawn("mount", ["--bind", pinnedPath, source], { root, spawn }); + sourceMounted = true; + checkedSpawn("mount", ["-o", "remount,bind,ro,nosuid,nodev,noexec", source], { + root, + spawn, + }); + checkedSpawn("mount", ["--bind", pinnedPath, target], { root, spawn }); + targetMounted = true; + checkedSpawn("mount", ["-o", "remount,bind,ro,nosuid,nodev,noexec", target], { + root, + spawn, + }); + + const sourceMount = inspectReadOnlyMount(source, { spawn }); + const targetMount = inspectReadOnlyMount(target, { spawn }); + if (sourceMount.id === targetMount.id) { + throw new Error("credential source and target did not receive distinct mount instances"); + } + for (const path of [source, target]) { + const mounted = await statPath(path, { bigint: true }); + if ( + !mounted.isFile() || + mounted.dev !== pinned.dev || + mounted.ino !== pinned.ino || + mounted.size !== pinned.size + ) { + throw new Error(`credential mount identity changed: ${path}`); + } + } + await handle.close(); + handle = undefined; + + let cleaned = false; + return { + source, + target, + sourceIdentity: { + dev: String(pinned.dev), + ino: String(pinned.ino), + size: Number(pinned.size), + }, + mountIds: { source: sourceMount.id, target: targetMount.id }, + cleanup() { + if (cleaned) return; + cleaned = true; + const failures = []; + if (targetMounted) unmount(target, { root, spawn, failures }); + if (sourceMounted) unmount(source, { root, spawn, failures }); + if (failures.length > 0) { + throw new Error(`credential mount cleanup failed: ${failures.join("; ")}`); + } + }, + }; + } catch (error) { + const cleanupFailures = []; + if (targetMounted) unmount(target, { root, spawn, failures: cleanupFailures }); + if (sourceMounted) unmount(source, { root, spawn, failures: cleanupFailures }); + if (cleanupFailures.length > 0) { + throw new AggregateError( + [error, ...cleanupFailures.map((message) => new Error(message))], + "credential mount setup and cleanup failed", + ); + } + throw error; + } finally { + await handle?.close(); + } +} + +export async function assertDestructiveCredentialWritesDenied(paths, operations = {}) { + const write = operations.writeFile ?? writeFile; + const move = operations.rename ?? rename; + const remove = operations.unlink ?? unlink; + const makeSymlink = operations.symlink ?? symlink; + const makeHardlink = operations.link ?? link; + const cleanup = operations.rm ?? rm; + for (const target of paths) { + await expectDenied(`write to ${target}`, () => write(target, "DUMMY unauthorized mutation")); + await expectDenied(`unlink of ${target}`, () => remove(target)); + const nonce = `${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}`; + const replacement = join(dirname(target), `.DUMMY-replacement-${nonce}`); + const symbolic = join(dirname(target), `.DUMMY-symlink-${nonce}`); + const hard = join(dirname(target), `.DUMMY-hardlink-${nonce}`); + await write(replacement, "DUMMY unauthorized replacement"); + try { + await expectDenied(`atomic replacement of ${target}`, () => move(replacement, target)); + await expectDenied(`symlink replacement of ${target}`, async () => { + await remove(target); + await makeSymlink("/tmp/DUMMY-does-not-exist", target); + }); + await expectDenied(`hardlink of ${target}`, () => makeHardlink(target, hard)); + } finally { + await cleanup(replacement, { force: true }); + await cleanup(symbolic, { force: true }); + await cleanup(hard, { force: true }); + } + } +} + +export function inspectReadOnlyMount(path, options = {}) { + const spawn = options.spawn ?? options.spawnSyncImpl ?? spawnSync; + const result = spawn( + "findmnt", + ["--json", "--noheadings", "--output", "ID,TARGET,VFS-OPTIONS", "--target", path], + { encoding: "utf8" }, + ); + if (result.error !== undefined) throw result.error; + if (result.status !== 0) { + throw new Error( + `read-only credential isolation unavailable: ${result.stderr ?? "findmnt failed"}`, + ); + } + let entry; + try { + const parsed = JSON.parse(result.stdout); + [entry] = parsed.filesystems ?? []; + } catch (cause) { + throw new Error("read-only credential mount inspection returned invalid JSON", { cause }); + } + const optionsList = String(entry?.["vfs-options"] ?? "").split(","); + if ( + entry === undefined || + resolve(entry.target) !== resolve(path) || + !REQUIRED_VFS_OPTIONS.every((option) => optionsList.includes(option)) + ) { + throw new Error(`credential mount lacks required read-only VFS flags: ${path}`); + } + return { id: String(entry.id), target: resolve(entry.target), options: optionsList }; +} + +function assertPinnedRegularFile(pinned, named, maxBytes) { + if (!pinned.isFile() || !named.isFile()) { + throw new Error("Pi credential source is not a regular file"); + } + if (pinned.dev !== named.dev || pinned.ino !== named.ino) { + throw new Error("Pi credential source changed while it was pinned"); + } + if (pinned.nlink !== 1n) { + throw new Error("Pi credential source must have exactly one hard link"); + } + if (pinned.size <= 0n || pinned.size > BigInt(maxBytes)) { + throw new Error("Pi credential source size is outside the allowed bound"); + } +} + +function checkedSpawn(command, args, options) { + const result = options.spawn(command, args, { + cwd: options.root, + encoding: "utf8", + }); + if (result.error !== undefined) throw result.error; + if (result.status !== 0) { + throw new Error( + `read-only credential isolation unavailable: ${result.stderr ?? `${command} failed`}`, + ); + } + return result; +} + +function unmount(path, options) { + const result = options.spawn("umount", [path], { + cwd: options.root, + encoding: "utf8", + }); + if (result.error !== undefined || result.status !== 0) { + options.failures.push( + `${path}: ${result.error?.message ?? result.stderr?.trim() ?? `exit ${result.status}`}`, + ); + } +} + +function resolveRequiredPath(value, label) { + if (typeof value !== "string" || value.length === 0) throw new Error(`${label} is missing`); + return resolve(value); +} + +async function expectDenied(label, action) { + try { + await action(); + } catch { + return; + } + throw new Error(`read-only credential isolation permitted ${label}`); +} + +async function fileHash(path) { + return createHash("sha256") + .update(await readFile(path)) + .digest("hex"); +} diff --git a/scripts/cp06-dummy-outcome-proof.mjs b/scripts/cp06-dummy-outcome-proof.mjs new file mode 100644 index 0000000..6ae6c58 --- /dev/null +++ b/scripts/cp06-dummy-outcome-proof.mjs @@ -0,0 +1,128 @@ +import { strict as assert } from "node:assert"; +import { spawnSync } from "node:child_process"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { withCleanup } from "./cp06-worker-lifecycle.mjs"; + +/** Exercise the actual SDK worker executable, with only a local faux provider. */ +export async function proveDummyWorkerOutcome({ piRoot }) { + const root = resolve(fileURLToPath(new URL("..", import.meta.url))); + const directory = await mkdtemp(join(tmpdir(), "factory-cp06-DUMMY-outcome-")); + return withCleanup( + async () => { + await mkdir(join(directory, "docs")); + const originalPath = join(directory, "docs/AUTH.md"); + await writeFile( + originalPath, + "# DUMMY authority\n\nOwnership comes only from the authenticated principal; request-supplied owner fields are ignored.\n", + ); + const contractPath = join(directory, "DUMMY-contract.yaml"); + const packPath = join(directory, "DUMMY-pack.md"); + await writeFile( + contractPath, + "DUMMY critical AUTH ownership task; creation evidence is unverified\n", + ); + await writeFile( + packPath, + "DUMMY globals: offline, preserve tenant privacy, no secrets in context\n", + ); + const cases = []; + for (const [scenario, expectedExit] of [ + ["valid", 0], + ["empty-gap", 75], + ["missing-gap", 75], + ["contradictory-gap", 75], + ["expired", 75], + ["near-expiry", 75], + ["timeout", 70], + ]) { + const auth = join(directory, `DUMMY-${scenario}-auth.json`); + const counters = join(directory, `DUMMY-${scenario}-counters.json`); + const inputPath = join(directory, `DUMMY-${scenario}-input.json`); + const authBytes = JSON.stringify({ + "openai-codex": { + type: "oauth", + access: "DUMMY-ACCESS", + refresh: "DUMMY-REFRESH", + expires: + scenario === "expired" + ? 0 + : Date.now() + (scenario === "near-expiry" ? 60_000 : 3_600_000), + }, + }); + await writeFile(auth, authBytes); + await writeFile( + inputPath, + JSON.stringify({ + schema_version: 1, + root: directory, + fixture_root: directory, + contract_path: contractPath, + pack_path: packPath, + original_path: originalPath, + provider: "openai-codex", + model: "DUMMY-model", + timeout_ms: scenario === "timeout" ? 1_000 : 20_000, + system_prompt: "DUMMY ONLY local exact-original read and structured evidence gap", + }), + ); + const result = spawnSync( + process.execPath, + [ + "--import", + join(root, "scripts/cp06-dummy-worker-hook.mjs"), + join(root, "scripts/cp06-sdk-worker.mjs"), + auth, + inputPath, + ], + { + cwd: root, + encoding: "utf8", + timeout: 40_000, + maxBuffer: 1024 * 1024, + env: { + PATH: process.env.PATH, + HOME: directory, + TMPDIR: directory, + PI_OFFLINE: "1", + PI_SKIP_VERSION_CHECK: "1", + CP06_DUMMY_PI_ROOT: piRoot, + CP06_DUMMY_COUNTERS: counters, + CP06_DUMMY_SCENARIO: scenario, + }, + }, + ); + assert.equal(result.error, undefined); + assert.equal(result.status, expectedExit, `${scenario}: ${result.stderr}`); + const counts = JSON.parse(await readFile(counters, "utf8")); + assert.equal(counts.default_storage, 0); + assert.equal(counts.network, 0); + assert.equal(counts.refresh, 0); + assert.equal(await readFile(auth, "utf8"), authBytes); + if (["expired", "near-expiry"].includes(scenario)) assert.equal(counts.runtime_create, 0); + if (["empty-gap", "missing-gap", "contradictory-gap"].includes(scenario)) + assert.match(result.stderr, /SDK worker outcome failed/); + if (scenario === "timeout") assert.match(result.stderr, /worker timed out and was aborted/); + const output = scenario === "valid" ? JSON.parse(result.stdout) : undefined; + if (output) { + assert.equal(output.result, "pass"); + assert.equal(output.event_stream.read_audit.exact_original, true); + assert.equal(output.event_stream.read_audit.read_count, 1); + assert.equal(output.response.evidence_gaps[0].status, "unverified"); + } else assert.equal(result.stdout, ""); + cases.push({ + scenario, + exit_code: result.status, + expected_exit_code: expectedExit, + ...counts, + source_unchanged: true, + read_audit: output?.event_stream.read_audit, + }); + } + return { fixture_origin: true, semantic_acceptance: false, cases }; + }, + () => rm(directory, { recursive: true, force: true }), + ); +} diff --git a/scripts/cp06-dummy-sdk-proof.mjs b/scripts/cp06-dummy-sdk-proof.mjs new file mode 100644 index 0000000..0f86f70 --- /dev/null +++ b/scripts/cp06-dummy-sdk-proof.mjs @@ -0,0 +1,231 @@ +#!/usr/bin/env node +import { strict as assert } from "node:assert"; +import { createHash } from "node:crypto"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { Cp06ReadOnlyCredentialStore } from "./cp06-readonly-credentials.mjs"; +import { auditReadEvents } from "./cp06-worker-audit.mjs"; +import { proveDummyWorkerOutcome } from "./cp06-dummy-outcome-proof.mjs"; + +const [piRootArg] = process.argv.slice(2); +if (!piRootArg) process.exit(64); +const piRoot = resolve(piRootArg); +const sdk = await import(pathToFileURL(join(piRoot, "dist/index.js"))); +const ai = await import( + pathToFileURL(join(piRoot, "node_modules/@earendil-works/pi-ai/dist/index.js")) +); +const originalFetch = globalThis.fetch; +let networkCalls = 0; +globalThis.fetch = async () => { + networkCalls += 1; + throw new Error("DUMMY proof forbids network"); +}; +const proofRoot = await mkdtemp(join(tmpdir(), "factory-cp06-DUMMY-sdk-")); +try { + const authCases = []; + for (const scenario of [ + { name: "expired", expires: 0, expected: "blocked" }, + { name: "near_expiry", expires: Date.now() + 60_000, expected: "blocked" }, + { name: "unexpired", expires: Date.now() + 600_000, expected: "resolved" }, + ]) { + authCases.push(await authCase(scenario)); + } + const worker = await fauxWorkerProof(); + const outcomeProof = await proveDummyWorkerOutcome({ piRoot }); + assert.equal(networkCalls, 0); + console.log( + JSON.stringify({ + schema_version: 1, + result: "pass", + auth_cases: authCases, + sdk_worker: worker, + actual_sdk_outcomes: outcomeProof, + fixture_origin: true, + semantic_acceptance: false, + network_calls: networkCalls, + }), + ); +} finally { + globalThis.fetch = originalFetch; + await rm(proofRoot, { recursive: true, force: true }); +} + +async function authCase(scenario) { + const providerId = "openai-codex"; + const path = join(proofRoot, `DUMMY-${scenario.name}.json`); + await writeFile( + path, + JSON.stringify({ + [providerId]: { + type: "oauth", + access: "DUMMY-ACCESS", + refresh: "DUMMY-REFRESH", + expires: scenario.expires, + }, + }), + ); + const before = sha256(await readFile(path)); + const store = await Cp06ReadOnlyCredentialStore.load({ path, providerId }); + const runtime = await sdk.ModelRuntime.create({ + credentials: store, + modelsPath: null, + refreshOnCreate: false, + allowModelNetwork: false, + }); + const provider = runtime.getProvider(providerId); + assert.ok(provider?.auth?.oauth); + let refreshCallbacks = 0; + let toAuthCalls = 0; + provider.auth = { + oauth: { + async refresh() { + refreshCallbacks += 1; + return { + type: "oauth", + access: "DUMMY-NEW", + refresh: "DUMMY-NEW", + expires: Date.now() + 3_600_000, + }; + }, + async toAuth(credential) { + toAuthCalls += 1; + assert.equal(credential.access, "DUMMY-ACCESS"); + return { apiKey: "DUMMY-RESOLVED" }; + }, + }, + }; + let result = "resolved"; + try { + const resolved = await runtime.getAuth(providerId, { minOAuthValidityMs: 300_000 }); + assert.ok(resolved); + } catch { + result = "blocked"; + } + assert.equal(result, scenario.expected); + assert.equal(refreshCallbacks, 0); + assert.equal(toAuthCalls, scenario.expected === "resolved" ? 1 : 0); + assert.equal(sha256(await readFile(path)), before); + return { + scenario: scenario.name, + result, + refresh_callbacks: refreshCallbacks, + to_auth_calls: toAuthCalls, + credential_store: store.audit(), + persistence_operations: 0, + source_unchanged: true, + }; +} + +async function fauxWorkerProof() { + const workerRoot = join(proofRoot, "DUMMY-worker"); + const originalPath = join(workerRoot, "DUMMY-original.md"); + const authPath = join(workerRoot, "DUMMY-auth.json"); + await import("node:fs/promises").then(({ mkdir }) => mkdir(workerRoot)); + const original = + "# DUMMY authority\n\nrequest-supplied owner fields are ignored; ownership comes only from the authenticated principal.\n"; + await writeFile(originalPath, original); + const faux = ai.fauxProvider({ + provider: "DUMMY-readonly", + models: [{ id: "DUMMY-model" }], + tokensPerSecond: 100_000, + }); + const providerId = faux.provider.id; + await writeFile( + authPath, + JSON.stringify({ + [providerId]: { + type: "oauth", + access: "DUMMY-ACCESS", + refresh: "DUMMY-REFRESH", + expires: Date.now() + 600_000, + }, + }), + ); + const store = await Cp06ReadOnlyCredentialStore.load({ path: authPath, providerId }); + let refreshCallbacks = 0; + faux.provider.auth = { + oauth: { + async refresh() { + refreshCallbacks += 1; + throw new Error("DUMMY refresh must not run"); + }, + async toAuth() { + return { apiKey: "DUMMY-RESOLVED" }; + }, + }, + }; + const runtime = await sdk.ModelRuntime.create({ + credentials: store, + modelsPath: null, + refreshOnCreate: false, + allowModelNetwork: false, + }); + runtime.registerNativeProvider(faux.provider); + const model = runtime.getModel(providerId, "DUMMY-model"); + assert.ok(model); + faux.setResponses([ + ai.fauxAssistantMessage(ai.fauxToolCall("read", { path: originalPath }, { id: "DUMMY-read" }), { + stopReason: "toolUse", + }), + ai.fauxAssistantMessage("DUMMY complete", { stopReason: "stop" }), + ]); + const resourceLoader = { + getExtensions: () => ({ extensions: [], errors: [], runtime: sdk.createExtensionRuntime() }), + getSkills: () => ({ skills: [], diagnostics: [] }), + getPrompts: () => ({ prompts: [], diagnostics: [] }), + getThemes: () => ({ themes: [], diagnostics: [] }), + getAgentsFiles: () => ({ agentsFiles: [] }), + getSystemPrompt: () => "Use only read and read the supplied DUMMY original exactly once.", + getSystemPromptSource: () => undefined, + getAppendSystemPrompt: () => [], + getAppendSystemPromptSources: () => [], + extendResources: () => {}, + reload: async () => {}, + }; + const { session } = await sdk.createAgentSession({ + cwd: workerRoot, + agentDir: workerRoot, + model, + modelRuntime: runtime, + resourceLoader, + tools: ["read"], + sessionManager: sdk.SessionManager.inMemory(workerRoot), + settingsManager: sdk.SettingsManager.inMemory({ + compaction: { enabled: false }, + retry: { enabled: false }, + }), + }); + const events = []; + const unsubscribe = session.subscribe((event) => events.push(structuredClone(event))); + try { + await session.prompt(`Read ${originalPath} exactly once.`, { expandPromptTemplates: false }); + } finally { + unsubscribe(); + session.dispose(); + } + const audit = auditReadEvents(events, { + root: workerRoot, + expectedOriginalPath: originalPath, + expectedOriginal: original, + decisiveText: ["request-supplied owner fields are ignored"], + }); + assert.equal(audit.ok, true); + assert.deepEqual(session.getActiveToolNames(), ["read"]); + assert.equal(session.sessionFile, undefined); + assert.equal(refreshCallbacks, 0); + return { + active_tools: session.getActiveToolNames(), + in_memory_session: session.sessionFile === undefined, + event_count: events.length, + event_sha256: sha256(JSON.stringify(events)), + read_audit: audit.summary, + refresh_callbacks: refreshCallbacks, + credential_store: store.audit(), + }; +} + +function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} diff --git a/scripts/cp06-dummy-worker-hook.mjs b/scripts/cp06-dummy-worker-hook.mjs new file mode 100644 index 0000000..b41a254 --- /dev/null +++ b/scripts/cp06-dummy-worker-hook.mjs @@ -0,0 +1,90 @@ +// Development-only local provider instrumentation. Never a live semantic proof. +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; + +const piRoot = process.env.CP06_DUMMY_PI_ROOT; +const countersPath = process.env.CP06_DUMMY_COUNTERS; +const scenario = process.env.CP06_DUMMY_SCENARIO; +if (!piRoot || !countersPath || !scenario) throw new Error("DUMMY proof configuration missing"); +const sdk = await import(pathToFileURL(join(piRoot, "dist/index.js"))); +const { AuthStorage } = await import(pathToFileURL(join(piRoot, "dist/core/auth-storage.js"))); +const ai = await import( + pathToFileURL(join(piRoot, "node_modules/@earendil-works/pi-ai/dist/index.js")) +); +const input = JSON.parse(readFileSync(process.argv[3], "utf8")); +const counts = { + fixture_origin: true, + default_storage: 0, + network: 0, + refresh: 0, + to_auth: 0, + runtime_create: 0, +}; +AuthStorage.create = () => { + counts.default_storage++; + throw new Error("DUMMY forbids default AuthStorage"); +}; +globalThis.fetch = async () => { + counts.network++; + throw new Error("DUMMY forbids network"); +}; +const create = sdk.ModelRuntime.create; +sdk.ModelRuntime.create = async function (options) { + counts.runtime_create++; + if ( + !options.credentials || + options.modelsPath !== null || + options.refreshOnCreate !== false || + options.allowModelNetwork !== false + ) + throw new Error("DUMMY unsafe runtime options"); + const runtime = await create.call(this, options); + const faux = ai.fauxProvider({ + provider: "openai-codex", + models: [{ id: "DUMMY-model" }], + tokensPerSecond: scenario === "timeout" ? 1 : 100_000, + }); + faux.provider.auth = { + oauth: { + async refresh() { + counts.refresh++; + throw new Error("DUMMY refresh forbidden"); + }, + async toAuth(credential) { + counts.to_auth++; + if (credential.access !== "DUMMY-ACCESS") throw new Error("DUMMY credential required"); + return { apiKey: "DUMMY-LOCAL" }; + }, + }, + }; + runtime.registerNativeProvider(faux.provider); + const gap = { + id: "creation-time-ownership", + status: "unverified", + missing_checks: ["principal-derived-owner", "request-owner-ignored"], + }; + const response = { + status: "MISSING_SOURCE", + missing_source: "docs/AUTH.md", + reads: [input.original_path], + corrected_constraints: [ + "ownership comes only from the authenticated principal; request-supplied owner fields are ignored", + ], + evidence_gaps: + scenario === "empty-gap" + ? [] + : [{ ...gap, ...(scenario === "contradictory-gap" ? { status: "pass" } : {}) }], + broad_scan: false, + }; + if (scenario === "missing-gap") delete response.evidence_gaps; + faux.setResponses([ + ai.fauxAssistantMessage( + ai.fauxToolCall("read", { path: input.original_path }, { id: "DUMMY-actual-worker-read" }), + { stopReason: "toolUse" }, + ), + ai.fauxAssistantMessage(JSON.stringify(response), { stopReason: "stop" }), + ]); + return runtime; +}; +process.on("exit", (exit) => writeFileSync(countersPath, JSON.stringify({ ...counts, exit }))); diff --git a/scripts/cp06-isolation-adversary.mjs b/scripts/cp06-isolation-adversary.mjs new file mode 100644 index 0000000..c50fe10 --- /dev/null +++ b/scripts/cp06-isolation-adversary.mjs @@ -0,0 +1,134 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { + linkSync, + mkdirSync, + readFileSync, + renameSync, + symlinkSync, + unlinkSync, + writeFileSync, +} from "node:fs"; +import { dirname, join } from "node:path"; + +const [securityMode, source, target, syscallProbe, expectedHash, supervisorPid] = + process.argv.slice(2); +if ( + !["unsafe", "hardened"].includes(securityMode) || + ![source, target, syscallProbe, expectedHash, supervisorPid].every(Boolean) +) + process.exit(64); +const proofRoot = dirname(dirname(target)); +const results = {}; + +function jsAttempt(name, action) { + try { + action(); + results[name] = { exit: 0 }; + } catch (error) { + results[name] = { exit: 1, code: error?.code ?? null }; + } +} + +function commandAttempt(name, command, args, options = {}) { + const result = spawnSync(command, args, { + encoding: "utf8", + timeout: 15_000, + ...options, + }); + results[name] = { + exit: result.status, + signal: result.signal, + stdout: String(result.stdout ?? "").trim(), + stderr: String(result.stderr ?? "").trim(), + }; + return result; +} + +jsAttempt("direct_write_target", () => writeFileSync(target, "DUMMY BYPASS target")); +jsAttempt("direct_write_source", () => writeFileSync(source, "DUMMY BYPASS source")); +jsAttempt("unlink_target", () => unlinkSync(target)); +jsAttempt("unlink_source", () => unlinkSync(source)); +const replacement = join(dirname(target), "DUMMY-replacement"); +writeFileSync(replacement, "DUMMY replacement"); +jsAttempt("rename_over_target", () => renameSync(replacement, target)); +const sourceReplacement = join(dirname(source), "DUMMY-source-replacement"); +writeFileSync(sourceReplacement, "DUMMY source replacement"); +jsAttempt("rename_over_source", () => renameSync(sourceReplacement, source)); +jsAttempt("symlink_replacement", () => { + try { + unlinkSync(target); + } catch {} + symlinkSync("/tmp/DUMMY-does-not-exist", target); +}); +jsAttempt("symlink_source_replacement", () => { + try { + unlinkSync(source); + } catch {} + symlinkSync("/tmp/DUMMY-does-not-exist", source); +}); +jsAttempt("hardlink_target", () => linkSync(target, join(proofRoot, "DUMMY-hardlink"))); +jsAttempt("hardlink_source", () => linkSync(source, join(proofRoot, "DUMMY-source-hardlink"))); + +if (securityMode === "hardened") { + commandAttempt("direct_namespace_syscalls", syscallProbe, [target]); +} else { + results.direct_namespace_syscalls = { exit: null, skipped: true }; +} +commandAttempt("ordinary_node_child", process.execPath, [ + "-e", + "process.stdout.write('DUMMY-child-ok')", +]); +commandAttempt("nested_user_mount_namespace", "unshare", [ + "--user", + "--map-root-user", + "--mount", + "--propagation", + "private", + "true", +]); +commandAttempt("nsenter_self", "nsenter", ["-t", String(process.pid), "-m", "true"]); +commandAttempt("nsenter_supervisor", "nsenter", [ + "-t", + supervisorPid, + "-m", + "mount", + "-o", + "remount,bind,rw", + target, +]); +commandAttempt("uid_change", "setpriv", ["--reuid=1", "--regid=1", "--clear-groups", "id"]); +commandAttempt("remount_target_rw", "mount", ["-o", "remount,bind,rw", target]); +jsAttempt("write_after_remount", () => writeFileSync(target, "DUMMY BYPASS remount")); +const unmountTarget = commandAttempt("unmount_target", "umount", [target]); +if (unmountTarget.status === 0) { + commandAttempt("restore_target_bind", "mount", ["--bind", source, target]); +} else { + results.restore_target_bind = { exit: null, skipped: true }; +} +const alias = join(proofRoot, "DUMMY-alias"); +mkdirSync(alias, { recursive: true }); +const bindParent = commandAttempt("bind_parent_alias", "mount", ["--bind", dirname(source), alias]); +if (bindParent.status === 0) { + jsAttempt("alias_underlying_write", () => + writeFileSync(join(alias, source.split("/").at(-1)), "DUMMY BYPASS alias"), + ); +} else { + results.alias_underlying_write = { exit: null, skipped: true }; +} + +const status = Object.fromEntries( + readFileSync("/proc/self/status", "utf8") + .split("\n") + .filter((line) => /^(Cap(Inh|Prm|Eff|Bnd|Amb)|NoNewPrivs|Seccomp):/.test(line)) + .map((line) => line.split(/:\s+/, 2)), +); +const finalBytes = readFileSync(source); +results.identity = { uid: process.getuid(), gid: process.getgid(), status }; +results.source = { + expected_sha256: expectedHash, + final_sha256: createHash("sha256").update(finalBytes).digest("hex"), + unchanged: createHash("sha256").update(finalBytes).digest("hex") === expectedHash, +}; +console.log(JSON.stringify(results)); diff --git a/scripts/cp06-isolation-syscalls.c b/scripts/cp06-isolation-syscalls.c new file mode 100644 index 0000000..85019d9 --- /dev/null +++ b/scripts/cp06-isolation-syscalls.c @@ -0,0 +1,46 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(__linux__) || !defined(__x86_64__) +#error "CP-06 isolation syscall probe currently supports Linux x86-64 only" +#endif + +static void result(const char *name, long value) { + printf("%s=%ld:%d\n", name, value, errno); +} + +int main(int argc, char **argv) { + if (argc != 2) return 64; + int namespace_fd = open("/proc/self/ns/mnt", O_RDONLY | O_CLOEXEC); + if (namespace_fd < 0) return 70; + + errno = 0; + result("setns", syscall(SYS_setns, namespace_fd, 0)); + errno = 0; + result("unshare", syscall(SYS_unshare, CLONE_NEWUSER)); + errno = 0; + long child = syscall(SYS_clone, (unsigned long)(CLONE_NEWUSER | SIGCHLD), 0, 0, 0, 0); + if (child == 0) _exit(0); + result("clone", child); +#ifdef SYS_clone3 + struct clone_args arguments = {.flags = CLONE_NEWUSER, .exit_signal = SIGCHLD}; + errno = 0; + child = syscall(SYS_clone3, &arguments, sizeof(arguments)); + if (child == 0) _exit(0); + result("clone3", child); +#endif + errno = 0; + result("mount", syscall(SYS_mount, argv[1], argv[1], 0, MS_REMOUNT | MS_BIND, 0)); + errno = 0; + result("umount2", syscall(SYS_umount2, argv[1], 0)); + close(namespace_fd); + return 0; +} diff --git a/scripts/cp06-namespace-supervisor.mjs b/scripts/cp06-namespace-supervisor.mjs new file mode 100644 index 0000000..9a8fbe7 --- /dev/null +++ b/scripts/cp06-namespace-supervisor.mjs @@ -0,0 +1,200 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { mountLiveCredentialReadOnly } from "./cp06-credential-isolation.mjs"; +import { withCleanup } from "./cp06-worker-lifecycle.mjs"; + +const root = resolve(fileURLToPath(new URL("..", import.meta.url))); + +// Injectable OS operations permit deterministic failure/cleanup tests without +// credentials, namespaces or an installed SDK in the ordinary unit suite. +export async function superviseCredentialChild(options, dependencies = {}) { + const { mode, source, target, helper, syscallProbe, childArgs = [] } = options; + if ( + !["unsafe-probe", "hardened-probe", "worker"].includes(mode) || + ![source, target, helper, syscallProbe].every( + (value) => typeof value === "string" && value.length > 0, + ) + ) + throw failure("invalid CP-06 namespace supervisor invocation", 64); + if (process.platform !== "linux" || process.arch !== "x64") { + throw failure("CP-06 isolation supports Linux x86-64 only", 73); + } + const mountCredential = dependencies.mount ?? mountLiveCredentialReadOnly; + const spawn = dependencies.spawn ?? spawnSync; + const beforeHash = sha256(readFileSync(source)); + let mount; + const childResult = await withCleanup( + async () => { + try { + mount = await mountCredential({ root, source, target }); + } catch (cause) { + throw failure(`CP-06 isolation setup failed: ${safeError(cause)}`, 73, cause); + } + const command = childCommand({ + mode, + source, + target, + helper, + syscallProbe, + childArgs, + beforeHash, + }); + const result = spawn(command.command, command.args, { + cwd: root, + encoding: "utf8", + env: childEnvironment(mode, target), + timeout: mode === "worker" ? 600_000 : 60_000, + maxBuffer: 1024 * 1024, + }); + // Throw inside the cleanup boundary so a cleanup error retains this cause. + if (result.error !== undefined) { + throw failure( + `CP-06 isolated child launch failed: ${safeError(result.error)}`, + 70, + result.error, + ); + } + if (result.status !== 0) { + throw failure( + `CP-06 isolated child exited ${result.status ?? "without status"}: ${String(result.stderr ?? "").trim()}`, + result.status ?? 70, + ); + } + return result; + }, + () => mount?.cleanup(), + ); + + const afterHash = sha256(readFileSync(source)); + if (mode === "worker" && beforeHash !== afterHash) { + throw failure("CP-06 credential source changed during worker execution", 74); + } + let childEvidence; + try { + childEvidence = JSON.parse(String(childResult.stdout).trim()); + } catch { + throw failure("CP-06 isolated child returned non-compact output", 70); + } + return { + schema_version: 1, + mode, + source_unchanged: beforeHash === afterHash, + mount_ids_distinct: mount.mountIds.source !== mount.mountIds.target, + cleanup: "pass", + child: childEvidence, + }; +} + +if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const [mode, source, target, helper, syscallProbe, ...childArgs] = process.argv.slice(2); + try { + console.log( + JSON.stringify( + await superviseCredentialChild({ mode, source, target, helper, syscallProbe, childArgs }), + ), + ); + } catch (error) { + process.stderr.write( + `${JSON.stringify({ + schema_version: 1, + code: error.code ?? "CP06_WORKER_FAILED", + exit_code: error.exitCode ?? 70, + message: safeError(error), + causes: + error instanceof AggregateError + ? error.errors.map((cause) => ({ + code: cause?.code ?? "CP06_WORKER_FAILED", + exit_code: cause?.exitCode ?? null, + })) + : [], + })}\n`, + ); + process.exitCode = error.exitCode ?? 70; + } +} + +function childCommand({ mode, source, target, helper, syscallProbe, childArgs, beforeHash }) { + if (mode === "unsafe-probe") { + return { + command: process.execPath, + args: [ + resolve(root, "scripts/cp06-isolation-adversary.mjs"), + "unsafe", + source, + target, + syscallProbe, + beforeHash, + String(process.pid), + ], + }; + } + const program = + mode === "hardened-probe" + ? [ + process.execPath, + resolve(root, "scripts/cp06-isolation-adversary.mjs"), + "hardened", + source, + target, + syscallProbe, + beforeHash, + String(process.pid), + ] + : [process.execPath, resolve(root, "scripts/cp06-sdk-worker.mjs"), target, ...childArgs]; + return { + command: "setpriv", + args: [ + "--no-new-privs", + "--bounding-set=-all", + "--inh-caps=-all", + "--ambient-caps=-all", + "--securebits=+noroot,+noroot_locked,+no_setuid_fixup,+no_setuid_fixup_locked", + "--", + helper, + ...program, + ], + }; +} + +function childEnvironment(childMode, credentialTarget) { + const environment = { + PATH: process.env.PATH ?? "/usr/bin:/bin", + LANG: process.env.LANG ?? "C.UTF-8", + LC_ALL: process.env.LC_ALL ?? "C.UTF-8", + PI_OFFLINE: "1", + PI_SKIP_VERSION_CHECK: "1", + PI_TELEMETRY: "0", + }; + if (childMode === "worker") { + environment.CP06_CREDENTIAL_TARGET = credentialTarget; + for (const name of [ + "SSL_CERT_FILE", + "SSL_CERT_DIR", + "NODE_EXTRA_CA_CERTS", + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + ]) { + if (process.env[name] !== undefined) environment[name] = process.env[name]; + } + } + return environment; +} + +function failure(message, exitCode, cause) { + return Object.assign(new Error(message, { cause }), { exitCode, code: "CP06_WORKER_FAILED" }); +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +function safeError(error) { + if (error instanceof AggregateError) + return `${error.message}: ${error.errors.map(safeError).join("; ")}`; + return error instanceof Error ? error.message : String(error); +} diff --git a/scripts/cp06-readonly-credentials.mjs b/scripts/cp06-readonly-credentials.mjs new file mode 100644 index 0000000..ad165d8 --- /dev/null +++ b/scripts/cp06-readonly-credentials.mjs @@ -0,0 +1,143 @@ +import { constants } from "node:fs"; +import { open } from "node:fs/promises"; + +const DEFAULT_MAX_BYTES = 1024 * 1024; + +export class Cp06AuthBlockedError extends Error { + constructor(message, options) { + super(message, options); + this.name = "Cp06AuthBlockedError"; + this.code = "CP06_AUTH_BLOCKED"; + } +} + +/** + * A deliberately tiny pi-ai CredentialStore for the CP-06 proof worker. + * It exposes one OAuth record and rejects every mutation before entering the + * callback that provider auth resolution would use to refresh the record. + */ +export class Cp06ReadOnlyCredentialStore { + #providerId; + #credential; + #audit = { reads: 0, lists: 0, modify_denials: 0, delete_denials: 0 }; + + static async load(options) { + const path = options?.path; + const providerId = options?.providerId; + const maxBytes = options?.maxBytes ?? DEFAULT_MAX_BYTES; + if (typeof path !== "string" || path.length === 0) { + throw blocked("credential target path is missing"); + } + if (typeof providerId !== "string" || providerId.length === 0) { + throw blocked("expected OAuth provider is missing"); + } + if (!Number.isSafeInteger(maxBytes) || maxBytes <= 0) { + throw blocked("credential size bound is invalid"); + } + + let handle; + try { + handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_CLOEXEC); + const before = await handle.stat({ bigint: true }); + if (!before.isFile()) throw blocked("credential target is not a regular file"); + if (before.size <= 0n || before.size > BigInt(maxBytes)) { + throw blocked("credential target size is outside the allowed bound"); + } + const bytes = await handle.readFile(); + const after = await handle.stat({ bigint: true }); + if ( + before.dev !== after.dev || + before.ino !== after.ino || + before.size !== after.size || + bytes.length !== Number(before.size) + ) { + throw blocked("credential target changed while it was read"); + } + let parsed; + try { + parsed = JSON.parse(bytes.toString("utf8")); + } catch (cause) { + throw blocked("credential target is not valid JSON", { cause }); + } + const credential = parsed?.[providerId]; + validateOAuthCredential(credential); + return new Cp06ReadOnlyCredentialStore(providerId, credential); + } catch (cause) { + if (cause instanceof Cp06AuthBlockedError) throw cause; + throw blocked("credential target could not be opened read-only", { cause }); + } finally { + await handle?.close(); + } + } + + constructor(providerId, credential) { + this.#providerId = providerId; + this.#credential = structuredClone(credential); + } + + assertUsableOAuth(options = {}) { + const now = options.now ?? Date.now(); + const minValidityMs = options.minValidityMs ?? 300_000; + if (!Number.isFinite(now) || !Number.isFinite(minValidityMs) || minValidityMs < 0) { + throw blocked("OAuth validity preflight parameters are invalid"); + } + const remainingValidityMs = this.#credential.expires - now; + if (remainingValidityMs < minValidityMs) { + throw blocked("OAuth credential is expired or too near expiry for a no-refresh run"); + } + return { + providerId: this.#providerId, + type: "oauth", + expires: this.#credential.expires, + remainingValidityMs, + }; + } + + async read(providerId, options) { + options?.signal?.throwIfAborted(); + this.#audit.reads += 1; + return providerId === this.#providerId ? structuredClone(this.#credential) : undefined; + } + + async list(options) { + options?.signal?.throwIfAborted(); + this.#audit.lists += 1; + return [{ providerId: this.#providerId, type: "oauth" }]; + } + + async modify(_providerId, _callback, options) { + options?.signal?.throwIfAborted(); + this.#audit.modify_denials += 1; + throw blocked("CP-06 read-only credential store denies refresh and mutation"); + } + + async delete(_providerId, options) { + options?.signal?.throwIfAborted(); + this.#audit.delete_denials += 1; + throw blocked("CP-06 read-only credential store denies credential deletion"); + } + + audit() { + return { ...this.#audit }; + } +} + +function validateOAuthCredential(credential) { + if ( + credential === null || + typeof credential !== "object" || + Array.isArray(credential) || + credential.type !== "oauth" || + typeof credential.access !== "string" || + credential.access.length === 0 || + typeof credential.refresh !== "string" || + credential.refresh.length === 0 || + !Number.isFinite(credential.expires) + ) { + throw blocked("expected provider does not contain one usable OAuth credential"); + } +} + +function blocked(message, options) { + return new Cp06AuthBlockedError(message, options); +} diff --git a/scripts/cp06-sdk-worker.mjs b/scripts/cp06-sdk-worker.mjs new file mode 100644 index 0000000..29c5e33 --- /dev/null +++ b/scripts/cp06-sdk-worker.mjs @@ -0,0 +1,278 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { accessSync, constants, readFileSync, realpathSync } from "node:fs"; +import { readFile } from "node:fs/promises"; +import { delimiter, dirname, join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { Cp06AuthBlockedError, Cp06ReadOnlyCredentialStore } from "./cp06-readonly-credentials.mjs"; +import { auditReadEvents } from "./cp06-worker-audit.mjs"; +import { auditWorkerOutcome } from "./cp06-worker-outcome.mjs"; + +const [credentialTargetArg, inputPathArg] = process.argv.slice(2); +if (!credentialTargetArg || !inputPathArg) process.exit(64); + +try { + const credentialTarget = resolve(credentialTargetArg); + const input = JSON.parse(await readFile(resolve(inputPathArg), "utf8")); + validateInput(input); + const pi = resolvePiPackage(); + const sdk = await import(pathToFileURL(join(pi.root, "dist/index.js"))); + assertSdkSurface(sdk); + + const contract = await readFile(input.contract_path, "utf8"); + const pack = await readFile(input.pack_path, "utf8"); + const original = await readFile(input.original_path); + const prompt = buildPrompt({ contract, pack, originalPath: input.original_path }); + const store = await Cp06ReadOnlyCredentialStore.load({ + path: credentialTarget, + providerId: input.provider, + }); + const minOAuthValidityMs = input.timeout_ms + 300_000; + const validity = store.assertUsableOAuth({ minValidityMs: minOAuthValidityMs }); + + const runtime = await sdk.ModelRuntime.create({ + credentials: store, + modelsPath: null, + refreshOnCreate: false, + allowModelNetwork: false, + }); + const model = runtime.getModel(input.provider, input.model); + if (model === undefined) throw blocked("configured Pi model is not built in to Pi 0.85.1"); + const auth = await runtime.getAuth(model, { minOAuthValidityMs }); + if (auth === undefined) throw blocked("expected OAuth provider did not resolve authentication"); + + const resourceLoader = emptyResourceLoader(sdk, input.system_prompt); + const settingsManager = sdk.SettingsManager.inMemory({ + compaction: { enabled: false }, + retry: { + enabled: false, + maxRetries: 0, + provider: { timeoutMs: input.timeout_ms, maxRetries: 0, maxRetryDelayMs: 0 }, + }, + defaultProjectTrust: "never", + enableAnalytics: false, + enableInstallTelemetry: false, + }); + const { session, extensionsResult } = await sdk.createAgentSession({ + cwd: input.root, + agentDir: dirname(credentialTarget), + model, + modelRuntime: runtime, + resourceLoader, + tools: ["read"], + sessionManager: sdk.SessionManager.inMemory(input.root), + settingsManager, + thinkingLevel: "high", + }); + if ((extensionsResult.extensions?.length ?? 0) !== 0 || session.sessionFile !== undefined) { + throw blocked("SDK worker unexpectedly loaded an extension or persistent session"); + } + if (JSON.stringify(session.getActiveToolNames()) !== JSON.stringify(["read"])) { + throw blocked("SDK worker active tool set is not exactly read"); + } + + const events = []; + const unsubscribe = session.subscribe((event) => events.push(structuredClone(event))); + let timedOut = false; + let timer; + try { + await Promise.race([ + session.prompt(prompt, { expandPromptTemplates: false, source: "rpc" }), + new Promise((_, reject) => { + timer = setTimeout(() => { + timedOut = true; + void session.abort(); + reject(new Error("CP06_WORKER_TIMEOUT")); + }, input.timeout_ms); + }), + ]); + } finally { + clearTimeout(timer); + if (timedOut || session.isStreaming) await session.abort(); + unsubscribe(); + session.dispose(); + } + + const responseText = finalAssistantText(events); + const response = parseResponse(responseText); + const readAudit = auditReadEvents(events, { + root: input.root, + expectedOriginalPath: input.original_path, + expectedOriginal: original, + decisiveText: ["request-supplied owner fields are ignored"], + }); + if (!readAudit.ok) { + throw blocked(`SDK event audit failed: ${readAudit.reason}`); + } + const outcome = auditWorkerOutcome(response, { + root: input.root, + referenceRoot: input.fixture_root, + expectedOriginalPath: input.original_path, + }); + if (!outcome.ok) throw blocked(`SDK worker outcome failed: ${outcome.reason}`); + + console.log( + JSON.stringify({ + schema_version: 1, + result: "pass", + pi_version: pi.version, + provider: input.provider, + model: input.model, + oauth_preflight: { + minimum_validity_ms: minOAuthValidityMs, + remaining_validity_ms: validity.remainingValidityMs, + refreshed: false, + }, + credential_store: store.audit(), + session: { in_memory: true, active_tools: session.getActiveToolNames() }, + event_stream: { + sha256: sha256(JSON.stringify(events)), + event_count: events.length, + read_audit: readAudit.summary, + }, + response, + response_sha256: sha256(responseText), + raw_transcript_retained: false, + }), + ); +} catch (error) { + if (error instanceof Cp06AuthBlockedError || error?.code === "CP06_AUTH_BLOCKED") { + process.stderr.write(`CP06_AUTH_BLOCKED: ${error.message}\n`); + process.exit(75); + } + process.stderr.write(`CP06_WORKER_FAILED: ${safeMessage(error)}\n`); + process.exit(70); +} + +function validateInput(input) { + if (input?.schema_version !== 1) throw blocked("worker input schema is unsupported"); + for (const key of ["root", "fixture_root", "contract_path", "pack_path", "original_path"]) + if (typeof input[key] !== "string" || !input[key].startsWith("/")) { + throw blocked(`worker input ${key} must be absolute`); + } + if (input.provider !== "openai-codex") throw blocked("worker provider must be openai-codex"); + if (typeof input.model !== "string" || input.model.length === 0) { + throw blocked("worker model is missing"); + } + if (!Number.isSafeInteger(input.timeout_ms) || input.timeout_ms < 1_000) { + throw blocked("worker timeout is invalid"); + } + if (typeof input.system_prompt !== "string" || input.system_prompt.length === 0) { + throw blocked("worker system prompt is missing"); + } +} + +function resolvePiPackage() { + const piExecutable = findExecutable("pi"); + const cli = realpathSync(piExecutable); + const packageRoot = findPiPackageRoot(cli); + const metadata = JSON.parse(readFileSync(join(packageRoot, "package.json"), "utf8")); + const version = spawnSync(piExecutable, ["--version"], { encoding: "utf8", timeout: 10_000 }); + if ( + metadata.name !== "@earendil-works/pi-coding-agent" || + metadata.version !== "0.85.1" || + version.status !== 0 || + version.stdout.trim() !== "0.85.1" + ) { + throw blocked("exact Pi 0.85.1 is required"); + } + return { root: packageRoot, version: metadata.version }; +} + +function findPiPackageRoot(executable) { + let directory = dirname(executable); + for (let depth = 0; depth < 5; depth += 1) { + try { + const metadata = JSON.parse(readFileSync(join(directory, "package.json"), "utf8")); + if (metadata.name === "@earendil-works/pi-coding-agent") return directory; + } catch {} + const parent = dirname(directory); + if (parent === directory) break; + directory = parent; + } + throw blocked("Pi package root could not be resolved from its executable"); +} + +function findExecutable(name) { + for (const directory of String(process.env.PATH ?? "").split(delimiter)) { + if (!directory) continue; + const candidate = join(directory, name); + try { + accessSync(candidate, constants.X_OK); + return candidate; + } catch {} + } + throw blocked("Pi executable is unavailable"); +} + +function assertSdkSurface(sdk) { + for (const name of [ + "ModelRuntime", + "createAgentSession", + "createExtensionRuntime", + "SessionManager", + "SettingsManager", + ]) { + if (sdk[name] === undefined) throw blocked(`Pi SDK export is unavailable: ${name}`); + } +} + +function emptyResourceLoader(sdk, systemPrompt) { + return { + getExtensions: () => ({ extensions: [], errors: [], runtime: sdk.createExtensionRuntime() }), + getSkills: () => ({ skills: [], diagnostics: [] }), + getPrompts: () => ({ prompts: [], diagnostics: [] }), + getThemes: () => ({ themes: [], diagnostics: [] }), + getAgentsFiles: () => ({ agentsFiles: [] }), + getSystemPrompt: () => systemPrompt, + getSystemPromptSource: () => undefined, + getAppendSystemPrompt: () => [], + getAppendSystemPromptSources: () => [], + extendResources: () => {}, + reload: async () => {}, + }; +} + +function buildPrompt({ contract, pack, originalPath }) { + return `You are a fresh task reviewer. You have no previous worker transcript. Review only the bounded handoff below. First inspect the task and pack. One concrete creation-time ownership rule was deliberately omitted from this handoff. Report MISSING_SOURCE, set missing_source to the exact canonical relative path docs/AUTH.md, and use the read tool exactly once, without offset or limit, on the supplied exact-original path; do not scan or read any other project file. Then return a concise JSON object with keys status, missing_source, reads, corrected_constraints, evidence_gaps, and broad_scan. The evidence_gaps field must be exactly [{"id":"creation-time-ownership","status":"unverified","missing_checks":["principal-derived-owner","request-owner-ignored"]}]: the supplied deterministic evidence does not test those creation-time behaviors. Recovering a requirement by reading is not evidence that the behavior passed. Do not add a passed/verified disposition or prose fields to this structured gap. Do not modify files.\n\nEXACT TASK CONTRACT\n---\n${contract}\n---\n\nBOUNDED PACK\n---\n${pack}\n---\n\nEXISTING DETERMINISTIC EVIDENCE\n---\nThe in-memory auth fixture currently reports absent/forged credentials as 401 and concealed cross-account GET/PUT/DELETE as 404 with unchanged state. Review whether the handoff states every decisive acceptance constraint; do not assume unshown creation semantics.\n---\n\nSUPPLIED EXACT-ORIGINAL PATH (read only if needed)\n${originalPath}\n`; +} + +function finalAssistantText(events) { + const messages = events + .filter((event) => event.type === "message_end" && event.message?.role === "assistant") + .map((event) => event.message); + const assistant = messages.at(-1); + if (assistant === undefined) + throw blocked("SDK event stream omitted the final assistant message"); + if (typeof assistant.content === "string") return assistant.content; + return Array.isArray(assistant.content) + ? assistant.content + .filter((block) => block?.type === "text" && typeof block.text === "string") + .map((block) => block.text) + .join("") + : ""; +} + +function parseResponse(output) { + try { + return JSON.parse(output.trim()); + } catch { + const fenced = output.trim().match(/```(?:json)?\s*([\s\S]*?)```/); + if (fenced === null) throw blocked("SDK worker did not return the required JSON response"); + return JSON.parse(fenced[1]); + } +} + +function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} + +function blocked(message) { + return new Cp06AuthBlockedError(message); +} + +function safeMessage(error) { + if (error?.message === "CP06_WORKER_TIMEOUT") return "worker timed out and was aborted"; + return error instanceof Error ? error.message : String(error); +} diff --git a/scripts/cp06-seccomp-exec.c b/scripts/cp06-seccomp-exec.c new file mode 100644 index 0000000..5ead15a --- /dev/null +++ b/scripts/cp06-seccomp-exec.c @@ -0,0 +1,123 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(__linux__) || !defined(__x86_64__) +#error "CP-06 isolation helper currently supports Linux x86-64 only" +#endif + +typedef void *scmp_filter_ctx; +struct scmp_arg_cmp { + unsigned int arg; + int op; + uint64_t datum_a; + uint64_t datum_b; +}; +extern scmp_filter_ctx seccomp_init(uint32_t def_action); +extern int seccomp_rule_add(scmp_filter_ctx, uint32_t, int, unsigned int, ...); +extern int seccomp_rule_add_array(scmp_filter_ctx, uint32_t, int, unsigned int, + const struct scmp_arg_cmp *); +extern int seccomp_load(scmp_filter_ctx); +extern void seccomp_release(scmp_filter_ctx); +extern int seccomp_syscall_resolve_name(const char *); + +#define SCMP_ACT_ALLOW 0x7fff0000U +#define SCMP_ACT_ERRNO(x) (0x00050000U | ((x) & 0x0000ffffU)) +#define SCMP_CMP_MASKED_EQ 7 + +static void fail(const char *message) { + perror(message); + _exit(70); +} + +static void deny(scmp_filter_ctx ctx, const char *name, int error) { + int nr = seccomp_syscall_resolve_name(name); + if (nr >= 0 && seccomp_rule_add(ctx, SCMP_ACT_ERRNO(error), nr, 0) < 0) { + fprintf(stderr, "cannot install seccomp rule for %s\n", name); + _exit(70); + } +} + +static void close_inherited_fds(void) { +#ifdef SYS_close_range + if (syscall(SYS_close_range, 3U, ~0U, 0U) == 0) return; + if (errno != ENOSYS) fail("close_range"); +#endif + long limit = sysconf(_SC_OPEN_MAX); + if (limit < 0 || limit > 1048576) limit = 1048576; + for (int fd = 3; fd < limit; ++fd) close(fd); +} + +int main(int argc, char **argv) { + if (argc < 2) { + fprintf(stderr, "usage: cp06-seccomp-exec COMMAND [ARG...]\n"); + return 64; + } + + pid_t parent = getppid(); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) != 0) fail("PR_SET_PDEATHSIG"); + if (getppid() != parent || parent == 1) { + fprintf(stderr, "CP-06 supervisor disappeared before isolation\n"); + return 70; + } + close_inherited_fds(); + if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) != 0) fail("PR_SET_NO_NEW_PRIVS"); + + scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW); + if (ctx == NULL) { + fprintf(stderr, "seccomp_init failed\n"); + return 70; + } + const char *blocked[] = { + "mount", "umount2", "pivot_root", + "move_mount", "open_tree", "fsopen", + "fsconfig", "fsmount", "fspick", + "mount_setattr", "unshare", "setns", + "open_by_handle_at", "name_to_handle_at", "ptrace", + "process_vm_readv", "process_vm_writev", "pidfd_getfd", + "bpf", "perf_event_open", "personality", + "kexec_load", "kexec_file_load", + }; + for (size_t i = 0; i < sizeof(blocked) / sizeof(blocked[0]); ++i) { + deny(ctx, blocked[i], EPERM); + } + + /* Node/libc can fall back to clone(2), where namespace bits are filtered. */ + deny(ctx, "clone3", ENOSYS); + int clone_nr = seccomp_syscall_resolve_name("clone"); + const uint64_t namespace_flags[] = { + CLONE_NEWNS, CLONE_NEWCGROUP, CLONE_NEWUTS, CLONE_NEWIPC, + CLONE_NEWUSER, CLONE_NEWPID, CLONE_NEWNET, + }; + for (size_t i = 0; + clone_nr >= 0 && i < sizeof(namespace_flags) / sizeof(namespace_flags[0]); + ++i) { + struct scmp_arg_cmp comparison = { + .arg = 0, + .op = SCMP_CMP_MASKED_EQ, + .datum_a = namespace_flags[i], + .datum_b = namespace_flags[i], + }; + if (seccomp_rule_add_array(ctx, SCMP_ACT_ERRNO(EPERM), clone_nr, 1, + &comparison) < 0) { + fprintf(stderr, "cannot install clone namespace filter\n"); + seccomp_release(ctx); + return 70; + } + } + + if (seccomp_load(ctx) < 0) { + seccomp_release(ctx); + fail("seccomp_load"); + } + seccomp_release(ctx); + execvp(argv[1], &argv[1]); + fail("execvp"); +} diff --git a/scripts/cp06-worker-audit.mjs b/scripts/cp06-worker-audit.mjs new file mode 100644 index 0000000..366b19c --- /dev/null +++ b/scripts/cp06-worker-audit.mjs @@ -0,0 +1,133 @@ +import { createHash } from "node:crypto"; +import { resolve } from "node:path"; + +export function isExpectedOriginalReference(value, options) { + return value === (options.expectedOriginalReference ?? "docs/AUTH.md"); +} + +export function auditReadEvents(events, options) { + const expectedPath = resolve(options.root, options.expectedOriginalPath); + const expectedBytes = Buffer.isBuffer(options.expectedOriginal) + ? options.expectedOriginal + : Buffer.from(options.expectedOriginal, "utf8"); + const expectedText = expectedBytes.toString("utf8"); + const decisiveText = options.decisiveText ?? []; + const starts = events + .map((event, index) => ({ event, index })) + .filter(({ event }) => event?.type === "tool_execution_start"); + const ends = events + .map((event, index) => ({ event, index })) + .filter(({ event }) => event?.type === "tool_execution_end"); + const readStarts = starts.filter(({ event }) => event.toolName === "read"); + const readEnds = ends.filter(({ event }) => event.toolName === "read"); + const otherToolCalls = starts.filter(({ event }) => event.toolName !== "read"); + const otherToolCompletions = ends.filter(({ event }) => event.toolName !== "read"); + const resolvedReads = readStarts.map(({ event, index }) => ({ + event, + index, + path: event.args?.path, + resolvedPath: + typeof event.args?.path === "string" ? resolve(options.root, event.args.path) : null, + })); + const originalReads = resolvedReads.filter((entry) => entry.resolvedPath === expectedPath); + const rejectedReads = resolvedReads.filter((entry) => entry.resolvedPath !== expectedPath); + const summary = { + read_count: readStarts.length, + project_read_count: originalReads.length, + project_read_paths: originalReads.map((entry) => entry.resolvedPath), + rejected_read_paths: rejectedReads.map((entry) => entry.path), + other_tool_calls: otherToolCalls.map(({ event }) => event.toolName), + other_tool_completions: otherToolCompletions.map(({ event }) => event.toolName), + completed_successfully: false, + exact_original: false, + expected_sha256: sha256(expectedBytes), + returned_sha256: null, + returned_bytes: null, + tool_call_id: null, + start_event_index: null, + end_event_index: null, + response_event_index: null, + }; + const reject = (reason) => ({ ok: false, reason, summary }); + + if (otherToolCalls.length > 0 || otherToolCompletions.length > 0) { + return reject( + `unexpected tool events ${[ + ...summary.other_tool_calls, + ...summary.other_tool_completions, + ].join(", ")}`, + ); + } + if (readStarts.length !== 1 || originalReads.length !== 1 || rejectedReads.length > 0) { + return reject( + `expected exactly one read of the supplied original, saw ${originalReads.length} original and ${rejectedReads.length} other reads`, + ); + } + + const start = originalReads[0]; + const callId = start.event.toolCallId; + summary.tool_call_id = typeof callId === "string" ? callId : null; + summary.start_event_index = start.index; + if (typeof callId !== "string" || callId.length === 0) { + return reject("original read start omitted toolCallId"); + } + if (start.event.args?.offset !== undefined) { + return reject("original read used an explicit offset"); + } + if (start.event.args?.limit !== undefined) { + return reject("original read used an explicit limit"); + } + + const matchingEnds = readEnds.filter(({ event }) => event.toolCallId === callId); + if (matchingEnds.length !== 1) { + return reject( + `expected one matching read completion for ${callId}, saw ${matchingEnds.length}`, + ); + } + if (readEnds.length !== 1) { + return reject("read event stream contains an unmatched or duplicate completion"); + } + const end = matchingEnds[0]; + summary.end_event_index = end.index; + if (end.index <= start.index) { + return reject("read completion did not occur after its matching start"); + } + if (end.event.isError !== false) { + return reject("matching read completion was not explicitly successful"); + } + const content = end.event.result?.content; + if ( + !Array.isArray(content) || + content.length !== 1 || + content[0]?.type !== "text" || + typeof content[0].text !== "string" + ) { + return reject("successful read completion omitted one authoritative text result"); + } + const returnedBytes = Buffer.from(content[0].text, "utf8"); + summary.returned_sha256 = sha256(returnedBytes); + summary.returned_bytes = returnedBytes.length; + if (!returnedBytes.equals(expectedBytes)) { + return reject("read result bytes do not equal the supplied exact original"); + } + if (decisiveText.some((text) => !expectedText.includes(text))) { + return reject("the supplied original does not contain every declared decisive constraint"); + } + + const finalResponse = events + .map((event, index) => ({ event, index })) + .filter(({ event }) => event?.type === "message_end" && event.message?.role === "assistant") + .at(-1); + summary.response_event_index = finalResponse?.index ?? null; + if (finalResponse === undefined || finalResponse.index <= end.index) { + return reject("final assistant response did not occur after the successful original read"); + } + + summary.completed_successfully = true; + summary.exact_original = true; + return { ok: true, summary }; +} + +function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} diff --git a/scripts/cp06-worker-lifecycle.mjs b/scripts/cp06-worker-lifecycle.mjs new file mode 100644 index 0000000..db2cf8e --- /dev/null +++ b/scripts/cp06-worker-lifecycle.mjs @@ -0,0 +1,34 @@ +export class Cp06CleanupError extends AggregateError { + constructor(primary, cleanup) { + super( + primary === undefined ? [cleanup] : [primary, cleanup], + "CP-06 evaluation cleanup failed", + { + cause: primary ?? cleanup, + }, + ); + this.code = "CP06_CLEANUP_FAILED"; + this.exitCode = 74; + } +} + +/** Await cleanup on success and failure; never lose the primary error. */ +export async function withCleanup(action, cleanup) { + let result; + let primary; + let failed = false; + try { + result = await action(); + } catch (error) { + failed = true; + primary = error; + } finally { + try { + await cleanup(); + } catch (error) { + throw new Cp06CleanupError(primary, error); + } + } + if (failed) throw primary; + return result; +} diff --git a/scripts/cp06-worker-outcome.mjs b/scripts/cp06-worker-outcome.mjs new file mode 100644 index 0000000..d9a05fd --- /dev/null +++ b/scripts/cp06-worker-outcome.mjs @@ -0,0 +1,37 @@ +import { isExpectedOriginalReference } from "./cp06-worker-audit.mjs"; + +export function auditWorkerOutcome(response, options) { + const corrected = Array.isArray(response?.corrected_constraints) + ? response.corrected_constraints.join(" ") + : ""; + if ( + response?.status !== "MISSING_SOURCE" || + !isExpectedOriginalReference(response.missing_source, options) || + response.broad_scan !== false || + !/ownership comes only from the authenticated principal/i.test(corrected) || + !/request-supplied owner fields? (?:are )?ignored/i.test(corrected) + ) { + return { ok: false, reason: "response did not satisfy the targeted-read oracle" }; + } + const gaps = response.evidence_gaps; + const gap = Array.isArray(gaps) && gaps.length === 1 ? gaps[0] : undefined; + const checks = gap?.missing_checks; + if ( + gap === null || + typeof gap !== "object" || + Array.isArray(gap) || + Object.keys(gap).sort().join(",") !== "id,missing_checks,status" || + gap.id !== "creation-time-ownership" || + gap.status !== "unverified" || + !Array.isArray(checks) || + checks.length !== 2 || + !checks.includes("principal-derived-owner") || + !checks.includes("request-owner-ignored") + ) { + return { + ok: false, + reason: "creation-time ownership evidence must remain explicitly unverified", + }; + } + return { ok: true }; +} diff --git a/scripts/prove-cp06-auth-security.mjs b/scripts/prove-cp06-auth-security.mjs new file mode 100644 index 0000000..2c50956 --- /dev/null +++ b/scripts/prove-cp06-auth-security.mjs @@ -0,0 +1,32 @@ +#!/usr/bin/env node +import { writeFile } from "node:fs/promises"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { Cp06IsolationUnsupportedError, runCp06SecurityPreflight } from "./cp06-auth-security.mjs"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const outputRoot = resolve(root, process.env.CP06_OUTPUT ?? ".factory/state/cp06-correction"); +try { + const proof = await runCp06SecurityPreflight({ + root, + outputRoot: resolve(outputRoot, "auth-security"), + reviewer: process.env.CP06_REVIEWER, + }); + const path = resolve(outputRoot, "auth-security", "evidence.json"); + await writeFile(path, `${JSON.stringify(proof, null, 2)}\n`); + console.log( + JSON.stringify({ + ok: true, + gate: "CP-06-read-only-auth-security", + evidence_path: path.startsWith(`${root}/`) ? path.slice(root.length + 1) : path, + result: proof.result, + }), + ); +} catch (error) { + if (error instanceof Cp06IsolationUnsupportedError) { + process.stderr.write(`CP06_ISOLATION_UNSUPPORTED: ${error.message}\n`); + process.exit(73); + } + process.stderr.write(`CP06_SECURITY_PROOF_FAILED: ${error.message}\n`); + process.exit(1); +} diff --git a/scripts/prove-cp06-worker-cleanup.mjs b/scripts/prove-cp06-worker-cleanup.mjs new file mode 100644 index 0000000..44b08af --- /dev/null +++ b/scripts/prove-cp06-worker-cleanup.mjs @@ -0,0 +1,139 @@ +#!/usr/bin/env node +import { strict as assert } from "node:assert"; +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { withCleanup } from "./cp06-worker-lifecycle.mjs"; + +const root = resolve(fileURLToPath(new URL("..", import.meta.url))); +const output = resolve( + root, + process.env.CP06_OUTPUT ?? ".factory/state/cp06-correction", + "worker-cleanup", +); +await mkdir(output, { recursive: true }); +const scratch = await mkdtemp(join(output, "DUMMY-cleanup-")); +const cases = await withCleanup( + async () => { + const cases = []; + for (const [scenario, expectedExit] of [ + ["expired", 75], + ["near-expiry", 75], + ["missing-source-setup", 70], + ]) { + const home = join(scratch, scenario); + const temporary = join(home, "tmp"); + const proofOutput = join(home, "proofs"); + const source = join(home, "DUMMY-auth.json"); + await mkdir(temporary, { recursive: true }); + const sourceBytes = JSON.stringify({ + "openai-codex": { + type: "oauth", + access: "DUMMY-ACCESS", + refresh: "DUMMY-REFRESH", + expires: scenario === "expired" ? 0 : Date.now() + 60_000, + }, + }); + if (scenario !== "missing-source-setup") await writeFile(source, sourceBytes); + await mkdir(join(proofOutput, "ten-pack/raw"), { recursive: true }); + await mkdir(join(proofOutput, "ten-pack/repo/docs"), { recursive: true }); + await writeFile( + join(proofOutput, "ten-pack/raw/auth-worker-missing.pack.md"), + "DUMMY cleanup-only handoff; no semantic acceptance claimed.\n", + ); + await writeFile( + join(proofOutput, "ten-pack/repo/docs/AUTH.md"), + await readFile(join(root, "test/fixtures/cp06-context/docs/AUTH.md")), + ); + const args = [join(root, "scripts/replay-cp06-worker.mjs")]; + const result = spawnSync(process.execPath, args, { + cwd: root, + encoding: "utf8", + timeout: 180_000, + maxBuffer: 1024 * 1024, + env: { + PATH: process.env.PATH, + HOME: home, + TMPDIR: temporary, + PI_OFFLINE: "1", + PI_SKIP_VERSION_CHECK: "1", + CP06_PI_AUTH_FILE: source, + CP06_OUTPUT: proofOutput, + CP06_REVIEWER: "DUMMY cleanup regression", + }, + }); + assert.equal(result.error, undefined); + assert.equal(result.status, expectedExit, result.stderr); + const blocked = JSON.parse(await readFile(join(proofOutput, "worker/blocked.json"), "utf8")); + assert.equal(blocked.result, "blocked"); + assert.equal(blocked.stage, "isolated-sdk-worker"); + assert.deepEqual(await readdir(temporary), [], `${scenario}: evaluation home leaked`); + if (scenario !== "missing-source-setup") + assert.equal(await readFile(source, "utf8"), sourceBytes); + await assert.rejects(readFile(join(proofOutput, "worker/evidence.json")), { code: "ENOENT" }); + cases.push({ + scenario, + command: [process.execPath, ...args], + exit_code: result.status, + expected_exit_code: expectedExit, + stage: blocked.stage, + worker_failure: blocked.worker_failure, + evaluation_home_removed: true, + source_unchanged: scenario === "missing-source-setup" ? null : true, + fixture_origin: true, + semantic_acceptance: false, + }); + } + return cases; + }, + () => rm(scratch, { recursive: true, force: true }), +); +const git = spawnSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }); +assert.equal(git.status, 0); +const sources = [ + "scripts/replay-cp06-worker.mjs", + "scripts/cp06-namespace-supervisor.mjs", + "scripts/cp06-worker-lifecycle.mjs", + "scripts/prove-cp06-worker-cleanup.mjs", +]; +const manifest = { + schema_version: 1, + gate: "CP06-DUMMY-worker-cleanup", + result: "pass", + tested_sha: git.stdout.trim(), + recorded_at: new Date().toISOString(), + reviewer: process.env.CP06_REVIEWER ?? "CP06 correction worker", + command: "npm run proof:cp06:worker-cleanup", + exit_code: 0, + fixture_origin: true, + semantic_acceptance: false, + cases, + source_sha256: Object.fromEntries( + await Promise.all( + sources.map(async (path) => [ + path, + createHash("sha256") + .update(await readFile(join(root, path))) + .digest("hex"), + ]), + ), + ), + limitations: [ + "Actual full-runner expired/near-expiry and missing-source setup cases use DUMMY auth only. Launch, timeout, audit and compound cleanup failures are additionally injected at the executable supervisor interface in the unit suite; SDK timeout is exercised by the DUMMY auth-security proof.", + ], +}; +await writeFile(join(output, "evidence.json"), `${JSON.stringify(manifest, null, 2)}\n`); +console.log( + JSON.stringify({ + result: "pass", + gate: manifest.gate, + tested_sha: manifest.tested_sha, + cases: cases.map(({ scenario, exit_code, evaluation_home_removed }) => ({ + scenario, + exit_code, + evaluation_home_removed, + })), + }), +); diff --git a/scripts/replay-cp06-p07.mjs b/scripts/replay-cp06-p07.mjs new file mode 100644 index 0000000..83ce136 --- /dev/null +++ b/scripts/replay-cp06-p07.mjs @@ -0,0 +1,135 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { writeFileSync } from "node:fs"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { spawnSync } from "node:child_process"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const outputRoot = resolve(root, process.env.CP06_OUTPUT ?? ".factory/state/cp06-correction"); +const rawRoot = join(outputRoot, "p07", "raw"); +await rm(join(outputRoot, "p07"), { recursive: true, force: true }); +await mkdir(rawRoot, { recursive: true }); + +const steps = []; +steps.push(run("build", "npm", ["run", "build"], 0)); +steps.push( + run( + "fail-before", + process.execPath, + ["test/fixtures/workflow/history/malformed-json-probe.mjs"], + 1, + ["SyntaxError", "not valid JSON"], + ), +); +steps.push( + run( + "fix-after", + process.execPath, + [ + "--test", + "--test-name-pattern=normal API fixture rejects malformed", + "dist/test/workflow.test.js", + ], + 0, + ), +); +steps.push( + run( + "critical-auth-retest", + process.execPath, + ["--test", "--test-name-pattern=critical auth fixture", "dist/test/workflow.test.js"], + 0, + ), +); + +const findingPath = "test/fixtures/workflow/history/finding.json"; +const finding = JSON.parse(await readFile(join(root, findingPath), "utf8")); +const testedSha = runText("git", ["rev-parse", "HEAD"]); +const manifest = { + schema_version: 1, + gate: "P-07", + tested_sha: testedSha, + recorded_at: new Date().toISOString(), + reviewer: process.env.CP06_REVIEWER ?? "Pi CP-06 correction worker", + finding: { + path: findingPath, + sha256: await fileHash(join(root, findingPath)), + record: finding, + }, + sequence: steps, + result: "pass", + limitations: [ + "Behavioral fixtures are in-memory and make no network or real-account access.", + "The historical module is a minimal retained reproduction, not a raw worker transcript.", + ], +}; +const manifestPath = join(outputRoot, "p07", "evidence.json"); +await writeFile(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`); +console.log( + JSON.stringify({ + ok: true, + gate: "P-07", + tested_sha: testedSha, + manifest_path: relative(manifestPath), + manifest_sha256: await fileHash(manifestPath), + sequence: steps.map(({ id, command, exit_code, output_path, output_sha256 }) => ({ + id, + command, + exit_code, + output_path, + output_sha256, + })), + }), +); + +function run(id, command, args, expectedExit, expectedOutputFragments = []) { + const result = spawnSync(command, args, { cwd: root, encoding: "utf8", env: process.env }); + const exitCode = result.status; + const output = `command: ${quote([command, ...args])}\nexit: ${exitCode}\n--- stdout ---\n${result.stdout ?? ""}\n--- stderr ---\n${result.stderr ?? ""}`; + const outputPath = join(rawRoot, `${id}.txt`); + writeFileSync(outputPath, output); + const missingFragments = expectedOutputFragments.filter((fragment) => !output.includes(fragment)); + if (result.error !== undefined || exitCode !== expectedExit || missingFragments.length > 0) { + process.stderr.write(output); + const suffix = + missingFragments.length > 0 + ? `; missing output fragments ${missingFragments.join(", ")}` + : ""; + throw new Error(`${id} exited ${exitCode}; expected ${expectedExit}${suffix}`); + } + return { + id, + command: quote([command, ...args]), + expected_exit_code: expectedExit, + exit_code: exitCode, + output_path: relative(outputPath), + output_sha256: sha256(output), + expected_output_fragments: expectedOutputFragments, + }; +} + +function runText(command, args) { + const result = spawnSync(command, args, { cwd: root, encoding: "utf8" }); + if (result.status !== 0) throw new Error(`${command} failed: ${result.stderr}`); + return result.stdout.trim(); +} + +function quote(parts) { + return parts.map((part) => (/[\s'"]/.test(part) ? JSON.stringify(part) : part)).join(" "); +} + +function relative(path) { + return path.startsWith(`${root}/`) ? path.slice(root.length + 1) : path; +} + +function sha256(text) { + return createHash("sha256").update(text).digest("hex"); +} + +async function fileHash(path) { + return createHash("sha256") + .update(await readFile(path)) + .digest("hex"); +} diff --git a/scripts/replay-cp06-ten-pack.mjs b/scripts/replay-cp06-ten-pack.mjs new file mode 100644 index 0000000..1353a7c --- /dev/null +++ b/scripts/replay-cp06-ten-pack.mjs @@ -0,0 +1,707 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { cp, mkdir, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { spawnSync } from "node:child_process"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const fixtureSource = join(root, "test/fixtures/cp06-context"); +const outputRoot = resolve(root, process.env.CP06_OUTPUT ?? ".factory/state/cp06-correction"); +const proofRoot = join(outputRoot, "ten-pack"); +const fixtureRoot = join(proofRoot, "repo"); +const rawRoot = join(proofRoot, "raw"); +const cli = join(root, "dist/src/cli.js"); +const ctx = resolveTool(process.env.CTX_BIN ?? "ctx"); +configureOfflineCtxModelDir(); +const reviewer = process.env.CP06_REVIEWER ?? "Pi CP-06 correction worker"; +const commands = []; + +await rm(proofRoot, { recursive: true, force: true }); +await mkdir(rawRoot, { recursive: true }); +await cp(fixtureSource, fixtureRoot, { recursive: true }); +await mkdir(join(fixtureRoot, ".factory/state"), { recursive: true }); +const oracle = JSON.parse(await readFile(join(fixtureRoot, "oracle.json"), "utf8")); +const fixedAuth = await readFile(join(fixtureRoot, ".factory/tasks/AUTH-001.yaml"), "utf8"); +const beforeAuth = await readFile(join(fixtureRoot, "cases/AUTH-before.yaml"), "utf8"); +const workerBeforeAuth = await readFile(join(fixtureRoot, "cases/AUTH-worker-before.yaml"), "utf8"); +await writeFile(join(fixtureRoot, ".factory/tasks/AUTH-001.yaml"), beforeAuth); + +run("git-init", "git", ["init", "-q", fixtureRoot], { cwd: root }); +run("git-author-name", "git", ["config", "user.name", "Factory CP-06 fixture"], { + cwd: fixtureRoot, +}); +run("git-author-email", "git", ["config", "user.email", "factory-cp06@example.invalid"], { + cwd: fixtureRoot, +}); +run("git-add", "git", ["add", "."], { cwd: fixtureRoot }); +run("git-commit", "git", ["commit", "-qm", "seed CP-06 ten-pack fixture"], { + cwd: fixtureRoot, +}); +run("ctx-init", ctx, ["init", fixtureRoot, "--json"], { cwd: fixtureRoot }); + +const globalSources = ["PROJECT.md", "ARCHITECTURE.md"]; +const topicalSources = oracle.tasks.map((entry) => entry.target); +for (const path of [ + ...globalSources, + ...topicalSources.filter((path) => path !== "docs/AUTH.md"), +]) { + run( + `ctx-add-${slug(path)}`, + ctx, + ["add", path, "--root", fixtureRoot, "--authority", "normative", "--priority", "100", "--json"], + { + cwd: fixtureRoot, + }, + ); +} +run("ctx-index-without-auth", ctx, ["index", "--root", fixtureRoot, "--json"], { + cwd: fixtureRoot, + timeout: 120_000, +}); +const initialStatus = jsonRun( + "ctx-status-without-auth", + ctx, + ["status", "--root", fixtureRoot, "--json"], + { + cwd: fixtureRoot, + }, +); +assertSemantic(initialStatus, "initial status"); + +const authQuery = [ + "Bind item ownership to authenticated principal", + "Creation ignores request-supplied ownership and cross-account operations remain concealed.", + "A-AUTH: Missing or forged credentials return 401 and cross-account read update and delete return non-mutating 404.", + "principal ownership tenancy forged credentials", +].join("\n"); +const broad = run( + "bad-unscoped-auth", + ctx, + ["pack", authQuery, "--root", fixtureRoot, "--token-budget", "3500", "--json"], + { + cwd: fixtureRoot, + expected: [0], + timeout: 120_000, + }, +); +const broadJson = parseJsonIfPresent(broad.stdout); +if (broadJson === null || !Array.isArray(broadJson.items)) { + throw new Error("bad-path probe did not return a successful real-CTX pack"); +} +const broadPaths = itemPaths(broadJson); +const broadIrrelevant = broadPaths.filter( + (path) => path.startsWith("docs/") && path !== "docs/AUTH.md", +); +if (broadIrrelevant.length === 0) { + throw new Error("bad-path probe did not expose unrelated topical retrieval"); +} +const broadIrrelevantEvidence = await Promise.all( + broadIrrelevant.map(async (path) => { + const item = broadJson.items.find( + (candidate) => candidate?.source?.provenance?.document_path === path, + ); + const claimedSha256 = item?.source?.provenance?.document_sha256; + const actualSha256 = await fileHash(join(fixtureRoot, path)); + if (claimedSha256 !== actualSha256) { + throw new Error(`bad-path provenance digest does not match exact original ${path}`); + } + return { path, document_sha256: actualSha256 }; + }), +); +if (broadPaths.includes("docs/AUTH.md")) { + throw new Error("bad-path probe unexpectedly retrieved deliberately unindexed docs/AUTH.md"); +} + +const partialDisconfirming = run( + "disconfirming-partial-ctx", + ctx, + [ + "pack", + "Delete an owned record without disclosing another principal's record", + "--root", + fixtureRoot, + "--document", + "docs/DELETION.md", + "--token-budget", + "1300", + "--json", + ], + { cwd: fixtureRoot, expected: [0], timeout: 120_000 }, +); +const partialDisconfirmingJson = parseJsonIfPresent(partialDisconfirming.stdout); +if (partialDisconfirmingJson?.completeness_status !== "PARTIAL") { + throw new Error("disconfirming real-CTX probe did not retain its asserted PARTIAL result"); +} + +const initialAuth = factoryContext("auth-initial-missing", "AUTH-001", 0); +const initialAuthPack = await readFile(join(fixtureRoot, initialAuth.pack_path), "utf8"); +await copyArtifact(initialAuth.pack_path, "auth-initial-missing.pack.md"); +await copyArtifact(initialAuth.receipt_path, "auth-initial-missing.receipt.json"); +if (initialAuthPack.includes("docs/AUTH.md")) { + throw new Error("initial AUTH handoff unexpectedly contains the missing decisive original"); +} +await writeFile(join(fixtureRoot, ".factory/tasks/AUTH-001.yaml"), workerBeforeAuth); +const workerInitialAuth = factoryContext("auth-worker-missing", "AUTH-001", 0); +const workerInitialPack = await readFile(join(fixtureRoot, workerInitialAuth.pack_path), "utf8"); +await copyArtifact(workerInitialAuth.pack_path, "auth-worker-missing.pack.md"); +await copyArtifact(workerInitialAuth.receipt_path, "auth-worker-missing.receipt.json"); +if ( + workerInitialPack.includes("docs/AUTH.md") || + workerInitialPack.includes("request-supplied ownership") +) { + throw new Error("fresh-worker handoff did not omit the decisive creation ownership rule"); +} +await writeFile(join(fixtureRoot, ".factory/tasks/AUTH-001.yaml"), beforeAuth); + +run( + "ctx-add-auth", + ctx, + [ + "add", + "docs/AUTH.md", + "--root", + fixtureRoot, + "--authority", + "normative", + "--priority", + "100", + "--json", + ], + { + cwd: fixtureRoot, + }, +); +run("ctx-index-with-auth", ctx, ["index", "--root", fixtureRoot, "--json"], { + cwd: fixtureRoot, + timeout: 120_000, +}); +const fixedStatus = jsonRun( + "ctx-status-with-auth", + ctx, + ["status", "--root", fixtureRoot, "--json"], + { + cwd: fixtureRoot, + }, +); +assertSemantic(fixedStatus, "fixed status"); +const doctor = jsonRun( + "ctx-doctor-offline", + ctx, + ["doctor", "--offline", "--root", fixtureRoot, "--json"], + { + cwd: fixtureRoot, + }, +); +if (doctor.offline_ready !== true || doctor.network_attempted !== false) { + throw new Error("CTX offline doctor did not prove offline readiness"); +} + +const globalOnlyTaskPath = join(fixtureRoot, ".factory/tasks/GLOBAL-ONLY.yaml"); +const globalOnlyQuery = [ + "Keep credentials and raw worker transcripts out of offline context", + "A worker applies the mandatory rule that credentials and raw worker transcripts never enter a context pack and normal operation is offline.", + "A-GLOBAL: The pack contains the mandatory credentials, raw transcript, and offline constraints without topical documents.", + "credentials raw worker transcripts context pack normal operation offline", +].join("\n"); +await writeFile( + globalOnlyTaskPath, + `schema_version: 1 +id: GLOBAL-ONLY +title: Keep credentials and raw worker transcripts out of offline context +outcome: A worker applies the mandatory rule that credentials and raw worker transcripts never enter a context pack and normal operation is offline. +depends_on: [] +complexity: bounded +risk: bounded +acceptance: + - id: A-GLOBAL + statement: The pack contains the mandatory credentials, raw transcript, and offline constraints without topical documents. +advances: [C-COPY] +context: + topics: [credentials, raw worker transcripts, context pack, normal operation, offline] + required: [] +evidence: + required: [integration] +delivery: project-default +`, +); +const emptySelectionFactory = factoryContext("empty-selection-factory", "GLOBAL-ONLY", 0); +const emptySelectionPack = await readFile( + join(fixtureRoot, emptySelectionFactory.pack_path), + "utf8", +); +const emptySelectionTopicalDigests = Object.keys(emptySelectionFactory.source_digests).filter( + (path) => path.startsWith("docs/"), +); +if ( + emptySelectionTopicalDigests.length > 0 || + /^## Retrieved context: docs\//m.test(emptySelectionPack) +) { + throw new Error("empty task-local selection widened Factory output to topical authority"); +} +const emptySelectionCtx = run( + "empty-selection-real-ctx", + ctx, + [ + "pack", + globalOnlyQuery, + "--root", + fixtureRoot, + "--document", + "PROJECT.md", + "--document", + "ARCHITECTURE.md", + "--token-budget", + "3500", + "--json", + ], + { cwd: fixtureRoot, expected: [0], timeout: 120_000 }, +); +const emptySelectionCtxJson = parseJsonIfPresent(emptySelectionCtx.stdout); +if (emptySelectionCtxJson?.completeness_status !== "COMPLETE") { + throw new Error("empty-selection real-CTX probe was not COMPLETE"); +} +const emptySelectionProvenance = itemPaths(emptySelectionCtxJson); +if ( + emptySelectionProvenance.length === 0 || + emptySelectionProvenance.some((path) => !globalSources.includes(path)) +) { + throw new Error( + `empty-selection real-CTX provenance escaped mandatory globals: ${emptySelectionProvenance.join(",")}`, + ); +} +const emptySelectionOriginals = await Promise.all( + emptySelectionProvenance.map(async (path) => { + const item = emptySelectionCtxJson.items.find( + (candidate) => candidate?.source?.provenance?.document_path === path, + ); + const actualSha256 = await fileHash(join(fixtureRoot, path)); + if (item?.source?.provenance?.document_sha256 !== actualSha256) { + throw new Error(`empty-selection provenance digest does not match ${path}`); + } + return { path, document_sha256: actualSha256 }; + }), +); +await rm(globalOnlyTaskPath); + +const counterfactualBefore = factoryContext("counterfactual-before", "AUTH-001", 0); +const counterfactualBeforePack = await readFile( + join(fixtureRoot, counterfactualBefore.pack_path), + "utf8", +); +await copyArtifact(counterfactualBefore.pack_path, "counterfactual-before.pack.md"); +await copyArtifact(counterfactualBefore.receipt_path, "counterfactual-before.receipt.json"); +await writeFile(join(fixtureRoot, ".factory/tasks/AUTH-001.yaml"), fixedAuth); +const counterfactualAfter = factoryContext("counterfactual-after", "AUTH-001", 0); +const counterfactualAfterPack = await readFile( + join(fixtureRoot, counterfactualAfter.pack_path), + "utf8", +); +if ( + counterfactualBeforePack.includes("docs/AUTH.md") || + !counterfactualAfterPack.includes("docs/AUTH.md") +) { + throw new Error("one-condition AUTH counterfactual did not add the decisive original"); +} + +const packs = []; +for (const task of oracle.tasks) { + const payload = factoryContext(`fixed-${task.id}`, task.id, 0); + const packAbsolute = join(fixtureRoot, payload.pack_path); + const receiptAbsolute = join(fixtureRoot, payload.receipt_path); + const pack = await readFile(packAbsolute, "utf8"); + const receipt = JSON.parse(await readFile(receiptAbsolute, "utf8")); + const provenance = packSources(pack); + const topical = provenance.filter((path) => path.startsWith("docs/")); + const irrelevant = topical.filter((path) => path !== task.target); + const missingTarget = !topical.includes(task.target); + if (missingTarget || irrelevant.length > 0) { + throw new Error( + `${task.id} relevance failed: missing=${missingTarget} irrelevant=${irrelevant.join(",")}`, + ); + } + const packCopy = join(rawRoot, `${task.id}.pack.md`); + const receiptCopy = join(rawRoot, `${task.id}.receipt.json`); + await cp(packAbsolute, packCopy); + await cp(receiptAbsolute, receiptCopy); + packs.push({ + task_id: task.id, + risk: task.risk, + contract: fileEvidence(join(fixtureRoot, `.factory/tasks/${task.id}.yaml`)), + target_source: fileEvidence(join(fixtureRoot, task.target)), + pack: await fileEvidenceAsync(packCopy), + receipt: await fileEvidenceAsync(receiptCopy), + token_budget: payload.token_budget, + estimated_tokens: payload.estimated_tokens, + byte_budget: payload.byte_budget, + bytes: payload.bytes, + ctx_generation: payload.ctx.index_generation, + retrieval_mode: payload.ctx.retrieval_mode, + provenance, + irrelevant_topical_sources: irrelevant, + missing_target: missingTarget, + source_digests: receipt.source_digests, + }); +} + +await rm(join(fixtureRoot, ".factory/state/context/AUTH-001.md"), { force: true }); +await rm(join(fixtureRoot, ".factory/state/context/AUTH-001.json"), { force: true }); +await writeFile( + join(fixtureRoot, ".factory/tasks/AUTH-001.yaml"), + fixedAuth.replace("docs/AUTH.md", "docs/ABSENT.md"), +); +const disconfirming = factoryContext("disconfirming-missing-original", "AUTH-001", 3); +let fabricated = true; +try { + await stat(join(fixtureRoot, ".factory/state/context/AUTH-001.md")); +} catch { + fabricated = false; +} +if (fabricated || disconfirming.error?.code !== "CONTEXT_BLOCKED") { + throw new Error("missing-original disconfirming case did not fail closed without a pack"); +} + +const testedSha = gitText(root, ["rev-parse", "HEAD"]); +const sourceFiles = [ + ".factory/project.yaml", + ".factory/completion.yaml", + "PROJECT.md", + "ARCHITECTURE.md", + "oracle.json", + "cases/AUTH-before.yaml", + "cases/AUTH-worker-before.yaml", + ...oracle.tasks.map((entry) => `.factory/tasks/${entry.id}.yaml`), + ...topicalSources, +]; +const manifest = { + schema_version: 1, + gate: "P-06", + tested_sha: testedSha, + recorded_at: new Date().toISOString(), + reviewer, + fixture: { + tracked_root: "test/fixtures/cp06-context", + files: Object.fromEntries( + await Promise.all( + sourceFiles.map(async (path) => [path, await fileHash(join(fixtureSource, path))]), + ), + ), + }, + ctx: { + version: commandText(ctx, ["--version"]), + index_generation: fixedStatus.index_generation, + retrieval_mode: fixedStatus.retrieval_mode, + active_channels: fixedStatus.active_channels, + offline_ready: doctor.offline_ready, + network_attempted: doctor.network_attempted, + }, + original_failed_measurement_preserved: { + packs: 10, + topical_excerpts: 24, + irrelevant_topical_excerpts: 15, + packs_with_irrelevant_excerpt: 10, + missing_targets: 1, + source: "docs/probes/CP-06.md at PR #8 head 8b945a4ed6fcbb43a8b7aa7ce3a6532a4a23d145", + disposition: "fail", + }, + diagnosis: { + bad_unscoped_path: { + exit_code: broad.exitCode, + output_path: broad.outputPath, + output_sha256: broad.outputSha256, + provenance: broadPaths, + irrelevant_topical_sources: broadIrrelevant, + irrelevant_topical_originals: broadIrrelevantEvidence, + missing_target: true, + }, + empty_selection: { + semantics: "empty task-local authority falls back to mandatory global document filters", + factory_exit_code: 0, + factory_output_path: commandById("empty-selection-factory").outputPath, + factory_output_sha256: commandById("empty-selection-factory").outputSha256, + factory_topical_source_digests: emptySelectionTopicalDigests, + ctx_exit_code: emptySelectionCtx.exitCode, + ctx_output_path: emptySelectionCtx.outputPath, + ctx_output_sha256: emptySelectionCtx.outputSha256, + ctx_completeness_status: emptySelectionCtxJson.completeness_status, + ctx_provenance: emptySelectionOriginals, + }, + one_condition_counterfactual: { + unchanged_fields: "all AUTH task fields except context.required", + before_required: ["PROJECT.md", "ARCHITECTURE.md"], + after_required: ["PROJECT.md", "ARCHITECTURE.md", "docs/AUTH.md"], + before_contract_sha256: sha256(beforeAuth), + after_contract_sha256: sha256(fixedAuth), + before_pack_sha256: sha256(counterfactualBeforePack), + after_pack_sha256: sha256(counterfactualAfterPack), + corrected_constraints: [ + "principal-derived ownership", + "request owner ignored", + "missing or forged credential is 401", + "cross-account read/update/delete is non-mutating non-disclosing 404", + ], + }, + disconfirming_partial_case: { + condition: "a successful document-scoped real-CTX call can remain PARTIAL", + exit_code: partialDisconfirming.exitCode, + completeness_status: partialDisconfirmingJson.completeness_status, + output_path: partialDisconfirming.outputPath, + output_sha256: partialDisconfirming.outputSha256, + }, + disconfirming_missing_original_case: { + condition: "declared required original docs/ABSENT.md does not exist", + exit_code: 3, + error_code: disconfirming.error.code, + output_path: commandById("disconfirming-missing-original").outputPath, + output_sha256: commandById("disconfirming-missing-original").outputSha256, + pack_fabricated: fabricated, + }, + }, + fixed_sample: { + pack_count: packs.length, + relevant_pack_count: packs.filter( + (entry) => !entry.missing_target && entry.irrelevant_topical_sources.length === 0, + ).length, + irrelevant_topical_excerpts: packs.reduce( + (sum, entry) => sum + entry.irrelevant_topical_sources.length, + 0, + ), + packs_with_irrelevant_excerpt: packs.filter( + (entry) => entry.irrelevant_topical_sources.length > 0, + ).length, + missing_targets: packs.filter((entry) => entry.missing_target).length, + packs, + }, + commands: commands.map(({ stdout, stderr, ...entry }) => entry), + result: "pass", + limitations: [ + "The semantic replay is verified on Linux with CTX 1.0.0 and the locally installed model.", + "The small fixture demonstrates this fixed sample, not universal retrieval precision.", + "Raw CTX JSON, packs, receipts, and command logs remain ignored scratch under .factory/state.", + ], +}; +const manifestPath = join(proofRoot, "evidence.json"); +await writeFile(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`); +console.log( + JSON.stringify({ + ok: true, + gate: "P-06", + tested_sha: testedSha, + manifest_path: relative(manifestPath), + manifest_sha256: await fileHash(manifestPath), + fixed_pack_count: packs.length, + relevant_pack_count: manifest.fixed_sample.relevant_pack_count, + irrelevant_topical_excerpts: manifest.fixed_sample.irrelevant_topical_excerpts, + missing_targets: manifest.fixed_sample.missing_targets, + bad_path_exit: broad.exitCode, + bad_path_irrelevant_topical_sources: broadIrrelevant, + disconfirming_partial_exit: partialDisconfirming.exitCode, + disconfirming_missing_original_exit: 3, + }), +); + +function configureOfflineCtxModelDir() { + if (process.env.CTX_MODEL_DIR !== undefined && process.env.CTX_MODEL_DIR !== "") return; + const modelName = "BAAI/bge-small-en-v1.5"; + const revision = "52398278842ec682c6f32300af41344b1c0b0bb2"; + for (const candidate of candidateModelRoots()) { + const modelDirectory = join(candidate, modelKey(modelName, revision)); + const manifestPath = join(modelDirectory, "manifest.json"); + if (!existsSync(manifestPath)) continue; + try { + const manifest = JSON.parse(readFileSync(manifestPath, "utf8")); + if ( + manifest.provider === "fastembed" && + manifest.model_name === modelName && + manifest.revision === revision && + manifest.dimensions === 384 + ) { + process.env.CTX_MODEL_DIR = candidate; + return; + } + } catch { + // Keep the proof fail-closed; the later CTX semantic assertion reports the blocker. + } + } +} + +function candidateModelRoots() { + const candidates = []; + const home = process.env.HOME; + if (home !== undefined && home !== "") candidates.push(join(home, ".cache/ctx/models")); + const homeMatch = root.match(/^\/(?:home|Users)\/[^/]+/u); + if (homeMatch !== null) candidates.push(join(homeMatch[0], ".cache/ctx/models")); + return [...new Set(candidates)]; +} + +function modelKey(modelName, revision) { + const readable = modelName.replace(/[^A-Za-z0-9._-]+/g, "--").replace(/^-|-$/g, ""); + const digest = createHash("sha256") + .update(`${modelName}\0${revision}`) + .digest("hex") + .slice(0, 12); + return `${readable}-${digest}`; +} + +function factoryContext(id, taskId, expectedExit) { + const result = run( + id, + process.execPath, + [cli, "context", taskId, "--root", fixtureRoot, "--ctx-bin", ctx, "--json"], + { + cwd: root, + expected: [expectedExit], + timeout: 120_000, + }, + ); + const parsed = parseJsonIfPresent(result.stdout); + if (parsed === null) throw new Error(`${id} did not return JSON`); + return parsed; +} + +function jsonRun(id, command, args, options) { + const result = run(id, command, args, options); + const parsed = parseJsonIfPresent(result.stdout); + if (parsed === null) throw new Error(`${id} did not return JSON`); + return parsed; +} + +function run(id, command, args, options = {}) { + const cwd = options.cwd ?? root; + const result = spawnSync(command, args, { + cwd, + encoding: "utf8", + env: process.env, + timeout: options.timeout ?? 30_000, + }); + const exitCode = result.status; + const expected = options.expected ?? [0]; + const output = `command: ${quote([command, ...args])}\ncwd: ${cwd}\nexit: ${exitCode}\n--- stdout ---\n${result.stdout ?? ""}\n--- stderr ---\n${result.stderr ?? ""}`; + const outputPath = join(rawRoot, `${id}.txt`); + requireWrite(outputPath, output); + const entry = { + id, + command: quote([command, ...args]), + cwd: relative(cwd), + exit_code: exitCode, + output_path: relative(outputPath), + output_sha256: sha256(output), + stdout: result.stdout ?? "", + stderr: result.stderr ?? "", + }; + commands.push(entry); + if (result.error !== undefined || !expected.includes(exitCode)) { + process.stderr.write(output); + throw new Error(`${id} exited ${exitCode}; expected ${expected.join(" or ")}`); + } + return { ...entry, exitCode, outputPath: entry.output_path, outputSha256: entry.output_sha256 }; +} + +function commandById(id) { + const result = commands.find((entry) => entry.id === id); + if (result === undefined) throw new Error(`missing command record ${id}`); + return { outputPath: result.output_path, outputSha256: result.output_sha256 }; +} + +function parseJsonIfPresent(text) { + try { + return JSON.parse(text); + } catch { + return null; + } +} + +function itemPaths(value) { + if (value === null || !Array.isArray(value.items)) return []; + return [ + ...new Set( + value.items + .map((item) => item?.source?.provenance?.document_path) + .filter((path) => typeof path === "string"), + ), + ]; +} + +function packSources(pack) { + const paths = []; + for (const match of pack.matchAll( + /^## (?:Required source|Retrieved context): ([^:\n]+)(?::\d+-\d+)?$/gm, + )) { + paths.push(match[1]); + } + return [...new Set(paths)]; +} + +function assertSemantic(status, label) { + if ( + status.category !== "CLEAN" || + status.retrieval_mode !== "HYBRID_SEMANTIC" || + !Array.isArray(status.active_channels) || + !status.active_channels.includes("semantic") + ) { + throw new Error(`${label} is not clean HYBRID_SEMANTIC retrieval`); + } +} + +async function copyArtifact(path, name) { + await cp(join(fixtureRoot, path), join(rawRoot, name)); +} + +function fileEvidence(path) { + const relativePath = relative(path).replace(relative(fixtureRoot), "test/fixtures/cp06-context"); + const bytes = requireRead(path); + return { path: relativePath, sha256: sha256(bytes) }; +} + +async function fileEvidenceAsync(path) { + return { path: relative(path), sha256: await fileHash(path) }; +} + +function gitText(cwd, args) { + return commandText("git", args, cwd); +} + +function resolveTool(command) { + if (command.includes("/")) return resolve(command); + return commandText("which", [command]); +} + +function commandText(command, args, cwd = root) { + const result = spawnSync(command, args, { cwd, encoding: "utf8" }); + if (result.status !== 0) throw new Error(`${command} failed: ${result.stderr}`); + return result.stdout.trim(); +} + +function quote(parts) { + return parts.map((part) => (/[\s'"]/.test(part) ? JSON.stringify(part) : part)).join(" "); +} + +function relative(path) { + return path.startsWith(`${root}/`) ? path.slice(root.length + 1) : path; +} + +function slug(path) { + return path + .replace(/[^A-Za-z0-9]+/g, "-") + .replace(/^-|-$/g, "") + .toLowerCase(); +} + +function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} + +async function fileHash(path) { + return createHash("sha256") + .update(await readFile(path)) + .digest("hex"); +} + +function requireRead(path) { + return readFileSync(path); +} + +function requireWrite(path, value) { + writeFileSync(path, value); +} diff --git a/scripts/replay-cp06-worker.mjs b/scripts/replay-cp06-worker.mjs new file mode 100644 index 0000000..d3a4448 --- /dev/null +++ b/scripts/replay-cp06-worker.mjs @@ -0,0 +1,215 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { homedir, tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { withCleanup } from "./cp06-worker-lifecycle.mjs"; +import { + Cp06IsolationUnsupportedError, + runCp06SecurityPreflight, + runIsolatedWorker, +} from "./cp06-auth-security.mjs"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const outputRoot = resolve(root, process.env.CP06_OUTPUT ?? ".factory/state/cp06-correction"); +const workerRoot = join(outputRoot, "worker"); +const securityRoot = join(outputRoot, "auth-security"); +const fixtureRoot = join(outputRoot, "ten-pack", "repo"); +const initialPackPath = join(outputRoot, "ten-pack", "raw", "auth-worker-missing.pack.md"); +const contractPath = join(root, "test/fixtures/cp06-context/cases/AUTH-worker-before.yaml"); +const originalPath = join(fixtureRoot, "docs/AUTH.md"); +const provider = "openai-codex"; +const model = process.env.CP06_PI_MODEL ?? "gpt-5.6-sol"; +const timeoutMs = 600_000; +const systemPrompt = + "You are a bounded CP-06 reviewer. Use exactly the read tool, read only the supplied exact original once, return only the requested JSON, and do not attempt any file mutation or broad scan."; + +try { + await main(); +} catch (error) { + process.stderr.write(`${error.code ?? "CP06_WORKER_FAILED"}: ${error.message}\n`); + process.exitCode = error.exitCode ?? (error instanceof Cp06IsolationUnsupportedError ? 73 : 1); +} + +async function main() { + await rm(workerRoot, { recursive: true, force: true }); + await mkdir(workerRoot, { recursive: true }); + let security; + try { + // This DUMMY-only gate includes the unsafe control, hardened evaluated-child + // attacks, vendor refresh counters, and offline SDK read audit. It completes + // before the real credential path is opened or any provider request is made. + security = await runCp06SecurityPreflight({ + root, + outputRoot: securityRoot, + reviewer: process.env.CP06_REVIEWER, + }); + } catch (error) { + await recordBlocked(error, "dummy-security-preflight"); + throw error; + } + + const workerInput = { + schema_version: 1, + root, + fixture_root: fixtureRoot, + contract_path: contractPath, + pack_path: initialPackPath, + original_path: originalPath, + provider, + model, + timeout_ms: timeoutMs, + system_prompt: systemPrompt, + }; + const inputPath = join(workerRoot, "input.json"); + await writeFile(inputPath, `${JSON.stringify(workerInput, null, 2)}\n`); + const evaluationHome = await mkdtemp(join(tmpdir(), "factory-cp06-sdk-worker-")); + const credentialTarget = join(evaluationHome, "pi-agent", "auth.json"); + const credentialSource = resolve( + process.env.CP06_PI_AUTH_FILE ?? + join(process.env.PI_CODING_AGENT_DIR ?? join(homedir(), ".pi", "agent"), "auth.json"), + ); + const binaries = { + helper: join(securityRoot, "cp06-seccomp-exec"), + syscallProbe: join(securityRoot, "cp06-isolation-syscalls"), + }; + let isolated; + try { + isolated = await withCleanup( + () => + runIsolatedWorker({ + root, + source: credentialSource, + target: credentialTarget, + binaries, + inputPath, + timeout: timeoutMs + 60_000, + }), + () => rm(evaluationHome, { recursive: true, force: true }), + ); + } catch (error) { + await recordBlocked(error, "isolated-sdk-worker"); + throw error; + } + if ( + isolated.mode !== "worker" || + isolated.source_unchanged !== true || + isolated.mount_ids_distinct !== true || + isolated.cleanup !== "pass" || + isolated.child?.result !== "pass" + ) { + const error = new Error("isolated SDK worker omitted a required safety assertion"); + await recordBlocked(error, "isolated-sdk-worker-audit"); + throw error; + } + + const testedSha = textCommand("git", ["rev-parse", "HEAD"]); + const manifest = { + schema_version: 1, + gate: "P-06-fresh-worker", + tested_sha: testedSha, + recorded_at: new Date().toISOString(), + reviewer: process.env.CP06_REVIEWER ?? "Pi CP-06 read-only auth correction worker", + worker: { + pi_version: isolated.child.pi_version, + interface: "public SDK with injected CredentialStore", + provider, + model, + in_memory_session: isolated.child.session.in_memory, + tools: isolated.child.session.active_tools, + refresh_on_create: false, + model_catalog_network: false, + }, + bounded_handoff: { + contract_path: relative(contractPath), + contract_sha256: await fileHash(contractPath), + pack_path: relative(initialPackPath), + pack_sha256: await fileHash(initialPackPath), + original_path: relative(originalPath), + original_sha256: await fileHash(originalPath), + deliberately_omitted_constraint_sha256: sha256("request-supplied owner fields are ignored"), + evidence: "in-memory auth 401 and concealed non-mutating cross-account 404 observations", + }, + isolation: { + supported_platform: "Linux x86-64", + dummy_security_preflight: security, + source_and_target_distinct_read_only_mounts: isolated.mount_ids_distinct, + evaluated_child_capabilities: "all sets empty", + evaluated_child_no_new_privs: true, + evaluated_child_seccomp: true, + credential_store: "injected callback-denying read-only CredentialStore", + oauth_preflight: isolated.child.oauth_preflight, + credential_store_audit: isolated.child.credential_store, + credential_source_unchanged: isolated.source_unchanged, + cleanup: isolated.cleanup, + destructive_real_source_probe: false, + raw_transcript_retained: false, + }, + event_stream: isolated.child.event_stream, + command: + "Pi 0.85.1 createAgentSession({modelRuntime: injectedReadOnlyRuntime, tools: ['read'], inMemorySession}) inside capability-free seccomp child", + exit_code: 0, + response_sha256: isolated.child.response_sha256, + response: isolated.child.response, + result: "pass", + limitation: + "Live model wording is nondeterministic. The proof runner is Linux x86-64 only and exits blocked before credential access/provider use when user/mount namespaces, libseccomp, exact Pi 0.85.1, or adequate unexpired OAuth validity are unavailable.", + }; + const manifestPath = join(workerRoot, "evidence.json"); + await writeFile(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`); + console.log( + JSON.stringify({ + ok: true, + gate: manifest.gate, + tested_sha: testedSha, + manifest_path: relative(manifestPath), + manifest_sha256: await fileHash(manifestPath), + exit_code: 0, + status: manifest.response.status, + reads: manifest.event_stream.read_audit.project_read_paths, + broad_scan: manifest.response.broad_scan, + }), + ); +} + +async function recordBlocked(error, stage) { + const status = { + schema_version: 1, + gate: "P-06-fresh-worker", + result: "blocked", + stage, + code: error?.code ?? "CP06_WORKER_FAILED", + cleanup_failed: error?.code === "CP06_CLEANUP_FAILED", + worker_failure: error?.workerFailure ?? null, + causes: + error instanceof AggregateError + ? error.errors.map((cause) => ({ + code: cause?.code ?? "CP06_WORKER_FAILED", + exit_code: cause?.exitCode ?? null, + })) + : [], + recorded_at: new Date().toISOString(), + raw_transcript_retained: false, + }; + await writeFile(join(workerRoot, "blocked.json"), `${JSON.stringify(status, null, 2)}\n`); +} + +function textCommand(command, args) { + const result = spawnSync(command, args, { cwd: root, encoding: "utf8" }); + if (result.status !== 0) throw new Error(`${command} failed`); + return result.stdout.trim(); +} + +function relative(path) { + return path.startsWith(`${root}/`) ? path.slice(root.length + 1) : path; +} + +async function fileHash(path) { + return sha256(await readFile(path)); +} + +function sha256(value) { + return createHash("sha256").update(value).digest("hex"); +} diff --git a/skills/factory/SKILL.md b/skills/factory/SKILL.md index ae94486..0f790f3 100644 --- a/skills/factory/SKILL.md +++ b/skills/factory/SKILL.md @@ -37,7 +37,21 @@ For each ready task, run: context --root --json ``` -Inspect the pack and receipt for the exact task contract, required original sources, source and contract digests, CTX generation/retrieval mode, relevance, and token/byte bounds. Regenerate after any source, contract, or index change. Never place secrets, credentials, source archives, or agent transcripts in a pack. +Before generation, make `context.required` the reviewed task-local source selection: include every known original that can decide an acceptance or risk condition, but no unrelated documents as insurance. Factory reads those originals exactly and constrains CTX retrieval to the same document paths. Inspect the pack and receipt for the exact task contract, required originals, source and contract digests, CTX generation/retrieval mode, relevance, and token/byte bounds. Regenerate after any source, contract, or index change. Never place secrets, credentials, source archives, or agent transcripts in a pack. + +Give a fresh worker only the task-local brief, exact contract, bounded pack, and existing evidence—not a previous agent transcript. If a decisive source is absent, have the worker name it and make one targeted exact-original read, verify provenance, record the corrected assumption, and add that exact path to `context.required` before regeneration when it is durable authority. Do not widen every pack, hide raw retrieved excerpts, or omit a critical constraint to make retrieval appear complete. + +## Risk and task review + +Classify by failure consequence, not diff size, and use the higher class when observed risk exceeds the contract: + +- **bounded:** ordinary project checks and Firstmate's standard delivery policy; +- **normal:** targeted integration/e2e evidence, then a fresh semantic review when feasible; +- **critical:** explicit trust-boundary constraints, isolated negative tests, independent review, and human resolution of consequential policy. Use disposable identities only—never real accounts or secrets. + +Before semantic review, record the exact deterministic commands, exits, artifacts, and tested SHA. Give the reviewer the task contract, focused diff, relevant architecture and exact-original correction reads, and observed evidence—not the worker transcript. A finding must cite an acceptance ID, demonstrated scenario, or concrete risk. Repair it and rerun the changed behavior plus affected checks, record an evidence-backed disagreement, or block; unresolved critical findings cannot pass. + +This is task-local evidence policy, not another shipping gate. Do not call, embed, or rerun no-mistakes from Factory, and do not alter Firstmate routing or approvals. Firstmate's configured delivery invokes its normal gate once; Factory receipts may reference those resulting artifacts. ## Preview, then publish diff --git a/src/core/context.ts b/src/core/context.ts index a5100f9..67fa172 100644 --- a/src/core/context.ts +++ b/src/core/context.ts @@ -139,6 +139,11 @@ export async function generateContextPack(options: { } const lexical = statusMode === "LEXICAL_ONLY"; + const retrievalDocuments = + input.task.context.required.length > 0 + ? input.task.context.required + : input.project.documents.required; + const retrievalIdentities = selectedRetrievalIdentities(input.sources, retrievalDocuments); const packArgs = [ "pack", contextQuery(input.task), @@ -146,6 +151,7 @@ export async function generateContextPack(options: { options.root, "--token-budget", String(remaining), + ...retrievalDocuments.flatMap((path) => ["--document", path]), ...(lexical ? ["--no-embeddings"] : []), "--json", ]; @@ -185,6 +191,7 @@ export async function generateContextPack(options: { generation, input.sources, allSourceDigests, + retrievalIdentities, ); const markdown = renderPack({ @@ -401,12 +408,31 @@ function assertSameSnapshot(path: string, captured: string | undefined, actual: } } +function selectedRetrievalIdentities(sources: SourceFile[], selectedPaths: string[]): Set { + const identities = new Set(); + const missing: string[] = []; + for (const path of selectedPaths) { + const source = sources.find((candidate) => candidate.aliases.includes(path)); + if (source === undefined) missing.push(path); + else identities.add(source.identity); + } + if (missing.length > 0) { + blocked("CTX retrieval scope did not match the validated mandatory sources", { + reason: "RETRIEVAL_SCOPE_INVALID", + selected_documents: selectedPaths, + missing_documents: missing, + }); + } + return identities; +} + async function verifiedRanges( root: string, pack: Record, generation: number, mandatory: SourceFile[], digests: Record, + allowedIdentities: Set, ): Promise { if (!Array.isArray(pack.items)) { blocked("CTX pack omitted its items array", { reason: "CTX_PACK_INVALID" }); @@ -469,6 +495,12 @@ async function verifiedRanges( end_offset: endOffset, }); } + if (!allowedIdentities.has(document.identity)) { + blocked("CTX excerpt provenance was outside the selected retrieval scope", { + reason: "RETRIEVAL_SCOPE_WIDENED", + path, + }); + } assertSameSnapshot(path, snapshots.get(document.identity), document.sha256); assertSameSnapshot(path, digests[path], document.sha256); snapshots.set(document.identity, document.sha256); diff --git a/test/context.test.ts b/test/context.test.ts index 610e943..8e3b698 100644 --- a/test/context.test.ts +++ b/test/context.test.ts @@ -67,7 +67,7 @@ evidence: delivery: project-default `; -test("context writes exact mandatory sources, merges overlapping excerpts, and records provenance", async () => { +test("context writes exact mandatory sources and records provenance", async () => { const fixture = await seeded("success with spaces"); try { const result = await factory(fixture, { mode: "semantic" }); @@ -91,10 +91,7 @@ test("context writes exact mandatory sources, merges overlapping excerpts, and r assert.ok(pack.includes(architecture)); assert.match(pack, /Source: `PROJECT\.md`/); assert.match(pack, /CTX generation: `7`/); - assert.equal( - count(pack, "The overlapping excerpt sentinel appears exactly once in the source."), - 1, - ); + assert.doesNotMatch(pack, /The overlapping excerpt sentinel/); await assert.rejects(stat(join(fixture.root, "CONTEXT-PWNED"))); const receipt = parseObject(await readFile(join(fixture.root, payload.receipt_path), "utf8")); @@ -107,7 +104,7 @@ test("context writes exact mandatory sources, merges overlapping excerpts, and r receipt.source_digests["ARCHITECTURE.md"], payload.source_digests["ARCHITECTURE.md"], ); - assert.match(receipt.source_digests["docs/EXTRA.md"], /^[a-f0-9]{64}$/); + assert.equal(receipt.source_digests["docs/EXTRA.md"], undefined); assert.equal(receipt.ctx.index_generation, 7); assert.equal(receipt.ctx.retrieval_mode, "HYBRID_SEMANTIC"); } finally { @@ -115,6 +112,79 @@ test("context writes exact mandatory sources, merges overlapping excerpts, and r } }); +test("context limits CTX retrieval to the task-local source selection", async () => { + const fixture = await seeded("source-filter"); + try { + await writeFile( + join(fixture.root, "docs/TARGET.md"), + "# Target authority\n\nOnly this task-local source is relevant.\n", + ); + await writeFile( + join(fixture.root, ".factory/tasks/CTX-001.yaml"), + taskContract.replace( + "required: [PROJECT.md, ARCHITECTURE.md]", + "required: [PROJECT.md, ARCHITECTURE.md, docs/TARGET.md]", + ), + ); + + const result = await factory(fixture, { mode: "source-filter" }); + assert.equal(result.exit, 0, result.stderr || result.stdout); + const payload = parseObject(result.stdout); + const pack = await readFile(join(fixture.root, payload.pack_path), "utf8"); + assert.match(pack, /Only this task-local source is relevant\./); + assert.doesNotMatch(pack, /The overlapping excerpt sentinel/); + assert.equal(payload.source_digests["docs/EXTRA.md"], undefined); + } finally { + await fixture.dispose(); + } +}); + +test("context blocks CTX provenance outside the task-local source selection", async () => { + const fixture = await seeded("source-filter-adversarial"); + try { + await writeFile( + join(fixture.root, "docs/TARGET.md"), + "# Target authority\n\nOnly this task-local source is relevant.\n", + ); + await writeFile( + join(fixture.root, ".factory/tasks/CTX-001.yaml"), + taskContract.replace( + "required: [PROJECT.md, ARCHITECTURE.md]", + "required: [PROJECT.md, ARCHITECTURE.md, docs/TARGET.md]", + ), + ); + + const result = await factory(fixture, { mode: "source-filter-adversarial" }); + assert.equal(result.exit, 3, result.stderr || result.stdout); + const payload = parseObject(result.stdout); + assert.equal(payload.error.code, "CONTEXT_BLOCKED"); + assert.equal(payload.error.details.reason, "RETRIEVAL_SCOPE_WIDENED"); + assert.equal(payload.error.details.path, "docs/EXTRA.md"); + await assertNoPack(fixture.root); + } finally { + await fixture.dispose(); + } +}); + +test("empty task-local selection constrains CTX to mandatory global authority", async () => { + const fixture = await seeded("empty-source-selection"); + try { + await writeFile( + join(fixture.root, ".factory/tasks/CTX-001.yaml"), + taskContract.replace("required: [PROJECT.md, ARCHITECTURE.md]", "required: []"), + ); + + const result = await factory(fixture, { mode: "empty-source-selection" }); + assert.equal(result.exit, 0, result.stderr || result.stdout); + const payload = parseObject(result.stdout); + const pack = await readFile(join(fixture.root, payload.pack_path), "utf8"); + assert.doesNotMatch(pack, /The overlapping excerpt sentinel/); + assert.equal(payload.source_digests["docs/EXTRA.md"], undefined); + } finally { + await fixture.dispose(); + } +}); + test("context permits explicitly reported lexical fallback only when retrieval is sufficient", async () => { const fixture = await seeded("lexical"); try { @@ -381,19 +451,15 @@ test("context deduplicates mandatory and retrieved symlink identities", async () const payload = parseObject(result.stdout); const pack = await readFile(join(fixture.root, payload.pack_path), "utf8"); assert.equal(count(pack, await readFile(join(fixture.root, "PROJECT.md"), "utf8")), 1); + assert.doesNotMatch(pack, /The overlapping excerpt sentinel/); + assert.ok(pack.includes("docs/PROJECT-ALIAS.md")); assert.equal( - count(pack, "The overlapping excerpt sentinel appears exactly once in the source."), - 1, + payload.source_digests["docs/PROJECT-ALIAS.md"], + createHash("sha256") + .update(await readFile(join(fixture.root, "docs/PROJECT-ALIAS.md"))) + .digest("hex"), ); - for (const path of ["docs/PROJECT-ALIAS.md", "docs/EXTRA-ALIAS.md"]) { - assert.ok(pack.includes(path)); - assert.equal( - payload.source_digests[path], - createHash("sha256") - .update(await readFile(join(fixture.root, path))) - .digest("hex"), - ); - } + assert.equal(payload.source_digests["docs/EXTRA-ALIAS.md"], undefined); } finally { await fixture.dispose(); } @@ -572,14 +638,19 @@ if (command === "doctor") { process.exit(0); } if (command === "pack") { - const relative = "docs/EXTRA.md"; + const selectedDocuments = args.flatMap((arg, index) => arg === "--document" ? [args[index + 1]] : []); + const sourceFiltered = mode === "source-filter" && selectedDocuments.includes("docs/TARGET.md"); + const adversarialScope = mode === "source-filter-adversarial"; + const globalsOnly = mode === "empty-source-selection" && selectedDocuments.length === 2 && selectedDocuments.includes("PROJECT.md") && selectedDocuments.includes("ARCHITECTURE.md"); + const relative = adversarialScope ? "docs/EXTRA.md" : sourceFiltered ? "docs/TARGET.md" : globalsOnly ? "PROJECT.md" : selectedDocuments[0] || "docs/EXTRA.md"; const full = fs.readFileSync(path.join(root, relative), "utf8"); - const firstEnd = full.indexOf("Gamma context ends here."); - const secondStart = full.indexOf("The overlapping excerpt sentinel"); + const sentinelEnd = full.indexOf("Gamma context ends here."); + const firstEnd = sourceFiltered || globalsOnly || sentinelEnd < 0 ? full.length : sentinelEnd; + const sentinelStart = full.indexOf("The overlapping excerpt sentinel"); + const secondStart = sourceFiltered || globalsOnly || sentinelStart < 0 ? 0 : sentinelStart; const item = (start, end, text = full.slice(start, end)) => ({source:{source_type:"excerpt",text,provenance:{document_path:relative,document_sha256:hash(full),start_line:1,end_line:6,start_offset:start,end_offset:end,range_sha256:hash(text),index_generation:7}},estimated_tokens:Math.ceil(Buffer.byteLength(text)/3)}); - const items = [item(0, firstEnd), item(secondStart, full.length), item(0, firstEnd)]; + const items = sourceFiltered || globalsOnly || sentinelEnd < 0 ? [item(0, full.length)] : [item(0, firstEnd), item(secondStart, full.length), item(0, firstEnd)]; if (mode === "aliases") { - items[1].source.provenance.document_path = "docs/EXTRA-ALIAS.md"; const original = fs.readFileSync(path.join(root, "docs/PROJECT-ALIAS.md"), "utf8"); items.push({source:{text:original,provenance:{document_path:"docs/PROJECT-ALIAS.md",document_sha256:hash(original),start_offset:0,end_offset:[...original].length,index_generation:7}}}); } diff --git a/test/cp06-credential-isolation.test.mjs b/test/cp06-credential-isolation.test.mjs new file mode 100644 index 0000000..cffe2f0 --- /dev/null +++ b/test/cp06-credential-isolation.test.mjs @@ -0,0 +1,149 @@ +import { strict as assert } from "node:assert"; +import { lstat, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { + assertDestructiveCredentialWritesDenied, + mountLiveCredentialReadOnly, +} from "../scripts/cp06-credential-isolation.mjs"; + +test("live credential mount pins one inode behind distinct read-only regular mounts", async () => { + const root = await mkdtemp(join(tmpdir(), "factory-cp06-live-mount-test-")); + const source = join(root, "auth-source.json"); + const target = join(root, "agent", "auth.json"); + const sourceBytes = '{"live":true}\n'; + await writeFile(source, sourceBytes); + const sourceStat = await lstat(source, { bigint: true }); + const calls = []; + let mountId = 40; + const spawnSyncImpl = (command, args) => { + calls.push([command, ...args]); + if (command === "findmnt") { + const path = args.at(-1); + return { + status: 0, + stdout: JSON.stringify({ + filesystems: [ + { + id: ++mountId, + target: path, + "vfs-options": "ro,nosuid,nodev,noexec,relatime", + }, + ], + }), + stderr: "", + }; + } + return { status: 0, stdout: "", stderr: "" }; + }; + const lstatImpl = async (path, options) => (path === target ? sourceStat : lstat(path, options)); + try { + const mount = await mountLiveCredentialReadOnly({ + root, + source, + target, + spawnSyncImpl, + lstatImpl, + }); + try { + assert.equal(await readFile(source, "utf8"), sourceBytes); + assert.notEqual(mount.mountIds.source, mount.mountIds.target); + assert.match(calls[0][2], new RegExp(`^/proc/${process.pid}/fd/\\d+$`)); + assert.deepEqual(calls[0].slice(0, 2), ["mount", "--bind"]); + assert.equal(calls[0][3], source); + assert.deepEqual(calls[1], ["mount", "-o", "remount,bind,ro,nosuid,nodev,noexec", source]); + assert.match(calls[2][2], new RegExp(`^/proc/${process.pid}/fd/\\d+$`)); + assert.equal(calls[2][3], target); + assert.deepEqual(calls[3], ["mount", "-o", "remount,bind,ro,nosuid,nodev,noexec", target]); + } finally { + mount.cleanup(); + } + assert.deepEqual(calls.slice(-2), [ + ["umount", target], + ["umount", source], + ]); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); + +test("mount setup failure reports cleanup failure instead of discarding it", async () => { + const root = await mkdtemp(join(tmpdir(), "factory-cp06-cleanup-test-")); + const source = join(root, "auth-source.json"); + const target = join(root, "agent", "auth.json"); + await writeFile(source, "DUMMY"); + let mountCalls = 0; + const spawnSyncImpl = (command) => { + if (command === "mount" && ++mountCalls === 2) { + return { status: 32, stdout: "", stderr: "DUMMY remount failure" }; + } + if (command === "umount") { + return { status: 32, stdout: "", stderr: "DUMMY cleanup failure" }; + } + return { status: 0, stdout: "", stderr: "" }; + }; + try { + await assert.rejects( + mountLiveCredentialReadOnly({ root, source, target, spawnSyncImpl }), + AggregateError, + ); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); + +test("unsupported isolation fails before credential access or provider work", async () => { + const { verifyIsolationPrerequisites, Cp06IsolationUnsupportedError } = + await import("../scripts/cp06-auth-security.mjs"); + let invocations = 0; + assert.throws( + () => + verifyIsolationPrerequisites({ + root: process.cwd(), + spawnSyncImpl(command) { + invocations += 1; + return { + status: command === "unshare" ? 1 : 0, + stdout: "", + stderr: "DUMMY namespaces unavailable", + }; + }, + }), + Cp06IsolationUnsupportedError, + ); + assert.equal(invocations, 1); +}); + +test("destructive DUMMY proof requires denied write unlink rename symlink and hardlink", async () => { + const calls = []; + const protectedPaths = new Set(["/dummy/source-auth.json", "/dummy/target-auth.json"]); + await assertDestructiveCredentialWritesDenied([...protectedPaths], { + async writeFile(path) { + calls.push(["write", path]); + if (protectedPaths.has(path)) throw new Error("EROFS"); + }, + async rename(from, to) { + calls.push(["rename", from, to]); + if (protectedPaths.has(to)) throw new Error("EBUSY"); + }, + async unlink(path) { + calls.push(["unlink", path]); + if (protectedPaths.has(path)) throw new Error("EBUSY"); + }, + async symlink(from, to) { + calls.push(["symlink", from, to]); + }, + async link(from, to) { + calls.push(["link", from, to]); + throw new Error("EXDEV"); + }, + async rm(path) { + calls.push(["rm", path]); + }, + }); + assert.equal(calls.filter(([operation]) => operation === "rename").length, 2); + assert.equal(calls.filter(([operation]) => operation === "unlink").length, 4); + assert.equal(calls.filter(([operation]) => operation === "link").length, 2); + assert.equal(calls.filter(([operation]) => operation === "symlink").length, 0); +}); diff --git a/test/cp06-readonly-credentials.test.mjs b/test/cp06-readonly-credentials.test.mjs new file mode 100644 index 0000000..b69059e --- /dev/null +++ b/test/cp06-readonly-credentials.test.mjs @@ -0,0 +1,126 @@ +import { strict as assert } from "node:assert"; +import { mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { + Cp06AuthBlockedError, + Cp06ReadOnlyCredentialStore, +} from "../scripts/cp06-readonly-credentials.mjs"; + +const providerId = "openai-codex"; + +async function withCredential(credential, action) { + const root = await mkdtemp(join(tmpdir(), "factory-cp06-credential-store-")); + const path = join(root, "DUMMY-auth.json"); + await writeFile( + path, + `${JSON.stringify({ + [providerId]: credential, + "ignored-provider": { type: "api_key", key: "DUMMY-IGNORED" }, + })}\n`, + { mode: 0o600 }, + ); + try { + return await action({ path, root }); + } finally { + await rm(root, { recursive: true, force: true }); + } +} + +function oauth(expires) { + return { + type: "oauth", + access: "DUMMY-ACCESS", + refresh: "DUMMY-REFRESH", + expires, + }; +} + +for (const [name, expires] of [ + ["expired", 0], + ["near-expiry", 1_060_000], +]) { + test(`${name} OAuth is rejected before refresh callback and persistence`, async () => { + await withCredential(oauth(expires), async ({ path }) => { + const before = await readFile(path); + const store = await Cp06ReadOnlyCredentialStore.load({ path, providerId }); + assert.throws( + () => store.assertUsableOAuth({ now: 1_000_000, minValidityMs: 300_000 }), + Cp06AuthBlockedError, + ); + let callbackCalls = 0; + await assert.rejects( + store.modify(providerId, async () => { + callbackCalls += 1; + return oauth(9_999_999); + }), + Cp06AuthBlockedError, + ); + assert.equal(callbackCalls, 0); + assert.deepEqual(await readFile(path), before); + assert.deepEqual(store.audit(), { + reads: 0, + lists: 0, + modify_denials: 1, + delete_denials: 0, + }); + }); + }); +} + +test("unexpired OAuth resolves from a clone without mutation", async () => { + await withCredential(oauth(2_000_000), async ({ path }) => { + const before = await readFile(path); + const store = await Cp06ReadOnlyCredentialStore.load({ path, providerId }); + const info = store.assertUsableOAuth({ now: 1_000_000, minValidityMs: 300_000 }); + assert.equal(info.remainingValidityMs, 1_000_000); + const credential = await store.read(providerId); + credential.access = "DUMMY-MUTATED-CLONE"; + assert.equal((await store.read(providerId)).access, "DUMMY-ACCESS"); + assert.equal(await store.read("ignored-provider"), undefined); + assert.deepEqual(await store.list(), [{ providerId, type: "oauth" }]); + assert.deepEqual(await readFile(path), before); + }); +}); + +test("delete is denied without persistence", async () => { + await withCredential(oauth(2_000_000), async ({ path }) => { + const before = await readFile(path); + const store = await Cp06ReadOnlyCredentialStore.load({ path, providerId }); + await assert.rejects(store.delete(providerId), Cp06AuthBlockedError); + assert.deepEqual(await readFile(path), before); + assert.equal(store.audit().delete_denials, 1); + }); +}); + +test("credential store rejects API keys, malformed OAuth, symlinks, and oversized files", async () => { + await withCredential({ type: "api_key", key: "DUMMY" }, async ({ path, root }) => { + await assert.rejects( + Cp06ReadOnlyCredentialStore.load({ path, providerId }), + Cp06AuthBlockedError, + ); + const link = join(root, "DUMMY-link.json"); + await symlink(path, link); + await assert.rejects( + Cp06ReadOnlyCredentialStore.load({ path: link, providerId }), + Cp06AuthBlockedError, + ); + const large = join(root, "DUMMY-large.json"); + await writeFile(large, "x".repeat(1025)); + await assert.rejects( + Cp06ReadOnlyCredentialStore.load({ path: large, providerId, maxBytes: 1024 }), + Cp06AuthBlockedError, + ); + }); + + await withCredential( + { type: "oauth", access: "DUMMY", refresh: "DUMMY", expires: "later" }, + async ({ path }) => { + await assert.rejects( + Cp06ReadOnlyCredentialStore.load({ path, providerId }), + Cp06AuthBlockedError, + ); + }, + ); +}); diff --git a/test/cp06-worker-audit.test.mjs b/test/cp06-worker-audit.test.mjs new file mode 100644 index 0000000..c562369 --- /dev/null +++ b/test/cp06-worker-audit.test.mjs @@ -0,0 +1,186 @@ +import { strict as assert } from "node:assert"; +import { readFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import test from "node:test"; +import { auditReadEvents, isExpectedOriginalReference } from "../scripts/cp06-worker-audit.mjs"; + +const root = process.cwd(); +const originalPath = resolve(root, "test/fixtures/cp06-context/docs/AUTH.md"); +const original = await readFile(originalPath); +const options = { + root, + expectedOriginalPath: originalPath, + expectedOriginal: original, + decisiveText: ["request-supplied owner fields are ignored"], +}; +const start = { + type: "tool_execution_start", + toolName: "read", + toolCallId: "read-auth", + args: { path: originalPath }, +}; +const finalMessage = { + type: "message_end", + message: { role: "assistant", content: [{ type: "text", text: "{}" }] }, +}; + +function successfulEnd(overrides = {}) { + return { + type: "tool_execution_end", + toolName: "read", + toolCallId: "read-auth", + isError: false, + result: { content: [{ type: "text", text: original.toString("utf8") }] }, + ...overrides, + }; +} + +for (const [name, events] of [ + ["omitted completion", [start, finalMessage]], + [ + "stub failed read completion", + [ + start, + successfulEnd({ isError: true, result: { content: [{ type: "text", text: "ENOENT" }] } }), + finalMessage, + ], + ], + [ + "partial read", + [ + { ...start, args: { path: originalPath, offset: 1, limit: 1 } }, + successfulEnd({ + result: { content: [{ type: "text", text: "# Authentication and tenancy" }] }, + }), + finalMessage, + ], + ], + [ + "explicit first-line offset with complete bytes", + [{ ...start, args: { path: originalPath, offset: 1 } }, successfulEnd(), finalMessage], + ], + [ + "explicit limit with complete bytes", + [ + { ...start, args: { path: originalPath, limit: Number.MAX_SAFE_INTEGER } }, + successfulEnd(), + finalMessage, + ], + ], + [ + "wrong starting range", + [ + { ...start, args: { path: originalPath, offset: 2 } }, + successfulEnd({ + result: { + content: [ + { + type: "text", + text: original.toString("utf8").split("\n").slice(1).join("\n"), + }, + ], + }, + }), + finalMessage, + ], + ], + [ + "mismatched completion id", + [start, successfulEnd({ toolCallId: "different-call" }), finalMessage], + ], + ["completion before start", [successfulEnd(), start, finalMessage]], + ["response before completion", [start, finalMessage, successfulEnd()]], + [ + "orphan non-read completion", + [ + start, + successfulEnd(), + { + type: "tool_execution_end", + toolName: "bash", + toolCallId: "orphan-bash", + isError: false, + result: { content: [{ type: "text", text: "unexpected" }] }, + }, + finalMessage, + ], + ], +]) { + test(`worker read audit rejects ${name}`, () => { + assert.equal(auditReadEvents(events, options).ok, false); + }); +} + +test("worker read audit rejects another project read", () => { + const events = [ + start, + successfulEnd(), + { + type: "tool_execution_start", + toolName: "read", + toolCallId: "read-extra", + args: { path: resolve(root, "PROJECT.md") }, + }, + { + type: "tool_execution_end", + toolName: "read", + toolCallId: "read-extra", + isError: false, + result: { content: [{ type: "text", text: "extra" }] }, + }, + finalMessage, + ]; + assert.equal(auditReadEvents(events, options).ok, false); +}); + +test("worker missing-source identity accepts only the canonical original path", () => { + const fixtureRoot = resolve(root, "test/fixtures/cp06-context"); + assert.equal( + isExpectedOriginalReference("docs/AUTH.md", { + root, + referenceRoot: fixtureRoot, + expectedOriginalPath: originalPath, + }), + true, + ); + assert.equal( + isExpectedOriginalReference("./docs/AUTH.md", { + root, + referenceRoot: fixtureRoot, + expectedOriginalPath: originalPath, + }), + false, + ); + assert.equal( + isExpectedOriginalReference(originalPath, { + root, + referenceRoot: fixtureRoot, + expectedOriginalPath: originalPath, + }), + false, + ); + assert.equal( + isExpectedOriginalReference("docs/OTHER.md", { + root, + referenceRoot: fixtureRoot, + expectedOriginalPath: originalPath, + }), + false, + ); + assert.equal( + isExpectedOriginalReference("unrelated but nonempty", { + root, + referenceRoot: fixtureRoot, + expectedOriginalPath: originalPath, + }), + false, + ); +}); + +test("worker read audit accepts one completed exact-original read before the response", () => { + const audit = auditReadEvents([start, successfulEnd(), finalMessage], options); + assert.equal(audit.ok, true); + assert.equal(audit.summary.completed_successfully, true); + assert.equal(audit.summary.exact_original, true); + assert.equal(audit.summary.expected_sha256, audit.summary.returned_sha256); +}); diff --git a/test/cp06-worker-lifecycle.test.mjs b/test/cp06-worker-lifecycle.test.mjs new file mode 100644 index 0000000..1355bfd --- /dev/null +++ b/test/cp06-worker-lifecycle.test.mjs @@ -0,0 +1,116 @@ +import { strict as assert } from "node:assert"; +import { mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { superviseCredentialChild } from "../scripts/cp06-namespace-supervisor.mjs"; +import { Cp06CleanupError, withCleanup } from "../scripts/cp06-worker-lifecycle.mjs"; + +for (const scenario of [ + "success", + "expired", + "near-expiry", + "setup", + "launch", + "timeout", + "audit", + "cleanup", + "primary-and-cleanup", +]) { + test(`supervisor awaits cleanup and preserves failures: ${scenario}`, async () => { + const directory = await mkdtemp(join(tmpdir(), "factory-cp06-DUMMY-lifecycle-")); + const source = join(directory, "DUMMY-source"); + const target = join(directory, "DUMMY-target"); + await writeFile(source, "DUMMY ONLY"); + let cleanups = 0; + let launches = 0; + const setupError = new Error("DUMMY setup failed before mount acquired"); + const cleanupError = new Error("DUMMY cleanup failed"); + const run = () => + superviseCredentialChild( + { + mode: "worker", + source, + target, + helper: "/DUMMY-helper", + syscallProbe: "/DUMMY-probe", + childArgs: ["/DUMMY-input"], + }, + { + async mount() { + if (scenario === "setup") throw setupError; + await writeFile(target, "DUMMY placeholder"); + return { + mountIds: { source: "1", target: "2" }, + async cleanup() { + await new Promise((done) => setTimeout(done, 5)); + cleanups++; + await rm(target); + if (["cleanup", "primary-and-cleanup"].includes(scenario)) throw cleanupError; + }, + }; + }, + spawn() { + launches++; + if (scenario === "launch") throw new Error("DUMMY spawn threw"); + if (scenario === "timeout") + return { + status: null, + error: Object.assign(new Error("DUMMY timeout"), { code: "ETIMEDOUT" }), + }; + if (["expired", "near-expiry", "primary-and-cleanup"].includes(scenario)) + return { status: 75, stderr: "DUMMY auth blocked" }; + return { + status: 0, + stdout: scenario === "audit" ? "invalid JSON" : '{"result":"pass"}', + }; + }, + }, + ); + try { + if (scenario === "success") { + const result = await run(); + assert.equal(result.cleanup, "pass"); + assert.equal(result.source_unchanged, true); + } else { + await assert.rejects(run, (error) => { + if (scenario === "setup") assert.equal(error.cause, setupError); + if (["expired", "near-expiry"].includes(scenario)) assert.equal(error.exitCode, 75); + if (scenario === "timeout") assert.equal(error.cause.code, "ETIMEDOUT"); + if (["cleanup", "primary-and-cleanup"].includes(scenario)) { + assert(error instanceof Cp06CleanupError); + assert.equal(error.exitCode, 74); + assert.equal(error.errors.at(-1), cleanupError); + if (scenario === "primary-and-cleanup") { + assert.equal(error.errors[0].exitCode, 75); + assert.match(error.errors[0].message, /DUMMY auth blocked/); + assert.equal(error.cause, error.errors[0]); + } + } + return true; + }); + } + assert.equal(cleanups, scenario === "setup" ? 0 : 1); + assert.equal(launches, scenario === "setup" ? 0 : 1); + await assert.rejects(stat(target), { code: "ENOENT" }); + assert.equal(await readFile(source, "utf8"), "DUMMY ONLY"); + } finally { + await rm(directory, { recursive: true, force: true }); + } + }); +} + +test("evaluation-home cleanup finishes before the blocked result is returned", async () => { + const directory = await mkdtemp(join(tmpdir(), "factory-cp06-DUMMY-home-")); + const primary = Object.assign(new Error("DUMMY blocked"), { exitCode: 75 }); + await assert.rejects( + withCleanup( + async () => { + throw primary; + }, + () => rm(directory, { recursive: true }), + ), + (error) => error === primary, + ); + await assert.rejects(stat(directory), { code: "ENOENT" }); +}); diff --git a/test/cp06-worker-outcome.test.mjs b/test/cp06-worker-outcome.test.mjs new file mode 100644 index 0000000..41a20ea --- /dev/null +++ b/test/cp06-worker-outcome.test.mjs @@ -0,0 +1,71 @@ +import { strict as assert } from "node:assert"; +import { resolve } from "node:path"; +import test from "node:test"; +import { auditWorkerOutcome } from "../scripts/cp06-worker-outcome.mjs"; + +const root = process.cwd(); +const options = { + root, + referenceRoot: resolve(root, "test/fixtures/cp06-context"), + expectedOriginalPath: resolve(root, "test/fixtures/cp06-context/docs/AUTH.md"), +}; +const gap = { + id: "creation-time-ownership", + status: "unverified", + missing_checks: ["principal-derived-owner", "request-owner-ignored"], +}; +const response = { + status: "MISSING_SOURCE", + missing_source: "docs/AUTH.md", + broad_scan: false, + corrected_constraints: [ + "Ownership comes only from the authenticated principal; request-supplied owner fields are ignored.", + ], + evidence_gaps: [gap], +}; + +test("worker outcome accepts the required structured unverified creation checks", () => { + assert.equal(auditWorkerOutcome(response, options).ok, true); + assert.equal( + auditWorkerOutcome( + { + ...response, + evidence_gaps: [{ ...gap, missing_checks: [...gap.missing_checks].reverse() }], + }, + options, + ).ok, + true, + ); +}); + +for (const [name, evidence_gaps] of [ + ["missing", undefined], + ["empty", []], + ["legacy prose without a check disposition", ["creation evidence missing"]], + ["contradictory passed disposition", [{ ...gap, status: "pass" }]], + ["contradictory verified flag", [{ ...gap, verified: true }]], + ["wrong gap", [{ ...gap, id: "read-time-ownership" }]], + ["omitted ignored-owner check", [{ ...gap, missing_checks: ["principal-derived-owner"] }]], + [ + "duplicate checks", + [{ ...gap, missing_checks: ["principal-derived-owner", "principal-derived-owner"] }], + ], + ["mixed unverified and passed", [gap, { ...gap, status: "pass" }]], +]) { + test(`worker outcome rejects ${name} creation evidence`, () => { + assert.equal(auditWorkerOutcome({ ...response, evidence_gaps }, options).ok, false); + }); +} + +test("creation evidence cannot replace source recovery or relax the read outcome", () => { + for (const change of [ + { status: "PASS" }, + { missing_source: "docs/OTHER.md" }, + { missing_source: "./docs/AUTH.md" }, + { missing_source: resolve(root, "test/fixtures/cp06-context/docs/AUTH.md") }, + { broad_scan: true }, + { corrected_constraints: [] }, + ]) + assert.equal(auditWorkerOutcome({ ...response, ...change }, options).ok, false); + assert.equal(auditWorkerOutcome(null, options).ok, false); +}); diff --git a/test/fixtures/cp06-context/.factory/completion.yaml b/test/fixtures/cp06-context/.factory/completion.yaml new file mode 100644 index 0000000..badf482 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/completion.yaml @@ -0,0 +1,73 @@ +schema_version: 1 +project: cp06-ten-pack +conditions: + - { + id: C-COPY, + outcome: Copy follows the style authority., + method: executable, + check_id: copy, + evidence: evidence/COPY.json, + } + - { + id: C-CONFIG, + outcome: Configuration precedence is stable., + method: executable, + check_id: config, + evidence: evidence/CONFIG.json, + } + - { + id: C-API, + outcome: API errors are stable., + method: executable, + check_id: api, + evidence: evidence/API.json, + } + - { + id: C-PERSIST, + outcome: Saved records survive restart., + method: executable, + check_id: persist, + evidence: evidence/PERSIST.json, + } + - { + id: C-UI, + outcome: Empty state is accessible., + method: executable, + check_id: ui, + evidence: evidence/UI.json, + } + - { + id: C-IMPORT, + outcome: Invalid import rows are reported., + method: executable, + check_id: import, + evidence: evidence/IMPORT.json, + } + - { + id: C-LOG, + outcome: Logs redact record content., + method: executable, + check_id: log, + evidence: evidence/LOG.json, + } + - { + id: C-EXPORT, + outcome: Export order is deterministic., + method: executable, + check_id: export, + evidence: evidence/EXPORT.json, + } + - { + id: C-DELETE, + outcome: Deletion is non-disclosing., + method: executable, + check_id: delete, + evidence: evidence/DELETE.json, + } + - { + id: C-AUTH, + outcome: Ownership derives from the principal., + method: executable, + check_id: auth, + evidence: evidence/AUTH.json, + } diff --git a/test/fixtures/cp06-context/.factory/project.yaml b/test/fixtures/cp06-context/.factory/project.yaml new file mode 100644 index 0000000..8a909d6 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/project.yaml @@ -0,0 +1,17 @@ +schema_version: 1 +id: cp06-ten-pack +name: CP-06 Ten Pack +intent: Prove that bounded worker packs contain only task-relevant authoritative knowledge. +audience: Factory reviewers. +goals: + - Generate ten bounded source-attributed packs. + - Correct a missing decisive source through one targeted required-original selection. +non_goals: + - Exercise a live account or network service. +constraints: + - Credentials and raw worker transcripts never enter a context pack. + - Normal fixture execution is offline and deterministic. + - Critical ownership constraints cannot be omitted to improve relevance metrics. +documents: + required: [PROJECT.md, ARCHITECTURE.md] +completion: .factory/completion.yaml diff --git a/test/fixtures/cp06-context/.factory/tasks/API-001.yaml b/test/fixtures/cp06-context/.factory/tasks/API-001.yaml new file mode 100644 index 0000000..8360cf2 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/API-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: API-001 +title: Stabilize profile API errors +outcome: Invalid profile requests return stable machine-readable errors. +depends_on: [] +complexity: bounded +risk: normal +acceptance: + - { id: A-API, statement: Malformed JSON and blank profiles return their specified 400 errors. } +advances: [C-API] +context: + topics: [profile API, malformed JSON, stable errors] + required: [PROJECT.md, ARCHITECTURE.md, docs/API.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/AUTH-001.yaml b/test/fixtures/cp06-context/.factory/tasks/AUTH-001.yaml new file mode 100644 index 0000000..a617d5d --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/AUTH-001.yaml @@ -0,0 +1,19 @@ +schema_version: 1 +id: AUTH-001 +title: Bind item ownership to authenticated principal +outcome: Creation ignores request-supplied ownership and cross-account operations remain concealed. +depends_on: [] +complexity: bounded +risk: critical +acceptance: + - { + id: A-AUTH, + statement: Missing or forged credentials return 401 and cross-account read update and delete return non-mutating 404., + } +advances: [C-AUTH] +context: + topics: [principal ownership, tenancy, forged credentials] + required: [PROJECT.md, ARCHITECTURE.md, docs/AUTH.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/CONFIG-001.yaml b/test/fixtures/cp06-context/.factory/tasks/CONFIG-001.yaml new file mode 100644 index 0000000..7f5c99f --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/CONFIG-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: CONFIG-001 +title: Enforce configuration precedence +outcome: Explicit options override project values and defaults. +depends_on: [] +complexity: bounded +risk: bounded +acceptance: + - { id: A-CONFIG, statement: Unknown keys fail and precedence is deterministic. } +advances: [C-CONFIG] +context: + topics: [configuration, precedence, unknown keys] + required: [PROJECT.md, ARCHITECTURE.md, docs/CONFIG.md] +evidence: + required: [unit] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/COPY-001.yaml b/test/fixtures/cp06-context/.factory/tasks/COPY-001.yaml new file mode 100644 index 0000000..f0bd6b0 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/COPY-001.yaml @@ -0,0 +1,19 @@ +schema_version: 1 +id: COPY-001 +title: Clarify empty-state copy +outcome: Empty-state text is direct and actionable. +depends_on: [] +complexity: bounded +risk: bounded +acceptance: + - { + id: A-COPY, + statement: Empty-state copy names the next action without implying an existing record., + } +advances: [C-COPY] +context: + topics: [empty state, interface copy, next action] + required: [PROJECT.md, ARCHITECTURE.md, docs/STYLE.md] +evidence: + required: [unit] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/DELETE-001.yaml b/test/fixtures/cp06-context/.factory/tasks/DELETE-001.yaml new file mode 100644 index 0000000..e88721d --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/DELETE-001.yaml @@ -0,0 +1,19 @@ +schema_version: 1 +id: DELETE-001 +title: Conceal cross-account deletion +outcome: Only an owner can delete a record or emit its audit marker. +depends_on: [] +complexity: bounded +risk: critical +acceptance: + - { + id: A-DELETE, + statement: A different principal receives 404 with no mutation or audit marker., + } +advances: [C-DELETE] +context: + topics: [deletion, non-disclosure, audit marker] + required: [PROJECT.md, ARCHITECTURE.md, docs/DELETION.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/EXPORT-001.yaml b/test/fixtures/cp06-context/.factory/tasks/EXPORT-001.yaml new file mode 100644 index 0000000..0b31956 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/EXPORT-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: EXPORT-001 +title: Produce deterministic exports +outcome: An unchanged dataset exports to identical UTF-8 JSON bytes. +depends_on: [] +complexity: bounded +risk: normal +acceptance: + - { id: A-EXPORT, statement: Record and object-key order are stable across repeated exports. } +advances: [C-EXPORT] +context: + topics: [UTF-8 JSON, deterministic export, stable order] + required: [PROJECT.md, ARCHITECTURE.md, docs/EXPORT.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/IMPORT-001.yaml b/test/fixtures/cp06-context/.factory/tasks/IMPORT-001.yaml new file mode 100644 index 0000000..fad3a48 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/IMPORT-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: IMPORT-001 +title: Report invalid import rows +outcome: Invalid rows are reported without changing valid-row order or the source. +depends_on: [] +complexity: bounded +risk: normal +acceptance: + - { id: A-IMPORT, statement: Stable row numbers and reasons identify invalid rows. } +advances: [C-IMPORT] +context: + topics: [import rows, validation reasons, input order] + required: [PROJECT.md, ARCHITECTURE.md, docs/IMPORT.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/LOG-001.yaml b/test/fixtures/cp06-context/.factory/tasks/LOG-001.yaml new file mode 100644 index 0000000..cd6180a --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/LOG-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: LOG-001 +title: Redact operational logs +outcome: Logs retain identifiers but omit credentials and record content. +depends_on: [] +complexity: bounded +risk: normal +acceptance: + - { id: A-LOG, statement: Denied-operation logs contain no private resource fields. } +advances: [C-LOG] +context: + topics: [operational logs, redaction, denied operation] + required: [PROJECT.md, ARCHITECTURE.md, docs/OBSERVABILITY.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/PERSIST-001.yaml b/test/fixtures/cp06-context/.factory/tasks/PERSIST-001.yaml new file mode 100644 index 0000000..11028f2 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/PERSIST-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: PERSIST-001 +title: Preserve records across restart +outcome: A successful save remains visible after restart. +depends_on: [] +complexity: bounded +risk: normal +acceptance: + - { id: A-PERSIST, statement: Restarting against the same store retains the saved record. } +advances: [C-PERSIST] +context: + topics: [save, restart, local store] + required: [PROJECT.md, ARCHITECTURE.md, docs/PERSISTENCE.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.factory/tasks/UI-001.yaml b/test/fixtures/cp06-context/.factory/tasks/UI-001.yaml new file mode 100644 index 0000000..52be0f7 --- /dev/null +++ b/test/fixtures/cp06-context/.factory/tasks/UI-001.yaml @@ -0,0 +1,16 @@ +schema_version: 1 +id: UI-001 +title: Make empty state accessible +outcome: The empty view exposes one labeled keyboard-reachable action. +depends_on: [] +complexity: bounded +risk: normal +acceptance: + - { id: A-UI, statement: Heading action label and focus order satisfy the empty-state contract. } +advances: [C-UI] +context: + topics: [empty view, keyboard focus, programmatic label] + required: [PROJECT.md, ARCHITECTURE.md, docs/UI.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/.gitignore b/test/fixtures/cp06-context/.gitignore new file mode 100644 index 0000000..1361cc5 --- /dev/null +++ b/test/fixtures/cp06-context/.gitignore @@ -0,0 +1,2 @@ +.factory/state/ +.ctx/ diff --git a/test/fixtures/cp06-context/ARCHITECTURE.md b/test/fixtures/cp06-context/ARCHITECTURE.md new file mode 100644 index 0000000..5836bbf --- /dev/null +++ b/test/fixtures/cp06-context/ARCHITECTURE.md @@ -0,0 +1,7 @@ +# Ten-pack fixture architecture + +The fixture uses an in-memory boundary around API, persistence, and presentation behavior. All operations are deterministic and make no network or account access. + +## Trust boundary + +Authenticated identity is authoritative for ownership. Item identifiers are scoped to that identity, so create-time collisions cannot disclose another account's identifiers. Cross-account operations neither disclose existence nor mutate state. diff --git a/test/fixtures/cp06-context/PROJECT.md b/test/fixtures/cp06-context/PROJECT.md new file mode 100644 index 0000000..1114195 --- /dev/null +++ b/test/fixtures/cp06-context/PROJECT.md @@ -0,0 +1,9 @@ +# Ten-pack fixture project + +The local service manages one person's saved records through stable APIs. Context packs must contain the exact task contract, the global privacy and offline constraints, and only task-relevant authority. + +## Global constraints + +- Credentials and raw worker transcripts never enter a context pack. +- Normal operation is offline. +- Errors use stable machine-readable codes. diff --git a/test/fixtures/cp06-context/cases/AUTH-before.yaml b/test/fixtures/cp06-context/cases/AUTH-before.yaml new file mode 100644 index 0000000..dfb7b9b --- /dev/null +++ b/test/fixtures/cp06-context/cases/AUTH-before.yaml @@ -0,0 +1,17 @@ +schema_version: 1 +id: AUTH-001 +title: Bind item ownership to authenticated principal +outcome: Creation ignores request-supplied ownership and cross-account operations remain concealed. +depends_on: [] +complexity: bounded +risk: critical +acceptance: + - id: A-AUTH + statement: Missing or forged credentials return 401 and cross-account read update and delete return non-mutating 404. +advances: [C-AUTH] +context: + topics: [principal ownership, tenancy, forged credentials] + required: [PROJECT.md, ARCHITECTURE.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/cases/AUTH-worker-before.yaml b/test/fixtures/cp06-context/cases/AUTH-worker-before.yaml new file mode 100644 index 0000000..2a4ff6e --- /dev/null +++ b/test/fixtures/cp06-context/cases/AUTH-worker-before.yaml @@ -0,0 +1,17 @@ +schema_version: 1 +id: AUTH-001 +title: Conceal cross-account item operations +outcome: Existing-item operations across accounts remain concealed and non-mutating. +depends_on: [] +complexity: bounded +risk: critical +acceptance: + - id: A-AUTH + statement: Missing or forged credentials return 401 and cross-account read update and delete return non-mutating 404. +advances: [C-AUTH] +context: + topics: [tenancy, forged credentials] + required: [PROJECT.md, ARCHITECTURE.md] +evidence: + required: [integration] +delivery: project-default diff --git a/test/fixtures/cp06-context/docs/API.md b/test/fixtures/cp06-context/docs/API.md new file mode 100644 index 0000000..5d31e72 --- /dev/null +++ b/test/fixtures/cp06-context/docs/API.md @@ -0,0 +1,3 @@ +# Profile API + +Malformed JSON returns `400 invalid_json`; a blank display name returns `400 invalid_profile`. A valid GET remains successful and unknown routes return `404`. diff --git a/test/fixtures/cp06-context/docs/AUTH.md b/test/fixtures/cp06-context/docs/AUTH.md new file mode 100644 index 0000000..500493e --- /dev/null +++ b/test/fixtures/cp06-context/docs/AUTH.md @@ -0,0 +1,3 @@ +# Authentication and tenancy + +Ownership comes only from the authenticated principal; request-supplied owner fields are ignored. Missing or forged fixture credentials return `401`. Cross-account read, update, and delete return a non-disclosing `404` and must not mutate state. diff --git a/test/fixtures/cp06-context/docs/CONFIG.md b/test/fixtures/cp06-context/docs/CONFIG.md new file mode 100644 index 0000000..e139c3d --- /dev/null +++ b/test/fixtures/cp06-context/docs/CONFIG.md @@ -0,0 +1,3 @@ +# Configuration + +Explicit command options override project configuration, which overrides built-in defaults. Unknown keys fail validation instead of being ignored. diff --git a/test/fixtures/cp06-context/docs/DELETION.md b/test/fixtures/cp06-context/docs/DELETION.md new file mode 100644 index 0000000..cfb6552 --- /dev/null +++ b/test/fixtures/cp06-context/docs/DELETION.md @@ -0,0 +1,3 @@ +# Deletion + +Deletion removes an owned record and emits an audit marker. A different principal receives non-disclosing `404`, no audit marker, and no state mutation. diff --git a/test/fixtures/cp06-context/docs/EXPORT.md b/test/fixtures/cp06-context/docs/EXPORT.md new file mode 100644 index 0000000..99edeb8 --- /dev/null +++ b/test/fixtures/cp06-context/docs/EXPORT.md @@ -0,0 +1,3 @@ +# Export + +Exports are UTF-8 JSON with records ordered by identifier and object keys in a stable order. Repeating an unchanged export produces identical bytes. diff --git a/test/fixtures/cp06-context/docs/IMPORT.md b/test/fixtures/cp06-context/docs/IMPORT.md new file mode 100644 index 0000000..5776c7f --- /dev/null +++ b/test/fixtures/cp06-context/docs/IMPORT.md @@ -0,0 +1,3 @@ +# Import + +Rows are validated independently. Invalid rows report stable row numbers and reasons while valid rows retain input order; no source file is modified. diff --git a/test/fixtures/cp06-context/docs/OBSERVABILITY.md b/test/fixtures/cp06-context/docs/OBSERVABILITY.md new file mode 100644 index 0000000..b6a599c --- /dev/null +++ b/test/fixtures/cp06-context/docs/OBSERVABILITY.md @@ -0,0 +1,3 @@ +# Observability + +Operational logs include request and result identifiers but redact credentials, titles, and record bodies. A denied operation records no private resource fields. diff --git a/test/fixtures/cp06-context/docs/PERSISTENCE.md b/test/fixtures/cp06-context/docs/PERSISTENCE.md new file mode 100644 index 0000000..881bfb4 --- /dev/null +++ b/test/fixtures/cp06-context/docs/PERSISTENCE.md @@ -0,0 +1,3 @@ +# Persistence + +A successful save is committed before the response is returned. Restarting the service and reopening the same local store preserves the record. diff --git a/test/fixtures/cp06-context/docs/STYLE.md b/test/fixtures/cp06-context/docs/STYLE.md new file mode 100644 index 0000000..814a96f --- /dev/null +++ b/test/fixtures/cp06-context/docs/STYLE.md @@ -0,0 +1,3 @@ +# Interface copy style + +Use direct labels and sentence-case actions. Empty-state copy says what the user can do next and never implies that a record already exists. diff --git a/test/fixtures/cp06-context/docs/UI.md b/test/fixtures/cp06-context/docs/UI.md new file mode 100644 index 0000000..a92922f --- /dev/null +++ b/test/fixtures/cp06-context/docs/UI.md @@ -0,0 +1,3 @@ +# Empty state UI + +The empty view has one visible heading, one actionable control, and a programmatic label. Keyboard focus reaches the action before decorative content. diff --git a/test/fixtures/cp06-context/oracle.json b/test/fixtures/cp06-context/oracle.json new file mode 100644 index 0000000..e2706c3 --- /dev/null +++ b/test/fixtures/cp06-context/oracle.json @@ -0,0 +1,16 @@ +{ + "schema_version": 1, + "rule": "PROJECT.md and ARCHITECTURE.md are relevant global authority; for each task only its declared target source is relevant topical authority; every other topical document is irrelevant.", + "tasks": [ + { "id": "COPY-001", "risk": "bounded", "target": "docs/STYLE.md" }, + { "id": "CONFIG-001", "risk": "bounded", "target": "docs/CONFIG.md" }, + { "id": "API-001", "risk": "normal", "target": "docs/API.md" }, + { "id": "PERSIST-001", "risk": "normal", "target": "docs/PERSISTENCE.md" }, + { "id": "UI-001", "risk": "normal", "target": "docs/UI.md" }, + { "id": "IMPORT-001", "risk": "normal", "target": "docs/IMPORT.md" }, + { "id": "LOG-001", "risk": "normal", "target": "docs/OBSERVABILITY.md" }, + { "id": "EXPORT-001", "risk": "normal", "target": "docs/EXPORT.md" }, + { "id": "DELETE-001", "risk": "critical", "target": "docs/DELETION.md" }, + { "id": "AUTH-001", "risk": "critical", "target": "docs/AUTH.md" } + ] +} diff --git a/test/fixtures/workflow/critical-auth-isolation.ts b/test/fixtures/workflow/critical-auth-isolation.ts new file mode 100644 index 0000000..ff9f732 --- /dev/null +++ b/test/fixtures/workflow/critical-auth-isolation.ts @@ -0,0 +1,75 @@ +export interface AuthResponse { + status: number; + body: Record; +} + +interface Item { + id: string; + owner: "account-a" | "account-b"; + title: string; +} + +const fixturePrincipals = new Map([ + ["fixture-token-a", "account-a"], + ["fixture-token-b", "account-b"], +]); + +export function createAuthIsolationFixture(): { + request(input: { + token?: string; + method: "GET" | "POST" | "PUT" | "DELETE"; + itemId: string; + title?: string; + requestedOwner?: string; + }): AuthResponse; + snapshot(): ReadonlyArray>; +} { + const itemKey = (owner: Item["owner"], id: string): string => `${owner}\u0000${id}`; + const items = new Map([ + [itemKey("account-a", "item-a"), { id: "item-a", owner: "account-a", title: "Private A" }], + [itemKey("account-b", "item-b"), { id: "item-b", owner: "account-b", title: "Private B" }], + ]); + + return { + request(input): AuthResponse { + const principal = input.token === undefined ? undefined : fixturePrincipals.get(input.token); + if (principal === undefined) { + return { status: 401, body: { error: "unauthorized" } }; + } + const key = itemKey(principal, input.itemId); + const item = items.get(key); + if (input.method === "POST") { + if (typeof input.title !== "string" || input.title.trim().length === 0) { + return { status: 400, body: { error: "invalid_title" } }; + } + if (item !== undefined) return { status: 409, body: { error: "already_exists" } }; + const created: Item = { + id: input.itemId, + owner: principal, + title: input.title.trim(), + }; + items.set(key, created); + return { status: 201, body: { id: created.id, title: created.title } }; + } + if (item === undefined) { + // Deliberately conceal whether another account owns this identifier. + return { status: 404, body: { error: "not_found" } }; + } + if (input.method === "GET") { + return { status: 200, body: { id: item.id, title: item.title } }; + } + if (input.method === "DELETE") { + items.delete(key); + return { status: 204, body: {} }; + } + if (typeof input.title !== "string" || input.title.trim().length === 0) { + return { status: 400, body: { error: "invalid_title" } }; + } + item.title = input.title.trim(); + return { status: 200, body: { id: item.id, title: item.title } }; + }, + snapshot(): ReadonlyArray> { + return [...items.values()].map((item) => ({ ...item })); + }, + }; +} diff --git a/test/fixtures/workflow/history/finding.json b/test/fixtures/workflow/history/finding.json new file mode 100644 index 0000000..9e29e8b --- /dev/null +++ b/test/fixtures/workflow/history/finding.json @@ -0,0 +1,11 @@ +{ + "schema_version": 1, + "finding_id": "P07-MALFORMED-JSON", + "acceptance_id": "P-07", + "reviewer": "CP-06 simulated task reviewer", + "scenario": "PUT /v1/profile receives malformed JSON", + "finding": "The normal API fixture throws SyntaxError instead of returning the stable 400 invalid_json response.", + "disposition": "repaired", + "repair": "Catch parse failure and return status 400 with error invalid_json.", + "retest": "Execute the same malformed-JSON behavioral assertion against the checked-in final fixture." +} diff --git a/test/fixtures/workflow/history/malformed-json-probe.mjs b/test/fixtures/workflow/history/malformed-json-probe.mjs new file mode 100644 index 0000000..23af151 --- /dev/null +++ b/test/fixtures/workflow/history/malformed-json-probe.mjs @@ -0,0 +1,7 @@ +import { strict as assert } from "node:assert"; +import { handleProfileRequest } from "./normal-api-before.mjs"; + +assert.deepEqual(handleProfileRequest({ method: "PUT", path: "/v1/profile", body: "not-json" }), { + status: 400, + body: { error: "invalid_json" }, +}); diff --git a/test/fixtures/workflow/history/normal-api-before.mjs b/test/fixtures/workflow/history/normal-api-before.mjs new file mode 100644 index 0000000..f478308 --- /dev/null +++ b/test/fixtures/workflow/history/normal-api-before.mjs @@ -0,0 +1,11 @@ +export function handleProfileRequest(input) { + if (input.path !== "/v1/profile") return { status: 404, body: { error: "not_found" } }; + if (input.method === "GET") return { status: 200, body: { display_name: "Fixture User" } }; + if (input.method !== "PUT") return { status: 405, body: { error: "method_not_allowed" } }; + const parsed = JSON.parse(input.body ?? ""); + const displayName = parsed?.display_name; + if (typeof displayName !== "string" || displayName.trim().length === 0) { + return { status: 400, body: { error: "invalid_profile" } }; + } + return { status: 200, body: { display_name: displayName.trim() } }; +} diff --git a/test/fixtures/workflow/normal-api.ts b/test/fixtures/workflow/normal-api.ts new file mode 100644 index 0000000..95bafda --- /dev/null +++ b/test/fixtures/workflow/normal-api.ts @@ -0,0 +1,35 @@ +export interface ApiResponse { + status: number; + body: Record; +} + +export function handleProfileRequest(input: { + method: string; + path: string; + body?: string; +}): ApiResponse { + if (input.path !== "/v1/profile") { + return { status: 404, body: { error: "not_found" } }; + } + if (input.method === "GET") { + return { status: 200, body: { display_name: "Fixture User" } }; + } + if (input.method !== "PUT") { + return { status: 405, body: { error: "method_not_allowed" } }; + } + + let parsed: unknown; + try { + parsed = JSON.parse(input.body ?? "") as unknown; + } catch { + return { status: 400, body: { error: "invalid_json" } }; + } + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + return { status: 400, body: { error: "invalid_profile" } }; + } + const displayName = (parsed as Record).display_name; + if (typeof displayName !== "string" || displayName.trim().length === 0) { + return { status: 400, body: { error: "invalid_profile" } }; + } + return { status: 200, body: { display_name: displayName.trim() } }; +} diff --git a/test/workflow.test.ts b/test/workflow.test.ts new file mode 100644 index 0000000..4d70997 --- /dev/null +++ b/test/workflow.test.ts @@ -0,0 +1,168 @@ +import { strict as assert } from "node:assert"; +import { spawnSync } from "node:child_process"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import test from "node:test"; +import { createAuthIsolationFixture } from "./fixtures/workflow/critical-auth-isolation.js"; +import { handleProfileRequest } from "./fixtures/workflow/normal-api.js"; + +const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), "../.."); + +test("normal API fixture preserves successful profile behavior", () => { + assert.deepEqual(handleProfileRequest({ method: "GET", path: "/v1/profile" }), { + status: 200, + body: { display_name: "Fixture User" }, + }); + assert.deepEqual( + handleProfileRequest({ + method: "PUT", + path: "/v1/profile", + body: JSON.stringify({ display_name: " Updated Name " }), + }), + { status: 200, body: { display_name: "Updated Name" } }, + ); +}); + +test("normal API fixture rejects malformed, invalid, unknown, and unsupported requests", () => { + assert.deepEqual(handleProfileRequest({ method: "PUT", path: "/v1/profile", body: "not-json" }), { + status: 400, + body: { error: "invalid_json" }, + }); + assert.deepEqual( + handleProfileRequest({ + method: "PUT", + path: "/v1/profile", + body: JSON.stringify({ display_name: " " }), + }), + { status: 400, body: { error: "invalid_profile" } }, + ); + assert.equal(handleProfileRequest({ method: "GET", path: "/v1/missing" }).status, 404); + assert.equal(handleProfileRequest({ method: "POST", path: "/v1/profile" }).status, 405); +}); + +test("retained P-07 finding fails before repair and passes through the final API", () => { + const before = spawnSync( + process.execPath, + ["test/fixtures/workflow/history/malformed-json-probe.mjs"], + { cwd: repoRoot, encoding: "utf8" }, + ); + assert.equal(before.status, 1, before.stdout + before.stderr); + assert.match(before.stdout + before.stderr, /SyntaxError/); + assert.deepEqual(handleProfileRequest({ method: "PUT", path: "/v1/profile", body: "not-json" }), { + status: 400, + body: { error: "invalid_json" }, + }); +}); + +test("critical auth fixture allows each account to observe its own item", () => { + const api = createAuthIsolationFixture(); + assert.deepEqual(api.request({ token: "fixture-token-a", method: "GET", itemId: "item-a" }), { + status: 200, + body: { id: "item-a", title: "Private A" }, + }); + assert.deepEqual(api.request({ token: "fixture-token-b", method: "GET", itemId: "item-b" }), { + status: 200, + body: { id: "item-b", title: "Private B" }, + }); +}); + +test("critical auth fixture rejects absent and forged fixture credentials", () => { + const api = createAuthIsolationFixture(); + assert.equal(api.request({ method: "GET", itemId: "item-a" }).status, 401); + assert.equal( + api.request({ token: "not-a-fixture-principal", method: "GET", itemId: "item-a" }).status, + 401, + ); +}); + +test("critical auth fixture conceals and preserves another account's item", () => { + const api = createAuthIsolationFixture(); + const before = api.snapshot(); + + assert.equal( + api.request({ token: "fixture-token-b", method: "GET", itemId: "item-a" }).status, + 404, + ); + assert.equal( + api.request({ + token: "fixture-token-b", + method: "PUT", + itemId: "item-a", + title: "Stolen", + requestedOwner: "account-b", + }).status, + 404, + ); + assert.equal( + api.request({ token: "fixture-token-b", method: "DELETE", itemId: "item-a" }).status, + 404, + ); + assert.deepEqual(api.snapshot(), before); +}); + +test("critical auth fixture prevents cross-tenant identifier collision probes", () => { + const api = createAuthIsolationFixture(); + const accountABefore = api + .snapshot() + .find((item) => item.owner === "account-a" && item.id === "item-a"); + + assert.deepEqual( + api.request({ + token: "fixture-token-b", + method: "POST", + itemId: "item-a", + title: "Private B with tenant-local id", + }), + { status: 201, body: { id: "item-a", title: "Private B with tenant-local id" } }, + ); + assert.deepEqual(api.request({ token: "fixture-token-a", method: "GET", itemId: "item-a" }), { + status: 200, + body: { id: "item-a", title: "Private A" }, + }); + assert.deepEqual(api.request({ token: "fixture-token-b", method: "GET", itemId: "item-a" }), { + status: 200, + body: { id: "item-a", title: "Private B with tenant-local id" }, + }); + assert.deepEqual( + api.snapshot().find((item) => item.owner === "account-a" && item.id === "item-a"), + accountABefore, + ); +}); + +test("critical auth fixture validates title before tenant-local collision state", () => { + const api = createAuthIsolationFixture(); + const before = api.snapshot(); + + assert.deepEqual( + api.request({ token: "fixture-token-b", method: "POST", itemId: "item-a", title: "" }), + { status: 400, body: { error: "invalid_title" } }, + ); + assert.deepEqual( + api.request({ token: "fixture-token-b", method: "POST", itemId: "unused", title: "" }), + { status: 400, body: { error: "invalid_title" } }, + ); + assert.deepEqual(api.snapshot(), before); +}); + +test("critical auth fixture derives new item ownership from the principal", () => { + const api = createAuthIsolationFixture(); + assert.equal( + api.request({ + token: "fixture-token-b", + method: "POST", + itemId: "item-new", + title: "Owned by B", + requestedOwner: "account-a", + }).status, + 201, + ); + assert.equal( + api.request({ token: "fixture-token-a", method: "GET", itemId: "item-new" }).status, + 404, + ); + assert.deepEqual(api.request({ token: "fixture-token-b", method: "GET", itemId: "item-new" }), { + status: 200, + body: { id: "item-new", title: "Owned by B" }, + }); + assert.equal(api.snapshot().find((item) => item.id === "item-new")?.owner, "account-b"); +});