From d111802ce7432dbe2f940d3c541fd09ffd9143e3 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:28:47 -0400 Subject: [PATCH 01/32] docs: add versioned agent-client capability matrix and policy controls Add a data-only matrix (src/agents/client-capabilities.ts) that classifies packaging, skill discovery, hooks, plugin MCP, standalone MCP, rules and UI for Codex, Claude Code, Cursor and standalone consumers. Each cell records status, evidence class, evidence ref, observed client version, source SHA and host label. The rendered reference doc carries companion-package layout and model-policy ownership decisions. Tests enforce that every cell is classified, verified cells cite an existing fixture-strength test, manifest or listing evidence never verifies loading, and manifest-declared components agree with the matrix. The plugin generation test gains a negative-controlled scan of plugins/hack and examples/basic for model, permission and output-style settings and non-primer hooks, plus a skill inventory check against the generator and docs/integrations.md, which now names all four shipped skills. Co-Authored-By: Claude Opus 5.5 --- docs/agent-guidance.md | 6 + docs/integrations.md | 9 +- docs/reference/agent-client-capabilities.md | 143 ++++++ src/agents/client-capabilities.ts | 517 ++++++++++++++++++++ tests/agent-client-capabilities.test.ts | 465 ++++++++++++++++++ tests/agent-plugin-generation.test.ts | 262 +++++++++- 6 files changed, 1398 insertions(+), 4 deletions(-) create mode 100644 docs/reference/agent-client-capabilities.md create mode 100644 src/agents/client-capabilities.ts create mode 100644 tests/agent-client-capabilities.test.ts diff --git a/docs/agent-guidance.md b/docs/agent-guidance.md index 44ae2c1d0..2e7c1a2eb 100644 --- a/docs/agent-guidance.md +++ b/docs/agent-guidance.md @@ -3,6 +3,10 @@ Hack owns Hack-specific instructions. User and project policy continues to own model selection, general coding standards, work-unit planning, evolution, and writeback. Do not ship provider model pins or copy global instruction packs into this plugin. +`tests/agent-plugin-generation.test.ts` enforces this for every file under +`plugins/hack` and `examples/basic`: no model or model-override keys, no permission, +approval or sandbox overrides, no output-style settings, and hooks that run only +`hack agent prime`. See [model-policy ownership](reference/agent-client-capabilities.md#model-policy-ownership). ## Sources and installed surfaces @@ -43,6 +47,8 @@ belong to their client adapters; shared behavior belongs to the canonical source 4. Run instruction-source, onboarding, hack-init-skill, shared-agent-skill, and plugin generation tests. Run normal typecheck/check/test gates for behavior changes. New packaging needs fresh client loading proof beyond a successful marketplace listing. + Record that proof, or the open gate, in `src/agents/client-capabilities.ts` and + regenerate the [capability matrix](reference/agent-client-capabilities.md). 5. Release preparation regenerates plugin assets after the version bump. Verify the published bundle and installed executable separately. diff --git a/docs/integrations.md b/docs/integrations.md index 90a7c5997..b9dc0f486 100644 --- a/docs/integrations.md +++ b/docs/integrations.md @@ -61,8 +61,11 @@ trust and approval rules. ## Optional native plugins Hack also ships an optional native plugin for Codex, Claude Code, and Cursor in -`plugins/hack`. It bundles the `hack-cli` and `hack-init` skills, the `hack mcp serve` -server, Claude primer hooks, and a Cursor rule that the agent can select when relevant. +`plugins/hack`. It bundles four skills (`hack-cli`, `hack-init`, `hack-install`, and +`hack-remote`), the `hack mcp serve` server, Claude primer hooks, and a Cursor rule +that the agent can select when relevant. The +[agent client capability matrix](reference/agent-client-capabilities.md) records +which of these components have been observed loading in each client. Install the Hack CLI first and make `hack` available on the client's PATH. Choose either the native plugin or the existing standalone setup for each client. @@ -95,7 +98,7 @@ outside its local-plugin root; use a copy rather than weakening that boundary. S [Cursor's plugin documentation](https://cursor.com/docs/plugins) for marketplace and local development support in the installed client version. -After installing, verify in a fresh session that both skills are discoverable, +After installing, verify in a fresh session that all four skills are discoverable, `hack mcp serve` can initialize, and the client's hooks or rule load. A successful marketplace registration alone does not prove those components loaded. The plugin's MCP fallback is for clients without shell access; use the Hack CLI when shell is available. diff --git a/docs/reference/agent-client-capabilities.md b/docs/reference/agent-client-capabilities.md new file mode 100644 index 000000000..638c81dbd --- /dev/null +++ b/docs/reference/agent-client-capabilities.md @@ -0,0 +1,143 @@ +# Agent client capability matrix + +This page classifies what Hack's optional agent integrations are known to do in +each client. The data lives in `src/agents/client-capabilities.ts`. +`tests/agent-client-capabilities.test.ts` checks the generated section below +against that data and enforces the classification rules. + +## Classification rules + +- **verified**: a named test file observed the client load the component, at + `isolated-fixture` or `authenticated-session` strength. +- **untested**: Hack ships the component, but no fixture or session has observed + a client load it at the recorded version. Each untested cell names its open gate. +- **unsupported**: Hack does not ship the component for that client. This says + nothing about what the client itself supports. + +Evidence classes, weakest first: + +| Class | Meaning | +| --- | --- | +| `source-read` | Classified by reading shipped files or client documentation. | +| `schema` | A test parses or generates the shipped file; no client loads it. | +| `isolated-fixture` | A test runs the real client with an isolated home and configuration. | +| `authenticated-session` | A signed-in client session used the component. | + +A manifest parse, a schema check or a marketplace listing is never evidence that a +component loaded. Each cell records the client version, a host label and the source +commit at which it was classified. That is one observation on one host. It does not +cover other machines or later client versions. A status for one client implies +nothing about another client, including UI parity. + +To reclassify a cell, update the data, increment +`AGENT_CLIENT_CAPABILITY_MATRIX_VERSION`, and regenerate the section below so the +test passes. Promote a cell to `verified` only after its named test passes against +the recorded client version. + + + +Matrix version 1, observed 2026-10-11. + +### Clients + +| Client | Manifest | +| --- | --- | +| Codex | `plugins/hack/.codex-plugin/plugin.json` | +| Claude Code | `plugins/hack/.claude-plugin/plugin.json` | +| Cursor | `plugins/hack/.cursor-plugin/plugin.json` | +| Standalone (portable manifest or generic stdio MCP client) | `plugins/hack/plugin.json` | + +### Capabilities + +| Capability | Meaning | +| --- | --- | +| Packaging | The client accepts, installs and enables the Hack plugin package in a session. | +| Skill discovery | A fresh session discovers the plugin's bundled skills (hack-cli, hack-init, hack-install, hack-remote). | +| Hooks | The client runs plugin-shipped hooks; Hack hooks may only run `hack agent prime`. | +| Plugin MCP | The client starts the plugin-declared `hack mcp serve` server and lists its tools. | +| Standalone MCP | The client loads an MCP entry written by `hack mcp install` (stdio or a selected MCP bundle) without the plugin. | +| Rules | The client loads a plugin-shipped rule file. | +| UI | The client renders plugin presentation metadata (display name, default prompts, brand color). | + +### Status summary + +| Capability | Codex | Claude Code | Cursor | Standalone (portable manifest or generic stdio MCP client) | +| --- | --- | --- | --- | --- | +| Packaging | untested | untested | untested | untested | +| Skill discovery | untested | untested | untested | untested | +| Hooks | unsupported | untested | unsupported | unsupported | +| Plugin MCP | untested | untested | untested | untested | +| Standalone MCP | untested | untested | untested | untested | +| Rules | unsupported | unsupported | untested | unsupported | +| UI | untested | unsupported | unsupported | unsupported | + +### Cells + +| Client | Capability | Status | Evidence class | Evidence | Client version | Host | Source | Note | Open gate | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| codex | Packaging | untested | schema | `plugins/hack/.codex-plugin/plugin.json` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest is generated and parsed by tests/agent-plugin-generation.test.ts; no client installed it. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| codex | Skill discovery | untested | schema | `plugins/hack/.codex-plugin/plugin.json` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest declares `skills: ./skills/`; skill content matches the canonical renderers. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| codex | Hooks | unsupported | source-read | `plugins/hack/.codex-plugin/plugin.json` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | Hack ships no Codex plugin hooks; the manifest declares no `hooks` key. This says nothing about Codex's own hook support. | none | +| codex | Plugin MCP | untested | schema | `plugins/hack/.mcp.json` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest points `mcpServers` at `.mcp.json` (`hack mcp serve`); no plugin session started it. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| codex | Standalone MCP | untested | isolated-fixture | `tests/mcp-bundle-client.test.ts` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | In the opt-in fixture, `codex mcp get hack --json` resolved the generated bundle adapter from an isolated CODEX_HOME. The fixture then stopped at the Claude step, so no tool call was observed in that run. Configuration discovery alone does not prove a Codex model session loaded the server or accepted project trust. | Pass tests/mcp-bundle-client.test.ts end to end at this client version; a Codex model session tool call remains a separate authenticated-session gate. | +| codex | Rules | unsupported | source-read | `plugins/hack/.codex-plugin/plugin.json` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | Hack ships no Codex plugin rule file. Project AGENTS.md snippets are standalone setup, not a plugin component. | none | +| codex | UI | untested | schema | `plugins/hack/.codex-plugin/plugin.json` | codex-cli 0.162.1 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest declares an `interface` block (display name, default prompts, brand color); rendering was not observed. | Observe the installed plugin's presentation in the Codex UI for this client version. | +| claude | Packaging | untested | schema | `plugins/hack/.claude-plugin/plugin.json` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest is generated and parsed by tests/agent-plugin-generation.test.ts; no client installed it. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| claude | Skill discovery | untested | schema | `plugins/hack/.claude-plugin/plugin.json` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest declares `skills: ./skills/`; skill content matches the canonical renderers. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| claude | Hooks | untested | schema | `plugins/hack/hooks/claude-hooks.json` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | SessionStart and PreCompact hooks run only `hack agent prime`; the generation test enforces that command allowlist. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| claude | Plugin MCP | untested | schema | `plugins/hack/.mcp.json` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest points `mcpServers` at `.mcp.json` (`hack mcp serve`); no plugin session started it. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| claude | Standalone MCP | untested | isolated-fixture | `tests/mcp-bundle-client.test.ts` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | The opt-in fixture failed at this version: in an isolated HOME, `claude mcp get hack` reports the project-scoped server as pending approval instead of connected, so no connection or tool call was observed. Project approval is a native trust gate and must not be bypassed. | Qualify Claude standalone MCP without bypassing project approval (for example, user scope in an isolated HOME) and pass the fixture end to end. | +| claude | Rules | unsupported | source-read | `plugins/hack/.claude-plugin/plugin.json` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | Hack ships no Claude plugin rule file. Project CLAUDE.md snippets are standalone setup, not a plugin component. | none | +| claude | UI | unsupported | source-read | `plugins/hack/.claude-plugin/plugin.json` | Claude Code 2.1.296 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest declares no presentation metadata. Codex UI metadata implies nothing for Claude Code. | none | +| cursor | Packaging | untested | schema | `plugins/hack/.cursor-plugin/plugin.json` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest is generated and parsed by tests/agent-plugin-generation.test.ts. docs/integrations.md records an earlier manual symlink-rejection observation (Cursor 3.18.25), not a fixture. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| cursor | Skill discovery | untested | schema | `plugins/hack/.cursor-plugin/plugin.json` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest declares `skills: ./skills/`; skill content matches the canonical renderers. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| cursor | Hooks | unsupported | source-read | `plugins/hack/.cursor-plugin/plugin.json` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Hack ships no Cursor plugin hooks; the manifest declares no `hooks` key. | none | +| cursor | Plugin MCP | untested | schema | `plugins/hack/.mcp.json` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest points `mcpServers` at `.mcp.json` (`hack mcp serve`); no plugin session started it. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| cursor | Standalone MCP | untested | schema | `tests/mcp-install.test.ts` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Tests check the `.cursor/mcp.json` entry Hack writes; Cursor itself never loaded it in a fixture. | Add an isolated Cursor fixture that loads the written entry and lists Hack tools. | +| cursor | Rules | untested | schema | `plugins/hack/rules/hack.mdc` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Agent-selectable rule (`alwaysApply: false`) matches the canonical renderer; loading was not observed. | Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count. | +| cursor | UI | unsupported | source-read | `plugins/hack/.cursor-plugin/plugin.json` | Cursor 3.24.12 | `macos-arm64-contributor` | `ec56769bdc04` | Manifest declares no presentation metadata. Codex UI metadata implies nothing for Cursor. | none | +| standalone | Packaging | untested | schema | `plugins/hack/plugin.json` | none | `macos-arm64-contributor` | `ec56769bdc04` | Portable agent-plugins manifest is derived from the Codex manifest; no portable consumer was observed. | Name a portable consumer and observe it install the package. | +| standalone | Skill discovery | untested | source-read | `plugins/hack/plugin.json` | none | `macos-arm64-contributor` | `ec56769bdc04` | The portable manifest declares no component paths; discovery of `skills/` by convention is unverified. | Name a portable consumer and observe skill discovery in a fresh session. | +| standalone | Hooks | unsupported | source-read | `plugins/hack/plugin.json` | none | `macos-arm64-contributor` | `ec56769bdc04` | The portable manifest carries hooks only through the OpenAI extension, and Hack ships none there. | none | +| standalone | Plugin MCP | untested | schema | `plugins/hack/mcp.json` | none | `macos-arm64-contributor` | `ec56769bdc04` | Portable MCP manifest adds an explicit stdio transport to the same launch as `.mcp.json`. | Name a portable consumer and observe it start the declared server. | +| standalone | Standalone MCP | untested | isolated-fixture | `tests/mcp-bundle-client.test.ts` | none | `macos-arm64-contributor` | `ec56769bdc04` | The fixture's reference MCP SDK client calls a real tool through the generated entry, but that step was not reached in the recorded run. The SDK client is not an agent. | Pass tests/mcp-bundle-client.test.ts end to end, or add a client-independent stdio fixture. | +| standalone | Rules | unsupported | source-read | `plugins/hack/plugin.json` | none | `macos-arm64-contributor` | `ec56769bdc04` | No portable rule component is shipped. | none | +| standalone | UI | unsupported | source-read | `plugins/hack/plugin.json` | none | `macos-arm64-contributor` | `ec56769bdc04` | Presentation metadata exists only in the OpenAI extension; no portable UI is claimed. | none | + + + +## Companion-package layout decision + +Hack keeps three separate companion layouts. Each layout has one authoritative +owner, and none of them replaces the others. Users choose one per client. + +| Layout | What it contains | Authoritative owner | Installed and updated by | +| --- | --- | --- | --- | +| Native plugin (`plugins/hack`) | Client manifests, four skills, Claude primer hooks, Cursor rule, `hack mcp serve` declaration | `scripts/generate-agent-plugins.ts`, rendering from `src/agents/instruction-source.ts` and the skill renderers in `src/agents/` | The client's plugin manager. Hack commands never install or update it. | +| Standalone setup | Project AGENTS.md/CLAUDE.md snippets, Cursor rules, shared skills, Claude primer hooks, MCP entries | `src/agents/instruction-source.ts` and its renderers, written by `hack setup` and `hack mcp install` | Explicit `hack setup` and `hack mcp install` commands in a chosen scope | +| MCP bundle | Native MCP adapter, owner and backend executables with a content-derived identity | `src/mcp/bundle.ts` (packaging and verification) | `hack mcp install --bundle ... --cli ...` writes the client entry; the bundle is never modified after selection | + +Decisions: + +- Shared guidance has one canonical source, `src/agents/instruction-source.ts`. The + plugin and the standalone setup are renderings of it, not separate sources. +- Installing the plugin does not migrate or remove standalone guidance. See + [integration setup](../integrations.md). +- The MCP bundle is an alternative launch for MCP only. It does not carry skills, + hooks or rules. +- This page records the current layout. Layout changes are separate work and must + update this matrix in the same change. + +## Model-policy ownership + +The user's own provider configuration and project policy own these settings. Hack +artifacts must not set them: + +- model selection, model overrides and reasoning effort +- permission modes, tool allowlists, approval policies and sandbox modes +- output styles + +Hack-shipped hooks may run only `hack agent prime`. +`tests/agent-plugin-generation.test.ts` enforces this for every file under +`plugins/hack` and `examples/basic`. It includes a negative control that proves the +check rejects each forbidden setting. See [maintaining agent guidance](../agent-guidance.md). diff --git a/src/agents/client-capabilities.ts b/src/agents/client-capabilities.ts new file mode 100644 index 000000000..288b8da81 --- /dev/null +++ b/src/agents/client-capabilities.ts @@ -0,0 +1,517 @@ +/** + * Versioned agent-client capability matrix for Hack's optional agent + * integrations. This module is data only. + * + * Invariants, enforced by tests/agent-client-capabilities.test.ts: + * - Every client x capability pair has exactly one cell. + * - A `verified` cell names an existing test file whose evidence class is + * `isolated-fixture` or stronger. A manifest parse, schema check or + * marketplace listing is never component-loading evidence. + * - For the named native-plugin clients, a component is `unsupported` exactly + * when that client's plugin manifest does not declare it. + * - docs/reference/agent-client-capabilities.md matches this data. + * + * Each cell records the client version, host label and source revision at + * which it was classified. Those values describe one observation on one host; + * they are not a cross-machine or long-term compatibility claim, and a status + * for one client implies nothing about another client's UI or loading behavior. + */ + +/** Increment when cells are reclassified or the cell shape changes. */ +export const AGENT_CLIENT_CAPABILITY_MATRIX_VERSION = 1; + +/** Date (UTC) of the observations recorded below. */ +export const AGENT_CLIENT_CAPABILITY_OBSERVED_ON = "2026-10-11"; + +export const AGENT_CLIENTS = [ + "codex", + "claude", + "cursor", + "standalone", +] as const; +export type AgentClient = (typeof AGENT_CLIENTS)[number]; + +export const AGENT_CLIENT_CAPABILITIES = [ + "packaging", + "skillDiscovery", + "hooks", + "pluginMcp", + "standaloneMcp", + "rules", + "ui", +] as const; +export type AgentClientCapability = (typeof AGENT_CLIENT_CAPABILITIES)[number]; + +export type CapabilityStatus = "verified" | "untested" | "unsupported"; + +/** + * Evidence classes, weakest first. `source-read` and `schema` describe files + * Hack ships; only `isolated-fixture` and `authenticated-session` observe a + * client loading a component. + */ +export const EVIDENCE_CLASSES = [ + "source-read", + "schema", + "isolated-fixture", + "authenticated-session", +] as const; +export type EvidenceClass = (typeof EVIDENCE_CLASSES)[number]; + +export interface AgentClientDefinition { + readonly client: AgentClient; + readonly label: string; + /** Repository-relative plugin manifest Hack ships for this client. */ + readonly manifestPath: string; +} + +export interface AgentClientCapabilityDefinition { + readonly capability: AgentClientCapability; + readonly label: string; + readonly meaning: string; +} + +export interface ClientCapabilityCell { + readonly client: AgentClient; + readonly capability: AgentClientCapability; + readonly status: CapabilityStatus; + readonly evidenceClass: EvidenceClass; + /** Repository-relative file that carries the evidence. */ + readonly evidenceRef: string; + /** Installed client version observed during classification, if any. */ + readonly observedClientVersion: string | null; + /** Commit of the Hack source that was classified. */ + readonly sourceSha: string; + /** Non-identifying label of the host that produced the observation. */ + readonly hostLabel: string; + readonly note: string; + /** Proof still required before the cell can be `verified`; null if none. */ + readonly openGate: string | null; +} + +export const AGENT_CLIENT_DEFINITIONS: readonly AgentClientDefinition[] = [ + { + client: "codex", + label: "Codex", + manifestPath: "plugins/hack/.codex-plugin/plugin.json", + }, + { + client: "claude", + label: "Claude Code", + manifestPath: "plugins/hack/.claude-plugin/plugin.json", + }, + { + client: "cursor", + label: "Cursor", + manifestPath: "plugins/hack/.cursor-plugin/plugin.json", + }, + { + client: "standalone", + label: "Standalone (portable manifest or generic stdio MCP client)", + manifestPath: "plugins/hack/plugin.json", + }, +]; + +export const AGENT_CLIENT_CAPABILITY_DEFINITIONS: readonly AgentClientCapabilityDefinition[] = + [ + { + capability: "packaging", + label: "Packaging", + meaning: + "The client accepts, installs and enables the Hack plugin package in a session.", + }, + { + capability: "skillDiscovery", + label: "Skill discovery", + meaning: + "A fresh session discovers the plugin's bundled skills (hack-cli, hack-init, hack-install, hack-remote).", + }, + { + capability: "hooks", + label: "Hooks", + meaning: + "The client runs plugin-shipped hooks; Hack hooks may only run `hack agent prime`.", + }, + { + capability: "pluginMcp", + label: "Plugin MCP", + meaning: + "The client starts the plugin-declared `hack mcp serve` server and lists its tools.", + }, + { + capability: "standaloneMcp", + label: "Standalone MCP", + meaning: + "The client loads an MCP entry written by `hack mcp install` (stdio or a selected MCP bundle) without the plugin.", + }, + { + capability: "rules", + label: "Rules", + meaning: "The client loads a plugin-shipped rule file.", + }, + { + capability: "ui", + label: "UI", + meaning: + "The client renders plugin presentation metadata (display name, default prompts, brand color).", + }, + ]; + +const SOURCE_SHA = "ec56769bdc046f5b64e621eb04613cfb00a65ced"; +const HOST = "macos-arm64-contributor"; +const CODEX_VERSION = "codex-cli 0.162.1"; +const CLAUDE_VERSION = "Claude Code 2.1.296"; +const CURSOR_VERSION = "Cursor 3.24.12"; +const BUNDLE_FIXTURE = "tests/mcp-bundle-client.test.ts"; + +const PLUGIN_LOAD_GATE = + "Install the plugin into an isolated client home, start a fresh session and observe the component load; a marketplace listing or manifest parse does not count."; + +export const AGENT_CLIENT_CAPABILITY_MATRIX: readonly ClientCapabilityCell[] = [ + // Codex + { + client: "codex", + capability: "packaging", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.codex-plugin/plugin.json", + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest is generated and parsed by tests/agent-plugin-generation.test.ts; no client installed it.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "codex", + capability: "skillDiscovery", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.codex-plugin/plugin.json", + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest declares `skills: ./skills/`; skill content matches the canonical renderers.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "codex", + capability: "hooks", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/.codex-plugin/plugin.json", + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Hack ships no Codex plugin hooks; the manifest declares no `hooks` key. This says nothing about Codex's own hook support.", + openGate: null, + }, + { + client: "codex", + capability: "pluginMcp", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.mcp.json", + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest points `mcpServers` at `.mcp.json` (`hack mcp serve`); no plugin session started it.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "codex", + capability: "standaloneMcp", + status: "untested", + evidenceClass: "isolated-fixture", + evidenceRef: BUNDLE_FIXTURE, + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "In the opt-in fixture, `codex mcp get hack --json` resolved the generated bundle adapter from an isolated CODEX_HOME. The fixture then stopped at the Claude step, so no tool call was observed in that run. Configuration discovery alone does not prove a Codex model session loaded the server or accepted project trust.", + openGate: + "Pass tests/mcp-bundle-client.test.ts end to end at this client version; a Codex model session tool call remains a separate authenticated-session gate.", + }, + { + client: "codex", + capability: "rules", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/.codex-plugin/plugin.json", + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Hack ships no Codex plugin rule file. Project AGENTS.md snippets are standalone setup, not a plugin component.", + openGate: null, + }, + { + client: "codex", + capability: "ui", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.codex-plugin/plugin.json", + observedClientVersion: CODEX_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest declares an `interface` block (display name, default prompts, brand color); rendering was not observed.", + openGate: + "Observe the installed plugin's presentation in the Codex UI for this client version.", + }, + // Claude Code + { + client: "claude", + capability: "packaging", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.claude-plugin/plugin.json", + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest is generated and parsed by tests/agent-plugin-generation.test.ts; no client installed it.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "claude", + capability: "skillDiscovery", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.claude-plugin/plugin.json", + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest declares `skills: ./skills/`; skill content matches the canonical renderers.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "claude", + capability: "hooks", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/hooks/claude-hooks.json", + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "SessionStart and PreCompact hooks run only `hack agent prime`; the generation test enforces that command allowlist.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "claude", + capability: "pluginMcp", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.mcp.json", + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest points `mcpServers` at `.mcp.json` (`hack mcp serve`); no plugin session started it.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "claude", + capability: "standaloneMcp", + status: "untested", + evidenceClass: "isolated-fixture", + evidenceRef: BUNDLE_FIXTURE, + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "The opt-in fixture failed at this version: in an isolated HOME, `claude mcp get hack` reports the project-scoped server as pending approval instead of connected, so no connection or tool call was observed. Project approval is a native trust gate and must not be bypassed.", + openGate: + "Qualify Claude standalone MCP without bypassing project approval (for example, user scope in an isolated HOME) and pass the fixture end to end.", + }, + { + client: "claude", + capability: "rules", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/.claude-plugin/plugin.json", + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Hack ships no Claude plugin rule file. Project CLAUDE.md snippets are standalone setup, not a plugin component.", + openGate: null, + }, + { + client: "claude", + capability: "ui", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/.claude-plugin/plugin.json", + observedClientVersion: CLAUDE_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest declares no presentation metadata. Codex UI metadata implies nothing for Claude Code.", + openGate: null, + }, + // Cursor + { + client: "cursor", + capability: "packaging", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.cursor-plugin/plugin.json", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest is generated and parsed by tests/agent-plugin-generation.test.ts. docs/integrations.md records an earlier manual symlink-rejection observation (Cursor 3.18.25), not a fixture.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "cursor", + capability: "skillDiscovery", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.cursor-plugin/plugin.json", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest declares `skills: ./skills/`; skill content matches the canonical renderers.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "cursor", + capability: "hooks", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/.cursor-plugin/plugin.json", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Hack ships no Cursor plugin hooks; the manifest declares no `hooks` key.", + openGate: null, + }, + { + client: "cursor", + capability: "pluginMcp", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/.mcp.json", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest points `mcpServers` at `.mcp.json` (`hack mcp serve`); no plugin session started it.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "cursor", + capability: "standaloneMcp", + status: "untested", + evidenceClass: "schema", + evidenceRef: "tests/mcp-install.test.ts", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Tests check the `.cursor/mcp.json` entry Hack writes; Cursor itself never loaded it in a fixture.", + openGate: + "Add an isolated Cursor fixture that loads the written entry and lists Hack tools.", + }, + { + client: "cursor", + capability: "rules", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/rules/hack.mdc", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Agent-selectable rule (`alwaysApply: false`) matches the canonical renderer; loading was not observed.", + openGate: PLUGIN_LOAD_GATE, + }, + { + client: "cursor", + capability: "ui", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/.cursor-plugin/plugin.json", + observedClientVersion: CURSOR_VERSION, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Manifest declares no presentation metadata. Codex UI metadata implies nothing for Cursor.", + openGate: null, + }, + // Standalone: no named client; the portable manifest or a generic stdio MCP client. + { + client: "standalone", + capability: "packaging", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/plugin.json", + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Portable agent-plugins manifest is derived from the Codex manifest; no portable consumer was observed.", + openGate: "Name a portable consumer and observe it install the package.", + }, + { + client: "standalone", + capability: "skillDiscovery", + status: "untested", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/plugin.json", + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "The portable manifest declares no component paths; discovery of `skills/` by convention is unverified.", + openGate: + "Name a portable consumer and observe skill discovery in a fresh session.", + }, + { + client: "standalone", + capability: "hooks", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/plugin.json", + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "The portable manifest carries hooks only through the OpenAI extension, and Hack ships none there.", + openGate: null, + }, + { + client: "standalone", + capability: "pluginMcp", + status: "untested", + evidenceClass: "schema", + evidenceRef: "plugins/hack/mcp.json", + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Portable MCP manifest adds an explicit stdio transport to the same launch as `.mcp.json`.", + openGate: + "Name a portable consumer and observe it start the declared server.", + }, + { + client: "standalone", + capability: "standaloneMcp", + status: "untested", + evidenceClass: "isolated-fixture", + evidenceRef: BUNDLE_FIXTURE, + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "The fixture's reference MCP SDK client calls a real tool through the generated entry, but that step was not reached in the recorded run. The SDK client is not an agent.", + openGate: + "Pass tests/mcp-bundle-client.test.ts end to end, or add a client-independent stdio fixture.", + }, + { + client: "standalone", + capability: "rules", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/plugin.json", + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "No portable rule component is shipped.", + openGate: null, + }, + { + client: "standalone", + capability: "ui", + status: "unsupported", + evidenceClass: "source-read", + evidenceRef: "plugins/hack/plugin.json", + observedClientVersion: null, + sourceSha: SOURCE_SHA, + hostLabel: HOST, + note: "Presentation metadata exists only in the OpenAI extension; no portable UI is claimed.", + openGate: null, + }, +]; diff --git a/tests/agent-client-capabilities.test.ts b/tests/agent-client-capabilities.test.ts new file mode 100644 index 000000000..4c5691c3e --- /dev/null +++ b/tests/agent-client-capabilities.test.ts @@ -0,0 +1,465 @@ +import { expect, test } from "bun:test"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; + +import { + AGENT_CLIENT_CAPABILITIES, + AGENT_CLIENT_CAPABILITY_DEFINITIONS, + AGENT_CLIENT_CAPABILITY_MATRIX, + AGENT_CLIENT_CAPABILITY_MATRIX_VERSION, + AGENT_CLIENT_CAPABILITY_OBSERVED_ON, + AGENT_CLIENT_DEFINITIONS, + AGENT_CLIENTS, + type AgentClient, + type AgentClientCapability, + type ClientCapabilityCell, + EVIDENCE_CLASSES, + type EvidenceClass, +} from "../src/agents/client-capabilities.ts"; +import { isRecord } from "../src/lib/guards.ts"; + +const repoRoot = join(import.meta.dir, ".."); +const referencePath = join( + repoRoot, + "docs", + "reference", + "agent-client-capabilities.md" +); + +type IssueCode = + | "missing-cell" + | "duplicate-cell" + | "missing-evidence-ref" + | "invalid-source-sha" + | "missing-host-label" + | "missing-client-version" + | "open-gate-missing" + | "verified-without-test" + | "verified-below-fixture" + | "verified-on-manifest-or-listing" + | "verified-with-open-gate" + | "manifest-disagreement"; + +interface ClassificationIssue { + readonly client: AgentClient; + readonly capability: AgentClientCapability; + readonly code: IssueCode; +} + +/** Manifest key that declares each plugin component for the named clients. */ +const MANIFEST_COMPONENT_KEYS: Partial> = + { + skillDiscovery: "skills", + hooks: "hooks", + pluginMcp: "mcpServers", + rules: "rules", + ui: "interface", + }; + +const NATIVE_PLUGIN_CLIENTS: readonly AgentClient[] = [ + "codex", + "claude", + "cursor", +]; + +function evidenceStrength(evidenceClass: EvidenceClass): number { + return EVIDENCE_CLASSES.indexOf(evidenceClass); +} + +/** Manifests, shipped component files and marketplace listings describe a + * package; they never show that a client loaded a component. */ +function isManifestOrListingRef(ref: string): boolean { + const name = ref.split("/").at(-1) ?? ref; + return ( + ref.startsWith("plugins/") || + ref.includes("marketplace") || + ["plugin.json", "mcp.json", ".mcp.json", "marketplace.json"].includes( + name + ) || + name.endsWith(".mdc") || + name === "SKILL.md" + ); +} + +function classificationIssues({ + cells, + manifests, + exists, +}: { + readonly cells: readonly ClientCapabilityCell[]; + readonly manifests: Readonly>>; + readonly exists: (repoRelativePath: string) => boolean; +}): ClassificationIssue[] { + const issues: ClassificationIssue[] = []; + for (const client of AGENT_CLIENTS) { + for (const capability of AGENT_CLIENT_CAPABILITIES) { + const matches = cells.filter( + (cell) => cell.client === client && cell.capability === capability + ); + if (matches.length === 0) { + issues.push({ client, capability, code: "missing-cell" }); + } else if (matches.length > 1) { + issues.push({ client, capability, code: "duplicate-cell" }); + } + } + } + for (const cell of cells) { + const report = (code: IssueCode) => + issues.push({ client: cell.client, capability: cell.capability, code }); + if (!(cell.evidenceRef && exists(cell.evidenceRef))) { + report("missing-evidence-ref"); + } + if (!/^[0-9a-f]{40}$/.test(cell.sourceSha)) { + report("invalid-source-sha"); + } + if (cell.hostLabel.trim() === "") { + report("missing-host-label"); + } + if (cell.client !== "standalone" && !cell.observedClientVersion) { + report("missing-client-version"); + } + if (cell.status === "untested" && !cell.openGate?.trim()) { + report("open-gate-missing"); + } + if (cell.status === "verified") { + if ( + !( + /^tests\/.+\.test\.ts$/.test(cell.evidenceRef) && + exists(cell.evidenceRef) + ) + ) { + report("verified-without-test"); + } + if ( + evidenceStrength(cell.evidenceClass) < + evidenceStrength("isolated-fixture") + ) { + report("verified-below-fixture"); + } + if (isManifestOrListingRef(cell.evidenceRef)) { + report("verified-on-manifest-or-listing"); + } + if (cell.openGate !== null) { + report("verified-with-open-gate"); + } + if (!cell.observedClientVersion) { + report("missing-client-version"); + } + } + if (NATIVE_PLUGIN_CLIENTS.includes(cell.client)) { + const manifest = manifests[cell.client] ?? {}; + const key = MANIFEST_COMPONENT_KEYS[cell.capability]; + const declared = + cell.capability === "packaging" + ? Object.keys(manifest).length > 0 + : key !== undefined && key in manifest; + if ( + (key !== undefined || cell.capability === "packaging") && + (cell.status === "unsupported") === declared + ) { + report("manifest-disagreement"); + } + } + } + return issues; +} + +async function loadManifests(): Promise< + Record> +> { + const manifests: Record> = {}; + for (const definition of AGENT_CLIENT_DEFINITIONS) { + const parsed: unknown = await Bun.file( + join(repoRoot, definition.manifestPath) + ).json(); + if (!isRecord(parsed)) { + throw new Error(`Expected a manifest object: ${definition.manifestPath}`); + } + manifests[definition.client] = parsed; + } + return manifests; +} + +const repoExists = (path: string) => existsSync(join(repoRoot, path)); + +function replaceCell( + client: AgentClient, + capability: AgentClientCapability, + change: Partial +): ClientCapabilityCell[] { + return AGENT_CLIENT_CAPABILITY_MATRIX.map((cell) => + cell.client === client && cell.capability === capability + ? { ...cell, ...change } + : cell + ); +} + +function codesFor( + issues: readonly ClassificationIssue[], + client: AgentClient, + capability: AgentClientCapability +): IssueCode[] { + return issues + .filter( + (issue) => issue.client === client && issue.capability === capability + ) + .map((issue) => issue.code); +} + +function escapeCell(value: string): string { + return value.replaceAll("|", "\\|").replaceAll("\n", " "); +} + +function clientLabel(client: AgentClient): string { + return ( + AGENT_CLIENT_DEFINITIONS.find((definition) => definition.client === client) + ?.label ?? client + ); +} + +function capabilityLabel(capability: AgentClientCapability): string { + return ( + AGENT_CLIENT_CAPABILITY_DEFINITIONS.find( + (definition) => definition.capability === capability + )?.label ?? capability + ); +} + +const GENERATED_BEGIN = ""; +const GENERATED_END = ""; + +function renderGeneratedSection(): string { + const lines = [ + GENERATED_BEGIN, + "", + `Matrix version ${AGENT_CLIENT_CAPABILITY_MATRIX_VERSION}, observed ${AGENT_CLIENT_CAPABILITY_OBSERVED_ON}.`, + "", + "### Clients", + "", + "| Client | Manifest |", + "| --- | --- |", + ...AGENT_CLIENT_DEFINITIONS.map( + (definition) => + `| ${escapeCell(definition.label)} | \`${definition.manifestPath}\` |` + ), + "", + "### Capabilities", + "", + "| Capability | Meaning |", + "| --- | --- |", + ...AGENT_CLIENT_CAPABILITY_DEFINITIONS.map( + (definition) => + `| ${definition.label} | ${escapeCell(definition.meaning)} |` + ), + "", + "### Status summary", + "", + `| Capability | ${AGENT_CLIENTS.map(clientLabel).map(escapeCell).join(" | ")} |`, + `| --- | ${AGENT_CLIENTS.map(() => "---").join(" | ")} |`, + ...AGENT_CLIENT_CAPABILITIES.map((capability) => { + const statuses = AGENT_CLIENTS.map( + (client) => + AGENT_CLIENT_CAPABILITY_MATRIX.find( + (cell) => cell.client === client && cell.capability === capability + )?.status ?? "missing" + ); + return `| ${capabilityLabel(capability)} | ${statuses.join(" | ")} |`; + }), + "", + "### Cells", + "", + "| Client | Capability | Status | Evidence class | Evidence | Client version | Host | Source | Note | Open gate |", + "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |", + ...AGENT_CLIENT_CAPABILITY_MATRIX.map((cell) => + [ + "", + cell.client, + capabilityLabel(cell.capability), + cell.status, + cell.evidenceClass, + `\`${cell.evidenceRef}\``, + escapeCell(cell.observedClientVersion ?? "none"), + `\`${cell.hostLabel}\``, + `\`${cell.sourceSha.slice(0, 12)}\``, + escapeCell(cell.note), + escapeCell(cell.openGate ?? "none"), + "", + ] + .join(" | ") + .trim() + ), + "", + GENERATED_END, + ]; + return lines.join("\n"); +} + +test("every client and capability pair is classified with complete evidence", async () => { + expect(AGENT_CLIENT_CAPABILITY_MATRIX).toHaveLength( + AGENT_CLIENTS.length * AGENT_CLIENT_CAPABILITIES.length + ); + const issues = classificationIssues({ + cells: AGENT_CLIENT_CAPABILITY_MATRIX, + manifests: await loadManifests(), + exists: repoExists, + }); + expect(issues).toEqual([]); +}); + +test("verified cells require an existing fixture-strength test file", async () => { + const manifests = await loadManifests(); + const accepted = replaceCell("codex", "standaloneMcp", { + status: "verified", + evidenceClass: "isolated-fixture", + evidenceRef: "tests/mcp-bundle-client.test.ts", + openGate: null, + }); + expect( + classificationIssues({ cells: accepted, manifests, exists: repoExists }) + ).toEqual([]); + + const missingTest = replaceCell("codex", "standaloneMcp", { + status: "verified", + evidenceClass: "isolated-fixture", + evidenceRef: "tests/does-not-exist.test.ts", + openGate: null, + }); + expect( + codesFor( + classificationIssues({ + cells: missingTest, + manifests, + exists: repoExists, + }), + "codex", + "standaloneMcp" + ) + ).toEqual(["missing-evidence-ref", "verified-without-test"]); + + const weakClass = replaceCell("codex", "standaloneMcp", { + status: "verified", + evidenceClass: "schema", + evidenceRef: "tests/mcp-install.test.ts", + openGate: null, + }); + expect( + codesFor( + classificationIssues({ cells: weakClass, manifests, exists: repoExists }), + "codex", + "standaloneMcp" + ) + ).toEqual(["verified-below-fixture"]); +}); + +test("negative control: plugin skills verified on manifest-only evidence fail", async () => { + const manifests = await loadManifests(); + const cells = replaceCell("claude", "skillDiscovery", { + status: "verified", + evidenceClass: "isolated-fixture", + evidenceRef: "plugins/hack/.claude-plugin/plugin.json", + openGate: null, + }); + expect( + codesFor( + classificationIssues({ cells, manifests, exists: repoExists }), + "claude", + "skillDiscovery" + ) + ).toEqual(["verified-without-test", "verified-on-manifest-or-listing"]); + + const listing = replaceCell("cursor", "packaging", { + status: "verified", + evidenceClass: "authenticated-session", + evidenceRef: ".claude-plugin/marketplace.json", + openGate: null, + }); + expect( + codesFor( + classificationIssues({ + cells: listing, + manifests, + exists: () => true, + }), + "cursor", + "packaging" + ) + ).toEqual(["verified-without-test", "verified-on-manifest-or-listing"]); +}); + +test("manifest-declared components agree with the matrix", async () => { + const manifests = await loadManifests(); + expect("hooks" in (manifests.codex ?? {})).toBe(false); + const codexHooks = replaceCell("codex", "hooks", { + status: "verified", + evidenceClass: "isolated-fixture", + evidenceRef: "tests/mcp-bundle-client.test.ts", + openGate: null, + }); + expect( + codesFor( + classificationIssues({ + cells: codexHooks, + manifests, + exists: repoExists, + }), + "codex", + "hooks" + ) + ).toEqual(["manifest-disagreement"]); + + const hiddenRules = replaceCell("cursor", "rules", { + status: "unsupported", + openGate: null, + }); + expect( + codesFor( + classificationIssues({ + cells: hiddenRules, + manifests, + exists: repoExists, + }), + "cursor", + "rules" + ) + ).toEqual(["manifest-disagreement"]); + + const missing = AGENT_CLIENT_CAPABILITY_MATRIX.filter( + (cell) => !(cell.client === "claude" && cell.capability === "ui") + ); + expect( + codesFor( + classificationIssues({ cells: missing, manifests, exists: repoExists }), + "claude", + "ui" + ) + ).toEqual(["missing-cell"]); + + const ungated = replaceCell("claude", "hooks", { openGate: null }); + expect( + codesFor( + classificationIssues({ cells: ungated, manifests, exists: repoExists }), + "claude", + "hooks" + ) + ).toEqual(["open-gate-missing"]); +}); + +test("reference doc matches the matrix and records ownership decisions", async () => { + const doc = await Bun.file(referencePath).text(); + const start = doc.indexOf(GENERATED_BEGIN); + const end = doc.indexOf(GENERATED_END); + expect(start).toBeGreaterThanOrEqual(0); + expect(end).toBeGreaterThan(start); + expect(doc.slice(start, end + GENERATED_END.length)).toBe( + renderGeneratedSection() + ); + expect(doc).toContain("## Companion-package layout decision"); + expect(doc).toContain("## Model-policy ownership"); + for (const owner of [ + "scripts/generate-agent-plugins.ts", + "src/agents/instruction-source.ts", + "src/mcp/bundle.ts", + ]) { + expect(doc).toContain(owner); + } +}); diff --git a/tests/agent-plugin-generation.test.ts b/tests/agent-plugin-generation.test.ts index 858a3c90c..38e4a30a1 100644 --- a/tests/agent-plugin-generation.test.ts +++ b/tests/agent-plugin-generation.test.ts @@ -1,5 +1,5 @@ import { afterEach, expect, test } from "bun:test"; -import { cp, mkdir, mkdtemp, rm, symlink } from "node:fs/promises"; +import { cp, mkdir, mkdtemp, readdir, rm, symlink } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -13,6 +13,7 @@ import { renderCodexSkill } from "../src/agents/codex-skill.ts"; import { renderHackInitSkill } from "../src/agents/hack-init-skill.ts"; import { renderHackInstallSkill } from "../src/agents/install-skill.ts"; import { renderHackRemoteSkill } from "../src/agents/remote-bootstrap-skill.ts"; +import { isRecord } from "../src/lib/guards.ts"; const repoRoot = join(import.meta.dir, ".."); @@ -217,3 +218,262 @@ test("release preparation updates both manifests and rendered guidance after bum expect(skill).toContain("hack CLI v9.8.7"); expect(skill).not.toContain("hack CLI v0.0.1"); }, 15_000); + +interface PolicyFinding { + readonly file: string; + readonly code: + | "model-setting" + | "permission-setting" + | "output-style-setting" + | "frontmatter-hooks" + | "hook-command"; +} + +const HOOK_COMMAND = "hack agent prime"; + +/** Model, permission and output-style policy belongs to the user's provider + * configuration, never to Hack-shipped artifacts. */ +function forbiddenSettingCode(key: string): PolicyFinding["code"] | null { + const normalized = key.toLowerCase().replace(/[^a-z]/g, ""); + if (normalized.startsWith("model") || normalized.endsWith("model")) { + return "model-setting"; + } + if ( + normalized.includes("permission") || + normalized.includes("approval") || + normalized.startsWith("sandbox") || + normalized === "allowedtools" || + normalized === "disallowedtools" || + normalized === "yolo" + ) { + return "permission-setting"; + } + if (normalized.includes("outputstyle")) { + return "output-style-setting"; + } + return null; +} + +/** Hook entries may only run the session primer as a plain command. */ +function hasDisallowedHook(value: unknown): boolean { + if (Array.isArray(value)) { + return value.some(hasDisallowedHook); + } + if (!isRecord(value)) { + return false; + } + if ( + ("type" in value || "command" in value) && + !(value.type === "command" && value.command === HOOK_COMMAND) + ) { + return true; + } + return Object.values(value).some(hasDisallowedHook); +} + +function structuredPolicyFindings( + file: string, + value: unknown +): PolicyFinding[] { + if (Array.isArray(value)) { + return value.flatMap((item) => structuredPolicyFindings(file, item)); + } + if (!isRecord(value)) { + return []; + } + const findings: PolicyFinding[] = []; + for (const [key, nested] of Object.entries(value)) { + const code = forbiddenSettingCode(key); + if (code) { + findings.push({ file, code }); + } + if (key === "hooks" && hasDisallowedHook(nested)) { + findings.push({ file, code: "hook-command" }); + } + findings.push(...structuredPolicyFindings(file, nested)); + } + return findings; +} + +function frontmatterKeys(text: string): string[] { + const match = /^---\n([\s\S]*?)\n---/.exec(text); + if (!match?.[1]) { + return []; + } + return match[1] + .split("\n") + .map((line) => /^([A-Za-z0-9_-]+):/.exec(line)?.[1]) + .filter((key): key is string => key !== undefined); +} + +/** Classify one shipped file with closed codes; never report file content. */ +function policyFindings(file: string, text: string): PolicyFinding[] { + const findings: PolicyFinding[] = []; + const name = file.split("/").at(-1) ?? file; + if (file.split("/").some((part) => /^output[-_]?styles?$/i.test(part))) { + findings.push({ file, code: "output-style-setting" }); + } + if (name.endsWith(".json")) { + findings.push(...structuredPolicyFindings(file, JSON.parse(text))); + } else if (name.endsWith(".jsonl")) { + for (const line of text.split("\n").filter((entry) => entry.trim())) { + findings.push(...structuredPolicyFindings(file, JSON.parse(line))); + } + } else if (name.endsWith(".toml")) { + findings.push(...structuredPolicyFindings(file, Bun.TOML.parse(text))); + } else if (name.endsWith(".md") || name.endsWith(".mdc")) { + for (const key of frontmatterKeys(text)) { + const code = forbiddenSettingCode(key); + if (code) { + findings.push({ file, code }); + } + if (key.toLowerCase() === "hooks") { + findings.push({ file, code: "frontmatter-hooks" }); + } + } + } + return findings; +} + +test("plugin bundle and examples never pin models, override permissions or run other hooks", async () => { + const findings: PolicyFinding[] = []; + let scanned = 0; + for (const root of ["plugins/hack", "examples/basic"]) { + const glob = new Bun.Glob("**/*"); + for await (const relative of glob.scan({ + cwd: join(repoRoot, root), + dot: true, + onlyFiles: true, + })) { + const file = `${root}/${relative}`; + findings.push( + ...policyFindings(file, await Bun.file(join(repoRoot, file)).text()) + ); + scanned += 1; + } + } + expect(scanned).toBeGreaterThan(10); + expect(findings).toEqual([]); + expect( + await Bun.file( + join(repoRoot, "plugins/hack/hooks/claude-hooks.json") + ).text() + ).toContain(HOOK_COMMAND); +}); + +test("negative control: the policy scan rejects each forbidden setting", () => { + const cases: readonly (readonly [string, string, PolicyFinding["code"]])[] = [ + ["plugin.json", JSON.stringify({ model: "opus" }), "model-setting"], + [ + "settings.json", + JSON.stringify({ env: { x: "1" }, modelOverride: "x" }), + "model-setting", + ], + ["config.toml", 'model = "gpt-5"\n', "model-setting"], + ["config.toml", 'model_reasoning_effort = "high"\n', "model-setting"], + ["config.toml", 'approval_policy = "never"\n', "permission-setting"], + [ + "config.toml", + 'sandbox_mode = "danger-full-access"\n', + "permission-setting", + ], + [ + "settings.json", + JSON.stringify({ permissions: { defaultMode: "bypassPermissions" } }), + "permission-setting", + ], + [ + "settings.json", + JSON.stringify({ outputStyle: "Explanatory" }), + "output-style-setting", + ], + ["output-styles/hack.md", "# Style\n", "output-style-setting"], + ["skills/x/SKILL.md", "---\nname: x\nmodel: opus\n---\n", "model-setting"], + [ + "skills/x/SKILL.md", + "---\nname: x\nallowed-tools: Bash\n---\n", + "permission-setting", + ], + [ + "skills/x/SKILL.md", + "---\nname: x\nhooks:\n Stop: []\n---\n", + "frontmatter-hooks", + ], + [ + "hooks/claude-hooks.json", + JSON.stringify({ + hooks: { + SessionStart: [ + { + hooks: [ + { type: "command", command: `${HOOK_COMMAND}; curl example` }, + ], + }, + ], + }, + }), + "hook-command", + ], + [ + "hooks/claude-hooks.json", + JSON.stringify({ + hooks: { Stop: [{ hooks: [{ type: "prompt", prompt: "x" }] }] }, + }), + "hook-command", + ], + ]; + for (const [file, text, code] of cases) { + expect(policyFindings(file, text).map((finding) => finding.code)).toContain( + code + ); + } + expect( + policyFindings( + "hooks/claude-hooks.json", + JSON.stringify({ + hooks: { + SessionStart: [ + { + matcher: "", + hooks: [{ type: "command", command: HOOK_COMMAND }], + }, + ], + }, + }) + ) + ).toEqual([]); +}); + +test("shipped skill inventory matches the generator and the integration doc", async () => { + tempDir = await mkdtemp(join(tmpdir(), "hack-agent-skills-")); + const pluginRoot = join(tempDir, "plugins", "hack"); + for (const directory of [ + ".codex-plugin", + ".claude-plugin", + ".cursor-plugin", + ]) { + await mkdir(join(pluginRoot, directory), { recursive: true }); + await cp( + join(repoRoot, "plugins/hack", directory, "plugin.json"), + join(pluginRoot, directory, "plugin.json") + ); + } + await cp( + join(repoRoot, "plugins/hack/.mcp.json"), + join(pluginRoot, ".mcp.json") + ); + await generateAgentPlugins({ repoRoot: tempDir, version: "0.0.0" }); + const skillNames = async (root: string) => + (await readdir(join(root, "skills"), { withFileTypes: true })) + .filter((entry) => entry.isDirectory()) + .map((entry) => entry.name) + .sort(); + const generated = await skillNames(pluginRoot); + expect(generated.length).toBeGreaterThan(0); + expect(await skillNames(join(repoRoot, "plugins/hack"))).toEqual(generated); + const doc = await Bun.file(join(repoRoot, "docs/integrations.md")).text(); + for (const name of generated) { + expect(doc).toContain(`\`${name}\``); + } + expect(doc).not.toContain("both skills"); +}, 15_000); From 58c485a92a06e237f0cc416332fbfae3dcb48a78 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:32:18 -0400 Subject: [PATCH 02/32] fix(mcp): harden run argv and tool input contract hack.project.run built `run `, so hack's own parser consumed child flags: `node --version` printed hack's version and `--branch other` retargeted the instance. Run now emits `run -- ` and every option value is fused as `--opt=value` across all tools. - Validate service, services, branch, profiles, logs service and open target against a compose-identifier pattern (no leading '-', no empty value; branches may also contain '/'). - tail, maxEvents, maxMs and timeoutMs are positive integers with hard ceilings (provisional F27 settings); effective limits are echoed in data.limits. - Every tool carries annotations (readOnly for list/status/logs/open, destructive for down/run/init, openWorld false). - Refusals return error{code, message} with existing codes (E_USAGE, E_PROJECT_NOT_FOUND) and spawned:false. A typed tools/call dispatcher validates before any handler runs, so refused calls leave no argv entry and no audit line. - Children spawn with HACK_NO_INTERACTIVE=1 on a copy of the frozen session env. Refs HACK-1133, HACK-1134 Co-Authored-By: Claude Opus 5.5 --- src/mcp/server.ts | 834 ++++++++++++++++++++---------- tests/mcp-session-context.test.ts | 13 +- tests/mcp.test.ts | 483 ++++++++++++++++- tests/project-run-command.test.ts | 37 ++ 4 files changed, 1078 insertions(+), 289 deletions(-) diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 779bd1b2c..c4b9b86a3 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -2,13 +2,23 @@ import { appendFile } from "node:fs/promises"; import { dirname, isAbsolute, relative, resolve } from "node:path"; import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; -import type { CallToolResult } from "@modelcontextprotocol/sdk/types.js"; +import type { RequestHandlerExtra } from "@modelcontextprotocol/sdk/shared/protocol.js"; +import { + CallToolRequestSchema, + type CallToolResult, + ErrorCode, + McpError, + type ServerNotification, + type ServerRequest, + type ToolAnnotations, +} from "@modelcontextprotocol/sdk/types.js"; import * as z from "zod/v4"; import pkg from "../../package.json"; import { type OnboardingMode, renderOnboardingPrompt, } from "../agents/onboarding-prompt.ts"; +import type { HackErrorCode } from "../lib/cli-result.ts"; import { ensureDir, pathExists } from "../lib/fs.ts"; import { isRecord } from "../lib/guards.ts"; import { resolveHackInvocation } from "../lib/hack-cli.ts"; @@ -52,15 +62,75 @@ type HackCommandResult = { readonly stderr: string; }; +/** Existing CLI error codes reused for pre-spawn MCP refusals (no new codes). */ +type McpRefusalCode = Extract; + type ToolResultPayload = HackCommandResult & { readonly ok: boolean; readonly data?: unknown; + /** Present (and false) only on refusals returned before any child spawned. */ + readonly spawned?: false; + readonly error?: { + readonly code: McpRefusalCode; + readonly message: string; + }; }; +type ProjectArgsResolution = + | { readonly ok: true; readonly args: string[] } + | { + readonly ok: false; + readonly code: McpRefusalCode; + readonly message: string; + }; + const DEFAULT_CMD_TIMEOUT_MS = 120_000; const DEFAULT_LOG_TAIL_EVENTS = 200; const DEFAULT_LOG_TAIL_MS = 5000; +/** + * Hard ceilings for numeric MCP inputs. These are provisional F27 experiment + * settings: values are refused above the ceiling (never clamped), and the + * effective limits a call ran with are echoed in `data.limits`. + */ +export const MCP_INPUT_CEILINGS = Object.freeze({ + tail: 10_000, + maxEvents: 10_000, + maxMs: 60_000, + timeoutMs: 600_000, +}); + +/** + * Compose-style identifier: starts with an ASCII letter or digit, so a value + * can never be read as an option (`-x`) or be empty. Applied to service, + * services, profiles, logs service and open target. + */ +const COMPOSE_IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_.-]*$/; +/** Same rule for branch names, which may also carry `/` (the CLI slugs it). */ +const BRANCH_IDENTIFIER_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_./-]*$/; +const IDENTIFIER_MAX_LENGTH = 253; + +const READ_ONLY_TOOL_ANNOTATIONS = Object.freeze({ + readOnlyHint: true, + openWorldHint: false, +}) satisfies ToolAnnotations; + +/** Changes local runtime state without removing it (up, restart). */ +const ADDITIVE_TOOL_ANNOTATIONS = Object.freeze({ + readOnlyHint: false, + destructiveHint: false, + idempotentHint: false, + openWorldHint: false, +}) satisfies ToolAnnotations; + +/** May stop, replace or overwrite local state (down, run, init). */ +const DESTRUCTIVE_TOOL_ANNOTATIONS = Object.freeze({ + readOnlyHint: false, + destructiveHint: true, + idempotentHint: false, + openWorldHint: false, +}) satisfies ToolAnnotations; + const toolOutputSchema = { ok: z.boolean(), outputTruncated: z.boolean().optional(), @@ -69,8 +139,50 @@ const toolOutputSchema = { stdout: z.string(), stderr: z.string(), data: z.unknown().optional(), + spawned: z.literal(false).optional(), + error: z + .object({ + code: z.enum(["E_USAGE", "E_PROJECT_NOT_FOUND"]), + message: z.string(), + }) + .optional(), } as const; +function composeIdentifierInput(description: string) { + return z + .string() + .max(IDENTIFIER_MAX_LENGTH) + .regex(COMPOSE_IDENTIFIER_PATTERN) + .describe(description); +} + +function branchIdentifierInput(description: string) { + return z + .string() + .max(IDENTIFIER_MAX_LENGTH) + .regex(BRANCH_IDENTIFIER_PATTERN) + .describe(description); +} + +function boundedLimitInput(opts: { + readonly ceiling: number; + readonly description: string; +}) { + return z + .number() + .int() + .positive() + .max(opts.ceiling) + .describe(`${opts.description} (1-${opts.ceiling})`); +} + +/** Child environment: the frozen session env plus forced non-interactivity. */ +function buildChildEnv( + env: Readonly> +): Record { + return { ...env, HACK_NO_INTERACTIVE: "1" }; +} + /** * Start the hack MCP server on stdio for local tool clients. */ @@ -193,6 +305,13 @@ function registerTools(opts: { readonly server: McpServer; readonly context: McpSessionContext; }): void { + const tools: HackToolRegistry = new Map(); + const register = ( + tool: HackToolDefinition + ): void => { + registerHackTool({ server: opts.server, tools, tool }); + }; + const projectSelectorInput = { projectName: z .string() @@ -209,39 +328,57 @@ function registerTools(opts: { }; const branchInput = { - branch: z.string().describe("Branch/worktree name").optional(), + branch: branchIdentifierInput("Branch/worktree name").optional(), }; const profileInput = { - profiles: z.array(z.string()).describe("Compose profile names").optional(), + profiles: z + .array(composeIdentifierInput("Compose profile name")) + .describe("Compose profile names") + .optional(), }; - opts.server.registerTool( - "hack.projects.list", - { - title: "List hack projects", - description: "List registered hack projects and runtime status.", - inputSchema: { - filter: z.string().describe("Filter by project name").optional(), - includeGlobal: z - .boolean() - .describe("Include global infra projects") - .optional(), - includeUnregistered: z - .boolean() - .describe("Include unregistered compose projects") - .optional(), - }, - outputSchema: toolOutputSchema, + const logSelectionInput = { + service: composeIdentifierInput("Single service name").optional(), + services: z + .array(composeIdentifierInput("Service name")) + .describe("Multiple service names") + .optional(), + query: z.string().describe("Raw LogQL selector/query").optional(), + loki: z.boolean().describe("Force Loki backend").optional(), + compose: z.boolean().describe("Force docker compose backend").optional(), + tail: boundedLimitInput({ + ceiling: MCP_INPUT_CEILINGS.tail, + description: "Tail last N log lines", + }).optional(), + since: z.string().describe("Start time (RFC3339 or duration)").optional(), + until: z.string().describe("End time (RFC3339 or duration)").optional(), + }; + + register({ + name: "hack.projects.list", + title: "List hack projects", + description: "List registered hack projects and runtime status.", + annotations: READ_ONLY_TOOL_ANNOTATIONS, + inputSchema: { + filter: z.string().describe("Filter by project name").optional(), + includeGlobal: z + .boolean() + .describe("Include global infra projects") + .optional(), + includeUnregistered: z + .boolean() + .describe("Include unregistered compose projects") + .optional(), }, - async ( + handler: async ( { filter, includeGlobal, includeUnregistered }, extra ): Promise => { const args = [ "projects", "--json", - ...(filter ? ["--project", filter] : []), + ...(filter ? [`--project=${filter}`] : []), ...(includeGlobal ? ["--include-global"] : []), ...(includeUnregistered ? ["--all"] : []), ]; @@ -257,29 +394,27 @@ function registerTools(opts: { result, data: parseJson(result.stdout), }); - } - ); - - opts.server.registerTool( - "hack.project.status", - { - title: "Project status", - description: "Get docker compose status for a project.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.status", + title: "Project status", + description: "Get docker compose status for a project.", + annotations: READ_ONLY_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -301,29 +436,27 @@ function registerTools(opts: { result, data: parseJson(result.stdout), }); - } - ); - - opts.server.registerTool( - "hack.project.up", - { - title: "Start project", - description: "Start project services in detached mode.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.up", + title: "Start project", + description: "Start project services in detached mode.", + annotations: ADDITIVE_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -342,29 +475,27 @@ function registerTools(opts: { }); return buildToolResult({ result }); - } - ); - - opts.server.registerTool( - "hack.project.down", - { - title: "Stop project", - description: "Stop project services.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.down", + title: "Stop project", + description: "Stop project services.", + annotations: DESTRUCTIVE_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -382,29 +513,27 @@ function registerTools(opts: { }); return buildToolResult({ result }); - } - ); - - opts.server.registerTool( - "hack.project.restart", - { - title: "Restart project", - description: "Restart project services.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.restart", + title: "Restart project", + description: "Restart project services.", + annotations: ADDITIVE_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -422,97 +551,79 @@ function registerTools(opts: { }); return buildToolResult({ result }); - } - ); - - opts.server.registerTool( - "hack.project.run", - { - title: "Run command in service container", - description: "Run a one-off command inside a service container.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - service: z.string().describe("Compose service name"), - cmd: z.array(z.string()).describe("Command arguments").optional(), - workdir: z - .string() - .describe("Working directory inside container") - .optional(), - timeoutMs: z.number().describe("Command timeout in ms").optional(), - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.run", + title: "Run command in service container", + description: + "Run a one-off command inside a service container. Command arguments are passed to the child after `--` and are never parsed as hack options.", + annotations: DESTRUCTIVE_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + service: composeIdentifierInput("Compose service name"), + cmd: z.array(z.string()).describe("Command arguments").optional(), + workdir: z + .string() + .describe("Working directory inside container") + .optional(), + timeoutMs: boundedLimitInput({ + ceiling: MCP_INPUT_CEILINGS.timeoutMs, + description: "Command timeout in ms", + }).optional(), + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } - const args = [ - "run", - ...resolved.args, - ...buildBranchArgs({ branch: input.branch }), - ...buildProfileArgs({ profiles: input.profiles }), - ...(input.workdir ? ["--workdir", input.workdir] : []), - input.service, - ...(input.cmd ?? []), - ]; - + const timeoutMs = input.timeoutMs ?? DEFAULT_CMD_TIMEOUT_MS; const result = await runHackCommand({ context: opts.context, signal: extra.signal, tool: "hack.project.run", - args, - timeoutMs: input.timeoutMs, + args: buildRunArgs({ + projectArgs: resolved.args, + branch: input.branch, + profiles: input.profiles, + workdir: input.workdir, + service: input.service, + cmd: input.cmd ?? [], + }), + timeoutMs, }); - return buildToolResult({ result }); - } - ); - - opts.server.registerTool( - "hack.project.logs.snapshot", - { - title: "Snapshot logs", - description: "Fetch a log snapshot and return structured log events.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - service: z.string().describe("Single service name").optional(), - services: z - .array(z.string()) - .describe("Multiple service names") - .optional(), - query: z.string().describe("Raw LogQL selector/query").optional(), - loki: z.boolean().describe("Force Loki backend").optional(), - compose: z - .boolean() - .describe("Force docker compose backend") - .optional(), - tail: z.number().describe("Tail last N log lines").optional(), - since: z - .string() - .describe("Start time (RFC3339 or duration)") - .optional(), - until: z.string().describe("End time (RFC3339 or duration)").optional(), - }, - outputSchema: toolOutputSchema, + return buildToolResult({ result, data: { limits: { timeoutMs } } }); }, - async (input, extra) => { + }); + + register({ + name: "hack.project.logs.snapshot", + title: "Snapshot logs", + description: "Fetch a log snapshot and return structured log events.", + annotations: READ_ONLY_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + ...logSelectionInput, + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -522,16 +633,7 @@ function registerTools(opts: { ...buildProfileArgs({ profiles: input.profiles }), "--json", "--no-follow", - ...(input.loki ? ["--loki"] : []), - ...(input.compose ? ["--compose"] : []), - ...(input.tail ? ["--tail", String(input.tail)] : []), - ...(input.query ? ["--query", input.query] : []), - ...(input.services - ? buildServicesArgs({ services: input.services }) - : []), - ...(input.since ? ["--since", input.since] : []), - ...(input.until ? ["--until", input.until] : []), - ...(input.service ? [input.service] : []), + ...buildLogSelectionArgs(input), ]; const result = await runHackCommand({ @@ -546,50 +648,39 @@ function registerTools(opts: { data: { events: parseJsonLines(result.stdout), count: countJsonLines(result.stdout), + limits: { tail: input.tail ?? null }, }, }); - } - ); - - opts.server.registerTool( - "hack.project.logs.tail", - { - title: "Tail logs", - description: "Stream logs briefly and return the collected events.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - ...profileInput, - service: z.string().describe("Single service name").optional(), - services: z - .array(z.string()) - .describe("Multiple service names") - .optional(), - query: z.string().describe("Raw LogQL selector/query").optional(), - loki: z.boolean().describe("Force Loki backend").optional(), - compose: z - .boolean() - .describe("Force docker compose backend") - .optional(), - tail: z.number().describe("Tail last N log lines").optional(), - since: z - .string() - .describe("Start time (RFC3339 or duration)") - .optional(), - until: z.string().describe("End time (RFC3339 or duration)").optional(), - maxEvents: z.number().describe("Max events to collect").optional(), - maxMs: z.number().describe("Max duration in ms").optional(), - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.logs.tail", + title: "Tail logs", + description: "Stream logs briefly and return the collected events.", + annotations: READ_ONLY_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + ...profileInput, + ...logSelectionInput, + maxEvents: boundedLimitInput({ + ceiling: MCP_INPUT_CEILINGS.maxEvents, + description: "Max events to collect", + }).optional(), + maxMs: boundedLimitInput({ + ceiling: MCP_INPUT_CEILINGS.maxMs, + description: "Max duration in ms", + }).optional(), + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -598,25 +689,18 @@ function registerTools(opts: { ...buildBranchArgs({ branch: input.branch }), ...buildProfileArgs({ profiles: input.profiles }), "--json", - ...(input.loki ? ["--loki"] : []), - ...(input.compose ? ["--compose"] : []), - ...(input.tail ? ["--tail", String(input.tail)] : []), - ...(input.query ? ["--query", input.query] : []), - ...(input.services - ? buildServicesArgs({ services: input.services }) - : []), - ...(input.since ? ["--since", input.since] : []), - ...(input.until ? ["--until", input.until] : []), - ...(input.service ? [input.service] : []), + ...buildLogSelectionArgs(input), ]; + const maxEvents = input.maxEvents ?? DEFAULT_LOG_TAIL_EVENTS; + const maxMs = input.maxMs ?? DEFAULT_LOG_TAIL_MS; const result = await runHackLogTail({ context: opts.context, signal: extra.signal, tool: "hack.project.logs.tail", args, - maxEvents: input.maxEvents, - maxMs: input.maxMs, + maxEvents, + maxMs, }); return buildToolResult({ @@ -632,31 +716,30 @@ function registerTools(opts: { count: result.events.length, stop_reason: result.stopReason, duration_ms: result.durationMs, + limits: { tail: input.tail ?? null, maxEvents, maxMs }, }, }); - } - ); - - opts.server.registerTool( - "hack.project.open", - { - title: "Resolve project URL", - description: "Return a URL for the project without launching a browser.", - inputSchema: { - ...projectSelectorInput, - ...branchInput, - target: z.string().describe("Target host/subdomain or URL").optional(), - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.open", + title: "Resolve project URL", + description: "Return a URL for the project without launching a browser.", + annotations: READ_ONLY_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + ...branchInput, + target: composeIdentifierInput("Target host or subdomain").optional(), + }, + handler: async (input, extra) => { const resolved = await resolveProjectArgs({ selection: toProjectSelection(input), cwd: opts.context.cwd, registryPath: opts.context.registryPath, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ @@ -678,35 +761,34 @@ function registerTools(opts: { result, data: parseJson(result.stdout), }); - } - ); - - opts.server.registerTool( - "hack.project.init", - { - title: "Initialize project", - description: "Initialize a repo using non-interactive defaults.", - inputSchema: { - ...projectSelectorInput, - name: z.string().describe("Project slug").optional(), - devHost: z.string().describe("DEV_HOST override").optional(), - oauth: z.boolean().describe("Enable OAuth alias host").optional(), - oauthTld: z.string().describe("OAuth alias TLD").optional(), - manual: z - .boolean() - .describe("Skip discovery and generate a minimal compose") - .optional(), - noDiscovery: z - .boolean() - .describe("Skip discovery even if available") - .optional(), - }, - outputSchema: toolOutputSchema, }, - async (input, extra) => { + }); + + register({ + name: "hack.project.init", + title: "Initialize project", + description: "Initialize a repo using non-interactive defaults.", + annotations: DESTRUCTIVE_TOOL_ANNOTATIONS, + inputSchema: { + ...projectSelectorInput, + name: z.string().describe("Project slug").optional(), + devHost: z.string().describe("DEV_HOST override").optional(), + oauth: z.boolean().describe("Enable OAuth alias host").optional(), + oauthTld: z.string().describe("OAuth alias TLD").optional(), + manual: z + .boolean() + .describe("Skip discovery and generate a minimal compose") + .optional(), + noDiscovery: z + .boolean() + .describe("Skip discovery even if available") + .optional(), + }, + handler: async (input, extra) => { const selection = toProjectSelection(input); if (!(selection.path || selection.repoRoot)) { return buildToolError({ + code: "E_USAGE", message: "Provide repoRoot or path for initialization.", }); } @@ -718,17 +800,17 @@ function registerTools(opts: { allowUnregistered: true, }); if (!resolved.ok) { - return buildToolError({ message: resolved.message }); + return buildToolError(resolved); } const args = [ "init", ...resolved.args, "--auto", - ...(input.name ? ["--name", input.name] : []), - ...(input.devHost ? ["--dev-host", input.devHost] : []), + ...(input.name ? [`--name=${input.name}`] : []), + ...(input.devHost ? [`--dev-host=${input.devHost}`] : []), ...(input.oauth ? ["--oauth"] : []), - ...(input.oauthTld ? ["--oauth-tld", input.oauthTld] : []), + ...(input.oauthTld ? [`--oauth-tld=${input.oauthTld}`] : []), ...(input.manual ? ["--manual"] : []), ...(input.noDiscovery ? ["--no-discovery"] : []), ]; @@ -742,10 +824,137 @@ function registerTools(opts: { }); return buildToolResult({ result }); + }, + }); + + installTypedToolDispatch({ server: opts.server, tools }); +} + +type ToolExtra = RequestHandlerExtra; + +type HackToolDefinition = { + readonly name: string; + readonly title: string; + readonly description: string; + readonly annotations: ToolAnnotations; + readonly inputSchema: Shape; + readonly handler: ( + input: z.output>, + extra: ToolExtra + ) => Promise; +}; + +type HackToolEntry = { + /** Validates raw arguments and refuses before the handler can spawn. */ + readonly call: (args: unknown, extra: ToolExtra) => Promise; +}; + +type HackToolRegistry = Map; + +/** + * Register one tool for discovery (`tools/list`) and keep its validating entry + * point for {@link installTypedToolDispatch}. The schema advertised by the SDK + * and the schema enforced at call time are the same object. + */ +function registerHackTool(opts: { + readonly server: McpServer; + readonly tools: HackToolRegistry; + readonly tool: HackToolDefinition; +}): void { + const { tool } = opts; + const schema = z.object(tool.inputSchema); + const entry: HackToolEntry = { + call: async (args, extra) => { + const parsed = await schema.safeParseAsync(args ?? {}); + if (!parsed.success) { + return buildToolError({ + code: "E_USAGE", + message: describeInvalidArguments(parsed.error), + }); + } + return await tool.handler(parsed.data, extra); + }, + }; + opts.tools.set(tool.name, entry); + const listedSchema: z.ZodObject = schema; + opts.server.registerTool( + tool.name, + { + title: tool.title, + description: tool.description, + annotations: { title: tool.title, ...tool.annotations }, + inputSchema: listedSchema, + outputSchema: toolOutputSchema, + }, + (args: unknown, extra: ToolExtra) => entry.call(args, extra) + ); +} + +/** + * Replace the SDK's generic `tools/call` dispatcher. The SDK rejects schema + * failures with untyped text before a handler runs; this dispatcher keeps that + * ordering (validation precedes any spawn) but returns the typed + * `{error, spawned: false}` refusal. Thrown handler failures and output-schema + * mismatches keep the SDK's untyped `isError` text result, because they may + * occur after a child was spawned. + */ +function installTypedToolDispatch(opts: { + readonly server: McpServer; + readonly tools: HackToolRegistry; +}): void { + const outputSchema = z.object(toolOutputSchema); + opts.server.server.setRequestHandler( + CallToolRequestSchema, + async (request, extra): Promise => { + const entry = opts.tools.get(request.params.name); + if (!entry) { + throw new McpError( + ErrorCode.InvalidParams, + `Tool ${request.params.name} not found` + ); + } + try { + const result = await entry.call(request.params.arguments, extra); + if ( + !( + result.isError || + outputSchema.safeParse(result.structuredContent).success + ) + ) { + return buildUntypedToolError( + `Output validation error: Invalid structured content for tool ${request.params.name}` + ); + } + return result; + } catch (error) { + return buildUntypedToolError( + error instanceof Error ? error.message : String(error) + ); + } } ); } +/** Closed, value-free summary: names the rejected fields, never echoes input. */ +function describeInvalidArguments(error: z.ZodError): string { + const fields = [ + ...new Set( + error.issues + .map((issue) => issue.path[0]) + .filter((key): key is string => typeof key === "string") + ), + ]; + const subject = + fields.length > 0 + ? `Invalid arguments: ${fields.join(", ")}.` + : "Invalid arguments."; + return `${subject} Identifiers must match ${COMPOSE_IDENTIFIER_PATTERN.source} (branches may also contain "/"); limits must be positive integers within their documented ceiling.`; +} + +function buildUntypedToolError(message: string): CallToolResult { + return { content: [{ type: "text", text: message }], isError: true }; +} + function buildToolResult(opts: { readonly result: HackCommandResult; readonly data?: unknown; @@ -770,16 +979,24 @@ function buildToolResult(opts: { }; } +/** + * Typed refusal for a request rejected before any child process exists. + * `spawned: false` is a guarantee: every caller returns before + * `runHackCommand`/`runHackLogTail`, so no argv, audit line or mutation occurs. + * `stderr` repeats the message for clients reading the earlier contract. + */ function buildToolError(opts: { + readonly code: McpRefusalCode; readonly message: string; - readonly command?: string; }): CallToolResult { const payload: ToolResultPayload = { ok: false, - command: opts.command ?? "", + command: "", exitCode: 1, stdout: "", stderr: opts.message, + spawned: false, + error: { code: opts.code, message: opts.message }, }; return { @@ -789,29 +1006,74 @@ function buildToolError(opts: { }; } +/** + * `hack run` argv. Every option value is fused (`--opt=value`) and `--` ends + * hack's option parsing, so nothing in `cmd` (for example `--branch other` or + * `--version`) can be consumed by hack or retarget the instance. + */ +export function buildRunArgs(opts: { + readonly projectArgs: readonly string[]; + readonly branch?: string; + readonly profiles?: readonly string[]; + readonly workdir?: string; + readonly service: string; + readonly cmd: readonly string[]; +}): string[] { + return [ + "run", + ...opts.projectArgs, + ...buildBranchArgs({ branch: opts.branch }), + ...buildProfileArgs({ profiles: opts.profiles }), + ...(opts.workdir ? [`--workdir=${opts.workdir}`] : []), + "--", + opts.service, + ...opts.cmd, + ]; +} + +function buildLogSelectionArgs(input: { + readonly loki?: boolean; + readonly compose?: boolean; + readonly tail?: number; + readonly query?: string; + readonly services?: readonly string[]; + readonly since?: string; + readonly until?: string; + readonly service?: string; +}): string[] { + return [ + ...(input.loki ? ["--loki"] : []), + ...(input.compose ? ["--compose"] : []), + ...(input.tail === undefined ? [] : [`--tail=${input.tail}`]), + ...(input.query ? [`--query=${input.query}`] : []), + ...(input.services ? buildServicesArgs({ services: input.services }) : []), + ...(input.since ? [`--since=${input.since}`] : []), + ...(input.until ? [`--until=${input.until}`] : []), + ...(input.service ? [input.service] : []), + ]; +} + function buildBranchArgs(opts: { readonly branch?: string }): string[] { - const branch = (opts.branch ?? "").trim(); - return branch.length > 0 ? ["--branch", branch] : []; + return opts.branch ? [`--branch=${opts.branch}`] : []; } function buildProfileArgs(opts: { readonly profiles?: readonly string[]; }): string[] { - const profiles = (opts.profiles ?? []).map((p) => p.trim()).filter(Boolean); + const profiles = opts.profiles ?? []; if (profiles.length === 0) { return []; } - return ["--profile", profiles.join(",")]; + return [`--profile=${profiles.join(",")}`]; } function buildServicesArgs(opts: { readonly services: readonly string[]; }): string[] { - const services = opts.services.map((s) => s.trim()).filter(Boolean); - if (services.length === 0) { + if (opts.services.length === 0) { return []; } - return ["--services", services.join(",")]; + return [`--services=${opts.services.join(",")}`]; } function toProjectSelection(input: ProjectSelection): ProjectSelection { @@ -827,10 +1089,7 @@ async function resolveProjectArgs(opts: { readonly selection: ProjectSelection; readonly cwd: string; readonly allowUnregistered?: boolean; -}): Promise< - | { readonly ok: true; readonly args: string[] } - | { readonly ok: false; readonly message: string } -> { +}): Promise { const selection = opts.selection; const picked = [ selection.projectName, @@ -840,6 +1099,7 @@ async function resolveProjectArgs(opts: { if (picked.length > 1) { return { ok: false, + code: "E_USAGE", message: "Use only one of projectName, repoRoot, or path.", }; } @@ -855,7 +1115,11 @@ async function resolveProjectArgs(opts: { if (pathLike) { const absPath = resolve(opts.cwd, pathLike); if (!(await pathExists(absPath))) { - return { ok: false, message: `Path not found: ${absPath}` }; + return { + ok: false, + code: "E_PROJECT_NOT_FOUND", + message: `Path not found: ${absPath}`, + }; } if (!opts.allowUnregistered) { const allowed = await isPathAllowed({ @@ -865,11 +1129,12 @@ async function resolveProjectArgs(opts: { if (!allowed) { return { ok: false, + code: "E_PROJECT_NOT_FOUND", message: "Path is outside registered hack projects.", }; } } - return { ok: true, args: ["--path", absPath] }; + return { ok: true, args: [`--path=${absPath}`] }; } const context = await findProjectContext(opts.cwd); @@ -885,25 +1150,24 @@ async function resolveProjectArgs(opts: { if (!allowed) { return { ok: false, + code: "E_PROJECT_NOT_FOUND", message: "No project context found. Provide projectName or path.", }; } } - return { ok: true, args: ["--path", opts.cwd] }; + return { ok: true, args: [`--path=${opts.cwd}`] }; } async function resolveProjectNameArgs(opts: { readonly projectName: string; readonly registryPath: string; -}): Promise< - | { readonly ok: true; readonly args: string[] } - | { readonly ok: false; readonly message: string } -> { +}): Promise { const name = normalizeProjectName(opts.projectName); if (!name) { return { ok: false, + code: "E_USAGE", message: "Invalid projectName: use a name containing letters or digits.", }; } @@ -915,16 +1179,17 @@ async function resolveProjectNameArgs(opts: { if (!project) { return { ok: false, + code: "E_PROJECT_NOT_FOUND", message: `Unknown project "${opts.projectName}". Run 'hack init' or 'hack projects' first.`, }; } - return { ok: true, args: ["--project", name] }; + return { ok: true, args: [`--project=${name}`] }; } catch (error) { if ( error instanceof AmbiguousProjectNameError || error instanceof AmbiguousProjectIdError ) { - return { ok: false, message: error.message }; + return { ok: false, code: "E_USAGE", message: error.message }; } throw error; } @@ -979,7 +1244,7 @@ async function runHackCommand(opts: { const proc = Bun.spawn(cmd, { detached: true, cwd: opts.cwd ?? opts.context.cwd, - env: opts.context.env, + env: buildChildEnv(opts.context.env), stdin: "ignore", stdout: "pipe", stderr: "pipe", @@ -1038,8 +1303,8 @@ async function runHackLogTail(opts: { readonly signal: AbortSignal; readonly tool: string; readonly args: readonly string[]; - readonly maxEvents?: number; - readonly maxMs?: number; + readonly maxEvents: number; + readonly maxMs: number; }): Promise<{ readonly outputTruncated?: boolean; readonly command: string; @@ -1057,14 +1322,13 @@ async function runHackLogTail(opts: { const proc = Bun.spawn(cmd, { detached: true, cwd: opts.context.cwd, - env: opts.context.env, + env: buildChildEnv(opts.context.env), stdin: "ignore", stdout: "pipe", stderr: "pipe", }); - const maxEvents = opts.maxEvents ?? DEFAULT_LOG_TAIL_EVENTS; - const maxMs = opts.maxMs ?? DEFAULT_LOG_TAIL_MS; + const { maxEvents, maxMs } = opts; const start = Date.now(); const events: Record[] = []; diff --git a/tests/mcp-session-context.test.ts b/tests/mcp-session-context.test.ts index cd34d4293..d9b47f70d 100644 --- a/tests/mcp-session-context.test.ts +++ b/tests/mcp-session-context.test.ts @@ -58,7 +58,7 @@ async function fixture(marker: string, delayMs = 20) { const command = join(project, "command"); await writeFile( command, - `#!${process.execPath}\nawait Bun.write("request-started", String(process.pid));\nawait Bun.sleep(${delayMs});\nconsole.log(JSON.stringify({ marker: process.env.SYNTHETIC_SESSION_MARKER, cwd: process.cwd(), args: Bun.argv.slice(2) }));\n` + `#!${process.execPath}\nawait Bun.write("request-started", String(process.pid));\nawait Bun.sleep(${delayMs});\nconsole.log(JSON.stringify({ marker: process.env.SYNTHETIC_SESSION_MARKER, noInteractive: process.env.HACK_NO_INTERACTIVE ?? null, cwd: process.cwd(), args: Bun.argv.slice(2) }));\n` ); await chmod(command, 0o755); const env = { @@ -67,6 +67,9 @@ async function fixture(marker: string, delayMs = 20) { HACK_HOME: state, HACK_GLOBAL_CONFIG_PATH: join(registryDir, "config.json"), HACK_MCP_COMMAND: "./command", + // Children must still see HACK_NO_INTERACTIVE=1: the server overrides + // this per spawn on a copy, never on the frozen session env. + HACK_NO_INTERACTIVE: "0", SYNTHETIC_SESSION_MARKER: marker, }; const server = createMcpServer({ cwd: project, env }); @@ -106,8 +109,9 @@ test("MCP sessions preserve independent cwd, environment, registry selection and ok: true, data: { marker: index === 0 ? "first" : "second", + noInteractive: "1", cwd: await realpath(item.project), - args: ["ps", "--project", "shared-name", "--json"], + args: ["ps", "--project=shared-name", "--json"], }, }); const audit = await readFile(join(item.state, "mcp-audit.log"), "utf8"); @@ -131,6 +135,11 @@ test("MCP path authorization uses only that session's registry", async () => { expect(denied.isError).toBe(true); expect(denied.structuredContent).toMatchObject({ stderr: "Path is outside registered hack projects.", + spawned: false, + error: { + code: "E_PROJECT_NOT_FOUND", + message: "Path is outside registered hack projects.", + }, }); const accepted = await second.client.callTool({ name: "hack.project.status", diff --git a/tests/mcp.test.ts b/tests/mcp.test.ts index 9d9a6e128..1b0227b26 100644 --- a/tests/mcp.test.ts +++ b/tests/mcp.test.ts @@ -1,11 +1,26 @@ -import { afterEach, expect } from "bun:test"; -import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { afterEach, expect, test } from "bun:test"; +import { + chmod, + mkdir, + mkdtemp, + readFile, + rm, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { Client } from "@modelcontextprotocol/sdk/client/index.js"; import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js"; +import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js"; +import { parseCliArgv, resolveCommand } from "../src/cli/command.ts"; +import { CLI_SPEC } from "../src/cli/spec.ts"; +import { + buildRunArgs, + createMcpServer, + MCP_INPUT_CEILINGS, +} from "../src/mcp/server.ts"; import { testIntegration } from "./helpers/ci.ts"; let tempDir: string | null = null; @@ -147,3 +162,467 @@ function buildHackStubScript(): string { "process.exit(1)", ].join("\n"); } + +// --------------------------------------------------------------------------- +// In-process argv and contract checks (HACK-1133 / HACK-1134). A stub hack +// binary records every argv it receives plus the child's interactivity env; +// the session writes its audit log under the fixture's HACK_HOME. +// --------------------------------------------------------------------------- + +const contractClients: Client[] = []; +const contractRoots: string[] = []; + +afterEach(async () => { + await Promise.all(contractClients.splice(0).map((mcp) => mcp.close())); + await Promise.all( + contractRoots + .splice(0) + .map((root) => rm(root, { recursive: true, force: true })) + ); +}); + +type ContractFixture = { + readonly client: Client; + readonly argvLog: string; + readonly auditLog: string; +}; + +type StubInvocation = { + readonly args: readonly string[]; + readonly noInteractive: string | null; +}; + +async function startContractFixture(): Promise { + const root = await mkdtemp(join(tmpdir(), "hack-mcp-contract-")); + contractRoots.push(root); + const project = join(root, "repo"); + const projectDir = join(project, ".hack"); + const state = join(root, "state"); + const registryDir = join(root, "registry"); + await mkdir(projectDir, { recursive: true }); + await mkdir(registryDir); + await writeFile(join(projectDir, "docker-compose.yml"), "services: {}\n"); + await writeFile( + join(projectDir, "hack.config.json"), + JSON.stringify({ name: "demo" }) + ); + await writeFile( + join(registryDir, "projects.json"), + JSON.stringify({ + version: 1, + projects: [ + { + id: "demo", + createdAt: "2026-01-01T00:00:00Z", + name: "demo", + repoRoot: project, + projectDirName: ".hack", + projectDir, + }, + ], + }) + ); + const argvLog = join(root, "argv.jsonl"); + const stub = join(root, "hack-stub"); + await writeFile( + stub, + [ + `#!${process.execPath}`, + 'import { appendFileSync } from "node:fs";', + "const args = Bun.argv.slice(2);", + `appendFileSync(${JSON.stringify(argvLog)}, \`\${JSON.stringify({ args, noInteractive: process.env.HACK_NO_INTERACTIVE ?? null })}\\n\`);`, + "console.log(JSON.stringify({ ok: true }));", + ].join("\n") + ); + await chmod(stub, 0o755); + const env: Record = { + ...process.env, + HOME: root, + HACK_HOME: state, + HACK_GLOBAL_CONFIG_PATH: join(registryDir, "config.json"), + HACK_MCP_COMMAND: stub, + }; + Reflect.deleteProperty(env, "HACK_NO_INTERACTIVE"); + const server = createMcpServer({ cwd: project, env }); + const client = new Client({ name: "hack-mcp-contract", version: "1" }); + contractClients.push(client); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + await server.connect(serverTransport); + await client.connect(clientTransport); + return { client, argvLog, auditLog: join(state, "mcp-audit.log") }; +} + +async function readStubInvocations( + argvLog: string +): Promise { + if (!(await Bun.file(argvLog).exists())) { + return []; + } + return (await readFile(argvLog, "utf8")) + .split("\n") + .filter((line) => line.length > 0) + .map((line) => JSON.parse(line) as StubInvocation); +} + +test("hack.project.run ends hack options with -- and passes child flags intact", async () => { + const fixture = await startContractFixture(); + const result = await fixture.client.callTool({ + name: "hack.project.run", + arguments: { + projectName: "demo", + branch: "feature/x", + profiles: ["tools", "extra"], + workdir: "/srv/app dir", + service: "api", + cmd: ["node", "--version", "--branch", "other"], + }, + }); + expect(result.isError).toBeUndefined(); + expect(result.structuredContent).toMatchObject({ + ok: true, + data: { limits: { timeoutMs: 120_000 } }, + }); + expect(await readStubInvocations(fixture.argvLog)).toEqual([ + { + args: [ + "run", + "--project=demo", + "--branch=feature/x", + "--profile=tools,extra", + "--workdir=/srv/app dir", + "--", + "api", + "node", + "--version", + "--branch", + "other", + ], + noInteractive: "1", + }, + ]); +}); + +test("hack's own parser never reads run child flags as hack options", () => { + const argv = buildRunArgs({ + projectArgs: ["--project=demo"], + service: "api", + cmd: ["node", "--version", "--branch", "other"], + }); + const parsed = parseCliArgv(CLI_SPEC, argv); + expect(parsed.values.version).toBeUndefined(); + expect(parsed.values.branch).toBeUndefined(); + expect(parsed.values.project).toBe("demo"); + const resolved = resolveCommand(CLI_SPEC, parsed.positionals); + expect(resolved.command?.name).toBe("run"); + expect(resolved.remainingPositionals).toEqual([ + "api", + "node", + "--version", + "--branch", + "other", + ]); +}); + +test("spawning tools force a non-interactive child and echo effective limits", async () => { + const markerBefore = process.env.HACK_NO_INTERACTIVE; + const fixture = await startContractFixture(); + const tail = await fixture.client.callTool({ + name: "hack.project.logs.tail", + arguments: { projectName: "demo", maxEvents: 5, maxMs: 1000, tail: 3 }, + }); + expect(tail.structuredContent).toMatchObject({ + ok: true, + data: { limits: { tail: 3, maxEvents: 5, maxMs: 1000 } }, + }); + const snapshot = await fixture.client.callTool({ + name: "hack.project.logs.snapshot", + arguments: { + projectName: "demo", + services: ["api", "worker"], + query: '{app="x"} |= "-y"', + }, + }); + expect(snapshot.structuredContent).toMatchObject({ + ok: true, + data: { limits: { tail: null } }, + }); + const ceiling = await fixture.client.callTool({ + name: "hack.project.run", + arguments: { + projectName: "demo", + service: "api", + timeoutMs: MCP_INPUT_CEILINGS.timeoutMs, + }, + }); + expect(ceiling.structuredContent).toMatchObject({ + ok: true, + data: { limits: { timeoutMs: MCP_INPUT_CEILINGS.timeoutMs } }, + }); + const invocations = await readStubInvocations(fixture.argvLog); + expect(invocations.map((entry) => entry.args)).toEqual([ + ["logs", "--project=demo", "--json", "--tail=3"], + [ + "logs", + "--project=demo", + "--json", + "--no-follow", + '--query={app="x"} |= "-y"', + "--services=api,worker", + ], + ["run", "--project=demo", "--", "api"], + ]); + expect(invocations.map((entry) => entry.noInteractive)).toEqual([ + "1", + "1", + "1", + ]); + // The child env is a copy: the server process env is never mutated. + expect(process.env.HACK_NO_INTERACTIVE).toBe(markerBefore); +}); + +const REFUSED_MARKER = "-REFUSED-VALUE-MARKER"; + +type RefusedCall = { + readonly label: string; + readonly name: string; + readonly arguments: Record; +}; + +const LIMIT_PROBES = [0, -1, 1.5] as const; + +function limitProbes(opts: { + readonly name: string; + readonly field: keyof typeof MCP_INPUT_CEILINGS; + readonly base: Record; +}): RefusedCall[] { + return [...LIMIT_PROBES, MCP_INPUT_CEILINGS[opts.field] + 1].map((value) => ({ + label: `${opts.name} ${opts.field}=${value}`, + name: opts.name, + arguments: { ...opts.base, [opts.field]: value }, + })); +} + +const refusedCalls: readonly RefusedCall[] = [ + { + label: "run leading-dash service", + name: "hack.project.run", + arguments: { projectName: "demo", service: REFUSED_MARKER }, + }, + { + label: "run empty service", + name: "hack.project.run", + arguments: { projectName: "demo", service: "" }, + }, + { + label: "run missing service", + name: "hack.project.run", + arguments: { projectName: "demo" }, + }, + { + label: "run leading-dash branch", + name: "hack.project.run", + arguments: { projectName: "demo", service: "api", branch: REFUSED_MARKER }, + }, + { + label: "status empty branch", + name: "hack.project.status", + arguments: { projectName: "demo", branch: "" }, + }, + { + label: "up leading-dash profile", + name: "hack.project.up", + arguments: { projectName: "demo", profiles: [REFUSED_MARKER] }, + }, + { + label: "down empty profile", + name: "hack.project.down", + arguments: { projectName: "demo", profiles: [""] }, + }, + { + label: "snapshot leading-dash service", + name: "hack.project.logs.snapshot", + arguments: { projectName: "demo", service: REFUSED_MARKER }, + }, + { + label: "snapshot leading-dash services entry", + name: "hack.project.logs.snapshot", + arguments: { projectName: "demo", services: ["api", REFUSED_MARKER] }, + }, + { + label: "open leading-dash target", + name: "hack.project.open", + arguments: { projectName: "demo", target: REFUSED_MARKER }, + }, + ...limitProbes({ + name: "hack.project.logs.tail", + field: "tail", + base: { projectName: "demo" }, + }), + ...limitProbes({ + name: "hack.project.logs.tail", + field: "maxEvents", + base: { projectName: "demo" }, + }), + ...limitProbes({ + name: "hack.project.logs.tail", + field: "maxMs", + base: { projectName: "demo" }, + }), + ...limitProbes({ + name: "hack.project.logs.snapshot", + field: "tail", + base: { projectName: "demo" }, + }), + ...limitProbes({ + name: "hack.project.run", + field: "timeoutMs", + base: { projectName: "demo", service: "api" }, + }), +]; + +test("invalid identifiers and limits are typed refusals that never spawn or audit", async () => { + const fixture = await startContractFixture(); + for (const call of refusedCalls) { + const result = await fixture.client.callTool({ + name: call.name, + arguments: call.arguments, + }); + expect({ label: call.label, isError: result.isError }).toEqual({ + label: call.label, + isError: true, + }); + expect(result.structuredContent).toMatchObject({ + ok: false, + spawned: false, + command: "", + exitCode: 1, + stdout: "", + error: { code: "E_USAGE" }, + }); + // Closed diagnostics: the rejected value is never echoed back. + expect(JSON.stringify(result)).not.toContain(REFUSED_MARKER); + } + expect(await readStubInvocations(fixture.argvLog)).toEqual([]); + expect(await Bun.file(fixture.auditLog).exists()).toBe(false); +}); + +test("project resolution refusals carry existing codes and never spawn", async () => { + const fixture = await startContractFixture(); + const unknown = await fixture.client.callTool({ + name: "hack.project.down", + arguments: { projectName: "missing-project" }, + }); + expect(unknown.structuredContent).toMatchObject({ + ok: false, + spawned: false, + error: { code: "E_PROJECT_NOT_FOUND" }, + }); + const ambiguous = await fixture.client.callTool({ + name: "hack.project.run", + arguments: { projectName: "demo", path: ".", service: "api" }, + }); + expect(ambiguous.structuredContent).toMatchObject({ + ok: false, + spawned: false, + stderr: "Use only one of projectName, repoRoot, or path.", + error: { + code: "E_USAGE", + message: "Use only one of projectName, repoRoot, or path.", + }, + }); + expect(await readStubInvocations(fixture.argvLog)).toEqual([]); + expect(await Bun.file(fixture.auditLog).exists()).toBe(false); +}); + +const READ_ONLY = { readOnlyHint: true, openWorldHint: false }; +const ADDITIVE = { + readOnlyHint: false, + destructiveHint: false, + idempotentHint: false, + openWorldHint: false, +}; +const DESTRUCTIVE = { + readOnlyHint: false, + destructiveHint: true, + idempotentHint: false, + openWorldHint: false, +}; + +test("tools/list publishes names, annotations and required inputs for every tool", async () => { + const fixture = await startContractFixture(); + const listed = await fixture.client.listTools(); + expect( + listed.tools.map((tool) => ({ + name: tool.name, + annotations: tool.annotations, + required: tool.inputSchema.required ?? [], + })) + ).toEqual([ + { + name: "hack.projects.list", + annotations: { title: "List hack projects", ...READ_ONLY }, + required: [], + }, + { + name: "hack.project.status", + annotations: { title: "Project status", ...READ_ONLY }, + required: [], + }, + { + name: "hack.project.up", + annotations: { title: "Start project", ...ADDITIVE }, + required: [], + }, + { + name: "hack.project.down", + annotations: { title: "Stop project", ...DESTRUCTIVE }, + required: [], + }, + { + name: "hack.project.restart", + annotations: { title: "Restart project", ...ADDITIVE }, + required: [], + }, + { + name: "hack.project.run", + annotations: { + title: "Run command in service container", + ...DESTRUCTIVE, + }, + required: ["service"], + }, + { + name: "hack.project.logs.snapshot", + annotations: { title: "Snapshot logs", ...READ_ONLY }, + required: [], + }, + { + name: "hack.project.logs.tail", + annotations: { title: "Tail logs", ...READ_ONLY }, + required: [], + }, + { + name: "hack.project.open", + annotations: { title: "Resolve project URL", ...READ_ONLY }, + required: [], + }, + { + name: "hack.project.init", + annotations: { title: "Initialize project", ...DESTRUCTIVE }, + required: [], + }, + ]); + const run = listed.tools.find((tool) => tool.name === "hack.project.run"); + expect(run?.inputSchema.properties?.timeoutMs).toMatchObject({ + type: "integer", + exclusiveMinimum: 0, + maximum: MCP_INPUT_CEILINGS.timeoutMs, + }); + expect(run?.inputSchema.properties?.service).toMatchObject({ + type: "string", + pattern: "^[A-Za-z0-9][A-Za-z0-9_.-]*$", + }); + expect(run?.outputSchema?.properties).toHaveProperty("spawned"); + expect(run?.outputSchema?.properties).toHaveProperty("error"); +}); diff --git a/tests/project-run-command.test.ts b/tests/project-run-command.test.ts index ac79e70c3..5b148b3dd 100644 --- a/tests/project-run-command.test.ts +++ b/tests/project-run-command.test.ts @@ -12,6 +12,9 @@ import { registerScopedModuleMock } from "./helpers/scoped-module-mock.ts"; const runCalls: Array<{ readonly composeFiles: readonly string[]; + readonly composeProject: string | null | undefined; + readonly service: string; + readonly cmdArgs: readonly string[]; readonly env: Record | undefined; readonly noDeps: boolean | undefined; }> = []; @@ -36,11 +39,17 @@ const runtimeBackendMock = await registerScopedModuleMock({ ps: async () => 0, run: async (opts: { readonly composeFiles: readonly string[]; + readonly composeProject?: string | null; + readonly service: string; + readonly cmdArgs: readonly string[]; readonly env?: Record; readonly noDeps?: boolean; }) => { runCalls.push({ composeFiles: [...opts.composeFiles], + composeProject: opts.composeProject, + service: opts.service, + cmdArgs: [...opts.cmdArgs], env: opts.env ? { ...opts.env } : undefined, noDeps: opts.noDeps, }); @@ -68,6 +77,7 @@ const loggerMock = await registerScopedModuleMock({ }); const { runCommand } = await import("../src/commands/project.ts"); +const { runCli } = await import("../src/cli/run.ts"); beforeAll(() => { runtimeBackendMock.activate(); @@ -117,6 +127,33 @@ test("run reaches the runtime without writing a contended registry", async () => ); }); +test("hack run -- passes child flags intact and keeps the default compose project", async () => { + const projectRoot = await createProject(); + // Without `--`, hack would consume --version (print and exit) and --branch + // (retarget the instance); the MCP run tool relies on this boundary. + const exitCode = await runCli([ + "run", + `--path=${projectRoot}`, + "--", + "api", + "node", + "--version", + "--branch", + "other", + ]); + + expect(exitCode).toBe(0); + expect(runCalls).toHaveLength(1); + expect(runCalls[0]?.service).toBe("api"); + expect(runCalls[0]?.cmdArgs).toEqual([ + "node", + "--version", + "--branch", + "other", + ]); + expect(runCalls[0]?.composeProject).toBeNull(); +}); + async function runPrintenv({ projectRoot, envName, From c6c0db35ab75ae02aa0a4b5d307425211efb1992 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:40:00 -0400 Subject: [PATCH 03/32] fix(env): harden legacy env key files and AES-GCM envelope parsing Legacy project env key reads (checkout-local, shared common-dir and inherited primary keys) now use the same lstat, O_NOFOLLOW, size-bound and stable-descriptor discipline as the native reader, extracted into src/lib/private-file.ts. Key priority is unchanged. A symlinked, dangling, oversized or non-regular key file is refused with a fixed message. New key files are created with open(O_CREAT|O_EXCL|O_NOFOLLOW|O_WRONLY, 0600), fchmod 0600 and fsync, so an existing entry or dangling symlink is never followed or replaced. A concurrent creator's key is adopted. Encryption (env add --secret and the legacy migration) refuses a git-tracked selected or inherited key with a fixed, value-free message before any key copy. Decryption with a tracked key stays allowed for v4 compatibility. Legacy v1 values and the encrypted_file store now require canonical base64, a 12-byte IV, a 16-byte tag (authTagLength pinned) and a bounded ciphertext; all parse, tamper and wrong-key failures use fixed redacted errors. The envelope format, key derivation and v1 decryptability are unchanged. Refs HACK-1127 Co-Authored-By: Claude Opus 5.5 --- src/lib/private-file.ts | 239 ++++++++++++++++ src/lib/project-env-config.ts | 379 +++++++++++++++---------- src/lib/secret-store.ts | 82 ++++-- tests/native-env-values.test.ts | 23 ++ tests/project-env-config.test.ts | 205 +++++++++++++ tests/project-env-worktree-key.test.ts | 266 +++++++++++++++++ tests/secret-store.test.ts | 152 ++++++++++ 7 files changed, 1171 insertions(+), 175 deletions(-) create mode 100644 src/lib/private-file.ts diff --git a/src/lib/private-file.ts b/src/lib/private-file.ts new file mode 100644 index 000000000..fd7f02d6e --- /dev/null +++ b/src/lib/private-file.ts @@ -0,0 +1,239 @@ +import { constants, type Stats } from "node:fs"; +import { + type FileHandle, + lstat, + open, + realpath, + unlink, +} from "node:fs/promises"; +import { dirname } from "node:path"; +import { isRecord } from "./guards.ts"; + +/** + * Bounded, no-follow reads and exclusive creates for private key material. + * + * Ownership: callers own the path choice and the refusal message. These helpers + * never follow a symlink at the final path component, never replace an existing + * entry, and never include file contents in errors. + */ + +export type BoundedPrivateFileReadOptions = { + readonly path: string; + /** + * Directories whose identity must stay stable across the read. With + * `canonical`, each one and the file must also be spelled by its realpath. + */ + readonly directories: readonly string[]; + /** Require the canonical (realpath) spelling for `path` and `directories`. */ + readonly canonical: boolean; + /** Largest accepted file size in bytes. */ + readonly maxBytes: number; + /** Builds the caller's fixed, value-free refusal. */ + readonly refusal: () => Error; + /** Runs between steps, e.g. to observe cancellation. */ + readonly checkpoint?: () => void; +}; + +function sameFileVersion(before: Stats, after: Stats): boolean { + return ( + before.dev === after.dev && + before.ino === after.ino && + before.size === after.size && + before.mtimeMs === after.mtimeMs && + before.ctimeMs === after.ctimeMs + ); +} + +function isMissingEntryError(error: unknown): boolean { + return isRecord(error) && error.code === "ENOENT"; +} + +/** + * Reads a small private text file without following a final-component symlink. + * Returns null when the entry is absent or contains only whitespace; otherwise + * the trimmed UTF-8 text. A symlink (dangling or not), a non-regular file, an + * unreadable or oversized file, invalid UTF-8, or a file that changes during + * the read throws `refusal()`; other filesystem errors propagate unchanged. + */ +export async function readBoundedPrivateFile( + opts: BoundedPrivateFileReadOptions +): Promise { + const checkpoint = opts.checkpoint ?? (() => undefined); + checkpoint(); + const directories = await Promise.all( + opts.directories.map(async (path) => { + const stats = await lstat(path); + if ( + !stats.isDirectory() || + (opts.canonical && (await realpath(path)) !== path) + ) { + throw opts.refusal(); + } + return { path, stats }; + }) + ); + const recheckDirectories = async () => { + for (const { path, stats } of directories) { + const current = await lstat(path); + if ( + !current.isDirectory() || + current.dev !== stats.dev || + current.ino !== stats.ino || + (opts.canonical && (await realpath(path)) !== path) + ) { + throw opts.refusal(); + } + } + checkpoint(); + }; + const observed = await lstat(opts.path).catch((error: unknown) => { + if (isMissingEntryError(error)) { + return null; + } + throw error; + }); + if (!observed) { + await recheckDirectories(); + return null; + } + if ( + !observed.isFile() || + observed.size > opts.maxBytes || + (observed.mode & 0o444) === 0 || + (opts.canonical && (await realpath(opts.path)) !== opts.path) + ) { + throw opts.refusal(); + } + const file = await open( + opts.path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); + try { + checkpoint(); + const before = await file.stat(); + if (!(before.isFile() && sameFileVersion(observed, before))) { + throw opts.refusal(); + } + const buffer = Buffer.alloc(before.size + 1); + let size = 0; + while (size < buffer.length) { + checkpoint(); + const { bytesRead } = await file.read( + buffer, + size, + buffer.length - size, + size + ); + if (bytesRead === 0) { + break; + } + size += bytesRead; + } + const after = await file.stat(); + const current = await lstat(opts.path); + if ( + size !== before.size || + !current.isFile() || + !sameFileVersion(before, after) || + !sameFileVersion(before, current) || + (opts.canonical && (await realpath(opts.path)) !== opts.path) + ) { + throw opts.refusal(); + } + await recheckDirectories(); + let text: string; + try { + text = new TextDecoder("utf-8", { fatal: true }).decode( + buffer.subarray(0, size) + ); + } catch { + throw opts.refusal(); + } + return text.trim() || null; + } finally { + await file.close(); + } +} + +/** + * Creates `path` with mode 0600 and writes `text`, durably. The open uses + * O_CREAT|O_EXCL|O_NOFOLLOW, so an existing entry, including a dangling + * symlink, fails with EEXIST and its target is never created or written. + * On a failed write the partially written file is removed only while the + * path still names the file this call created. + */ +export async function createPrivateFileExclusive(opts: { + readonly path: string; + readonly text: string; +}): Promise { + const file = await open( + opts.path, + constants.O_CREAT | + constants.O_EXCL | + constants.O_NOFOLLOW | + constants.O_WRONLY, + 0o600 + ); + let created: Stats | null = null; + let complete = false; + try { + created = await file.stat(); + // The umask can only remove bits; restore owner read/write explicitly. + await file.chmod(0o600); + await file.writeFile(opts.text, "utf8"); + await file.sync(); + complete = true; + } finally { + await file.close(); + if (!complete && created) { + const current = await lstat(opts.path).catch(() => null); + if ( + current?.isFile() && + current.dev === created.dev && + current.ino === created.ino + ) { + await unlink(opts.path).catch(() => undefined); + } + } + } + await syncDirectory({ path: dirname(opts.path) }); +} + +/** Best effort: persist the directory entry where the platform allows it. */ +async function syncDirectory(opts: { readonly path: string }): Promise { + let directory: FileHandle | null = null; + try { + directory = await open(opts.path, constants.O_RDONLY); + await directory.sync(); + } catch { + // Some filesystems refuse directory fsync; the file itself is already synced. + } finally { + await directory?.close().catch(() => undefined); + } +} + +/** + * Decodes standard padded base64 only when the text is its canonical encoding + * (re-encoding the bytes reproduces the text exactly). Whitespace, URL-safe or + * invalid characters, missing or extra padding, and non-zero trailing bits are + * refused. The length is bounded before any decoding allocation. + */ +export function decodeCanonicalBase64(opts: { + readonly text: string; + readonly maxBytes: number; +}): Buffer | null { + if ( + opts.text.length % 4 !== 0 || + opts.text.length > Math.ceil(opts.maxBytes / 3) * 4 + ) { + return null; + } + const decoded = Buffer.from(opts.text, "base64"); + if ( + decoded.length > opts.maxBytes || + decoded.toString("base64") !== opts.text + ) { + return null; + } + return decoded; +} diff --git a/src/lib/project-env-config.ts b/src/lib/project-env-config.ts index 2e9c9895d..1d756962a 100644 --- a/src/lib/project-env-config.ts +++ b/src/lib/project-env-config.ts @@ -4,16 +4,7 @@ import { createHash, randomBytes, } from "node:crypto"; -import { constants, type Stats } from "node:fs"; -import { - chmod, - lstat, - open, - readdir, - realpath, - rm, - stat, -} from "node:fs/promises"; +import { lstat, readdir, rm, stat } from "node:fs/promises"; import { basename, dirname, relative, resolve } from "node:path"; import { YAML } from "bun"; import { @@ -63,6 +54,11 @@ import { acquireNativeManagedEnvFile, assertNativeProjectInputRoot, } from "./native-project-inputs.ts"; +import { + createPrivateFileExclusive, + decodeCanonicalBase64, + readBoundedPrivateFile, +} from "./private-file.ts"; import { readProjectDefaultEnvConfig } from "./project.ts"; import { assertLegacyProjectDirectory, @@ -87,6 +83,9 @@ const PROJECT_ENV_SCOPE_PATTERN = /^[a-z0-9][a-z0-9._-]*$/; const PROJECT_ENV_SECRET_PREFIX = "v1" as const; const PROJECT_ENV_ALGORITHM = "aes-256-gcm"; const PROJECT_ENV_IV_BYTES = 12; +const PROJECT_ENV_TAG_BYTES = 16; +/** Decoded ciphertext bound; a value never exceeds its 1 MiB env layer. */ +const PROJECT_ENV_CIPHERTEXT_LIMIT = NATIVE_CONFIG_INPUT_LIMIT; const PROJECT_ENV_SECRET_KEY_ENV = "HACK_ENV_SECRET_KEY"; export const PROJECT_ENV_HOST_SCOPE = "host" as const; @@ -2699,28 +2698,29 @@ async function resolveProjectEnvKey(opts: { signal: opts.nativeSelection.signal, }); } + // Priority: checkout-local, shared common-dir, primary checkout, environment. const keyPath = resolveProjectEnvKeyPath({ projectRoot: opts.projectRoot }); + const localKey = await readLegacyProjectEnvKeyFile({ path: keyPath }); + if (localKey) { + return localKey; + } const sharedKeyPath = await resolveProjectEnvSharedKeyPath({ projectRoot: opts.projectRoot, }); - const text = await readTextFile(keyPath); - const sharedText = - sharedKeyPath === null ? null : await readTextFile(sharedKeyPath); - const envFallback = process.env[PROJECT_ENV_SECRET_KEY_ENV]?.trim() ?? ""; + const sharedKey = + sharedKeyPath === null + ? null + : await readLegacyProjectEnvKeyFile({ path: sharedKeyPath }); + if (sharedKey) { + return sharedKey; + } const inheritedKey = await resolveInheritedProjectEnvKey({ projectRoot: opts.projectRoot, }); - const trimmed = text?.trim() ?? ""; - if (trimmed.length > 0) { - return trimmed; - } - const sharedTrimmed = sharedText?.trim() ?? ""; - if (sharedTrimmed.length > 0) { - return sharedTrimmed; - } if (inheritedKey) { return inheritedKey.keyText; } + const envFallback = process.env[PROJECT_ENV_SECRET_KEY_ENV]?.trim() ?? ""; if (envFallback.length > 0) { return envFallback; } @@ -2787,108 +2787,80 @@ async function resolveNativeProjectEnvKey(opts: { return null; } -function sameNativeKeyFile(before: Stats, after: Stats): boolean { - return ( - before.dev === after.dev && - before.ino === after.ino && - before.size === after.size && - before.mtimeMs === after.mtimeMs && - before.ctimeMs === after.ctimeMs - ); -} - /** Keys stay inside their existing owner; reject redirected, oversized or changing descriptor reads. */ async function readNativeProjectEnvKeyFile(opts: { readonly path: string; readonly directories: readonly string[]; readonly signal?: AbortSignal; }): Promise { - checkNativeEnvCancellation(opts.signal); - const directories = await Promise.all( - opts.directories.map(async (path) => { - const stats = await lstat(path); - if (!stats.isDirectory() || (await realpath(path)) !== path) { - throw nativeEnvValuesError(); - } - return { path, stats }; - }) - ); - const recheckDirectories = async () => { - for (const { path, stats } of directories) { - const current = await lstat(path); - if ( - !current.isDirectory() || - current.dev !== stats.dev || - current.ino !== stats.ino || - (await realpath(path)) !== path - ) { - throw nativeEnvValuesError(); - } - } - checkNativeEnvCancellation(opts.signal); - }; - const observed = await lstat(opts.path).catch((error: unknown) => { - if (isRecord(error) && error.code === "ENOENT") { - return null; - } - throw error; + return await readBoundedPrivateFile({ + path: opts.path, + directories: opts.directories, + canonical: true, + maxBytes: NATIVE_CONFIG_INPUT_LIMIT, + refusal: nativeEnvValuesError, + checkpoint: () => checkNativeEnvCancellation(opts.signal), }); - if (!observed) { - await recheckDirectories(); - return null; +} + +/** + * Legacy key files are refused unless they are regular, non-symlink, bounded + * UTF-8 files, so a committed `.hack.secret.key` symlink never selects another + * file as the key. Directory components keep their legacy spelling (temporary + * and home directories are often reached through symlinks); only the final + * component is pinned. Absent and whitespace-only files return null. + */ +async function readLegacyProjectEnvKeyFile(opts: { + readonly path: string; +}): Promise { + try { + return await readBoundedPrivateFile({ + path: opts.path, + directories: [], + canonical: false, + maxBytes: NATIVE_CONFIG_INPUT_LIMIT, + refusal: projectEnvKeyFileError, + }); + } catch { + throw projectEnvKeyFileError(); } - if ( - !observed.isFile() || - observed.size > NATIVE_CONFIG_INPUT_LIMIT || - (observed.mode & 0o444) === 0 || - (await realpath(opts.path)) !== opts.path - ) { - throw nativeEnvValuesError(); +} + +/** Fixed, value-free refusal for legacy key selection, creation and encryption. */ +export class ProjectEnvKeyRefusalError extends Error { + constructor(message: string) { + super(message); + this.name = "ProjectEnvKeyRefusalError"; } - const file = await open( - opts.path, - constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK +} + +function projectEnvKeyFileError(): ProjectEnvKeyRefusalError { + return new ProjectEnvKeyRefusalError( + `Refusing project env key: ${PROJECT_ENV_KEY_FILENAME} must be a regular, readable, non-symlink, non-empty UTF-8 file of at most 1 MiB. Remove or replace it; key contents are never included in this message.` ); - try { - checkNativeEnvCancellation(opts.signal); - const before = await file.stat(); - if (!(before.isFile() && sameNativeKeyFile(observed, before))) { - throw nativeEnvValuesError(); - } - const buffer = Buffer.alloc(before.size + 1); - let size = 0; - while (size < buffer.length) { - checkNativeEnvCancellation(opts.signal); - const { bytesRead } = await file.read( - buffer, - size, - buffer.length - size, - size - ); - if (bytesRead === 0) { - break; - } - size += bytesRead; - } - const after = await file.stat(); - const current = await lstat(opts.path); - if ( - size !== before.size || - !current.isFile() || - !sameNativeKeyFile(before, after) || - !sameNativeKeyFile(before, current) || - (await realpath(opts.path)) !== opts.path - ) { - throw nativeEnvValuesError(); - } - await recheckDirectories(); - return ( - new TextDecoder("utf-8", { fatal: true }) - .decode(buffer.subarray(0, size)) - .trim() || null - ); - } finally { - await file.close(); +} + +function projectEnvTrackedKeyError(): ProjectEnvKeyRefusalError { + return new ProjectEnvKeyRefusalError( + `Refusing to encrypt with a git-tracked ${PROJECT_ENV_KEY_FILENAME}: a committed key is not secret. Run 'hack doctor --fix' to untrack it, rotate the key, and re-encrypt existing secrets.` + ); +} + +/** + * Encryption refuses a key file that git tracks. Decryption with such a key + * stays allowed for v4 compatibility; `hack doctor` reports the tracked key. + * An undeterminable status (git unavailable, path outside a work tree) is not + * treated as tracked. + */ +async function assertProjectEnvKeyUntracked(opts: { + readonly keyPath: string; +}): Promise { + const tracked = await isGitTrackedProjectEnvPath({ + projectRoot: dirname(opts.keyPath), + path: opts.keyPath, + }); + if (tracked === true) { + throw projectEnvTrackedKeyError(); } } @@ -2913,6 +2885,11 @@ export type EnsureProjectEnvSecretKeyResult = { * A checkout-local key is never silently created in a linked worktree: if the * shared location cannot be resolved (degraded git) or written, the fallback * local write carries a loud divergence warning (also returned in `warnings`). + * + * This is the encryption-side key owner. Key files are read without following + * symlinks, a git-tracked selected or adopted key is refused before any write, + * and new key files are created exclusively with mode 0600 (an existing entry, + * including a dangling symlink, is never replaced or followed). */ export async function ensureProjectEnvSecretKey(opts: { readonly projectRoot: string; @@ -2923,25 +2900,25 @@ export async function ensureProjectEnvSecretKey(opts: { projectRoot: opts.projectRoot, }); const sharedKeyPath = await resolveSharedKeyPathFromLocation(sharedLocation); - const existing = await readTextFile(keyPath); - if (existing !== null && existing.trim().length > 0) { + const existing = await readLegacyProjectEnvKeyFile({ path: keyPath }); + if (existing !== null) { + await assertProjectEnvKeyUntracked({ keyPath }); return { keyPath, - keyText: existing.trim(), + keyText: existing, created: false, warnings: [], }; } const sharedExisting = - sharedKeyPath === null ? null : await readTextFile(sharedKeyPath); - if ( - sharedKeyPath !== null && - sharedExisting !== null && - sharedExisting.trim().length > 0 - ) { + sharedKeyPath === null + ? null + : await readLegacyProjectEnvKeyFile({ path: sharedKeyPath }); + if (sharedKeyPath !== null && sharedExisting !== null) { + await assertProjectEnvKeyUntracked({ keyPath: sharedKeyPath }); return { keyPath: sharedKeyPath, - keyText: sharedExisting.trim(), + keyText: sharedExisting, created: false, warnings: [], }; @@ -2951,6 +2928,8 @@ export async function ensureProjectEnvSecretKey(opts: { projectRoot: opts.projectRoot, }); if (inheritedKey) { + // Refuse before copying a committed key into the shared location. + await assertProjectEnvKeyUntracked({ keyPath: inheritedKey.keyPath }); const written = await writeProjectEnvKeyWithFallback({ projectRoot: opts.projectRoot, preferredKeyPath: sharedKeyPath, @@ -2960,7 +2939,7 @@ export async function ensureProjectEnvSecretKey(opts: { }); return { keyPath: written.keyPath, - keyText: inheritedKey.keyText, + keyText: written.keyText, created: false, warnings: written.warnings, }; @@ -2976,8 +2955,9 @@ export async function ensureProjectEnvSecretKey(opts: { }); return { keyPath: written.keyPath, - keyText, - created: true, + keyText: written.keyText, + // A concurrent writer's key is adopted rather than replaced. + created: written.keyText === keyText, warnings: written.warnings, }; } @@ -3005,19 +2985,26 @@ async function writeProjectEnvKeyWithFallback(opts: { readonly localKeyPath: string; readonly keyText: string; readonly sharedLocation: ProjectEnvSharedKeyLocation | null; -}): Promise<{ readonly keyPath: string; readonly warnings: string[] }> { +}): Promise<{ + readonly keyPath: string; + readonly keyText: string; + readonly warnings: string[]; +}> { const warnings: string[] = []; if (opts.preferredKeyPath !== null) { try { - await writeProjectEnvKeyFile({ + const keyText = await writeProjectEnvKeyFile({ projectRoot: opts.projectRoot, path: opts.preferredKeyPath, keyText: opts.keyText, }); - return { keyPath: opts.preferredKeyPath, warnings }; + return { keyPath: opts.preferredKeyPath, keyText, warnings }; } catch (error: unknown) { - if (error instanceof ProjectInputSelectionError) { + if ( + error instanceof ProjectInputSelectionError || + error instanceof ProjectEnvKeyRefusalError + ) { throw error; } const message = error instanceof Error ? error.message : String(error); @@ -3034,7 +3021,7 @@ async function writeProjectEnvKeyWithFallback(opts: { ); } - await writeProjectEnvKeyFile({ + const keyText = await writeProjectEnvKeyFile({ projectRoot: opts.projectRoot, path: opts.localKeyPath, keyText: opts.keyText, @@ -3046,19 +3033,68 @@ async function writeProjectEnvKeyWithFallback(opts: { comment: "# project env key", }); emitProjectEnvKeyWarnings({ warnings }); - return { keyPath: opts.localKeyPath, warnings }; + return { keyPath: opts.localKeyPath, keyText, warnings }; } +/** + * Creates the key file exclusively (O_CREAT|O_EXCL|O_NOFOLLOW, mode 0600, + * fsync) and returns the key text now stored at `path`. When the path is + * already occupied, a valid untracked regular key file (e.g. a concurrent + * writer's) is adopted; anything else, including a symlink, is refused. + */ async function writeProjectEnvKeyFile(opts: { readonly projectRoot: string; readonly path: string; readonly keyText: string; -}): Promise { +}): Promise { await assertLegacyProjectInputFamily({ projectRoot: opts.projectRoot }); await ensureDir(dirname(opts.path)); await assertLegacyProjectInputFamily({ projectRoot: opts.projectRoot }); - await writeTextFile(opts.path, `${opts.keyText}\n`); - await chmod(opts.path, 0o600); + try { + await createPrivateFileExclusive({ + path: opts.path, + text: `${opts.keyText}\n`, + }); + return opts.keyText; + } catch (error: unknown) { + if (!(isRecord(error) && error.code === "EEXIST")) { + throw error; + } + } + return await adoptExistingProjectEnvKeyFile({ path: opts.path }); +} + +const PROJECT_ENV_KEY_ADOPTION_ATTEMPTS = 20; +const PROJECT_ENV_KEY_ADOPTION_DELAY_MS = 25; + +/** + * A concurrent creator may expose its key file before writing the key, so a + * regular file is re-read for a bounded period (about half a second). A + * symlink or other entry, or a file still empty or invalid afterwards, is + * refused; it is never replaced. + */ +async function adoptExistingProjectEnvKeyFile(opts: { + readonly path: string; +}): Promise { + for ( + let attempt = 0; + attempt < PROJECT_ENV_KEY_ADOPTION_ATTEMPTS; + attempt += 1 + ) { + const entry = await lstat(opts.path).catch(() => null); + if (!entry?.isFile()) { + break; + } + const existing = await readLegacyProjectEnvKeyFile({ + path: opts.path, + }).catch(() => null); + if (existing !== null) { + await assertProjectEnvKeyUntracked({ keyPath: opts.path }); + return existing; + } + await Bun.sleep(PROJECT_ENV_KEY_ADOPTION_DELAY_MS); + } + throw projectEnvKeyFileError(); } function emitProjectEnvKeyWarnings(opts: { @@ -3080,8 +3116,8 @@ async function resolveInheritedProjectEnvKey(opts: { } const keyPath = resolveProjectEnvKeyPath({ projectRoot: primaryRoot }); - const keyText = (await readTextFile(keyPath))?.trim() ?? ""; - if (keyText.length === 0) { + const keyText = await readLegacyProjectEnvKeyFile({ path: keyPath }); + if (keyText === null) { return null; } @@ -3097,7 +3133,9 @@ function encryptProjectEnvValue(opts: { }): string { const iv = randomBytes(PROJECT_ENV_IV_BYTES); const key = deriveProjectEnvKey({ keyText: opts.keyText }); - const cipher = createCipheriv(PROJECT_ENV_ALGORITHM, key, iv); + const cipher = createCipheriv(PROJECT_ENV_ALGORITHM, key, iv, { + authTagLength: PROJECT_ENV_TAG_BYTES, + }); const ciphertext = Buffer.concat([ cipher.update(opts.plaintext, "utf8"), cipher.final(), @@ -3111,34 +3149,65 @@ function encryptProjectEnvValue(opts: { ].join(":"); } +function invalidSecureEnvValueError(): Error { + return new Error("Invalid secure env value."); +} + +/** + * Parses and opens a v1 value `v1:::`. Each segment must + * be canonical padded base64; the IV must be 12 bytes, the tag 16 bytes and the + * ciphertext at most the env input bound. An empty ciphertext segment is the + * authenticated empty plaintext. Malformed, oversized, truncated, tampered and + * wrong-key inputs all throw the same fixed, value-free error. + */ function decryptProjectEnvValue(opts: { readonly ciphertext: string; readonly keyText: string; }): string { const parts = opts.ciphertext.split(":"); const [prefix, ivText, tagText, ciphertextText] = parts; - // AES-GCM authenticates an empty plaintext with an empty ciphertext segment. if ( parts.length !== 4 || prefix !== PROJECT_ENV_SECRET_PREFIX || - !ivText || - !tagText || + ivText === undefined || + tagText === undefined || ciphertextText === undefined ) { - throw new Error("Invalid secure env value."); + throw invalidSecureEnvValueError(); + } + const iv = decodeCanonicalBase64({ + text: ivText, + maxBytes: PROJECT_ENV_IV_BYTES, + }); + const tag = decodeCanonicalBase64({ + text: tagText, + maxBytes: PROJECT_ENV_TAG_BYTES, + }); + const ciphertext = decodeCanonicalBase64({ + text: ciphertextText, + maxBytes: PROJECT_ENV_CIPHERTEXT_LIMIT, + }); + if ( + iv?.length !== PROJECT_ENV_IV_BYTES || + tag?.length !== PROJECT_ENV_TAG_BYTES || + ciphertext === null + ) { + throw invalidSecureEnvValueError(); + } + try { + const key = deriveProjectEnvKey({ keyText: opts.keyText }); + const decipher = createDecipheriv(PROJECT_ENV_ALGORITHM, key, iv, { + authTagLength: PROJECT_ENV_TAG_BYTES, + }); + decipher.setAuthTag(tag); + const plaintext = Buffer.concat([ + decipher.update(ciphertext), + decipher.final(), + ]); + return plaintext.toString("utf8"); + } catch { + throw invalidSecureEnvValueError(); } - const key = deriveProjectEnvKey({ keyText: opts.keyText }); - const decipher = createDecipheriv( - PROJECT_ENV_ALGORITHM, - key, - Buffer.from(ivText, "base64") - ); - decipher.setAuthTag(Buffer.from(tagText, "base64")); - const plaintext = Buffer.concat([ - decipher.update(Buffer.from(ciphertextText, "base64")), - decipher.final(), - ]); - return plaintext.toString("utf8"); } function deriveProjectEnvKey(opts: { readonly keyText: string }): Buffer { diff --git a/src/lib/secret-store.ts b/src/lib/secret-store.ts index bc82e9ae6..847492244 100644 --- a/src/lib/secret-store.ts +++ b/src/lib/secret-store.ts @@ -15,6 +15,7 @@ import type { import { readControlPlaneConfig } from "../control-plane/sdk/config.ts"; import { resolveGlobalHackDir } from "./config-paths.ts"; import { ensureDir, readTextFile } from "./fs.ts"; +import { decodeCanonicalBase64 } from "./private-file.ts"; const ENCRYPTED_FILE_KEY_ENV = "HACK_SECRETS_FILE_KEY"; const DISABLE_ENCRYPTED_FILE_KEYCHAIN_FALLBACK_ENV = @@ -27,6 +28,9 @@ const ENCRYPTED_STORE_VERSION = 1 as const; const PLAINTEXT_STORE_VERSION = 1 as const; const ENCRYPTED_ALGORITHM = "aes-256-gcm"; const ENCRYPTED_IV_BYTES = 12; +const ENCRYPTED_TAG_BYTES = 16; +/** Decoded ciphertext bound for the whole encrypted store. */ +const ENCRYPTED_CIPHERTEXT_LIMIT = 16 * 1024 * 1024; let cachedEncryptedFileKey: string | null = null; let encryptedFileKeyFailureCooldownUntilMs = 0; @@ -38,6 +42,13 @@ type EncryptedStorePayload = { readonly ciphertext: string; }; +/** Envelope fields decoded and size-checked; the bytes are not yet authenticated. */ +type ParsedEncryptedEnvelope = { + readonly iv: Buffer; + readonly tag: Buffer; + readonly ciphertext: Buffer; +}; + type PlaintextStore = { readonly version: typeof PLAINTEXT_STORE_VERSION; readonly entries: Record; @@ -712,10 +723,17 @@ async function readEncryptedEntries(input: { `Invalid encrypted secret store payload at ${input.filePath}.` ); } - const plaintext = decryptStore({ - envelope, - encryptionKey: input.encryptionKey, - }); + let plaintext: string; + try { + plaintext = decryptStore({ + envelope, + encryptionKey: input.encryptionKey, + }); + } catch { + throw new Error( + `Cannot decrypt encrypted secret store at ${input.filePath}: the key does not match or the payload was modified.` + ); + } return parsePlaintextStore(plaintext).entries; } @@ -736,7 +754,14 @@ async function writeEncryptedEntries(input: { await Bun.write(input.filePath, `${JSON.stringify(envelope, null, 2)}\n`); } -function parseEncryptedEnvelope(value: unknown): EncryptedStorePayload | null { +/** + * Accepts only the v1 envelope with canonical padded base64 fields, a 12-byte + * IV, a 16-byte tag and a bounded ciphertext. Anything else returns null so the + * caller reports a fixed, value-free error. + */ +function parseEncryptedEnvelope( + value: unknown +): ParsedEncryptedEnvelope | null { if (!(value && typeof value === "object")) { return null; } @@ -750,13 +775,26 @@ function parseEncryptedEnvelope(value: unknown): EncryptedStorePayload | null { ) { return null; } - return { - version: ENCRYPTED_STORE_VERSION, - algorithm: ENCRYPTED_ALGORITHM, - iv: maybe.iv, - tag: maybe.tag, - ciphertext: maybe.ciphertext, - }; + const iv = decodeCanonicalBase64({ + text: maybe.iv, + maxBytes: ENCRYPTED_IV_BYTES, + }); + const tag = decodeCanonicalBase64({ + text: maybe.tag, + maxBytes: ENCRYPTED_TAG_BYTES, + }); + const ciphertext = decodeCanonicalBase64({ + text: maybe.ciphertext, + maxBytes: ENCRYPTED_CIPHERTEXT_LIMIT, + }); + if ( + iv?.length !== ENCRYPTED_IV_BYTES || + tag?.length !== ENCRYPTED_TAG_BYTES || + ciphertext === null + ) { + return null; + } + return { iv, tag, ciphertext }; } function parsePlaintextStore(value: string): PlaintextStore { @@ -797,7 +835,9 @@ function encryptStore(input: { }): EncryptedStorePayload { const iv = randomBytes(ENCRYPTED_IV_BYTES); const key = deriveEncryptionKey({ source: input.encryptionKey }); - const cipher = createCipheriv(ENCRYPTED_ALGORITHM, key, iv); + const cipher = createCipheriv(ENCRYPTED_ALGORITHM, key, iv, { + authTagLength: ENCRYPTED_TAG_BYTES, + }); const ciphertext = Buffer.concat([ cipher.update(input.plaintext, "utf8"), cipher.final(), @@ -813,17 +853,19 @@ function encryptStore(input: { } function decryptStore(input: { - readonly envelope: EncryptedStorePayload; + readonly envelope: ParsedEncryptedEnvelope; readonly encryptionKey: string; }): string { const key = deriveEncryptionKey({ source: input.encryptionKey }); - const iv = Buffer.from(input.envelope.iv, "base64"); - const tag = Buffer.from(input.envelope.tag, "base64"); - const ciphertext = Buffer.from(input.envelope.ciphertext, "base64"); - const decipher = createDecipheriv(ENCRYPTED_ALGORITHM, key, iv); - decipher.setAuthTag(tag); + const decipher = createDecipheriv( + ENCRYPTED_ALGORITHM, + key, + input.envelope.iv, + { authTagLength: ENCRYPTED_TAG_BYTES } + ); + decipher.setAuthTag(input.envelope.tag); const plaintext = Buffer.concat([ - decipher.update(ciphertext), + decipher.update(input.envelope.ciphertext), decipher.final(), ]); return plaintext.toString("utf8"); diff --git a/tests/native-env-values.test.ts b/tests/native-env-values.test.ts index a0d35af4a..e3aaeef77 100644 --- a/tests/native-env-values.test.ts +++ b/tests/native-env-values.test.ts @@ -508,6 +508,29 @@ for (const kind of [ }); } +// GCM verifies a truncated tag unless the tag length is pinned; both owners pin 16 bytes. +for (const kind of ["truncated-tag", "noncanonical-base64"] as const) { + test(`authentic ciphertext with a ${kind} refuses with fixed redacted diagnostics`, async () => { + const p = await project(); + await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); + const authentic = await encrypted(); + if (!(isRecord(authentic) && typeof authentic.secure === "string")) { + throw new Error("Synthetic ciphertext absent"); + } + const [prefix, iv, tag = "", ciphertext] = authentic.secure.split(":"); + const changedTag = + kind === "truncated-tag" + ? Buffer.from(tag, "base64").subarray(0, 12).toString("base64") + : ` ${tag}`; + await layer(p, "hack.env.default.yaml", { + global: { + SECRET: { secure: [prefix, iv, changedTag, ciphertext].join(":") }, + }, + }); + await refuses(() => values(p)); + }); +} + test("an encrypted empty value preserves presence", async () => { const p = await project(); await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); diff --git a/tests/project-env-config.test.ts b/tests/project-env-config.test.ts index 0ed20d6df..ee3da7b8a 100644 --- a/tests/project-env-config.test.ts +++ b/tests/project-env-config.test.ts @@ -10,6 +10,7 @@ import { } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; +import { YAML } from "bun"; import { PROJECT_COMPOSE_FILENAME, @@ -1548,3 +1549,207 @@ test("Compose env repair rechecks native selection after reading the repair inpu ).rejects.toThrow("E_NATIVE_PROJECT_CONFLICT"); expect(await readFile(repo.composeFile, "utf8")).toBe(contents); }); + +const ENVELOPE_KEY = "synthetic-envelope-key-for-tests-only"; +const ENVELOPE_SENTINEL = "synthetic-envelope-plaintext-never-in-errors"; + +type EnvelopeParts = { + readonly iv: Buffer; + readonly tag: Buffer; + readonly ciphertext: Buffer; +}; + +/** Encrypts the sentinel with the real mutation owner and returns the v1 value. */ +async function encryptedEnvelopeFixture(): Promise<{ + readonly repo: Awaited>; + readonly secure: string; +}> { + const repo = await createRepo(); + await writeFile( + resolve(repo.projectRoot, PROJECT_ENV_KEY_FILENAME), + `${ENVELOPE_KEY}\n` + ); + await setProjectEnvValue({ + projectRoot: repo.projectRoot, + projectDir: repo.projectDir, + envName: null, + scope: "global", + key: "TOKEN", + value: ENVELOPE_SENTINEL, + secret: true, + }); + const parsed: unknown = YAML.parse( + await readFile(resolve(repo.projectDir, "hack.env.default.yaml"), "utf8") + ); + const secure = ( + parsed as { values?: { global?: { TOKEN?: { secure?: unknown } } } } + ).values?.global?.TOKEN?.secure; + if (typeof secure !== "string") { + throw new Error("missing synthetic envelope"); + } + return { repo, secure }; +} + +function envelopeParts(secure: string): EnvelopeParts { + const [, iv = "", tag = "", ciphertext = ""] = secure.split(":"); + return { + iv: Buffer.from(iv, "base64"), + tag: Buffer.from(tag, "base64"), + ciphertext: Buffer.from(ciphertext, "base64"), + }; +} + +function envelope(parts: EnvelopeParts): string { + return [ + "v1", + parts.iv.toString("base64"), + parts.tag.toString("base64"), + parts.ciphertext.toString("base64"), + ].join(":"); +} + +async function writeSecureValue(opts: { + readonly repo: Awaited>; + readonly secure: string; +}): Promise { + await writeFile( + resolve(opts.repo.projectDir, "hack.env.default.yaml"), + JSON.stringify({ + version: 1, + environment: "default", + secretsprovider: "project_key", + values: { global: { TOKEN: { secure: opts.secure } } }, + }) + ); +} + +const ENVELOPE_MUTATIONS: ReadonlyArray< + readonly [string, (secure: string) => string] +> = [ + ...[4, 8, 12].map( + (bytes) => + [ + `${bytes}-byte tag`, + (secure: string) => { + const parts = envelopeParts(secure); + return envelope({ ...parts, tag: parts.tag.subarray(0, bytes) }); + }, + ] as const + ), + [ + "16-byte IV", + (secure) => { + const parts = envelopeParts(secure); + return envelope({ + ...parts, + iv: Buffer.concat([parts.iv, Buffer.alloc(4)]), + }); + }, + ], + [ + "base64 with !", + (secure) => { + const [prefix, iv, tag, ciphertext = ""] = secure.split(":"); + return [prefix, iv, tag, `!${ciphertext.slice(1)}`].join(":"); + }, + ], + [ + "base64 with whitespace", + (secure) => { + const [prefix, iv = "", tag, ciphertext] = secure.split(":"); + return [prefix, `${iv.slice(0, 4)} ${iv.slice(4)}`, tag, ciphertext].join( + ":" + ); + }, + ], + [ + "truncated ciphertext", + (secure) => { + const parts = envelopeParts(secure); + return envelope({ + ...parts, + ciphertext: parts.ciphertext.subarray(0, -1), + }); + }, + ], + [ + "bit-flipped ciphertext", + (secure) => { + const parts = envelopeParts(secure); + const ciphertext = Buffer.from(parts.ciphertext); + ciphertext[0] = (ciphertext[0] ?? 0) ^ 1; + return envelope({ ...parts, ciphertext }); + }, + ], + [ + "oversized ciphertext", + (secure) => + envelope({ + ...envelopeParts(secure), + ciphertext: Buffer.alloc(1024 * 1024 + 1, 7), + }), + ], +]; + +for (const [name, mutate] of ENVELOPE_MUTATIONS) { + test(`legacy secure env value with ${name} refuses with a fixed redacted error`, async () => { + const { repo, secure } = await encryptedEnvelopeFixture(); + const mutated = mutate(secure); + expect(mutated).not.toBe(secure); + await writeSecureValue({ repo, secure: mutated }); + + const error = await resolveProjectEnvConfig({ + projectRoot: repo.projectRoot, + projectDir: repo.projectDir, + envName: null, + serviceNames: ["api"], + }).then( + () => null, + (caught: unknown) => caught + ); + + expect(error).toBeInstanceOf(Error); + const message = error instanceof Error ? error.message : ""; + expect(message).toBe("Invalid secure env value."); + for (const privateText of [ENVELOPE_SENTINEL, ENVELOPE_KEY]) { + expect(String(error)).not.toContain(privateText); + } + }); +} + +test("legacy secure env value with the wrong key refuses with the same fixed error", async () => { + const { repo } = await encryptedEnvelopeFixture(); + await writeFile( + resolve(repo.projectRoot, PROJECT_ENV_KEY_FILENAME), + "synthetic-wrong-key\n" + ); + await expect( + resolveProjectEnvConfig({ + projectRoot: repo.projectRoot, + projectDir: repo.projectDir, + envName: null, + serviceNames: ["api"], + }) + ).rejects.toThrow("Invalid secure env value."); +}); + +test("legacy v1 secure values round-trip, including the empty value", async () => { + const { repo } = await encryptedEnvelopeFixture(); + await setProjectEnvValue({ + projectRoot: repo.projectRoot, + projectDir: repo.projectDir, + envName: null, + scope: "global", + key: "EMPTY", + value: "", + secret: true, + }); + const resolved = await resolveProjectEnvConfig({ + projectRoot: repo.projectRoot, + projectDir: repo.projectDir, + envName: null, + serviceNames: ["api"], + }); + expect(resolved?.globalEnv.TOKEN).toBe(ENVELOPE_SENTINEL); + expect(resolved?.globalEnv.EMPTY).toBe(""); +}); diff --git a/tests/project-env-worktree-key.test.ts b/tests/project-env-worktree-key.test.ts index cc7233384..3bda56e8c 100644 --- a/tests/project-env-worktree-key.test.ts +++ b/tests/project-env-worktree-key.test.ts @@ -1,11 +1,13 @@ import { afterEach, expect, test } from "bun:test"; import { chmod, + lstat, mkdir, mkdtemp, readFile, rm, stat, + symlink, writeFile, } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -16,8 +18,10 @@ import { PROJECT_CONFIG_FILENAME, PROJECT_ENV_KEY_FILENAME, } from "../src/constants.ts"; +import { createPrivateFileExclusive } from "../src/lib/private-file.ts"; import { ensureProjectEnvSecretKey, + ProjectEnvKeyRefusalError, resolveProjectEnvConfig, resolveProjectEnvSharedKeyLocation, setProjectEnvValue, @@ -241,3 +245,265 @@ test.skipIf(process.getuid?.() === 0)( expect(ensured.warnings[0]).toContain(location.path); } ); + +const KEY_SENTINEL = "synthetic-committed-key-text-never-in-output"; +const VALUE_SENTINEL = "synthetic-secret-value-never-in-output"; + +type CliResult = { + readonly exitCode: number; + readonly stdout: string; + readonly stderr: string; +}; + +/** Runs the current-branch CLI with an isolated HOME and global config. */ +async function runHackCli(opts: { + readonly sandbox: string; + readonly args: readonly string[]; +}): Promise { + const home = resolve(opts.sandbox, "home"); + await mkdir(home, { recursive: true }); + const env: Record = { + ...process.env, + HOME: home, + HACK_GLOBAL_CONFIG_PATH: resolve(home, "hack.config.json"), + }; + Reflect.deleteProperty(env, "HACK_ENV_SECRET_KEY"); + const proc = Bun.spawn( + ["bun", resolve(import.meta.dir, "../index.ts"), ...opts.args], + { + cwd: opts.sandbox, + env, + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + } + ); + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { exitCode, stdout, stderr }; +} + +function envAddSecretArgs(projectRoot: string): readonly string[] { + return [ + "env", + "add", + "API_TOKEN", + VALUE_SENTINEL, + "--secret", + "--path", + projectRoot, + ]; +} + +function expectNoPrivateText(result: CliResult): void { + for (const text of [KEY_SENTINEL, VALUE_SENTINEL]) { + expect(result.stdout).not.toContain(text); + expect(result.stderr).not.toContain(text); + } +} + +async function commitKeyFile(primaryRoot: string): Promise { + await runGit(["add", "-f", PROJECT_ENV_KEY_FILENAME], primaryRoot); + await runGit(["commit", "-m", "commit key"], primaryRoot); +} + +test("env add --secret refuses a dangling key symlink and never creates its target", async () => { + const fixture = await createWorktreeFixture(); + const leakPath = resolve(fixture.primaryRoot, "leak.txt"); + await symlink( + "leak.txt", + resolve(fixture.primaryRoot, PROJECT_ENV_KEY_FILENAME) + ); + + const result = await runHackCli({ + sandbox: fixture.sandbox, + args: envAddSecretArgs(fixture.primaryRoot), + }); + + expect(result.exitCode).not.toBe(0); + expect(result.stderr).toContain("Refusing project env key"); + expect(await Bun.file(leakPath).exists()).toBe(false); + expect( + await Bun.file( + resolve(fixture.primaryRoot, ".hack", "hack.env.default.yaml") + ).exists() + ).toBe(false); + expectNoPrivateText(result); +}); + +test("env add --secret refuses a key symlink to a tracked file without disclosing it", async () => { + const fixture = await createWorktreeFixture(); + const trackedPath = resolve(fixture.primaryRoot, "committed-key.txt"); + await writeFile(trackedPath, `${KEY_SENTINEL}\n`); + await runGit(["add", "committed-key.txt"], fixture.primaryRoot); + await runGit(["commit", "-m", "track key text"], fixture.primaryRoot); + await symlink( + "committed-key.txt", + resolve(fixture.primaryRoot, PROJECT_ENV_KEY_FILENAME) + ); + + const result = await runHackCli({ + sandbox: fixture.sandbox, + args: envAddSecretArgs(fixture.primaryRoot), + }); + + expect(result.exitCode).not.toBe(0); + expect(result.stderr).toContain("Refusing project env key"); + expect(await readFile(trackedPath, "utf8")).toBe(`${KEY_SENTINEL}\n`); + expectNoPrivateText(result); +}); + +test("env add --secret refuses to encrypt with a committed key and keeps the key text out of output", async () => { + const fixture = await createWorktreeFixture(); + await writeFile( + resolve(fixture.primaryRoot, PROJECT_ENV_KEY_FILENAME), + `${KEY_SENTINEL}\n`, + { mode: 0o600 } + ); + await commitKeyFile(fixture.primaryRoot); + + const result = await runHackCli({ + sandbox: fixture.sandbox, + args: envAddSecretArgs(fixture.primaryRoot), + }); + + expect(result.exitCode).not.toBe(0); + expect(result.stderr).toContain("Refusing to encrypt with a git-tracked"); + expect( + await Bun.file( + resolve(fixture.primaryRoot, ".hack", "hack.env.default.yaml") + ).exists() + ).toBe(false); + expectNoPrivateText(result); +}); + +test("a linked worktree refuses a committed primary key before copying it to the shared location", async () => { + const fixture = await createWorktreeFixture(); + await writeFile( + resolve(fixture.primaryRoot, PROJECT_ENV_KEY_FILENAME), + `${KEY_SENTINEL}\n` + ); + await commitKeyFile(fixture.primaryRoot); + + const error = await ensureProjectEnvSecretKey({ + projectRoot: fixture.linkedRoot, + }).then( + () => null, + (caught: unknown) => caught + ); + + expect(error).toBeInstanceOf(ProjectEnvKeyRefusalError); + expect(String(error)).not.toContain(KEY_SENTINEL); + const location = await resolveProjectEnvSharedKeyLocation({ + projectRoot: fixture.linkedRoot, + }); + expect(location).not.toBeNull(); + expect(await Bun.file(location?.path ?? "").exists()).toBe(false); +}); + +// v4 compatibility choice: decrypting with a tracked key keeps working (hack +// doctor reports the committed key); only encryption refuses it. +test("decrypting with a git-tracked key stays allowed for v4 compatibility while encryption refuses", async () => { + const fixture = await createWorktreeFixture(); + const projectDir = resolve(fixture.primaryRoot, ".hack"); + await setProjectEnvValue({ + projectRoot: fixture.primaryRoot, + projectDir, + envName: null, + scope: "api", + key: "API_TOKEN", + value: VALUE_SENTINEL, + secret: true, + }); + await commitKeyFile(fixture.primaryRoot); + + const resolved = await resolveProjectEnvConfig({ + projectRoot: fixture.primaryRoot, + projectDir, + envName: null, + serviceNames: ["api"], + }); + expect(resolved?.serviceEnv.api?.API_TOKEN).toBe(VALUE_SENTINEL); + + await expect( + setProjectEnvValue({ + projectRoot: fixture.primaryRoot, + projectDir, + envName: null, + scope: "api", + key: "OTHER_TOKEN", + value: VALUE_SENTINEL, + secret: true, + }) + ).rejects.toThrow("Refusing to encrypt with a git-tracked"); +}); + +test("a new checkout-local key is created 0600 even under a restrictive umask", async () => { + const fixture = await createWorktreeFixture(); + const previousUmask = process.umask(0o277); + let ensured: Awaited>; + try { + ensured = await ensureProjectEnvSecretKey({ + projectRoot: fixture.primaryRoot, + }); + } finally { + process.umask(previousUmask); + } + expect(ensured.created).toBe(true); + expect(ensured.keyPath).toBe( + resolve(fixture.primaryRoot, PROJECT_ENV_KEY_FILENAME) + ); + const info = await lstat(ensured.keyPath); + expect(info.isFile()).toBe(true); + expect(info.mode & 0o777).toBe(0o600); + expect((await readFile(ensured.keyPath, "utf8")).trim()).toBe( + ensured.keyText + ); +}); + +test("concurrent key creation across checkouts converges on one shared key", async () => { + const fixture = await createWorktreeFixture(); + const results = await Promise.all([ + ensureProjectEnvSecretKey({ projectRoot: fixture.linkedRoot }), + ensureProjectEnvSecretKey({ projectRoot: fixture.linkedRoot }), + ]); + const location = await resolveProjectEnvSharedKeyLocation({ + projectRoot: fixture.linkedRoot, + }); + const stored = (await readFile(location?.path ?? "", "utf8")).trim(); + for (const result of results) { + expect(result.keyPath).toBe(location?.path ?? ""); + expect(result.keyText).toBe(stored); + } + expect(results.filter((result) => result.created).length).toBe(1); +}); + +test("a whitespace-only key file is refused instead of being replaced", async () => { + const fixture = await createWorktreeFixture(); + const keyPath = resolve(fixture.primaryRoot, PROJECT_ENV_KEY_FILENAME); + await writeFile(keyPath, "\n"); + + await expect( + ensureProjectEnvSecretKey({ projectRoot: fixture.primaryRoot }) + ).rejects.toBeInstanceOf(ProjectEnvKeyRefusalError); + expect(await readFile(keyPath, "utf8")).toBe("\n"); +}); + +// Negative control: replacing createPrivateFileExclusive with Bun.write makes +// this test fail, because Bun.write follows the dangling link and creates it. +test("exclusive private-file creation never follows a dangling symlink", async () => { + const sandbox = await mkdtemp(join(tmpdir(), "hack-private-file-")); + tempDirs.add(sandbox); + const leakPath = resolve(sandbox, "leak.txt"); + const linkPath = resolve(sandbox, PROJECT_ENV_KEY_FILENAME); + await symlink("leak.txt", linkPath); + + await expect( + createPrivateFileExclusive({ path: linkPath, text: `${KEY_SENTINEL}\n` }) + ).rejects.toMatchObject({ code: "EEXIST" }); + expect(await Bun.file(leakPath).exists()).toBe(false); + expect((await lstat(linkPath)).isSymbolicLink()).toBe(true); +}); diff --git a/tests/secret-store.test.ts b/tests/secret-store.test.ts index 0a41f02c0..6db50d880 100644 --- a/tests/secret-store.test.ts +++ b/tests/secret-store.test.ts @@ -300,6 +300,158 @@ test("encrypted_file backend returns recovery guidance when key material is miss await expect(access(keyPath)).rejects.toThrow(); }); +const STORE_KEY = "synthetic-store-key-for-tests-only"; +const STORE_SENTINEL = "synthetic-store-value-never-in-errors"; + +type StoreEnvelope = { + readonly version: number; + readonly algorithm: string; + readonly iv: string; + readonly tag: string; + readonly ciphertext: string; +}; + +/** Writes one sentinel entry through the real store and returns its envelope. */ +async function encryptedStoreFixture(): Promise<{ + readonly storePath: string; + readonly envelope: StoreEnvelope; +}> { + if (!tempDir) { + throw new Error("Missing temp dir"); + } + process.env.HACK_SECRETS_FILE_KEY = STORE_KEY; + const storePath = join(tempDir, "secrets.enc.json"); + await writeGlobalConfig({ + value: { + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath }, + }, + }, + }, + }); + const store = await resolveSecretStore({ projectName: "app" }); + await store.set({ key: "TOKEN", value: STORE_SENTINEL }); + await store.set({ key: "EMPTY", value: "" }); + const envelope = JSON.parse( + await readFile(storePath, "utf8") + ) as StoreEnvelope; + return { storePath, envelope }; +} + +function reencode( + envelope: StoreEnvelope, + field: "iv" | "tag" | "ciphertext", + change: (bytes: Buffer) => Buffer +): StoreEnvelope { + return { + ...envelope, + [field]: change(Buffer.from(envelope[field], "base64")).toString("base64"), + }; +} + +const STORE_MUTATIONS: ReadonlyArray< + readonly [string, (envelope: StoreEnvelope) => StoreEnvelope] +> = [ + ...[4, 8, 12].map( + (bytes) => + [ + `${bytes}-byte tag`, + (envelope: StoreEnvelope) => + reencode(envelope, "tag", (tag) => tag.subarray(0, bytes)), + ] as const + ), + [ + "16-byte IV", + (envelope) => + reencode(envelope, "iv", (iv) => Buffer.concat([iv, Buffer.alloc(4)])), + ], + [ + "base64 with !", + (envelope) => ({ + ...envelope, + ciphertext: `!${envelope.ciphertext.slice(1)}`, + }), + ], + [ + "base64 with whitespace", + (envelope) => ({ + ...envelope, + tag: `${envelope.tag.slice(0, 4)}\n${envelope.tag.slice(4)}`, + }), + ], + [ + "truncated ciphertext", + (envelope) => + reencode(envelope, "ciphertext", (ciphertext) => + ciphertext.subarray(0, -1) + ), + ], + [ + "bit-flipped ciphertext", + (envelope) => + reencode(envelope, "ciphertext", (ciphertext) => { + const flipped = Buffer.from(ciphertext); + flipped[0] = (flipped[0] ?? 0) ^ 1; + return flipped; + }), + ], + [ + "oversized ciphertext", + (envelope) => ({ + ...envelope, + ciphertext: Buffer.alloc(16 * 1024 * 1024 + 1, 7).toString("base64"), + }), + ], +]; + +for (const [name, mutate] of STORE_MUTATIONS) { + test(`encrypted_file envelope with ${name} refuses with a fixed redacted error`, async () => { + const { storePath, envelope } = await encryptedStoreFixture(); + const mutated = mutate(envelope); + expect(mutated).not.toEqual(envelope); + await writeFile(storePath, `${JSON.stringify(mutated)}\n`); + + const store = await resolveSecretStore({ projectName: "app" }); + const error = await store.get({ key: "TOKEN" }).then( + () => null, + (caught: unknown) => caught + ); + + expect(error).toBeInstanceOf(Error); + const message = error instanceof Error ? error.message : ""; + expect(message).toMatch( + /^(Invalid encrypted secret store payload|Cannot decrypt encrypted secret store) at / + ); + for (const privateText of [STORE_SENTINEL, STORE_KEY]) { + expect(String(error)).not.toContain(privateText); + } + }); +} + +test("encrypted_file envelope with the wrong key refuses with a fixed redacted error", async () => { + await encryptedStoreFixture(); + process.env.HACK_SECRETS_FILE_KEY = "synthetic-wrong-store-key"; + const store = await resolveSecretStore({ projectName: "app" }); + const error = await store.get({ key: "TOKEN" }).then( + () => null, + (caught: unknown) => caught + ); + expect(error).toBeInstanceOf(Error); + expect(String(error)).toContain("Cannot decrypt encrypted secret store at "); + expect(String(error)).not.toContain(STORE_SENTINEL); +}); + +test("encrypted_file v1 envelope round-trips, including the empty value", async () => { + const { envelope } = await encryptedStoreFixture(); + expect(Buffer.from(envelope.iv, "base64").length).toBe(12); + expect(Buffer.from(envelope.tag, "base64").length).toBe(16); + const store = await resolveSecretStore({ projectName: "app" }); + expect(await store.get({ key: "TOKEN" })).toBe(STORE_SENTINEL); + expect(await store.get({ key: "EMPTY" })).toBe(""); +}); + async function writeGlobalConfig(input: { readonly value: Record; }): Promise { From ffeecce2c17b41c301fc9d4dcaf9664cda7cb148 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:50:29 -0400 Subject: [PATCH 04/32] fix(agents): generate capability matrix doc and parse all guidance config Address verifier findings on the agent-client capability matrix: - Move the matrix renderer into scripts/generate-agent-client-capabilities.ts and add `bun run docs:agent-client-capabilities` (with `--check`), which rewrites only the generated section. The test compares the committed section with the rendered one and covers marker failures with closed codes. - Scope the model/permission policy scan to generated guidance surfaces (plugins/hack, .claude-plugin, .cursor-plugin, .agents through its skill symlinks, and the example agent configuration) instead of example compose files. Parse YAML and markdown frontmatter as YAML, fail closed on structured-config formats the scan cannot parse, and count a file as scanned only when it was parsed. - Treat reasoning-effort keys (reasoning_effort, model_reasoning_effort, frontmatter `effort`, nested effort/reasoning keys) as forbidden model settings, with negative controls. Co-Authored-By: Claude Opus 5.5 --- docs/agent-guidance.md | 12 +- docs/reference/agent-client-capabilities.md | 28 ++- package.json | 1 + scripts/generate-agent-client-capabilities.ts | 219 +++++++++++++++++ src/agents/client-capabilities.ts | 3 +- tests/agent-client-capabilities.test.ts | 179 ++++++-------- tests/agent-plugin-generation.test.ts | 230 +++++++++++++++--- 7 files changed, 515 insertions(+), 157 deletions(-) create mode 100644 scripts/generate-agent-client-capabilities.ts diff --git a/docs/agent-guidance.md b/docs/agent-guidance.md index 2e7c1a2eb..c6b9cbd2c 100644 --- a/docs/agent-guidance.md +++ b/docs/agent-guidance.md @@ -3,10 +3,11 @@ Hack owns Hack-specific instructions. User and project policy continues to own model selection, general coding standards, work-unit planning, evolution, and writeback. Do not ship provider model pins or copy global instruction packs into this plugin. -`tests/agent-plugin-generation.test.ts` enforces this for every file under -`plugins/hack` and `examples/basic`: no model or model-override keys, no permission, -approval or sandbox overrides, no output-style settings, and hooks that run only -`hack agent prime`. See [model-policy ownership](reference/agent-client-capabilities.md#model-policy-ownership). +`tests/agent-plugin-generation.test.ts` enforces this for the generated guidance +surfaces (`plugins/hack`, the client marketplace manifests, `.agents`, and the +example agent configuration): no model, model-override or reasoning-effort keys, no +permission, approval or sandbox overrides, no output-style settings, and hooks that +run only `hack agent prime`. See [model-policy ownership](reference/agent-client-capabilities.md#model-policy-ownership). ## Sources and installed surfaces @@ -48,7 +49,8 @@ belong to their client adapters; shared behavior belongs to the canonical source generation tests. Run normal typecheck/check/test gates for behavior changes. New packaging needs fresh client loading proof beyond a successful marketplace listing. Record that proof, or the open gate, in `src/agents/client-capabilities.ts` and - regenerate the [capability matrix](reference/agent-client-capabilities.md). + run `bun run docs:agent-client-capabilities` to regenerate the + [capability matrix](reference/agent-client-capabilities.md). 5. Release preparation regenerates plugin assets after the version bump. Verify the published bundle and installed executable separately. diff --git a/docs/reference/agent-client-capabilities.md b/docs/reference/agent-client-capabilities.md index 638c81dbd..8c408b2c1 100644 --- a/docs/reference/agent-client-capabilities.md +++ b/docs/reference/agent-client-capabilities.md @@ -2,8 +2,10 @@ This page classifies what Hack's optional agent integrations are known to do in each client. The data lives in `src/agents/client-capabilities.ts`. -`tests/agent-client-capabilities.test.ts` checks the generated section below -against that data and enforces the classification rules. +`scripts/generate-agent-client-capabilities.ts` renders the generated section +below from that data. `tests/agent-client-capabilities.test.ts` fails when the +committed section differs from the rendered one, and enforces the classification +rules. ## Classification rules @@ -30,9 +32,11 @@ cover other machines or later client versions. A status for one client implies nothing about another client, including UI parity. To reclassify a cell, update the data, increment -`AGENT_CLIENT_CAPABILITY_MATRIX_VERSION`, and regenerate the section below so the -test passes. Promote a cell to `verified` only after its named test passes against -the recorded client version. +`AGENT_CLIENT_CAPABILITY_MATRIX_VERSION`, and run +`bun run docs:agent-client-capabilities` to rewrite the section below. Add +`-- --check` to report a stale section without writing. Edit only the prose +outside the generated markers by hand. Promote a cell to `verified` only after its +named test passes against the recorded client version. @@ -133,11 +137,17 @@ Decisions: The user's own provider configuration and project policy own these settings. Hack artifacts must not set them: -- model selection, model overrides and reasoning effort +- model selection, model overrides and reasoning effort (for example + `model_reasoning_effort`, `reasoning_effort`, or `effort` in skill frontmatter) - permission modes, tool allowlists, approval policies and sandbox modes - output styles Hack-shipped hooks may run only `hack agent prime`. -`tests/agent-plugin-generation.test.ts` enforces this for every file under -`plugins/hack` and `examples/basic`. It includes a negative control that proves the -check rejects each forbidden setting. See [maintaining agent guidance](../agent-guidance.md). +`tests/agent-plugin-generation.test.ts` enforces this for the generated guidance +surfaces: `plugins/hack`, `.claude-plugin`, `.cursor-plugin`, `.agents` (including +the skills it links to), and the agent configuration in `examples/basic/.claude`, +`.codex` and `.cursor`. It parses JSON, JSON Lines, TOML, YAML and markdown +frontmatter. It fails on structured-config formats it cannot parse, such as INI, +JSONC, XML or `.env`, and counts a file as scanned only when it parsed it. +A negative control proves the check rejects each forbidden setting. See +[maintaining agent guidance](../agent-guidance.md). diff --git a/package.json b/package.json index 73053cacc..02d64946b 100644 --- a/package.json +++ b/package.json @@ -72,6 +72,7 @@ "test:e2e:dispatch": "bun run scripts/dispatch-e2e.ts", "test:e2e:railway": "bun run scripts/railway-node-e2e.ts", "docs:cli-reference": "bun run scripts/generate-cli-reference.ts", + "docs:agent-client-capabilities": "bun run scripts/generate-agent-client-capabilities.ts", "generate:agent-plugins": "bun run scripts/generate-agent-plugins.ts" }, "devDependencies": { diff --git a/scripts/generate-agent-client-capabilities.ts b/scripts/generate-agent-client-capabilities.ts new file mode 100644 index 000000000..2ad6a7af7 --- /dev/null +++ b/scripts/generate-agent-client-capabilities.ts @@ -0,0 +1,219 @@ +import { resolve } from "node:path"; + +import { + AGENT_CLIENT_CAPABILITIES, + AGENT_CLIENT_CAPABILITY_DEFINITIONS, + AGENT_CLIENT_CAPABILITY_MATRIX, + AGENT_CLIENT_CAPABILITY_MATRIX_VERSION, + AGENT_CLIENT_CAPABILITY_OBSERVED_ON, + AGENT_CLIENT_DEFINITIONS, + AGENT_CLIENTS, + type AgentClient, + type AgentClientCapability, + type ClientCapabilityCell, +} from "../src/agents/client-capabilities.ts"; + +/** + * Renders the generated section of docs/reference/agent-client-capabilities.md + * from src/agents/client-capabilities.ts. + * + * Run `bun run docs:agent-client-capabilities` to rewrite the section in place, + * or add `--check` to exit non-zero when the committed section is stale. + * tests/agent-client-capabilities.test.ts fails on the same drift. Only the + * text between the markers is generated; the surrounding prose is hand-written. + */ + +export const AGENT_CLIENT_CAPABILITIES_REFERENCE_PATH = + "docs/reference/agent-client-capabilities.md"; +export const GENERATED_BEGIN = + ""; +export const GENERATED_END = + ""; + +/** Closed failure codes; messages never include document content or paths. */ +export type GeneratedSectionError = + | "missing-markers" + | "duplicate-markers" + | "markers-out-of-order"; + +const GENERATED_SECTION_ERROR_MESSAGES: Readonly< + Record +> = { + "missing-markers": "The reference doc is missing a generated-section marker.", + "duplicate-markers": + "The reference doc has more than one generated-section marker pair.", + "markers-out-of-order": + "The reference doc's generated-section end marker precedes its start marker.", +}; + +export class GeneratedSectionMarkerError extends Error { + readonly code: GeneratedSectionError; + + constructor(code: GeneratedSectionError) { + super(GENERATED_SECTION_ERROR_MESSAGES[code]); + this.name = "GeneratedSectionMarkerError"; + this.code = code; + } +} + +function escapeCell(value: string): string { + return value.replaceAll("|", "\\|").replaceAll("\n", " "); +} + +function clientLabel(client: AgentClient): string { + return ( + AGENT_CLIENT_DEFINITIONS.find((definition) => definition.client === client) + ?.label ?? client + ); +} + +function capabilityLabel(capability: AgentClientCapability): string { + return ( + AGENT_CLIENT_CAPABILITY_DEFINITIONS.find( + (definition) => definition.capability === capability + )?.label ?? capability + ); +} + +function renderCellRow(cell: ClientCapabilityCell): string { + return [ + "", + cell.client, + capabilityLabel(cell.capability), + cell.status, + cell.evidenceClass, + `\`${cell.evidenceRef}\``, + escapeCell(cell.observedClientVersion ?? "none"), + `\`${cell.hostLabel}\``, + `\`${cell.sourceSha.slice(0, 12)}\``, + escapeCell(cell.note), + escapeCell(cell.openGate ?? "none"), + "", + ] + .join(" | ") + .trim(); +} + +/** Renders the generated section, markers included, from the given cells. */ +export function renderAgentClientCapabilitySection({ + cells = AGENT_CLIENT_CAPABILITY_MATRIX, +}: { + readonly cells?: readonly ClientCapabilityCell[]; +} = {}): string { + const lines = [ + GENERATED_BEGIN, + "", + `Matrix version ${AGENT_CLIENT_CAPABILITY_MATRIX_VERSION}, observed ${AGENT_CLIENT_CAPABILITY_OBSERVED_ON}.`, + "", + "### Clients", + "", + "| Client | Manifest |", + "| --- | --- |", + ...AGENT_CLIENT_DEFINITIONS.map( + (definition) => + `| ${escapeCell(definition.label)} | \`${definition.manifestPath}\` |` + ), + "", + "### Capabilities", + "", + "| Capability | Meaning |", + "| --- | --- |", + ...AGENT_CLIENT_CAPABILITY_DEFINITIONS.map( + (definition) => + `| ${definition.label} | ${escapeCell(definition.meaning)} |` + ), + "", + "### Status summary", + "", + `| Capability | ${AGENT_CLIENTS.map(clientLabel).map(escapeCell).join(" | ")} |`, + `| --- | ${AGENT_CLIENTS.map(() => "---").join(" | ")} |`, + ...AGENT_CLIENT_CAPABILITIES.map((capability) => { + const statuses = AGENT_CLIENTS.map( + (client) => + cells.find( + (cell) => cell.client === client && cell.capability === capability + )?.status ?? "missing" + ); + return `| ${capabilityLabel(capability)} | ${statuses.join(" | ")} |`; + }), + "", + "### Cells", + "", + "| Client | Capability | Status | Evidence class | Evidence | Client version | Host | Source | Note | Open gate |", + "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |", + ...cells.map(renderCellRow), + "", + GENERATED_END, + ]; + return lines.join("\n"); +} + +function markerRange(document: string): { start: number; end: number } { + const start = document.indexOf(GENERATED_BEGIN); + const end = document.indexOf(GENERATED_END); + if (start < 0 || end < 0) { + throw new GeneratedSectionMarkerError("missing-markers"); + } + if ( + document.indexOf(GENERATED_BEGIN, start + 1) >= 0 || + document.indexOf(GENERATED_END, end + 1) >= 0 + ) { + throw new GeneratedSectionMarkerError("duplicate-markers"); + } + if (end < start) { + throw new GeneratedSectionMarkerError("markers-out-of-order"); + } + return { start, end: end + GENERATED_END.length }; +} + +/** Returns the committed generated section, markers included. */ +export function extractGeneratedSection(document: string): string { + const { start, end } = markerRange(document); + return document.slice(start, end); +} + +/** Replaces the generated section, keeping all hand-written prose intact. */ +export function replaceGeneratedSection({ + document, + section, +}: { + readonly document: string; + readonly section: string; +}): string { + const { start, end } = markerRange(document); + return document.slice(0, start) + section + document.slice(end); +} + +async function main(): Promise { + const check = process.argv.slice(2).includes("--check"); + const docPath = resolve( + import.meta.dir, + "..", + AGENT_CLIENT_CAPABILITIES_REFERENCE_PATH + ); + const document = await Bun.file(docPath).text(); + const next = replaceGeneratedSection({ + document, + section: renderAgentClientCapabilitySection(), + }); + if (check) { + if (next !== document) { + process.stderr.write( + "The agent client capability reference is stale. Run `bun run docs:agent-client-capabilities`.\n" + ); + process.exit(1); + } + process.stdout.write("agent client capability reference is current\n"); + return; + } + if (next === document) { + process.stdout.write("agent client capability reference is current\n"); + return; + } + await Bun.write(docPath, next); + process.stdout.write(`wrote ${AGENT_CLIENT_CAPABILITIES_REFERENCE_PATH}\n`); +} + +if (import.meta.main) { + await main(); +} diff --git a/src/agents/client-capabilities.ts b/src/agents/client-capabilities.ts index 288b8da81..657ee65f2 100644 --- a/src/agents/client-capabilities.ts +++ b/src/agents/client-capabilities.ts @@ -9,7 +9,8 @@ * marketplace listing is never component-loading evidence. * - For the named native-plugin clients, a component is `unsupported` exactly * when that client's plugin manifest does not declare it. - * - docs/reference/agent-client-capabilities.md matches this data. + * - docs/reference/agent-client-capabilities.md matches this data; rewrite its + * generated section with `bun run docs:agent-client-capabilities`. * * Each cell records the client version, host label and source revision at * which it was classified. Those values describe one observation on one host; diff --git a/tests/agent-client-capabilities.test.ts b/tests/agent-client-capabilities.test.ts index 4c5691c3e..8b1653ea4 100644 --- a/tests/agent-client-capabilities.test.ts +++ b/tests/agent-client-capabilities.test.ts @@ -2,12 +2,18 @@ import { expect, test } from "bun:test"; import { existsSync } from "node:fs"; import { join } from "node:path"; +import { + AGENT_CLIENT_CAPABILITIES_REFERENCE_PATH, + extractGeneratedSection, + GENERATED_BEGIN, + GENERATED_END, + GeneratedSectionMarkerError, + renderAgentClientCapabilitySection, + replaceGeneratedSection, +} from "../scripts/generate-agent-client-capabilities.ts"; import { AGENT_CLIENT_CAPABILITIES, - AGENT_CLIENT_CAPABILITY_DEFINITIONS, AGENT_CLIENT_CAPABILITY_MATRIX, - AGENT_CLIENT_CAPABILITY_MATRIX_VERSION, - AGENT_CLIENT_CAPABILITY_OBSERVED_ON, AGENT_CLIENT_DEFINITIONS, AGENT_CLIENTS, type AgentClient, @@ -19,12 +25,7 @@ import { import { isRecord } from "../src/lib/guards.ts"; const repoRoot = join(import.meta.dir, ".."); -const referencePath = join( - repoRoot, - "docs", - "reference", - "agent-client-capabilities.md" -); +const referencePath = join(repoRoot, AGENT_CLIENT_CAPABILITIES_REFERENCE_PATH); type IssueCode = | "missing-cell" @@ -206,94 +207,6 @@ function codesFor( .map((issue) => issue.code); } -function escapeCell(value: string): string { - return value.replaceAll("|", "\\|").replaceAll("\n", " "); -} - -function clientLabel(client: AgentClient): string { - return ( - AGENT_CLIENT_DEFINITIONS.find((definition) => definition.client === client) - ?.label ?? client - ); -} - -function capabilityLabel(capability: AgentClientCapability): string { - return ( - AGENT_CLIENT_CAPABILITY_DEFINITIONS.find( - (definition) => definition.capability === capability - )?.label ?? capability - ); -} - -const GENERATED_BEGIN = ""; -const GENERATED_END = ""; - -function renderGeneratedSection(): string { - const lines = [ - GENERATED_BEGIN, - "", - `Matrix version ${AGENT_CLIENT_CAPABILITY_MATRIX_VERSION}, observed ${AGENT_CLIENT_CAPABILITY_OBSERVED_ON}.`, - "", - "### Clients", - "", - "| Client | Manifest |", - "| --- | --- |", - ...AGENT_CLIENT_DEFINITIONS.map( - (definition) => - `| ${escapeCell(definition.label)} | \`${definition.manifestPath}\` |` - ), - "", - "### Capabilities", - "", - "| Capability | Meaning |", - "| --- | --- |", - ...AGENT_CLIENT_CAPABILITY_DEFINITIONS.map( - (definition) => - `| ${definition.label} | ${escapeCell(definition.meaning)} |` - ), - "", - "### Status summary", - "", - `| Capability | ${AGENT_CLIENTS.map(clientLabel).map(escapeCell).join(" | ")} |`, - `| --- | ${AGENT_CLIENTS.map(() => "---").join(" | ")} |`, - ...AGENT_CLIENT_CAPABILITIES.map((capability) => { - const statuses = AGENT_CLIENTS.map( - (client) => - AGENT_CLIENT_CAPABILITY_MATRIX.find( - (cell) => cell.client === client && cell.capability === capability - )?.status ?? "missing" - ); - return `| ${capabilityLabel(capability)} | ${statuses.join(" | ")} |`; - }), - "", - "### Cells", - "", - "| Client | Capability | Status | Evidence class | Evidence | Client version | Host | Source | Note | Open gate |", - "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |", - ...AGENT_CLIENT_CAPABILITY_MATRIX.map((cell) => - [ - "", - cell.client, - capabilityLabel(cell.capability), - cell.status, - cell.evidenceClass, - `\`${cell.evidenceRef}\``, - escapeCell(cell.observedClientVersion ?? "none"), - `\`${cell.hostLabel}\``, - `\`${cell.sourceSha.slice(0, 12)}\``, - escapeCell(cell.note), - escapeCell(cell.openGate ?? "none"), - "", - ] - .join(" | ") - .trim() - ), - "", - GENERATED_END, - ]; - return lines.join("\n"); -} - test("every client and capability pair is classified with complete evidence", async () => { expect(AGENT_CLIENT_CAPABILITY_MATRIX).toHaveLength( AGENT_CLIENTS.length * AGENT_CLIENT_CAPABILITIES.length @@ -444,15 +357,18 @@ test("manifest-declared components agree with the matrix", async () => { ).toEqual(["open-gate-missing"]); }); -test("reference doc matches the matrix and records ownership decisions", async () => { +test("committed reference section matches the generator output", async () => { const doc = await Bun.file(referencePath).text(); - const start = doc.indexOf(GENERATED_BEGIN); - const end = doc.indexOf(GENERATED_END); - expect(start).toBeGreaterThanOrEqual(0); - expect(end).toBeGreaterThan(start); - expect(doc.slice(start, end + GENERATED_END.length)).toBe( - renderGeneratedSection() + expect(extractGeneratedSection(doc)).toBe( + renderAgentClientCapabilitySection() ); + expect( + replaceGeneratedSection({ + document: doc, + section: renderAgentClientCapabilitySection(), + }) + ).toBe(doc); + expect(doc).toContain("bun run docs:agent-client-capabilities"); expect(doc).toContain("## Companion-package layout decision"); expect(doc).toContain("## Model-policy ownership"); for (const owner of [ @@ -462,4 +378,59 @@ test("reference doc matches the matrix and records ownership decisions", async ( ]) { expect(doc).toContain(owner); } + const packageJson: unknown = await Bun.file( + join(repoRoot, "package.json") + ).json(); + const scripts = isRecord(packageJson) ? packageJson.scripts : undefined; + expect( + isRecord(scripts) ? scripts["docs:agent-client-capabilities"] : undefined + ).toBe("bun run scripts/generate-agent-client-capabilities.ts"); +}); + +test("negative control: a reclassified cell makes the committed section stale", async () => { + const doc = await Bun.file(referencePath).text(); + const reclassified = replaceCell("claude", "skillDiscovery", { + status: "verified", + openGate: null, + }); + const rendered = renderAgentClientCapabilitySection({ cells: reclassified }); + expect(extractGeneratedSection(doc)).not.toBe(rendered); + const rewritten = replaceGeneratedSection({ + document: doc, + section: rendered, + }); + expect(extractGeneratedSection(rewritten)).toBe(rendered); + expect(rewritten.slice(0, doc.indexOf(GENERATED_BEGIN))).toBe( + doc.slice(0, doc.indexOf(GENERATED_BEGIN)) + ); + expect(rewritten.slice(rewritten.indexOf(GENERATED_END))).toBe( + doc.slice(doc.indexOf(GENERATED_END)) + ); +}); + +test("generator refuses documents without exactly one ordered marker pair", () => { + const section = renderAgentClientCapabilitySection(); + const codeFor = (document: string): string | null => { + try { + replaceGeneratedSection({ document, section }); + return null; + } catch (error) { + return error instanceof GeneratedSectionMarkerError + ? error.code + : "other"; + } + }; + expect(codeFor("# Title\n")).toBe("missing-markers"); + expect(codeFor(`${GENERATED_BEGIN}\n`)).toBe("missing-markers"); + expect( + codeFor( + `${GENERATED_BEGIN}\n${GENERATED_END}\n${GENERATED_BEGIN}\n${GENERATED_END}\n` + ) + ).toBe("duplicate-markers"); + expect(codeFor(`${GENERATED_END}\n${GENERATED_BEGIN}\n`)).toBe( + "markers-out-of-order" + ); + expect(codeFor(`a\n${GENERATED_BEGIN}\nold\n${GENERATED_END}\nb\n`)).toBe( + null + ); }); diff --git a/tests/agent-plugin-generation.test.ts b/tests/agent-plugin-generation.test.ts index 38e4a30a1..c55602276 100644 --- a/tests/agent-plugin-generation.test.ts +++ b/tests/agent-plugin-generation.test.ts @@ -2,6 +2,7 @@ import { afterEach, expect, test } from "bun:test"; import { cp, mkdir, mkdtemp, readdir, rm, symlink } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { parse as parseYaml } from "yaml"; import { generateAgentPlugins, @@ -226,16 +227,60 @@ interface PolicyFinding { | "permission-setting" | "output-style-setting" | "frontmatter-hooks" - | "hook-command"; + | "hook-command" + | "unparsed-config"; +} + +/** How the scan treated one file. Only `parsed` files count as scanned. */ +type PolicyScanKind = "parsed" | "not-config"; + +interface PolicyScan { + readonly kind: PolicyScanKind; + readonly findings: readonly PolicyFinding[]; } const HOOK_COMMAND = "hack agent prime"; -/** Model, permission and output-style policy belongs to the user's provider - * configuration, never to Hack-shipped artifacts. */ +/** + * Generated agent guidance surfaces: the native plugin bundle, the client + * marketplace manifests, the shared `.agents` tree (skills reach it through + * symlinks) and the agent configuration in the basic example. Example compose + * and runtime files are application configuration, not agent guidance. + */ +const GUIDANCE_SURFACES = [ + "plugins/hack", + ".claude-plugin", + ".cursor-plugin", + ".agents", + "examples/basic/.claude", + "examples/basic/.codex", + "examples/basic/.cursor", +] as const; + +/** Structured-config formats the scan cannot parse. Shipping one fails the + * scan instead of counting it as inspected. */ +const UNPARSED_CONFIG_EXTENSIONS = new Set([ + "cfg", + "conf", + "env", + "ini", + "json5", + "jsonc", + "plist", + "properties", + "xml", +]); + +/** Model, reasoning-effort, permission and output-style policy belongs to the + * user's provider configuration, never to Hack-shipped artifacts. */ function forbiddenSettingCode(key: string): PolicyFinding["code"] | null { const normalized = key.toLowerCase().replace(/[^a-z]/g, ""); - if (normalized.startsWith("model") || normalized.endsWith("model")) { + if ( + normalized.startsWith("model") || + normalized.endsWith("model") || + normalized.includes("effort") || + normalized.startsWith("reasoning") + ) { return "model-setting"; } if ( @@ -295,65 +340,110 @@ function structuredPolicyFindings( return findings; } -function frontmatterKeys(text: string): string[] { - const match = /^---\n([\s\S]*?)\n---/.exec(text); - if (!match?.[1]) { - return []; - } - return match[1] - .split("\n") - .map((line) => /^([A-Za-z0-9_-]+):/.exec(line)?.[1]) - .filter((key): key is string => key !== undefined); +const FRONTMATTER = /^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/; + +/** Parses markdown frontmatter as YAML; returns null when there is none. */ +function frontmatter(text: string): unknown { + const match = FRONTMATTER.exec(text); + return match ? parseYaml(match[1] ?? "") : null; +} + +function extensionOf(name: string): string { + const dot = name.lastIndexOf("."); + return dot > 0 ? name.slice(dot + 1).toLowerCase() : ""; } /** Classify one shipped file with closed codes; never report file content. */ -function policyFindings(file: string, text: string): PolicyFinding[] { +function policyScan(file: string, text: string): PolicyScan { const findings: PolicyFinding[] = []; const name = file.split("/").at(-1) ?? file; if (file.split("/").some((part) => /^output[-_]?styles?$/i.test(part))) { findings.push({ file, code: "output-style-setting" }); } - if (name.endsWith(".json")) { - findings.push(...structuredPolicyFindings(file, JSON.parse(text))); - } else if (name.endsWith(".jsonl")) { - for (const line of text.split("\n").filter((entry) => entry.trim())) { - findings.push(...structuredPolicyFindings(file, JSON.parse(line))); + const extension = extensionOf(name); + let parsed: unknown = null; + let kind: PolicyScanKind = "parsed"; + if (extension === "json") { + parsed = JSON.parse(text); + } else if (extension === "jsonl") { + parsed = text + .split("\n") + .filter((line) => line.trim()) + .map((line): unknown => JSON.parse(line)); + } else if (extension === "toml") { + parsed = Bun.TOML.parse(text); + } else if (extension === "yaml" || extension === "yml") { + parsed = parseYaml(text); + } else if (extension === "md" || extension === "mdc") { + parsed = frontmatter(text); + if (parsed === null) { + kind = "not-config"; + } else if (isRecord(parsed) && "hooks" in parsed) { + findings.push({ file, code: "frontmatter-hooks" }); } - } else if (name.endsWith(".toml")) { - findings.push(...structuredPolicyFindings(file, Bun.TOML.parse(text))); - } else if (name.endsWith(".md") || name.endsWith(".mdc")) { - for (const key of frontmatterKeys(text)) { - const code = forbiddenSettingCode(key); - if (code) { - findings.push({ file, code }); - } - if (key.toLowerCase() === "hooks") { - findings.push({ file, code: "frontmatter-hooks" }); - } + } else { + kind = "not-config"; + if ( + UNPARSED_CONFIG_EXTENSIONS.has(extension) || + name.toLowerCase().startsWith(".env") + ) { + findings.push({ file, code: "unparsed-config" }); } } - return findings; + findings.push(...structuredPolicyFindings(file, parsed)); + return { kind, findings }; } -test("plugin bundle and examples never pin models, override permissions or run other hooks", async () => { +function policyFindings(file: string, text: string): PolicyFinding[] { + return [...policyScan(file, text).findings]; +} + +test("generated guidance surfaces never pin models, override permissions or run other hooks", async () => { const findings: PolicyFinding[] = []; - let scanned = 0; - for (const root of ["plugins/hack", "examples/basic"]) { + const parsedFiles: string[] = []; + for (const root of GUIDANCE_SURFACES) { const glob = new Bun.Glob("**/*"); for await (const relative of glob.scan({ cwd: join(repoRoot, root), dot: true, onlyFiles: true, + followSymlinks: true, })) { const file = `${root}/${relative}`; - findings.push( - ...policyFindings(file, await Bun.file(join(repoRoot, file)).text()) + const scan = policyScan( + file, + await Bun.file(join(repoRoot, file)).text() ); - scanned += 1; + findings.push(...scan.findings); + if (scan.kind === "parsed") { + parsedFiles.push(file); + } } } - expect(scanned).toBeGreaterThan(10); expect(findings).toEqual([]); + // Scope guard: every known config surface was actually parsed, including + // YAML and markdown frontmatter reached through the `.agents` symlinks. + for (const file of [ + "plugins/hack/plugin.json", + "plugins/hack/.mcp.json", + "plugins/hack/.codex-plugin/plugin.json", + "plugins/hack/.claude-plugin/plugin.json", + "plugins/hack/.cursor-plugin/plugin.json", + "plugins/hack/hooks/claude-hooks.json", + "plugins/hack/rules/hack.mdc", + "plugins/hack/skills/hack-cli/SKILL.md", + ".claude-plugin/marketplace.json", + ".cursor-plugin/marketplace.json", + ".agents/plugins/marketplace.json", + ".agents/skills/hack-repo-tla/SKILL.md", + ".agents/skills/hack-repo-tla/agents/openai.yaml", + "examples/basic/.claude/settings.json", + "examples/basic/.codex/config.toml", + "examples/basic/.cursor/mcp.json", + ]) { + expect(parsedFiles).toContain(file); + } + expect(parsedFiles.some((file) => file.includes("compose"))).toBe(false); expect( await Bun.file( join(repoRoot, "plugins/hack/hooks/claude-hooks.json") @@ -361,6 +451,35 @@ test("plugin bundle and examples never pin models, override permissions or run o ).toContain(HOOK_COMMAND); }); +test("policy scan counts only parsed files and fails closed on unparsed config", () => { + expect(policyScan("plugins/hack/README.md", "# Hack\n").kind).toBe( + "not-config" + ); + expect(policyScan("skills/x/scripts/run", "#!/bin/sh\n").kind).toBe( + "not-config" + ); + expect(policyScan("skills/x/SKILL.md", "---\nname: x\n---\n").kind).toBe( + "parsed" + ); + expect(policyScan("skills/x/agents/openai.yaml", "a: 1\n").kind).toBe( + "parsed" + ); + for (const file of [ + "plugins/hack/settings.ini", + "plugins/hack/client.conf", + "plugins/hack/config.jsonc", + "plugins/hack/.env", + "plugins/hack/.env.local", + ]) { + const scan = policyScan(file, "model=x\n"); + expect(scan.kind).toBe("not-config"); + expect(scan.findings.map((finding) => finding.code)).toEqual([ + "unparsed-config", + ]); + } + expect(() => policyScan("skills/x/agents/openai.yaml", "a: [\n")).toThrow(); +}); + test("negative control: the policy scan rejects each forbidden setting", () => { const cases: readonly (readonly [string, string, PolicyFinding["code"]])[] = [ ["plugin.json", JSON.stringify({ model: "opus" }), "model-setting"], @@ -371,6 +490,41 @@ test("negative control: the policy scan rejects each forbidden setting", () => { ], ["config.toml", 'model = "gpt-5"\n', "model-setting"], ["config.toml", 'model_reasoning_effort = "high"\n', "model-setting"], + ["config.toml", 'reasoning_effort = "high"\n', "model-setting"], + ["config.toml", 'plan_mode_reasoning_effort = "low"\n', "model-setting"], + [ + "settings.json", + JSON.stringify({ env: { CLAUDE_CODE_EFFORT_LEVEL: "high" } }), + "model-setting", + ], + [ + "settings.json", + JSON.stringify({ reasoning: { effort: "high" } }), + "model-setting", + ], + ["skills/x/SKILL.md", "---\nname: x\neffort: high\n---\n", "model-setting"], + [ + "skills/x/SKILL.md", + "---\nname: x\nmetadata:\n reasoningEffort: high\n---\n", + "model-setting", + ], + ["agents/openai.yaml", "model: gpt-5\n", "model-setting"], + ["agents/openai.yaml", "model_reasoning_effort: high\n", "model-setting"], + [ + "agents/openai.yaml", + "policy:\n approval_policy: never\n", + "permission-setting", + ], + [ + "agents/openai.yml", + "sandbox_mode: danger-full-access\n", + "permission-setting", + ], + [ + "hooks/hooks.yaml", + "hooks:\n Stop:\n - type: command\n command: curl example\n", + "hook-command", + ], ["config.toml", 'approval_policy = "never"\n', "permission-setting"], [ "config.toml", From 56bac3a020b4aae71fbd901b5e127b0053e7a3c1 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:51:10 -0400 Subject: [PATCH 05/32] fix(mcp): make typed refusals value-free and keep unknown-tool in-band Every pre-spawn refusal now comes from a closed catalogue (MCP_REFUSALS): error {code, reason, message} uses an existing code, a closed reason and a fixed message, and stderr repeats that message. Project names, ids, paths and registry values are no longer echoed. Schema refusals name only keys of the tool's own input schema. An unknown tool name returns the SDK's in-band isError text result again instead of a JSON-RPC -32602 protocol error, with a fixed message that omits the requested name. A canary test drives every refusal reason and the unknown-tool path and asserts the canary never appears in the result, with no spawn or audit line. Co-Authored-By: Claude Opus 5.5 --- src/mcp/server.ts | 178 +++++++++++++++++++---------- tests/mcp-session-context.test.ts | 1 + tests/mcp.test.ts | 179 +++++++++++++++++++++++++++++- 3 files changed, 296 insertions(+), 62 deletions(-) diff --git a/src/mcp/server.ts b/src/mcp/server.ts index c4b9b86a3..b3dcefcc2 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -7,7 +7,6 @@ import { CallToolRequestSchema, type CallToolResult, ErrorCode, - McpError, type ServerNotification, type ServerRequest, type ToolAnnotations, @@ -65,6 +64,73 @@ type HackCommandResult = { /** Existing CLI error codes reused for pre-spawn MCP refusals (no new codes). */ type McpRefusalCode = Extract; +/** + * Closed catalogue of pre-spawn refusals. Each reason maps to one existing + * error code and one fixed message. Messages are value-free: they never repeat + * caller-supplied project names, ids, paths, services or argv, nor registry + * values derived from them. `invalid_arguments` may append the rejected schema + * field names, which are drawn only from the tool's own input schema keys. + */ +export const MCP_REFUSALS = Object.freeze({ + invalid_arguments: { + code: "E_USAGE", + message: "Invalid arguments.", + }, + conflicting_project_selection: { + code: "E_USAGE", + message: "Use only one of projectName, repoRoot, or path.", + }, + invalid_project_name: { + code: "E_USAGE", + message: "Invalid projectName: use a name containing letters or digits.", + }, + ambiguous_project: { + code: "E_USAGE", + message: + "Ambiguous project selection: multiple registrations match. Select by path and inspect 'hack projects' before renaming or pruning an entry.", + }, + project_not_found: { + code: "E_PROJECT_NOT_FOUND", + message: "Unknown project. Run 'hack init' or 'hack projects' first.", + }, + path_not_found: { + code: "E_PROJECT_NOT_FOUND", + message: "Path not found.", + }, + path_not_registered: { + code: "E_PROJECT_NOT_FOUND", + message: "Path is outside registered hack projects.", + }, + no_project_context: { + code: "E_PROJECT_NOT_FOUND", + message: "No project context found. Provide projectName or path.", + }, + init_path_required: { + code: "E_USAGE", + message: "Provide repoRoot or path for initialization.", + }, +} as const satisfies Record< + string, + { readonly code: McpRefusalCode; readonly message: string } +>); + +export type McpRefusalReason = keyof typeof MCP_REFUSALS; + +/** Reasons whose message is exactly the catalogue text. */ +type FixedRefusalReason = Exclude; + +const MCP_REFUSAL_REASONS = Object.keys(MCP_REFUSALS) as [ + McpRefusalReason, + ...McpRefusalReason[], +]; + +/** + * In-band result for a `tools/call` naming an unregistered tool. Matches the + * SDK dispatcher's earlier shape (untyped `isError` text carrying the + * InvalidParams code) but never repeats the requested tool name. + */ +export const MCP_UNKNOWN_TOOL_MESSAGE = `MCP error ${ErrorCode.InvalidParams}: Tool not found`; + type ToolResultPayload = HackCommandResult & { readonly ok: boolean; readonly data?: unknown; @@ -72,17 +138,14 @@ type ToolResultPayload = HackCommandResult & { readonly spawned?: false; readonly error?: { readonly code: McpRefusalCode; + readonly reason: McpRefusalReason; readonly message: string; }; }; type ProjectArgsResolution = | { readonly ok: true; readonly args: string[] } - | { - readonly ok: false; - readonly code: McpRefusalCode; - readonly message: string; - }; + | { readonly ok: false; readonly reason: FixedRefusalReason }; const DEFAULT_CMD_TIMEOUT_MS = 120_000; const DEFAULT_LOG_TAIL_EVENTS = 200; @@ -143,6 +206,7 @@ const toolOutputSchema = { error: z .object({ code: z.enum(["E_USAGE", "E_PROJECT_NOT_FOUND"]), + reason: z.enum(MCP_REFUSAL_REASONS), message: z.string(), }) .optional(), @@ -787,10 +851,7 @@ function registerTools(opts: { handler: async (input, extra) => { const selection = toProjectSelection(input); if (!(selection.path || selection.repoRoot)) { - return buildToolError({ - code: "E_USAGE", - message: "Provide repoRoot or path for initialization.", - }); + return buildToolError({ reason: "init_path_required" }); } const resolved = await resolveProjectArgs({ @@ -863,13 +924,19 @@ function registerHackTool(opts: { }): void { const { tool } = opts; const schema = z.object(tool.inputSchema); + const schemaKeys: ReadonlySet = new Set( + Object.keys(tool.inputSchema) + ); const entry: HackToolEntry = { call: async (args, extra) => { const parsed = await schema.safeParseAsync(args ?? {}); if (!parsed.success) { return buildToolError({ - code: "E_USAGE", - message: describeInvalidArguments(parsed.error), + reason: "invalid_arguments", + message: describeInvalidArguments({ + error: parsed.error, + schemaKeys, + }), }); } return await tool.handler(parsed.data, extra); @@ -897,6 +964,10 @@ function registerHackTool(opts: { * `{error, spawned: false}` refusal. Thrown handler failures and output-schema * mismatches keep the SDK's untyped `isError` text result, because they may * occur after a child was spawned. + * + * An unknown tool name keeps the SDK 1.26 contract: an in-band `isError` text + * result (not a JSON-RPC protocol error), with the fixed + * {@link MCP_UNKNOWN_TOOL_MESSAGE} that omits the requested name. */ function installTypedToolDispatch(opts: { readonly server: McpServer; @@ -908,10 +979,7 @@ function installTypedToolDispatch(opts: { async (request, extra): Promise => { const entry = opts.tools.get(request.params.name); if (!entry) { - throw new McpError( - ErrorCode.InvalidParams, - `Tool ${request.params.name} not found` - ); + return buildUntypedToolError(MCP_UNKNOWN_TOOL_MESSAGE); } try { const result = await entry.call(request.params.arguments, extra); @@ -935,19 +1003,29 @@ function installTypedToolDispatch(opts: { ); } -/** Closed, value-free summary: names the rejected fields, never echoes input. */ -function describeInvalidArguments(error: z.ZodError): string { +/** + * Closed, value-free summary: names the rejected fields, never echoes input. + * Field names are kept only when they are keys of the tool's own schema, so a + * caller-chosen key can never reach the message. + */ +function describeInvalidArguments(opts: { + readonly error: z.ZodError; + readonly schemaKeys: ReadonlySet; +}): string { const fields = [ ...new Set( - error.issues + opts.error.issues .map((issue) => issue.path[0]) - .filter((key): key is string => typeof key === "string") + .filter( + (key): key is string => + typeof key === "string" && opts.schemaKeys.has(key) + ) ), ]; const subject = fields.length > 0 ? `Invalid arguments: ${fields.join(", ")}.` - : "Invalid arguments."; + : MCP_REFUSALS.invalid_arguments.message; return `${subject} Identifiers must match ${COMPOSE_IDENTIFIER_PATTERN.source} (branches may also contain "/"); limits must be positive integers within their documented ceiling.`; } @@ -984,19 +1062,27 @@ function buildToolResult(opts: { * `spawned: false` is a guarantee: every caller returns before * `runHackCommand`/`runHackLogTail`, so no argv, audit line or mutation occurs. * `stderr` repeats the message for clients reading the earlier contract. + * The message is the reason's fixed text from {@link MCP_REFUSALS}; only + * `invalid_arguments` may pass its schema-key-only summary instead. */ -function buildToolError(opts: { - readonly code: McpRefusalCode; - readonly message: string; -}): CallToolResult { +function buildToolError( + opts: + | { readonly reason: FixedRefusalReason } + | { readonly reason: "invalid_arguments"; readonly message: string } +): CallToolResult { + const { code } = MCP_REFUSALS[opts.reason]; + const message = + opts.reason === "invalid_arguments" + ? opts.message + : MCP_REFUSALS[opts.reason].message; const payload: ToolResultPayload = { ok: false, command: "", exitCode: 1, stdout: "", - stderr: opts.message, + stderr: message, spawned: false, - error: { code: opts.code, message: opts.message }, + error: { code, reason: opts.reason, message }, }; return { @@ -1097,11 +1183,7 @@ async function resolveProjectArgs(opts: { selection.path, ].filter((value) => value !== undefined); if (picked.length > 1) { - return { - ok: false, - code: "E_USAGE", - message: "Use only one of projectName, repoRoot, or path.", - }; + return { ok: false, reason: "conflicting_project_selection" }; } if (selection.projectName !== undefined) { @@ -1115,11 +1197,7 @@ async function resolveProjectArgs(opts: { if (pathLike) { const absPath = resolve(opts.cwd, pathLike); if (!(await pathExists(absPath))) { - return { - ok: false, - code: "E_PROJECT_NOT_FOUND", - message: `Path not found: ${absPath}`, - }; + return { ok: false, reason: "path_not_found" }; } if (!opts.allowUnregistered) { const allowed = await isPathAllowed({ @@ -1127,11 +1205,7 @@ async function resolveProjectArgs(opts: { registryPath: opts.registryPath, }); if (!allowed) { - return { - ok: false, - code: "E_PROJECT_NOT_FOUND", - message: "Path is outside registered hack projects.", - }; + return { ok: false, reason: "path_not_registered" }; } } return { ok: true, args: [`--path=${absPath}`] }; @@ -1148,11 +1222,7 @@ async function resolveProjectArgs(opts: { registryPath: opts.registryPath, }); if (!allowed) { - return { - ok: false, - code: "E_PROJECT_NOT_FOUND", - message: "No project context found. Provide projectName or path.", - }; + return { ok: false, reason: "no_project_context" }; } } @@ -1165,11 +1235,7 @@ async function resolveProjectNameArgs(opts: { }): Promise { const name = normalizeProjectName(opts.projectName); if (!name) { - return { - ok: false, - code: "E_USAGE", - message: "Invalid projectName: use a name containing letters or digits.", - }; + return { ok: false, reason: "invalid_project_name" }; } try { const project = await resolveRegisteredProjectByName({ @@ -1177,11 +1243,7 @@ async function resolveProjectNameArgs(opts: { registryPath: opts.registryPath, }); if (!project) { - return { - ok: false, - code: "E_PROJECT_NOT_FOUND", - message: `Unknown project "${opts.projectName}". Run 'hack init' or 'hack projects' first.`, - }; + return { ok: false, reason: "project_not_found" }; } return { ok: true, args: [`--project=${name}`] }; } catch (error) { @@ -1189,7 +1251,7 @@ async function resolveProjectNameArgs(opts: { error instanceof AmbiguousProjectNameError || error instanceof AmbiguousProjectIdError ) { - return { ok: false, code: "E_USAGE", message: error.message }; + return { ok: false, reason: "ambiguous_project" }; } throw error; } diff --git a/tests/mcp-session-context.test.ts b/tests/mcp-session-context.test.ts index d9b47f70d..0c6423df0 100644 --- a/tests/mcp-session-context.test.ts +++ b/tests/mcp-session-context.test.ts @@ -138,6 +138,7 @@ test("MCP path authorization uses only that session's registry", async () => { spawned: false, error: { code: "E_PROJECT_NOT_FOUND", + reason: "path_not_registered", message: "Path is outside registered hack projects.", }, }); diff --git a/tests/mcp.test.ts b/tests/mcp.test.ts index 1b0227b26..e6570aa68 100644 --- a/tests/mcp.test.ts +++ b/tests/mcp.test.ts @@ -20,6 +20,9 @@ import { buildRunArgs, createMcpServer, MCP_INPUT_CEILINGS, + MCP_REFUSALS, + MCP_UNKNOWN_TOOL_MESSAGE, + type McpRefusalReason, } from "../src/mcp/server.ts"; import { testIntegration } from "./helpers/ci.ts"; @@ -187,12 +190,27 @@ type ContractFixture = { readonly auditLog: string; }; +type RegistryEntryFixture = { + readonly id: string; + readonly name: string; + readonly repoRoot: string; +}; + +type ContractFixtureOptions = { + /** Extra registry rows beside the registered `demo` project. */ + readonly extraProjects?: readonly RegistryEntryFixture[]; + /** Session cwd relative to the fixture root (default: the demo repo). */ + readonly cwdName?: string; +}; + type StubInvocation = { readonly args: readonly string[]; readonly noInteractive: string | null; }; -async function startContractFixture(): Promise { +async function startContractFixture( + opts: ContractFixtureOptions = {} +): Promise { const root = await mkdtemp(join(tmpdir(), "hack-mcp-contract-")); contractRoots.push(root); const project = join(root, "repo"); @@ -219,9 +237,22 @@ async function startContractFixture(): Promise { projectDirName: ".hack", projectDir, }, + ...(opts.extraProjects ?? []).map((entry) => ({ + id: entry.id, + createdAt: "2026-01-01T00:00:00Z", + name: entry.name, + repoRoot: join(root, entry.repoRoot), + projectDirName: ".hack", + projectDir: join(root, entry.repoRoot, ".hack"), + })), ], }) ); + let cwd = project; + if (opts.cwdName !== undefined) { + cwd = join(root, opts.cwdName); + await mkdir(cwd, { recursive: true }); + } const argvLog = join(root, "argv.jsonl"); const stub = join(root, "hack-stub"); await writeFile( @@ -243,14 +274,14 @@ async function startContractFixture(): Promise { HACK_MCP_COMMAND: stub, }; Reflect.deleteProperty(env, "HACK_NO_INTERACTIVE"); - const server = createMcpServer({ cwd: project, env }); + const server = createMcpServer({ cwd, env }); const client = new Client({ name: "hack-mcp-contract", version: "1" }); contractClients.push(client); const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); await server.connect(serverTransport); await client.connect(clientTransport); - return { client, argvLog, auditLog: join(state, "mcp-audit.log") }; + return { client, argvLog, auditLog: join(state, "mcp-audit.log"), root }; } async function readStubInvocations( @@ -516,7 +547,7 @@ test("project resolution refusals carry existing codes and never spawn", async ( expect(unknown.structuredContent).toMatchObject({ ok: false, spawned: false, - error: { code: "E_PROJECT_NOT_FOUND" }, + error: { code: "E_PROJECT_NOT_FOUND", reason: "project_not_found" }, }); const ambiguous = await fixture.client.callTool({ name: "hack.project.run", @@ -528,6 +559,7 @@ test("project resolution refusals carry existing codes and never spawn", async ( stderr: "Use only one of projectName, repoRoot, or path.", error: { code: "E_USAGE", + reason: "conflicting_project_selection", message: "Use only one of projectName, repoRoot, or path.", }, }); @@ -535,6 +567,145 @@ test("project resolution refusals carry existing codes and never spawn", async ( expect(await Bun.file(fixture.auditLog).exists()).toBe(false); }); +/** + * Lowercase alphanumeric so project-name normalization keeps it intact; any + * echo of caller input (or of registry values it selected) would expose it. + */ +const CANARY = "zqxcanary4821"; + +type CanaryRefusal = { + readonly reason: McpRefusalReason; + readonly name: string; + readonly arguments: Record; + readonly fixture?: ContractFixtureOptions; +}; + +const canaryRefusals: readonly CanaryRefusal[] = [ + { + reason: "invalid_arguments", + name: "hack.project.run", + arguments: { projectName: "demo", service: `-${CANARY}`, cmd: [CANARY] }, + }, + { + reason: "conflicting_project_selection", + name: "hack.project.status", + arguments: { projectName: CANARY, path: `/nonexistent/${CANARY}` }, + }, + { + reason: "invalid_project_name", + name: "hack.project.status", + arguments: { projectName: `%%%${"!".repeat(3)}%%%` }, + }, + { + reason: "project_not_found", + name: "hack.project.status", + arguments: { projectName: CANARY }, + }, + { + reason: "ambiguous_project", + name: "hack.project.status", + arguments: { projectName: CANARY }, + fixture: { + extraProjects: [ + { id: `${CANARY}-a`, name: CANARY, repoRoot: `${CANARY}-a` }, + { id: `${CANARY}-b`, name: CANARY, repoRoot: `${CANARY}-b` }, + ], + }, + }, + { + reason: "ambiguous_project", + name: "hack.project.status", + arguments: { projectName: "idclash" }, + fixture: { + extraProjects: [ + { id: CANARY, name: "idclash", repoRoot: `${CANARY}-a` }, + { id: CANARY, name: "idclash-twin", repoRoot: `${CANARY}-b` }, + ], + }, + }, + { + reason: "path_not_found", + name: "hack.project.status", + arguments: { path: `/nonexistent/${CANARY}` }, + }, + { + reason: "path_not_registered", + name: "hack.project.logs.snapshot", + arguments: { path: ".", service: "api" }, + fixture: { cwdName: CANARY }, + }, + { + reason: "no_project_context", + name: "hack.project.status", + arguments: {}, + fixture: { cwdName: CANARY }, + }, + { + reason: "init_path_required", + name: "hack.project.init", + arguments: { name: CANARY, devHost: `${CANARY}.hack` }, + }, +]; + +test("every typed refusal is value-free: fixed message, closed reason, no canary", async () => { + const covered = new Set(); + for (const refusal of canaryRefusals) { + const fixture = await startContractFixture(refusal.fixture); + const result = await fixture.client.callTool({ + name: refusal.name, + arguments: refusal.arguments, + }); + const expected = MCP_REFUSALS[refusal.reason]; + const structured = result.structuredContent as { + readonly stderr: string; + readonly error: { readonly message: string }; + }; + expect({ reason: refusal.reason, isError: result.isError }).toEqual({ + reason: refusal.reason, + isError: true, + }); + expect(structured).toMatchObject({ + ok: false, + spawned: false, + command: "", + stdout: "", + error: { code: expected.code, reason: refusal.reason }, + }); + if (refusal.reason === "invalid_arguments") { + expect(structured.error.message).toStartWith( + "Invalid arguments: service." + ); + } else { + expect(structured.error.message).toBe(expected.message); + } + expect(structured.stderr).toBe(structured.error.message); + const serialized = JSON.stringify(result); + expect(serialized).not.toContain(CANARY); + expect(serialized).not.toContain("%%%"); + expect(serialized).not.toContain(fixture.root); + expect(await readStubInvocations(fixture.argvLog)).toEqual([]); + expect(await Bun.file(fixture.auditLog).exists()).toBe(false); + covered.add(refusal.reason); + } + const coveredReasons: readonly string[] = [...covered].sort(); + expect(coveredReasons).toEqual(Object.keys(MCP_REFUSALS).sort()); +}); + +test("an unknown tool keeps the in-band isError shape without echoing its name", async () => { + const fixture = await startContractFixture(); + const result = await fixture.client.callTool({ + name: `hack.${CANARY}`, + arguments: { projectName: CANARY }, + }); + expect(result).toEqual({ + content: [{ type: "text", text: MCP_UNKNOWN_TOOL_MESSAGE }], + isError: true, + }); + expect(MCP_UNKNOWN_TOOL_MESSAGE).toBe("MCP error -32602: Tool not found"); + expect(JSON.stringify(result)).not.toContain(CANARY); + expect(await readStubInvocations(fixture.argvLog)).toEqual([]); +}); + const READ_ONLY = { readOnlyHint: true, openWorldHint: false }; const ADDITIVE = { readOnlyHint: false, From cd76b26cff38c860045917c3994e4099137d333e Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:51:41 -0400 Subject: [PATCH 06/32] test(env): pin AES-GCM IV, size and tag-length guards with discriminating tests Seal authentic envelopes with 8- and 16-byte IVs and with ciphertext one byte over each parser bound, so only the parser guards can refuse them, and assert through a createDecipheriv spy that structural refusals happen before any decipher and that successful opens pass authTagLength 16. Covers the legacy project env, encrypted_file store and native env value parsers, with at-bound 12-byte-IV controls. Co-Authored-By: Claude Opus 5.5 --- tests/native-env-values.test.ts | 98 ++++++++++++++ tests/project-env-config.test.ts | 159 ++++++++++++++++++++++- tests/secret-store.test.ts | 212 ++++++++++++++++++++++++++++++- 3 files changed, 467 insertions(+), 2 deletions(-) diff --git a/tests/native-env-values.test.ts b/tests/native-env-values.test.ts index e3aaeef77..4497f14d9 100644 --- a/tests/native-env-values.test.ts +++ b/tests/native-env-values.test.ts @@ -1,4 +1,5 @@ import { afterEach, beforeEach, expect, mock, spyOn, test } from "bun:test"; +import * as nodeCrypto from "node:crypto"; import * as fs from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -531,6 +532,103 @@ for (const kind of ["truncated-tag", "noncanonical-base64"] as const) { }); } +/** + * Seals plaintext exactly as the v1 owner does but with a caller-chosen IV + * length. GCM accepts any non-empty IV, so refusing these authentic values can + * only come from the parser's own IV guard, not from GCM authentication. + */ +function sealed(opts: { + readonly plaintext: string; + readonly ivBytes: number; +}): ProjectEnvStoredValue { + const key = nodeCrypto.createHash("sha256").update(KEY, "utf8").digest(); + const iv = nodeCrypto.randomBytes(opts.ivBytes); + const cipher = nodeCrypto.createCipheriv("aes-256-gcm", key, iv, { + authTagLength: 16, + }); + const ciphertext = Buffer.concat([ + cipher.update(opts.plaintext, "utf8"), + cipher.final(), + ]); + return { + secure: [ + "v1", + iv.toString("base64"), + cipher.getAuthTag().toString("base64"), + ciphertext.toString("base64"), + ].join(":"), + }; +} + +test("a test-sealed 12-byte-IV value decrypts with the tag length pinned (control)", async () => { + const p = await project(); + await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); + await layer(p, "hack.env.default.yaml", { + global: { SECRET: sealed({ plaintext: SENTINEL, ivBytes: 12 }) }, + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + expect((await values(p)).globalEnv).toEqual({ SECRET: SENTINEL }); + expect(decipher).toHaveBeenCalled(); + for (const call of decipher.mock.calls) { + const options: unknown = call[3]; + expect(options).toEqual({ authTagLength: 16 }); + } +}); + +for (const ivBytes of [8, 16]) { + test(`an authentic value sealed with a ${ivBytes}-byte IV refuses before decipher`, async () => { + const p = await project(); + await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); + await layer(p, "hack.env.default.yaml", { + global: { SECRET: sealed({ plaintext: SENTINEL, ivBytes }) }, + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + await refuses(() => values(p)); + expect(decipher).not.toHaveBeenCalled(); + }); +} + +test("a truncated tag refuses before decipher on the native path", async () => { + const p = await project(); + await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); + const authentic = sealed({ plaintext: SENTINEL, ivBytes: 12 }); + if (!(isRecord(authentic) && typeof authentic.secure === "string")) { + throw new Error("Synthetic ciphertext absent"); + } + const [prefix, iv, tag = "", ciphertext] = authentic.secure.split(":"); + await layer(p, "hack.env.default.yaml", { + global: { + SECRET: { + secure: [ + prefix, + iv, + Buffer.from(tag, "base64").subarray(0, 12).toString("base64"), + ciphertext, + ].join(":"), + }, + }, + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + await refuses(() => values(p)); + expect(decipher).not.toHaveBeenCalled(); +}); + +// The native layer read is bounded at 1 MiB, below the encoded size of a +// ciphertext over the 1 MiB parser bound, so this refusal comes from the +// bounded read; the shared parser bound is discriminated on the legacy path. +test("an authentic value over the ciphertext bound refuses without decipher", async () => { + const p = await project(); + await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); + await layer(p, "hack.env.default.yaml", { + global: { + SECRET: sealed({ plaintext: "a".repeat(1024 * 1024 + 1), ivBytes: 12 }), + }, + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + await refuses(() => values(p)); + expect(decipher).not.toHaveBeenCalled(); +}); + test("an encrypted empty value preserves presence", async () => { const p = await project(); await fs.writeFile(join(p, PROJECT_ENV_KEY_FILENAME), KEY); diff --git a/tests/project-env-config.test.ts b/tests/project-env-config.test.ts index ee3da7b8a..b0a555c93 100644 --- a/tests/project-env-config.test.ts +++ b/tests/project-env-config.test.ts @@ -1,4 +1,5 @@ -import { afterEach, expect, test } from "bun:test"; +import { afterEach, expect, spyOn, test } from "bun:test"; +import * as nodeCrypto from "node:crypto"; import { writeFileSync } from "node:fs"; import { mkdir, @@ -1753,3 +1754,159 @@ test("legacy v1 secure values round-trip, including the empty value", async () = expect(resolved?.globalEnv.TOKEN).toBe(ENVELOPE_SENTINEL); expect(resolved?.globalEnv.EMPTY).toBe(""); }); + +/** Byte bound shared by the legacy and native v1 ciphertext parsers. */ +const ENVELOPE_CIPHERTEXT_LIMIT = 1024 * 1024; + +/** + * Seals plaintext exactly as the v1 owner does (AES-256-GCM, SHA-256 of the + * trimmed key text, 16-byte tag) but with a caller-chosen IV length. Node and + * Bun accept any non-empty GCM IV, so a refusal of these authentic envelopes + * can only come from the parser's own IV and size guards, not from GCM + * authentication. + */ +function sealProjectEnvValue(opts: { + readonly plaintext: string; + readonly ivBytes: number; +}): string { + const key = nodeCrypto + .createHash("sha256") + .update(ENVELOPE_KEY, "utf8") + .digest(); + const iv = nodeCrypto.randomBytes(opts.ivBytes); + const cipher = nodeCrypto.createCipheriv("aes-256-gcm", key, iv, { + authTagLength: 16, + }); + const ciphertext = Buffer.concat([ + cipher.update(opts.plaintext, "utf8"), + cipher.final(), + ]); + return envelope({ iv, tag: cipher.getAuthTag(), ciphertext }); +} + +type LegacyTokenOutcome = + | { readonly value: string | undefined } + | { readonly error: unknown }; + +async function resolveLegacyToken(opts: { + readonly repo: Awaited>; +}): Promise { + try { + const resolved = await resolveProjectEnvConfig({ + projectRoot: opts.repo.projectRoot, + projectDir: opts.repo.projectDir, + envName: null, + serviceNames: ["api"], + }); + return { value: resolved?.globalEnv.TOKEN }; + } catch (error: unknown) { + return { error }; + } +} + +function expectFixedEnvelopeRefusal(outcome: LegacyTokenOutcome): void { + const error = "error" in outcome ? outcome.error : null; + expect(error).toBeInstanceOf(Error); + expect(error instanceof Error ? error.message : "").toBe( + "Invalid secure env value." + ); + for (const privateText of [ENVELOPE_SENTINEL, ENVELOPE_KEY]) { + expect(String(error)).not.toContain(privateText); + } +} + +test("legacy test-sealed 12-byte-IV value at the ciphertext bound decrypts with the tag length pinned (control)", async () => { + const { repo } = await encryptedEnvelopeFixture(); + const plaintext = `${ENVELOPE_SENTINEL}${"a".repeat( + ENVELOPE_CIPHERTEXT_LIMIT - ENVELOPE_SENTINEL.length + )}`; + await writeSecureValue({ + repo, + secure: sealProjectEnvValue({ plaintext, ivBytes: 12 }), + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + const outcome = await resolveLegacyToken({ repo }); + expect("value" in outcome ? outcome.value : null).toBe(plaintext); + // The tag length is pinned by option rather than left to the runtime default. + expect(decipher).toHaveBeenCalled(); + for (const call of decipher.mock.calls) { + const options: unknown = call[3]; + expect(options).toEqual({ authTagLength: 16 }); + } + } finally { + decipher.mockRestore(); + } +}); + +for (const ivBytes of [8, 16]) { + test(`legacy authentic value sealed with a ${ivBytes}-byte IV refuses before decipher`, async () => { + const { repo } = await encryptedEnvelopeFixture(); + await writeSecureValue({ + repo, + secure: sealProjectEnvValue({ plaintext: ENVELOPE_SENTINEL, ivBytes }), + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + expectFixedEnvelopeRefusal(await resolveLegacyToken({ repo })); + expect(decipher).not.toHaveBeenCalled(); + } finally { + decipher.mockRestore(); + } + }); +} + +test("legacy authentic value one byte over the ciphertext bound refuses before decipher", async () => { + const { repo } = await encryptedEnvelopeFixture(); + await writeSecureValue({ + repo, + secure: sealProjectEnvValue({ + plaintext: "a".repeat(ENVELOPE_CIPHERTEXT_LIMIT + 1), + ivBytes: 12, + }), + }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + expectFixedEnvelopeRefusal(await resolveLegacyToken({ repo })); + expect(decipher).not.toHaveBeenCalled(); + } finally { + decipher.mockRestore(); + } +}); + +// Structural mutations must be refused by the explicit parser guards before any +// decipher exists, so the refusal does not depend on runtime GCM defaults. +for (const name of [ + "4-byte tag", + "8-byte tag", + "12-byte tag", + "16-byte IV", + "base64 with !", + "base64 with whitespace", + "oversized ciphertext", +]) { + test(`legacy secure env value with ${name} refuses before decipher`, async () => { + const mutate = ENVELOPE_MUTATIONS.find(([label]) => label === name)?.[1]; + if (!mutate) { + throw new Error("missing envelope mutation"); + } + const { repo, secure } = await encryptedEnvelopeFixture(); + await writeSecureValue({ repo, secure: mutate(secure) }); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + expectFixedEnvelopeRefusal(await resolveLegacyToken({ repo })); + expect(decipher).not.toHaveBeenCalled(); + } finally { + decipher.mockRestore(); + } + }); +} + +test("legacy secure env value with a 16-byte tag of the wrong value refuses", async () => { + const { repo, secure } = await encryptedEnvelopeFixture(); + const parts = envelopeParts(secure); + const tag = Buffer.from(parts.tag); + tag[15] = (tag[15] ?? 0) ^ 1; + await writeSecureValue({ repo, secure: envelope({ ...parts, tag }) }); + expectFixedEnvelopeRefusal(await resolveLegacyToken({ repo })); +}); diff --git a/tests/secret-store.test.ts b/tests/secret-store.test.ts index 6db50d880..bf3e30a12 100644 --- a/tests/secret-store.test.ts +++ b/tests/secret-store.test.ts @@ -1,4 +1,5 @@ -import { afterEach, beforeEach, expect, test } from "bun:test"; +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import * as nodeCrypto from "node:crypto"; import { access, mkdir, @@ -452,6 +453,215 @@ test("encrypted_file v1 envelope round-trips, including the empty value", async expect(await store.get({ key: "EMPTY" })).toBe(""); }); +/** Byte bound of the encrypted_file ciphertext parser. */ +const STORE_CIPHERTEXT_LIMIT = 16 * 1024 * 1024; + +function storeKey(): Buffer { + return nodeCrypto.createHash("sha256").update(STORE_KEY, "utf8").digest(); +} + +/** Opens a store envelope the way the v1 owner does, to reuse its plaintext. */ +function openStoreEnvelope(envelope: StoreEnvelope): string { + const decipher = nodeCrypto.createDecipheriv( + "aes-256-gcm", + storeKey(), + Buffer.from(envelope.iv, "base64"), + { authTagLength: 16 } + ); + decipher.setAuthTag(Buffer.from(envelope.tag, "base64")); + return Buffer.concat([ + decipher.update(Buffer.from(envelope.ciphertext, "base64")), + decipher.final(), + ]).toString("utf8"); +} + +/** + * Seals a store plaintext exactly as the v1 owner does but with a caller-chosen + * IV length. GCM accepts any non-empty IV, so a refusal of these authentic + * envelopes can only come from the parser's own IV and size guards. + */ +function sealStoreEnvelope(opts: { + readonly plaintext: string; + readonly ivBytes: number; +}): StoreEnvelope { + const iv = nodeCrypto.randomBytes(opts.ivBytes); + const cipher = nodeCrypto.createCipheriv("aes-256-gcm", storeKey(), iv, { + authTagLength: 16, + }); + const ciphertext = Buffer.concat([ + cipher.update(opts.plaintext, "utf8"), + cipher.final(), + ]); + return { + version: 1, + algorithm: "aes-256-gcm", + iv: iv.toString("base64"), + tag: cipher.getAuthTag().toString("base64"), + ciphertext: ciphertext.toString("base64"), + }; +} + +/** Pads the real store plaintext with an extra entry to exactly `bytes` bytes. */ +function storePlaintextOfLength(opts: { + readonly envelope: StoreEnvelope; + readonly bytes: number; +}): string { + const parsed = JSON.parse(openStoreEnvelope(opts.envelope)) as { + readonly version: number; + readonly entries: Record; + }; + const base = JSON.stringify({ + ...parsed, + entries: { ...parsed.entries, PAD: "" }, + }); + const padded = JSON.stringify({ + ...parsed, + entries: { ...parsed.entries, PAD: "a".repeat(opts.bytes - base.length) }, + }); + expect(Buffer.byteLength(padded, "utf8")).toBe(opts.bytes); + return padded; +} + +async function readStoreToken(): Promise< + { readonly value: string | null } | { readonly error: unknown } +> { + try { + const store = await resolveSecretStore({ projectName: "app" }); + return { value: await store.get({ key: "TOKEN" }) }; + } catch (error: unknown) { + return { error }; + } +} + +function expectFixedStoreRefusal( + outcome: { readonly value: string | null } | { readonly error: unknown } +): void { + const error = "error" in outcome ? outcome.error : null; + expect(error).toBeInstanceOf(Error); + expect(error instanceof Error ? error.message : "").toMatch( + /^Invalid encrypted secret store payload at / + ); + for (const privateText of [STORE_SENTINEL, STORE_KEY]) { + expect(String(error)).not.toContain(privateText); + } +} + +test("encrypted_file test-sealed 12-byte-IV envelope at the ciphertext bound opens with the tag length pinned (control)", async () => { + const { storePath, envelope } = await encryptedStoreFixture(); + await writeFile( + storePath, + `${JSON.stringify( + sealStoreEnvelope({ + plaintext: storePlaintextOfLength({ + envelope, + bytes: STORE_CIPHERTEXT_LIMIT, + }), + ivBytes: 12, + }) + )}\n` + ); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + const outcome = await readStoreToken(); + expect("value" in outcome ? outcome.value : null).toBe(STORE_SENTINEL); + // The tag length is pinned by option rather than left to the runtime default. + expect(decipher).toHaveBeenCalled(); + for (const call of decipher.mock.calls) { + const options: unknown = call[3]; + expect(options).toEqual({ authTagLength: 16 }); + } + } finally { + decipher.mockRestore(); + } +}); + +for (const ivBytes of [8, 16]) { + test(`encrypted_file authentic envelope sealed with a ${ivBytes}-byte IV refuses before decipher`, async () => { + const { storePath, envelope } = await encryptedStoreFixture(); + await writeFile( + storePath, + `${JSON.stringify( + sealStoreEnvelope({ plaintext: openStoreEnvelope(envelope), ivBytes }) + )}\n` + ); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + expectFixedStoreRefusal(await readStoreToken()); + expect(decipher).not.toHaveBeenCalled(); + } finally { + decipher.mockRestore(); + } + }); +} + +test("encrypted_file authentic envelope one byte over the ciphertext bound refuses before decipher", async () => { + const { storePath, envelope } = await encryptedStoreFixture(); + await writeFile( + storePath, + `${JSON.stringify( + sealStoreEnvelope({ + plaintext: storePlaintextOfLength({ + envelope, + bytes: STORE_CIPHERTEXT_LIMIT + 1, + }), + ivBytes: 12, + }) + )}\n` + ); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + expectFixedStoreRefusal(await readStoreToken()); + expect(decipher).not.toHaveBeenCalled(); + } finally { + decipher.mockRestore(); + } +}); + +// Structural mutations must be refused by the explicit parser guards before any +// decipher exists, so the refusal does not depend on runtime GCM defaults. +for (const name of [ + "4-byte tag", + "8-byte tag", + "12-byte tag", + "16-byte IV", + "base64 with !", + "base64 with whitespace", + "oversized ciphertext", +]) { + test(`encrypted_file envelope with ${name} refuses before decipher`, async () => { + const mutate = STORE_MUTATIONS.find(([label]) => label === name)?.[1]; + if (!mutate) { + throw new Error("missing store mutation"); + } + const { storePath, envelope } = await encryptedStoreFixture(); + await writeFile(storePath, `${JSON.stringify(mutate(envelope))}\n`); + const decipher = spyOn(nodeCrypto, "createDecipheriv"); + try { + expectFixedStoreRefusal(await readStoreToken()); + expect(decipher).not.toHaveBeenCalled(); + } finally { + decipher.mockRestore(); + } + }); +} + +test("encrypted_file envelope with a 16-byte tag of the wrong value refuses", async () => { + const { storePath, envelope } = await encryptedStoreFixture(); + const tag = Buffer.from(envelope.tag, "base64"); + tag[15] = (tag[15] ?? 0) ^ 1; + await writeFile( + storePath, + `${JSON.stringify({ ...envelope, tag: tag.toString("base64") })}\n` + ); + const outcome = await readStoreToken(); + const error = "error" in outcome ? outcome.error : null; + expect(error).toBeInstanceOf(Error); + expect(String(error)).toContain("Cannot decrypt encrypted secret store at "); + for (const privateText of [STORE_SENTINEL, STORE_KEY]) { + expect(String(error)).not.toContain(privateText); + } +}); + async function writeGlobalConfig(input: { readonly value: Record; }): Promise { From bcf88ec9a35a35225f4882dd5f108c0ae74ca7cf Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 22:05:39 -0400 Subject: [PATCH 07/32] fix(init): stop writing repo-local agent files by default Interactive `hack init` now prints a non-mutating note that points at the optional Hack plugin and `hack agent onboard`. Repository-local export of Cursor rules, Claude hooks, Codex skills, AGENTS.md/CLAUDE.md sections and project MCP config is a separate opt-in whose confirm defaults to no and whose target list starts empty. The decision moves out of project.ts into src/agents/init-agent-integrations.ts. Doctor and the agent-prime freshness inventory now treat absent optional integrations as current. Doctor flags a path only when it exists and lacks the current revision marker (AGENTS.md/CLAUDE.md only when they carry a Hack-managed section). integration-sync counts only stale or errored checks; user-owned docs without a Hack section are not drift; retired legacy artifacts still are. Co-Authored-By: Claude Opus 5.5 --- docs/guides/init-project.md | 18 ++ src/agents/init-agent-integrations.ts | 258 +++++++++++++++++++++ src/cli/integration-sync.ts | 17 +- src/commands/doctor.ts | 73 ++++-- src/commands/project.ts | 155 +------------ src/mcp/agent-docs.ts | 3 + tests/doctor-command.test.ts | 69 ++++++ tests/init-agent-integrations.test.ts | 318 ++++++++++++++++++++++++++ tests/integration-sync.test.ts | 123 ++++++++++ 9 files changed, 866 insertions(+), 168 deletions(-) create mode 100644 src/agents/init-agent-integrations.ts create mode 100644 tests/init-agent-integrations.test.ts create mode 100644 tests/integration-sync.test.ts diff --git a/docs/guides/init-project.md b/docs/guides/init-project.md index 86b91774f..6072fc289 100644 --- a/docs/guides/init-project.md +++ b/docs/guides/init-project.md @@ -41,6 +41,24 @@ Note: - HTTP services via `https://*.hack.local` hostnames (matching your Caddy labels) - non-HTTP services via Compose service hostnames (e.g. `db`, `redis`) +## Coding agents + +Interactive `hack init` does not write coding-agent files by default. It prints a +note recommending the optional Hack plugin for your client (see +[Integrations](../integrations.md#optional-native-plugins)) and +`hack agent onboard` for agent-assisted project setup. Non-interactive runs neither +prompt nor write agent files. + +Repository-local export is a separate opt-in for clients that cannot use the plugin. +Its confirm defaults to "no" and its target list starts empty. Only the targets you +select are written into the repository: `.cursor/rules/hack.mdc`, +`.claude/settings.local.json` plus the `hack-init` skill, `.codex/skills/hack-cli`, +the managed section of `AGENTS.md`/`CLAUDE.md`, or project MCP config. + +`hack doctor` and `hack agent prime --check` treat absent optional integrations as +current. They report review needed only for an installed integration that is stale +or unreadable, or for a retired legacy artifact. + ## Local domain compatibility For new projects using the default `hack.local`, `--oauth` adds diff --git a/src/agents/init-agent-integrations.ts b/src/agents/init-agent-integrations.ts new file mode 100644 index 000000000..ec40da43a --- /dev/null +++ b/src/agents/init-agent-integrations.ts @@ -0,0 +1,258 @@ +import { confirm, isCancel, multiselect, note } from "@clack/prompts"; +import { upsertAgentDocs } from "../mcp/agent-docs.ts"; +import { installMcpConfig, type McpTarget } from "../mcp/install.ts"; +import { logger } from "../ui/logger.ts"; +import { installClaudeHooks } from "./claude.ts"; +import { installCodexSkill } from "./codex-skill.ts"; +import { installCursorRules } from "./cursor.ts"; + +/** + * Coding-agent step of interactive `hack init`. + * + * Contract: the default path is non-mutating. It prints a note that points at + * the optional native plugin and `hack agent onboard`, and writes nothing under + * `.cursor`, `.claude`, `.codex`, `AGENTS.md`, `CLAUDE.md` or `.mcp.json`. + * Repository-local export is a separate, explicit opt-in: its confirm defaults + * to "no" and its target list starts empty, so only targets the user selects + * are written. Non-interactive runs neither prompt nor write. + */ + +/** Repository-local agent files `hack init` can export on explicit request. */ +export type InitAgentExportTarget = + | "cursor" + | "claude" + | "codex" + | "agents" + | "mcp"; + +export type InitAgentExportOption = { + readonly value: InitAgentExportTarget; + readonly label: string; +}; + +export const INIT_AGENT_EXPORT_CONFIRM = { + message: + "Also write Hack agent files into this repository? (most setups only need the plugin)", + initialValue: false, +} as const; + +export const INIT_AGENT_EXPORT_SELECT_MESSAGE = + "Select repository files to write (none selected by default):"; + +export const INIT_AGENT_EXPORT_OPTIONS: readonly InitAgentExportOption[] = [ + { + value: "cursor", + label: "Cursor rules: writes .cursor/rules/hack.mdc into this repository", + }, + { + value: "claude", + label: + "Claude Code hooks: writes .claude/settings.local.json and .claude/skills/hack-init into this repository", + }, + { + value: "codex", + label: "Codex skill: writes .codex/skills/hack-cli into this repository", + }, + { + value: "agents", + label: "Agent docs: writes AGENTS.md and CLAUDE.md into this repository", + }, + { + value: "mcp", + label: + "MCP config (no-shell clients): writes .mcp.json, .cursor/mcp.json or .codex/config.toml into this repository", + }, +]; + +/** Repository writes resolved from an explicit export selection. */ +export type InitAgentExportPlan = { + readonly cursor: boolean; + readonly claude: boolean; + readonly codex: boolean; + readonly agentDocs: boolean; + readonly mcpTargets: readonly McpTarget[]; +}; + +const ALL_MCP_TARGETS: readonly McpTarget[] = ["cursor", "claude", "codex"]; + +/** + * Resolve which repository files an explicit selection writes. An empty + * selection writes nothing. MCP config follows the selected clients, or all + * three clients when MCP is the only client-neutral choice. + */ +export function planInitAgentExport(opts: { + readonly selected: readonly InitAgentExportTarget[]; +}): InitAgentExportPlan { + const selection = new Set(opts.selected); + const clientHints = ALL_MCP_TARGETS.filter((target) => selection.has(target)); + let mcpTargets: readonly McpTarget[] = []; + if (selection.has("mcp")) { + mcpTargets = clientHints.length > 0 ? clientHints : ALL_MCP_TARGETS; + } + return { + cursor: selection.has("cursor"), + claude: selection.has("claude"), + codex: selection.has("codex"), + agentDocs: selection.has("agents"), + mcpTargets, + }; +} + +/** Non-mutating default guidance shown by interactive `hack init`. */ +export function renderInitAgentIntegrationNote(): string { + return [ + "No coding-agent files were written to this repository.", + "Recommended: install the optional Hack plugin for your client, for example:", + " codex plugin add hack@hack-dance", + " claude plugin install hack@hack-dance", + "Cursor: install Hack from Customize. See docs/integrations.md.", + "", + "Then let your agent finish project setup:", + " hack agent onboard", + ].join("\n"); +} + +export type InitAgentIntegrationPrompts = { + readonly note: (opts: { + readonly message: string; + readonly title: string; + }) => void; + readonly confirm: (opts: { + readonly message: string; + readonly initialValue: boolean; + }) => Promise; + readonly multiselect: (opts: { + readonly message: string; + readonly options: readonly InitAgentExportOption[]; + readonly initialValues: readonly InitAgentExportTarget[]; + readonly required: boolean; + }) => Promise; +}; + +export type InitAgentIntegrationOutcome = + | { readonly kind: "skipped" } + | { readonly kind: "note-only" } + | { readonly kind: "exported"; readonly plan: InitAgentExportPlan }; + +const clackPrompts: InitAgentIntegrationPrompts = { + note: ({ message, title }) => note(message, title), + confirm: (opts) => + confirm({ message: opts.message, initialValue: opts.initialValue }), + multiselect: (opts) => + multiselect({ + message: opts.message, + options: opts.options.map((option) => ({ ...option })), + initialValues: [...opts.initialValues], + required: opts.required, + }), +}; + +/** + * Run the coding-agent step of `hack init`. `guard` runs before every write + * (and once up front) so legacy/native project input ownership is rechecked. + */ +export async function runInitAgentIntegrations(opts: { + readonly repoRoot: string; + readonly interactive: boolean; + readonly guard: (opts: { readonly projectRoot: string }) => Promise; + readonly prompts?: InitAgentIntegrationPrompts; +}): Promise { + await opts.guard({ projectRoot: opts.repoRoot }); + if (!opts.interactive) { + return { kind: "skipped" }; + } + const prompts = opts.prompts ?? clackPrompts; + prompts.note({ + message: renderInitAgentIntegrationNote(), + title: "Coding agents", + }); + + const shouldExport = await prompts.confirm(INIT_AGENT_EXPORT_CONFIRM); + if (isCancel(shouldExport) || shouldExport !== true) { + return { kind: "note-only" }; + } + + const selected = await prompts.multiselect({ + message: INIT_AGENT_EXPORT_SELECT_MESSAGE, + options: INIT_AGENT_EXPORT_OPTIONS, + initialValues: [], + required: false, + }); + if (isCancel(selected) || selected.length === 0) { + return { kind: "note-only" }; + } + + const plan = planInitAgentExport({ selected }); + await applyInitAgentExport({ + repoRoot: opts.repoRoot, + plan, + guard: opts.guard, + }); + return { kind: "exported", plan }; +} + +async function applyInitAgentExport(opts: { + readonly repoRoot: string; + readonly plan: InitAgentExportPlan; + readonly guard: (opts: { readonly projectRoot: string }) => Promise; +}): Promise { + const projectRoot = opts.repoRoot; + if (opts.plan.cursor) { + await opts.guard({ projectRoot }); + const result = await installCursorRules({ scope: "project", projectRoot }); + logInstallResult({ label: "Cursor rules", ...result }); + } + if (opts.plan.claude) { + await opts.guard({ projectRoot }); + const result = await installClaudeHooks({ scope: "project", projectRoot }); + logInstallResult({ label: "Claude hooks", ...result }); + } + if (opts.plan.codex) { + await opts.guard({ projectRoot }); + const result = await installCodexSkill({ scope: "project", projectRoot }); + logInstallResult({ label: "Codex skill", ...result }); + } + if (opts.plan.agentDocs) { + await opts.guard({ projectRoot }); + const results = await upsertAgentDocs({ + projectRoot, + targets: ["agents", "claude"], + }); + for (const result of results) { + logInstallResult({ label: "Agent docs", ...result }); + } + } + if (opts.plan.mcpTargets.length > 0) { + await opts.guard({ projectRoot }); + const results = await installMcpConfig({ + targets: opts.plan.mcpTargets, + scope: "project", + projectRoot, + }); + for (const result of results) { + logInstallResult({ + label: "MCP config", + status: result.status, + path: result.path ?? "unknown path", + message: result.message, + }); + } + } +} + +function logInstallResult(opts: { + readonly label: string; + readonly status: string; + readonly path: string; + readonly message?: string; +}): void { + if (opts.status === "error") { + logger.warn({ message: opts.message ?? `Failed to update ${opts.label}` }); + return; + } + if (opts.status === "noop") { + logger.info({ message: `No changes for ${opts.label} (${opts.path})` }); + return; + } + logger.success({ message: `Updated ${opts.label} at ${opts.path}` }); +} diff --git a/src/cli/integration-sync.ts b/src/cli/integration-sync.ts index f3a4e664c..ac72dd368 100644 --- a/src/cli/integration-sync.ts +++ b/src/cli/integration-sync.ts @@ -44,7 +44,7 @@ export function renderAgentIntegrationFreshnessNotice(opts: { } return [ `Hack integration inventory: review needed (CLI v${opts.report.cliVersion}).`, - "Some integrations are missing, stale, or could not be checked. Missing optional integrations need not be installed.", + "Some installed integrations are stale, retired, or could not be checked. Missing optional integrations need not be installed.", `Inspect affected paths: ${opts.report.verifyCommand}`, "Repair only the affected integration and scope authorized for this task.", `For an explicitly requested full refresh: ${opts.report.fixCommand}`, @@ -101,18 +101,27 @@ async function detectIntegrationDrift(opts: { }; } +// Optional integrations are not required: an absent ("missing") integration is +// current. Only present-but-stale or unverifiable ("error") checks are drift. +// Retired legacy artifacts are drift whenever they are present. function hasSingleCheckDrift(status: string): boolean { - return status !== "noop" && status !== "absent"; + return status === "stale" || status === "error"; } function hasMcpDrift(opts: { readonly checks: readonly McpCheckResult[]; }): boolean { - return opts.checks.some((check) => check.status !== "present"); + // MCP config has no revision marker, so only unreadable configs count. + return opts.checks.some((check) => check.status === "error"); } function hasDocDrift(opts: { readonly checks: readonly AgentDocCheckResult[]; }): boolean { - return opts.checks.some((check) => check.status !== "present"); + // A user-owned AGENTS.md/CLAUDE.md without a Hack section is not drift. + return opts.checks.some( + (check) => + check.status === "stale" || + (check.status === "error" && check.unmanaged !== true) + ); } diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index ac9a19896..4b50f6ef4 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -3526,44 +3526,87 @@ async function maybeRepairHackd(): Promise { } } +/** + * Optional integrations are current when absent: a path is stale only when it + * exists and lacks the current revision marker. AGENTS.md/CLAUDE.md are + * user-owned, so they count only when they carry a Hack-managed section. + * Retained legacy artifacts are always stale. + */ export async function inspectDoctorAgentIntegrations(opts: { readonly projectRoot: string | null; readonly homeDir?: string; }): Promise<{ readonly status: "current" | "stale" }> { const home = (opts.homeDir ?? process.env.HOME ?? homedir()).trim(); - const paths = [ + const hackOwnedPaths = [ ...(opts.projectRoot ? [ resolve(opts.projectRoot, ".cursor", "rules", "hack.mdc"), resolve(opts.projectRoot, ".codex", "skills", "hack-cli", "SKILL.md"), - resolve(opts.projectRoot, "AGENTS.md"), - resolve(opts.projectRoot, "CLAUDE.md"), ] : []), resolve(home, ".cursor", "rules", "hack.mdc"), resolve(home, ".codex", "skills", "hack-cli", "SKILL.md"), resolve(home, ".ai", "skills", "hack-cli", "SKILL.md"), ]; + const sharedDocPaths = opts.projectRoot + ? [ + resolve(opts.projectRoot, "AGENTS.md"), + resolve(opts.projectRoot, "CLAUDE.md"), + ] + : []; const marker = `Content revision: \`${HACK_AGENT_INTEGRATION_CONTENT_REVISION}\``; - const [contents, legacyProject, legacyUser] = await Promise.all([ - Promise.all(paths.map((path) => readTextFile(path))), - opts.projectRoot - ? checkLegacyProjectAgentArtifacts({ projectRoot: opts.projectRoot }) - : Promise.resolve([]), - checkLegacyUserAgentArtifacts({ home }), - ]); + const [ownedContents, docContents, legacyProject, legacyUser] = + await Promise.all([ + Promise.all( + hackOwnedPaths.map((path) => readOptionalIntegrationText(path)) + ), + Promise.all( + sharedDocPaths.map((path) => readOptionalIntegrationText(path)) + ), + opts.projectRoot + ? checkLegacyProjectAgentArtifacts({ projectRoot: opts.projectRoot }) + : Promise.resolve([]), + checkLegacyUserAgentArtifacts({ home }), + ]); const hasLegacyArtifacts = [...legacyProject, ...legacyUser].some( (check) => check.status !== "absent" ); + // Present but unreadable paths are treated as stale rather than absent. + const hasStalePath = + ownedContents.some( + (content) => + content.present && + (content.text === null || !content.text.includes(marker)) + ) || + docContents.some( + (content) => + content.present && + (content.text === null || + (content.text.includes(HACK_AGENT_DOCS_SECTION_MARKER) && + !content.text.includes(marker))) + ); return { - status: - contents.every((content) => content?.includes(marker)) && - !hasLegacyArtifacts - ? "current" - : "stale", + status: hasStalePath || hasLegacyArtifacts ? "stale" : "current", }; } +const HACK_AGENT_DOCS_SECTION_MARKER = ""; + +async function readOptionalIntegrationText( + path: string +): Promise< + | { readonly present: false } + | { readonly present: true; readonly text: string | null } +> { + const text = await readTextFile(path); + if (text !== null) { + return { present: true, text }; + } + return (await pathExists(path)) + ? { present: true, text: null } + : { present: false }; +} + async function resolveDaemonReportForDoctorFix(): Promise< ReturnType > { diff --git a/src/commands/project.ts b/src/commands/project.ts index 38e6bf887..2eb690dbf 100644 --- a/src/commands/project.ts +++ b/src/commands/project.ts @@ -21,16 +21,13 @@ import { autocompleteMultiselect, confirm, isCancel, - multiselect, note, password, select, text, } from "@clack/prompts"; import { YAML } from "bun"; -import { installClaudeHooks } from "../agents/claude.ts"; -import { installCodexSkill } from "../agents/codex-skill.ts"; -import { installCursorRules } from "../agents/cursor.ts"; +import { runInitAgentIntegrations } from "../agents/init-agent-integrations.ts"; import { type OnboardingWith, parseOnboardingWith, @@ -274,9 +271,6 @@ import { buildWorktreeRetargetWarning, findSameCheckoutRetargetConflicts, } from "../lib/worktree-runtime-target.ts"; -import { upsertAgentDocs } from "../mcp/agent-docs.ts"; -import type { McpTarget } from "../mcp/install.ts"; -import { installMcpConfig } from "../mcp/install.ts"; import { resolveMux } from "../mux/mux-resolver.ts"; import { renderProjectConfigJson, @@ -3160,7 +3154,11 @@ async function handleInit({ ); } - await maybeSetupAgentIntegrations({ repoRoot }); + await runInitAgentIntegrations({ + repoRoot, + interactive: canPrompt(), + guard: assertLegacyProjectInputFamily, + }); note( [ @@ -3503,147 +3501,6 @@ function validateOauthTld(opts: { readonly value: string }): string | null { return null; } -type SetupIntegration = "cursor" | "claude" | "codex" | "agents" | "mcp"; - -async function maybeSetupAgentIntegrations(opts: { - readonly repoRoot: string; -}): Promise { - await assertLegacyProjectInputFamily({ projectRoot: opts.repoRoot }); - if (!canPrompt()) { - return; - } - const shouldSetup = await confirm({ - message: "Set up coding agent integrations? (Cursor/Claude/Codex)", - initialValue: true, - }); - if (isCancel(shouldSetup) || !shouldSetup) { - return; - } - - const selected = await multiselect({ - message: "Select integrations to install:", - required: true, - options: [ - { value: "cursor", label: "Cursor rules (.cursor/rules/hack.mdc)" }, - { - value: "claude", - label: "Claude Code hooks (.claude/settings.local.json)", - }, - { value: "codex", label: "Codex skill (.codex/skills/hack-cli)" }, - { value: "agents", label: "AGENTS.md / CLAUDE.md snippets" }, - { value: "mcp", label: "MCP config (no-shell clients)" }, - ], - initialValues: ["cursor", "claude", "codex"], - }); - if (isCancel(selected) || selected.length === 0) { - return; - } - - const selection = new Set(selected); - - if (selection.has("cursor")) { - await assertLegacyProjectInputFamily({ projectRoot: opts.repoRoot }); - const result = await installCursorRules({ - scope: "project", - projectRoot: opts.repoRoot, - }); - logInstallResult({ - label: "Cursor rules", - status: result.status, - path: result.path, - message: result.message, - }); - } - - if (selection.has("claude")) { - await assertLegacyProjectInputFamily({ projectRoot: opts.repoRoot }); - const result = await installClaudeHooks({ - scope: "project", - projectRoot: opts.repoRoot, - }); - logInstallResult({ - label: "Claude hooks", - status: result.status, - path: result.path, - message: result.message, - }); - } - - if (selection.has("codex")) { - await assertLegacyProjectInputFamily({ projectRoot: opts.repoRoot }); - const result = await installCodexSkill({ - scope: "project", - projectRoot: opts.repoRoot, - }); - logInstallResult({ - label: "Codex skill", - status: result.status, - path: result.path, - message: result.message, - }); - } - - if (selection.has("agents")) { - await assertLegacyProjectInputFamily({ projectRoot: opts.repoRoot }); - const results = await upsertAgentDocs({ - projectRoot: opts.repoRoot, - targets: ["agents", "claude"], - }); - for (const result of results) { - logInstallResult({ - label: "Agent docs", - status: result.status, - path: result.path, - message: result.message, - }); - } - } - - if (selection.has("mcp")) { - await assertLegacyProjectInputFamily({ projectRoot: opts.repoRoot }); - const targetHints = selected.filter( - (value) => value === "cursor" || value === "claude" || value === "codex" - ); - const targets = ( - targetHints.length > 0 ? targetHints : ["cursor", "claude", "codex"] - ) as McpTarget[]; - - const results = await installMcpConfig({ - targets, - scope: "project", - projectRoot: opts.repoRoot, - }); - - for (const result of results) { - logInstallResult({ - label: "MCP config", - status: result.status, - path: result.path ?? "unknown path", - message: result.message, - }); - } - } -} - -function logInstallResult(opts: { - readonly label: string; - readonly status: string; - readonly path: string; - readonly message?: string; -}): void { - if (opts.status === "error") { - logger.warn({ message: opts.message ?? `Failed to update ${opts.label}` }); - return; - } - - if (opts.status === "noop") { - logger.info({ message: `No changes for ${opts.label} (${opts.path})` }); - return; - } - - logger.success({ message: `Updated ${opts.label} at ${opts.path}` }); -} - // Exported for direct unit-testing of the auto (non-interactive) discovery // + validation pipeline without going through the CLI prompt flow. export interface ComposeWizardInput { diff --git a/src/mcp/agent-docs.ts b/src/mcp/agent-docs.ts index eb3e24ca0..7668f8827 100644 --- a/src/mcp/agent-docs.ts +++ b/src/mcp/agent-docs.ts @@ -20,6 +20,8 @@ export type AgentDocCheckResult = { readonly status: "present" | "stale" | "missing" | "error"; readonly path: string; readonly message?: string; + /** Set when the file exists but has no Hack-managed section at all. */ + readonly unmanaged?: true; }; export type AgentDocRemoveResult = { @@ -118,6 +120,7 @@ export async function checkAgentDocs(opts: { status: "error", path, message: "Missing hack agent-docs markers.", + unmanaged: true, }); continue; } diff --git a/tests/doctor-command.test.ts b/tests/doctor-command.test.ts index d67bc805b..9f3f9d32a 100644 --- a/tests/doctor-command.test.ts +++ b/tests/doctor-command.test.ts @@ -300,6 +300,75 @@ test("doctor audits global agent guidance without a project", async () => { } }); +test("doctor treats absent optional agent integrations as current", async () => { + const root = await mkdtemp(join(tmpdir(), "hack-doctor-absent-agents-")); + const home = join(root, "home"); + const project = join(root, "repo"); + const marker = `Content revision: \`${HACK_AGENT_INTEGRATION_CONTENT_REVISION}\``; + try { + await mkdir(home, { recursive: true }); + await mkdir(project, { recursive: true }); + + // Empty project plus empty HOME. + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "current" }); + + // Plugin-only: client plugin state and user-authored docs, but no + // standalone Hack rules, skills or managed doc sections. + const pluginDir = join(home, ".claude", "plugins", "hack"); + await mkdir(pluginDir, { recursive: true }); + await writeFile(join(pluginDir, "plugin.json"), "{}\n"); + await writeFile(join(project, "AGENTS.md"), "# Team notes\n"); + await writeFile(join(project, "CLAUDE.md"), "Use bun.\n"); + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "current" }); + + // A current Hack-managed section stays current. + await writeFile( + join(project, "AGENTS.md"), + `# Team notes\n\n${marker}\n\n` + ); + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "current" }); + + // One present Hack-managed section without the current revision is stale. + await writeFile( + join(project, "AGENTS.md"), + "# Team notes\n\nold\n\n" + ); + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "stale" }); + await writeFile(join(project, "AGENTS.md"), "# Team notes\n"); + + // One present Hack-owned file without the current revision is stale. + const projectRules = join(project, ".cursor", "rules", "hack.mdc"); + await mkdir(join(projectRules, ".."), { recursive: true }); + await writeFile(projectRules, "stale\n"); + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "stale" }); + await writeFile(projectRules, `${marker}\n`); + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "current" }); + + // A retired legacy project artifact is stale even when nothing else is. + await writeFile( + join(project, "CLAUDE.md"), + "\nRetired\n\n" + ); + await expect( + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }) + ).resolves.toEqual({ status: "stale" }); + } finally { + await rm(root, { recursive: true, force: true }); + } +}); + test("doctor guidance routes runtime hygiene drift to projects prune", () => { const guidance = buildDoctorRecoveryGuidance({ results: [ diff --git a/tests/init-agent-integrations.test.ts b/tests/init-agent-integrations.test.ts new file mode 100644 index 000000000..61519df06 --- /dev/null +++ b/tests/init-agent-integrations.test.ts @@ -0,0 +1,318 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, relative } from "node:path"; +import { + INIT_AGENT_EXPORT_CONFIRM, + INIT_AGENT_EXPORT_OPTIONS, + type InitAgentExportTarget, + type InitAgentIntegrationPrompts, + planInitAgentExport, + renderInitAgentIntegrationNote, + runInitAgentIntegrations, +} from "../src/agents/init-agent-integrations.ts"; + +/** + * Every test runs against a fixture project root and a fixture HOME so the + * real installers can execute without touching the contributor's + * ~/.claude, ~/.codex or ~/.cursor configuration. + */ + +const AGENT_PATHS = [ + ".cursor", + ".claude", + ".codex", + "AGENTS.md", + "CLAUDE.md", + ".mcp.json", +] as const; + +let root: string | null = null; +let savedHome: string | undefined; + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "hack-init-agents-")); + await mkdir(join(root, "home"), { recursive: true }); + await mkdir(join(root, "repo", ".hack"), { recursive: true }); + await writeFile( + join(root, "repo", ".hack", "docker-compose.yml"), + "services: {}\n" + ); + savedHome = process.env.HOME; + process.env.HOME = join(root, "home"); +}); + +afterEach(async () => { + if (savedHome === undefined) { + // biome-ignore lint/performance/noDelete: restoring an unset env var + delete process.env.HOME; + } else { + process.env.HOME = savedHome; + } + if (root) { + await rm(root, { recursive: true, force: true }); + root = null; + } +}); + +function fixture(): { readonly repo: string; readonly home: string } { + if (!root) { + throw new Error("fixture not initialized"); + } + return { repo: join(root, "repo"), home: join(root, "home") }; +} + +async function listFiles(dir: string): Promise { + const entries = await readdir(dir, { recursive: true, withFileTypes: true }); + return entries + .filter((entry) => entry.isFile()) + .map((entry) => relative(dir, join(entry.parentPath, entry.name))) + .sort(); +} + +type PromptLog = { + notes: string[]; + confirms: { message: string; initialValue: boolean }[]; + multiselects: { + initialValues: readonly InitAgentExportTarget[]; + required: boolean; + labels: readonly string[]; + }[]; +}; + +/** Simulates a user pressing Enter on every prompt (accepting each default). */ +function defaultAnswerPrompts(log: PromptLog): InitAgentIntegrationPrompts { + return { + note: ({ message }) => { + log.notes.push(message); + }, + confirm: (opts) => { + log.confirms.push({ ...opts }); + return Promise.resolve(opts.initialValue); + }, + multiselect: (opts) => { + log.multiselects.push({ + initialValues: opts.initialValues, + required: opts.required, + labels: opts.options.map((option) => option.label), + }); + return Promise.resolve([...opts.initialValues]); + }, + }; +} + +/** Simulates a user who opts in and selects exactly `selected`. */ +function optInPrompts(opts: { + readonly log: PromptLog; + readonly selected: readonly InitAgentExportTarget[]; +}): InitAgentIntegrationPrompts { + const base = defaultAnswerPrompts(opts.log); + return { + ...base, + confirm: (confirmOpts) => { + opts.log.confirms.push({ ...confirmOpts }); + return Promise.resolve(true); + }, + multiselect: async (selectOpts) => { + await base.multiselect(selectOpts); + return [...opts.selected]; + }, + }; +} + +function emptyLog(): PromptLog { + return { notes: [], confirms: [], multiselects: [] }; +} + +const noopGuard = (): Promise => Promise.resolve(); + +test("default answers print a note and write no agent files", async () => { + const { repo, home } = fixture(); + const log = emptyLog(); + const before = await listFiles(repo); + + const outcome = await runInitAgentIntegrations({ + repoRoot: repo, + interactive: true, + guard: noopGuard, + prompts: defaultAnswerPrompts(log), + }); + + expect(outcome).toEqual({ kind: "note-only" }); + expect(log.notes).toEqual([renderInitAgentIntegrationNote()]); + expect(log.notes[0]).toContain("hack agent onboard"); + expect(log.notes[0]).toContain("plugin"); + expect(log.confirms).toHaveLength(1); + expect(log.confirms[0]?.initialValue).toBe(false); + expect(log.multiselects).toEqual([]); + + expect(await listFiles(repo)).toEqual(before); + for (const path of AGENT_PATHS) { + expect(await Bun.file(join(repo, path)).exists()).toBe(false); + expect( + await readdir(join(repo, path)).then( + () => true, + () => false + ) + ).toBe(false); + } + expect(await listFiles(home)).toEqual([]); +}); + +test("export prompt has no preselection and labels repository writes", () => { + expect(INIT_AGENT_EXPORT_CONFIRM.initialValue).toBe(false); + for (const option of INIT_AGENT_EXPORT_OPTIONS) { + expect(option.label).toContain("into this repository"); + } + expect(INIT_AGENT_EXPORT_OPTIONS.map((option) => option.value)).toEqual([ + "cursor", + "claude", + "codex", + "agents", + "mcp", + ]); +}); + +test("opting in without selecting a target writes nothing", async () => { + const { repo, home } = fixture(); + const log = emptyLog(); + const before = await listFiles(repo); + + const outcome = await runInitAgentIntegrations({ + repoRoot: repo, + interactive: true, + guard: noopGuard, + prompts: optInPrompts({ log, selected: [] }), + }); + + expect(outcome).toEqual({ kind: "note-only" }); + expect(log.multiselects).toHaveLength(1); + expect(log.multiselects[0]?.initialValues).toEqual([]); + expect(log.multiselects[0]?.required).toBe(false); + expect(await listFiles(repo)).toEqual(before); + expect(await listFiles(home)).toEqual([]); +}); + +test("non-interactive init neither prompts nor writes", async () => { + const { repo, home } = fixture(); + const log = emptyLog(); + const before = await listFiles(repo); + let guardCalls = 0; + + const outcome = await runInitAgentIntegrations({ + repoRoot: repo, + interactive: false, + guard: () => { + guardCalls += 1; + return Promise.resolve(); + }, + prompts: defaultAnswerPrompts(log), + }); + + expect(outcome).toEqual({ kind: "skipped" }); + expect(guardCalls).toBe(1); + expect(log).toEqual(emptyLog()); + expect(await listFiles(repo)).toEqual(before); + expect(await listFiles(home)).toEqual([]); +}); + +test("explicit Cursor opt-in writes only the Cursor rules file", async () => { + const { repo, home } = fixture(); + const log = emptyLog(); + const before = await listFiles(repo); + let guardCalls = 0; + + const outcome = await runInitAgentIntegrations({ + repoRoot: repo, + interactive: true, + guard: () => { + guardCalls += 1; + return Promise.resolve(); + }, + prompts: optInPrompts({ log, selected: ["cursor"] }), + }); + + expect(outcome.kind).toBe("exported"); + expect(guardCalls).toBe(2); + expect(await listFiles(repo)).toEqual( + [...before, join(".cursor", "rules", "hack.mdc")].sort() + ); + expect(await listFiles(home)).toEqual([]); +}); + +test("explicit Claude opt-in writes only Claude project files", async () => { + const { repo, home } = fixture(); + const before = await listFiles(repo); + + await runInitAgentIntegrations({ + repoRoot: repo, + interactive: true, + guard: noopGuard, + prompts: optInPrompts({ log: emptyLog(), selected: ["claude"] }), + }); + + const added = (await listFiles(repo)).filter( + (path) => !before.includes(path) + ); + expect(added.length).toBeGreaterThan(0); + for (const path of added) { + expect(path.startsWith(".claude/")).toBe(true); + } + expect(added).toContain(join(".claude", "settings.local.json")); + expect(await listFiles(home)).toEqual([]); +}); + +test("a failing ownership guard prevents every export write", async () => { + const { repo, home } = fixture(); + const before = await listFiles(repo); + let guardCalls = 0; + + await expect( + runInitAgentIntegrations({ + repoRoot: repo, + interactive: true, + guard: () => { + guardCalls += 1; + return guardCalls > 1 + ? Promise.reject(new Error("project input changed")) + : Promise.resolve(); + }, + prompts: optInPrompts({ + log: emptyLog(), + selected: ["cursor", "claude", "codex", "agents", "mcp"], + }), + }) + ).rejects.toThrow("project input changed"); + + expect(await listFiles(repo)).toEqual(before); + expect(await listFiles(home)).toEqual([]); +}); + +test("planInitAgentExport maps selections to exactly the selected writes", () => { + expect(planInitAgentExport({ selected: [] })).toEqual({ + cursor: false, + claude: false, + codex: false, + agentDocs: false, + mcpTargets: [], + }); + expect(planInitAgentExport({ selected: ["agents"] })).toEqual({ + cursor: false, + claude: false, + codex: false, + agentDocs: true, + mcpTargets: [], + }); + expect(planInitAgentExport({ selected: ["claude", "mcp"] })).toEqual({ + cursor: false, + claude: true, + codex: false, + agentDocs: false, + mcpTargets: ["claude"], + }); + expect(planInitAgentExport({ selected: ["mcp"] }).mcpTargets).toEqual([ + "cursor", + "claude", + "codex", + ]); +}); diff --git a/tests/integration-sync.test.ts b/tests/integration-sync.test.ts new file mode 100644 index 000000000..3559c5508 --- /dev/null +++ b/tests/integration-sync.test.ts @@ -0,0 +1,123 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { installCursorRules } from "../src/agents/cursor.ts"; +import { inspectAgentIntegrationFreshness } from "../src/cli/integration-sync.ts"; + +/** + * Freshness treats optional integrations as current when absent. Every test + * uses a fixture project root and fixture HOME, never the contributor's HOME. + */ + +let root: string | null = null; +let savedHome: string | undefined; + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "hack-integration-sync-")); + await mkdir(join(root, "home"), { recursive: true }); + await mkdir(join(root, "repo"), { recursive: true }); + savedHome = process.env.HOME; + process.env.HOME = join(root, "home"); +}); + +afterEach(async () => { + if (savedHome === undefined) { + // biome-ignore lint/performance/noDelete: restoring an unset env var + delete process.env.HOME; + } else { + process.env.HOME = savedHome; + } + if (root) { + await rm(root, { recursive: true, force: true }); + root = null; + } +}); + +function fixture(): { readonly repo: string; readonly home: string } { + if (!root) { + throw new Error("fixture not initialized"); + } + return { repo: join(root, "repo"), home: join(root, "home") }; +} + +async function status(projectRoot: string | null): Promise { + return (await inspectAgentIntegrationFreshness({ projectRoot })).status; +} + +test("empty project and empty HOME are current", async () => { + const { repo } = fixture(); + expect(await status(repo)).toBe("current"); + expect(await status(null)).toBe("current"); +}); + +test("plugin-only setups without standalone files are current", async () => { + const { repo, home } = fixture(); + const pluginDir = join(home, ".claude", "plugins", "hack"); + await mkdir(pluginDir, { recursive: true }); + await writeFile(join(pluginDir, "plugin.json"), "{}\n"); + await writeFile( + join(home, ".claude", "settings.json"), + `${JSON.stringify({ enabledPlugins: { "hack@hack-dance": true } })}\n` + ); + await mkdir(join(home, ".codex"), { recursive: true }); + await writeFile( + join(home, ".codex", "config.toml"), + '[plugins."hack@hack-dance"]\nenabled = true\n' + ); + await writeFile(join(repo, "AGENTS.md"), "# Team notes\n"); + await writeFile(join(repo, "CLAUDE.md"), "Use bun.\n"); + await writeFile( + join(repo, ".mcp.json"), + `${JSON.stringify({ mcpServers: { other: { command: "other" } } })}\n` + ); + + expect(await status(repo)).toBe("current"); +}); + +test("a current installed integration stays current", async () => { + const { repo } = fixture(); + await installCursorRules({ scope: "project", projectRoot: repo }); + expect(await status(repo)).toBe("current"); +}); + +test("one stale present file is stale", async () => { + const { repo } = fixture(); + await installCursorRules({ scope: "project", projectRoot: repo }); + const rules = join(repo, ".cursor", "rules", "hack.mdc"); + await writeFile(rules, `${await Bun.file(rules).text()}\nSTALE-PROBE\n`); + expect(await status(repo)).toBe("stale"); +}); + +test("a stale Hack-managed agent doc section is stale", async () => { + const { repo } = fixture(); + await writeFile( + join(repo, "AGENTS.md"), + "# Team notes\n\nold\n\n" + ); + expect(await status(repo)).toBe("stale"); +}); + +test("an unreadable MCP config counts as drift", async () => { + const { repo } = fixture(); + await writeFile(join(repo, ".mcp.json"), "{ not json\n"); + expect(await status(repo)).toBe("stale"); +}); + +test("a retired legacy artifact is stale", async () => { + const { repo, home } = fixture(); + await writeFile( + join(repo, "CLAUDE.md"), + "\nRetired\n\n" + ); + expect(await status(repo)).toBe("stale"); + + await rm(join(repo, "CLAUDE.md")); + const legacySkillDir = join(home, ".codex", "skills", "hack-tickets"); + await mkdir(legacySkillDir, { recursive: true }); + await writeFile( + join(legacySkillDir, "SKILL.md"), + "---\nname: hack-tickets\n---\n" + ); + expect(await status(null)).toBe("stale"); +}); From c005262b854373c0447f794f5116c5bb50f4ad5b Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 22:13:17 -0400 Subject: [PATCH 08/32] fix(env): resolve --env base and default the same way on every command config plan and config validate only treated the literal "base" as the base selection, so `--env default` (and DEFAULT/none) reached the native compiler as an overlay named "default" while env get, env list and the runtime commands selected base layers. Every CLI command that accepts --env now resolves it through one shared resolver (src/cli/env-selection.ts): omitted keeps the configured default overlay, base/default/none in any case select base layers, other names are normalized, and a non-empty value with no usable name characters is refused with the fixed "Invalid --env value." usage error instead of silently selecting base. Authored native dispatch uses the same resolver, and session workspace names stay env-base. docs/env.md gains a command and delivery matrix with one row per CLI_SPEC command that accepts --env; tests/env-command-matrix.test.ts keeps the rows and selector flags in sync with CLI_SPEC. The canonical agent guidance gains an env-composition section linking to the matrix; the content revision, plugin bundle, repository examples and CLI reference are regenerated. Co-Authored-By: Claude Opus 5.5 --- .codex/skills/hack-cli/SKILL.md | 12 +- .cursor/rules/hack.mdc | 12 +- docs/env.md | 55 ++++- docs/reference/cli.md | 42 ++-- examples/basic/AGENTS.md | 11 +- examples/basic/CLAUDE.md | 11 +- plugins/hack/rules/hack.mdc | 10 +- plugins/hack/skills/hack-cli/SKILL.md | 10 +- src/agents/instruction-source.ts | 12 ++ src/agents/integration-revision.ts | 2 +- src/cli/env-selection.ts | 41 ++++ src/cli/options.ts | 4 +- src/commands/config-plan.ts | 6 +- src/commands/config-validate.ts | 7 +- src/commands/env.ts | 12 +- src/commands/project.ts | 12 +- src/commands/session.ts | 12 +- src/lib/native-compose-command.ts | 6 +- src/lib/project.ts | 14 +- tests/env-command-matrix.test.ts | 184 ++++++++++++++++ tests/env-selection-parity.test.ts | 288 ++++++++++++++++++++++++++ 21 files changed, 688 insertions(+), 75 deletions(-) create mode 100644 src/cli/env-selection.ts create mode 100644 tests/env-command-matrix.test.ts create mode 100644 tests/env-selection-parity.test.ts diff --git a/.codex/skills/hack-cli/SKILL.md b/.codex/skills/hack-cli/SKILL.md index 573e55440..8dd7c9d10 100644 --- a/.codex/skills/hack-cli/SKILL.md +++ b/.codex/skills/hack-cli/SKILL.md @@ -12,9 +12,9 @@ Use `hack` as the primary interface for local-first development. ## Integration freshness -- These instructions were generated by hack CLI v4.1.1; use the content revision and command behavior to assess freshness. +- These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `c2a3f9669bb6` (version alone is not a freshness guarantee). +- Content revision: `c262dbc28ebc` (version alone is not a freshness guarantee). ## Product boundary @@ -144,6 +144,14 @@ Use `hack` as the primary interface for local-first development. - Execute in workspace with injected env: `hack session exec --env qa --service api "bun db:migrate"` - Stop workspace: `hack session stop ` +## Env composition + +- Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last. +- Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. +- Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. +- Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. +- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix + ## Branch instances (parallel envs) - Use a branch instance when you need two versions running at once (PR review, experiments, migrations) or want to keep a stable environment while testing another branch. diff --git a/.cursor/rules/hack.mdc b/.cursor/rules/hack.mdc index 740f7aeec..6795c812a 100644 --- a/.cursor/rules/hack.mdc +++ b/.cursor/rules/hack.mdc @@ -7,9 +7,9 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is ## Integration freshness -- These instructions were generated by hack CLI v4.1.1; use the content revision and command behavior to assess freshness. +- These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `c2a3f9669bb6` (version alone is not a freshness guarantee). +- Content revision: `c262dbc28ebc` (version alone is not a freshness guarantee). ## Product boundary @@ -64,6 +64,14 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - Interactive host shell with injected env: `hack host shell --env --scope `. - Browser/host URL: use `hack open --json` (OAuth aliases are preferred when enabled). Container-to-container traffic should use Compose DNS. +## Env composition + +- Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last. +- Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. +- Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. +- Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. +- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix + ## Agent integration maintenance - Ordinary Hack commands, `hack update`, `hack doctor --fix`, and primer hooks never install plugins or mutate agent integration files. diff --git a/docs/env.md b/docs/env.md index 1d29bafb6..a9e1eed56 100644 --- a/docs/env.md +++ b/docs/env.md @@ -136,7 +136,13 @@ Projects can set a default overlay in `.hack/hack.config.json`: } ``` -Use `--env=base` to bypass that default and read only `.hack/hack.env.default.yaml`. +Use `--env=base` to bypass that default and read only `.hack/hack.env.default.yaml` +(plus local overrides). `default` and `none` are aliases for `base`, in any case, on every +command that accepts `--env`; an overlay file named after one of those spellings is never +selected by name. Other names are normalized (for example ` QA ` selects `qa`), and a +non-empty value with no usable name characters is refused with `Invalid --env value.` +See the [command and delivery matrix](#command-and-delivery-matrix) for what each command +does with the selection. ## Metadata-only planning API @@ -206,6 +212,53 @@ Use `--target compose` if you explicitly want the container-oriented compose vie For host execution, `--scope` means "which env scope should Hack inject", not "where should this run." For example, `hack host exec --scope api -- bun db:migrate` still runs on the host. +## Command and delivery matrix + +Every command that accepts `--env` resolves it with the same rule: omitted selects +the configured default overlay (`env.defaultOverlay`); `base`, `default` or `none` +selects base layers only; any other value selects that normalized overlay name. The +table lists every such command; `tests/env-command-matrix.test.ts` fails when a +command gains or loses `--env` without a matching row, or when a row's selectors +drift from the CLI. Commands not listed here refuse `--env` as an option that is not +valid for them (arguments after a standalone `--` belong to the child command). + +The **Selectors** column lists the env-related flags each command accepts. +`--service` selects a Compose service scope, `--scope` selects the env scope injected +into a host command, and `--target` on env and host commands chooses the `host` or +`compose` env view. On `up`, `down` and `restart`, `--target` is execution routing +(`auto|local|remote`), not an env view. + +| Command | Selectors | Resolution timing | Refresh path | Native and compose limits | +| --- | --- | --- | --- | --- | +| `hack up` | `--env`, `--target` | Once, when services are created; values go into Compose containers and lifecycle hooks/processes, and the selection is recorded for the running stack. | `hack restart` or `hack env apply --service `; editing env files does not change running containers. | Authored native projects receive the same selection before dispatch. | +| `hack down` | `--env`, `--target` | At teardown, for lifecycle down hooks only; omitted reuses a named overlay recorded for the running stack, else the configured default. | None; nothing is delivered to containers. | The native runtime backend refuses `--env`. | +| `hack restart` | `--env`, `--target` | When services are recreated; omitted reuses a named overlay recorded for the running stack, else the configured default, and an explicit `--env` (including `base`) wins. | This is the refresh path for the whole stack or the named services. | Authored native projects receive the same selection before dispatch. | +| `hack run` | `--env` | Per invocation, for the new one-off container. | Each run resolves again. | The native runtime backend refuses `--env`; restart with the selection first. | +| `hack exec` | `--env` | Checked at invocation against the env recorded for the running stack; the command runs with the env the container was created with. | `hack restart --env ` | Compose refuses when the requested env differs from the env recorded for the running stack; the native runtime backend refuses `--env`. | +| `hack env list` | `--env`, `--service` | Per invocation, read-only; secret values are masked unless `--show-secrets`. | Not needed. | Reads files; no runtime required. | +| `hack env get` | `--env`, `--service` | Per invocation; prints exactly one plaintext value to stdout. | Not needed. | Reads files; refuses native project inputs. | +| `hack env explain` | `--env`, `--service`, `--target` | Per invocation; reports source, precedence and delivery without the value. | Not needed. | Reads files; no runtime required. | +| `hack env apply` | `--env`, `--service` | Runs `hack restart ` with the same `--env`, recreating one service. | This is the refresh path for one service. | `--service` is required. | +| `hack env add` | `--env`, `--service` | Selects the file written (`hack.env..yaml`, or the base file for `base`/`default`/`none`; `--local` writes the local variant). | Running workloads keep old values until `hack env apply` or `hack restart`. | Mutates only env files. | +| `hack env set` | `--env`, `--service` | Alias for `hack env add`. | Same as `hack env add`. | Mutates only env files. | +| `hack env unset` | `--env`, `--service` | Selects the file the key is removed from, as for `hack env add`. | Same as `hack env add`. | Mutates only env files. | +| `hack env materialize` | `--env`, `--service` | Writes `.hack/.env` once, at invocation. | Rerun `hack env materialize`. | Compatibility output for external tools; runtime injection does not depend on it. | +| `hack env exec` | `--env`, `--service`, `--target` | Per invocation, into one host command (host view by default). | Each invocation resolves again. | Runs on the host, not in a container. | +| `hack env shell` | `--env`, `--service`, `--target` | When the shell starts. | Start a new shell. | Runs on the host, not in a container. | +| `hack host exec` | `--env`, `--scope`, `--target` | Per invocation, into one host command (host view by default). | Each invocation resolves again. | Runs on the host, not in a container. | +| `hack host shell` | `--env`, `--scope`, `--target` | When the shell starts. | Start a new shell. | Runs on the host, not in a container. | +| `hack config validate` | `--env` | Per invocation; resolves the native selection without reading managed env documents. | Not needed. | Authored native projects only; cannot be combined with `--file`. | +| `hack config plan` | `--env` | Per invocation; metadata only (names, scopes, secret flags), never decrypted. | Not needed. | Authored native projects only. | +| `hack session start` | `--env`, `--service` | When the workspace is created; the name gains `.env-` (`.env-base` for base spellings), and `--up` runs `hack up -d` with the same selection. | Start a new workspace or pass `--env` to `hack session exec`. | Injects into the workspace shell, not into containers. | +| `hack session exec` | `--env`, `--service` | Per command; omitted reuses the selection encoded in the workspace name. | Each command resolves again. | Injects into the workspace command, not into containers. | + +MCP tools accept no env selector. They run the CLI without `--env`, so each row's +omitted-`--env` behavior applies. + +An overlay name is a label for a set of env files. It implies no production authority +and no network or instance isolation: a `prod` overlay still runs on the local stack, +and two overlays share the same instance unless you choose a separate branch instance. + ## Common commands Inspect resolved env: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index a135c0c9d..dddd00ecb 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -107,7 +107,7 @@ hack up [services...] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--branch ` | Run against a branch-specific instance (compose name + hostnames) | | `--detach, -d` | Run in background (docker compose up -d) | | `--profile ` | Enable one or more compose profiles (comma-separated) | @@ -134,7 +134,7 @@ hack down [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--branch ` | Run against a branch-specific instance (compose name + hostnames) | | `--profile ` | Enable one or more compose profiles (comma-separated) | | `--target ` | Execution target routing (auto routes to remote when project execution mode requires it) | @@ -169,7 +169,7 @@ hack restart [services...] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--branch ` | Run against a branch-specific instance (compose name + hostnames) | | `--profile ` | Enable one or more compose profiles (comma-separated) | | `--target ` | Execution target routing (auto routes to remote when project execution mode requires it) | @@ -250,7 +250,7 @@ hack run [cmd...] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--branch ` | Run against a branch-specific instance (compose name + hostnames) | | `--workdir ` | Working directory inside the container (docker compose run -w) | | `--profile ` | Enable one or more compose profiles (comma-separated) | @@ -282,7 +282,7 @@ hack exec [cmd...] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--branch ` | Run against a branch-specific instance (compose name + hostnames) | | `--workdir ` | Working directory inside the container (docker compose run -w) | | `--profile ` | Enable one or more compose profiles (comma-separated) | @@ -593,7 +593,7 @@ Uses the matching bundled Rust compiler. Without --file, discovers a native proj | `--profile ` | Comma-separated declared native profiles | | `--domain ` | Select the generated native routing domain suffix | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--json` | Output JSON (machine-readable) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | | `--help, -h` | Show help | @@ -618,7 +618,7 @@ Resolves the native project, permitted local env selection and routing preview, | `--profile ` | Comma-separated declared native profiles | | `--domain ` | Select the generated native routing domain suffix | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--json` | Output JSON (machine-readable) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | | `--help, -h` | Show help | @@ -752,7 +752,7 @@ Reuse the default project workspace when it already exists, or create an isolate | `--new` | Create an isolated workspace instead of reusing the default project workspace | | `--name ` | Suffix for an isolated workspace name (for example: agent-1 -> project--agent-1) | | `--detach, -d` | Create or reuse the workspace without attaching (for GUI/non-TTY use) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--service ` | Inject env for the selected scope (global or a service name) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | | `--help, -h` | Show help | @@ -831,7 +831,7 @@ Queue a command in the workspace without opening a new interactive attach flow. | Option | Description | | --- | --- | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--service ` | Inject env for the selected scope (global or a service name) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | | `--help, -h` | Show help | @@ -2233,7 +2233,7 @@ hack env list [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--json` | Output JSON (machine-readable) | | `--show-secrets` | Print secret values (keychain) in plaintext | | `--service ` | Target scope (global or a discovered service name) | @@ -2265,7 +2265,7 @@ Read one global or service value without starting a runtime. Empty values succee | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--service ` | Target scope (global or a discovered service name) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | | `--help, -h` | Show help | @@ -2293,7 +2293,7 @@ hack env explain [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--json` | Output JSON (machine-readable) | | `--service ` | Target scope (global or a discovered service name) | | `--target ` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) | @@ -2317,7 +2317,7 @@ hack env apply [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--json` | Output JSON (machine-readable) | | `--service ` | Target scope (global or a discovered service name) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | @@ -2347,7 +2347,7 @@ hack env add [key] [value] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--secret` | Store value as an encrypted secure entry | | `--local` | Write to the worktree-local env override file instead of the shared repo env file | | `--service ` | Target scope (global or a discovered service name) | @@ -2378,7 +2378,7 @@ hack env set [key] [value] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--secret` | Store value as an encrypted secure entry | | `--local` | Write to the worktree-local env override file instead of the shared repo env file | | `--service ` | Target scope (global or a discovered service name) | @@ -2402,7 +2402,7 @@ hack env materialize [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--service ` | Target scope (global or a discovered service name) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | | `--help, -h` | Show help | @@ -2432,7 +2432,7 @@ Inject the selected Hack env overlay directly into a one-off host command withou | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--service ` | Target scope (global or a discovered service name) | | `--target ` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) | | `--shell ` | Run a shell command string via /bin/sh -lc after env injection so `$VAR` expansion happens inside the child shell | @@ -2460,7 +2460,7 @@ Start an interactive host shell with the selected Hack env overlay injected into | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--service ` | Target scope (global or a discovered service name) | | `--target ` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | @@ -2489,7 +2489,7 @@ hack env unset [key] [options] | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--local` | Write to the worktree-local env override file instead of the shared repo env file | | `--service ` | Target scope (global or a discovered service name) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | @@ -2594,7 +2594,7 @@ Run a one-off command on the host with the selected Hack env overlay injected. U | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--scope ` | Resolve values for one env scope while still running the command on the host | | `--target ` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) | | `--shell ` | Run a shell command string via /bin/sh -lc after env injection so `$VAR` expansion happens inside the child shell | @@ -2622,7 +2622,7 @@ Start an interactive host shell with the selected Hack env overlay injected. Use | --- | --- | | `--path, -p ` | Run a project command against a repo path (overrides cwd search) | | `--project ` | Target a registered project by name (from ~/.hack/projects.json) | -| `--env ` | Apply an optional env overlay by name (use 'base' to bypass overlays) | +| `--env ` | Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only | | `--scope ` | Resolve values for one env scope while still running the command on the host | | `--target ` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) | | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) | diff --git a/examples/basic/AGENTS.md b/examples/basic/AGENTS.md index f6d651237..6c61ad366 100644 --- a/examples/basic/AGENTS.md +++ b/examples/basic/AGENTS.md @@ -4,9 +4,9 @@ Use `hack` as the single interface for local-first runtime orchestration (compose, DNS/TLS, logs, env, and persistent project workspaces). Integration freshness: -- These instructions were generated by hack CLI v4.1.1; use the content revision and command behavior to assess freshness. +- These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `c2a3f9669bb6` (version alone is not a freshness guarantee). +- Content revision: `c262dbc28ebc` (version alone is not a freshness guarantee). Product boundary: - Supported CLI surface: project init, up/down/restart, open, logs, env, host exec/shell, sessions, doctor, and daemon. @@ -122,6 +122,13 @@ Workspaces (mux-managed, tmux-first by default): - Execute in workspace with injected env: `hack session exec --env qa --service api "bun db:migrate"` - Stop workspace: `hack session stop ` +Env composition: +- Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last. +- Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. +- Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. +- Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. +- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix + Branch instances (parallel envs): - Use a branch instance when you need two versions running at once (PR review, experiments, migrations) or want to keep a stable environment while testing another branch. - Target one with `--branch ` on up/open/logs/down (for example: `hack up --branch --detach`). diff --git a/examples/basic/CLAUDE.md b/examples/basic/CLAUDE.md index f6d651237..6c61ad366 100644 --- a/examples/basic/CLAUDE.md +++ b/examples/basic/CLAUDE.md @@ -4,9 +4,9 @@ Use `hack` as the single interface for local-first runtime orchestration (compose, DNS/TLS, logs, env, and persistent project workspaces). Integration freshness: -- These instructions were generated by hack CLI v4.1.1; use the content revision and command behavior to assess freshness. +- These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `c2a3f9669bb6` (version alone is not a freshness guarantee). +- Content revision: `c262dbc28ebc` (version alone is not a freshness guarantee). Product boundary: - Supported CLI surface: project init, up/down/restart, open, logs, env, host exec/shell, sessions, doctor, and daemon. @@ -122,6 +122,13 @@ Workspaces (mux-managed, tmux-first by default): - Execute in workspace with injected env: `hack session exec --env qa --service api "bun db:migrate"` - Stop workspace: `hack session stop ` +Env composition: +- Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last. +- Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. +- Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. +- Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. +- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix + Branch instances (parallel envs): - Use a branch instance when you need two versions running at once (PR review, experiments, migrations) or want to keep a stable environment while testing another branch. - Target one with `--branch ` on up/open/logs/down (for example: `hack up --branch --detach`). diff --git a/plugins/hack/rules/hack.mdc b/plugins/hack/rules/hack.mdc index 59bf46940..cd07cd697 100644 --- a/plugins/hack/rules/hack.mdc +++ b/plugins/hack/rules/hack.mdc @@ -14,7 +14,7 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `c2a3f9669bb6` (version alone is not a freshness guarantee). +- Content revision: `c262dbc28ebc` (version alone is not a freshness guarantee). ## Product boundary @@ -69,6 +69,14 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - Interactive host shell with injected env: `hack host shell --env --scope `. - Browser/host URL: use `hack open --json` (OAuth aliases are preferred when enabled). Container-to-container traffic should use Compose DNS. +## Env composition + +- Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last. +- Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. +- Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. +- Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. +- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix + ## Agent integration maintenance - Ordinary Hack commands, `hack update`, `hack doctor --fix`, and primer hooks never install plugins or mutate agent integration files. diff --git a/plugins/hack/skills/hack-cli/SKILL.md b/plugins/hack/skills/hack-cli/SKILL.md index 56186dfbc..8dd7c9d10 100644 --- a/plugins/hack/skills/hack-cli/SKILL.md +++ b/plugins/hack/skills/hack-cli/SKILL.md @@ -14,7 +14,7 @@ Use `hack` as the primary interface for local-first development. - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `c2a3f9669bb6` (version alone is not a freshness guarantee). +- Content revision: `c262dbc28ebc` (version alone is not a freshness guarantee). ## Product boundary @@ -144,6 +144,14 @@ Use `hack` as the primary interface for local-first development. - Execute in workspace with injected env: `hack session exec --env qa --service api "bun db:migrate"` - Stop workspace: `hack session stop ` +## Env composition + +- Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last. +- Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. +- Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. +- Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. +- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix + ## Branch instances (parallel envs) - Use a branch instance when you need two versions running at once (PR review, experiments, migrations) or want to keep a stable environment while testing another branch. diff --git a/src/agents/instruction-source.ts b/src/agents/instruction-source.ts index 16a6c948c..a5b1d3f4d 100644 --- a/src/agents/instruction-source.ts +++ b/src/agents/instruction-source.ts @@ -230,6 +230,18 @@ export const INSTRUCTION_SECTIONS: readonly InstructionSection[] = [ "Stop workspace: `hack session stop `", ], }, + { + id: "env-composition", + title: "Env composition", + surfaces: ALL_SURFACES, + bullets: [ + "Layer order: `.hack/hack.env.default.yaml`, the selected `.hack/hack.env..yaml`, then worktree-local `.hack/hack.env.local.yaml` and `.hack/hack.env..local.yaml`; service values override `global`, and host commands apply `host` values last.", + "Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation.", + "Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret.", + "Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`.", + "Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix", + ], + }, { id: "branch-instances", diff --git a/src/agents/integration-revision.ts b/src/agents/integration-revision.ts index fbc9c7bdb..19fe0790f 100644 --- a/src/agents/integration-revision.ts +++ b/src/agents/integration-revision.ts @@ -3,4 +3,4 @@ * source test recomputes this value and fails whenever guidance changes * without a revision update. */ -export const HACK_AGENT_INTEGRATION_CONTENT_REVISION = "c2a3f9669bb6"; +export const HACK_AGENT_INTEGRATION_CONTENT_REVISION = "c262dbc28ebc"; diff --git a/src/cli/env-selection.ts b/src/cli/env-selection.ts new file mode 100644 index 000000000..e45b9d08b --- /dev/null +++ b/src/cli/env-selection.ts @@ -0,0 +1,41 @@ +import { + ENV_BASE_SELECTION_ALIASES, + parseEnvConfigSelection, +} from "../lib/project.ts"; +import { CliUsageError } from "./command.ts"; + +const INVALID_ENV_SELECTION_MESSAGE = "Invalid --env value."; + +/** + * Resolve a CLI `--env` value through the one shared selection grammar. + * + * - `undefined` (option omitted): callers apply the configured default overlay. + * - `null`: base layers only. `base`, `default` and `none` (any case) and an + * empty value bypass the configured default overlay. + * - `string`: the canonical overlay name (`normalizeEnvConfigName`). + * + * A non-empty value with no usable overlay characters is refused with a fixed, + * value-free usage error instead of silently selecting the base layers. + * Every CLI command that accepts `--env` must resolve it here so that + * `default`/`base` mean the same layers everywhere. + */ +export function resolveRequestedEnvName(input: { + readonly envOption: string | undefined; +}): string | null | undefined { + const selection = parseEnvConfigSelection(input.envOption); + if (input.envOption === undefined) { + return selection; + } + if (selection === undefined) { + throw new CliUsageError(INVALID_ENV_SELECTION_MESSAGE); + } + const trimmed = input.envOption.trim(); + if ( + selection === null && + trimmed.length > 0 && + !ENV_BASE_SELECTION_ALIASES.has(trimmed.toLowerCase()) + ) { + throw new CliUsageError(INVALID_ENV_SELECTION_MESSAGE); + } + return selection; +} diff --git a/src/cli/options.ts b/src/cli/options.ts index eff1d50be..ceb52ddda 100644 --- a/src/cli/options.ts +++ b/src/cli/options.ts @@ -23,9 +23,9 @@ export const optEnv = defineOption({ name: "env", type: "string", long: "--env", - valueHint: "", + valueHint: "", description: - "Apply an optional env overlay by name (use 'base' to bypass overlays)", + "Select an env overlay by name; omit for the configured default overlay, or use 'base' or 'default' for base layers only", } as const); export const optBranch = defineOption({ diff --git a/src/commands/config-plan.ts b/src/commands/config-plan.ts index 178d513b3..52951068f 100644 --- a/src/commands/config-plan.ts +++ b/src/commands/config-plan.ts @@ -5,6 +5,7 @@ import { defineOption, withHandler, } from "../cli/command.ts"; +import { resolveRequestedEnvName } from "../cli/env-selection.ts"; import { optEnv, optJson, optPath } from "../cli/options.ts"; import { HackCliError } from "../lib/cli-result.ts"; import { @@ -49,6 +50,9 @@ export const configPlanCommand = withHandler(spec, async ({ ctx, args }) => { if (profiles?.some((name) => name.length === 0)) { throw new CliUsageError("Native profile names must not be empty."); } + const explicitOverlay = resolveRequestedEnvName({ + envOption: args.options.env, + }); const controller = new AbortController(); const cancel = () => controller.abort(); process.once("SIGINT", cancel); @@ -58,7 +62,7 @@ export const configPlanCommand = withHandler(spec, async ({ ctx, args }) => { startDir: resolve(ctx.cwd, args.options.path ?? "."), profiles, explicitDomain: args.options.domain, - explicitOverlay: args.options.env === "base" ? null : args.options.env, + explicitOverlay, signal: controller.signal, }); renderPlan({ result, json: args.options.json === true }); diff --git a/src/commands/config-validate.ts b/src/commands/config-validate.ts index 706e13689..3fe36e1df 100644 --- a/src/commands/config-validate.ts +++ b/src/commands/config-validate.ts @@ -5,6 +5,7 @@ import { defineOption, withHandler, } from "../cli/command.ts"; +import { resolveRequestedEnvName } from "../cli/env-selection.ts"; import { optEnv, optJson, optPath } from "../cli/options.ts"; import { HackCliError } from "../lib/cli-result.ts"; import { @@ -77,6 +78,9 @@ export const configValidateCommand = withHandler( if (profiles?.some((name) => name.length === 0)) { throw new CliUsageError("Native profile names must not be empty."); } + const explicitOverlay = resolveRequestedEnvName({ + envOption: args.options.env, + }); const controller = new AbortController(); const cancel = () => controller.abort(); process.once("SIGINT", cancel); @@ -88,8 +92,7 @@ export const configValidateCommand = withHandler( startDir: resolve(ctx.cwd, args.options.path ?? "."), profiles, explicitDomain: args.options.domain, - explicitOverlay: - args.options.env === "base" ? null : args.options.env, + explicitOverlay, signal: controller.signal, }) : await compileNativeConfig({ diff --git a/src/commands/env.ts b/src/commands/env.ts index d867a9e84..3e9e3a23f 100644 --- a/src/commands/env.ts +++ b/src/commands/env.ts @@ -9,6 +9,7 @@ import { defineOption, withHandler, } from "../cli/command.ts"; +import { resolveRequestedEnvName } from "../cli/env-selection.ts"; import { optEnv, optJson, optPath, optProject } from "../cli/options.ts"; import { readControlPlaneConfig } from "../control-plane/sdk/config.ts"; import { updateGlobalConfig } from "../lib/config.ts"; @@ -50,7 +51,6 @@ import type { ProjectContext } from "../lib/project.ts"; import { defaultProjectSlugFromPath, findProjectContext, - parseEnvConfigSelection, readProjectConfig, sanitizeProjectSlug, } from "../lib/project.ts"; @@ -628,16 +628,6 @@ async function resolveProjectName(project: ProjectContext): Promise { return sanitizeProjectSlug(raw.length > 0 ? raw : derived); } -function resolveRequestedEnvName(input: { - readonly envOption: string | undefined; -}): string | null | undefined { - const envName = parseEnvConfigSelection(input.envOption); - if (input.envOption !== undefined && envName === undefined) { - throw new CliUsageError("Invalid --env value."); - } - return envName; -} - async function resolveConfiguredSecretStore(input: { readonly project: ProjectContext; readonly projectName: string; diff --git a/src/commands/project.ts b/src/commands/project.ts index 38e6bf887..112e59158 100644 --- a/src/commands/project.ts +++ b/src/commands/project.ts @@ -77,6 +77,7 @@ import { defineOption, withHandler, } from "../cli/command.ts"; +import { resolveRequestedEnvName } from "../cli/env-selection.ts"; import { optBranch, optDetach, @@ -219,7 +220,6 @@ import { findProjectContext, findRepoRootForInit, type ProjectLifecycleCommand, - parseEnvConfigSelection, readProjectConfig, readProjectDevHost, resolveProjectOauthAliasHost, @@ -676,16 +676,6 @@ function resolveStartDir(ctx: CliContext, pathOpt: string | undefined): string { return pathOpt ? resolve(ctx.cwd, pathOpt) : ctx.cwd; } -function resolveRequestedEnvName(input: { - readonly envOption: string | undefined; -}): string | null | undefined { - const envName = parseEnvConfigSelection(input.envOption); - if (input.envOption !== undefined && envName === undefined) { - throw new CliUsageError("Invalid --env value."); - } - return envName; -} - export function resolveRequestedComposeServices(opts: { readonly requested: readonly string[] | undefined; readonly available: readonly string[]; diff --git a/src/commands/session.ts b/src/commands/session.ts index d260b5a8a..d300eac4c 100644 --- a/src/commands/session.ts +++ b/src/commands/session.ts @@ -11,6 +11,7 @@ import { defineOption, withHandler, } from "../cli/command.ts"; +import { resolveRequestedEnvName } from "../cli/env-selection.ts"; import { optEnv, optJson, optPretty } from "../cli/options.ts"; import { readHackEnvContract, @@ -22,7 +23,6 @@ import { defaultProjectSlugFromPath, findProjectContext, type ProjectContext, - parseEnvConfigSelection, readProjectConfig, } from "../lib/project.ts"; import { @@ -880,16 +880,6 @@ function resolveRunUpCwd(opts: { return opts.project.repoRoot; } -function resolveRequestedEnvName(opts: { - readonly envOption: string | undefined; -}): string | null | undefined { - const envName = parseEnvConfigSelection(opts.envOption); - if (opts.envOption !== undefined && envName === undefined) { - throw new CliUsageError("Invalid --env value."); - } - return envName; -} - function resolveRequestedServiceName(opts: { readonly serviceOption: string | undefined; }): string | null { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index e6ee1586d..a89a0f178 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -2,6 +2,7 @@ import { randomBytes } from "node:crypto"; import type { LogOutputFormat } from "../backends/log-backend.ts"; import type { RuntimeBaseOptions } from "../backends/runtime-backend.ts"; import { CliUsageError } from "../cli/command.ts"; +import { resolveRequestedEnvName } from "../cli/env-selection.ts"; import { emitCliResult, errorResult, @@ -135,7 +136,6 @@ import { import { waitNativeComposeReady } from "./native-compose-wait-ready.ts"; import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; -import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; @@ -2319,7 +2319,9 @@ async function dispatchNativeComposeCommand( const options = { ...input, overlay: - input.overlay === null ? null : parseEnvConfigSelection(input.overlay), + input.overlay === null + ? null + : resolveRequestedEnvName({ envOption: input.overlay }), }; const adopted = await tryLegacyComposeAdoptedCommand(input); if (adopted !== null) { diff --git a/src/lib/project.ts b/src/lib/project.ts index ba163a112..7a54cea5d 100644 --- a/src/lib/project.ts +++ b/src/lib/project.ts @@ -122,6 +122,17 @@ export function normalizeEnvConfigName(input: string): string | null { return normalized.length > 0 ? normalized : null; } +/** + * `--env` spellings (case-insensitive) that select the base layers and bypass + * the configured default overlay. An overlay file named for one of these + * spellings is never selected by name. + */ +export const ENV_BASE_SELECTION_ALIASES: ReadonlySet = new Set([ + "base", + "default", + "none", +]); + export function parseEnvConfigSelection( input: string | undefined ): string | null | undefined { @@ -134,8 +145,7 @@ export function parseEnvConfigSelection( return null; } - const lowered = trimmed.toLowerCase(); - if (lowered === "base" || lowered === "default" || lowered === "none") { + if (ENV_BASE_SELECTION_ALIASES.has(trimmed.toLowerCase())) { return null; } diff --git a/tests/env-command-matrix.test.ts b/tests/env-command-matrix.test.ts new file mode 100644 index 000000000..2bfdadae8 --- /dev/null +++ b/tests/env-command-matrix.test.ts @@ -0,0 +1,184 @@ +import { describe, expect, test } from "bun:test"; +import { resolve } from "node:path"; +import type { AnyCommandSpec } from "../src/cli/command.ts"; +import { optEnv } from "../src/cli/options.ts"; +import { runCli } from "../src/cli/run.ts"; +import { CLI_SPEC } from "../src/cli/spec.ts"; + +/** + * Source check for the "Command and delivery matrix" in docs/env.md: its rows + * must be exactly the CLI_SPEC command paths that accept `--env`, and each + * row's Selectors cell must list exactly the env-related flags the command + * accepts. A command that gains or loses `--env` fails here until the matrix + * row is added or removed. + */ + +const DOC_PATH = resolve(import.meta.dir, "..", "docs", "env.md"); +const SECTION_HEADING = "## Command and delivery matrix"; +const SELECTOR_FLAGS: ReadonlySet = new Set([ + "--env", + "--service", + "--scope", + "--target", +]); +const ROW_PATTERN = /^\| `hack ([a-z][a-z -]*)` \| ([^|]*) \|/; +const FLAG_PATTERN = /`(--[a-z-]+)`/g; + +type EnvCommand = { + readonly path: string; + readonly spec: AnyCommandSpec; +}; + +function envCommands( + commands: readonly AnyCommandSpec[], + prefix: readonly string[] = [] +): EnvCommand[] { + const found: EnvCommand[] = []; + for (const command of commands) { + const path = [...prefix, command.name]; + if (command.options.some((option) => option.long === "--env")) { + found.push({ path: path.join(" "), spec: command }); + } + found.push(...envCommands(command.subcommands, path)); + } + return found; +} + +function matrixRows(doc: string): Map> { + const start = doc.indexOf(SECTION_HEADING); + expect(start).toBeGreaterThanOrEqual(0); + const rest = doc.slice(start + SECTION_HEADING.length); + const end = rest.indexOf("\n## "); + const section = end >= 0 ? rest.slice(0, end) : rest; + const rows = new Map>(); + for (const line of section.split("\n")) { + const match = ROW_PATTERN.exec(line); + if (!match) { + continue; + } + const path = match[1] ?? ""; + expect(rows.has(path), `duplicate matrix row: hack ${path}`).toBe(false); + rows.set( + path, + new Set( + [...(match[2] ?? "").matchAll(FLAG_PATTERN)].map( + (flag) => flag[1] ?? "" + ) + ) + ); + } + return rows; +} + +function selectorFlags(spec: AnyCommandSpec): ReadonlySet { + return new Set( + spec.options + .map((option) => option.long) + .filter((long) => SELECTOR_FLAGS.has(long)) + ); +} + +/** Differences between the documented matrix and CLI_SPEC; empty when in sync. */ +function matrixDrift(doc: string): string[] { + const rows = matrixRows(doc); + const commands = envCommands(CLI_SPEC.commands); + const drift: string[] = []; + for (const { path, spec } of commands) { + const documented = rows.get(path); + if (!documented) { + drift.push(`missing row: hack ${path}`); + continue; + } + const expected = [...selectorFlags(spec)].sort(); + const actual = [...documented].sort(); + if (JSON.stringify(expected) !== JSON.stringify(actual)) { + drift.push( + `selector drift for hack ${path}: expected ${expected.join(",")}, documented ${actual.join(",")}` + ); + } + } + const known = new Set(commands.map((command) => command.path)); + for (const path of rows.keys()) { + if (!known.has(path)) { + drift.push(`row without --env in CLI_SPEC: hack ${path}`); + } + } + return drift; +} + +describe("env command and delivery matrix", () => { + test("docs/env.md rows equal the CLI_SPEC commands that accept --env", async () => { + const doc = await Bun.file(DOC_PATH).text(); + expect(matrixDrift(doc)).toEqual([]); + expect(matrixRows(doc).size).toBe(envCommands(CLI_SPEC.commands).length); + }); + + test("every --env option shares the one optEnv definition", () => { + for (const { path, spec } of envCommands(CLI_SPEC.commands)) { + const option = spec.options.find( + (candidate) => candidate.long === "--env" + ); + expect(option, `hack ${path} defines its own --env`).toBe(optEnv); + } + }); + + test("negative control: removing or adding a matrix row is detected", async () => { + const doc = await Bun.file(DOC_PATH).text(); + const configPlanRow = doc + .split("\n") + .find((line) => line.startsWith("| `hack config plan` |")); + expect(configPlanRow).toBeDefined(); + expect(matrixDrift(doc.replace(`${configPlanRow}\n`, ""))).toEqual([ + "missing row: hack config plan", + ]); + expect( + matrixDrift( + doc.replace( + `${configPlanRow}\n`, + `${configPlanRow}\n| \`hack logs\` | \`--env\` | x | x | x |\n` + ) + ) + ).toEqual(["row without --env in CLI_SPEC: hack logs"]); + expect( + matrixDrift( + doc.replace( + `${configPlanRow}\n`, + `${configPlanRow?.replace("| `--env` |", "| `--env`, `--service` |")}\n` + ) + ) + ).toEqual([ + "selector drift for hack config plan: expected --env, documented --env,--service", + ]); + }); + + test("negative control: a command without --env refuses the flag as unsupported", async () => { + const commands: readonly AnyCommandSpec[] = CLI_SPEC.commands; + const ps = commands.find((command) => command.name === "ps"); + expect(ps).toBeDefined(); + expect(ps?.options.some((option) => option.long === "--env")).toBe(false); + expect(ps?.allowUnknownOptions).not.toBe(true); + const stdout: Buffer[] = []; + const originalOut = process.stdout.write; + const originalErr = process.stderr.write; + process.stdout.write = ((chunk: string | Uint8Array) => { + stdout.push(Buffer.from(chunk)); + return true; + }) as typeof process.stdout.write; + process.stderr.write = (() => true) as typeof process.stderr.write; + let code: number; + try { + code = await runCli(["ps", "--env", "qa", "--json"]); + } finally { + process.stdout.write = originalOut; + process.stderr.write = originalErr; + } + expect(code).not.toBe(0); + expect(JSON.parse(Buffer.concat(stdout).toString())).toEqual({ + ok: false, + error: { + code: "E_USAGE", + message: 'Option(s) not valid for "ps": --env', + }, + }); + }); +}); diff --git a/tests/env-selection-parity.test.ts b/tests/env-selection-parity.test.ts new file mode 100644 index 000000000..52f174ba4 --- /dev/null +++ b/tests/env-selection-parity.test.ts @@ -0,0 +1,288 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + mkdir, + mkdtemp, + readdir, + realpath, + rm, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { CliUsageError } from "../src/cli/command.ts"; +import { resolveRequestedEnvName } from "../src/cli/env-selection.ts"; +import { optEnv } from "../src/cli/options.ts"; +import { runCli } from "../src/cli/run.ts"; +import { __testOnlySessionCommand } from "../src/commands/session.ts"; +import { tryNativeComposeCommand } from "../src/lib/native-compose-command.ts"; +import { restoreEnv } from "./helpers/env.ts"; +import { requireTestConfigCompiler } from "./helpers/test-config-compiler.ts"; + +/** + * One `--env` grammar across commands: `base`, `default` and `none` (any case) + * select the base layers and bypass the configured default overlay, and an + * unusable name is refused by every surface instead of silently selecting base. + * + * Compose-style projects back `env get`/`env list`; the authored native + * project backs metadata-only `config plan` through the real compiler. Both + * configure `qa` as the default overlay so omission and base differ. + */ + +const realCompiler = requireTestConfigCompiler(); +const BASE_SPELLINGS: string[] = ["base", "default", "DEFAULT", "none"]; +const INVALID_NAME = "!!!"; +const ENV_NAMES = [ + "HACK_GLOBAL_CONFIG_PATH", + "HACK_CONFIG_COMPILER_BINARY", + "HACK_ENV_SECRET_KEY", + "HACK_SECRETS_FILE_KEY", + "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK", + "HACK_EXECUTION_MODE", + "HACK_HOME", + "HOME", + "CI", +] as const; +const savedEnv = new Map(); +let root = ""; +let composeRoot = ""; +let nativeRoot = ""; + +beforeEach(async () => { + // Native inputs must live in unredirected directories (no /var symlink). + root = await realpath( + await mkdtemp(join(tmpdir(), "hack-env-selection-parity-")) + ); + composeRoot = join(root, "compose"); + nativeRoot = join(root, "native"); + for (const name of ENV_NAMES) { + savedEnv.set(name, process.env[name]); + Reflect.deleteProperty(process.env, name); + } + process.env.HOME = join(root, "home"); + process.env.HACK_HOME = join(root, "hack-home"); + process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; + process.env.HACK_GLOBAL_CONFIG_PATH = join(root, "global.json"); + process.env.HACK_CONFIG_COMPILER_BINARY = realCompiler; + await writeFile(process.env.HACK_GLOBAL_CONFIG_PATH, "{}"); + + await mkdir(join(composeRoot, ".hack"), { recursive: true }); + await writeFile( + join(composeRoot, ".hack/hack.config.json"), + JSON.stringify({ name: "parity-compose", env: { defaultOverlay: "qa" } }) + ); + await writeFile( + join(composeRoot, ".hack/docker-compose.yml"), + "services:\n api:\n image: alpine:3.20\n" + ); + await writeLayers(composeRoot); + + await mkdir(join(nativeRoot, ".hack"), { recursive: true }); + await writeFile( + join(nativeRoot, ".hack/hack.project.json"), + JSON.stringify({ + schema_version: 1, + name: "parity-native", + environment: { default_overlay: "qa" }, + worktree: { inherit_local: false, auto_branch: false }, + services: { web: { image: `sha256:${"a".repeat(64)}` } }, + }) + ); + await writeLayers(nativeRoot); +}); + +afterEach(async () => { + for (const [name, value] of savedEnv) { + restoreEnv(name, value); + } + savedEnv.clear(); + await rm(root, { recursive: true, force: true }); +}); + +async function writeLayers(projectRoot: string): Promise { + for (const name of ["default", "qa"]) { + await writeFile( + join(projectRoot, `.hack/hack.env.${name}.yaml`), + JSON.stringify({ + version: 1, + environment: name, + secretsprovider: "project_key", + values: { global: { VALUE: name === "default" ? "base" : "qa" } }, + }) + ); + } +} + +async function capture(args: readonly string[]) { + const stdout: Buffer[] = []; + const stderr: Buffer[] = []; + const originalOut = process.stdout.write; + const originalErr = process.stderr.write; + process.stdout.write = ((chunk: string | Uint8Array) => { + stdout.push(Buffer.from(chunk)); + return true; + }) as typeof process.stdout.write; + process.stderr.write = ((chunk: string | Uint8Array) => { + stderr.push(Buffer.from(chunk)); + return true; + }) as typeof process.stderr.write; + try { + const code = await runCli(args); + return { + code, + stdout: Buffer.concat(stdout).toString(), + stderr: Buffer.concat(stderr).toString(), + }; + } finally { + process.stdout.write = originalOut; + process.stderr.write = originalErr; + } +} + +function envArgs(value: string | undefined): readonly string[] { + return value === undefined ? [] : ["--env", value]; +} + +/** Selected overlay per surface; `null` is base layers only. */ +async function selections(value: string | undefined) { + const plan = await capture([ + "config", + "plan", + "--path", + nativeRoot, + ...envArgs(value), + "--json", + ]); + const get = await capture([ + "env", + "get", + "VALUE", + "--path", + composeRoot, + ...envArgs(value), + ]); + const list = await capture([ + "env", + "list", + "--path", + composeRoot, + ...envArgs(value), + "--json", + ]); + return { plan, get, list }; +} + +function planOverlay(result: { readonly stdout: string }): unknown { + const parsed: unknown = JSON.parse(result.stdout); + expect(parsed).toHaveProperty("ok", true); + expect(parsed).toHaveProperty("environment_plan.overlay"); + return (parsed as { environment_plan: { overlay: unknown } }).environment_plan + .overlay; +} + +function listOverlay(result: { readonly stdout: string }): unknown { + const parsed: unknown = JSON.parse(result.stdout); + expect(parsed).toHaveProperty("env_selection.effective"); + return (parsed as { env_selection: { effective: unknown } }).env_selection + .effective; +} + +test("omitting --env selects the configured default overlay on every surface", async () => { + const { plan, get, list } = await selections(undefined); + expect([plan.code, get.code, list.code]).toEqual([0, 0, 0]); + expect(planOverlay(plan)).toBe("qa"); + expect(get.stdout).toBe("qa"); + expect(listOverlay(list)).toBe("qa"); +}); + +test.each( + BASE_SPELLINGS +)("--env %s selects base layers on config plan, env get and env list alike", async (spelling) => { + const { plan, get, list } = await selections(spelling); + expect([plan.code, get.code, list.code]).toEqual([0, 0, 0]); + // Regression: config plan previously forwarded `default`/`DEFAULT`/`none` + // as an overlay name, selecting `.hack/hack.env.default.yaml` as an overlay. + expect(planOverlay(plan)).toBeNull(); + expect(get.stdout).toBe("base"); + expect(listOverlay(list)).toBeNull(); +}); + +test("an unusable --env name is refused by every surface without reading values", async () => { + const before = await readdir(join(nativeRoot, ".hack")); + const { plan, get, list } = await selections(INVALID_NAME); + for (const result of [plan, get, list]) { + expect(result.code).not.toBe(0); + } + // JSON surfaces report the fixed usage envelope; `get` keeps stdout empty. + for (const result of [plan, list]) { + expect(JSON.parse(result.stdout)).toEqual({ + ok: false, + error: { code: "E_USAGE", message: "Invalid --env value." }, + }); + } + expect(get.stdout).toBe(""); + expect(get.stderr).toContain("Unable to read env value."); + const validate = await capture([ + "config", + "validate", + "--path", + nativeRoot, + "--env", + INVALID_NAME, + "--json", + ]); + expect(validate.code).not.toBe(0); + expect(JSON.parse(validate.stdout)).toHaveProperty("error.code", "E_USAGE"); + expect(await readdir(join(nativeRoot, ".hack"))).toEqual(before); +}); + +test("config validate shares the base spellings with config plan", async () => { + for (const spelling of BASE_SPELLINGS) { + const result = await capture([ + "config", + "validate", + "--path", + nativeRoot, + "--env", + spelling, + "--json", + ]); + expect(result.code).toBe(0); + expect(JSON.parse(result.stdout)).toHaveProperty( + "local_resolution.overlay", + null + ); + } +}); + +test("authored native dispatch refuses an unusable --env before selecting a project", async () => { + const error: unknown = await tryNativeComposeCommand({ + cwd: nativeRoot, + path: nativeRoot, + operation: "ps", + overlay: INVALID_NAME, + }).catch((value: unknown) => value); + expect(error).toBeInstanceOf(CliUsageError); + expect(error).toHaveProperty("message", "Invalid --env value."); +}); + +test("the shared resolver maps base spellings to null and keeps workspace names env-base", () => { + const { buildScopedWorkspaceBaseName } = __testOnlySessionCommand; + expect(resolveRequestedEnvName({ envOption: undefined })).toBeUndefined(); + expect(resolveRequestedEnvName({ envOption: " QA " })).toBe("qa"); + expect(() => resolveRequestedEnvName({ envOption: INVALID_NAME })).toThrow( + "Invalid --env value." + ); + for (const spelling of BASE_SPELLINGS) { + const envName = resolveRequestedEnvName({ envOption: spelling }); + expect(envName).toBeNull(); + expect( + buildScopedWorkspaceBaseName({ + projectName: "parity", + envOptionSpecified: true, + envName, + serviceName: null, + }) + ).toBe("parity.env-base"); + } + expect(optEnv.valueHint).toBe(""); +}); From 6136946efe6118e5bcd321bd0350e35b350bf55d Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 22:35:27 -0400 Subject: [PATCH 09/32] fix(agents): count corrupted and unreadable integrations as drift Address the W10 verifier findings so integration sync and doctor agree: - checkAgentDocs marks a doc `unmanaged` only when it has neither the Hack start nor end marker. A doc with exactly one marker is a corrupted managed section and now reports a malformed-marker error, which sync counts as drift. Doctor treats the same doc as stale. - Add readOptionalTextFile, which uses lstat before reading so callers can tell "absent" from "present but unreadable". The Cursor rules, Codex skill, shared skill, hack-init skill and agent-doc checks now report an unreadable existing file as `error` (drift) instead of `missing` (optional, current). Doctor uses the same helper. - Add regression tests for start-only and end-only docs, plus mode-000 Hack-owned files and docs in fixture HOME/project roots, for both sync and doctor. Default `hack init` still writes no repository files. Co-Authored-By: Claude Opus 5.5 --- src/agents/codex-skill.ts | 13 +++- src/agents/cursor.ts | 13 +++- src/agents/hack-init-skill.ts | 14 +++- src/agents/shared-skill.ts | 12 ++- src/cli/integration-sync.ts | 7 +- src/commands/doctor.ts | 44 ++++++++--- src/lib/fs.ts | 44 ++++++++++- src/mcp/agent-docs.ts | 133 +++++++++++++++++++++------------ tests/doctor-command.test.ts | 109 ++++++++++++++++++++++++++- tests/integration-sync.test.ts | 70 ++++++++++++++++- 10 files changed, 387 insertions(+), 72 deletions(-) diff --git a/src/agents/codex-skill.ts b/src/agents/codex-skill.ts index 4f8b80fa5..ecfefa820 100644 --- a/src/agents/codex-skill.ts +++ b/src/agents/codex-skill.ts @@ -4,7 +4,7 @@ import { dirname, resolve } from "node:path"; import { ensureDir, pathExists, - readTextFile, + readOptionalTextFile, writeTextFileIfChanged, } from "../lib/fs.ts"; import { @@ -108,7 +108,16 @@ export async function checkCodexSkill(opts: { } const path = resolved.path; - const content = await readTextFile(path); + const file = await readOptionalTextFile(path); + if (file.kind === "unreadable") { + return { + scope: opts.scope, + status: "error", + path, + message: "Hack-managed file exists but could not be read.", + }; + } + const content = file.kind === "text" ? file.text : ""; if (!content) { return { scope: opts.scope, status: "missing", path }; } diff --git a/src/agents/cursor.ts b/src/agents/cursor.ts index 274628359..97cdb365c 100644 --- a/src/agents/cursor.ts +++ b/src/agents/cursor.ts @@ -4,7 +4,7 @@ import { dirname, resolve } from "node:path"; import { ensureDir, pathExists, - readTextFile, + readOptionalTextFile, writeTextFileIfChanged, } from "../lib/fs.ts"; import { @@ -80,7 +80,16 @@ export async function checkCursorRules(opts: { } const path = resolved.path; - const content = await readTextFile(path); + const file = await readOptionalTextFile(path); + if (file.kind === "unreadable") { + return { + scope: opts.scope, + status: "error", + path, + message: "Hack-managed file exists but could not be read.", + }; + } + const content = file.kind === "text" ? file.text : ""; if (!content) { return { scope: opts.scope, status: "missing", path }; } diff --git a/src/agents/hack-init-skill.ts b/src/agents/hack-init-skill.ts index 2f3c9e670..1744320ab 100644 --- a/src/agents/hack-init-skill.ts +++ b/src/agents/hack-init-skill.ts @@ -4,7 +4,7 @@ import { dirname, resolve } from "node:path"; import { ensureDir, pathExists, - readTextFile, + readOptionalTextFile, writeTextFileIfChanged, } from "../lib/fs.ts"; import { normalizeInstructionText } from "./instruction-source.ts"; @@ -129,7 +129,17 @@ export async function checkHackInitSkill(opts: { } const path = resolved.path; - const content = await readTextFile(path); + const file = await readOptionalTextFile(path); + if (file.kind === "unreadable") { + return { + client: opts.client, + scope: opts.scope, + status: "error", + path, + message: "Hack-managed file exists but could not be read.", + }; + } + const content = file.kind === "text" ? file.text : ""; if (!content) { return { client: opts.client, scope: opts.scope, status: "missing", path }; } diff --git a/src/agents/shared-skill.ts b/src/agents/shared-skill.ts index 5b5a8828f..8175eef4d 100644 --- a/src/agents/shared-skill.ts +++ b/src/agents/shared-skill.ts @@ -4,7 +4,7 @@ import { dirname, resolve } from "node:path"; import { ensureDir, pathExists, - readTextFile, + readOptionalTextFile, writeTextFileIfChanged, } from "../lib/fs.ts"; import { renderCodexSkill } from "./codex-skill.ts"; @@ -56,7 +56,15 @@ export async function checkSharedHackSkill(): Promise { return { status: "error", path: SKILL_FILENAME, message: resolved.message }; } - const content = await readTextFile(resolved.path); + const file = await readOptionalTextFile(resolved.path); + if (file.kind === "unreadable") { + return { + status: "error", + path: resolved.path, + message: "Hack-managed file exists but could not be read.", + }; + } + const content = file.kind === "text" ? file.text : ""; if (!content) { return { status: "missing", path: resolved.path }; } diff --git a/src/cli/integration-sync.ts b/src/cli/integration-sync.ts index ac72dd368..977ad121b 100644 --- a/src/cli/integration-sync.ts +++ b/src/cli/integration-sync.ts @@ -103,7 +103,9 @@ async function detectIntegrationDrift(opts: { // Optional integrations are not required: an absent ("missing") integration is // current. Only present-but-stale or unverifiable ("error") checks are drift. -// Retired legacy artifacts are drift whenever they are present. +// A Hack-owned file that exists but cannot be read is reported as "error" by +// its check, so it counts as drift (matching doctor). Retired legacy +// artifacts are drift whenever they are present. function hasSingleCheckDrift(status: string): boolean { return status === "stale" || status === "error"; } @@ -118,7 +120,8 @@ function hasMcpDrift(opts: { function hasDocDrift(opts: { readonly checks: readonly AgentDocCheckResult[]; }): boolean { - // A user-owned AGENTS.md/CLAUDE.md without a Hack section is not drift. + // A user-owned AGENTS.md/CLAUDE.md with neither Hack marker is not drift. + // A doc with only one marker (corrupted section) or an unreadable doc is. return opts.checks.some( (check) => check.status === "stale" || diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 4b50f6ef4..77a494861 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -80,6 +80,7 @@ import { ensureDir, ensureGitignoreEntry, pathExists, + readOptionalTextFile, readTextFile, writeTextFileIfChanged, } from "../lib/fs.ts"; @@ -3528,9 +3529,10 @@ async function maybeRepairHackd(): Promise { /** * Optional integrations are current when absent: a path is stale only when it - * exists and lacks the current revision marker. AGENTS.md/CLAUDE.md are - * user-owned, so they count only when they carry a Hack-managed section. - * Retained legacy artifacts are always stale. + * exists and lacks the current revision marker, or exists but cannot be read. + * AGENTS.md/CLAUDE.md are user-owned, so they count only when they carry a + * Hack start or end marker; a section with only one of the two markers is + * corrupted and stale. Retained legacy artifacts are always stale. */ export async function inspectDoctorAgentIntegrations(opts: { readonly projectRoot: string | null; @@ -3582,15 +3584,35 @@ export async function inspectDoctorAgentIntegrations(opts: { (content) => content.present && (content.text === null || - (content.text.includes(HACK_AGENT_DOCS_SECTION_MARKER) && - !content.text.includes(marker))) + isStaleAgentDocSection({ text: content.text, marker })) ); return { status: hasStalePath || hasLegacyArtifacts ? "stale" : "current", }; } -const HACK_AGENT_DOCS_SECTION_MARKER = ""; +const HACK_AGENT_DOCS_SECTION_START = ""; +const HACK_AGENT_DOCS_SECTION_END = ""; + +/** + * A user-owned doc with neither Hack marker is not stale. A doc with only one + * marker is a corrupted managed section; a complete section is stale when it + * lacks the current revision marker. + */ +function isStaleAgentDocSection(opts: { + readonly text: string; + readonly marker: string; +}): boolean { + const hasStart = opts.text.includes(HACK_AGENT_DOCS_SECTION_START); + const hasEnd = opts.text.includes(HACK_AGENT_DOCS_SECTION_END); + if (!(hasStart || hasEnd)) { + return false; + } + if (!(hasStart && hasEnd)) { + return true; + } + return !opts.text.includes(opts.marker); +} async function readOptionalIntegrationText( path: string @@ -3598,13 +3620,11 @@ async function readOptionalIntegrationText( | { readonly present: false } | { readonly present: true; readonly text: string | null } > { - const text = await readTextFile(path); - if (text !== null) { - return { present: true, text }; + const file = await readOptionalTextFile(path); + if (file.kind === "absent") { + return { present: false }; } - return (await pathExists(path)) - ? { present: true, text: null } - : { present: false }; + return { present: true, text: file.kind === "text" ? file.text : null }; } async function resolveDaemonReportForDoctorFix(): Promise< diff --git a/src/lib/fs.ts b/src/lib/fs.ts index 7b0e49ef4..f289e353a 100644 --- a/src/lib/fs.ts +++ b/src/lib/fs.ts @@ -1,4 +1,4 @@ -import { mkdir, stat } from "node:fs/promises"; +import { lstat, mkdir, stat } from "node:fs/promises"; export async function pathExists(absolutePath: string): Promise { try { @@ -28,6 +28,48 @@ export async function readTextFile( } } +/** + * Result of reading an optional file that distinguishes a path that does not + * exist from one that exists but cannot be read. + */ +export type OptionalTextFile = + | { readonly kind: "absent" } + | { readonly kind: "unreadable" } + | { readonly kind: "text"; readonly text: string }; + +/** + * Read an optional text file without collapsing "present but unreadable" into + * "absent". + * + * Existence is decided with `lstat` before reading, so a path is `absent` only + * when it (or a parent component) does not exist. Any other lstat failure + * (for example a parent directory without search permission), a dangling + * symlink, or a read failure (for example mode 000) is `unreadable`. Callers + * that check Hack-managed files must treat `unreadable` as drift rather than + * as an optional integration that was never installed. + */ +export async function readOptionalTextFile( + absolutePath: string +): Promise { + try { + await lstat(absolutePath); + } catch (error: unknown) { + return isMissingPathError(error) + ? { kind: "absent" } + : { kind: "unreadable" }; + } + const text = await readTextFile(absolutePath); + return text === null ? { kind: "unreadable" } : { kind: "text", text }; +} + +function isMissingPathError(error: unknown): boolean { + if (typeof error !== "object" || error === null || !("code" in error)) { + return false; + } + const code = (error as { readonly code: unknown }).code; + return code === "ENOENT" || code === "ENOTDIR"; +} + export async function writeTextFile( absolutePath: string, content: string diff --git a/src/mcp/agent-docs.ts b/src/mcp/agent-docs.ts index 7668f8827..ee529022d 100644 --- a/src/mcp/agent-docs.ts +++ b/src/mcp/agent-docs.ts @@ -4,7 +4,12 @@ import { normalizeInstructionText, renderInstructionSections, } from "../agents/instruction-source.ts"; -import { pathExists, readTextFile, writeTextFileIfChanged } from "../lib/fs.ts"; +import { + pathExists, + readOptionalTextFile, + readTextFile, + writeTextFileIfChanged, +} from "../lib/fs.ts"; export type AgentDocTarget = "agents" | "claude"; @@ -20,7 +25,11 @@ export type AgentDocCheckResult = { readonly status: "present" | "stale" | "missing" | "error"; readonly path: string; readonly message?: string; - /** Set when the file exists but has no Hack-managed section at all. */ + /** + * Set only when the file is readable and carries neither Hack marker, so it + * is a user-owned doc without a Hack-managed section. A doc with exactly one + * marker is a corrupted managed section and is never `unmanaged`. + */ readonly unmanaged?: true; }; @@ -108,46 +117,7 @@ export async function checkAgentDocs(opts: { for (const target of opts.targets) { const path = resolveAgentDocPath({ projectRoot: opts.projectRoot, target }); try { - const existing = await readTextFile(path); - if (!existing) { - results.push({ target, status: "missing", path }); - continue; - } - - if (!hasAgentDocSnippet({ content: existing })) { - results.push({ - target, - status: "error", - path, - message: "Missing hack agent-docs markers.", - unmanaged: true, - }); - continue; - } - - const region = extractSnippetRegion({ content: existing }); - if (region === null) { - results.push({ - target, - status: "error", - path, - message: "Malformed hack agent-docs markers.", - }); - continue; - } - - if (normalizeInstructionText({ text: region }) !== expected) { - results.push({ - target, - status: "stale", - path, - message: - "hack agent-docs content is out of date. Run: hack setup sync", - }); - continue; - } - - results.push({ target, status: "present", path }); + results.push(await checkAgentDocTarget({ target, path, expected })); } catch (error: unknown) { const message = error instanceof Error ? error.message : "Failed to read file"; @@ -158,6 +128,72 @@ export async function checkAgentDocs(opts: { return results; } +/** + * Classify one agent doc. Absent or empty docs are `missing`; an unreadable + * doc is `error`; a readable doc with neither Hack marker is a user-owned + * `unmanaged` error; a doc with only one marker is a corrupted (managed) + * section and is `error` without `unmanaged`. + */ +async function checkAgentDocTarget(opts: { + readonly target: AgentDocTarget; + readonly path: string; + readonly expected: string; +}): Promise { + const { target, path } = opts; + const file = await readOptionalTextFile(path); + if (file.kind === "unreadable") { + return { + target, + status: "error", + path, + message: "Agent doc exists but could not be read.", + }; + } + const existing = file.kind === "text" ? file.text : ""; + if (!existing) { + return { target, status: "missing", path }; + } + + const markers = inspectAgentDocMarkers({ content: existing }); + if (!(markers.hasStart || markers.hasEnd)) { + return { + target, + status: "error", + path, + message: "Missing hack agent-docs markers.", + unmanaged: true, + }; + } + if (!(markers.hasStart && markers.hasEnd)) { + return { + target, + status: "error", + path, + message: + "Malformed hack agent-docs markers: only one of the start/end markers is present.", + }; + } + + const region = extractSnippetRegion({ content: existing }); + if (region === null) { + return { + target, + status: "error", + path, + message: "Malformed hack agent-docs markers.", + }; + } + if (normalizeInstructionText({ text: region }) !== opts.expected) { + return { + target, + status: "stale", + path, + message: "hack agent-docs content is out of date. Run: hack setup sync", + }; + } + return { target, status: "present", path }; +} + /** * Remove the hack snippet from agent docs. */ @@ -290,11 +326,14 @@ function extractSnippetRegion(opts: { return match ? match[0] : null; } -function hasAgentDocSnippet(opts: { readonly content: string }): boolean { - return ( - opts.content.includes(DOC_MARKER_START) && - opts.content.includes(DOC_MARKER_END) - ); +function inspectAgentDocMarkers(opts: { readonly content: string }): { + readonly hasStart: boolean; + readonly hasEnd: boolean; +} { + return { + hasStart: opts.content.includes(DOC_MARKER_START), + hasEnd: opts.content.includes(DOC_MARKER_END), + }; } function escapeRegex(opts: { readonly value: string }): string { diff --git a/tests/doctor-command.test.ts b/tests/doctor-command.test.ts index 9f3f9d32a..6d7be786c 100644 --- a/tests/doctor-command.test.ts +++ b/tests/doctor-command.test.ts @@ -1,5 +1,12 @@ import { expect, test } from "bun:test"; -import { mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises"; +import { + chmod, + mkdir, + mkdtemp, + readdir, + rm, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { HACK_AGENT_INTEGRATION_CONTENT_REVISION } from "../src/agents/integration-revision.ts"; @@ -369,6 +376,106 @@ test("doctor treats absent optional agent integrations as current", async () => } }); +/** Run a doctor agent-integration check against a fixture HOME and project. */ +async function withDoctorAgentFixture( + run: (paths: { + readonly home: string; + readonly project: string; + readonly inspect: () => Promise<{ readonly status: string }>; + }) => Promise +): Promise { + const root = await mkdtemp(join(tmpdir(), "hack-doctor-agent-edge-")); + const home = join(root, "home"); + const project = join(root, "repo"); + const savedHome = process.env.HOME; + process.env.HOME = home; + try { + await mkdir(home, { recursive: true }); + await mkdir(project, { recursive: true }); + await run({ + home, + project, + inspect: () => + inspectDoctorAgentIntegrations({ projectRoot: project, homeDir: home }), + }); + } finally { + if (savedHome === undefined) { + // biome-ignore lint/performance/noDelete: restoring an unset env var + delete process.env.HOME; + } else { + process.env.HOME = savedHome; + } + await rm(root, { recursive: true, force: true }); + } +} + +test("doctor treats a doc with only one Hack marker as stale", async () => { + const marker = `Content revision: \`${HACK_AGENT_INTEGRATION_CONTENT_REVISION}\``; + await withDoctorAgentFixture(async ({ project, inspect }) => { + const doc = join(project, "AGENTS.md"); + + // Start marker only, even with the current revision text inside. + await writeFile( + doc, + `# notes\n\n${marker}\nno end marker\n` + ); + await expect(inspect()).resolves.toEqual({ status: "stale" }); + + // End marker only. + await writeFile( + doc, + `# notes\n${marker}\nno start marker\n\n` + ); + await expect(inspect()).resolves.toEqual({ status: "stale" }); + + // Neither marker: user-owned doc, not stale. + await writeFile(doc, "# notes\n"); + await expect(inspect()).resolves.toEqual({ status: "current" }); + }); +}); + +// Root can read mode-000 files, so the unreadable fixture cannot be built. +const doctorRunsAsRoot = process.getuid?.() === 0; + +test.skipIf(doctorRunsAsRoot)( + "doctor treats a present but unreadable Hack-owned file as stale", + async () => { + const marker = `Content revision: \`${HACK_AGENT_INTEGRATION_CONTENT_REVISION}\``; + await withDoctorAgentFixture(async ({ project, inspect }) => { + const rules = join(project, ".cursor", "rules", "hack.mdc"); + await mkdir(join(rules, ".."), { recursive: true }); + await writeFile(rules, `${marker}\n`); + await expect(inspect()).resolves.toEqual({ status: "current" }); + + await chmod(rules, 0o000); + try { + await expect(inspect()).resolves.toEqual({ status: "stale" }); + } finally { + await chmod(rules, 0o644); + } + await expect(inspect()).resolves.toEqual({ status: "current" }); + }); + } +); + +test.skipIf(doctorRunsAsRoot)( + "doctor treats a present but unreadable agent doc as stale", + async () => { + await withDoctorAgentFixture(async ({ project, inspect }) => { + const doc = join(project, "CLAUDE.md"); + await writeFile(doc, "# notes\n"); + await expect(inspect()).resolves.toEqual({ status: "current" }); + + await chmod(doc, 0o000); + try { + await expect(inspect()).resolves.toEqual({ status: "stale" }); + } finally { + await chmod(doc, 0o644); + } + }); + } +); + test("doctor guidance routes runtime hygiene drift to projects prune", () => { const guidance = buildDoctorRecoveryGuidance({ results: [ diff --git a/tests/integration-sync.test.ts b/tests/integration-sync.test.ts index 3559c5508..89a4155fb 100644 --- a/tests/integration-sync.test.ts +++ b/tests/integration-sync.test.ts @@ -1,8 +1,9 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { installCursorRules } from "../src/agents/cursor.ts"; +import { installSharedHackSkill } from "../src/agents/shared-skill.ts"; import { inspectAgentIntegrationFreshness } from "../src/cli/integration-sync.ts"; /** @@ -121,3 +122,70 @@ test("a retired legacy artifact is stale", async () => { ); expect(await status(null)).toBe("stale"); }); + +test("a doc with only the Hack start marker is a corrupted section and stale", async () => { + const { repo } = fixture(); + await writeFile( + join(repo, "AGENTS.md"), + "# notes\n\nold content without end\n" + ); + expect(await status(repo)).toBe("stale"); +}); + +test("a doc with only the Hack end marker is a corrupted section and stale", async () => { + const { repo } = fixture(); + await writeFile( + join(repo, "CLAUDE.md"), + "# notes\nold content without start\n\n" + ); + expect(await status(repo)).toBe("stale"); +}); + +// Root can read mode-000 files, so the unreadable fixture cannot be built. +const runsAsRoot = process.getuid?.() === 0; + +test.skipIf(runsAsRoot)( + "an installed but unreadable Hack-owned project file counts as drift", + async () => { + const { repo } = fixture(); + await installCursorRules({ scope: "project", projectRoot: repo }); + const rules = join(repo, ".cursor", "rules", "hack.mdc"); + expect(await status(repo)).toBe("current"); + await chmod(rules, 0o000); + try { + expect(await status(repo)).toBe("stale"); + } finally { + await chmod(rules, 0o644); + } + expect(await status(repo)).toBe("current"); + } +); + +test.skipIf(runsAsRoot)( + "an installed but unreadable Hack-owned HOME file counts as drift", + async () => { + const { home } = fixture(); + await installSharedHackSkill(); + const skill = join(home, ".ai", "skills", "hack-cli", "SKILL.md"); + expect(await status(null)).toBe("current"); + await chmod(skill, 0o000); + try { + expect(await status(null)).toBe("stale"); + } finally { + await chmod(skill, 0o644); + } + } +); + +test.skipIf(runsAsRoot)("an unreadable agent doc counts as drift", async () => { + const { repo } = fixture(); + const doc = join(repo, "AGENTS.md"); + await writeFile(doc, "# Team notes\n"); + expect(await status(repo)).toBe("current"); + await chmod(doc, 0o000); + try { + expect(await status(repo)).toBe("stale"); + } finally { + await chmod(doc, 0o644); + } +}); From 9854d1b5be525c6cd4308575480d06de3bfdfd2d Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 23:02:09 -0400 Subject: [PATCH 10/32] test(env): pin the console logger backend for CLI refusal tests The env add --secret refusal tests asserted the refusal text on stderr, but the default logger backend depends on the host: gum (stderr) when installed, clack (stdout) otherwise. Hosted macOS runners have no gum, so the refusal reached stdout and stderr was empty only in CI. Pin HACK_LOGGER=console in the CLI helper, matching the other CLI-spawning tests, so the stderr assertions hold on every host. Co-Authored-By: Claude Opus 5.5 --- tests/project-env-worktree-key.test.ts | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/project-env-worktree-key.test.ts b/tests/project-env-worktree-key.test.ts index 3bda56e8c..00cf7fa34 100644 --- a/tests/project-env-worktree-key.test.ts +++ b/tests/project-env-worktree-key.test.ts @@ -255,7 +255,14 @@ type CliResult = { readonly stderr: string; }; -/** Runs the current-branch CLI with an isolated HOME and global config. */ +/** + * Runs the current-branch CLI with an isolated HOME and global config. + * + * The logger backend is pinned to `console` (plain stderr). Without the pin + * the backend depends on the host: `gum` writes to stderr when installed, but + * hosted runners lack gum and fall back to clack, which writes to stdout, so + * refusal text would leave stderr empty only in CI. + */ async function runHackCli(opts: { readonly sandbox: string; readonly args: readonly string[]; @@ -266,6 +273,7 @@ async function runHackCli(opts: { ...process.env, HOME: home, HACK_GLOBAL_CONFIG_PATH: resolve(home, "hack.config.json"), + HACK_LOGGER: "console", }; Reflect.deleteProperty(env, "HACK_ENV_SECRET_KEY"); const proc = Bun.spawn( From 6e1ff6398112de11507cdb626089a77afd20df5c Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 23:12:24 -0400 Subject: [PATCH 11/32] test(mcp): assert closed ambiguous_project refusal in project-name suite The project-name boundary suite still expected the old value-echoing "Ambiguous project name/ID" stderr from MCP. Assert the closed MCP_REFUSALS contract instead: exact code, reason and fixed message, spawned: false, no argv, and none of the duplicated names, registry ids or fixture root anywhere in the serialized result. Co-Authored-By: Claude Opus 5.5 --- tests/project-name-command.test.ts | 74 ++++++++++++++++++++++++------ 1 file changed, 60 insertions(+), 14 deletions(-) diff --git a/tests/project-name-command.test.ts b/tests/project-name-command.test.ts index ef699cd85..e03ea752a 100644 --- a/tests/project-name-command.test.ts +++ b/tests/project-name-command.test.ts @@ -3,7 +3,11 @@ import { chmod, readFile, rm, writeFile } from "node:fs/promises"; import { join, resolve } from "node:path"; import { Client } from "@modelcontextprotocol/sdk/client/index.js"; import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js"; -import { createMcpServer } from "../src/mcp/server.ts"; +import { + createMcpServer, + MCP_REFUSALS, + type McpRefusalReason, +} from "../src/mcp/server.ts"; import { projectNameFixture } from "./helpers/project-name-fixture.ts"; // The same boundary suite can qualify the current branch's compiled binary. @@ -127,6 +131,37 @@ for (const selector of selectors) { }, 15_000); } +/** + * Asserts the closed pre-spawn refusal contract: fixed catalogue code and + * message, no child spawned, and none of the given caller or registry values + * anywhere in the serialized result. + */ +function expectClosedRefusal(opts: { + readonly result: Awaited>; + readonly reason: McpRefusalReason; + readonly absent: readonly string[]; +}) { + const expected = MCP_REFUSALS[opts.reason]; + expect(opts.result.isError).toBe(true); + expect(opts.result.structuredContent).toEqual({ + ok: false, + command: "", + exitCode: 1, + stdout: "", + stderr: expected.message, + spawned: false, + error: { + code: expected.code, + reason: opts.reason, + message: expected.message, + }, + }); + const serialized = JSON.stringify(opts.result); + for (const value of opts.absent) { + expect(serialized).not.toContain(value); + } +} + async function connectMcp() { const command = join(fixture.root, "fixture-cli"); // Spawn the actual current-source CLI (or explicitly selected current build). @@ -165,19 +200,25 @@ test("MCP selectors survive real CLI forwarding and reject aliases with multiple const contender = await fixture.createProject("other", "my-app"); await fixture.writeRegistry([target.entry, contender.entry]); const before = await readFile(fixture.registryPath, "utf8"); + const registered = [ + "my_app", + "my-app", + target.entry.id, + contender.entry.id, + fixture.root, + ]; for (const projectName of ["my_app", "my-app", "", "!!!"]) { const result = await client.callTool({ name: "hack.project.open", arguments: { projectName }, }); - expect(result.isError).toBe(true); - expect(result.structuredContent).toMatchObject({ - ok: false, - stderr: expect.stringContaining( + expectClosedRefusal({ + result, + reason: projectName === "" || projectName === "!!!" - ? "Invalid projectName" - : "Ambiguous project name" - ), + ? "invalid_project_name" + : "ambiguous_project", + absent: projectName === "" ? registered : [...registered, projectName], }); } expect(await readFile(fixture.registryPath, "utf8")).toBe(before); @@ -195,12 +236,17 @@ for (const reversed of [false, true]) { name: "hack.project.open", arguments: { projectName }, }); - expect(result.isError).toBe(true); - expect(result.structuredContent).toMatchObject({ - ok: false, - exitCode: 1, - stdout: "", - stderr: expect.stringContaining("Ambiguous project ID"), + expectClosedRefusal({ + result, + reason: "ambiguous_project", + absent: [ + projectName, + "my_app", + "my-app", + contender.entry.name, + target.entry.id, + fixture.root, + ], }); } expect(await readFile(fixture.registryPath, "utf8")).toBe(before); From 1f8b51e0c29e2213d609e8689cc923c512cdbeef Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 21:12:53 -0400 Subject: [PATCH 12/32] chore(x01): temporary witness cost trace for hosted measurement Temporary measurement only; reverted before the final commit. Writes closed labels and durations per native command to a per-scenario JSONL file and prints a summary in the docker-e2e job. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 19 ++ scripts/x01-trace-summary.ts | 268 ++++++++++++++++++ src/lib/native-compose-command-storage.ts | 12 +- src/lib/native-compose-command.ts | 8 +- src/lib/native-compose-ownership.ts | 5 +- src/lib/native-compose-phase-trace.ts | 8 +- .../native-compose-storage-witness-docker.ts | 43 ++- src/lib/native-compose-storage-witness.ts | 34 ++- src/lib/shell.ts | 17 +- src/lib/x01-cost-probe.ts | 171 +++++++++++ tests/e2e/harness.ts | 9 + .../e2e/native-config-networks-acceptance.ts | 4 + 12 files changed, 570 insertions(+), 28 deletions(-) create mode 100644 scripts/x01-trace-summary.ts create mode 100644 src/lib/x01-cost-probe.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 457db89f4..34c2387cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -306,7 +306,10 @@ jobs: HACK_E2E_CAPSULE_DIR: ${{ runner.temp }}/process-policy-capsules HACK_E2E_CLI_BIN: ${{ github.workspace }}/dist/hack HACK_E2E_SHARD: ${{ matrix.shard }}/4 + # TEMPORARY X01 measurement; reverted before the final commit. + HACK_X01_TRACE_DIR: ${{ runner.temp }}/x01-trace run: | + mkdir -p "$HACK_X01_TRACE_DIR" HACK_E2E_DOCKER_HOST="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" HACK_E2E_NETWORK_SOURCE_REVISION="$(git rev-parse HEAD)" HACK_E2E_NETWORK_CLI_SHA256="$(sha256sum dist/hack | cut -d ' ' -f1)" @@ -315,6 +318,22 @@ jobs: HACK_E2E_COMPOSE_PLUGIN_SHA256="$(sha256sum "$HACK_E2E_COMPOSE_PLUGIN_PATH" | cut -d ' ' -f1)" export HACK_E2E_DOCKER_HOST HACK_E2E_NETWORK_SOURCE_REVISION HACK_E2E_NETWORK_CLI_SHA256 HACK_E2E_NETWORK_COMPILER_SHA256 HACK_E2E_COMPOSE_PLUGIN_PATH HACK_E2E_COMPOSE_PLUGIN_SHA256 HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker --shard="$HACK_E2E_SHARD" + # TEMPORARY X01 measurement; reverted before the final commit. + - name: X01 witness cost summary + if: always() + run: | + for f in "$RUNNER_TEMP"/x01-trace/*.jsonl; do + [ -f "$f" ] || continue + bun scripts/x01-trace-summary.ts "$f" "$(basename "$f" .jsonl)" || true + done + - name: X01 witness cost trace artifact + if: always() + uses: actions/upload-artifact@v4 + with: + name: x01-trace-shard-${{ matrix.shard }} + path: ${{ runner.temp }}/x01-trace + if-no-files-found: ignore + retention-days: 7 - name: Preserve process-policy capsule and startup diagnostic # Closed fixed-field JSON only (counts, enums, booleans); the gh log viewer truncates it inline. if: failure() diff --git a/scripts/x01-trace-summary.ts b/scripts/x01-trace-summary.ts new file mode 100644 index 000000000..ba276c07f --- /dev/null +++ b/scripts/x01-trace-summary.ts @@ -0,0 +1,268 @@ +// TEMPORARY X01 measurement summary. Reverted before the final commit. +// Reads closed-label JSONL written by src/lib/x01-cost-probe.ts. +type Event = { + readonly p: number; + readonly v: string; + readonly k: string; + readonly s?: string; + readonly l?: string; + readonly ms?: number; + readonly t?: number; + readonly c?: number; + readonly line?: { + readonly phase?: string; + readonly boundary?: string; + readonly durationMs?: number | null; + }; +}; +const WITNESS = new Set([ + "storage-prepare", + "storage-verify", + "carrier-new", + "inspect", + "provision", + "finish", + "witness-verify", + "witness-prepare", + "witness-enroll", +]); +const isWitness = (label: string) => + WITNESS.has(label) || label.startsWith("invoke:"); +const LEAF = new Set(["probe", "run", "exec", "hash"]); +const PHASE = /^(oneoff|storage|guard|finish|finalize)\.[a-z-]+$/; + +const path = process.argv[2]; +const title = process.argv[3] ?? path; +if (!path) { + throw new Error("usage: x01-trace-summary [title]"); +} +const text = await Bun.file(path).text(); +const events: Event[] = []; +for (const line of text.split("\n")) { + if (line.trim() === "") { + continue; + } + try { + events.push(JSON.parse(line) as Event); + } catch { + // skip partial line + } +} +const byProcess = new Map(); +for (const event of events) { + const list = byProcess.get(event.p) ?? []; + list.push(event); + byProcess.set(event.p, list); +} +type Proc = { + readonly pid: number; + readonly verb: string; + readonly totalMs: number; + readonly witnessMs: number; + readonly events: Event[]; + readonly kind: string; +}; +function topWitnessMs(list: Event[]): number { + let total = 0; + for (const event of list) { + if (event.k !== "scope" || !event.l || !isWitness(event.l)) { + continue; + } + const parts = (event.s ?? "").split(">"); + parts.pop(); + if (parts.some(isWitness)) { + continue; + } + total += event.ms ?? 0; + } + return total; +} +const procs: Proc[] = []; +for (const [pid, list] of byProcess) { + const exit = list.find((event) => event.k === "exit"); + const verb = list[0]?.v ?? "other"; + const labels = new Set(list.map((event) => event.l ?? "")); + let kind = verb; + if (verb === "up" || verb === "restart" || verb === "run") { + if (labels.has("invoke:seed")) { + kind = `${verb}(cold)`; + } else if (labels.has("invoke:verify")) { + kind = `${verb}(warm)`; + } else { + kind = `${verb}(no-witness)`; + } + } else if (verb === "exec") { + kind = labels.has("invoke:verify") ? "exec(storage)" : "exec(image)"; + } + procs.push({ + pid, + verb, + totalMs: exit?.t ?? Math.max(...list.map((event) => event.t ?? 0)), + witnessMs: topWitnessMs(list), + events: list, + kind, + }); +} +const fmt = (ms: number) => (ms / 1000).toFixed(1); +const out: string[] = []; +out.push(`#### X01 trace: ${title}`); +out.push(`events=${events.length} processes=${procs.length}`); +out.push(""); +out.push( + "kind | n | avg total s | sum total s | avg witness s | sum witness s" +); +const kinds = [...new Set(procs.map((proc) => proc.kind))].sort(); +for (const kind of kinds) { + const selected = procs.filter((proc) => proc.kind === kind); + const total = selected.reduce((sum, proc) => sum + proc.totalMs, 0); + const witness = selected.reduce((sum, proc) => sum + proc.witnessMs, 0); + out.push( + `${kind} | ${selected.length} | ${fmt(total / selected.length)} | ${fmt(total)} | ${fmt(witness / selected.length)} | ${fmt(witness)}` + ); +} +const allTotal = procs.reduce((sum, proc) => sum + proc.totalMs, 0); +const allWitness = procs.reduce((sum, proc) => sum + proc.witnessMs, 0); +out.push( + `ALL | ${procs.length} | - | ${fmt(allTotal)} | - | ${fmt(allWitness)}` +); +out.push(""); + +function leafTable(selected: Proc[], heading: string, limit = 24) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (!(LEAF.has(event.k) && event.l)) { + continue; + } + const key = `${event.k}:${event.l}`; + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push(`${heading} (per process avg over n=${selected.length})`); + out.push("leaf | calls/proc | s/proc | ms/call"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function leafByScope(selected: Proc[], heading: string, limit = 40) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (!(LEAF.has(event.k) && event.l)) { + continue; + } + const parts = (event.s ?? "") + .split(">") + .filter( + (part) => + isWitness(part) || part.startsWith("cmd:") || PHASE.test(part) + ); + const inner = parts.slice(-2).join(">") || "(none)"; + const key = `${inner} :: ${event.k}:${event.l}`; + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push( + `${heading} leaves by scope (per process avg over n=${selected.length})` + ); + out.push("scope :: leaf | calls/proc | s/proc | ms/call"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function scopeTable(selected: Proc[], heading: string, limit = 30) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (event.k !== "scope" || !event.l) { + continue; + } + const parts = (event.s ?? "").split(">"); + const key = parts + .filter( + (part) => + PHASE.test(part) || + isWitness(part) || + part.startsWith("xattr:") || + part.startsWith("cmd:") || + part.startsWith("exec-") + ) + .join(">"); + if (key === "") { + continue; + } + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push(`${heading} scopes (per process avg over n=${selected.length})`); + out.push("scope path | n/proc | s/proc | ms/each"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function phaseRows(proc: Proc) { + const rows = new Map(); + for (const event of proc.events) { + if ( + event.k === "phase" && + event.line?.boundary !== "begin" && + event.line?.phase + ) { + const row = rows.get(event.line.phase) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.line.durationMs ?? 0; + rows.set(event.line.phase, row); + } + } + return rows; +} +for (const kind of [ + "up(cold)", + "up(warm)", + "exec(storage)", + "restart(warm)", + "run(warm)", + "run(cold)", + "down", +]) { + const selected = procs.filter((proc) => proc.kind === kind); + if (selected.length === 0) { + continue; + } + const sample = selected[0] as Proc; + out.push( + `### ${kind}: first sample total ${fmt(sample.totalMs)} s, witness ${fmt(sample.witnessMs)} s` + ); + out.push("phase-trace phase | spans | s"); + for (const [phase, row] of phaseRows(sample)) { + out.push(`${phase} | ${row.n} | ${fmt(row.ms)}`); + } + out.push(""); + scopeTable([sample], `${kind} first sample`, 40); + leafTable([sample], `${kind} first sample leaves`); + scopeTable(selected, `${kind} all`, 30); + leafTable(selected, `${kind} all leaves`); + leafByScope(selected, `${kind} all`); +} +console.log(out.join("\n")); diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index 58bc48bf8..ce45846c8 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -23,6 +23,7 @@ import { } from "./native-compose-storage-witness.ts"; import { createNativeComposeDockerStorageXattrCarrier } from "./native-compose-storage-witness-docker.ts"; import type { NativeComposeStorageXattrCarrier } from "./native-compose-storage-witness-xattr-carrier.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; type Selection = { readonly name: string; readonly storage: string }; type Document = Readonly>; @@ -129,7 +130,12 @@ async function volumeNames(signal: AbortSignal): Promise { /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ -export async function prepareNativeComposeCommandStorage(opts: { +export function prepareNativeComposeCommandStorage( + opts: Parameters[0] +): ReturnType { + return x01Scope("storage-prepare", () => x01Prepare(opts)); +} +async function x01Prepare(opts: { readonly store: NativeComposeGenerationStore; readonly mutation: NativeComposeMutation; readonly operation: "up" | "restart" | "run" | "exec"; @@ -220,7 +226,9 @@ export async function prepareNativeComposeCommandStorage(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = async (generation: NativeComposeGeneration) => { + const verify = (generation: NativeComposeGeneration) => + x01Scope("storage-verify", () => x01Verify(generation)); + const x01Verify = async (generation: NativeComposeGeneration) => { await assertFresh?.(); const before = await store.loadCurrent(); if (before.storageWitnessesPending) { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index e6ee1586d..9701043c2 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -137,6 +137,7 @@ import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; export type NativeComposeCommandOptions = { @@ -942,7 +943,8 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = async () => { + const assertSaved = () => x01Scope("exec-assert-saved", x01AssertSaved); + const x01AssertSaved = async () => { const latest = await store.loadCurrent(); assertStartupAvailable(latest); if ( @@ -959,7 +961,9 @@ async function savedCommand(opts: { signal, assertSaved, run: () => - runSavedProcess({ options, generation, document, base, signal }), + x01Scope("exec-process", () => + runSavedProcess({ options, generation, document, base, signal }) + ), }); }); } diff --git a/src/lib/native-compose-ownership.ts b/src/lib/native-compose-ownership.ts index 0019b3fc3..d8bc9dc8a 100644 --- a/src/lib/native-compose-ownership.ts +++ b/src/lib/native-compose-ownership.ts @@ -3,6 +3,7 @@ import { isRecord } from "./guards.ts"; import { nativeComposeVolumeCreatedAt } from "./native-compose-retained-storage.ts"; import { beginNativeCpuChild } from "./native-cpu-diagnostics.ts"; import { findExecutableInPath } from "./shell.ts"; +import { x01Label, x01Time } from "./x01-cost-probe.ts"; const PROJECT = /^[a-z0-9][a-z0-9_-]*$/; const SERVICE = /^[a-z0-9][a-z0-9._-]{0,62}$/; @@ -1232,7 +1233,7 @@ export function createNativeComposeProbe( probeRefused("E_NATIVE_COMPOSE_PROBE", "operation"); } const environment = { ...process.env }; - return async (args) => { + const x01Inner = async (args: readonly string[]): Promise => { checkInterruption(opts.signal, false); const remainingTime = deadline - Date.now(); if (remainingTime <= 0) { @@ -1310,6 +1311,8 @@ export function createNativeComposeProbe( observeCpu(probeExitCode(completion[2])); } }; + return (args) => + x01Time("probe", x01Label(args, false), () => x01Inner(args)); } function probeExitCode( diff --git a/src/lib/native-compose-phase-trace.ts b/src/lib/native-compose-phase-trace.ts index 540f833b0..943b79320 100644 --- a/src/lib/native-compose-phase-trace.ts +++ b/src/lib/native-compose-phase-trace.ts @@ -1,5 +1,6 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { writeSync } from "node:fs"; +import { x01Enabled, x01Phase, x01Scope } from "./x01-cost-probe.ts"; export const NATIVE_COMPOSE_PHASE_TRACE = "HACK_NATIVE_COMPOSE_PHASE_TRACE"; const PHASES = [ @@ -135,6 +136,8 @@ export function withNativeComposePhaseTrace(operation: () => Promise) { writeSync(2, line); }, }); + } else if (x01Enabled()) { + trace = createNativeComposePhaseTrace({ write: x01Phase }); } } catch { // Diagnostic setup never changes selection, results or effect authority. @@ -147,5 +150,8 @@ export function measureNativeComposePhase( phase: Phase, operation: () => Promise ): Promise { - return context.getStore()?.measure(phase, operation) ?? operation(); + return x01Scope( + phase, + () => context.getStore()?.measure(phase, operation) ?? operation() + ); } diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index 0c04e7ba5..da785e7b4 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -96,6 +96,7 @@ import { refuseNativeComposeStorageXattr as refuse, } from "./native-compose-storage-witness-xattr-codec.ts"; import { findExecutableInPath, type RunExitEvent, run } from "./shell.ts"; +import { x01Label, x01Scope, x01Time } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const PROCESS = @@ -813,7 +814,12 @@ async function groupAbsent( await new Promise((resolve) => setTimeout(resolve, 20)); } } -async function commandExecutable( +function commandExecutable( + path: string +): Promise { + return x01Time("hash", x01Label([path]), () => x01CommandExecutable(path)); +} +async function x01CommandExecutable( path: string ): Promise { const canonical = await realpath(path); @@ -960,7 +966,14 @@ async function commandBoot( } /** No callback can reject before the shared owner awaits exit. A rejected owner or * unproven group retains the journal; daemon effects are never inferred from exit. */ -export async function runNativeComposeStorageDockerCommand(supplied: { +export function runNativeComposeStorageDockerCommand( + supplied: Parameters[0] +): ReturnType { + return x01Scope(`cmd:${x01Label(supplied.args)}`, () => + x01RunCommandInner(supplied) + ); +} +async function x01RunCommandInner(supplied: { readonly context: Pick; readonly files: Pick< Files, @@ -1305,7 +1318,12 @@ function matchesScope( * image; other daemons and uncached images refuse rather than pull or emulate. * PostgreSQL data roots and crash durability remain unqualified. No module * initialization executes Docker or the helper. */ -export async function createNativeComposeDockerStorageXattrCarrier( +export function createNativeComposeDockerStorageXattrCarrier( + opts: Context +): Promise { + return x01Scope("carrier-new", () => x01CreateCarrier(opts)); +} +async function x01CreateCarrier( opts: Context ): Promise { const context: Context = Object.freeze({ ...opts }); @@ -2024,16 +2042,19 @@ export async function createNativeComposeDockerStorageXattrCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect, - provision, - invoke, + inspect: (selection) => x01Scope("inspect", () => inspect(selection)), + provision: (selection) => + x01Scope("provision", () => provision(selection)), + invoke: (input) => + x01Scope(`invoke:${input.request.operation}`, () => invoke(input)), finish: context.originalCommandRecords === true - ? async ({ input, completion }) => { - const finish = finalizers.get(input) ?? refuse(); - finalizers.delete(input); - await finish(completion); - } + ? ({ input, completion }) => + x01Scope("finish", async () => { + const finish = finalizers.get(input) ?? refuse(); + finalizers.delete(input); + await finish(completion); + }) : undefined, }, signal: context.signal, diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index b8a242f3b..0650f4b49 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -78,6 +78,7 @@ import { nativeComposeStorageXattrValueMatches, sameNativeComposeStorageXattrRoot, } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; export type { NativeComposeStorageWitnessReference } from "./native-compose-storage-witness-state.ts"; @@ -493,7 +494,16 @@ function requireXattrAdmission( refuse(); } } -async function invokeXattr(opts: { +function invokeXattr( + opts: Parameters[0] +): ReturnType { + const operation = + isRecord(opts.request) && typeof opts.request.operation === "string" + ? opts.request.operation + : "unknown"; + return x01Scope(`xattr:${operation}`, () => x01InvokeXattr(opts)); +} +async function x01InvokeXattr(opts: { readonly carrier: NativeComposeStorageXattrCarrier; readonly saved: XattrExpectation; readonly volume: NativeComposeRetainedVolume; @@ -1387,17 +1397,21 @@ export async function prepareNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - return await prepareNativeComposeStorageWitness({ - ...input, - xattrCarrier: carrier, - }); + return await x01Scope("witness-prepare", () => + prepareNativeComposeStorageWitness({ + ...input, + xattrCarrier: carrier, + }) + ); } export async function enrollNativeComposeStorageXattrWitness(opts: { readonly enrollment: NativeComposeStorageWitnessEnrollment; }): Promise { - return await enrollNativeComposeStorageWitness({ - enrollment: opts.enrollment, - }); + return await x01Scope("witness-enroll", () => + enrollNativeComposeStorageWitness({ + enrollment: opts.enrollment, + }) + ); } export async function verifyNativeComposeStorageXattrWitness( opts: Omit< @@ -1408,7 +1422,9 @@ export async function verifyNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - await verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }); + await x01Scope("witness-verify", () => + verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }) + ); } /** Private saved observation seam. It never performs a kernel read, repairs an diff --git a/src/lib/shell.ts b/src/lib/shell.ts index ef9faf1ba..ff27e6b85 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -4,6 +4,7 @@ import { } from "./native-cpu-diagnostics.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; +import { x01Label, x01Time } from "./x01-cost-probe.ts"; export interface ExecResult { readonly exitCode: number; @@ -40,7 +41,13 @@ function buildSpawnEnv( return override ? { ...base, ...override } : base; } -export async function exec( +export function exec( + cmd: readonly string[], + opts: ExecOptions = {} +): Promise { + return x01Time("exec", x01Label(cmd), () => x01ExecInner(cmd, opts)); +} +async function x01ExecInner( cmd: readonly string[], opts: ExecOptions = {} ): Promise { @@ -123,7 +130,13 @@ export type RunExitEvent = { readonly maxRssBytes: number | null; }; -export async function run( +export function run( + cmd: readonly string[], + opts: RunOptions = {} +): Promise { + return x01Time("run", x01Label(cmd), () => x01RunInner(cmd, opts)); +} +async function x01RunInner( cmd: readonly string[], opts: RunOptions = {} ): Promise { diff --git a/src/lib/x01-cost-probe.ts b/src/lib/x01-cost-probe.ts new file mode 100644 index 000000000..48848c688 --- /dev/null +++ b/src/lib/x01-cost-probe.ts @@ -0,0 +1,171 @@ +// TEMPORARY X01 measurement probe. Reverted before the final commit. +// Writes closed labels and durations only (no ids, paths or values). +import { AsyncLocalStorage } from "node:async_hooks"; +import { appendFileSync } from "node:fs"; + +const FILE = process.env.HACK_X01_TRACE_FILE; +const VERBS = new Set([ + "up", + "down", + "exec", + "run", + "restart", + "ps", + "logs", + "status", + "init", +]); +const SUBS = new Set([ + "info", + "image", + "container", + "volume", + "network", + "compose", + "create", + "start", + "rm", + "inspect", + "ls", + "ps", + "exec", + "run", + "logs", + "version", + "events", + "stop", + "kill", + "wait", + "up", + "down", + "config", + "pull", + "build", + "restart", + "port", + "cp", + "top", +]); +const verb = (() => { + for (const arg of process.argv.slice(2)) { + if (VERBS.has(arg)) { + return arg; + } + } + return "other"; +})(); +const scopes = new AsyncLocalStorage(); +let enabled = typeof FILE === "string" && FILE.startsWith("/"); +let wroteStart = false; + +function append(record: Record): void { + appendFileSync( + FILE as string, + `${JSON.stringify({ p: process.pid, v: verb, ...record })}\n` + ); +} + +function write(record: Record): void { + if (!enabled) { + return; + } + try { + if (!wroteStart) { + wroteStart = true; + append({ k: "start", t: Math.round(performance.now()) }); + process.on("exit", (code) => { + try { + append({ k: "exit", c: code, t: Math.round(performance.now()) }); + } catch { + // measurement only + } + }); + } + append({ + s: scopes.getStore() ?? "", + t: Math.round(performance.now()), + ...record, + }); + } catch { + enabled = false; + } +} + +export function x01Enabled(): boolean { + return enabled; +} + +/** Raw line from the phase trace, re-wrapped with process identity. */ +export function x01Phase(line: string): void { + try { + write({ k: "phase", line: JSON.parse(line) as unknown }); + } catch { + enabled = false; + } +} + +export function x01Label(argv: readonly string[], binary = true): string { + const parts: string[] = []; + let rest = argv; + if (binary) { + const name = (argv[0] ?? "").split("/").at(-1) ?? ""; + parts.push( + ["docker", "ps", "sysctl", "sh", "git", "bun", "tmux"].includes(name) + ? name + : "bin" + ); + rest = argv.slice(1); + } else { + parts.push("docker"); + } + for (const token of rest) { + if (parts.length >= 3) { + break; + } + if (token.startsWith("-")) { + continue; + } + if (SUBS.has(token)) { + parts.push(token); + } else { + break; + } + } + return parts.join(" "); +} + +export async function x01Time( + kind: string, + label: string, + operation: () => Promise +): Promise { + if (!enabled) { + return await operation(); + } + const started = performance.now(); + let ok = false; + try { + const result = await operation(); + ok = true; + return result; + } finally { + write({ + k: kind, + l: label, + ms: Math.round((performance.now() - started) * 10) / 10, + ok, + }); + } +} + +export async function x01Scope( + label: string, + operation: () => Promise +): Promise { + if (!enabled) { + return await operation(); + } + const parent = scopes.getStore(); + const path = parent ? `${parent}>${label}` : label; + return await scopes.run(path, () => x01Time("scope", label, operation)); +} diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index 2201457b1..e7599c528 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,6 +43,8 @@ const PASSTHROUGH_ENV_KEYS = [ "LC_ALL", "DOCKER_HOST", "DOCKER_CONFIG", + // TEMPORARY X01 measurement passthrough; reverted before the final commit. + "HACK_X01_TRACE_FILE", ] as const; export type CliResult = { @@ -768,6 +770,13 @@ export async function runScenarios( }; process.stdout.write(`-- ${scenario.name} (${scenario.tier}) --\n`); + // TEMPORARY X01 measurement: one trace file per scenario. + if (process.env.HACK_X01_TRACE_DIR?.startsWith("/")) { + process.env.HACK_X01_TRACE_FILE = join( + process.env.HACK_X01_TRACE_DIR, + `${scenario.name}.jsonl` + ); + } let outcome: ScenarioOutcome; let preserveFailedFixture = false; try { diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index 2728a6059..a21575961 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -1153,6 +1153,10 @@ export const nativeConfigNetworksScenario: Scenario = { NO_COLOR: "1", HACK_RUNTIME_BACKEND: "compose", HACK_DAEMON_DISABLE_DOCKER_EVENTS: "1", + // TEMPORARY X01 measurement passthrough; reverted before the final commit. + ...(process.env.HACK_X01_TRACE_FILE + ? { HACK_X01_TRACE_FILE: process.env.HACK_X01_TRACE_FILE } + : {}), }; const engine = Bun.which("docker"); const git = Bun.which("git"); From 0911e6de6f7c57a31c01296c40f20e238ac60b7d Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 23:36:51 -0400 Subject: [PATCH 13/32] perf(storage): coalesce witness helper work per native command A storage-bearing saved exec ran three full witness proofs (nine helper carriers) and a warm up/restart seven. Within one command and one mutation lease, an adjacent repeat with only read-only checks between it and the previous proof now reuses that completed kernel proof instead of three new helper carriers: - saved exec: the pre-spawn check reuses the preparation proof; the proof after the process stays fresh (3 -> 2 proofs); - up/restart without build, routing or file material: the first pre-spawn guard reuses the enrollment proof (warm 7 -> 6, cold seed path 6 -> 5). Reuse requires the same lease authority, engine, generation and reference and a byte-identical opening material binding to the one that opened and closed the recorded proof. Every host check (anchors, expectation, completion, binding, carrier artifact and lifetime) still runs; any verification consumes the record first, so a failure or binding change forces a fresh proof. Ordinary verification always proves. Co-Authored-By: Claude Opus 5.5 --- src/lib/native-compose-command-storage.ts | 35 ++++-- src/lib/native-compose-command.ts | 28 ++++- src/lib/native-compose-storage-witness.ts | 112 ++++++++++++++---- tests/helpers/native-compose-command.ts | 1 + ...compose-storage-activation-command.test.ts | 58 +++++++++ ...ompose-storage-witness-xattr-owner.test.ts | 108 +++++++++++++++++ 6 files changed, 308 insertions(+), 34 deletions(-) diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index ce45846c8..d9b58c132 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -128,6 +128,11 @@ async function volumeNames(signal: AbortSignal): Promise { return names; } +export type NativeComposeCommandStorageVerifyOptions = { + /** Only when nothing but read-only checks ran since this lease's previous proof. */ + readonly adjacentProof?: true; +}; + /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ export function prepareNativeComposeCommandStorage( @@ -226,9 +231,14 @@ async function x01Prepare(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = (generation: NativeComposeGeneration) => - x01Scope("storage-verify", () => x01Verify(generation)); - const x01Verify = async (generation: NativeComposeGeneration) => { + const verify = ( + generation: NativeComposeGeneration, + options: NativeComposeCommandStorageVerifyOptions = {} + ) => x01Scope("storage-verify", () => x01Verify(generation, options)); + const x01Verify = async ( + generation: NativeComposeGeneration, + options: NativeComposeCommandStorageVerifyOptions = {} + ) => { await assertFresh?.(); const before = await store.loadCurrent(); if (before.storageWitnessesPending) { @@ -248,6 +258,9 @@ async function x01Prepare(opts: { engineId, reference: entry.reference, carrier: await carrier(), + ...(options.adjacentProof === true + ? { adjacentProof: true as const } + : {}), }); } const after = await store.loadCurrent(); @@ -328,18 +341,24 @@ export type NativeComposeCommandStorage = Awaited< ReturnType >; -/** Saved exec does not publish a ready receipt. Cancellation preserves its known - * exit and starts no late helper; every future admission re-verifies content. */ +/** Saved exec does not publish a ready receipt. The opening check directly + * follows command storage preparation under the same lease, with only read-only + * source and ownership checks between, so it may reuse that adjacent proof. The + * closing check after the process always performs a fresh proof. Cancellation + * preserves its known exit and starts no late helper; every future admission + * re-verifies content. */ export async function runNativeComposeStorageVerifiedExec(opts: { readonly signal: AbortSignal; - readonly assertSaved: () => Promise; + readonly assertSaved: ( + options: NativeComposeCommandStorageVerifyOptions + ) => Promise; readonly run: () => Promise; }): Promise { const { signal, assertSaved, run } = opts; - await assertSaved(); + await assertSaved({ adjacentProof: true }); const code = await run(); if (!signal.aborted) { - await assertSaved(); + await assertSaved({}); } return code; } diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index 9701043c2..0aa92b81e 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -26,6 +26,7 @@ import { } from "./native-compose-build.ts"; import { type NativeComposeCommandStorage, + type NativeComposeCommandStorageVerifyOptions, nativeComposePlanStorage, nativeComposeSavedExecUsesReadLease, prepareNativeComposeCommandStorage, @@ -943,8 +944,11 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = () => x01Scope("exec-assert-saved", x01AssertSaved); - const x01AssertSaved = async () => { + const assertSaved = (proof: NativeComposeCommandStorageVerifyOptions) => + x01Scope("exec-assert-saved", () => x01AssertSaved(proof)); + const x01AssertSaved = async ( + proof: NativeComposeCommandStorageVerifyOptions + ) => { const latest = await store.loadCurrent(); assertStartupAvailable(latest); if ( @@ -955,7 +959,7 @@ async function savedCommand(opts: { } await store.readGenerationDocument(generation); await assertNativeComposeOwned(selection); - await storage?.verify(generation); + await storage?.verify(generation, proof); }; return await runNativeComposeStorageVerifiedExec({ signal, @@ -1665,6 +1669,15 @@ async function executePreparedGeneration(opts: { }); }, }); + // Up/restart enrollment ends with a storage proof. With no build, routing or + // file material, the first pre-spawn guard follows it with only read-only + // source and ownership checks (arming is a no-op), so that guard may reuse the + // adjacent proof. Every later guard and finalization proof stays fresh. + let adjacentGuardProof = + (operation === "up" || operation === "restart") && + builds.length === 0 && + routing === null && + files === null; const result = await mutation.runEffect({ generation, operation, @@ -1699,7 +1712,14 @@ async function executePreparedGeneration(opts: { assertFresh, assertOwned: () => ownership.assertOwned(), verifyStorage: storage - ? () => storage.verify(generation) + ? () => { + const adjacentProof = adjacentGuardProof; + adjacentGuardProof = false; + return storage.verify( + generation, + adjacentProof ? { adjacentProof } : {} + ); + } : undefined, }), }); diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index 0650f4b49..bfcbfe4e2 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -1245,7 +1245,62 @@ export async function enrollNativeComposeStorageWitness(opts: { }).catch((error) => normalizedRefusal(error, stage)); } -/** Read-only verification. No missing journal/marker, including interrupted enrollment, is seeded or repaired. */ +/** + * Completed directory-xattr kernel proofs of one mutation lease. A proof is keyed + * by its exact engine, generation and reference and holds the material binding + * that opened and closed it unchanged. The lease's authority object is the + * WeakMap key, so a later command, lease or revoked authority never sees it. + */ +const leaseProofs = new WeakMap< + NativeComposeMaterialAuthority, + Map +>(); +/** Take (and clear) this verification's lease record before any check, so a + * failed or refused verification never leaves an older proof reusable. */ +function takeLeaseProof(opts: { + readonly authority: NativeComposeMaterialAuthority; + readonly engineId: string; + readonly generation: NativeComposeGeneration; + readonly reference: NativeComposeStorageWitnessReference; + readonly adjacent: boolean; +}) { + const proofs = leaseProofs.get(opts.authority) ?? new Map(); + leaseProofs.set(opts.authority, proofs); + const key = JSON.stringify([ + opts.engineId, + opts.generation.generationId, + opts.generation.inputRevision, + opts.reference, + ]); + const recorded = proofs.get(key); + proofs.delete(key); + return { + /** Run the kernel proof unless an adjacent caller opens with the recorded binding. */ + async prove(opening: string, run: () => Promise): Promise { + if (!(opts.adjacent && recorded === opening)) { + await run(); + } + }, + /** Record a completed proof only when its binding never moved. */ + record(opening: string, closing: string): void { + if (opening === closing) { + proofs.set(key, opening); + } + }, + }; +} + +/** + * Read-only verification. No missing journal/marker, including interrupted enrollment, is seeded or repaired. + * + * `adjacentProof` is only for a caller whose code since its previous storage + * proof under this same lease performed no hook, workload, engine or storage + * effect. Such a repeat runs every host check (anchors, expectation, completion, + * binding) but reuses the completed kernel proof instead of three new helper + * carriers, and only when its opening material binding is byte-identical to the + * binding that opened and closed that proof. Any other verification, and any + * mismatch, performs the full proof and replaces the lease record. + */ export async function verifyNativeComposeStorageWitness(opts: { readonly authority: NativeComposeMaterialAuthority; readonly generation: NativeComposeGeneration; @@ -1256,6 +1311,7 @@ export async function verifyNativeComposeStorageWitness(opts: { readonly archive: Uint8Array; }>; readonly xattrCarrier?: NativeComposeStorageXattrCarrier; + readonly adjacentProof?: true; }): Promise { const { authority, generation, engineId, observe, xattrCarrier } = opts; if (!nativeComposeStorageWitnessReferenceValid(opts.reference)) { @@ -1284,11 +1340,19 @@ export async function verifyNativeComposeStorageWitness(opts: { return await runNativeComposeMaterialAction({ authority, run: async () => { + const lease = takeLeaseProof({ + authority, + engineId, + generation, + reference, + adjacent: opts.adjacentProof === true, + }); const current = await assertNativeComposeMaterialAuthority({ authority, generation, phase: "inspect", }); + const opening = JSON.stringify(current); const held = await directories({ binding: current, name: reference.volume.name, @@ -1334,27 +1398,29 @@ export async function verifyNativeComposeStorageWitness(opts: { if (!xattrCarrier) { return refuse(); } - await observeXattr({ - carrier: xattrCarrier, - saved, - volume: reference.volume, - stopped: false, - journal: last(held), - check: async () => { - await checkedExpectation(held, reference.expectation); - await readCompletion(); - await recheckDirectories(held); - const latest = await assertNativeComposeMaterialAuthority({ - authority, - generation, - phase: "inspect", - }); - if (!matchesBinding(saved.binding, latest, engineId)) { - return refuse(); - } - return latest; - }, - }); + await lease.prove(opening, () => + observeXattr({ + carrier: xattrCarrier, + saved, + volume: reference.volume, + stopped: false, + journal: last(held), + check: async () => { + await checkedExpectation(held, reference.expectation); + await readCompletion(); + await recheckDirectories(held); + const latest = await assertNativeComposeMaterialAuthority({ + authority, + generation, + phase: "inspect", + }); + if (!matchesBinding(saved.binding, latest, engineId)) { + return refuse(); + } + return latest; + }, + }) + ); } await checkedExpectation(held, reference.expectation); await readCompletion(); @@ -1369,6 +1435,8 @@ export async function verifyNativeComposeStorageWitness(opts: { } if (xattrCarrier) { checkNativeComposeStorageXattrCarrierLifetime(xattrCarrier); + // A v3 reference with a carrier always reached the kernel proof above. + lease.record(opening, JSON.stringify(latest)); } } finally { await close(held); diff --git a/tests/helpers/native-compose-command.ts b/tests/helpers/native-compose-command.ts index 37e01b083..460e97148 100644 --- a/tests/helpers/native-compose-command.ts +++ b/tests/helpers/native-compose-command.ts @@ -177,6 +177,7 @@ if(args[0]==="compose") { } if(oneoff && args.includes("run")) {const doc=await Bun.file(args[args.indexOf("-f")+1]).json();await Bun.write(engine,JSON.stringify(doc));await Bun.write(root+"/oneoff-name",args[args.indexOf("--name")+1]);process.exit(0);} if(args.includes("down")) {await rm(engine,{force:true});await appendFile(root+"/order","engine-stopped\\n");process.exit(0);} + if(args.includes("exec") && await Bun.file(engine).exists()) {${opts.storage ? 'await appendFile(root+"/storage-events","exec\\n");' : ""}process.exit(0);} process.exit(99); } if(oneoff && args[0]==="container" && args[1]==="rm") {if(args.length!==3||args[2]!=="c".repeat(64))process.exit(96);await Bun.write(root+"/oneoff-removed","removed");process.exit(0);} diff --git a/tests/native-compose-storage-activation-command.test.ts b/tests/native-compose-storage-activation-command.test.ts index 6ad088fdd..93267d2d7 100644 --- a/tests/native-compose-storage-activation-command.test.ts +++ b/tests/native-compose-storage-activation-command.test.ts @@ -378,3 +378,61 @@ test("public down recover completes only removed readonly original work and next } }); }, 500_000); + +// One proof is root, verify and root-after through three fresh helper carriers. +// Warm up/restart keep the opening, both effect-phase, enrollment and both +// finalization proofs; only the first pre-spawn guard reuses the adjacent +// enrollment proof (no build, routing or file material). Saved exec keeps its +// opening proof, reuses it for the adjacent pre-spawn check and proves again +// after the process. A cold up seeds once and keeps every later proof. +test( + "source CLI helper invocations per command keep every non-adjacent proof", + async () => + await ownedColdCase(async (owner) => { + const root = await fixture("", false, { + storage, + noHooks: true, + cleanupAllowed: owner.cleanupAllowed, + }); + const eventsFile = join(root, "storage-events"); + const events = async () => + (await Bun.file(eventsFile).exists()) + ? (await Bun.file(eventsFile).text()).trim().split("\n") + : []; + const counted = async (args: readonly string[]) => { + const before = (await events()).length; + const result = await owner.invoke(root, args); + expect(result.code, await disposition(root, result.code)).toBe(0); + return (await events()).slice(before); + }; + const proofs = (count: number) => + Array.from({ length: count }, () => ["root", "verify", "root"]).flat(); + expect(await counted(["up", "--detach", "--json"])).toEqual([ + "root", + "seed", + ...proofs(3), + "workload", + ...proofs(2), + ]); + expect(await counted(["up", "--detach", "--json"])).toEqual([ + ...proofs(4), + "workload", + ...proofs(2), + ]); + expect(await counted(["exec", "web", "--", "true"])).toEqual([ + ...proofs(1), + "exec", + ...proofs(1), + ]); + expect(await counted(["restart", "--json"])).toEqual([ + ...proofs(4), + "workload", + ...proofs(2), + ]); + expect( + (await operations(root)).filter((op) => op === "seed") + ).toHaveLength(1); + expect(await Bun.file(join(root, "carriers")).json()).toEqual([]); + }), + 390_000 +); diff --git a/tests/native-compose-storage-witness-xattr-owner.test.ts b/tests/native-compose-storage-witness-xattr-owner.test.ts index 7692b8405..2cb45c0bc 100644 --- a/tests/native-compose-storage-witness-xattr-owner.test.ts +++ b/tests/native-compose-storage-witness-xattr-owner.test.ts @@ -2869,3 +2869,111 @@ test("ordinary inspection authority cannot enter saved carrier recovery observat }); expect(observed).toBe(0); }, 30_000); + +function adjacentVerifier(opts: { + readonly transport: ReturnType; + readonly generation: NativeComposeGeneration; + readonly reference: Parameters< + typeof verifyNativeComposeStorageXattrWitness + >[0]["reference"]; +}) { + return (mutation: NativeComposeMutation, adjacent: boolean) => + verifyNativeComposeStorageXattrWitness({ + authority: mutation.materialAuthority, + generation: opts.generation, + engineId, + reference: opts.reference, + carrier: opts.transport.carrier, + ...(adjacent ? { adjacentProof: true as const } : {}), + }); +} + +test("adjacent storage proof is reused only within its own lease", async () => { + const { store, transport, generation, reference } = await active(); + const verify = adjacentVerifier({ transport, generation, reference }); + transport.calls.length = 0; + await store.withMutation(async (mutation) => { + // Nothing to reuse yet: the first adjacent request performs the proof. + await verify(mutation, true); + expect(helperOperations(transport)).toEqual(["root", "verify", "root"]); + await verify(mutation, true); + await verify(mutation, true); + expect(helperOperations(transport)).toHaveLength(3); + // Ordinary verification always performs a fresh proof. + await verify(mutation, false); + expect(helperOperations(transport)).toHaveLength(6); + }); + await store.withMutation(async (mutation) => { + await verify(mutation, true); + expect(helperOperations(transport)).toHaveLength(9); + }); + expect(transport.calls).not.toContain("seed"); + expect(transport.calls).not.toContain("provision"); +}, 30_000); + +test("adjacent storage proof still runs every host check and never replaces the next fresh proof", async () => { + const { store, transport, generation, reference } = await active(); + const verify = adjacentVerifier({ transport, generation, reference }); + const slot = join( + store.identity.checkoutRoot, + ".hack", + ".internal", + "native-compose", + store.identity.instanceId, + "storage-witnesses", + createHash("sha256").update(volume.name).digest("hex") + ); + transport.calls.length = 0; + await store.withMutation(async (mutation) => { + await verify(mutation, false); + // Host evidence is rechecked on reuse; a missing completion record refuses. + await rename(join(slot, "enrolled.json"), join(slot, "moved.json")); + await expect(verify(mutation, true)).rejects.toThrow("values omitted"); + await rename(join(slot, "moved.json"), join(slot, "enrolled.json")); + // The refusal cleared the lease record, so the repeat proves again. + await verify(mutation, true); + expect(helperOperations(transport)).toHaveLength(6); + // A reused kernel proof stands only for its adjacent caller. + transport.attributes.clear(); + await verify(mutation, true); + expect(helperOperations(transport)).toHaveLength(6); + await expect(verify(mutation, false)).rejects.toThrow("values omitted"); + await expect(verify(mutation, true)).rejects.toThrow("values omitted"); + }); + expect(transport.calls).not.toContain("seed"); +}, 30_000); + +test("adjacent storage proof is not reused across a changed material binding", async () => { + const { store, transport, generation, reference } = await active(); + const verify = adjacentVerifier({ transport, generation, reference }); + transport.calls.length = 0; + await store.withMutation(async (mutation) => { + await verify(mutation, false); + expect(helperOperations(transport)).toHaveLength(3); + // Intent publication changes the receipt before the effect body runs. + const result = await mutation.runEffect({ + generation, + operation: "up", + assertFresh: async () => {}, + assertOwned: async () => {}, + captureStorage: () => [volume], + storageWitnesses: { + kind: "directory-xattr", + engineId, + carrier: transport.carrier, + }, + effect: async () => { + const entry = helperOperations(transport).length; + // The effect-entry proof under the published intent is adjacent. + await verify(mutation, true); + expect(helperOperations(transport)).toHaveLength(entry); + return { value: 0, outcome: "complete" }; + }, + }); + expect(result).toEqual({ value: 0, outcome: "complete" }); + // After completion the receipt moved again; nothing earlier is reusable. + const finished = helperOperations(transport).length; + await verify(mutation, true); + expect(helperOperations(transport)).toHaveLength(finished + 3); + }); +}, 30_000); From eb9a678597fe7ad46fb4ca938b70bea2b819e2c6 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Sun, 11 Oct 2026 23:42:17 -0400 Subject: [PATCH 14/32] chore(x01): revert temporary witness cost trace Reverts the hosted measurement commit (closed-label cost trace, phase-trace routing, per-scenario summary step and artifact) now that the per-phase and per-call attribution is recorded in the pull request. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 19 -- scripts/x01-trace-summary.ts | 268 ------------------ src/lib/native-compose-command-storage.ts | 14 +- src/lib/native-compose-command.ts | 9 +- src/lib/native-compose-ownership.ts | 5 +- src/lib/native-compose-phase-trace.ts | 8 +- .../native-compose-storage-witness-docker.ts | 43 +-- src/lib/native-compose-storage-witness.ts | 34 +-- src/lib/shell.ts | 17 +- src/lib/x01-cost-probe.ts | 171 ----------- tests/e2e/harness.ts | 9 - .../e2e/native-config-networks-acceptance.ts | 4 - 12 files changed, 28 insertions(+), 573 deletions(-) delete mode 100644 scripts/x01-trace-summary.ts delete mode 100644 src/lib/x01-cost-probe.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 34c2387cc..457db89f4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -306,10 +306,7 @@ jobs: HACK_E2E_CAPSULE_DIR: ${{ runner.temp }}/process-policy-capsules HACK_E2E_CLI_BIN: ${{ github.workspace }}/dist/hack HACK_E2E_SHARD: ${{ matrix.shard }}/4 - # TEMPORARY X01 measurement; reverted before the final commit. - HACK_X01_TRACE_DIR: ${{ runner.temp }}/x01-trace run: | - mkdir -p "$HACK_X01_TRACE_DIR" HACK_E2E_DOCKER_HOST="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" HACK_E2E_NETWORK_SOURCE_REVISION="$(git rev-parse HEAD)" HACK_E2E_NETWORK_CLI_SHA256="$(sha256sum dist/hack | cut -d ' ' -f1)" @@ -318,22 +315,6 @@ jobs: HACK_E2E_COMPOSE_PLUGIN_SHA256="$(sha256sum "$HACK_E2E_COMPOSE_PLUGIN_PATH" | cut -d ' ' -f1)" export HACK_E2E_DOCKER_HOST HACK_E2E_NETWORK_SOURCE_REVISION HACK_E2E_NETWORK_CLI_SHA256 HACK_E2E_NETWORK_COMPILER_SHA256 HACK_E2E_COMPOSE_PLUGIN_PATH HACK_E2E_COMPOSE_PLUGIN_SHA256 HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker --shard="$HACK_E2E_SHARD" - # TEMPORARY X01 measurement; reverted before the final commit. - - name: X01 witness cost summary - if: always() - run: | - for f in "$RUNNER_TEMP"/x01-trace/*.jsonl; do - [ -f "$f" ] || continue - bun scripts/x01-trace-summary.ts "$f" "$(basename "$f" .jsonl)" || true - done - - name: X01 witness cost trace artifact - if: always() - uses: actions/upload-artifact@v4 - with: - name: x01-trace-shard-${{ matrix.shard }} - path: ${{ runner.temp }}/x01-trace - if-no-files-found: ignore - retention-days: 7 - name: Preserve process-policy capsule and startup diagnostic # Closed fixed-field JSON only (counts, enums, booleans); the gh log viewer truncates it inline. if: failure() diff --git a/scripts/x01-trace-summary.ts b/scripts/x01-trace-summary.ts deleted file mode 100644 index ba276c07f..000000000 --- a/scripts/x01-trace-summary.ts +++ /dev/null @@ -1,268 +0,0 @@ -// TEMPORARY X01 measurement summary. Reverted before the final commit. -// Reads closed-label JSONL written by src/lib/x01-cost-probe.ts. -type Event = { - readonly p: number; - readonly v: string; - readonly k: string; - readonly s?: string; - readonly l?: string; - readonly ms?: number; - readonly t?: number; - readonly c?: number; - readonly line?: { - readonly phase?: string; - readonly boundary?: string; - readonly durationMs?: number | null; - }; -}; -const WITNESS = new Set([ - "storage-prepare", - "storage-verify", - "carrier-new", - "inspect", - "provision", - "finish", - "witness-verify", - "witness-prepare", - "witness-enroll", -]); -const isWitness = (label: string) => - WITNESS.has(label) || label.startsWith("invoke:"); -const LEAF = new Set(["probe", "run", "exec", "hash"]); -const PHASE = /^(oneoff|storage|guard|finish|finalize)\.[a-z-]+$/; - -const path = process.argv[2]; -const title = process.argv[3] ?? path; -if (!path) { - throw new Error("usage: x01-trace-summary [title]"); -} -const text = await Bun.file(path).text(); -const events: Event[] = []; -for (const line of text.split("\n")) { - if (line.trim() === "") { - continue; - } - try { - events.push(JSON.parse(line) as Event); - } catch { - // skip partial line - } -} -const byProcess = new Map(); -for (const event of events) { - const list = byProcess.get(event.p) ?? []; - list.push(event); - byProcess.set(event.p, list); -} -type Proc = { - readonly pid: number; - readonly verb: string; - readonly totalMs: number; - readonly witnessMs: number; - readonly events: Event[]; - readonly kind: string; -}; -function topWitnessMs(list: Event[]): number { - let total = 0; - for (const event of list) { - if (event.k !== "scope" || !event.l || !isWitness(event.l)) { - continue; - } - const parts = (event.s ?? "").split(">"); - parts.pop(); - if (parts.some(isWitness)) { - continue; - } - total += event.ms ?? 0; - } - return total; -} -const procs: Proc[] = []; -for (const [pid, list] of byProcess) { - const exit = list.find((event) => event.k === "exit"); - const verb = list[0]?.v ?? "other"; - const labels = new Set(list.map((event) => event.l ?? "")); - let kind = verb; - if (verb === "up" || verb === "restart" || verb === "run") { - if (labels.has("invoke:seed")) { - kind = `${verb}(cold)`; - } else if (labels.has("invoke:verify")) { - kind = `${verb}(warm)`; - } else { - kind = `${verb}(no-witness)`; - } - } else if (verb === "exec") { - kind = labels.has("invoke:verify") ? "exec(storage)" : "exec(image)"; - } - procs.push({ - pid, - verb, - totalMs: exit?.t ?? Math.max(...list.map((event) => event.t ?? 0)), - witnessMs: topWitnessMs(list), - events: list, - kind, - }); -} -const fmt = (ms: number) => (ms / 1000).toFixed(1); -const out: string[] = []; -out.push(`#### X01 trace: ${title}`); -out.push(`events=${events.length} processes=${procs.length}`); -out.push(""); -out.push( - "kind | n | avg total s | sum total s | avg witness s | sum witness s" -); -const kinds = [...new Set(procs.map((proc) => proc.kind))].sort(); -for (const kind of kinds) { - const selected = procs.filter((proc) => proc.kind === kind); - const total = selected.reduce((sum, proc) => sum + proc.totalMs, 0); - const witness = selected.reduce((sum, proc) => sum + proc.witnessMs, 0); - out.push( - `${kind} | ${selected.length} | ${fmt(total / selected.length)} | ${fmt(total)} | ${fmt(witness / selected.length)} | ${fmt(witness)}` - ); -} -const allTotal = procs.reduce((sum, proc) => sum + proc.totalMs, 0); -const allWitness = procs.reduce((sum, proc) => sum + proc.witnessMs, 0); -out.push( - `ALL | ${procs.length} | - | ${fmt(allTotal)} | - | ${fmt(allWitness)}` -); -out.push(""); - -function leafTable(selected: Proc[], heading: string, limit = 24) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (!(LEAF.has(event.k) && event.l)) { - continue; - } - const key = `${event.k}:${event.l}`; - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push(`${heading} (per process avg over n=${selected.length})`); - out.push("leaf | calls/proc | s/proc | ms/call"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function leafByScope(selected: Proc[], heading: string, limit = 40) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (!(LEAF.has(event.k) && event.l)) { - continue; - } - const parts = (event.s ?? "") - .split(">") - .filter( - (part) => - isWitness(part) || part.startsWith("cmd:") || PHASE.test(part) - ); - const inner = parts.slice(-2).join(">") || "(none)"; - const key = `${inner} :: ${event.k}:${event.l}`; - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push( - `${heading} leaves by scope (per process avg over n=${selected.length})` - ); - out.push("scope :: leaf | calls/proc | s/proc | ms/call"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function scopeTable(selected: Proc[], heading: string, limit = 30) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (event.k !== "scope" || !event.l) { - continue; - } - const parts = (event.s ?? "").split(">"); - const key = parts - .filter( - (part) => - PHASE.test(part) || - isWitness(part) || - part.startsWith("xattr:") || - part.startsWith("cmd:") || - part.startsWith("exec-") - ) - .join(">"); - if (key === "") { - continue; - } - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push(`${heading} scopes (per process avg over n=${selected.length})`); - out.push("scope path | n/proc | s/proc | ms/each"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function phaseRows(proc: Proc) { - const rows = new Map(); - for (const event of proc.events) { - if ( - event.k === "phase" && - event.line?.boundary !== "begin" && - event.line?.phase - ) { - const row = rows.get(event.line.phase) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.line.durationMs ?? 0; - rows.set(event.line.phase, row); - } - } - return rows; -} -for (const kind of [ - "up(cold)", - "up(warm)", - "exec(storage)", - "restart(warm)", - "run(warm)", - "run(cold)", - "down", -]) { - const selected = procs.filter((proc) => proc.kind === kind); - if (selected.length === 0) { - continue; - } - const sample = selected[0] as Proc; - out.push( - `### ${kind}: first sample total ${fmt(sample.totalMs)} s, witness ${fmt(sample.witnessMs)} s` - ); - out.push("phase-trace phase | spans | s"); - for (const [phase, row] of phaseRows(sample)) { - out.push(`${phase} | ${row.n} | ${fmt(row.ms)}`); - } - out.push(""); - scopeTable([sample], `${kind} first sample`, 40); - leafTable([sample], `${kind} first sample leaves`); - scopeTable(selected, `${kind} all`, 30); - leafTable(selected, `${kind} all leaves`); - leafByScope(selected, `${kind} all`); -} -console.log(out.join("\n")); diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index d9b58c132..c9384e64c 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -23,7 +23,6 @@ import { } from "./native-compose-storage-witness.ts"; import { createNativeComposeDockerStorageXattrCarrier } from "./native-compose-storage-witness-docker.ts"; import type { NativeComposeStorageXattrCarrier } from "./native-compose-storage-witness-xattr-carrier.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; type Selection = { readonly name: string; readonly storage: string }; type Document = Readonly>; @@ -135,12 +134,7 @@ export type NativeComposeCommandStorageVerifyOptions = { /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ -export function prepareNativeComposeCommandStorage( - opts: Parameters[0] -): ReturnType { - return x01Scope("storage-prepare", () => x01Prepare(opts)); -} -async function x01Prepare(opts: { +export async function prepareNativeComposeCommandStorage(opts: { readonly store: NativeComposeGenerationStore; readonly mutation: NativeComposeMutation; readonly operation: "up" | "restart" | "run" | "exec"; @@ -231,11 +225,7 @@ async function x01Prepare(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = ( - generation: NativeComposeGeneration, - options: NativeComposeCommandStorageVerifyOptions = {} - ) => x01Scope("storage-verify", () => x01Verify(generation, options)); - const x01Verify = async ( + const verify = async ( generation: NativeComposeGeneration, options: NativeComposeCommandStorageVerifyOptions = {} ) => { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index 0aa92b81e..8c5dcaf82 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -138,7 +138,6 @@ import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; export type NativeComposeCommandOptions = { @@ -944,9 +943,7 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = (proof: NativeComposeCommandStorageVerifyOptions) => - x01Scope("exec-assert-saved", () => x01AssertSaved(proof)); - const x01AssertSaved = async ( + const assertSaved = async ( proof: NativeComposeCommandStorageVerifyOptions ) => { const latest = await store.loadCurrent(); @@ -965,9 +962,7 @@ async function savedCommand(opts: { signal, assertSaved, run: () => - x01Scope("exec-process", () => - runSavedProcess({ options, generation, document, base, signal }) - ), + runSavedProcess({ options, generation, document, base, signal }), }); }); } diff --git a/src/lib/native-compose-ownership.ts b/src/lib/native-compose-ownership.ts index d8bc9dc8a..0019b3fc3 100644 --- a/src/lib/native-compose-ownership.ts +++ b/src/lib/native-compose-ownership.ts @@ -3,7 +3,6 @@ import { isRecord } from "./guards.ts"; import { nativeComposeVolumeCreatedAt } from "./native-compose-retained-storage.ts"; import { beginNativeCpuChild } from "./native-cpu-diagnostics.ts"; import { findExecutableInPath } from "./shell.ts"; -import { x01Label, x01Time } from "./x01-cost-probe.ts"; const PROJECT = /^[a-z0-9][a-z0-9_-]*$/; const SERVICE = /^[a-z0-9][a-z0-9._-]{0,62}$/; @@ -1233,7 +1232,7 @@ export function createNativeComposeProbe( probeRefused("E_NATIVE_COMPOSE_PROBE", "operation"); } const environment = { ...process.env }; - const x01Inner = async (args: readonly string[]): Promise => { + return async (args) => { checkInterruption(opts.signal, false); const remainingTime = deadline - Date.now(); if (remainingTime <= 0) { @@ -1311,8 +1310,6 @@ export function createNativeComposeProbe( observeCpu(probeExitCode(completion[2])); } }; - return (args) => - x01Time("probe", x01Label(args, false), () => x01Inner(args)); } function probeExitCode( diff --git a/src/lib/native-compose-phase-trace.ts b/src/lib/native-compose-phase-trace.ts index 943b79320..540f833b0 100644 --- a/src/lib/native-compose-phase-trace.ts +++ b/src/lib/native-compose-phase-trace.ts @@ -1,6 +1,5 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { writeSync } from "node:fs"; -import { x01Enabled, x01Phase, x01Scope } from "./x01-cost-probe.ts"; export const NATIVE_COMPOSE_PHASE_TRACE = "HACK_NATIVE_COMPOSE_PHASE_TRACE"; const PHASES = [ @@ -136,8 +135,6 @@ export function withNativeComposePhaseTrace(operation: () => Promise) { writeSync(2, line); }, }); - } else if (x01Enabled()) { - trace = createNativeComposePhaseTrace({ write: x01Phase }); } } catch { // Diagnostic setup never changes selection, results or effect authority. @@ -150,8 +147,5 @@ export function measureNativeComposePhase( phase: Phase, operation: () => Promise ): Promise { - return x01Scope( - phase, - () => context.getStore()?.measure(phase, operation) ?? operation() - ); + return context.getStore()?.measure(phase, operation) ?? operation(); } diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index da785e7b4..0c04e7ba5 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -96,7 +96,6 @@ import { refuseNativeComposeStorageXattr as refuse, } from "./native-compose-storage-witness-xattr-codec.ts"; import { findExecutableInPath, type RunExitEvent, run } from "./shell.ts"; -import { x01Label, x01Scope, x01Time } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const PROCESS = @@ -814,12 +813,7 @@ async function groupAbsent( await new Promise((resolve) => setTimeout(resolve, 20)); } } -function commandExecutable( - path: string -): Promise { - return x01Time("hash", x01Label([path]), () => x01CommandExecutable(path)); -} -async function x01CommandExecutable( +async function commandExecutable( path: string ): Promise { const canonical = await realpath(path); @@ -966,14 +960,7 @@ async function commandBoot( } /** No callback can reject before the shared owner awaits exit. A rejected owner or * unproven group retains the journal; daemon effects are never inferred from exit. */ -export function runNativeComposeStorageDockerCommand( - supplied: Parameters[0] -): ReturnType { - return x01Scope(`cmd:${x01Label(supplied.args)}`, () => - x01RunCommandInner(supplied) - ); -} -async function x01RunCommandInner(supplied: { +export async function runNativeComposeStorageDockerCommand(supplied: { readonly context: Pick; readonly files: Pick< Files, @@ -1318,12 +1305,7 @@ function matchesScope( * image; other daemons and uncached images refuse rather than pull or emulate. * PostgreSQL data roots and crash durability remain unqualified. No module * initialization executes Docker or the helper. */ -export function createNativeComposeDockerStorageXattrCarrier( - opts: Context -): Promise { - return x01Scope("carrier-new", () => x01CreateCarrier(opts)); -} -async function x01CreateCarrier( +export async function createNativeComposeDockerStorageXattrCarrier( opts: Context ): Promise { const context: Context = Object.freeze({ ...opts }); @@ -2042,19 +2024,16 @@ async function x01CreateCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect: (selection) => x01Scope("inspect", () => inspect(selection)), - provision: (selection) => - x01Scope("provision", () => provision(selection)), - invoke: (input) => - x01Scope(`invoke:${input.request.operation}`, () => invoke(input)), + inspect, + provision, + invoke, finish: context.originalCommandRecords === true - ? ({ input, completion }) => - x01Scope("finish", async () => { - const finish = finalizers.get(input) ?? refuse(); - finalizers.delete(input); - await finish(completion); - }) + ? async ({ input, completion }) => { + const finish = finalizers.get(input) ?? refuse(); + finalizers.delete(input); + await finish(completion); + } : undefined, }, signal: context.signal, diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index bfcbfe4e2..269c21547 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -78,7 +78,6 @@ import { nativeComposeStorageXattrValueMatches, sameNativeComposeStorageXattrRoot, } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; export type { NativeComposeStorageWitnessReference } from "./native-compose-storage-witness-state.ts"; @@ -494,16 +493,7 @@ function requireXattrAdmission( refuse(); } } -function invokeXattr( - opts: Parameters[0] -): ReturnType { - const operation = - isRecord(opts.request) && typeof opts.request.operation === "string" - ? opts.request.operation - : "unknown"; - return x01Scope(`xattr:${operation}`, () => x01InvokeXattr(opts)); -} -async function x01InvokeXattr(opts: { +async function invokeXattr(opts: { readonly carrier: NativeComposeStorageXattrCarrier; readonly saved: XattrExpectation; readonly volume: NativeComposeRetainedVolume; @@ -1465,21 +1455,17 @@ export async function prepareNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - return await x01Scope("witness-prepare", () => - prepareNativeComposeStorageWitness({ - ...input, - xattrCarrier: carrier, - }) - ); + return await prepareNativeComposeStorageWitness({ + ...input, + xattrCarrier: carrier, + }); } export async function enrollNativeComposeStorageXattrWitness(opts: { readonly enrollment: NativeComposeStorageWitnessEnrollment; }): Promise { - return await x01Scope("witness-enroll", () => - enrollNativeComposeStorageWitness({ - enrollment: opts.enrollment, - }) - ); + return await enrollNativeComposeStorageWitness({ + enrollment: opts.enrollment, + }); } export async function verifyNativeComposeStorageXattrWitness( opts: Omit< @@ -1490,9 +1476,7 @@ export async function verifyNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - await x01Scope("witness-verify", () => - verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }) - ); + await verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }); } /** Private saved observation seam. It never performs a kernel read, repairs an diff --git a/src/lib/shell.ts b/src/lib/shell.ts index ff27e6b85..ef9faf1ba 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -4,7 +4,6 @@ import { } from "./native-cpu-diagnostics.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; -import { x01Label, x01Time } from "./x01-cost-probe.ts"; export interface ExecResult { readonly exitCode: number; @@ -41,13 +40,7 @@ function buildSpawnEnv( return override ? { ...base, ...override } : base; } -export function exec( - cmd: readonly string[], - opts: ExecOptions = {} -): Promise { - return x01Time("exec", x01Label(cmd), () => x01ExecInner(cmd, opts)); -} -async function x01ExecInner( +export async function exec( cmd: readonly string[], opts: ExecOptions = {} ): Promise { @@ -130,13 +123,7 @@ export type RunExitEvent = { readonly maxRssBytes: number | null; }; -export function run( - cmd: readonly string[], - opts: RunOptions = {} -): Promise { - return x01Time("run", x01Label(cmd), () => x01RunInner(cmd, opts)); -} -async function x01RunInner( +export async function run( cmd: readonly string[], opts: RunOptions = {} ): Promise { diff --git a/src/lib/x01-cost-probe.ts b/src/lib/x01-cost-probe.ts deleted file mode 100644 index 48848c688..000000000 --- a/src/lib/x01-cost-probe.ts +++ /dev/null @@ -1,171 +0,0 @@ -// TEMPORARY X01 measurement probe. Reverted before the final commit. -// Writes closed labels and durations only (no ids, paths or values). -import { AsyncLocalStorage } from "node:async_hooks"; -import { appendFileSync } from "node:fs"; - -const FILE = process.env.HACK_X01_TRACE_FILE; -const VERBS = new Set([ - "up", - "down", - "exec", - "run", - "restart", - "ps", - "logs", - "status", - "init", -]); -const SUBS = new Set([ - "info", - "image", - "container", - "volume", - "network", - "compose", - "create", - "start", - "rm", - "inspect", - "ls", - "ps", - "exec", - "run", - "logs", - "version", - "events", - "stop", - "kill", - "wait", - "up", - "down", - "config", - "pull", - "build", - "restart", - "port", - "cp", - "top", -]); -const verb = (() => { - for (const arg of process.argv.slice(2)) { - if (VERBS.has(arg)) { - return arg; - } - } - return "other"; -})(); -const scopes = new AsyncLocalStorage(); -let enabled = typeof FILE === "string" && FILE.startsWith("/"); -let wroteStart = false; - -function append(record: Record): void { - appendFileSync( - FILE as string, - `${JSON.stringify({ p: process.pid, v: verb, ...record })}\n` - ); -} - -function write(record: Record): void { - if (!enabled) { - return; - } - try { - if (!wroteStart) { - wroteStart = true; - append({ k: "start", t: Math.round(performance.now()) }); - process.on("exit", (code) => { - try { - append({ k: "exit", c: code, t: Math.round(performance.now()) }); - } catch { - // measurement only - } - }); - } - append({ - s: scopes.getStore() ?? "", - t: Math.round(performance.now()), - ...record, - }); - } catch { - enabled = false; - } -} - -export function x01Enabled(): boolean { - return enabled; -} - -/** Raw line from the phase trace, re-wrapped with process identity. */ -export function x01Phase(line: string): void { - try { - write({ k: "phase", line: JSON.parse(line) as unknown }); - } catch { - enabled = false; - } -} - -export function x01Label(argv: readonly string[], binary = true): string { - const parts: string[] = []; - let rest = argv; - if (binary) { - const name = (argv[0] ?? "").split("/").at(-1) ?? ""; - parts.push( - ["docker", "ps", "sysctl", "sh", "git", "bun", "tmux"].includes(name) - ? name - : "bin" - ); - rest = argv.slice(1); - } else { - parts.push("docker"); - } - for (const token of rest) { - if (parts.length >= 3) { - break; - } - if (token.startsWith("-")) { - continue; - } - if (SUBS.has(token)) { - parts.push(token); - } else { - break; - } - } - return parts.join(" "); -} - -export async function x01Time( - kind: string, - label: string, - operation: () => Promise -): Promise { - if (!enabled) { - return await operation(); - } - const started = performance.now(); - let ok = false; - try { - const result = await operation(); - ok = true; - return result; - } finally { - write({ - k: kind, - l: label, - ms: Math.round((performance.now() - started) * 10) / 10, - ok, - }); - } -} - -export async function x01Scope( - label: string, - operation: () => Promise -): Promise { - if (!enabled) { - return await operation(); - } - const parent = scopes.getStore(); - const path = parent ? `${parent}>${label}` : label; - return await scopes.run(path, () => x01Time("scope", label, operation)); -} diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index e7599c528..2201457b1 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,8 +43,6 @@ const PASSTHROUGH_ENV_KEYS = [ "LC_ALL", "DOCKER_HOST", "DOCKER_CONFIG", - // TEMPORARY X01 measurement passthrough; reverted before the final commit. - "HACK_X01_TRACE_FILE", ] as const; export type CliResult = { @@ -770,13 +768,6 @@ export async function runScenarios( }; process.stdout.write(`-- ${scenario.name} (${scenario.tier}) --\n`); - // TEMPORARY X01 measurement: one trace file per scenario. - if (process.env.HACK_X01_TRACE_DIR?.startsWith("/")) { - process.env.HACK_X01_TRACE_FILE = join( - process.env.HACK_X01_TRACE_DIR, - `${scenario.name}.jsonl` - ); - } let outcome: ScenarioOutcome; let preserveFailedFixture = false; try { diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index a21575961..2728a6059 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -1153,10 +1153,6 @@ export const nativeConfigNetworksScenario: Scenario = { NO_COLOR: "1", HACK_RUNTIME_BACKEND: "compose", HACK_DAEMON_DISABLE_DOCKER_EVENTS: "1", - // TEMPORARY X01 measurement passthrough; reverted before the final commit. - ...(process.env.HACK_X01_TRACE_FILE - ? { HACK_X01_TRACE_FILE: process.env.HACK_X01_TRACE_FILE } - : {}), }; const engine = Bun.which("docker"); const git = Bun.which("git"); From 7b2ccfc8db529200f09f0455dd90cc5d9df9d5e5 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 00:21:00 -0400 Subject: [PATCH 15/32] test(e2e): restore the networks scenario budget after witness coalescing The networks scenario took 3409 s on next with three proofs per storage exec and ran 2275 s on the hosted engine with the coalesced proofs, so the 80 min budget raise from the witness recovery change returns to 60 min. The budget bounds a hung engine; no assertion or command deadline changes. Co-Authored-By: Claude Opus 5.5 --- tests/e2e/native-config-networks-acceptance.ts | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index 2728a6059..633cb0449 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -42,8 +42,9 @@ const FIXTURE = "hack.e2e.native-config-networks"; const OUTPUT_LIMIT = 2 * 1024 * 1024; const COMMAND_TIMEOUT = 180_000; // Witness-bearing exec/up verification runs three fresh helper carriers per proof; on the -// hosted Linux engine that is ~12-15 s per exec and ~35 s per up across three checkouts. -const SCENARIO_TIMEOUT = 80 * 60_000; +// hosted Linux engine a storage exec (two proofs) takes ~10 s and a warm up (six or seven) +// ~30 s across three checkouts, about 38 minutes for this scenario. +const SCENARIO_TIMEOUT = 60 * 60_000; const CLEANUP_TIMEOUT = 3 * 60_000; const BUN_TAG = "oven/bun:1.4.2-slim"; const CADDY_TAG = "lucaslorentz/caddy-docker-proxy:2.10.0-alpine"; From 37f093a5118e7ba19eb5c5c7d6b96e7a5249f76e Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 02:10:55 -0400 Subject: [PATCH 16/32] perf(storage): make witness engine and executable fences cheaper Storage witness proofs keep every fence; three exact costs are cut: - Engine-ID fences: for an explicit DOCKER_HOST=unix:///... selection, the first fence of a command under one exact binding (selection, docker executable, client config and socket identities) still uses `docker info` and must equal one direct GET /info on that socket; only then do later fences under that unchanged binding each make one fresh direct request instead of a CLI process that lists every client plugin. A refused admission keeps the CLI, a moved binding re-admits through the CLI, and an admitted direct failure refuses without fallback. Two exact back-to-back ID reads with nothing between them are collapsed (holder observation into carrier inspect; provision fence into the target observation). - Recorded command executables: the content digest is cached for one command by canonical path and complete stat identity (dev, ino, size, mtime and ctime in ns, mode, uid); every identity check still runs, any change re-reads, and racily fresh files are never cached. - Target observations: the witness's closing re-observation of a deferred-finish verify invocation is answered by the carrier's own closing observation of the same query when the rechecked material binding is unchanged and only reads ran between (one observation per proof). Co-Authored-By: Claude Opus 5.5 --- src/lib/native-compose-engine-identity.ts | 146 ++++++++++++---- ...tive-compose-storage-command-executable.ts | 133 +++++++++++++++ ...compose-storage-witness-docker-identity.ts | 51 ++++++ .../native-compose-storage-witness-docker.ts | 161 ++++++++++-------- ...e-compose-storage-witness-xattr-carrier.ts | 16 +- src/lib/native-compose-storage-witness.ts | 27 ++- tests/helpers/native-compose-command.ts | 5 +- 7 files changed, 414 insertions(+), 125 deletions(-) create mode 100644 src/lib/native-compose-storage-command-executable.ts create mode 100644 src/lib/native-compose-storage-witness-docker-identity.ts diff --git a/src/lib/native-compose-engine-identity.ts b/src/lib/native-compose-engine-identity.ts index 6682491f4..63618b2cc 100644 --- a/src/lib/native-compose-engine-identity.ts +++ b/src/lib/native-compose-engine-identity.ts @@ -273,20 +273,16 @@ function readId(opts: { }); } -/** - * Replace only `docker info` transport for an explicit local Unix selection. - * Every call makes a fresh bounded daemon-ID request; no result is cached. Other - * selections return null before admission and retain their existing CLI path. - * Admitted errors never fall back. Socket, executable and selection bindings are - * checked before and after each request under the caller's existing time budget. - */ -export function createNativeComposeEngineIdentityObserver( - opts: { - readonly signal?: AbortSignal; - readonly timeoutMs?: number; - readonly environment?: Environment; - } = {} -): (() => Promise) | null { +type BoundObserver = { + /** Opaque digest of the exact selection, executable, config and socket binding. */ + readonly key: string; + readonly read: () => Promise; +}; +function bindObserver(opts: { + readonly signal?: AbortSignal; + readonly timeoutMs?: number; + readonly environment?: Environment; +}): BoundObserver | null { const environment = opts.environment ?? process.env; const signal = opts.signal; const selected = selection(environment); @@ -328,29 +324,107 @@ export function createNativeComposeEngineIdentityObserver( check(signal, deadline); }; recheck(); - return async () => { - if (reading) { - return refuse(); - } - reading = true; - try { - recheck(); - const result = await readId({ - socketPath: socket.canonical, - signal, - deadline, - limit: remaining, - }); - remaining -= result.bytes; - recheck(); - return result.id; - } catch { - return refuse(); - } finally { - reading = false; - } - }; + const key = new Bun.CryptoHasher("sha256") + .update(JSON.stringify([selected, binary, config, socket, executable])) + .digest("hex"); + return Object.freeze({ + key, + read: async () => { + if (reading) { + return refuse(); + } + reading = true; + try { + recheck(); + const result = await readId({ + socketPath: socket.canonical, + signal, + deadline, + limit: remaining, + }); + remaining -= result.bytes; + recheck(); + return result.id; + } catch { + return refuse(); + } finally { + reading = false; + } + }, + }); } catch { return refuse(); } } + +/** + * Replace only `docker info` transport for an explicit local Unix selection. + * Every call makes a fresh bounded daemon-ID request; no result is cached. Other + * selections return null before admission and retain their existing CLI path. + * Admitted errors never fall back. Socket, executable and selection bindings are + * checked before and after each request under the caller's existing time budget. + */ +export function createNativeComposeEngineIdentityObserver( + opts: { + readonly signal?: AbortSignal; + readonly timeoutMs?: number; + readonly environment?: Environment; + } = {} +): (() => Promise) | null { + return bindObserver(opts)?.read ?? null; +} + +/** + * Command-scoped transport for repeated `docker info --format '{{json .ID}}'` + * fences. Each fence still makes exactly one fresh engine-ID read; no ID is cached. + * + * Admission is per exact binding: the selection values, `docker` executable path + * and identity, client-config path/content and socket path identities. The first + * fence under an eligible binding returns the caller's own Docker CLI read and + * compares it with one direct `GET /info` on the same explicit Unix socket. Only + * an equal ID admits that binding; a direct failure or a different ID keeps the + * CLI for that binding for the rest of the command. Later fences under an admitted + * binding use one fresh direct request each, and its failures refuse without CLI + * fallback. A changed binding is a new binding and is admitted through the CLI + * again. Ineligible selections, and bindings that cannot be established, always + * use the CLI. The returned text has the CLI's `{{json .ID}}` line shape. + */ +export function createNativeComposeEngineIdentityAdmissions(): (opts: { + readonly cli: () => Promise; + readonly signal?: AbortSignal; + readonly timeoutMs: number; + readonly environment: Environment; +}) => Promise { + const admitted = new Map(); + return async (opts) => { + let bound: BoundObserver | null; + try { + bound = bindObserver({ + signal: opts.signal, + timeoutMs: opts.timeoutMs, + environment: opts.environment, + }); + } catch { + bound = null; + } + if (bound === null) { + return await opts.cli(); + } + const state = admitted.get(bound.key); + if (state === true) { + return `${JSON.stringify(await bound.read())}\n`; + } + const text = await opts.cli(); + if (state === undefined) { + let equal = false; + try { + const value: unknown = JSON.parse(text); + equal = typeof value === "string" && value === (await bound.read()); + } catch { + equal = false; + } + admitted.set(bound.key, equal); + } + return text; + }; +} diff --git a/src/lib/native-compose-storage-command-executable.ts b/src/lib/native-compose-storage-command-executable.ts new file mode 100644 index 000000000..a8817d685 --- /dev/null +++ b/src/lib/native-compose-storage-command-executable.ts @@ -0,0 +1,133 @@ +import { constants } from "node:fs"; +import { type FileHandle, lstat, open, realpath } from "node:fs/promises"; +import type { NativeComposeStorageCommandExecutable } from "./native-compose-storage-command-record.ts"; +import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; + +/** Coarser than any supported filesystem timestamp granularity (1 s on HFS+/ext3). */ +const RACY_MS = 2000; +const LIMIT = 268_435_456n; + +async function digestContent(file: FileHandle, size: number): Promise { + const hash = new Bun.CryptoHasher("sha256"); + const bytes = Buffer.alloc(65_536); + let position = 0; + while (position < size) { + const read = await file.read( + bytes, + 0, + Math.min(bytes.length, size - position), + position + ); + if (read.bytesRead <= 0) { + return refuse(); + } + hash.update(bytes.subarray(0, read.bytesRead)); + position += read.bytesRead; + } + return hash.digest("hex"); +} + +/** + * Identity and SHA-256 of a recorded command executable (the Docker CLI and + * `/bin/sh`), with every open, type, link-count, size, write-mode, before/after + * stat, named-path and canonical-path check run on each call. + * + * Only the content read is cached, for the life of one CLI process (one native + * command), keyed by the canonical path and the complete stat identity: device, + * inode, size, mtime and ctime in nanoseconds, mode and owner. Any change to any + * of them re-reads the content. Like Git's racily clean index entries, a file + * whose mtime or ctime is within two seconds of the read is never cached, so a + * write inside the same timestamp tick as the hashed bytes cannot be hidden. + * `now` and `digest` exist for focused tests only. + */ +export function createNativeComposeStorageCommandExecutableReader( + opts: { + readonly now?: () => number; + readonly digest?: (file: FileHandle, size: number) => Promise; + } = {} +): (path: string) => Promise { + const now = opts.now ?? Date.now; + const digest = opts.digest ?? digestContent; + const settled = new Map< + string, + { readonly identity: string; readonly hash: string } + >(); + return async (path) => { + const canonical = await realpath(path); + const file = await open( + canonical, + constants.O_RDONLY | constants.O_NOFOLLOW + ); + try { + const before = await file.stat({ bigint: true }); + if ( + !before.isFile() || + before.nlink !== 1n || + before.size <= 0n || + before.size > LIMIT || + (before.mode & 0o022n) !== 0n + ) { + return refuse(); + } + const identity = JSON.stringify( + [ + before.dev, + before.ino, + before.size, + before.mtimeNs, + before.ctimeNs, + before.mode, + before.uid, + ].map(String) + ); + const size = Number(before.size); + const started = now(); + const cached = settled.get(canonical); + const hash = + cached?.identity === identity ? cached.hash : await digest(file, size); + const after = await file.stat({ bigint: true }), + named = await lstat(canonical, { bigint: true }); + if ( + !( + before.dev === after.dev && + before.ino === after.ino && + before.dev === named.dev && + before.ino === named.ino + ) || + before.size !== after.size || + before.mtimeNs !== after.mtimeNs || + before.ctimeNs !== after.ctimeNs || + before.mode !== after.mode || + before.uid !== after.uid || + (await realpath(path)) !== canonical + ) { + settled.delete(canonical); + return refuse(); + } + const newest = Math.max( + Number(before.mtimeNs / 1_000_000n), + Number(before.ctimeNs / 1_000_000n) + ); + if (newest < started - RACY_MS) { + settled.set(canonical, { identity, hash }); + } else { + settled.delete(canonical); + } + return { + path: canonical, + dev: before.dev.toString(), + ino: before.ino.toString(), + uid: Number(before.uid), + mode: Number(before.mode), + size, + hash, + }; + } finally { + await file.close(); + } + }; +} + +/** The command's single reader; a CLI process runs exactly one native command. */ +export const readNativeComposeStorageCommandExecutable = + createNativeComposeStorageCommandExecutableReader(); diff --git a/src/lib/native-compose-storage-witness-docker-identity.ts b/src/lib/native-compose-storage-witness-docker-identity.ts new file mode 100644 index 000000000..e718b33c7 --- /dev/null +++ b/src/lib/native-compose-storage-witness-docker-identity.ts @@ -0,0 +1,51 @@ +import { createNativeComposeEngineIdentityAdmissions } from "./native-compose-engine-identity.ts"; +import type { createNativeComposeProbe } from "./native-compose-ownership.ts"; + +type Probe = ReturnType; +const IDENTITY = ["info", "--format", "{{json .ID}}"] as const; + +/** One admission table per CLI process; a process runs exactly one native command. */ +const commandAdmissions = createNativeComposeEngineIdentityAdmissions(); + +function identityQuery(args: readonly string[]): boolean { + return ( + args.length === IDENTITY.length && + args.every((value, index) => value === IDENTITY[index]) + ); +} + +/** + * Storage-witness probe whose exact engine-ID query keeps one fresh read per + * fence but may use the command's admitted direct Unix transport + * (`createNativeComposeEngineIdentityAdmissions`). Every other query, every + * ineligible selection and every unadmitted binding goes to the supplied CLI + * probe unchanged. The direct read shares this probe's signal and deadline, and + * the same environment snapshot the CLI probe captured. Recovery observation + * keeps its own CLI probes. + */ +export function nativeComposeStorageIdentityProbe(opts: { + readonly probe: Probe; + readonly signal?: AbortSignal; + readonly timeoutMs: number; + readonly admissions?: ReturnType< + typeof createNativeComposeEngineIdentityAdmissions + >; +}): Probe { + const { probe, signal } = opts; + const admissions = opts.admissions ?? commandAdmissions; + const deadline = Date.now() + opts.timeoutMs; + const environment = Object.freeze({ ...process.env }); + return (args) => { + const remaining = Math.min(60_000, deadline - Date.now()); + if (!identityQuery(args) || remaining < 1) { + // An expired probe keeps the CLI probe's own timeout refusal. + return probe(args); + } + return admissions({ + cli: () => probe(args), + signal, + timeoutMs: remaining, + environment, + }); + }; +} diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index 0c04e7ba5..c8e062188 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -40,6 +40,7 @@ import { type NativeComposeStorageCarrierCompletion, type NativeComposeStorageReadonlyCarrierIntent, } from "./native-compose-storage-carrier-journal.ts"; +import { readNativeComposeStorageCommandExecutable } from "./native-compose-storage-command-executable.ts"; import { type NativeComposeStorageCommandObservation, readNativeComposeStorageCommandObservation, @@ -67,6 +68,7 @@ import { nativeComposeStorageDockerImageReference, selectNativeComposeStorageDockerDependency, } from "./native-compose-storage-witness-docker-artifact.ts"; +import { nativeComposeStorageIdentityProbe } from "./native-compose-storage-witness-docker-identity.ts"; import { assertNativeComposeStorageDockerCarrierVolume, inspectNativeComposeStorageDockerVolume, @@ -813,68 +815,9 @@ async function groupAbsent( await new Promise((resolve) => setTimeout(resolve, 20)); } } -async function commandExecutable( - path: string -): Promise { - const canonical = await realpath(path); - const file = await open(canonical, constants.O_RDONLY | constants.O_NOFOLLOW); - try { - const before = await file.stat({ bigint: true }); - if ( - !before.isFile() || - before.nlink !== 1n || - before.size <= 0n || - before.size > 268_435_456n || - (before.mode & 0o022n) !== 0n - ) { - return refuse(); - } - const hash = new Bun.CryptoHasher("sha256"); - const bytes = Buffer.alloc(65_536); - let position = 0; - const size = Number(before.size); - while (position < size) { - const read = await file.read( - bytes, - 0, - Math.min(bytes.length, size - position), - position - ); - if (read.bytesRead <= 0) { - return refuse(); - } - hash.update(bytes.subarray(0, read.bytesRead)); - position += read.bytesRead; - } - const after = await file.stat({ bigint: true }), - named = await lstat(canonical, { bigint: true }); - if ( - !( - before.dev === after.dev && - before.ino === after.ino && - before.dev === named.dev && - before.ino === named.ino - ) || - before.size !== after.size || - before.mtimeNs !== after.mtimeNs || - before.ctimeNs !== after.ctimeNs || - (await realpath(path)) !== canonical - ) { - return refuse(); - } - return { - path: canonical, - dev: before.dev.toString(), - ino: before.ino.toString(), - uid: Number(before.uid), - mode: Number(before.mode), - size, - hash: hash.digest("hex"), - }; - } finally { - await file.close(); - } -} +/** Every identity check runs per call; only an unchanged stat identity skips the + * content read for the rest of this command. */ +const commandExecutable = readNativeComposeStorageCommandExecutable; async function boundedHostFile(path: string): Promise { const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); try { @@ -1330,11 +1273,14 @@ export async function createNativeComposeDockerStorageXattrCarrier( if (!cleanup) { lifetime(context); } - return createNativeComposeProbe({ - signal: cleanup ? undefined : context.signal, - timeoutMs: cleanup - ? 10_000 - : Math.min(15_000, context.deadline - Date.now()), + const signal = cleanup ? undefined : context.signal; + const timeoutMs = cleanup + ? 10_000 + : Math.min(15_000, context.deadline - Date.now()); + return nativeComposeStorageIdentityProbe({ + probe: createNativeComposeProbe({ signal, timeoutMs }), + signal, + timeoutMs, }); }; let dependency: ReturnType; @@ -1368,7 +1314,7 @@ export async function createNativeComposeDockerStorageXattrCarrier( throw normalized; } const selectedArtifact = dependency.artifact; - const inspect = async (selection: Selection) => { + const observe = async (selection: Selection) => { const stage: NativeComposeEffectRefusal["stage"] = "storage-target"; try { const before = await current(context, selection); @@ -1384,7 +1330,7 @@ export async function createNativeComposeDockerStorageXattrCarrier( return refuse(); } lifetime(context); - return target; + return { binding: after, target }; } catch (error) { retainNativeComposeEffectRefusal(error, { stage, @@ -1393,6 +1339,49 @@ export async function createNativeComposeDockerStorageXattrCarrier( throw error; } }; + const inspect = async (selection: Selection) => + (await observe(selection)).target; + // The closing target observation of the latest deferred-finish verify + // invocation, after which that invocation performed no effect. Every port + // entry consumes it, so it can answer only the next port call. + let closing: { + readonly invocationId: string; + readonly name: string; + readonly storage: string; + readonly binding: NativeComposeMaterialBinding; + readonly target: unknown; + } | null = null; + /** An exact repeat (same query, same rechecked material binding, no effect + * since: the caller attests reads only) answers with that closing observation. */ + const inspectPort = async ( + selection: Selection, + options?: { readonly afterInvocation?: string } + ) => { + const closed = closing; + closing = null; + if ( + closed && + options?.afterInvocation === closed.invocationId && + selection.name === closed.name && + selection.storage === closed.storage + ) { + try { + tool(); + const binding = await current(context, selection); + if (same(binding, closed.binding)) { + lifetime(context); + return closed.target; + } + } catch (error) { + retainNativeComposeEffectRefusal(error, { + stage: "storage-target", + reason: nativeComposeEffectReason(error), + }); + throw error; + } + } + return await inspect(selection); + }; const provision = async ( selection: Selection & { readonly engineId: string; @@ -1400,6 +1389,7 @@ export async function createNativeComposeDockerStorageXattrCarrier( readonly ownerToken: string; } ) => { + closing = null; const stage: NativeComposeEffectRefusal["stage"] = "storage-provision"; try { const before = await current(context, selection, true); @@ -1465,8 +1455,9 @@ export async function createNativeComposeDockerStorageXattrCarrier( requestText: "", captures: [], }; + // The target observation opens with this same engine-ID fence; nothing + // runs between, so a separate identical read here would be a duplicate. const read = probe(); - await engine(read, context); const finalAbsent = await observeNativeComposeStorageDockerTarget({ probe: read, current: before, @@ -1556,6 +1547,7 @@ export async function createNativeComposeDockerStorageXattrCarrier( (completion: NativeComposeStorageCarrierCompletion) => Promise >(); const invoke = async (input: NativeComposeStorageXattrInvocation) => { + closing = null; let stage: NativeComposeEffectRefusal["stage"] = "storage-target"; try { const before = await current(context, input.target, !input.readonly); @@ -1668,12 +1660,17 @@ export async function createNativeComposeDockerStorageXattrCarrier( if (!(created.exitCode === 0 && ID.test(id))) { return refuse(); } + /** `engineFenced` only when the caller's immediately preceding step, with no + * await between, ended with the identical engine-ID fence. */ const inspectCarrier = async ( - cleanup = false + cleanup = false, + engineFenced = false ): Promise => { try { const observed = probe(cleanup); - await engine(observed, context); + if (!engineFenced) { + await engine(observed, context); + } const value: unknown = JSON.parse( await observed([ "container", @@ -1735,7 +1732,8 @@ export async function createNativeComposeDockerStorageXattrCarrier( if (!same(holders, input.target)) { return refuse(); } - const ready = await inspectCarrier(); + // The holder observation just closed with the same engine-ID fence. + const ready = await inspectCarrier(false, true); if ( !( ready.createdAt === known.createdAt && @@ -1931,7 +1929,8 @@ export async function createNativeComposeDockerStorageXattrCarrier( if (selected !== "") { return refuse(); } - const after = await inspect(input.target); + const closed = await observe(input.target); + const after = closed.target; if (!same(after, input.target)) { return refuse(); } @@ -1978,6 +1977,17 @@ export async function createNativeComposeDockerStorageXattrCarrier( await retireInvocationFiles(files); } completed = true; + if (deferred) { + // No effect follows this closing observation before return: retirement + // waits for the journal completion and finish(). + closing = { + invocationId: input.invocationId, + name: input.target.name, + storage: input.target.storage, + binding: closed.binding, + target: after, + }; + } return { artifact: selectedArtifact, carrierId: id, @@ -2024,12 +2034,13 @@ export async function createNativeComposeDockerStorageXattrCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect, + inspect: inspectPort, provision, invoke, finish: context.originalCommandRecords === true ? async ({ input, completion }) => { + closing = null; const finish = finalizers.get(input) ?? refuse(); finalizers.delete(input); await finish(completion); diff --git a/src/lib/native-compose-storage-witness-xattr-carrier.ts b/src/lib/native-compose-storage-witness-xattr-carrier.ts index b3fe8936d..b229444a2 100644 --- a/src/lib/native-compose-storage-witness-xattr-carrier.ts +++ b/src/lib/native-compose-storage-witness-xattr-carrier.ts @@ -112,10 +112,18 @@ export type NativeComposeStorageXattrInvocation = { }; /** Trusted engine transport. Shape-valid ports alone do not qualify artifacts or runtime behavior. */ export type NativeComposeStorageXattrPorts = { - readonly inspect: (selection: { - readonly name: string; - readonly storage: string; - }) => Promise; + /** `afterInvocation` attests that the caller performed only reads since that + * invocation of this carrier returned. A carrier whose invocation performed no + * effect after its own closing observation of the same selection may answer + * with that observation when the material binding is unchanged; any other + * call observes again. */ + readonly inspect: ( + selection: { + readonly name: string; + readonly storage: string; + }, + options?: { readonly afterInvocation?: string } + ) => Promise; /** Exactly one original cold attempt. Idempotent create alone cannot establish admission. */ readonly provision?: (selection: { readonly name: string; diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index 269c21547..7586eb73b 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -450,18 +450,23 @@ async function inspectXattr(opts: { readonly saved: XattrExpectation; readonly stopped: boolean; readonly check: XattrCheck; + /** Only when nothing but reads ran since this carrier invocation returned. */ + readonly afterInvocation?: string; }): Promise { checkXattrArtifact(opts.saved, opts.carrier); const before = await opts.check(); checkNativeComposeStorageXattrCarrierLifetime(opts.carrier); - const value = await nativeComposeStorageXattrCarrierPorts( - opts.carrier - ).inspect( - Object.freeze({ - name: opts.saved.binding.name, - storage: opts.saved.binding.storage, - }) - ); + const selection = Object.freeze({ + name: opts.saved.binding.name, + storage: opts.saved.binding.storage, + }); + const ports = nativeComposeStorageXattrCarrierPorts(opts.carrier); + const value = await (opts.afterInvocation === undefined + ? ports.inspect(selection) + : ports.inspect( + selection, + Object.freeze({ afterInvocation: opts.afterInvocation }) + )); const current = await opts.check(); checkNativeComposeStorageXattrCarrierLifetime(opts.carrier); if (JSON.stringify(before) !== JSON.stringify(current)) { @@ -555,7 +560,11 @@ async function invokeXattr(opts: { carrier: opts.carrier, current: latest, }); - const after = await inspectXattr(opts); + // Only reads ran since invoke returned (binding check and result validation). + const after = await inspectXattr({ + ...opts, + afterInvocation: input.invocationId, + }); if (JSON.stringify(after) !== JSON.stringify(target)) { return refuse(); } diff --git a/tests/helpers/native-compose-command.ts b/tests/helpers/native-compose-command.ts index 460e97148..33dc84cc7 100644 --- a/tests/helpers/native-compose-command.ts +++ b/tests/helpers/native-compose-command.ts @@ -226,7 +226,9 @@ export async function invoke( root: string, args = ["up", "--detach", "--json"], startupTimeoutMs = 1000, - commandTimeoutMs = 15_000 + commandTimeoutMs = 15_000, + /** Docker client selection only (for example an explicit Unix DOCKER_HOST). */ + extraEnv: Readonly> = {} ) { const child = Bun.spawn( [ @@ -250,6 +252,7 @@ export async function invoke( HACK_COMPOSE_STARTUP_TIMEOUT_MS: String(startupTimeoutMs), CI: "1", HACK_EXECUTION_MODE: "non_interactive", + ...extraEnv, }, stdin: "ignore", stdout: "pipe", From 6a4e118e853c3fb341af2ff7c24e41630b96135f Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 02:10:28 -0400 Subject: [PATCH 17/32] chore(x03): temporary witness probe trace for hosted measurement Temporary measurement only; reverted before the final commit. Re-applies the X01 closed-label trace (reverting eb9a6785) and adds point events for direct engine-ID reads and admissions, cached executable digests, target observations and reused closing observations, plus a shard 1 identity fence benchmark. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 23 ++ scripts/x01-trace-summary.ts | 268 ++++++++++++++++++ scripts/x03-identity-bench.sh | 29 ++ src/lib/native-compose-command-storage.ts | 14 +- src/lib/native-compose-command.ts | 9 +- src/lib/native-compose-engine-identity.ts | 9 +- src/lib/native-compose-ownership.ts | 5 +- src/lib/native-compose-phase-trace.ts | 8 +- ...tive-compose-storage-command-executable.ts | 4 + ...ompose-storage-witness-docker-inventory.ts | 8 +- .../native-compose-storage-witness-docker.ts | 45 ++- src/lib/native-compose-storage-witness.ts | 34 ++- src/lib/shell.ts | 17 +- src/lib/x01-cost-probe.ts | 176 ++++++++++++ tests/e2e/harness.ts | 9 + .../e2e/native-config-networks-acceptance.ts | 4 + 16 files changed, 630 insertions(+), 32 deletions(-) create mode 100644 scripts/x01-trace-summary.ts create mode 100644 scripts/x03-identity-bench.sh create mode 100644 src/lib/x01-cost-probe.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 457db89f4..09052d11e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -300,13 +300,20 @@ jobs: - name: Verify owned bridge policy and aliases through real compiler and Compose normalization if: matrix.shard == 4 run: HACK_TEST_COMPOSE_CONFIG=1 bun test tests/native-compose-renderer-config.test.ts --test-name-pattern '^one authored bridge keeps internal policy and static aliases through compiler and Compose normalization$' + # TEMPORARY X03 measurement; reverted before the final commit. + - name: X03 engine identity fence benchmark + if: matrix.shard == 1 + run: bash scripts/x03-identity-bench.sh || true - name: Run local and Docker E2E shard env: HACK_E2E_PROCESS_POLICY_REFUSAL_CAPSULE: "1" HACK_E2E_CAPSULE_DIR: ${{ runner.temp }}/process-policy-capsules HACK_E2E_CLI_BIN: ${{ github.workspace }}/dist/hack HACK_E2E_SHARD: ${{ matrix.shard }}/4 + # TEMPORARY X01 measurement; reverted before the final commit. + HACK_X01_TRACE_DIR: ${{ runner.temp }}/x01-trace run: | + mkdir -p "$HACK_X01_TRACE_DIR" HACK_E2E_DOCKER_HOST="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" HACK_E2E_NETWORK_SOURCE_REVISION="$(git rev-parse HEAD)" HACK_E2E_NETWORK_CLI_SHA256="$(sha256sum dist/hack | cut -d ' ' -f1)" @@ -315,6 +322,22 @@ jobs: HACK_E2E_COMPOSE_PLUGIN_SHA256="$(sha256sum "$HACK_E2E_COMPOSE_PLUGIN_PATH" | cut -d ' ' -f1)" export HACK_E2E_DOCKER_HOST HACK_E2E_NETWORK_SOURCE_REVISION HACK_E2E_NETWORK_CLI_SHA256 HACK_E2E_NETWORK_COMPILER_SHA256 HACK_E2E_COMPOSE_PLUGIN_PATH HACK_E2E_COMPOSE_PLUGIN_SHA256 HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker --shard="$HACK_E2E_SHARD" + # TEMPORARY X01 measurement; reverted before the final commit. + - name: X01 witness cost summary + if: always() + run: | + for f in "$RUNNER_TEMP"/x01-trace/*.jsonl; do + [ -f "$f" ] || continue + bun scripts/x01-trace-summary.ts "$f" "$(basename "$f" .jsonl)" || true + done + - name: X01 witness cost trace artifact + if: always() + uses: actions/upload-artifact@v4 + with: + name: x01-trace-shard-${{ matrix.shard }} + path: ${{ runner.temp }}/x01-trace + if-no-files-found: ignore + retention-days: 7 - name: Preserve process-policy capsule and startup diagnostic # Closed fixed-field JSON only (counts, enums, booleans); the gh log viewer truncates it inline. if: failure() diff --git a/scripts/x01-trace-summary.ts b/scripts/x01-trace-summary.ts new file mode 100644 index 000000000..ba276c07f --- /dev/null +++ b/scripts/x01-trace-summary.ts @@ -0,0 +1,268 @@ +// TEMPORARY X01 measurement summary. Reverted before the final commit. +// Reads closed-label JSONL written by src/lib/x01-cost-probe.ts. +type Event = { + readonly p: number; + readonly v: string; + readonly k: string; + readonly s?: string; + readonly l?: string; + readonly ms?: number; + readonly t?: number; + readonly c?: number; + readonly line?: { + readonly phase?: string; + readonly boundary?: string; + readonly durationMs?: number | null; + }; +}; +const WITNESS = new Set([ + "storage-prepare", + "storage-verify", + "carrier-new", + "inspect", + "provision", + "finish", + "witness-verify", + "witness-prepare", + "witness-enroll", +]); +const isWitness = (label: string) => + WITNESS.has(label) || label.startsWith("invoke:"); +const LEAF = new Set(["probe", "run", "exec", "hash"]); +const PHASE = /^(oneoff|storage|guard|finish|finalize)\.[a-z-]+$/; + +const path = process.argv[2]; +const title = process.argv[3] ?? path; +if (!path) { + throw new Error("usage: x01-trace-summary [title]"); +} +const text = await Bun.file(path).text(); +const events: Event[] = []; +for (const line of text.split("\n")) { + if (line.trim() === "") { + continue; + } + try { + events.push(JSON.parse(line) as Event); + } catch { + // skip partial line + } +} +const byProcess = new Map(); +for (const event of events) { + const list = byProcess.get(event.p) ?? []; + list.push(event); + byProcess.set(event.p, list); +} +type Proc = { + readonly pid: number; + readonly verb: string; + readonly totalMs: number; + readonly witnessMs: number; + readonly events: Event[]; + readonly kind: string; +}; +function topWitnessMs(list: Event[]): number { + let total = 0; + for (const event of list) { + if (event.k !== "scope" || !event.l || !isWitness(event.l)) { + continue; + } + const parts = (event.s ?? "").split(">"); + parts.pop(); + if (parts.some(isWitness)) { + continue; + } + total += event.ms ?? 0; + } + return total; +} +const procs: Proc[] = []; +for (const [pid, list] of byProcess) { + const exit = list.find((event) => event.k === "exit"); + const verb = list[0]?.v ?? "other"; + const labels = new Set(list.map((event) => event.l ?? "")); + let kind = verb; + if (verb === "up" || verb === "restart" || verb === "run") { + if (labels.has("invoke:seed")) { + kind = `${verb}(cold)`; + } else if (labels.has("invoke:verify")) { + kind = `${verb}(warm)`; + } else { + kind = `${verb}(no-witness)`; + } + } else if (verb === "exec") { + kind = labels.has("invoke:verify") ? "exec(storage)" : "exec(image)"; + } + procs.push({ + pid, + verb, + totalMs: exit?.t ?? Math.max(...list.map((event) => event.t ?? 0)), + witnessMs: topWitnessMs(list), + events: list, + kind, + }); +} +const fmt = (ms: number) => (ms / 1000).toFixed(1); +const out: string[] = []; +out.push(`#### X01 trace: ${title}`); +out.push(`events=${events.length} processes=${procs.length}`); +out.push(""); +out.push( + "kind | n | avg total s | sum total s | avg witness s | sum witness s" +); +const kinds = [...new Set(procs.map((proc) => proc.kind))].sort(); +for (const kind of kinds) { + const selected = procs.filter((proc) => proc.kind === kind); + const total = selected.reduce((sum, proc) => sum + proc.totalMs, 0); + const witness = selected.reduce((sum, proc) => sum + proc.witnessMs, 0); + out.push( + `${kind} | ${selected.length} | ${fmt(total / selected.length)} | ${fmt(total)} | ${fmt(witness / selected.length)} | ${fmt(witness)}` + ); +} +const allTotal = procs.reduce((sum, proc) => sum + proc.totalMs, 0); +const allWitness = procs.reduce((sum, proc) => sum + proc.witnessMs, 0); +out.push( + `ALL | ${procs.length} | - | ${fmt(allTotal)} | - | ${fmt(allWitness)}` +); +out.push(""); + +function leafTable(selected: Proc[], heading: string, limit = 24) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (!(LEAF.has(event.k) && event.l)) { + continue; + } + const key = `${event.k}:${event.l}`; + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push(`${heading} (per process avg over n=${selected.length})`); + out.push("leaf | calls/proc | s/proc | ms/call"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function leafByScope(selected: Proc[], heading: string, limit = 40) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (!(LEAF.has(event.k) && event.l)) { + continue; + } + const parts = (event.s ?? "") + .split(">") + .filter( + (part) => + isWitness(part) || part.startsWith("cmd:") || PHASE.test(part) + ); + const inner = parts.slice(-2).join(">") || "(none)"; + const key = `${inner} :: ${event.k}:${event.l}`; + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push( + `${heading} leaves by scope (per process avg over n=${selected.length})` + ); + out.push("scope :: leaf | calls/proc | s/proc | ms/call"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function scopeTable(selected: Proc[], heading: string, limit = 30) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (event.k !== "scope" || !event.l) { + continue; + } + const parts = (event.s ?? "").split(">"); + const key = parts + .filter( + (part) => + PHASE.test(part) || + isWitness(part) || + part.startsWith("xattr:") || + part.startsWith("cmd:") || + part.startsWith("exec-") + ) + .join(">"); + if (key === "") { + continue; + } + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push(`${heading} scopes (per process avg over n=${selected.length})`); + out.push("scope path | n/proc | s/proc | ms/each"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function phaseRows(proc: Proc) { + const rows = new Map(); + for (const event of proc.events) { + if ( + event.k === "phase" && + event.line?.boundary !== "begin" && + event.line?.phase + ) { + const row = rows.get(event.line.phase) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.line.durationMs ?? 0; + rows.set(event.line.phase, row); + } + } + return rows; +} +for (const kind of [ + "up(cold)", + "up(warm)", + "exec(storage)", + "restart(warm)", + "run(warm)", + "run(cold)", + "down", +]) { + const selected = procs.filter((proc) => proc.kind === kind); + if (selected.length === 0) { + continue; + } + const sample = selected[0] as Proc; + out.push( + `### ${kind}: first sample total ${fmt(sample.totalMs)} s, witness ${fmt(sample.witnessMs)} s` + ); + out.push("phase-trace phase | spans | s"); + for (const [phase, row] of phaseRows(sample)) { + out.push(`${phase} | ${row.n} | ${fmt(row.ms)}`); + } + out.push(""); + scopeTable([sample], `${kind} first sample`, 40); + leafTable([sample], `${kind} first sample leaves`); + scopeTable(selected, `${kind} all`, 30); + leafTable(selected, `${kind} all leaves`); + leafByScope(selected, `${kind} all`); +} +console.log(out.join("\n")); diff --git a/scripts/x03-identity-bench.sh b/scripts/x03-identity-bench.sh new file mode 100644 index 000000000..050dfd5c1 --- /dev/null +++ b/scripts/x03-identity-bench.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# TEMPORARY X03 measurement; reverted before the final commit. +# Decomposes one engine-ID fence on the hosted engine. Prints timings only. +set -euo pipefail +host="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" +socket="${host#unix://}" +n=40 +bench() { + local label="$1" + shift + local start end + start=$(date +%s%N) + for _ in $(seq "$n"); do + "$@" >/dev/null + done + end=$(date +%s%N) + echo "x03-bench ${label}: $(((end - start) / n / 1000)) us/call over ${n}" +} +echo "x03-bench selection: $(case "$host" in unix://*) echo unix ;; *) echo other ;; esac), DOCKER_HOST set: ${DOCKER_HOST:+yes}" +echo "x03-bench client plugins: $(docker info --format '{{len .ClientInfo.Plugins}}')" +bench "docker info --format {{json .ID}}" docker info --format '{{json .ID}}' +bench "docker version --format server" docker version --format '{{.Server.Version}}' +bench "docker volume ls" docker volume ls --format '{{json .Name}}' +bench "docker context inspect (client only)" docker context inspect --format '{{json .Endpoints.docker.Host}}' +bench "curl GET /info (direct Unix)" curl -sf --unix-socket "$socket" http://docker/info +bench "curl GET /_ping (direct Unix)" curl -sf --unix-socket "$socket" http://docker/_ping +sha="$(command -v docker)" +bench "sha256sum docker CLI" sha256sum "$sha" +echo "x03-bench docker CLI size: $(stat -Lc %s "$sha") bytes" diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index c9384e64c..d9b58c132 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -23,6 +23,7 @@ import { } from "./native-compose-storage-witness.ts"; import { createNativeComposeDockerStorageXattrCarrier } from "./native-compose-storage-witness-docker.ts"; import type { NativeComposeStorageXattrCarrier } from "./native-compose-storage-witness-xattr-carrier.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; type Selection = { readonly name: string; readonly storage: string }; type Document = Readonly>; @@ -134,7 +135,12 @@ export type NativeComposeCommandStorageVerifyOptions = { /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ -export async function prepareNativeComposeCommandStorage(opts: { +export function prepareNativeComposeCommandStorage( + opts: Parameters[0] +): ReturnType { + return x01Scope("storage-prepare", () => x01Prepare(opts)); +} +async function x01Prepare(opts: { readonly store: NativeComposeGenerationStore; readonly mutation: NativeComposeMutation; readonly operation: "up" | "restart" | "run" | "exec"; @@ -225,7 +231,11 @@ export async function prepareNativeComposeCommandStorage(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = async ( + const verify = ( + generation: NativeComposeGeneration, + options: NativeComposeCommandStorageVerifyOptions = {} + ) => x01Scope("storage-verify", () => x01Verify(generation, options)); + const x01Verify = async ( generation: NativeComposeGeneration, options: NativeComposeCommandStorageVerifyOptions = {} ) => { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index 8c5dcaf82..0aa92b81e 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -138,6 +138,7 @@ import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; export type NativeComposeCommandOptions = { @@ -943,7 +944,9 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = async ( + const assertSaved = (proof: NativeComposeCommandStorageVerifyOptions) => + x01Scope("exec-assert-saved", () => x01AssertSaved(proof)); + const x01AssertSaved = async ( proof: NativeComposeCommandStorageVerifyOptions ) => { const latest = await store.loadCurrent(); @@ -962,7 +965,9 @@ async function savedCommand(opts: { signal, assertSaved, run: () => - runSavedProcess({ options, generation, document, base, signal }), + x01Scope("exec-process", () => + runSavedProcess({ options, generation, document, base, signal }) + ), }); }); } diff --git a/src/lib/native-compose-engine-identity.ts b/src/lib/native-compose-engine-identity.ts index 63618b2cc..205641d63 100644 --- a/src/lib/native-compose-engine-identity.ts +++ b/src/lib/native-compose-engine-identity.ts @@ -3,6 +3,7 @@ import { request } from "node:http"; import { dirname, isAbsolute, join, normalize, parse } from "node:path"; import { isRecord } from "./guards.ts"; import { bindNativeComposeEngineConfig } from "./native-compose-engine-config.ts"; +import { x01Mark, x01Time } from "./x01-cost-probe.ts"; const ENGINE_ID = /^[A-Za-z0-9:-]{1,128}$/; const AMBIGUOUS_PATH = /[%?#\0\r\n]/; @@ -411,18 +412,22 @@ export function createNativeComposeEngineIdentityAdmissions(): (opts: { return await opts.cli(); } const state = admitted.get(bound.key); + const direct = bound.read; if (state === true) { - return `${JSON.stringify(await bound.read())}\n`; + return `${JSON.stringify(await x01Time("ident", "direct", direct))}\n`; } const text = await opts.cli(); if (state === undefined) { let equal = false; try { const value: unknown = JSON.parse(text); - equal = typeof value === "string" && value === (await bound.read()); + equal = + typeof value === "string" && + value === (await x01Time("ident", "admission", direct)); } catch { equal = false; } + x01Mark("ident-admit", equal ? "admitted" : "refused"); admitted.set(bound.key, equal); } return text; diff --git a/src/lib/native-compose-ownership.ts b/src/lib/native-compose-ownership.ts index 0019b3fc3..d8bc9dc8a 100644 --- a/src/lib/native-compose-ownership.ts +++ b/src/lib/native-compose-ownership.ts @@ -3,6 +3,7 @@ import { isRecord } from "./guards.ts"; import { nativeComposeVolumeCreatedAt } from "./native-compose-retained-storage.ts"; import { beginNativeCpuChild } from "./native-cpu-diagnostics.ts"; import { findExecutableInPath } from "./shell.ts"; +import { x01Label, x01Time } from "./x01-cost-probe.ts"; const PROJECT = /^[a-z0-9][a-z0-9_-]*$/; const SERVICE = /^[a-z0-9][a-z0-9._-]{0,62}$/; @@ -1232,7 +1233,7 @@ export function createNativeComposeProbe( probeRefused("E_NATIVE_COMPOSE_PROBE", "operation"); } const environment = { ...process.env }; - return async (args) => { + const x01Inner = async (args: readonly string[]): Promise => { checkInterruption(opts.signal, false); const remainingTime = deadline - Date.now(); if (remainingTime <= 0) { @@ -1310,6 +1311,8 @@ export function createNativeComposeProbe( observeCpu(probeExitCode(completion[2])); } }; + return (args) => + x01Time("probe", x01Label(args, false), () => x01Inner(args)); } function probeExitCode( diff --git a/src/lib/native-compose-phase-trace.ts b/src/lib/native-compose-phase-trace.ts index 540f833b0..943b79320 100644 --- a/src/lib/native-compose-phase-trace.ts +++ b/src/lib/native-compose-phase-trace.ts @@ -1,5 +1,6 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { writeSync } from "node:fs"; +import { x01Enabled, x01Phase, x01Scope } from "./x01-cost-probe.ts"; export const NATIVE_COMPOSE_PHASE_TRACE = "HACK_NATIVE_COMPOSE_PHASE_TRACE"; const PHASES = [ @@ -135,6 +136,8 @@ export function withNativeComposePhaseTrace(operation: () => Promise) { writeSync(2, line); }, }); + } else if (x01Enabled()) { + trace = createNativeComposePhaseTrace({ write: x01Phase }); } } catch { // Diagnostic setup never changes selection, results or effect authority. @@ -147,5 +150,8 @@ export function measureNativeComposePhase( phase: Phase, operation: () => Promise ): Promise { - return context.getStore()?.measure(phase, operation) ?? operation(); + return x01Scope( + phase, + () => context.getStore()?.measure(phase, operation) ?? operation() + ); } diff --git a/src/lib/native-compose-storage-command-executable.ts b/src/lib/native-compose-storage-command-executable.ts index a8817d685..780dae040 100644 --- a/src/lib/native-compose-storage-command-executable.ts +++ b/src/lib/native-compose-storage-command-executable.ts @@ -2,6 +2,7 @@ import { constants } from "node:fs"; import { type FileHandle, lstat, open, realpath } from "node:fs/promises"; import type { NativeComposeStorageCommandExecutable } from "./native-compose-storage-command-record.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Label, x01Mark } from "./x01-cost-probe.ts"; /** Coarser than any supported filesystem timestamp granularity (1 s on HFS+/ext3). */ const RACY_MS = 2000; @@ -83,6 +84,9 @@ export function createNativeComposeStorageCommandExecutableReader( const size = Number(before.size); const started = now(); const cached = settled.get(canonical); + if (cached?.identity === identity) { + x01Mark("hash-cached", x01Label([canonical])); + } const hash = cached?.identity === identity ? cached.hash : await digest(file, size); const after = await file.stat({ bigint: true }), diff --git a/src/lib/native-compose-storage-witness-docker-inventory.ts b/src/lib/native-compose-storage-witness-docker-inventory.ts index b5bdb7410..f292021ce 100644 --- a/src/lib/native-compose-storage-witness-docker-inventory.ts +++ b/src/lib/native-compose-storage-witness-docker-inventory.ts @@ -8,6 +8,7 @@ import { type NativeComposeStorageXattrTarget, } from "./native-compose-storage-witness-xattr-carrier.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$/; @@ -164,7 +165,12 @@ export async function assertNativeComposeStorageDockerCarrierVolume(opts: { /** Complete holder reads include foreign containers. The two inventories and selected * volume reread are observed fences; they do not claim an atomic daemon transaction. */ -export async function observeNativeComposeStorageDockerTarget(opts: { +export function observeNativeComposeStorageDockerTarget( + opts: Parameters[0] +): ReturnType { + return x01Scope("target", () => x01ObserveTarget(opts)); +} +async function x01ObserveTarget(opts: { readonly probe: Probe; readonly selection: Selection; readonly current: NativeComposeMaterialBinding; diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index c8e062188..d955faf06 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -98,6 +98,7 @@ import { refuseNativeComposeStorageXattr as refuse, } from "./native-compose-storage-witness-xattr-codec.ts"; import { findExecutableInPath, type RunExitEvent, run } from "./shell.ts"; +import { x01Label, x01Mark, x01Scope, x01Time } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const PROCESS = @@ -817,7 +818,10 @@ async function groupAbsent( } /** Every identity check runs per call; only an unchanged stat identity skips the * content read for the rest of this command. */ -const commandExecutable = readNativeComposeStorageCommandExecutable; +const commandExecutable = (path: string) => + x01Time("hash", x01Label([path]), () => + readNativeComposeStorageCommandExecutable(path) + ); async function boundedHostFile(path: string): Promise { const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); try { @@ -903,7 +907,14 @@ async function commandBoot( } /** No callback can reject before the shared owner awaits exit. A rejected owner or * unproven group retains the journal; daemon effects are never inferred from exit. */ -export async function runNativeComposeStorageDockerCommand(supplied: { +export function runNativeComposeStorageDockerCommand( + supplied: Parameters[0] +): ReturnType { + return x01Scope(`cmd:${x01Label(supplied.args)}`, () => + x01RunCommandInner(supplied) + ); +} +async function x01RunCommandInner(supplied: { readonly context: Pick; readonly files: Pick< Files, @@ -1248,7 +1259,12 @@ function matchesScope( * image; other daemons and uncached images refuse rather than pull or emulate. * PostgreSQL data roots and crash durability remain unqualified. No module * initialization executes Docker or the helper. */ -export async function createNativeComposeDockerStorageXattrCarrier( +export function createNativeComposeDockerStorageXattrCarrier( + opts: Context +): Promise { + return x01Scope("carrier-new", () => x01CreateCarrier(opts)); +} +async function x01CreateCarrier( opts: Context ): Promise { const context: Context = Object.freeze({ ...opts }); @@ -1370,6 +1386,7 @@ export async function createNativeComposeDockerStorageXattrCarrier( const binding = await current(context, selection); if (same(binding, closed.binding)) { lifetime(context); + x01Mark("target-reused", "closing"); return closed.target; } } catch (error) { @@ -2034,17 +2051,21 @@ export async function createNativeComposeDockerStorageXattrCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect: inspectPort, - provision, - invoke, + inspect: (selection, options) => + x01Scope("inspect", () => inspectPort(selection, options)), + provision: (selection) => + x01Scope("provision", () => provision(selection)), + invoke: (input) => + x01Scope(`invoke:${input.request.operation}`, () => invoke(input)), finish: context.originalCommandRecords === true - ? async ({ input, completion }) => { - closing = null; - const finish = finalizers.get(input) ?? refuse(); - finalizers.delete(input); - await finish(completion); - } + ? ({ input, completion }) => + x01Scope("finish", async () => { + closing = null; + const finish = finalizers.get(input) ?? refuse(); + finalizers.delete(input); + await finish(completion); + }) : undefined, }, signal: context.signal, diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index 7586eb73b..1fc68abf7 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -78,6 +78,7 @@ import { nativeComposeStorageXattrValueMatches, sameNativeComposeStorageXattrRoot, } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; export type { NativeComposeStorageWitnessReference } from "./native-compose-storage-witness-state.ts"; @@ -498,7 +499,16 @@ function requireXattrAdmission( refuse(); } } -async function invokeXattr(opts: { +function invokeXattr( + opts: Parameters[0] +): ReturnType { + const operation = + isRecord(opts.request) && typeof opts.request.operation === "string" + ? opts.request.operation + : "unknown"; + return x01Scope(`xattr:${operation}`, () => x01InvokeXattr(opts)); +} +async function x01InvokeXattr(opts: { readonly carrier: NativeComposeStorageXattrCarrier; readonly saved: XattrExpectation; readonly volume: NativeComposeRetainedVolume; @@ -1464,17 +1474,21 @@ export async function prepareNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - return await prepareNativeComposeStorageWitness({ - ...input, - xattrCarrier: carrier, - }); + return await x01Scope("witness-prepare", () => + prepareNativeComposeStorageWitness({ + ...input, + xattrCarrier: carrier, + }) + ); } export async function enrollNativeComposeStorageXattrWitness(opts: { readonly enrollment: NativeComposeStorageWitnessEnrollment; }): Promise { - return await enrollNativeComposeStorageWitness({ - enrollment: opts.enrollment, - }); + return await x01Scope("witness-enroll", () => + enrollNativeComposeStorageWitness({ + enrollment: opts.enrollment, + }) + ); } export async function verifyNativeComposeStorageXattrWitness( opts: Omit< @@ -1485,7 +1499,9 @@ export async function verifyNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - await verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }); + await x01Scope("witness-verify", () => + verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }) + ); } /** Private saved observation seam. It never performs a kernel read, repairs an diff --git a/src/lib/shell.ts b/src/lib/shell.ts index ef9faf1ba..ff27e6b85 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -4,6 +4,7 @@ import { } from "./native-cpu-diagnostics.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; +import { x01Label, x01Time } from "./x01-cost-probe.ts"; export interface ExecResult { readonly exitCode: number; @@ -40,7 +41,13 @@ function buildSpawnEnv( return override ? { ...base, ...override } : base; } -export async function exec( +export function exec( + cmd: readonly string[], + opts: ExecOptions = {} +): Promise { + return x01Time("exec", x01Label(cmd), () => x01ExecInner(cmd, opts)); +} +async function x01ExecInner( cmd: readonly string[], opts: ExecOptions = {} ): Promise { @@ -123,7 +130,13 @@ export type RunExitEvent = { readonly maxRssBytes: number | null; }; -export async function run( +export function run( + cmd: readonly string[], + opts: RunOptions = {} +): Promise { + return x01Time("run", x01Label(cmd), () => x01RunInner(cmd, opts)); +} +async function x01RunInner( cmd: readonly string[], opts: RunOptions = {} ): Promise { diff --git a/src/lib/x01-cost-probe.ts b/src/lib/x01-cost-probe.ts new file mode 100644 index 000000000..17b2f5552 --- /dev/null +++ b/src/lib/x01-cost-probe.ts @@ -0,0 +1,176 @@ +// TEMPORARY X01 measurement probe. Reverted before the final commit. +// Writes closed labels and durations only (no ids, paths or values). +import { AsyncLocalStorage } from "node:async_hooks"; +import { appendFileSync } from "node:fs"; + +const FILE = process.env.HACK_X01_TRACE_FILE; +const VERBS = new Set([ + "up", + "down", + "exec", + "run", + "restart", + "ps", + "logs", + "status", + "init", +]); +const SUBS = new Set([ + "info", + "image", + "container", + "volume", + "network", + "compose", + "create", + "start", + "rm", + "inspect", + "ls", + "ps", + "exec", + "run", + "logs", + "version", + "events", + "stop", + "kill", + "wait", + "up", + "down", + "config", + "pull", + "build", + "restart", + "port", + "cp", + "top", +]); +const verb = (() => { + for (const arg of process.argv.slice(2)) { + if (VERBS.has(arg)) { + return arg; + } + } + return "other"; +})(); +const scopes = new AsyncLocalStorage(); +let enabled = typeof FILE === "string" && FILE.startsWith("/"); +let wroteStart = false; + +function append(record: Record): void { + appendFileSync( + FILE as string, + `${JSON.stringify({ p: process.pid, v: verb, ...record })}\n` + ); +} + +function write(record: Record): void { + if (!enabled) { + return; + } + try { + if (!wroteStart) { + wroteStart = true; + append({ k: "start", t: Math.round(performance.now()) }); + process.on("exit", (code) => { + try { + append({ k: "exit", c: code, t: Math.round(performance.now()) }); + } catch { + // measurement only + } + }); + } + append({ + s: scopes.getStore() ?? "", + t: Math.round(performance.now()), + ...record, + }); + } catch { + enabled = false; + } +} + +export function x01Enabled(): boolean { + return enabled; +} + +/** Raw line from the phase trace, re-wrapped with process identity. */ +export function x01Phase(line: string): void { + try { + write({ k: "phase", line: JSON.parse(line) as unknown }); + } catch { + enabled = false; + } +} + +export function x01Label(argv: readonly string[], binary = true): string { + const parts: string[] = []; + let rest = argv; + if (binary) { + const name = (argv[0] ?? "").split("/").at(-1) ?? ""; + parts.push( + ["docker", "ps", "sysctl", "sh", "git", "bun", "tmux"].includes(name) + ? name + : "bin" + ); + rest = argv.slice(1); + } else { + parts.push("docker"); + } + for (const token of rest) { + if (parts.length >= 3) { + break; + } + if (token.startsWith("-")) { + continue; + } + if (SUBS.has(token)) { + parts.push(token); + } else { + break; + } + } + return parts.join(" "); +} + +export async function x01Time( + kind: string, + label: string, + operation: () => Promise +): Promise { + if (!enabled) { + return await operation(); + } + const started = performance.now(); + let ok = false; + try { + const result = await operation(); + ok = true; + return result; + } finally { + write({ + k: kind, + l: label, + ms: Math.round((performance.now() - started) * 10) / 10, + ok, + }); + } +} + +/** TEMPORARY X03 measurement: a closed-label point event. */ +export function x01Mark(kind: string, label: string): void { + write({ k: kind, l: label }); +} + +export async function x01Scope( + label: string, + operation: () => Promise +): Promise { + if (!enabled) { + return await operation(); + } + const parent = scopes.getStore(); + const path = parent ? `${parent}>${label}` : label; + return await scopes.run(path, () => x01Time("scope", label, operation)); +} diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index 2201457b1..e7599c528 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,6 +43,8 @@ const PASSTHROUGH_ENV_KEYS = [ "LC_ALL", "DOCKER_HOST", "DOCKER_CONFIG", + // TEMPORARY X01 measurement passthrough; reverted before the final commit. + "HACK_X01_TRACE_FILE", ] as const; export type CliResult = { @@ -768,6 +770,13 @@ export async function runScenarios( }; process.stdout.write(`-- ${scenario.name} (${scenario.tier}) --\n`); + // TEMPORARY X01 measurement: one trace file per scenario. + if (process.env.HACK_X01_TRACE_DIR?.startsWith("/")) { + process.env.HACK_X01_TRACE_FILE = join( + process.env.HACK_X01_TRACE_DIR, + `${scenario.name}.jsonl` + ); + } let outcome: ScenarioOutcome; let preserveFailedFixture = false; try { diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index 633cb0449..6c5194702 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -1154,6 +1154,10 @@ export const nativeConfigNetworksScenario: Scenario = { NO_COLOR: "1", HACK_RUNTIME_BACKEND: "compose", HACK_DAEMON_DISABLE_DOCKER_EVENTS: "1", + // TEMPORARY X01 measurement passthrough; reverted before the final commit. + ...(process.env.HACK_X01_TRACE_FILE + ? { HACK_X01_TRACE_FILE: process.env.HACK_X01_TRACE_FILE } + : {}), }; const engine = Bun.which("docker"); const git = Bun.which("git"); From 2b11ed3c88ba06c0f2d291855bcba4a896fd587a Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 02:39:54 -0400 Subject: [PATCH 18/32] fix(storage): keep closed probe cancellation for direct engine-ID fences An admitted direct engine-ID read that fails because the caller cancelled or its probe deadline expired now reports the CLI probe's existing closed cancel/timeout classification; that probe refuses before spawning anything. Every other admitted direct failure still refuses without CLI fallback. Co-Authored-By: Claude Opus 5.5 --- src/lib/native-compose-storage-witness-docker-identity.ts | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/lib/native-compose-storage-witness-docker-identity.ts b/src/lib/native-compose-storage-witness-docker-identity.ts index e718b33c7..b77d53f29 100644 --- a/src/lib/native-compose-storage-witness-docker-identity.ts +++ b/src/lib/native-compose-storage-witness-docker-identity.ts @@ -46,6 +46,13 @@ export function nativeComposeStorageIdentityProbe(opts: { signal, timeoutMs: remaining, environment, + }).catch((error: unknown) => { + // Admitted failures never fall back. Cancellation and expiry keep the CLI + // probe's closed classification: that probe refuses them before spawning. + if (signal?.aborted || Date.now() >= deadline) { + return probe(args); + } + throw error; }); }; } From a777bc40ff036d93992f13463ac9a247ef5d44ee Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 02:40:01 -0400 Subject: [PATCH 19/32] test(storage): pin witness fence counts and refusal controls - Per command on the synthetic engine: 81 of 86 engine-ID fences and 20 of 21 target observations per proof, with every proof kept; an admitted explicit Unix selection uses one CLI fence per command. - A changed daemon ID between two admitted fences refuses before any helper, workload or state change; a refused admission keeps every fence on the CLI. - A target change right after the verify carrier's closing observation is refused by the next remaining observation. - Admission, re-admission, no-fallback and cancellation controls on a real Unix server; executable digest reuse, replacement, same-size tamper, mode, racy-fresh and mid-command replacement controls. - Reference docs describe the three cuts and their limits. Co-Authored-By: Claude Opus 5.5 --- docs/reference/native-compose-ownership.md | 10 + .../native-compose-storage-xattr-carrier.md | 40 ++++ .../helpers/native-compose-storage-docker.ts | 18 +- tests/native-compose-engine-identity.test.ts | 214 +++++++++++++++++ ...compose-storage-command-executable.test.ts | 118 ++++++++++ ...ive-compose-storage-command-record.test.ts | 66 +++++- .../native-compose-storage-fence-cost.test.ts | 215 ++++++++++++++++++ 7 files changed, 679 insertions(+), 2 deletions(-) create mode 100644 tests/native-compose-storage-command-executable.test.ts create mode 100644 tests/native-compose-storage-fence-cost.test.ts diff --git a/docs/reference/native-compose-ownership.md b/docs/reference/native-compose-ownership.md index 0cc7a0ba3..23aa78198 100644 --- a/docs/reference/native-compose-ownership.md +++ b/docs/reference/native-compose-ownership.md @@ -127,6 +127,16 @@ falls back to the CLI after a failure. Ambient HTTP proxy variables do not chang the Unix destination. Its deadline and cumulative 8 MiB response limit remain control-channel budgets; cancellation or failure closes only its owned connection. +Storage-witness fences in the Docker carrier repeat the engine-ID read hundreds +of times per command, so they use the same direct transport through +command-scoped admission instead of a fixed observer. Each fence still makes one +fresh read. The first fence under one exact binding (the selection values, +`docker` executable, config and socket identities above) is the existing Docker +CLI read and must equal one direct `GET /info`; only an equal ID admits that +binding for the rest of the command. A different or failed admission read keeps +the CLI for that binding, a changed binding is admitted through the CLI again, +and an admitted direct failure refuses without fallback. + Real Unix-server controls live in `tests/native-compose-engine-identity.test.ts`; existing CLI-transport and proxy ownership controls remain in `tests/native-compose-ingress.test.ts`. These tests establish the bounded transport diff --git a/docs/reference/native-compose-storage-xattr-carrier.md b/docs/reference/native-compose-storage-xattr-carrier.md index 8a44dadad..059d03e74 100644 --- a/docs/reference/native-compose-storage-xattr-carrier.md +++ b/docs/reference/native-compose-storage-xattr-carrier.md @@ -225,6 +225,46 @@ retains the private journal and input. Journal bytes do not replace these actual stop and absence observations. Ordinary command admission calls this port on the cataloged platforms; PostgreSQL data roots and crash durability remain unqualified. +### Fence cost within one command + +Every engine-ID fence, target observation and executable identity check still +runs; three exact costs are removed. Each change is local to one CLI process, +which runs exactly one native command. + +- Engine-ID fences (`docker info --format '{{json .ID}}'`) in the Docker + carrier use command-scoped admission for an explicit `DOCKER_HOST=unix:///...` + selection that the ingress transport also admits (see + [ownership](native-compose-ownership.md#ingress-daemon-identity-transport)). + The first fence under one exact binding (selection values, `docker` + executable, client config and socket identities) is the Docker CLI read and + must equal one direct `GET /info` on that socket. Only then does each later + fence under the unchanged binding make one fresh direct request instead of a + CLI process that also lists every client plugin. A different or failed + admission read keeps the CLI for that binding; a moved binding is admitted + through the CLI again; an admitted direct failure refuses without fallback. + No ID is cached. Saved recovery observation keeps its CLI probes. +- Two identical engine-ID reads with nothing awaited between them are one fence: + the holder observation's closing read also fences the carrier inspect that + immediately follows it, and the cold-provision target observation's opening + read is the provision fence. +- The recorded command executables (Docker CLI and `/bin/sh`) are opened and + checked on every call (type, link count, size, write mode, before/after stat, + named path, canonical path). Only their content digest is reused, keyed by the + canonical path and the complete stat identity (device, inode, size, mtime and + ctime in nanoseconds, mode, owner); any change re-reads the content, and a + file changed within two seconds of the read is never reused. +- After a readonly verify invocation with deferred finish, the carrier performs + no effect after its own closing target observation. The witness's immediate + re-observation of the same query, with only reads between, is answered with + that observation when the rechecked material binding is unchanged. Every other + observation, including the finisher's before file retirement and the opening + observation of the next invocation, stays fresh: each effect is still + immediately preceded and followed by a fresh target observation. + +On the synthetic engine a proof keeps 81 of 86 engine-ID fences and 20 of 21 +target observations; `tests/native-compose-storage-fence-cost.test.ts` pins the +per-command counts and the refusal controls. + Holder selection includes normalized ancestor and descendant bind paths; similarly named sibling roots stay independent. Command stdin and captures use held no-follow descriptors, checked after the final admission await and held until owned group diff --git a/tests/helpers/native-compose-storage-docker.ts b/tests/helpers/native-compose-storage-docker.ts index f103f6573..c65abf414 100644 --- a/tests/helpers/native-compose-storage-docker.ts +++ b/tests/helpers/native-compose-storage-docker.ts @@ -36,7 +36,17 @@ if(storageArgs[0]==="volume"&&storageArgs[1]==="create"){ console.log(name);process.exit(0); } if(storageArgs[0]==="volume"&&storageArgs[1]==="ls"&&argument("--format")==="{{json .Name}}"){ - for(const row of await readRows(storageVolumes))emit(row.name);process.exit(0); + for(const row of await readRows(storageVolumes))emit(row.name); + // The verify carrier's closing observation lists twice; then the target changes. + if(await Bun.file(storageRoot+"/fault-close-armed").exists()){ + const listed=Number(await Bun.file(storageRoot+"/fault-close-armed").text()||0)+1; + if(listed<2)await Bun.write(storageRoot+"/fault-close-armed",String(listed)); + else{ + const rows=await readRows(storageVolumes);rows[0].createdAt="2026-01-01T00:00:00Z";await writeRows(storageVolumes,rows); + await (await import("node:fs/promises")).rename(storageRoot+"/fault-close-armed",storageRoot+"/fault-target-changed"); + } + } + process.exit(0); } if(storageArgs[0]==="volume"&&storageArgs[1]==="inspect"&&argument("--format").includes(".Mountpoint")){ const rows=await readRows(storageVolumes); @@ -90,6 +100,12 @@ if(storageArgs[0]==="rm"){ await Bun.write(storageRoot+"/fault-post-remove",id); } } + if(await Bun.file(storageRoot+"/fault-after-verify-close").exists()&&removed.host.Mounts[1].ReadOnly){ + const program=removed.host.Mounts[0].Source,owner=program.slice(0,program.indexOf("/storage-carriers/")); + const hash=(await import("node:crypto")).createHash("sha256").update(removed.host.Mounts[1].Source).digest("hex"); + const journal=await Bun.file(owner+"/storage-witnesses/"+hash+"/carrier.json").json(); + if(journal.intent?.operation==="verify")await (await import("node:fs/promises")).rename(storageRoot+"/fault-after-verify-close",storageRoot+"/fault-close-armed"); + } await writeRows(storageCarriers,rows.filter(row=>row.id!==id));console.log(id);process.exit(0); } if(storageArgs[0]==="container"&&storageArgs[1]==="ls"&&argument("--format")==="{{json .ID}}"){ diff --git a/tests/native-compose-engine-identity.test.ts b/tests/native-compose-engine-identity.test.ts index e42757772..6b64461da 100644 --- a/tests/native-compose-engine-identity.test.ts +++ b/tests/native-compose-engine-identity.test.ts @@ -23,11 +23,14 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { bindNativeComposeEngineSocketPath, + createNativeComposeEngineIdentityAdmissions, createNativeComposeEngineIdentityObserver, NativeComposeEngineIdentityError, } from "../src/lib/native-compose-engine-identity.ts"; import { observeNativeComposeIngress } from "../src/lib/native-compose-ingress.ts"; +import { createNativeComposeProbe } from "../src/lib/native-compose-ownership.ts"; import { NativeComposeRoutingError } from "../src/lib/native-compose-routing.ts"; +import { nativeComposeStorageIdentityProbe } from "../src/lib/native-compose-storage-witness-docker-identity.ts"; import { restoreEnv } from "./helpers/env.ts"; const ENGINE = "synthetic-engine"; @@ -657,3 +660,214 @@ test("pure socket path binding pins mode and refuses ambiguous or relative selec expect(bound.path).toBe(socket); expect(requests).toHaveLength(0); }); + +// Command-scoped storage-witness admissions. A CLI stand-in counts each Docker +// CLI fence; the real Unix server counts each direct fence. +function cliStandIn(id: () => string = () => ENGINE) { + let calls = 0; + return { + calls: () => calls, + read: () => { + calls++; + return Promise.resolve(`${JSON.stringify(id())}\n`); + }, + }; +} +function fence( + admissions: ReturnType, + cli: ReturnType +) { + return admissions({ cli: cli.read, timeoutMs: 5000, environment }); +} +test("storage fences answer the first one with the CLI and admit only an equal direct ID", async () => { + const admissions = createNativeComposeEngineIdentityAdmissions(); + const cli = cliStandIn(); + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + expect([cli.calls(), requests.length]).toEqual([1, 1]); + for (let index = 0; index < 3; index++) { + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + } + // One fresh direct request per later fence; no further CLI process. + expect([cli.calls(), requests.length, connections.size]).toEqual([1, 4, 4]); +}); +test("a changed daemon ID between two admitted fences is returned, never cached", async () => { + const admissions = createNativeComposeEngineIdentityAdmissions(); + const cli = cliStandIn(); + await fence(admissions, cli); + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + handler = (_req, res) => + res.end(JSON.stringify({ ID: "replacement-engine", private: CANARY })); + expect(await fence(admissions, cli)).toBe('"replacement-engine"\n'); + handler = (_req, res) => res.end(JSON.stringify({ ID: ENGINE })); + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + expect([cli.calls(), requests.length]).toEqual([1, 4]); +}); +test("a different or failed admission read keeps the CLI for that binding", async () => { + for (const reply of [ + (_req: IncomingMessage, res: ServerResponse) => + res.end(JSON.stringify({ ID: "other-engine" })), + (_req: IncomingMessage, res: ServerResponse) => { + res.statusCode = 500; + res.end(); + }, + ]) { + handler = reply; + requests = []; + const admissions = createNativeComposeEngineIdentityAdmissions(); + const cli = cliStandIn(); + for (let index = 0; index < 3; index++) { + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + } + expect([cli.calls(), requests.length]).toEqual([3, 1]); + } +}); +test("admitted direct failures refuse without CLI fallback", async () => { + const admissions = createNativeComposeEngineIdentityAdmissions(); + const cli = cliStandIn(); + await fence(admissions, cli); + handler = (_req, res) => { + res.statusCode = 500; + res.end(CANARY); + }; + await refusal(() => fence(admissions, cli)); + expect([cli.calls(), requests.length]).toEqual([1, 2]); +}); +test("a moved config, socket or executable binding is admitted through the CLI again", async () => { + const path = await config("{}"); + const admissions = createNativeComposeEngineIdentityAdmissions(); + const cli = cliStandIn(); + await fence(admissions, cli); + await fence(admissions, cli); + expect([cli.calls(), requests.length]).toEqual([1, 2]); + await rename(path, join(root, "previous-config.json")); + await writeFile(path, "{}", { mode: 0o600 }); + await fence(admissions, cli); + await fence(admissions, cli); + expect([cli.calls(), requests.length]).toEqual([2, 4]); + await rename(socket, join(root, "original.sock")); + await listen(socket); + await fence(admissions, cli); + await fence(admissions, cli); + expect([cli.calls(), requests.length]).toEqual([3, 6]); + await rename(docker, join(root, "original-docker")); + await writeFile(docker, "#!/bin/sh\nexit 98\n", { mode: 0o700 }); + await fence(admissions, cli); + await fence(admissions, cli); + expect([cli.calls(), requests.length]).toEqual([4, 8]); +}); +test("ineligible or unbindable selections and CLI admission failures stay on the CLI", async () => { + for (const selection of [ + { DOCKER_HOST: undefined }, + { DOCKER_CONTEXT: "selected-context" }, + { DOCKER_HOST: `unix://${join(root, "missing.sock")}` }, + ]) { + const admissions = createNativeComposeEngineIdentityAdmissions(); + const cli = cliStandIn(); + for (let index = 0; index < 2; index++) { + expect( + await admissions({ + cli: cli.read, + timeoutMs: 5000, + environment: { ...environment, ...selection }, + }) + ).toBe(`"${ENGINE}"\n`); + } + expect(cli.calls()).toBe(2); + } + expect(requests).toHaveLength(0); + const admissions = createNativeComposeEngineIdentityAdmissions(); + const failure = new Error("synthetic CLI fence failure"); + await expect( + admissions({ + cli: () => Promise.reject(failure), + timeoutMs: 5000, + environment, + }) + ).rejects.toBe(failure); + // A failed CLI admission records nothing: the next fence admits again. + const cli = cliStandIn(); + await fence(admissions, cli); + await fence(admissions, cli); + expect([cli.calls(), requests.length]).toEqual([1, 2]); +}); +/** The storage probe snapshots process.env, like the CLI probe it wraps. */ +async function withSelectedProcessEnvironment(body: () => Promise) { + const keys = [ + "PATH", + "HOME", + "DOCKER_HOST", + "DOCKER_CONFIG", + "DOCKER_CONTEXT", + "DOCKER_TLS", + "DOCKER_TLS_VERIFY", + "DOCKER_CUSTOM_HEADERS", + "DOCKER_CERT_PATH", + "DOCKER_API_VERSION", + ] as const; + const before = keys.map((key) => process.env[key]); + try { + for (const key of keys) { + restoreEnv(key, environment[key]); + } + await body(); + } finally { + for (const [index, key] of keys.entries()) { + restoreEnv(key, before[index]); + } + } +} +test("cancelling an admitted direct fence keeps the CLI probe's closed cancellation without a process", async () => { + await withSelectedProcessEnvironment(async () => { + const admissions = createNativeComposeEngineIdentityAdmissions(); + await fence(admissions, cliStandIn()); + const controller = new AbortController(); + const probe = nativeComposeStorageIdentityProbe({ + probe: createNativeComposeProbe({ + signal: controller.signal, + timeoutMs: 5000, + }), + signal: controller.signal, + timeoutMs: 5000, + admissions, + }); + handler = () => controller.abort(); + const spawn = spyOn(Bun, "spawn"); + try { + await expect( + probe(["info", "--format", "{{json .ID}}"]) + ).rejects.toMatchObject({ code: "E_NATIVE_COMPOSE_PROBE_CANCELLED" }); + expect(spawn).not.toHaveBeenCalled(); + expect(requests).toHaveLength(2); + } finally { + spawn.mockRestore(); + } + }); +}); +test("storage identity probe routes only the exact engine-ID query", async () => { + const routed: string[] = []; + await withSelectedProcessEnvironment(async () => { + const admissions = createNativeComposeEngineIdentityAdmissions(); + const probe = nativeComposeStorageIdentityProbe({ + probe: (args) => { + routed.push(args.join(" ")); + return Promise.resolve(`${JSON.stringify(ENGINE)}\n`); + }, + timeoutMs: 5000, + admissions, + }); + for (const args of [ + ["info", "--format", "{{json .ID}}"], + ["info", "--format", "{{json .ID}}"], + ["info", "--format", '{"id":{{json .ID}}}'], + ["volume", "ls", "--format", "{{json .Name}}"], + ]) { + await probe(args); + } + expect(routed).toEqual([ + "info --format {{json .ID}}", + 'info --format {"id":{{json .ID}}}', + "volume ls --format {{json .Name}}", + ]); + expect(requests).toHaveLength(2); + }); +}); diff --git a/tests/native-compose-storage-command-executable.test.ts b/tests/native-compose-storage-command-executable.test.ts new file mode 100644 index 000000000..28cc95d99 --- /dev/null +++ b/tests/native-compose-storage-command-executable.test.ts @@ -0,0 +1,118 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + chmod, + link, + mkdtemp, + readFile, + realpath, + rename, + rm, + symlink, + utimes, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createNativeComposeStorageCommandExecutableReader } from "../src/lib/native-compose-storage-command-executable.ts"; + +let root: string; +let path: string; +beforeEach(async () => { + root = await realpath(await mkdtemp(join(tmpdir(), "command-executable-"))); + path = join(root, "docker"); + await writeFile(path, "#!/bin/sh\nexit 0\n", { mode: 0o700 }); +}); +afterEach(async () => { + await rm(root, { recursive: true, force: true }); +}); + +/** A reader whose clock treats every file as settled, counting content reads. */ +function reader(opts: { readonly settled?: boolean } = {}) { + let digests = 0; + const read = createNativeComposeStorageCommandExecutableReader({ + now: opts.settled === false ? Date.now : () => Date.now() + 60_000, + digest: async (file, size) => { + digests++; + const bytes = Buffer.alloc(size); + await file.read(bytes, 0, size, 0); + return new Bun.CryptoHasher("sha256").update(bytes).digest("hex"); + }, + }); + return { read, digests: () => digests }; +} + +test("an unchanged stat identity reuses the digest and still returns the full identity", async () => { + const { read, digests } = reader(); + const first = await read(path); + expect(await read(path)).toEqual(first); + expect(await read(path)).toEqual(first); + expect(digests()).toBe(1); + expect(first).toMatchObject({ + path, + hash: new Bun.CryptoHasher("sha256") + .update(await readFile(path)) + .digest("hex"), + }); +}); +test("replacing the executable between reads is a new identity and is re-read", async () => { + const { read, digests } = reader(); + const first = await read(path); + // Same bytes, new inode: the recorded identity no longer matches. + await rename(path, join(root, "original")); + await writeFile(path, "#!/bin/sh\nexit 0\n", { mode: 0o700 }); + const same = await read(path); + expect(same.ino).not.toBe(first.ino); + expect(same.hash).toBe(first.hash); + // Different bytes behind the same path: a different digest. + await rename(path, join(root, "second")); + await writeFile(path, "#!/bin/sh\nexit 7\n", { mode: 0o700 }); + const replaced = await read(path); + expect(replaced.hash).not.toBe(first.hash); + expect(digests()).toBe(3); +}); +test("same-size in-place tamper with restored mtime is re-read through ctime", async () => { + const { read, digests } = reader(); + const first = await read(path); + const before = await Bun.file(path).stat(); + const bytes = await readFile(path, "utf8"); + await writeFile(path, bytes.replace("exit 0", "exit 9")); + await utimes(path, before.atime, before.mtime); + const tampered = await read(path); + expect([tampered.ino, tampered.size]).toEqual([first.ino, first.size]); + expect(tampered.hash).not.toBe(first.hash); + expect(digests()).toBe(2); +}); +test("mode changes are re-read and group or other writable files still refuse", async () => { + const { read, digests } = reader(); + const first = await read(path); + await chmod(path, 0o500); + expect((await read(path)).mode).not.toBe(first.mode); + expect(digests()).toBe(2); + await chmod(path, 0o720); + await expect(read(path)).rejects.toThrow("values omitted"); + await chmod(path, 0o700); + await link(path, join(root, "second-link")); + await expect(read(path)).rejects.toThrow("values omitted"); +}); +test("a racily fresh executable is never cached", async () => { + const { read, digests } = reader({ settled: false }); + await read(path); + await read(path); + await read(path); + expect(digests()).toBe(3); +}); +test("a changed canonical target is a different cache entry", async () => { + const { read, digests } = reader(); + const other = join(root, "other"); + await writeFile(other, "#!/bin/sh\nexit 3\n", { mode: 0o700 }); + const alias = join(root, "alias"); + await symlink(path, alias); + const first = await read(alias); + expect(first.path).toBe(path); + await rm(alias); + await symlink(other, alias); + const second = await read(alias); + expect(second.path).toBe(other); + expect(second.hash).not.toBe(first.hash); + expect(digests()).toBe(2); +}); diff --git a/tests/native-compose-storage-command-record.test.ts b/tests/native-compose-storage-command-record.test.ts index c70a66b52..2bc06d105 100644 --- a/tests/native-compose-storage-command-record.test.ts +++ b/tests/native-compose-storage-command-record.test.ts @@ -1,5 +1,12 @@ import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; -import { existsSync, readFileSync } from "node:fs"; +import { + existsSync, + readFileSync, + renameSync, + statSync, + utimesSync, + writeFileSync, +} from "node:fs"; import { mkdtemp, readFile, @@ -409,6 +416,63 @@ async function transport( } } } +test.each(["unchanged", "replaced", "tampered"] as const)( + "a recorded executable %s after its digest was cached: only unchanged runs the target", + (change) => + ownedCase(async () => { + const f = await fixture(), + marker = join(f.root, "effect"); + const bin = await realpath( + await mkdtemp(join(tmpdir(), "native-command-executable-")) + ); + roots.push(bin); + const program = join(bin, "program"); + const text = '#!/bin/sh\nprintf complete > "$1"; printf value\n'; + await writeFile(program, text, { mode: 0o700 }); + // Settle past the racy window so the arming digest is cached for this command. + await new Promise((resolve) => setTimeout(resolve, 2100)); + try { + const run = runNativeComposeStorageDockerCommand({ + context: { + signal: new AbortController().signal, + deadline: Date.now() + 5000, + }, + files: f.files, + args: [program, marker], + timeoutMs: 5000, + // After the record is armed with the cached digest, before spawn. + beforeSpawn: () => { + if (change === "replaced") { + renameSync(program, join(bin, "original")); + writeFileSync(program, text, { mode: 0o700 }); + } else if (change === "tampered") { + const before = statSync(program); + writeFileSync(program, text.replace("value", "VALUE")); + utimesSync(program, before.atime, before.mtime); + } + }, + assertAdmitted: async () => {}, + originalCommand: { owner: f.owner, kind: "create", carrier: null }, + }); + if (change === "unchanged") { + expect(await run).toEqual({ exitCode: 0, stdout: "value" }); + expect(await readFile(marker, "utf8")).toBe("complete"); + expect((await f.read()).commands[0]?.settlement).toMatchObject({ + exitCode: 0, + }); + return; + } + await expect(run).rejects.toThrow("values omitted"); + expect(existsSync(marker)).toBe(false); + expect((await f.read()).commands[0]?.settlement).toBeNull(); + } finally { + if (!nativeComposeStorageCommandOwnerConfirmed(f.owner)) { + retained.add(f.root); + } + } + })(), + 30_000 +); test( "real fast command cannot execute before its durable stopped-child publication", ownedCase(async () => { diff --git a/tests/native-compose-storage-fence-cost.test.ts b/tests/native-compose-storage-fence-cost.test.ts new file mode 100644 index 000000000..3f096511b --- /dev/null +++ b/tests/native-compose-storage-fence-cost.test.ts @@ -0,0 +1,215 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdir, writeFile } from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import type { Socket } from "node:net"; +import { join } from "node:path"; +import { openNativeComposeGenerationStore } from "../src/lib/native-compose-generation.ts"; +import { fixture, invoke } from "./helpers/native-compose-command.ts"; + +const storage = { data: { kind: "persistent", scope: "worktree" } } as const; +const IDENTITY = JSON.stringify(["info", "--format", "{{json .ID}}"]); +// Every storage target observation lists volumes exactly twice (open and close). +const OBSERVATION = JSON.stringify([ + "volume", + "ls", + "--format", + "{{json .Name}}", +]); +// Per storage proof (root, verify, root-after carriers) on this synthetic engine. +// Before X03: 86 engine-ID fences and 21 target observations per proof. +const PROOF_FENCES = 81; +const PROOF_OBSERVATIONS = 20; +// The command's saved file-engine observation reads the ID twice, unchanged. +const FILE_ENGINE_FENCES = 2; + +const servers: Server[] = []; +const peers = new Set(); +afterEach(async () => { + for (const peer of peers) { + peer.destroy(); + } + peers.clear(); + for (const server of servers.splice(0)) { + await new Promise((resolve) => server.close(() => resolve())); + } +}); +/** A real Unix `GET /info` endpoint for the same engine the synthetic CLI reports. */ +async function engine( + root: string, + id: (request: number) => string = () => "fixture-engine" +) { + const socket = join(root, "engine.sock"); + const client = join(root, "client"); + await mkdir(client, { mode: 0o700 }); + await writeFile(join(client, "config.json"), "{}\n", { mode: 0o600 }); + let served = 0; + const server = createServer((req, res) => { + const request = served++; + if (req.method !== "GET" || req.url !== "/info") { + res.statusCode = 404; + res.end(); + return; + } + res.end(JSON.stringify({ ID: id(request) })); + }); + server.on("connection", (peer) => peers.add(peer)); + servers.push(server); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(socket, resolve); + }); + return { + env: { DOCKER_HOST: `unix://${socket}`, DOCKER_CONFIG: client }, + served: () => served, + }; +} +async function lines(root: string, name: string): Promise { + const file = Bun.file(join(root, name)); + return (await file.exists()) + ? (await file.text()).trim().split("\n").filter(Boolean) + : []; +} +async function counted( + root: string, + args: readonly string[], + selected?: Awaited> +) { + const before = (await lines(root, "requests")).length; + const direct = selected?.served() ?? 0; + const result = await invoke( + root, + [...args], + 30_000, + 120_000, + selected?.env ?? {} + ); + const calls = (await lines(root, "requests")).slice(before); + return { + code: result.code, + cli: calls.filter((line) => line === IDENTITY).length, + direct: (selected?.served() ?? 0) - direct, + observations: calls.filter((line) => line === OBSERVATION).length / 2, + }; +} +async function saved(root: string) { + const store = await openNativeComposeGenerationStore({ + projectRoot: root, + instance: null, + mode: "saved", + }); + try { + return await store.loadCurrent(); + } finally { + await store.close(); + } +} +async function enrolled() { + const root = await fixture("", false, { storage, noHooks: true }); + expect((await invoke(root, undefined, 30_000, 120_000)).code).toBe(0); + return root; +} + +test("storage fence count per command: every proof kept, exact duplicates gone, one CLI admission", async () => { + const root = await enrolled(); + // CLI transport (no explicit Unix selection): every fence is a `docker info`. + expect(await counted(root, ["exec", "web", "--", "true"])).toEqual({ + code: 0, + cli: FILE_ENGINE_FENCES + 2 * PROOF_FENCES, + direct: 0, + observations: 2 * PROOF_OBSERVATIONS, + }); + expect(await counted(root, ["up", "--detach", "--json"])).toEqual({ + code: 0, + cli: FILE_ENGINE_FENCES + 6 * PROOF_FENCES, + direct: 0, + observations: 6 * PROOF_OBSERVATIONS, + }); + // Admitted Unix selection: the same fences, one fresh read each. Only the + // first storage fence is a CLI read, checked against one extra direct read. + const selected = await engine(root); + expect(await counted(root, ["exec", "web", "--", "true"], selected)).toEqual({ + code: 0, + cli: FILE_ENGINE_FENCES + 1, + direct: 2 * PROOF_FENCES, + observations: 2 * PROOF_OBSERVATIONS, + }); + expect(await counted(root, ["up", "--detach", "--json"], selected)).toEqual({ + code: 0, + cli: FILE_ENGINE_FENCES + 1, + direct: 6 * PROOF_FENCES, + observations: 6 * PROOF_OBSERVATIONS, + }); +}, 390_000); + +test("a refused admission keeps every storage fence on the Docker CLI for the command", async () => { + const root = await enrolled(); + // The socket answers for a different engine than the CLI's selection. + const selected = await engine(root, () => "other-engine"); + expect(await counted(root, ["exec", "web", "--", "true"], selected)).toEqual({ + code: 0, + cli: FILE_ENGINE_FENCES + 2 * PROOF_FENCES, + direct: 1, + observations: 2 * PROOF_OBSERVATIONS, + }); +}, 390_000); + +test("a changed daemon ID between two admitted fences refuses before the workload", async () => { + const root = await enrolled(); + const before = await saved(root); + const carriers = await Bun.file(join(root, "carriers")).text(); + const operations = (await lines(root, "storage-operations")).length; + // Admission reads the original ID; every later fence sees another engine. + const selected = await engine(root, (request) => + request === 0 ? "fixture-engine" : "replacement-engine" + ); + const requestsBefore = (await lines(root, "requests")).length; + const result = await counted(root, ["up", "--detach", "--json"], selected); + expect(result.code).toBe(1); + // The admission and the next fence: the opening target observation refuses. + expect(result.direct).toBe(2); + const calls = (await lines(root, "requests")).slice(requestsBefore); + expect( + calls.some((line) => { + const args: string[] = JSON.parse(line); + return ( + args[0] === "compose" || + args[0] === "create" || + args[0] === "start" || + args[0] === "rm" + ); + }) + ).toBe(false); + expect((await lines(root, "storage-operations")).length).toBe(operations); + expect(await Bun.file(join(root, "carriers")).text()).toBe(carriers); + const after = await saved(root); + expect(after.storageWitnesses).toEqual(before.storageWitnesses); + expect(after.storageWitnessesPending).toBe(false); + expect(after.pending).toBeNull(); +}, 390_000); + +test("a target change after the verify carrier's closing observation is still detected", async () => { + const root = await enrolled(); + const events = (await lines(root, "storage-events")).length; + // Change the selected volume right after the verify carrier's own closing + // observation, where the witness's former duplicate re-observation stood. + await writeFile(join(root, "fault-after-verify-close"), "0"); + const requestsBefore = (await lines(root, "requests")).length; + const result = await counted(root, ["exec", "web", "--", "true"]); + expect(await Bun.file(join(root, "fault-target-changed")).exists()).toBe( + true + ); + expect(result.code).not.toBe(0); + // The carrier's closing observation (two listings, the change after the + // second) completed; a later complete observation saw the change and refused. + const calls = (await lines(root, "requests")).slice(requestsBefore); + const removed = calls.findLastIndex( + (line) => (JSON.parse(line) as string[])[0] === "rm" + ); + expect( + calls.slice(removed).filter((line) => line === OBSERVATION) + ).toHaveLength(4); + // The opening proof refused: no workload exec and no later proof ran. + const after = (await lines(root, "storage-events")).slice(events); + expect(after).not.toContain("exec"); + expect(after).toEqual(["root", "verify"]); +}, 390_000); From 994ddb894ef407319a2f6ba4bbdd04511e07ea40 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 02:41:07 -0400 Subject: [PATCH 20/32] chore(x03): revert temporary witness probe trace Measurement is complete; the hosted trace run was 38211327829. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 23 -- scripts/x01-trace-summary.ts | 268 ------------------ scripts/x03-identity-bench.sh | 29 -- src/lib/native-compose-command-storage.ts | 14 +- src/lib/native-compose-command.ts | 9 +- src/lib/native-compose-engine-identity.ts | 9 +- src/lib/native-compose-ownership.ts | 5 +- src/lib/native-compose-phase-trace.ts | 8 +- ...tive-compose-storage-command-executable.ts | 4 - ...ompose-storage-witness-docker-inventory.ts | 8 +- .../native-compose-storage-witness-docker.ts | 45 +-- src/lib/native-compose-storage-witness.ts | 34 +-- src/lib/shell.ts | 17 +- src/lib/x01-cost-probe.ts | 176 ------------ tests/e2e/harness.ts | 9 - .../e2e/native-config-networks-acceptance.ts | 4 - 16 files changed, 32 insertions(+), 630 deletions(-) delete mode 100644 scripts/x01-trace-summary.ts delete mode 100644 scripts/x03-identity-bench.sh delete mode 100644 src/lib/x01-cost-probe.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 09052d11e..457db89f4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -300,20 +300,13 @@ jobs: - name: Verify owned bridge policy and aliases through real compiler and Compose normalization if: matrix.shard == 4 run: HACK_TEST_COMPOSE_CONFIG=1 bun test tests/native-compose-renderer-config.test.ts --test-name-pattern '^one authored bridge keeps internal policy and static aliases through compiler and Compose normalization$' - # TEMPORARY X03 measurement; reverted before the final commit. - - name: X03 engine identity fence benchmark - if: matrix.shard == 1 - run: bash scripts/x03-identity-bench.sh || true - name: Run local and Docker E2E shard env: HACK_E2E_PROCESS_POLICY_REFUSAL_CAPSULE: "1" HACK_E2E_CAPSULE_DIR: ${{ runner.temp }}/process-policy-capsules HACK_E2E_CLI_BIN: ${{ github.workspace }}/dist/hack HACK_E2E_SHARD: ${{ matrix.shard }}/4 - # TEMPORARY X01 measurement; reverted before the final commit. - HACK_X01_TRACE_DIR: ${{ runner.temp }}/x01-trace run: | - mkdir -p "$HACK_X01_TRACE_DIR" HACK_E2E_DOCKER_HOST="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" HACK_E2E_NETWORK_SOURCE_REVISION="$(git rev-parse HEAD)" HACK_E2E_NETWORK_CLI_SHA256="$(sha256sum dist/hack | cut -d ' ' -f1)" @@ -322,22 +315,6 @@ jobs: HACK_E2E_COMPOSE_PLUGIN_SHA256="$(sha256sum "$HACK_E2E_COMPOSE_PLUGIN_PATH" | cut -d ' ' -f1)" export HACK_E2E_DOCKER_HOST HACK_E2E_NETWORK_SOURCE_REVISION HACK_E2E_NETWORK_CLI_SHA256 HACK_E2E_NETWORK_COMPILER_SHA256 HACK_E2E_COMPOSE_PLUGIN_PATH HACK_E2E_COMPOSE_PLUGIN_SHA256 HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker --shard="$HACK_E2E_SHARD" - # TEMPORARY X01 measurement; reverted before the final commit. - - name: X01 witness cost summary - if: always() - run: | - for f in "$RUNNER_TEMP"/x01-trace/*.jsonl; do - [ -f "$f" ] || continue - bun scripts/x01-trace-summary.ts "$f" "$(basename "$f" .jsonl)" || true - done - - name: X01 witness cost trace artifact - if: always() - uses: actions/upload-artifact@v4 - with: - name: x01-trace-shard-${{ matrix.shard }} - path: ${{ runner.temp }}/x01-trace - if-no-files-found: ignore - retention-days: 7 - name: Preserve process-policy capsule and startup diagnostic # Closed fixed-field JSON only (counts, enums, booleans); the gh log viewer truncates it inline. if: failure() diff --git a/scripts/x01-trace-summary.ts b/scripts/x01-trace-summary.ts deleted file mode 100644 index ba276c07f..000000000 --- a/scripts/x01-trace-summary.ts +++ /dev/null @@ -1,268 +0,0 @@ -// TEMPORARY X01 measurement summary. Reverted before the final commit. -// Reads closed-label JSONL written by src/lib/x01-cost-probe.ts. -type Event = { - readonly p: number; - readonly v: string; - readonly k: string; - readonly s?: string; - readonly l?: string; - readonly ms?: number; - readonly t?: number; - readonly c?: number; - readonly line?: { - readonly phase?: string; - readonly boundary?: string; - readonly durationMs?: number | null; - }; -}; -const WITNESS = new Set([ - "storage-prepare", - "storage-verify", - "carrier-new", - "inspect", - "provision", - "finish", - "witness-verify", - "witness-prepare", - "witness-enroll", -]); -const isWitness = (label: string) => - WITNESS.has(label) || label.startsWith("invoke:"); -const LEAF = new Set(["probe", "run", "exec", "hash"]); -const PHASE = /^(oneoff|storage|guard|finish|finalize)\.[a-z-]+$/; - -const path = process.argv[2]; -const title = process.argv[3] ?? path; -if (!path) { - throw new Error("usage: x01-trace-summary [title]"); -} -const text = await Bun.file(path).text(); -const events: Event[] = []; -for (const line of text.split("\n")) { - if (line.trim() === "") { - continue; - } - try { - events.push(JSON.parse(line) as Event); - } catch { - // skip partial line - } -} -const byProcess = new Map(); -for (const event of events) { - const list = byProcess.get(event.p) ?? []; - list.push(event); - byProcess.set(event.p, list); -} -type Proc = { - readonly pid: number; - readonly verb: string; - readonly totalMs: number; - readonly witnessMs: number; - readonly events: Event[]; - readonly kind: string; -}; -function topWitnessMs(list: Event[]): number { - let total = 0; - for (const event of list) { - if (event.k !== "scope" || !event.l || !isWitness(event.l)) { - continue; - } - const parts = (event.s ?? "").split(">"); - parts.pop(); - if (parts.some(isWitness)) { - continue; - } - total += event.ms ?? 0; - } - return total; -} -const procs: Proc[] = []; -for (const [pid, list] of byProcess) { - const exit = list.find((event) => event.k === "exit"); - const verb = list[0]?.v ?? "other"; - const labels = new Set(list.map((event) => event.l ?? "")); - let kind = verb; - if (verb === "up" || verb === "restart" || verb === "run") { - if (labels.has("invoke:seed")) { - kind = `${verb}(cold)`; - } else if (labels.has("invoke:verify")) { - kind = `${verb}(warm)`; - } else { - kind = `${verb}(no-witness)`; - } - } else if (verb === "exec") { - kind = labels.has("invoke:verify") ? "exec(storage)" : "exec(image)"; - } - procs.push({ - pid, - verb, - totalMs: exit?.t ?? Math.max(...list.map((event) => event.t ?? 0)), - witnessMs: topWitnessMs(list), - events: list, - kind, - }); -} -const fmt = (ms: number) => (ms / 1000).toFixed(1); -const out: string[] = []; -out.push(`#### X01 trace: ${title}`); -out.push(`events=${events.length} processes=${procs.length}`); -out.push(""); -out.push( - "kind | n | avg total s | sum total s | avg witness s | sum witness s" -); -const kinds = [...new Set(procs.map((proc) => proc.kind))].sort(); -for (const kind of kinds) { - const selected = procs.filter((proc) => proc.kind === kind); - const total = selected.reduce((sum, proc) => sum + proc.totalMs, 0); - const witness = selected.reduce((sum, proc) => sum + proc.witnessMs, 0); - out.push( - `${kind} | ${selected.length} | ${fmt(total / selected.length)} | ${fmt(total)} | ${fmt(witness / selected.length)} | ${fmt(witness)}` - ); -} -const allTotal = procs.reduce((sum, proc) => sum + proc.totalMs, 0); -const allWitness = procs.reduce((sum, proc) => sum + proc.witnessMs, 0); -out.push( - `ALL | ${procs.length} | - | ${fmt(allTotal)} | - | ${fmt(allWitness)}` -); -out.push(""); - -function leafTable(selected: Proc[], heading: string, limit = 24) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (!(LEAF.has(event.k) && event.l)) { - continue; - } - const key = `${event.k}:${event.l}`; - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push(`${heading} (per process avg over n=${selected.length})`); - out.push("leaf | calls/proc | s/proc | ms/call"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function leafByScope(selected: Proc[], heading: string, limit = 40) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (!(LEAF.has(event.k) && event.l)) { - continue; - } - const parts = (event.s ?? "") - .split(">") - .filter( - (part) => - isWitness(part) || part.startsWith("cmd:") || PHASE.test(part) - ); - const inner = parts.slice(-2).join(">") || "(none)"; - const key = `${inner} :: ${event.k}:${event.l}`; - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push( - `${heading} leaves by scope (per process avg over n=${selected.length})` - ); - out.push("scope :: leaf | calls/proc | s/proc | ms/call"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function scopeTable(selected: Proc[], heading: string, limit = 30) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (event.k !== "scope" || !event.l) { - continue; - } - const parts = (event.s ?? "").split(">"); - const key = parts - .filter( - (part) => - PHASE.test(part) || - isWitness(part) || - part.startsWith("xattr:") || - part.startsWith("cmd:") || - part.startsWith("exec-") - ) - .join(">"); - if (key === "") { - continue; - } - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push(`${heading} scopes (per process avg over n=${selected.length})`); - out.push("scope path | n/proc | s/proc | ms/each"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function phaseRows(proc: Proc) { - const rows = new Map(); - for (const event of proc.events) { - if ( - event.k === "phase" && - event.line?.boundary !== "begin" && - event.line?.phase - ) { - const row = rows.get(event.line.phase) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.line.durationMs ?? 0; - rows.set(event.line.phase, row); - } - } - return rows; -} -for (const kind of [ - "up(cold)", - "up(warm)", - "exec(storage)", - "restart(warm)", - "run(warm)", - "run(cold)", - "down", -]) { - const selected = procs.filter((proc) => proc.kind === kind); - if (selected.length === 0) { - continue; - } - const sample = selected[0] as Proc; - out.push( - `### ${kind}: first sample total ${fmt(sample.totalMs)} s, witness ${fmt(sample.witnessMs)} s` - ); - out.push("phase-trace phase | spans | s"); - for (const [phase, row] of phaseRows(sample)) { - out.push(`${phase} | ${row.n} | ${fmt(row.ms)}`); - } - out.push(""); - scopeTable([sample], `${kind} first sample`, 40); - leafTable([sample], `${kind} first sample leaves`); - scopeTable(selected, `${kind} all`, 30); - leafTable(selected, `${kind} all leaves`); - leafByScope(selected, `${kind} all`); -} -console.log(out.join("\n")); diff --git a/scripts/x03-identity-bench.sh b/scripts/x03-identity-bench.sh deleted file mode 100644 index 050dfd5c1..000000000 --- a/scripts/x03-identity-bench.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env bash -# TEMPORARY X03 measurement; reverted before the final commit. -# Decomposes one engine-ID fence on the hosted engine. Prints timings only. -set -euo pipefail -host="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" -socket="${host#unix://}" -n=40 -bench() { - local label="$1" - shift - local start end - start=$(date +%s%N) - for _ in $(seq "$n"); do - "$@" >/dev/null - done - end=$(date +%s%N) - echo "x03-bench ${label}: $(((end - start) / n / 1000)) us/call over ${n}" -} -echo "x03-bench selection: $(case "$host" in unix://*) echo unix ;; *) echo other ;; esac), DOCKER_HOST set: ${DOCKER_HOST:+yes}" -echo "x03-bench client plugins: $(docker info --format '{{len .ClientInfo.Plugins}}')" -bench "docker info --format {{json .ID}}" docker info --format '{{json .ID}}' -bench "docker version --format server" docker version --format '{{.Server.Version}}' -bench "docker volume ls" docker volume ls --format '{{json .Name}}' -bench "docker context inspect (client only)" docker context inspect --format '{{json .Endpoints.docker.Host}}' -bench "curl GET /info (direct Unix)" curl -sf --unix-socket "$socket" http://docker/info -bench "curl GET /_ping (direct Unix)" curl -sf --unix-socket "$socket" http://docker/_ping -sha="$(command -v docker)" -bench "sha256sum docker CLI" sha256sum "$sha" -echo "x03-bench docker CLI size: $(stat -Lc %s "$sha") bytes" diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index d9b58c132..c9384e64c 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -23,7 +23,6 @@ import { } from "./native-compose-storage-witness.ts"; import { createNativeComposeDockerStorageXattrCarrier } from "./native-compose-storage-witness-docker.ts"; import type { NativeComposeStorageXattrCarrier } from "./native-compose-storage-witness-xattr-carrier.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; type Selection = { readonly name: string; readonly storage: string }; type Document = Readonly>; @@ -135,12 +134,7 @@ export type NativeComposeCommandStorageVerifyOptions = { /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ -export function prepareNativeComposeCommandStorage( - opts: Parameters[0] -): ReturnType { - return x01Scope("storage-prepare", () => x01Prepare(opts)); -} -async function x01Prepare(opts: { +export async function prepareNativeComposeCommandStorage(opts: { readonly store: NativeComposeGenerationStore; readonly mutation: NativeComposeMutation; readonly operation: "up" | "restart" | "run" | "exec"; @@ -231,11 +225,7 @@ async function x01Prepare(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = ( - generation: NativeComposeGeneration, - options: NativeComposeCommandStorageVerifyOptions = {} - ) => x01Scope("storage-verify", () => x01Verify(generation, options)); - const x01Verify = async ( + const verify = async ( generation: NativeComposeGeneration, options: NativeComposeCommandStorageVerifyOptions = {} ) => { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index 0aa92b81e..8c5dcaf82 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -138,7 +138,6 @@ import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; export type NativeComposeCommandOptions = { @@ -944,9 +943,7 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = (proof: NativeComposeCommandStorageVerifyOptions) => - x01Scope("exec-assert-saved", () => x01AssertSaved(proof)); - const x01AssertSaved = async ( + const assertSaved = async ( proof: NativeComposeCommandStorageVerifyOptions ) => { const latest = await store.loadCurrent(); @@ -965,9 +962,7 @@ async function savedCommand(opts: { signal, assertSaved, run: () => - x01Scope("exec-process", () => - runSavedProcess({ options, generation, document, base, signal }) - ), + runSavedProcess({ options, generation, document, base, signal }), }); }); } diff --git a/src/lib/native-compose-engine-identity.ts b/src/lib/native-compose-engine-identity.ts index 205641d63..63618b2cc 100644 --- a/src/lib/native-compose-engine-identity.ts +++ b/src/lib/native-compose-engine-identity.ts @@ -3,7 +3,6 @@ import { request } from "node:http"; import { dirname, isAbsolute, join, normalize, parse } from "node:path"; import { isRecord } from "./guards.ts"; import { bindNativeComposeEngineConfig } from "./native-compose-engine-config.ts"; -import { x01Mark, x01Time } from "./x01-cost-probe.ts"; const ENGINE_ID = /^[A-Za-z0-9:-]{1,128}$/; const AMBIGUOUS_PATH = /[%?#\0\r\n]/; @@ -412,22 +411,18 @@ export function createNativeComposeEngineIdentityAdmissions(): (opts: { return await opts.cli(); } const state = admitted.get(bound.key); - const direct = bound.read; if (state === true) { - return `${JSON.stringify(await x01Time("ident", "direct", direct))}\n`; + return `${JSON.stringify(await bound.read())}\n`; } const text = await opts.cli(); if (state === undefined) { let equal = false; try { const value: unknown = JSON.parse(text); - equal = - typeof value === "string" && - value === (await x01Time("ident", "admission", direct)); + equal = typeof value === "string" && value === (await bound.read()); } catch { equal = false; } - x01Mark("ident-admit", equal ? "admitted" : "refused"); admitted.set(bound.key, equal); } return text; diff --git a/src/lib/native-compose-ownership.ts b/src/lib/native-compose-ownership.ts index d8bc9dc8a..0019b3fc3 100644 --- a/src/lib/native-compose-ownership.ts +++ b/src/lib/native-compose-ownership.ts @@ -3,7 +3,6 @@ import { isRecord } from "./guards.ts"; import { nativeComposeVolumeCreatedAt } from "./native-compose-retained-storage.ts"; import { beginNativeCpuChild } from "./native-cpu-diagnostics.ts"; import { findExecutableInPath } from "./shell.ts"; -import { x01Label, x01Time } from "./x01-cost-probe.ts"; const PROJECT = /^[a-z0-9][a-z0-9_-]*$/; const SERVICE = /^[a-z0-9][a-z0-9._-]{0,62}$/; @@ -1233,7 +1232,7 @@ export function createNativeComposeProbe( probeRefused("E_NATIVE_COMPOSE_PROBE", "operation"); } const environment = { ...process.env }; - const x01Inner = async (args: readonly string[]): Promise => { + return async (args) => { checkInterruption(opts.signal, false); const remainingTime = deadline - Date.now(); if (remainingTime <= 0) { @@ -1311,8 +1310,6 @@ export function createNativeComposeProbe( observeCpu(probeExitCode(completion[2])); } }; - return (args) => - x01Time("probe", x01Label(args, false), () => x01Inner(args)); } function probeExitCode( diff --git a/src/lib/native-compose-phase-trace.ts b/src/lib/native-compose-phase-trace.ts index 943b79320..540f833b0 100644 --- a/src/lib/native-compose-phase-trace.ts +++ b/src/lib/native-compose-phase-trace.ts @@ -1,6 +1,5 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { writeSync } from "node:fs"; -import { x01Enabled, x01Phase, x01Scope } from "./x01-cost-probe.ts"; export const NATIVE_COMPOSE_PHASE_TRACE = "HACK_NATIVE_COMPOSE_PHASE_TRACE"; const PHASES = [ @@ -136,8 +135,6 @@ export function withNativeComposePhaseTrace(operation: () => Promise) { writeSync(2, line); }, }); - } else if (x01Enabled()) { - trace = createNativeComposePhaseTrace({ write: x01Phase }); } } catch { // Diagnostic setup never changes selection, results or effect authority. @@ -150,8 +147,5 @@ export function measureNativeComposePhase( phase: Phase, operation: () => Promise ): Promise { - return x01Scope( - phase, - () => context.getStore()?.measure(phase, operation) ?? operation() - ); + return context.getStore()?.measure(phase, operation) ?? operation(); } diff --git a/src/lib/native-compose-storage-command-executable.ts b/src/lib/native-compose-storage-command-executable.ts index 780dae040..a8817d685 100644 --- a/src/lib/native-compose-storage-command-executable.ts +++ b/src/lib/native-compose-storage-command-executable.ts @@ -2,7 +2,6 @@ import { constants } from "node:fs"; import { type FileHandle, lstat, open, realpath } from "node:fs/promises"; import type { NativeComposeStorageCommandExecutable } from "./native-compose-storage-command-record.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Label, x01Mark } from "./x01-cost-probe.ts"; /** Coarser than any supported filesystem timestamp granularity (1 s on HFS+/ext3). */ const RACY_MS = 2000; @@ -84,9 +83,6 @@ export function createNativeComposeStorageCommandExecutableReader( const size = Number(before.size); const started = now(); const cached = settled.get(canonical); - if (cached?.identity === identity) { - x01Mark("hash-cached", x01Label([canonical])); - } const hash = cached?.identity === identity ? cached.hash : await digest(file, size); const after = await file.stat({ bigint: true }), diff --git a/src/lib/native-compose-storage-witness-docker-inventory.ts b/src/lib/native-compose-storage-witness-docker-inventory.ts index f292021ce..b5bdb7410 100644 --- a/src/lib/native-compose-storage-witness-docker-inventory.ts +++ b/src/lib/native-compose-storage-witness-docker-inventory.ts @@ -8,7 +8,6 @@ import { type NativeComposeStorageXattrTarget, } from "./native-compose-storage-witness-xattr-carrier.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$/; @@ -165,12 +164,7 @@ export async function assertNativeComposeStorageDockerCarrierVolume(opts: { /** Complete holder reads include foreign containers. The two inventories and selected * volume reread are observed fences; they do not claim an atomic daemon transaction. */ -export function observeNativeComposeStorageDockerTarget( - opts: Parameters[0] -): ReturnType { - return x01Scope("target", () => x01ObserveTarget(opts)); -} -async function x01ObserveTarget(opts: { +export async function observeNativeComposeStorageDockerTarget(opts: { readonly probe: Probe; readonly selection: Selection; readonly current: NativeComposeMaterialBinding; diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index d955faf06..c8e062188 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -98,7 +98,6 @@ import { refuseNativeComposeStorageXattr as refuse, } from "./native-compose-storage-witness-xattr-codec.ts"; import { findExecutableInPath, type RunExitEvent, run } from "./shell.ts"; -import { x01Label, x01Mark, x01Scope, x01Time } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const PROCESS = @@ -818,10 +817,7 @@ async function groupAbsent( } /** Every identity check runs per call; only an unchanged stat identity skips the * content read for the rest of this command. */ -const commandExecutable = (path: string) => - x01Time("hash", x01Label([path]), () => - readNativeComposeStorageCommandExecutable(path) - ); +const commandExecutable = readNativeComposeStorageCommandExecutable; async function boundedHostFile(path: string): Promise { const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); try { @@ -907,14 +903,7 @@ async function commandBoot( } /** No callback can reject before the shared owner awaits exit. A rejected owner or * unproven group retains the journal; daemon effects are never inferred from exit. */ -export function runNativeComposeStorageDockerCommand( - supplied: Parameters[0] -): ReturnType { - return x01Scope(`cmd:${x01Label(supplied.args)}`, () => - x01RunCommandInner(supplied) - ); -} -async function x01RunCommandInner(supplied: { +export async function runNativeComposeStorageDockerCommand(supplied: { readonly context: Pick; readonly files: Pick< Files, @@ -1259,12 +1248,7 @@ function matchesScope( * image; other daemons and uncached images refuse rather than pull or emulate. * PostgreSQL data roots and crash durability remain unqualified. No module * initialization executes Docker or the helper. */ -export function createNativeComposeDockerStorageXattrCarrier( - opts: Context -): Promise { - return x01Scope("carrier-new", () => x01CreateCarrier(opts)); -} -async function x01CreateCarrier( +export async function createNativeComposeDockerStorageXattrCarrier( opts: Context ): Promise { const context: Context = Object.freeze({ ...opts }); @@ -1386,7 +1370,6 @@ async function x01CreateCarrier( const binding = await current(context, selection); if (same(binding, closed.binding)) { lifetime(context); - x01Mark("target-reused", "closing"); return closed.target; } } catch (error) { @@ -2051,21 +2034,17 @@ async function x01CreateCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect: (selection, options) => - x01Scope("inspect", () => inspectPort(selection, options)), - provision: (selection) => - x01Scope("provision", () => provision(selection)), - invoke: (input) => - x01Scope(`invoke:${input.request.operation}`, () => invoke(input)), + inspect: inspectPort, + provision, + invoke, finish: context.originalCommandRecords === true - ? ({ input, completion }) => - x01Scope("finish", async () => { - closing = null; - const finish = finalizers.get(input) ?? refuse(); - finalizers.delete(input); - await finish(completion); - }) + ? async ({ input, completion }) => { + closing = null; + const finish = finalizers.get(input) ?? refuse(); + finalizers.delete(input); + await finish(completion); + } : undefined, }, signal: context.signal, diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index 1fc68abf7..7586eb73b 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -78,7 +78,6 @@ import { nativeComposeStorageXattrValueMatches, sameNativeComposeStorageXattrRoot, } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; export type { NativeComposeStorageWitnessReference } from "./native-compose-storage-witness-state.ts"; @@ -499,16 +498,7 @@ function requireXattrAdmission( refuse(); } } -function invokeXattr( - opts: Parameters[0] -): ReturnType { - const operation = - isRecord(opts.request) && typeof opts.request.operation === "string" - ? opts.request.operation - : "unknown"; - return x01Scope(`xattr:${operation}`, () => x01InvokeXattr(opts)); -} -async function x01InvokeXattr(opts: { +async function invokeXattr(opts: { readonly carrier: NativeComposeStorageXattrCarrier; readonly saved: XattrExpectation; readonly volume: NativeComposeRetainedVolume; @@ -1474,21 +1464,17 @@ export async function prepareNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - return await x01Scope("witness-prepare", () => - prepareNativeComposeStorageWitness({ - ...input, - xattrCarrier: carrier, - }) - ); + return await prepareNativeComposeStorageWitness({ + ...input, + xattrCarrier: carrier, + }); } export async function enrollNativeComposeStorageXattrWitness(opts: { readonly enrollment: NativeComposeStorageWitnessEnrollment; }): Promise { - return await x01Scope("witness-enroll", () => - enrollNativeComposeStorageWitness({ - enrollment: opts.enrollment, - }) - ); + return await enrollNativeComposeStorageWitness({ + enrollment: opts.enrollment, + }); } export async function verifyNativeComposeStorageXattrWitness( opts: Omit< @@ -1499,9 +1485,7 @@ export async function verifyNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - await x01Scope("witness-verify", () => - verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }) - ); + await verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }); } /** Private saved observation seam. It never performs a kernel read, repairs an diff --git a/src/lib/shell.ts b/src/lib/shell.ts index ff27e6b85..ef9faf1ba 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -4,7 +4,6 @@ import { } from "./native-cpu-diagnostics.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; -import { x01Label, x01Time } from "./x01-cost-probe.ts"; export interface ExecResult { readonly exitCode: number; @@ -41,13 +40,7 @@ function buildSpawnEnv( return override ? { ...base, ...override } : base; } -export function exec( - cmd: readonly string[], - opts: ExecOptions = {} -): Promise { - return x01Time("exec", x01Label(cmd), () => x01ExecInner(cmd, opts)); -} -async function x01ExecInner( +export async function exec( cmd: readonly string[], opts: ExecOptions = {} ): Promise { @@ -130,13 +123,7 @@ export type RunExitEvent = { readonly maxRssBytes: number | null; }; -export function run( - cmd: readonly string[], - opts: RunOptions = {} -): Promise { - return x01Time("run", x01Label(cmd), () => x01RunInner(cmd, opts)); -} -async function x01RunInner( +export async function run( cmd: readonly string[], opts: RunOptions = {} ): Promise { diff --git a/src/lib/x01-cost-probe.ts b/src/lib/x01-cost-probe.ts deleted file mode 100644 index 17b2f5552..000000000 --- a/src/lib/x01-cost-probe.ts +++ /dev/null @@ -1,176 +0,0 @@ -// TEMPORARY X01 measurement probe. Reverted before the final commit. -// Writes closed labels and durations only (no ids, paths or values). -import { AsyncLocalStorage } from "node:async_hooks"; -import { appendFileSync } from "node:fs"; - -const FILE = process.env.HACK_X01_TRACE_FILE; -const VERBS = new Set([ - "up", - "down", - "exec", - "run", - "restart", - "ps", - "logs", - "status", - "init", -]); -const SUBS = new Set([ - "info", - "image", - "container", - "volume", - "network", - "compose", - "create", - "start", - "rm", - "inspect", - "ls", - "ps", - "exec", - "run", - "logs", - "version", - "events", - "stop", - "kill", - "wait", - "up", - "down", - "config", - "pull", - "build", - "restart", - "port", - "cp", - "top", -]); -const verb = (() => { - for (const arg of process.argv.slice(2)) { - if (VERBS.has(arg)) { - return arg; - } - } - return "other"; -})(); -const scopes = new AsyncLocalStorage(); -let enabled = typeof FILE === "string" && FILE.startsWith("/"); -let wroteStart = false; - -function append(record: Record): void { - appendFileSync( - FILE as string, - `${JSON.stringify({ p: process.pid, v: verb, ...record })}\n` - ); -} - -function write(record: Record): void { - if (!enabled) { - return; - } - try { - if (!wroteStart) { - wroteStart = true; - append({ k: "start", t: Math.round(performance.now()) }); - process.on("exit", (code) => { - try { - append({ k: "exit", c: code, t: Math.round(performance.now()) }); - } catch { - // measurement only - } - }); - } - append({ - s: scopes.getStore() ?? "", - t: Math.round(performance.now()), - ...record, - }); - } catch { - enabled = false; - } -} - -export function x01Enabled(): boolean { - return enabled; -} - -/** Raw line from the phase trace, re-wrapped with process identity. */ -export function x01Phase(line: string): void { - try { - write({ k: "phase", line: JSON.parse(line) as unknown }); - } catch { - enabled = false; - } -} - -export function x01Label(argv: readonly string[], binary = true): string { - const parts: string[] = []; - let rest = argv; - if (binary) { - const name = (argv[0] ?? "").split("/").at(-1) ?? ""; - parts.push( - ["docker", "ps", "sysctl", "sh", "git", "bun", "tmux"].includes(name) - ? name - : "bin" - ); - rest = argv.slice(1); - } else { - parts.push("docker"); - } - for (const token of rest) { - if (parts.length >= 3) { - break; - } - if (token.startsWith("-")) { - continue; - } - if (SUBS.has(token)) { - parts.push(token); - } else { - break; - } - } - return parts.join(" "); -} - -export async function x01Time( - kind: string, - label: string, - operation: () => Promise -): Promise { - if (!enabled) { - return await operation(); - } - const started = performance.now(); - let ok = false; - try { - const result = await operation(); - ok = true; - return result; - } finally { - write({ - k: kind, - l: label, - ms: Math.round((performance.now() - started) * 10) / 10, - ok, - }); - } -} - -/** TEMPORARY X03 measurement: a closed-label point event. */ -export function x01Mark(kind: string, label: string): void { - write({ k: kind, l: label }); -} - -export async function x01Scope( - label: string, - operation: () => Promise -): Promise { - if (!enabled) { - return await operation(); - } - const parent = scopes.getStore(); - const path = parent ? `${parent}>${label}` : label; - return await scopes.run(path, () => x01Time("scope", label, operation)); -} diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index e7599c528..2201457b1 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,8 +43,6 @@ const PASSTHROUGH_ENV_KEYS = [ "LC_ALL", "DOCKER_HOST", "DOCKER_CONFIG", - // TEMPORARY X01 measurement passthrough; reverted before the final commit. - "HACK_X01_TRACE_FILE", ] as const; export type CliResult = { @@ -770,13 +768,6 @@ export async function runScenarios( }; process.stdout.write(`-- ${scenario.name} (${scenario.tier}) --\n`); - // TEMPORARY X01 measurement: one trace file per scenario. - if (process.env.HACK_X01_TRACE_DIR?.startsWith("/")) { - process.env.HACK_X01_TRACE_FILE = join( - process.env.HACK_X01_TRACE_DIR, - `${scenario.name}.jsonl` - ); - } let outcome: ScenarioOutcome; let preserveFailedFixture = false; try { diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index 6c5194702..633cb0449 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -1154,10 +1154,6 @@ export const nativeConfigNetworksScenario: Scenario = { NO_COLOR: "1", HACK_RUNTIME_BACKEND: "compose", HACK_DAEMON_DISABLE_DOCKER_EVENTS: "1", - // TEMPORARY X01 measurement passthrough; reverted before the final commit. - ...(process.env.HACK_X01_TRACE_FILE - ? { HACK_X01_TRACE_FILE: process.env.HACK_X01_TRACE_FILE } - : {}), }; const engine = Bun.which("docker"); const git = Bun.which("git"); From 3edbd546e686a5573f26bd0e4473b28a34d37213 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 04:00:38 -0400 Subject: [PATCH 21/32] feat(agents): package offline skill references and the HACK-P pattern catalogue Plugin skills can now ship references at skills//references/*.md from a typed registry (src/agents/skill-references.ts). The generator writes each reference with a recomputable content revision, removes stale ones, and links them from the plugin SKILL.md. Standalone hack-cli installs ship no references, so their render (and ownership fingerprint) is unchanged. The first reference is the hack-cli pattern catalogue, rendered from the typed register of all twelve HACK-P cards with dispositions, owners, missing contracts and value-free provenance. Tests check completeness, CLI_SPEC command resolution, a banned-recipe scan outside negative-example blocks, and an offline link contract over every packaged skill markdown file. External URLs in hack-install and hack-remote move under "Further reading". docs/agent-guidance.md records the reference contract and the promote/defer decision for the lifecycle, cleanup, checks, sessions, hack-run and hack-feedback skill candidates. Co-Authored-By: Claude Opus 5.5 --- docs/agent-guidance.md | 29 ++ plugins/hack/skills/hack-cli/SKILL.md | 4 + .../skills/hack-cli/references/patterns.md | 217 +++++++++ plugins/hack/skills/hack-install/SKILL.md | 11 +- plugins/hack/skills/hack-remote/SKILL.md | 9 +- scripts/generate-agent-plugins.ts | 71 ++- src/agents/install-skill.ts | 11 +- src/agents/pattern-catalogue.ts | 423 ++++++++++++++++++ src/agents/remote-bootstrap-skill.ts | 9 +- src/agents/skill-references.ts | 98 ++++ tests/agent-plugin-generation.test.ts | 10 +- tests/helpers/skill-links.ts | 172 +++++++ tests/pattern-catalogue.test.ts | 217 +++++++++ tests/skill-references.test.ts | 130 ++++++ 14 files changed, 1398 insertions(+), 13 deletions(-) create mode 100644 plugins/hack/skills/hack-cli/references/patterns.md create mode 100644 src/agents/pattern-catalogue.ts create mode 100644 src/agents/skill-references.ts create mode 100644 tests/helpers/skill-links.ts create mode 100644 tests/pattern-catalogue.test.ts create mode 100644 tests/skill-references.test.ts diff --git a/docs/agent-guidance.md b/docs/agent-guidance.md index 44ae2c1d0..09f8622d0 100644 --- a/docs/agent-guidance.md +++ b/docs/agent-guidance.md @@ -66,3 +66,32 @@ belong to their client adapters; shared behavior belongs to the canonical source preserve customized files. See [integration setup](integrations.md). - Historical Factory validation artifacts describe past runs; they are evidence, not current contributor policy. Do not rewrite history to resemble the current workflow. + +## Packaged references and skill candidates + +Plugin skills can ship markdown references at `skills//references/*.md`. +Register each one in `src/agents/skill-references.ts` (skill, file, render); the +generator writes it with a recomputable content revision and links it from the +plugin's SKILL.md. Standalone `hack setup` installs ship no references, so their +`hack-cli` render stays unlinked. References must resolve inside the installed +package and work offline: relative links stay inside `plugins/hack`, never point +at repository `docs/` or absolute paths, and external URLs appear only under a +"Further reading" heading. `tests/skill-references.test.ts` enforces this. + +The first reference is the `hack-cli` pattern catalogue, rendered from the typed +register in `src/agents/pattern-catalogue.ts`. It keeps all twelve HACK-P cards +(HACK-1145..HACK-1156). A card that is not supported names its missing contract +and domain owner; the disposition does not waive that owner's runtime gates. +`tests/pattern-catalogue.test.ts` checks completeness, CLI_SPEC command +resolution and the banned-recipe scan. + +Skill candidates and where their patterns live now: + +| Candidate | Decision | Patterns live in | Promotion owner and missing contract | +| --- | --- | --- | --- | +| lifecycle | Defer | `hack-cli` "Lifecycle + startup"; catalogue HACK-P01, P04, P05, P06 | WU29: identity-aware adoption and ownership-safe shared host lifecycles | +| cleanup | Defer | `hack-cli` "Lifecycle + startup" (`--prune-caches`); catalogue HACK-P09, P10 | WU12/WU30: reference-aware retention and cache provenance | +| checks | Defer | `hack-cli` "Running things"; catalogue HACK-P02, P10, P11 | WU11: qualified portable toolchain fixture with a distinct trigger | +| sessions | Defer | `hack-cli` "Workspaces" and [sessions](sessions.md) | WU18: trigger evaluations showing `hack-cli` misroutes session work | +| hack-run | Defer | `hack-cli` "Running things" | WU26: decides between a hack-run skill and references | +| hack-feedback | Defer | Not shipped | WU25: privacy-reviewed feedback drafts | diff --git a/plugins/hack/skills/hack-cli/SKILL.md b/plugins/hack/skills/hack-cli/SKILL.md index d9ded7b32..e35765aac 100644 --- a/plugins/hack/skills/hack-cli/SKILL.md +++ b/plugins/hack/skills/hack-cli/SKILL.md @@ -187,3 +187,7 @@ Use `hack` as the primary interface for local-first development. - Init patterns: `hack agent patterns` - MCP (no-shell only): `hack setup mcp` - MCP install: choose the requested client and scope with `hack mcp install --help`; do not configure other clients as a side effect. + +## References + +- [Pattern catalogue](references/patterns.md): reusable project patterns with what Hack supports today, the missing contract and its owner. diff --git a/plugins/hack/skills/hack-cli/references/patterns.md b/plugins/hack/skills/hack-cli/references/patterns.md new file mode 100644 index 000000000..b191f8189 --- /dev/null +++ b/plugins/hack/skills/hack-cli/references/patterns.md @@ -0,0 +1,217 @@ +# Hack pattern catalogue + +Reusable patterns observed across Hack projects, with what Hack supports today. +Use a card's commands only within its disposition. A partial, retained, deferred +or rejected card names the missing contract and its owner; do not improvise the +missing part with ad hoc scripts. + +| Id | Pattern | Disposition | Owner | +| --- | --- | --- | --- | +| HACK-P01 | One-shot dependency and generated-artifact bootstrap | Partial | WU07/WU11 | +| HACK-P02 | Portable toolchain without an always-on application | Partial | WU11 | +| HACK-P03 | Cloud prerequisite preflight | Partial | WU27 | +| HACK-P04 | Shared host service with identity-aware adoption | Retained as example | WU29 | +| HACK-P05 | Branch-owned external routing claim | Deferred | WU29 | +| HACK-P06 | Opt-in public tunnel with end-to-end readiness | Deferred | WU29 | +| HACK-P07 | Laptop and headless-runner capability overlays | Partial | WU27/WU30 | +| HACK-P08 | Tracked defaults plus idempotent machine-local hydration | Deferred | WU27/WU14 | +| HACK-P09 | Disposable run workspaces with reference-aware retention | Partial | WU12/WU30 | +| HACK-P10 | Pinned tool image plus independent dependency caches | Partial | WU30 | +| HACK-P11 | Stable agent guidance and typed operation results | Partial | WU19/WU21 | +| HACK-P12 | Multi-lane admission and durable run provenance | Deferred | WU13/WU04 | + +## HACK-P01: One-shot dependency and generated-artifact bootstrap + +- Use when: Services need installed dependencies or generated artifacts before the application starts. +- Disposition: Partial. Missing: locked-v1 covers one immutable payload per volume. Atomic publication of worktree-specific generated artifacts, env-dependent installers, and two-worktree or interrupted-writer qualification remain open. +- Owner: WU07/WU11; tracked in HACK-1145. +- Commands: `hack up --json`, `hack ps --json`, `hack logs --no-follow --tail 100` +- Prerequisites: A Compose initializer service that consumers require through `service_completed_successfully`; for the opt-in `hack.dependencies.cache-protocol: locked-v1` label, a pinned installer image and an immutable dependency payload. +- Env ownership: locked-v1 runs the initializer and verifier with a clean environment and refuses explicit `environment`/`env_file`; credential- or env-dependent installers keep their existing setup. +- Readiness: Installation, generated-artifact publication and application readiness are separate steps. Completion is the initializer's successful exit plus its verifier, not a populated cache. +- Cleanup: Ordinary `hack down` keeps dependency volumes. Keep worktree-specific generated output in its own always-run step and directory. +- Provenance: 4 consumers (HACK-P01 research register). + + +> Negative control: A populated cache directory is not a completion marker. Concurrent writers to one volume and failed publication need ownership and recovery; consumers must see a complete old or complete new artifact. + + +## HACK-P02: Portable toolchain without an always-on application + +- Use when: Checks, builds or tests need pinned tools in a container, but the project has no long-running app to start. +- Disposition: Partial. Missing: Cross-worktree cache reuse with bounded eviction, a source receipt for linked worktrees, and owned cache accounting remain open. +- Owner: WU11; tracked in HACK-1146. +- Commands: `hack run --profile -- ` +- Prerequisites: A one-shot Compose service behind an opt-in profile with pinned tool versions, explicit resource bounds and separate named volumes for reusable build caches. +- Env ownership: Mount no Docker socket, user home, SSH agent or credential directory; keep project secrets out of the image build context. +- Readiness: The task's own exit status is the result. Check the `hack run` exit status directly, not through a pipe filter, so a failed check stays a failure. +- Cleanup: The service is one-shot. Caches live in named volumes that `hack down` preserves; a source-identity override used for tests must not reach release evidence. +- Provenance: 4 consumers (HACK-P02 research register). + + +> Negative control: A test-only source identity override must not escape into release evidence, and a linked worktree's Git pointer needs deliberate handling rather than an assumed `.git` directory. + + +## HACK-P03: Cloud prerequisite preflight + +- Use when: Startup depends on cloud credentials or tokens that may be missing, expired or scoped wrongly. +- Disposition: Partial. Missing: Hack has no credential availability, expiry or scope check and no machine-identity path for headless runners; the WU27 env contract owns both. +- Owner: WU27; tracked in HACK-1147. +- Commands: `hack doctor --json`, `hack env explain --json`, `hack env explain --target compose --json` +- Prerequisites: Required key names declared in Hack env; the credential tool and its native sign-in owned by the user. +- Env ownership: Report key names, winning source and repair direction, never values. Check the container view separately from the host view. +- Readiness: Preflight runs before startup and fails with the missing name; a present value is not proof that it is valid or unexpired. +- Cleanup: Preflight starts nothing and leaves no state. +- Provenance: 3 consumers (HACK-P03 research register). + + +> Negative control: A token on the host does not prove the container receives it, and a headless runner cannot depend on interactive browser sign-in. + + +## HACK-P04: Shared host service with identity-aware adoption + +- Use when: Several projects or branches share a host helper such as a session issuer or local proxy. +- Disposition: Retained as example. Missing: Today's adoption is port-only. Health- and upstream-checked adoption with a recorded adopted-versus-owned state is the WU29 lifecycle ownership contract. +- Owner: WU29; tracked in HACK-1148. +- Commands: `hack up --json`, `hack logs --no-follow --tail 100` +- Prerequisites: A lifecycle process with `singleton` ports. Hack adopts an existing helper only when every configured port is occupied and the config opts into `onConflict: "adopt"`. +- Env ownership: The helper's credentials stay with its owner; adoption must not forward broad host env. +- Readiness: Pair adoption with a `lifecycle.up.before` hook that checks the listener's identity and expected upstream and fails on mismatch. +- Cleanup: Hack stops only processes it started; an adopted helper is not Hack-owned and is never stopped by Hack. +- Provenance: single-consumer generalization (HACK-P04 research register). + + +> Negative control: Port-only reuse can attach to a foreign service. Process liveness or a placeholder process is not readiness, and no foreign process may be killed to free a port. + + +## HACK-P05: Branch-owned external routing claim + +- Use when: A branch needs an external route or callback claim tied to its own instance. +- Disposition: Deferred. Missing: Hack has no branch-scoped external claim/release contract with bounded retry, stop-during-retry and PID-reuse safety. +- Owner: WU29; tracked in HACK-1149. +- Commands: none yet; do not substitute a hand-written recipe. +- Prerequisites: Explicit project and branch identity for the claim; no recipe is published until the contract exists. +- Env ownership: Claim credentials stay host-scoped and are not written into tracked files. +- Readiness: A claim must be a bounded, observable operation with a retry budget. +- Cleanup: Release only the claim and process group that this branch owns. +- Provenance: single-consumer generalization (HACK-P05 research register). + + +> Negative control: The observed consumer uses a fixed default URL, unbounded retry and broad name-based cleanup. Retain it as a negative example, not a copyable recipe. + + +## HACK-P06: Opt-in public tunnel with end-to-end readiness + +- Use when: A developer occasionally needs the local application reachable from the public internet. +- Disposition: Deferred. Missing: Hack has no opt-in tunnel lifecycle with external-route readiness and owned-child shutdown. +- Owner: WU29; tracked in HACK-1150. +- Commands: none yet; do not substitute a hand-written recipe. +- Prerequisites: Default off; host-scoped tunnel credentials, the tunnel tool and the intended CA. +- Env ownership: Pass only the tunnel's own credential; never forward broad host env. +- Readiness: Probe the exposed application route end to end; a missing credential, tool or CA fails clearly. +- Cleanup: Shutdown leaves no owned tunnel child; disabled mode starts no tunnel. +- Provenance: single-consumer generalization (HACK-P06 research register). + + +> Negative control: A running tunnel process or open TCP listener alone does not prove the external route reaches the right upstream. + + +## HACK-P07: Laptop and headless-runner capability overlays + +- Use when: The same application topology runs on a laptop and on a headless runner with different auth, network or helpers. +- Disposition: Partial. Missing: Hack has env overlays and value-free inspection but no laptop/headless capability contract or headless machine identity; WU27 and WU30 own them. +- Owner: WU27/WU30; tracked in HACK-1151. +- Commands: `hack env explain --env --json`, `hack env explain --env --target compose --json`, `hack up --env --json` +- Prerequisites: One shared Compose topology; env overlays that express only the auth, network and helper differences. +- Env ownership: Overlays own the differences; inspect the effective value source per target instead of reading host env. +- Readiness: Verify container metadata and dependency reachability on each target; the runner must not need browser sign-in. +- Cleanup: Overlays add no state of their own; inherited worktree overrides must be visible and deliberate. +- Provenance: single-consumer generalization (HACK-P07 research register). + + +> Negative control: Do not change locality flags just to bypass a readiness check. Host env is not evidence of container delivery. + + +## HACK-P08: Tracked defaults plus idempotent machine-local hydration + +- Use when: A repository tracks env defaults and each machine adds local values on top. +- Disposition: Deferred. Missing: Hack has no semantic no-op detection or atomic machine-local import contract for env hydration. +- Owner: WU27/WU14; tracked in HACK-1152. +- Commands: none yet; do not substitute a hand-written recipe. +- Prerequisites: Tracked defaults, a machine-local layer and preserved operator overrides. +- Env ownership: Defaults are tracked; hydration writes only the local layer and keeps values out of argv. +- Readiness: Repeat hydration leaves the tracked tree clean; an interrupted import still leaves a resolvable layer. +- Cleanup: Unchanged values cause no rewrite. +- Provenance: single-consumer generalization (HACK-P08 research register). + + +> Negative control: Re-encrypting unchanged entries causes diff and stash churn, and values passed in command arguments are not a secure import mechanism. + + +## HACK-P09: Disposable run workspaces with reference-aware retention + +- Use when: Runs or branches create workspaces and caches that should be reclaimed once nothing uses them. +- Disposition: Partial. Missing: Generation/run ownership, active references, holds and interrupted-reclamation recovery are not implemented for run workspaces. +- Owner: WU12/WU30; tracked in HACK-1153. +- Commands: `hack down --prune-caches`, `hack branch remove `, `hack projects prune --dry-run` +- Prerequisites: Explicitly disposable cache volumes (`hack.cache.disposable: "true"` or Next.js build output) and a reviewed dry run before pruning. +- Env ownership: Reclamation reads no env values. +- Readiness: Not applicable; retention is a cleanup contract. +- Cleanup: `--prune-caches` is confirmation-gated and removes only exact, project-owned disposable volumes; sessions and application data keep their own retention. +- Provenance: single-consumer generalization (HACK-P09 research register). + + +> Negative control: Stopped or Docker-reclaimable does not mean disposable. Active, pending, held or unknown resources must survive. + + +## HACK-P10: Pinned tool image plus independent dependency caches + +- Use when: A runner or toolchain image must stay pinned while application dependencies and credentials vary. +- Disposition: Partial. Missing: Hack does not yet distinguish L1, L2 and cold misses, detect a tampered cache, or emit stage timing and provenance receipts. +- Owner: WU30; tracked in HACK-1154. +- Commands: `hack run --profile -- `, `hack up --json` +- Prerequisites: Tool manifests pinned in the image; application dependencies in separate cache volumes; credentials outside both. +- Env ownership: Credentials are injected at run time and never baked into the tool image or caches. +- Readiness: Each run reports which stage ran; a changed lockfile or tool pin rebuilds through the same command. +- Cleanup: Tool images and dependency caches are retained; prune only explicitly disposable caches. +- Provenance: 3 consumers (HACK-P10 research register). + + +> Negative control: Code that has landed is not an enabled, verified deployment, and a cache hit does not prove the installer was skipped. + + +## HACK-P11: Stable agent guidance and typed operation results + +- Use when: Agents run project workflows and need versioned guidance plus machine-readable results. +- Disposition: Partial. Missing: A guidance freeze during execution and a typed terminal result for resumed background operations remain open. +- Owner: WU19/WU21; tracked in HACK-1155. +- Commands: `hack up --json`, `hack down --json`, `hack restart --json`, `hack doctor --json`, `hack run -- ` +- Prerequisites: Generated guidance from Hack's canonical source with a content revision; `--json` on lifecycle and diagnostic commands. +- Env ownership: Results report names and states, not values. +- Readiness: Pass child argv after `--` in `hack run -- `. Check the exit status or `--json` envelope directly rather than through a pipe filter. +- Cleanup: Guidance updates are explicit `hack setup` operations, never side effects of ordinary commands. +- Provenance: 2 consumers (HACK-P11 research register). + + +> Negative control: Filtered shell text can hide a failed child, and historical dispatch, PR or gateway requests must not silently expand current product scope. + + +## HACK-P12: Multi-lane admission and durable run provenance + +- Use when: Several heavy runs compete for scarce resources and need queued admission with durable ownership. +- Disposition: Deferred. Missing: Hack has no multi-lane admission, lease or durable run-provenance contract. +- Owner: WU13/WU04; tracked in HACK-1156. +- Commands: none yet; do not substitute a hand-written recipe. +- Prerequisites: Separate job admission, scarce-resource leases and project-registry operations. +- Env ownership: Admission and provenance records hold no env values. +- Readiness: Report queue time separately from execution time with the exact workload. +- Cleanup: Cancellation and worker restart must not duplicate ownership or lose a continuation. +- Provenance: 2 consumers (HACK-P12 research register). + + +> Negative control: A global registry lock is not a workload scheduler, and idle host, idle lane and idle application are different signals. + + +--- + +Content revision: `091b4e80a9b6`. Generated by Hack; edit the canonical source and regenerate the plugin. diff --git a/plugins/hack/skills/hack-install/SKILL.md b/plugins/hack/skills/hack-install/SKILL.md index c9da3dcd6..02f401b77 100644 --- a/plugins/hack/skills/hack-install/SKILL.md +++ b/plugins/hack/skills/hack-install/SKILL.md @@ -25,8 +25,8 @@ On a Homebrew host, use the official formula `hack-dance/tap/hack`. Inspect `brew install hack-dance/tap/hack` or `brew upgrade hack-dance/tap/hack` as appropriate. Run Homebrew as the user, never with sudo. -For another supported host, read the installation instructions at -https://github.com/hack-dance/hack and inspect the official release's platform +For another supported host, read the installation instructions in the Hack +repository (see Further reading) and inspect the official release's platform assets. Pin a release for reproducible setup, save and review its installer, and follow its documented verification. Do not guess asset names or run an unreviewed download through a shell. Managed containers use the slim path in @@ -34,7 +34,7 @@ unreviewed download through a shell. Managed containers use the slim path in ## Native prerelease alongside stable -Read https://github.com/hack-dance/hack/blob/next/docs/guides/candidate-install.md +Read the candidate installation guide (see Further reading) and verify the selected release exists. The current packaged native installer supports Apple Silicon macOS, not arbitrary Linux cloud hosts. Use an explicit published `5.0.0-next.N` tag and its reviewed `scripts/install-prerelease.py`. @@ -79,3 +79,8 @@ Report the executable, version, channel and target. Inspect project status and an actual app request, and an appropriate log check. A successful install or MCP listing alone does not prove app readiness. Keep local HTTPS trust prompts native; browser access is a separate check from a successful CLI HTTPS request. + +## Further reading + +- [Hack repository and releases](https://github.com/hack-dance/hack) +- [Candidate installation guide](https://github.com/hack-dance/hack/blob/next/docs/guides/candidate-install.md) diff --git a/plugins/hack/skills/hack-remote/SKILL.md b/plugins/hack/skills/hack-remote/SKILL.md index 2f8dd3dab..531d4d0bd 100644 --- a/plugins/hack/skills/hack-remote/SKILL.md +++ b/plugins/hack/skills/hack-remote/SKILL.md @@ -30,9 +30,9 @@ Use `hack_cli` and `project_root` below as local shell variables containing thos Request `HACK_ENV_SECRET_KEY` through the target's approved secret manager or direct process injection. The user enters the value in its native secret UI, never in chat. Hack needs the real key bytes locally to decrypt the committed env; an HTTPS proxy placeholder cannot replace them. Do not print the key, pass it as a command argument, write it into a plugin, image, shell profile, or cached file, or export decrypted values for later reuse. Use non-production credentials and review the chosen command's own logging. -Current Codex Cloud uses an Install script and Start skill. Request the key as a direct environment variable, preferably a personal value scoped to the chosen environment; it is available to the task. Network secrets substitute HTTPS credentials and cannot decrypt Hack env. Keep prepared files free of keys and plaintext secrets. After approved publication, verify a fresh task. See [current Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environments). +Current Codex Cloud uses an Install script and Start skill. Request the key as a direct environment variable, preferably a personal value scoped to the chosen environment; it is available to the task. Network secrets substitute HTTPS credentials and cannot decrypt Hack env. Keep prepared files free of keys and plaintext secrets. After approved publication, verify a fresh task. See current Cloud environments under Further reading. -Legacy Codex Cloud secrets are setup-only and removed before the agent phase; a setup export does not persist. Do not bypass that boundary by saving the key or decrypted env in cached files. Runtime decryption needs a separately authorized injection mechanism, or the task must use a secret-free workflow. See [legacy Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environment). +Legacy Codex Cloud secrets are setup-only and removed before the agent phase; a setup export does not persist. Do not bypass that boundary by saving the key or decrypted env in cached files. Runtime decryption needs a separately authorized injection mechanism, or the task must use a secret-free workflow. See legacy Cloud environments under Further reading. Cloud tasks must have this skill in their repository or supported cloud environment; a personal plugin installed on a local computer is not proof it is loaded remotely. Confirm the available skills in the target task. @@ -55,3 +55,8 @@ When the selected runtime is ready and the project's lifecycle hooks have been r ``` Check an actual application endpoint or repository health command with a bounded timeout, then perform one meaningful development operation. Follow the project's approved network access and authentication flow; do not bypass provider restrictions. Report the CLI version/path, target, workflow, env overlay name, readiness result, and remaining blocker without secrets. Installation, a listed MCP server, or a running container alone does not prove the app works. Stop only resources created and owned by this task when cleanup is requested; preserve reusable caches and application data. + +## Further reading + +- [Current Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environments) +- [Legacy Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environment) diff --git a/scripts/generate-agent-plugins.ts b/scripts/generate-agent-plugins.ts index a0fa434ef..1a8e85205 100644 --- a/scripts/generate-agent-plugins.ts +++ b/scripts/generate-agent-plugins.ts @@ -1,3 +1,4 @@ +import { readdir, rm } from "node:fs/promises"; import { dirname, resolve } from "node:path"; import { renderCodexSkill } from "../src/agents/codex-skill.ts"; @@ -5,6 +6,13 @@ import { renderCursorRules } from "../src/agents/cursor.ts"; import { renderHackInitSkill } from "../src/agents/hack-init-skill.ts"; import { renderHackInstallSkill } from "../src/agents/install-skill.ts"; import { renderHackRemoteSkill } from "../src/agents/remote-bootstrap-skill.ts"; +import { + type PluginSkillName, + referencesForSkill, + renderSkillReference, + skillReferencePath, + withSkillReferenceLinks, +} from "../src/agents/skill-references.ts"; import { isRecord } from "../src/lib/guards.ts"; export async function generateAgentPlugins({ @@ -19,7 +27,7 @@ export async function generateAgentPlugins({ const generatedSkills = [ { path: resolve(pluginRoot, "skills", "hack-cli", "SKILL.md"), - content: renderCodexSkill(), + content: renderPluginSkill("hack-cli"), }, { path: resolve(pluginRoot, "skills", "hack-init", "SKILL.md"), @@ -43,6 +51,7 @@ export async function generateAgentPlugins({ await Bun.$`mkdir -p ${dirname(skill.path)}`; await Bun.write(skill.path, skill.content); } + await writeSkillReferences({ pluginRoot }); const manifestPaths = [ resolve(pluginRoot, ".codex-plugin", "plugin.json"), @@ -137,6 +146,66 @@ export function portableMcpManifest( }; } +/** + * Render a plugin skill's SKILL.md. The plugin variant of `hack-cli` adds + * links to its packaged references; standalone installs keep + * {@link renderCodexSkill} because they ship no references. + */ +export function renderPluginSkill(skill: "hack-cli"): string { + return withSkillReferenceLinks({ skill, content: renderCodexSkill() }); +} + +const PLUGIN_SKILLS: readonly PluginSkillName[] = [ + "hack-cli", + "hack-init", + "hack-install", + "hack-remote", +]; + +/** Every packaged skill reference keyed by its plugin-relative path. */ +export function renderPluginReferences(): ReadonlyMap { + return new Map( + PLUGIN_SKILLS.flatMap((skill) => + referencesForSkill(skill).map( + (reference) => + [ + skillReferencePath(reference), + renderSkillReference(reference), + ] as const + ) + ) + ); +} + +/** + * Write every registered reference and remove stale generated ones, so each + * bundled `references/` directory holds exactly the registry's markdown files. + */ +async function writeSkillReferences({ + pluginRoot, +}: { + readonly pluginRoot: string; +}): Promise { + const references = renderPluginReferences(); + for (const skill of PLUGIN_SKILLS) { + const directory = resolve(pluginRoot, "skills", skill, "references"); + const existing = await readdir(directory).catch((): string[] => []); + for (const name of existing) { + if ( + name.endsWith(".md") && + !references.has(`skills/${skill}/references/${name}`) + ) { + await rm(resolve(directory, name)); + } + } + } + for (const [relativePath, content] of references) { + const path = resolve(pluginRoot, relativePath); + await Bun.$`mkdir -p ${dirname(path)}`; + await Bun.write(path, content); + } +} + export function renderPluginRule(): string { return [ "---", diff --git a/src/agents/install-skill.ts b/src/agents/install-skill.ts index 9717ba9f6..0ea2c0977 100644 --- a/src/agents/install-skill.ts +++ b/src/agents/install-skill.ts @@ -27,8 +27,8 @@ On a Homebrew host, use the official formula \`hack-dance/tap/hack\`. Inspect \`brew install hack-dance/tap/hack\` or \`brew upgrade hack-dance/tap/hack\` as appropriate. Run Homebrew as the user, never with sudo. -For another supported host, read the installation instructions at -https://github.com/hack-dance/hack and inspect the official release's platform +For another supported host, read the installation instructions in the Hack +repository (see Further reading) and inspect the official release's platform assets. Pin a release for reproducible setup, save and review its installer, and follow its documented verification. Do not guess asset names or run an unreviewed download through a shell. Managed containers use the slim path in @@ -36,7 +36,7 @@ unreviewed download through a shell. Managed containers use the slim path in ## Native prerelease alongside stable -Read https://github.com/hack-dance/hack/blob/next/docs/guides/candidate-install.md +Read the candidate installation guide (see Further reading) and verify the selected release exists. The current packaged native installer supports Apple Silicon macOS, not arbitrary Linux cloud hosts. Use an explicit published \`5.0.0-next.N\` tag and its reviewed \`scripts/install-prerelease.py\`. @@ -81,5 +81,10 @@ Report the executable, version, channel and target. Inspect project status and an actual app request, and an appropriate log check. A successful install or MCP listing alone does not prove app readiness. Keep local HTTPS trust prompts native; browser access is a separate check from a successful CLI HTTPS request. + +## Further reading + +- [Hack repository and releases](https://github.com/hack-dance/hack) +- [Candidate installation guide](https://github.com/hack-dance/hack/blob/next/docs/guides/candidate-install.md) `; } diff --git a/src/agents/pattern-catalogue.ts b/src/agents/pattern-catalogue.ts new file mode 100644 index 000000000..82e365656 --- /dev/null +++ b/src/agents/pattern-catalogue.ts @@ -0,0 +1,423 @@ +/** + * Typed register of the twelve reusable project patterns (HACK-P01..P12) + * found across Hack consumers, with each card's current disposition. + * + * Invariants (enforced by tests/pattern-catalogue.test.ts): + * - All twelve ids are present exactly once; the register stays complete even + * for recipes Hack cannot support yet. + * - A card that is not `supported` names its domain owner and the missing + * contract. Such a disposition never waives the owner's runtime gates. + * - Every `hack` invocation a card names resolves in CLI_SPEC on this branch. + * - Unsafe recipes (broad process kills, kill-by-port, placeholder liveness, + * secrets in argv) appear only inside negative-example blocks. + * + * Consumer names stay in the Linear issues; the shipped reference records + * only how many consumers a card generalizes from. + */ + +/** How far Hack supports a pattern today. */ +export type PatternDisposition = + | "supported" + | "partial" + | "retain-as-example" + | "deferred" + | "rejected"; + +/** Where a card's generalization comes from. */ +export type PatternProvenance = { + /** Number of distinct consumer projects that exhibit the pattern. */ + readonly consumers: number; + /** Research register entry; consumer detail lives in the Linear issue. */ + readonly source: string; +}; + +export type PatternCard = { + /** Stable research id, `HACK-P01`..`HACK-P12`. */ + readonly id: string; + /** Linear issue that tracks the card. */ + readonly linearId: string; + readonly title: string; + readonly useWhen: string; + readonly disposition: PatternDisposition; + /** + * `hack` invocations that exist on this CLI and apply to the pattern. For a + * card that is not `supported`, these cover only the part Hack owns today. + */ + readonly supportedCommands: readonly string[]; + readonly prerequisites: string; + readonly envOwnership: string; + readonly readiness: string; + readonly cleanup: string; + /** What looks like success but is not; rendered as a negative example. */ + readonly negativeControl: string; + /** Work unit(s) that own the runtime mechanism. */ + readonly domainOwner: string; + /** Contract Hack still lacks; `null` only for `supported` cards. */ + readonly missingContract: string | null; + readonly provenance: PatternProvenance; +}; + +/** The complete pattern register, in id order. */ +export const PATTERN_CATALOGUE: readonly PatternCard[] = [ + { + id: "HACK-P01", + linearId: "HACK-1145", + title: "One-shot dependency and generated-artifact bootstrap", + useWhen: + "Services need installed dependencies or generated artifacts before the application starts.", + disposition: "partial", + supportedCommands: [ + "hack up --json", + "hack ps --json", + "hack logs --no-follow --tail 100", + ], + prerequisites: + "A Compose initializer service that consumers require through `service_completed_successfully`; for the opt-in `hack.dependencies.cache-protocol: locked-v1` label, a pinned installer image and an immutable dependency payload.", + envOwnership: + "locked-v1 runs the initializer and verifier with a clean environment and refuses explicit `environment`/`env_file`; credential- or env-dependent installers keep their existing setup.", + readiness: + "Installation, generated-artifact publication and application readiness are separate steps. Completion is the initializer's successful exit plus its verifier, not a populated cache.", + cleanup: + "Ordinary `hack down` keeps dependency volumes. Keep worktree-specific generated output in its own always-run step and directory.", + negativeControl: + "A populated cache directory is not a completion marker. Concurrent writers to one volume and failed publication need ownership and recovery; consumers must see a complete old or complete new artifact.", + domainOwner: "WU07/WU11", + missingContract: + "locked-v1 covers one immutable payload per volume. Atomic publication of worktree-specific generated artifacts, env-dependent installers, and two-worktree or interrupted-writer qualification remain open.", + provenance: { consumers: 4, source: "HACK-P01 research register" }, + }, + { + id: "HACK-P02", + linearId: "HACK-1146", + title: "Portable toolchain without an always-on application", + useWhen: + "Checks, builds or tests need pinned tools in a container, but the project has no long-running app to start.", + disposition: "partial", + supportedCommands: ["hack run --profile -- "], + prerequisites: + "A one-shot Compose service behind an opt-in profile with pinned tool versions, explicit resource bounds and separate named volumes for reusable build caches.", + envOwnership: + "Mount no Docker socket, user home, SSH agent or credential directory; keep project secrets out of the image build context.", + readiness: + "The task's own exit status is the result. Check the `hack run` exit status directly, not through a pipe filter, so a failed check stays a failure.", + cleanup: + "The service is one-shot. Caches live in named volumes that `hack down` preserves; a source-identity override used for tests must not reach release evidence.", + negativeControl: + "A test-only source identity override must not escape into release evidence, and a linked worktree's Git pointer needs deliberate handling rather than an assumed `.git` directory.", + domainOwner: "WU11", + missingContract: + "Cross-worktree cache reuse with bounded eviction, a source receipt for linked worktrees, and owned cache accounting remain open.", + provenance: { consumers: 4, source: "HACK-P02 research register" }, + }, + { + id: "HACK-P03", + linearId: "HACK-1147", + title: "Cloud prerequisite preflight", + useWhen: + "Startup depends on cloud credentials or tokens that may be missing, expired or scoped wrongly.", + disposition: "partial", + supportedCommands: [ + "hack doctor --json", + "hack env explain --json", + "hack env explain --target compose --json", + ], + prerequisites: + "Required key names declared in Hack env; the credential tool and its native sign-in owned by the user.", + envOwnership: + "Report key names, winning source and repair direction, never values. Check the container view separately from the host view.", + readiness: + "Preflight runs before startup and fails with the missing name; a present value is not proof that it is valid or unexpired.", + cleanup: "Preflight starts nothing and leaves no state.", + negativeControl: + "A token on the host does not prove the container receives it, and a headless runner cannot depend on interactive browser sign-in.", + domainOwner: "WU27", + missingContract: + "Hack has no credential availability, expiry or scope check and no machine-identity path for headless runners; the WU27 env contract owns both.", + provenance: { consumers: 3, source: "HACK-P03 research register" }, + }, + { + id: "HACK-P04", + linearId: "HACK-1148", + title: "Shared host service with identity-aware adoption", + useWhen: + "Several projects or branches share a host helper such as a session issuer or local proxy.", + disposition: "retain-as-example", + supportedCommands: ["hack up --json", "hack logs --no-follow --tail 100"], + prerequisites: + 'A lifecycle process with `singleton` ports. Hack adopts an existing helper only when every configured port is occupied and the config opts into `onConflict: "adopt"`.', + envOwnership: + "The helper's credentials stay with its owner; adoption must not forward broad host env.", + readiness: + "Pair adoption with a `lifecycle.up.before` hook that checks the listener's identity and expected upstream and fails on mismatch.", + cleanup: + "Hack stops only processes it started; an adopted helper is not Hack-owned and is never stopped by Hack.", + negativeControl: + "Port-only reuse can attach to a foreign service. Process liveness or a placeholder process is not readiness, and no foreign process may be killed to free a port.", + domainOwner: "WU29", + missingContract: + "Today's adoption is port-only. Health- and upstream-checked adoption with a recorded adopted-versus-owned state is the WU29 lifecycle ownership contract.", + provenance: { consumers: 1, source: "HACK-P04 research register" }, + }, + { + id: "HACK-P05", + linearId: "HACK-1149", + title: "Branch-owned external routing claim", + useWhen: + "A branch needs an external route or callback claim tied to its own instance.", + disposition: "deferred", + supportedCommands: [], + prerequisites: + "Explicit project and branch identity for the claim; no recipe is published until the contract exists.", + envOwnership: + "Claim credentials stay host-scoped and are not written into tracked files.", + readiness: + "A claim must be a bounded, observable operation with a retry budget.", + cleanup: "Release only the claim and process group that this branch owns.", + negativeControl: + "The observed consumer uses a fixed default URL, unbounded retry and broad name-based cleanup. Retain it as a negative example, not a copyable recipe.", + domainOwner: "WU29", + missingContract: + "Hack has no branch-scoped external claim/release contract with bounded retry, stop-during-retry and PID-reuse safety.", + provenance: { consumers: 1, source: "HACK-P05 research register" }, + }, + { + id: "HACK-P06", + linearId: "HACK-1150", + title: "Opt-in public tunnel with end-to-end readiness", + useWhen: + "A developer occasionally needs the local application reachable from the public internet.", + disposition: "deferred", + supportedCommands: [], + prerequisites: + "Default off; host-scoped tunnel credentials, the tunnel tool and the intended CA.", + envOwnership: + "Pass only the tunnel's own credential; never forward broad host env.", + readiness: + "Probe the exposed application route end to end; a missing credential, tool or CA fails clearly.", + cleanup: + "Shutdown leaves no owned tunnel child; disabled mode starts no tunnel.", + negativeControl: + "A running tunnel process or open TCP listener alone does not prove the external route reaches the right upstream.", + domainOwner: "WU29", + missingContract: + "Hack has no opt-in tunnel lifecycle with external-route readiness and owned-child shutdown.", + provenance: { consumers: 1, source: "HACK-P06 research register" }, + }, + { + id: "HACK-P07", + linearId: "HACK-1151", + title: "Laptop and headless-runner capability overlays", + useWhen: + "The same application topology runs on a laptop and on a headless runner with different auth, network or helpers.", + disposition: "partial", + supportedCommands: [ + "hack env explain --env --json", + "hack env explain --env --target compose --json", + "hack up --env --json", + ], + prerequisites: + "One shared Compose topology; env overlays that express only the auth, network and helper differences.", + envOwnership: + "Overlays own the differences; inspect the effective value source per target instead of reading host env.", + readiness: + "Verify container metadata and dependency reachability on each target; the runner must not need browser sign-in.", + cleanup: + "Overlays add no state of their own; inherited worktree overrides must be visible and deliberate.", + negativeControl: + "Do not change locality flags just to bypass a readiness check. Host env is not evidence of container delivery.", + domainOwner: "WU27/WU30", + missingContract: + "Hack has env overlays and value-free inspection but no laptop/headless capability contract or headless machine identity; WU27 and WU30 own them.", + provenance: { consumers: 1, source: "HACK-P07 research register" }, + }, + { + id: "HACK-P08", + linearId: "HACK-1152", + title: "Tracked defaults plus idempotent machine-local hydration", + useWhen: + "A repository tracks env defaults and each machine adds local values on top.", + disposition: "deferred", + supportedCommands: [], + prerequisites: + "Tracked defaults, a machine-local layer and preserved operator overrides.", + envOwnership: + "Defaults are tracked; hydration writes only the local layer and keeps values out of argv.", + readiness: + "Repeat hydration leaves the tracked tree clean; an interrupted import still leaves a resolvable layer.", + cleanup: "Unchanged values cause no rewrite.", + negativeControl: + "Re-encrypting unchanged entries causes diff and stash churn, and values passed in command arguments are not a secure import mechanism.", + domainOwner: "WU27/WU14", + missingContract: + "Hack has no semantic no-op detection or atomic machine-local import contract for env hydration.", + provenance: { consumers: 1, source: "HACK-P08 research register" }, + }, + { + id: "HACK-P09", + linearId: "HACK-1153", + title: "Disposable run workspaces with reference-aware retention", + useWhen: + "Runs or branches create workspaces and caches that should be reclaimed once nothing uses them.", + disposition: "partial", + supportedCommands: [ + "hack down --prune-caches", + "hack branch remove ", + "hack projects prune --dry-run", + ], + prerequisites: + 'Explicitly disposable cache volumes (`hack.cache.disposable: "true"` or Next.js build output) and a reviewed dry run before pruning.', + envOwnership: "Reclamation reads no env values.", + readiness: "Not applicable; retention is a cleanup contract.", + cleanup: + "`--prune-caches` is confirmation-gated and removes only exact, project-owned disposable volumes; sessions and application data keep their own retention.", + negativeControl: + "Stopped or Docker-reclaimable does not mean disposable. Active, pending, held or unknown resources must survive.", + domainOwner: "WU12/WU30", + missingContract: + "Generation/run ownership, active references, holds and interrupted-reclamation recovery are not implemented for run workspaces.", + provenance: { consumers: 1, source: "HACK-P09 research register" }, + }, + { + id: "HACK-P10", + linearId: "HACK-1154", + title: "Pinned tool image plus independent dependency caches", + useWhen: + "A runner or toolchain image must stay pinned while application dependencies and credentials vary.", + disposition: "partial", + supportedCommands: [ + "hack run --profile -- ", + "hack up --json", + ], + prerequisites: + "Tool manifests pinned in the image; application dependencies in separate cache volumes; credentials outside both.", + envOwnership: + "Credentials are injected at run time and never baked into the tool image or caches.", + readiness: + "Each run reports which stage ran; a changed lockfile or tool pin rebuilds through the same command.", + cleanup: + "Tool images and dependency caches are retained; prune only explicitly disposable caches.", + negativeControl: + "Code that has landed is not an enabled, verified deployment, and a cache hit does not prove the installer was skipped.", + domainOwner: "WU30", + missingContract: + "Hack does not yet distinguish L1, L2 and cold misses, detect a tampered cache, or emit stage timing and provenance receipts.", + provenance: { consumers: 3, source: "HACK-P10 research register" }, + }, + { + id: "HACK-P11", + linearId: "HACK-1155", + title: "Stable agent guidance and typed operation results", + useWhen: + "Agents run project workflows and need versioned guidance plus machine-readable results.", + disposition: "partial", + supportedCommands: [ + "hack up --json", + "hack down --json", + "hack restart --json", + "hack doctor --json", + "hack run -- ", + ], + prerequisites: + "Generated guidance from Hack's canonical source with a content revision; `--json` on lifecycle and diagnostic commands.", + envOwnership: "Results report names and states, not values.", + readiness: + "Pass child argv after `--` in `hack run -- `. Check the exit status or `--json` envelope directly rather than through a pipe filter.", + cleanup: + "Guidance updates are explicit `hack setup` operations, never side effects of ordinary commands.", + negativeControl: + "Filtered shell text can hide a failed child, and historical dispatch, PR or gateway requests must not silently expand current product scope.", + domainOwner: "WU19/WU21", + missingContract: + "A guidance freeze during execution and a typed terminal result for resumed background operations remain open.", + provenance: { consumers: 2, source: "HACK-P11 research register" }, + }, + { + id: "HACK-P12", + linearId: "HACK-1156", + title: "Multi-lane admission and durable run provenance", + useWhen: + "Several heavy runs compete for scarce resources and need queued admission with durable ownership.", + disposition: "deferred", + supportedCommands: [], + prerequisites: + "Separate job admission, scarce-resource leases and project-registry operations.", + envOwnership: "Admission and provenance records hold no env values.", + readiness: + "Report queue time separately from execution time with the exact workload.", + cleanup: + "Cancellation and worker restart must not duplicate ownership or lose a continuation.", + negativeControl: + "A global registry lock is not a workload scheduler, and idle host, idle lane and idle application are different signals.", + domainOwner: "WU13/WU04", + missingContract: + "Hack has no multi-lane admission, lease or durable run-provenance contract.", + provenance: { consumers: 2, source: "HACK-P12 research register" }, + }, +]; + +/** Markers that delimit negative examples in the rendered catalogue. */ +export const NEGATIVE_EXAMPLE_START = ""; +export const NEGATIVE_EXAMPLE_END = ""; + +const DISPOSITION_LABELS: Readonly> = { + supported: "Supported", + partial: "Partial", + "retain-as-example": "Retained as example", + deferred: "Deferred", + rejected: "Rejected", +}; + +/** Render the catalogue as the packaged `hack-cli` reference. */ +export function renderPatternCatalogue(): string { + const lines = [ + "# Hack pattern catalogue", + "", + "Reusable patterns observed across Hack projects, with what Hack supports today.", + "Use a card's commands only within its disposition. A partial, retained, deferred", + "or rejected card names the missing contract and its owner; do not improvise the", + "missing part with ad hoc scripts.", + "", + "| Id | Pattern | Disposition | Owner |", + "| --- | --- | --- | --- |", + ...PATTERN_CATALOGUE.map( + (card) => + `| ${card.id} | ${card.title} | ${DISPOSITION_LABELS[card.disposition]} | ${card.domainOwner} |` + ), + "", + ...PATTERN_CATALOGUE.flatMap((card) => renderCard(card)), + ]; + return `${lines.join("\n").trimEnd()}\n`; +} + +function renderCard(card: PatternCard): string[] { + const commands = + card.supportedCommands.length === 0 + ? "none yet; do not substitute a hand-written recipe." + : card.supportedCommands.map((command) => `\`${command}\``).join(", "); + const consumers = + card.provenance.consumers === 1 + ? "single-consumer generalization" + : `${card.provenance.consumers} consumers`; + const disposition = + card.missingContract === null + ? DISPOSITION_LABELS[card.disposition] + : `${DISPOSITION_LABELS[card.disposition]}. Missing: ${card.missingContract}`; + return [ + `## ${card.id}: ${card.title}`, + "", + `- Use when: ${card.useWhen}`, + `- Disposition: ${disposition}`, + `- Owner: ${card.domainOwner}; tracked in ${card.linearId}.`, + `- Commands: ${commands}`, + `- Prerequisites: ${card.prerequisites}`, + `- Env ownership: ${card.envOwnership}`, + `- Readiness: ${card.readiness}`, + `- Cleanup: ${card.cleanup}`, + `- Provenance: ${consumers} (${card.provenance.source}).`, + "", + NEGATIVE_EXAMPLE_START, + `> Negative control: ${card.negativeControl}`, + NEGATIVE_EXAMPLE_END, + "", + ]; +} diff --git a/src/agents/remote-bootstrap-skill.ts b/src/agents/remote-bootstrap-skill.ts index cf9274285..0d93db797 100644 --- a/src/agents/remote-bootstrap-skill.ts +++ b/src/agents/remote-bootstrap-skill.ts @@ -37,9 +37,9 @@ export function renderHackRemoteSkill(): string { "", "Request `HACK_ENV_SECRET_KEY` through the target's approved secret manager or direct process injection. The user enters the value in its native secret UI, never in chat. Hack needs the real key bytes locally to decrypt the committed env; an HTTPS proxy placeholder cannot replace them. Do not print the key, pass it as a command argument, write it into a plugin, image, shell profile, or cached file, or export decrypted values for later reuse. Use non-production credentials and review the chosen command's own logging.", "", - "Current Codex Cloud uses an Install script and Start skill. Request the key as a direct environment variable, preferably a personal value scoped to the chosen environment; it is available to the task. Network secrets substitute HTTPS credentials and cannot decrypt Hack env. Keep prepared files free of keys and plaintext secrets. After approved publication, verify a fresh task. See [current Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environments).", + "Current Codex Cloud uses an Install script and Start skill. Request the key as a direct environment variable, preferably a personal value scoped to the chosen environment; it is available to the task. Network secrets substitute HTTPS credentials and cannot decrypt Hack env. Keep prepared files free of keys and plaintext secrets. After approved publication, verify a fresh task. See current Cloud environments under Further reading.", "", - "Legacy Codex Cloud secrets are setup-only and removed before the agent phase; a setup export does not persist. Do not bypass that boundary by saving the key or decrypted env in cached files. Runtime decryption needs a separately authorized injection mechanism, or the task must use a secret-free workflow. See [legacy Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environment).", + "Legacy Codex Cloud secrets are setup-only and removed before the agent phase; a setup export does not persist. Do not bypass that boundary by saving the key or decrypted env in cached files. Runtime decryption needs a separately authorized injection mechanism, or the task must use a secret-free workflow. See legacy Cloud environments under Further reading.", "", "Cloud tasks must have this skill in their repository or supported cloud environment; a personal plugin installed on a local computer is not proof it is loaded remotely. Confirm the available skills in the target task.", "", @@ -63,5 +63,10 @@ export function renderHackRemoteSkill(): string { "", "Check an actual application endpoint or repository health command with a bounded timeout, then perform one meaningful development operation. Follow the project's approved network access and authentication flow; do not bypass provider restrictions. Report the CLI version/path, target, workflow, env overlay name, readiness result, and remaining blocker without secrets. Installation, a listed MCP server, or a running container alone does not prove the app works. Stop only resources created and owned by this task when cleanup is requested; preserve reusable caches and application data.", "", + "## Further reading", + "", + "- [Current Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environments)", + "- [Legacy Cloud environments](https://learn.chatgpt.com/docs/environments/cloud-environment)", + "", ].join("\n"); } diff --git a/src/agents/skill-references.ts b/src/agents/skill-references.ts new file mode 100644 index 000000000..57792bea0 --- /dev/null +++ b/src/agents/skill-references.ts @@ -0,0 +1,98 @@ +import { createHash } from "node:crypto"; + +import { renderPatternCatalogue } from "./pattern-catalogue.ts"; + +/** Plugin skills that can carry packaged references. */ +export type PluginSkillName = + | "hack-cli" + | "hack-init" + | "hack-install" + | "hack-remote"; + +/** + * A markdown reference shipped inside the plugin package at + * `skills//references/`. + * + * References are offline-first: they resolve inside the installed package, + * link only to files inside it, and keep external URLs under a + * "Further reading" heading. Each one is linked from its skill's SKILL.md. + */ +export type SkillReference = { + readonly skill: PluginSkillName; + /** Lowercase kebab-case markdown file name, e.g. `patterns.md`. */ + readonly file: string; + /** Link text used in the skill's References section. */ + readonly title: string; + /** One-line description shown next to the link. */ + readonly summary: string; + /** Canonical body; the packaged file adds a revision footer. */ + readonly render: () => string; +}; + +/** Every packaged skill reference, in generation order. */ +export const SKILL_REFERENCES: readonly SkillReference[] = [ + { + skill: "hack-cli", + file: "patterns.md", + title: "Pattern catalogue", + summary: + "reusable project patterns with what Hack supports today, the missing contract and its owner.", + render: renderPatternCatalogue, + }, +]; + +const REFERENCE_FILE_PATTERN = /^[a-z0-9][a-z0-9-]*\.md$/; + +/** References that belong to one skill. */ +export function referencesForSkill( + skill: PluginSkillName +): readonly SkillReference[] { + return SKILL_REFERENCES.filter((reference) => reference.skill === skill); +} + +/** Path of a reference relative to the plugin root. */ +export function skillReferencePath(reference: SkillReference): string { + assertReferenceFile(reference.file); + return `skills/${reference.skill}/references/${reference.file}`; +} + +/** First 12 hex characters of the SHA-256 of a reference body. */ +export function skillReferenceRevision(body: string): string { + return createHash("sha256").update(body).digest("hex").slice(0, 12); +} + +/** + * Render the packaged reference file: the canonical body plus a content + * revision that a reader (or test) can recompute from the body above it. + */ +export function renderSkillReference(reference: SkillReference): string { + assertReferenceFile(reference.file); + const body = reference.render(); + return `${body.trimEnd()}\n\n---\n\nContent revision: \`${skillReferenceRevision(body)}\`. Generated by Hack; edit the canonical source and regenerate the plugin.\n`; +} + +/** + * Append a References section that links each of the skill's packaged + * references. Skills without references are returned unchanged, so the + * standalone installs (which ship no references) never gain a dangling link. + */ +export function withSkillReferenceLinks(opts: { + readonly skill: PluginSkillName; + readonly content: string; +}): string { + const references = referencesForSkill(opts.skill); + if (references.length === 0) { + return opts.content; + } + const links = references.map((reference) => { + assertReferenceFile(reference.file); + return `- [${reference.title}](references/${reference.file}): ${reference.summary}`; + }); + return `${opts.content.trimEnd()}\n\n## References\n\n${links.join("\n")}\n`; +} + +function assertReferenceFile(file: string): void { + if (!REFERENCE_FILE_PATTERN.test(file)) { + throw new Error("Invalid skill reference file name"); + } +} diff --git a/tests/agent-plugin-generation.test.ts b/tests/agent-plugin-generation.test.ts index 858a3c90c..891b70366 100644 --- a/tests/agent-plugin-generation.test.ts +++ b/tests/agent-plugin-generation.test.ts @@ -7,9 +7,10 @@ import { generateAgentPlugins, portableMcpManifest, portablePluginManifest, + renderPluginReferences, renderPluginRule, + renderPluginSkill, } from "../scripts/generate-agent-plugins.ts"; -import { renderCodexSkill } from "../src/agents/codex-skill.ts"; import { renderHackInitSkill } from "../src/agents/hack-init-skill.ts"; import { renderHackInstallSkill } from "../src/agents/install-skill.ts"; import { renderHackRemoteSkill } from "../src/agents/remote-bootstrap-skill.ts"; @@ -84,8 +85,13 @@ test("semantic-release commits all generated client manifests", async () => { test("bundled skills and rule match current canonical guidance", async () => { const root = join(repoRoot, "plugins", "hack"); expect(await Bun.file(join(root, "skills/hack-cli/SKILL.md")).text()).toBe( - renderCodexSkill() + renderPluginSkill("hack-cli") ); + const references = renderPluginReferences(); + expect(references.size).toBeGreaterThan(0); + for (const [path, content] of references) { + expect(await Bun.file(join(root, path)).text()).toBe(content); + } expect(await Bun.file(join(root, "skills/hack-init/SKILL.md")).text()).toBe( renderHackInitSkill() ); diff --git a/tests/helpers/skill-links.ts b/tests/helpers/skill-links.ts new file mode 100644 index 000000000..fcd41f668 --- /dev/null +++ b/tests/helpers/skill-links.ts @@ -0,0 +1,172 @@ +import { existsSync } from "node:fs"; +import { dirname, isAbsolute, relative, resolve, sep } from "node:path"; + +/** + * Offline link contract for the packaged plugin skills. + * + * Every markdown file under `/skills` is checked: + * - relative links resolve to an existing file inside the plugin root (no + * `../` escape into the repository, no `docs/` path that mirrors repository + * docs, no absolute paths); + * - external URLs appear only under a "Further reading" heading, so core + * guidance never depends on the network; + * - every `skills//references/*.md` file is linked from that skill's + * SKILL.md. + * + * Inline code and fenced blocks are literals (commands, schema ids), not + * links, and are ignored. Issues carry closed codes and plugin-relative paths. + */ +export type SkillLinkIssueCode = + | "absolute-link" + | "escapes-plugin" + | "repo-docs-link" + | "missing-target" + | "external-outside-further-reading" + | "unlinked-reference" + | "missing-skill"; + +export type SkillLinkIssue = { + readonly file: string; + readonly code: SkillLinkIssueCode; +}; + +const FENCED_BLOCK = /```[\s\S]*?```/g; +const INLINE_CODE = /`[^`\n]*`/g; +const HEADING = /^#{1,6}\s+(.+?)\s*#*\s*$/; +const MARKDOWN_LINK = /\]\(\s*]+)>?(?:\s+"[^"]*")?\s*\)/g; +const URL = /\b[a-z][a-z0-9+.-]*:\/\/[^\s)>\]]+/gi; +const SCHEME = /^[a-z][a-z0-9+.-]*:/i; +const FURTHER_READING = /^further reading$/i; + +export async function findSkillLinkIssues(opts: { + readonly pluginRoot: string; +}): Promise { + const issues: SkillLinkIssue[] = []; + const files: string[] = []; + for await (const path of new Bun.Glob("skills/**/*.md").scan({ + cwd: opts.pluginRoot, + onlyFiles: true, + })) { + files.push(path); + } + files.sort(); + + const linkedFromSkill = new Map>(); + for (const file of files) { + const absolute = resolve(opts.pluginRoot, file); + const targets = new Set(); + for (const link of scanLinks(await Bun.file(absolute).text())) { + const outcome = classifyLink({ + pluginRoot: opts.pluginRoot, + from: absolute, + ...link, + }); + if (outcome.kind === "issue") { + issues.push({ file, code: outcome.code }); + } else if (outcome.kind === "local") { + targets.add(outcome.path); + } + } + if (file.endsWith("/SKILL.md")) { + linkedFromSkill.set(dirname(absolute), targets); + } + } + + for (const file of files) { + const code = referenceLinkIssue({ + pluginRoot: opts.pluginRoot, + file, + linkedFromSkill, + }); + if (code) { + issues.push({ file, code }); + } + } + return issues; +} + +type LinkOutcome = + | { readonly kind: "ignored" } + | { readonly kind: "local"; readonly path: string } + | { readonly kind: "issue"; readonly code: SkillLinkIssueCode }; + +function classifyLink(opts: { + readonly pluginRoot: string; + readonly from: string; + readonly target: string; + readonly furtherReading: boolean; +}): LinkOutcome { + if (SCHEME.test(opts.target) || opts.target.startsWith("//")) { + return opts.furtherReading + ? { kind: "ignored" } + : { kind: "issue", code: "external-outside-further-reading" }; + } + const path = opts.target.split("#")[0]?.split("?")[0] ?? ""; + if (path === "") { + return { kind: "ignored" }; + } + if (isAbsolute(path) || path.startsWith("~")) { + return { kind: "issue", code: "absolute-link" }; + } + const resolved = resolve(dirname(opts.from), path); + const inside = relative(opts.pluginRoot, resolved); + if (inside === ".." || inside.startsWith(`..${sep}`) || isAbsolute(inside)) { + return { kind: "issue", code: "escapes-plugin" }; + } + if (path.split("/").includes("docs")) { + return { kind: "issue", code: "repo-docs-link" }; + } + if (!existsSync(resolved)) { + return { kind: "issue", code: "missing-target" }; + } + return { kind: "local", path: resolved }; +} + +/** A `skills//references/.md` must be linked from its SKILL.md. */ +function referenceLinkIssue(opts: { + readonly pluginRoot: string; + readonly file: string; + readonly linkedFromSkill: ReadonlyMap>; +}): SkillLinkIssueCode | null { + const parts = opts.file.split("/"); + if (parts.length !== 4 || parts[2] !== "references") { + return null; + } + const links = opts.linkedFromSkill.get( + resolve(opts.pluginRoot, "skills", parts[1] ?? "") + ); + if (!links) { + return "missing-skill"; + } + return links.has(resolve(opts.pluginRoot, opts.file)) + ? null + : "unlinked-reference"; +} + +/** Yield link targets and URLs with whether they sit under "Further reading". */ +function* scanLinks( + text: string +): Generator<{ readonly target: string; readonly furtherReading: boolean }> { + const prose = text + .replace(FENCED_BLOCK, (block) => block.replace(/[^\n]/g, " ")) + .replace(INLINE_CODE, (code) => " ".repeat(code.length)); + let furtherReading = false; + for (const line of prose.split("\n")) { + const heading = HEADING.exec(line); + if (heading) { + furtherReading = FURTHER_READING.test(heading[1] ?? ""); + continue; + } + const seen = new Set(); + for (const match of line.matchAll(MARKDOWN_LINK)) { + const target = match[1] ?? ""; + seen.add(target); + yield { target, furtherReading }; + } + for (const match of line.matchAll(URL)) { + if (!seen.has(match[0])) { + yield { target: match[0], furtherReading }; + } + } + } +} diff --git a/tests/pattern-catalogue.test.ts b/tests/pattern-catalogue.test.ts new file mode 100644 index 000000000..96e6df6b9 --- /dev/null +++ b/tests/pattern-catalogue.test.ts @@ -0,0 +1,217 @@ +import { expect, test } from "bun:test"; +import { join } from "node:path"; + +import { + NEGATIVE_EXAMPLE_END, + NEGATIVE_EXAMPLE_START, + PATTERN_CATALOGUE, + type PatternCard, + type PatternDisposition, + renderPatternCatalogue, +} from "../src/agents/pattern-catalogue.ts"; +import { CLI_SPEC } from "../src/cli/spec.ts"; +import { + extractHackInvocations, + findGuidanceCommandIssues, +} from "./helpers/cli-guidance-commands.ts"; + +const EXPECTED_IDS = Array.from( + { length: 12 }, + (_, index) => `HACK-P${String(index + 1).padStart(2, "0")}` +); +const OWNER = /^WU\d+(?:\/WU\d+)*$/; + +/** Closed id problems: missing or duplicated cards fail the register. */ +function catalogueIdIssues(cards: readonly PatternCard[]): string[] { + const ids = cards.map((card) => card.id); + return [ + ...EXPECTED_IDS.filter((id) => !ids.includes(id)).map( + (id) => `missing ${id}` + ), + ...ids + .filter((id, index) => ids.indexOf(id) !== index) + .map((id) => `duplicate ${id}`), + ...ids + .filter((id) => !EXPECTED_IDS.includes(id)) + .map((id) => `unexpected ${id}`), + ]; +} + +/** Recipes that must never appear outside a negative-example block. */ +const BANNED_RECIPES: readonly (readonly [string, RegExp])[] = [ + ["broad-process-kill", /\b(?:pkill|killall)\b/], + ["kill-by-port", /\blsof\b[^\n|]*-t[^\n|]*\|\s*xargs\s+(?:-\S+\s+)*kill\b/], + ["placeholder-liveness-tail", /\btail\s+-f\s+\/dev\/null\b/], + ["placeholder-liveness-sleep", /\bsleep\s+infinity\b/], + [ + "argv-secret-flag", + /--(?:password|passwd|token|secret|api-key|apikey|access-key)(?:=|\s+)\S/i, + ], + [ + "argv-secret-assignment", + /\b[A-Z][A-Z0-9_]*(?:TOKEN|SECRET|PASSWORD|PASSWD|API_KEY)=\S/, + ], + ["argv-env-value", /\bhack\s+env\s+(?:set|add)\s+\S+\s+\S+/], +]; + +function stripNegativeExamples(text: string): string { + let result = ""; + let rest = text; + for (;;) { + const start = rest.indexOf(NEGATIVE_EXAMPLE_START); + if (start === -1) { + return result + rest; + } + const end = rest.indexOf(NEGATIVE_EXAMPLE_END, start); + if (end === -1) { + // An unterminated block exempts nothing. + return result + rest; + } + result += rest.slice(0, start); + rest = rest.slice(end + NEGATIVE_EXAMPLE_END.length); + } +} + +function bannedRecipeFindings(text: string): string[] { + const scanned = stripNegativeExamples(text); + return BANNED_RECIPES.filter(([, pattern]) => pattern.test(scanned)).map( + ([code]) => code + ); +} + +test("the catalogue holds exactly the twelve HACK-P cards in order", () => { + expect(catalogueIdIssues(PATTERN_CATALOGUE)).toEqual([]); + expect(PATTERN_CATALOGUE.map((card) => card.id)).toEqual(EXPECTED_IDS); + expect(PATTERN_CATALOGUE.map((card) => card.linearId)).toEqual( + Array.from({ length: 12 }, (_, index) => `HACK-${1145 + index}`) + ); +}); + +test("negative control: deleting or duplicating a card fails the id check", () => { + expect(catalogueIdIssues(PATTERN_CATALOGUE.slice(1))).toEqual([ + "missing HACK-P01", + ]); + const first = PATTERN_CATALOGUE[0]; + if (!first) { + throw new Error("empty catalogue"); + } + expect(catalogueIdIssues([...PATTERN_CATALOGUE, first])).toEqual([ + "duplicate HACK-P01", + ]); +}); + +test("starting dispositions match the recorded decisions", () => { + const expected: Readonly> = { + "HACK-P01": "partial", + "HACK-P02": "partial", + "HACK-P03": "partial", + "HACK-P04": "retain-as-example", + "HACK-P05": "deferred", + "HACK-P06": "deferred", + "HACK-P07": "partial", + "HACK-P08": "deferred", + "HACK-P09": "partial", + "HACK-P10": "partial", + "HACK-P11": "partial", + "HACK-P12": "deferred", + }; + expect( + Object.fromEntries( + PATTERN_CATALOGUE.map((card) => [card.id, card.disposition]) + ) + ).toEqual(expected); + expect( + PATTERN_CATALOGUE.find((card) => card.id === "HACK-P04")?.domainOwner + ).toBe("WU29"); +}); + +test("every card is complete and every unsupported card names its owner and missing contract", () => { + for (const card of PATTERN_CATALOGUE) { + for (const field of [ + card.title, + card.useWhen, + card.prerequisites, + card.envOwnership, + card.readiness, + card.cleanup, + card.negativeControl, + card.provenance.source, + ]) { + expect(field.trim().length, card.id).toBeGreaterThan(0); + } + expect(card.domainOwner, card.id).toMatch(OWNER); + expect(card.provenance.consumers, card.id).toBeGreaterThan(0); + if (card.disposition === "supported") { + expect(card.missingContract, card.id).toBeNull(); + } else { + expect(card.missingContract?.trim().length ?? 0, card.id).toBeGreaterThan( + 0 + ); + } + } +}); + +test("every named hack command resolves in CLI_SPEC", () => { + for (const card of PATTERN_CATALOGUE) { + for (const command of card.supportedCommands) { + expect(command.startsWith("hack "), `${card.id}: ${command}`).toBe(true); + expect( + findGuidanceCommandIssues({ + cli: CLI_SPEC, + rendered: `\`${command}\``, + }), + `${card.id}: ${command}` + ).toEqual([]); + } + } + const rendered = renderPatternCatalogue(); + expect(extractHackInvocations(rendered).length).toBeGreaterThan(0); + expect(findGuidanceCommandIssues({ cli: CLI_SPEC, rendered })).toEqual([]); +}); + +test("banned recipes appear only inside negative-example blocks", () => { + expect(bannedRecipeFindings(renderPatternCatalogue())).toEqual([]); + const rendered = renderPatternCatalogue(); + for (const card of PATTERN_CATALOGUE) { + expect(rendered).toContain( + `${NEGATIVE_EXAMPLE_START}\n> Negative control: ${card.negativeControl}\n${NEGATIVE_EXAMPLE_END}` + ); + } +}); + +test("negative control: the banned-recipe scan flags each recipe outside a negative block", () => { + const fixtures: Readonly> = { + "broad-process-kill": "pkill -f node", + "kill-by-port": "lsof -t -i :3000 | xargs kill", + "placeholder-liveness-tail": "command: tail -f /dev/null", + "placeholder-liveness-sleep": "command: sleep infinity", + "argv-secret-flag": "tool login --token abc123", + "argv-secret-assignment": "API_TOKEN=abc123 hack up", + "argv-env-value": "hack env set DATABASE_URL postgres://x", + }; + for (const [code, text] of Object.entries(fixtures)) { + expect(bannedRecipeFindings(text)).toEqual([code]); + expect( + bannedRecipeFindings( + `${NEGATIVE_EXAMPLE_START}\n${text}\n${NEGATIVE_EXAMPLE_END}` + ) + ).toEqual([]); + expect(bannedRecipeFindings(`${NEGATIVE_EXAMPLE_START}\n${text}`)).toEqual([ + code, + ]); + } +}); + +test("the packaged hack-cli reference is the rendered catalogue", async () => { + const packaged = await Bun.file( + join( + import.meta.dir, + "..", + "plugins/hack/skills/hack-cli/references/patterns.md" + ) + ).text(); + expect(packaged.startsWith(renderPatternCatalogue().trimEnd())).toBe(true); + for (const card of PATTERN_CATALOGUE) { + expect(packaged).toContain(`## ${card.id}: ${card.title}`); + } +}); diff --git a/tests/skill-references.test.ts b/tests/skill-references.test.ts new file mode 100644 index 000000000..c650199e6 --- /dev/null +++ b/tests/skill-references.test.ts @@ -0,0 +1,130 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdir, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { renderPluginSkill } from "../scripts/generate-agent-plugins.ts"; +import { renderCodexSkill } from "../src/agents/codex-skill.ts"; +import { + renderSkillReference, + SKILL_REFERENCES, + skillReferencePath, + skillReferenceRevision, + withSkillReferenceLinks, +} from "../src/agents/skill-references.ts"; +import { findSkillLinkIssues } from "./helpers/skill-links.ts"; + +const pluginRoot = join(import.meta.dir, "..", "plugins", "hack"); + +let tempDir: string | null = null; + +afterEach(async () => { + if (tempDir) { + await rm(tempDir, { recursive: true, force: true }); + tempDir = null; + } +}); + +async function fixturePlugin( + files: Readonly> +): Promise { + tempDir = await mkdtemp(join(tmpdir(), "hack-skill-links-")); + const root = join(tempDir, "plugins", "hack"); + await mkdir(join(tempDir, "docs"), { recursive: true }); + await Bun.write(join(tempDir, "docs", "env.md"), "# Env\n"); + for (const [path, text] of Object.entries(files)) { + await mkdir(join(root, path, ".."), { recursive: true }); + await Bun.write(join(root, path), text); + } + return root; +} + +test("packaged skill links stay inside the plugin and work offline", async () => { + expect(await findSkillLinkIssues({ pluginRoot })).toEqual([]); +}); + +test("every packaged reference is linked and carries a recomputable revision", async () => { + expect(SKILL_REFERENCES.length).toBeGreaterThan(0); + for (const reference of SKILL_REFERENCES) { + const packaged = await Bun.file( + join(pluginRoot, skillReferencePath(reference)) + ).text(); + expect(packaged).toBe(renderSkillReference(reference)); + expect(packaged).toContain( + `Content revision: \`${skillReferenceRevision(reference.render())}\`` + ); + const skill = await Bun.file( + join(pluginRoot, "skills", reference.skill, "SKILL.md") + ).text(); + expect(skill).toContain(`](references/${reference.file})`); + } +}); + +test("only the plugin skill links references; standalone renders stay unchanged", () => { + expect(renderCodexSkill()).not.toContain("references/"); + const plugin = renderPluginSkill("hack-cli"); + expect(plugin.startsWith(renderCodexSkill().trimEnd())).toBe(true); + expect(plugin.slice(renderCodexSkill().trimEnd().length)).toBe( + "\n\n## References\n\n- [Pattern catalogue](references/patterns.md): reusable project patterns with what Hack supports today, the missing contract and its owner.\n" + ); + const unchanged = "---\nname: hack-init\n---\n"; + expect( + withSkillReferenceLinks({ skill: "hack-init", content: unchanged }) + ).toBe(unchanged); +}); + +test("negative control: a link into repository docs fails the link test", async () => { + const root = await fixturePlugin({ + "skills/x/SKILL.md": "# X\n\nSee [env](../../docs/env.md).\n", + "skills/y/SKILL.md": "# Y\n\nSee [env](../../../../docs/env.md).\n", + }); + expect(await findSkillLinkIssues({ pluginRoot: root })).toEqual([ + { file: "skills/x/SKILL.md", code: "repo-docs-link" }, + { file: "skills/y/SKILL.md", code: "escapes-plugin" }, + ]); +}); + +test("negative control: absolute, missing, external and unlinked references fail", async () => { + const root = await fixturePlugin({ + "skills/x/SKILL.md": [ + "# X", + "", + "- [abs](/etc/hosts)", + "- [gone](references/gone.md)", + "- [web](https://example.com/guide)", + "- Read https://example.com/bare too.", + "- `https://schemas.hack/literal.json` is a literal, not a link.", + "", + "```sh", + "curl https://example.com/in-a-fence", + "```", + "", + "## Further reading", + "", + "- [ok](https://example.com/ok)", + "", + ].join("\n"), + "skills/x/references/orphan.md": "# Orphan\n", + }); + expect(await findSkillLinkIssues({ pluginRoot: root })).toEqual([ + { file: "skills/x/SKILL.md", code: "absolute-link" }, + { file: "skills/x/SKILL.md", code: "missing-target" }, + { file: "skills/x/SKILL.md", code: "external-outside-further-reading" }, + { file: "skills/x/SKILL.md", code: "external-outside-further-reading" }, + { file: "skills/x/references/orphan.md", code: "unlinked-reference" }, + ]); +}); + +test("reference file names are constrained to plain markdown names", () => { + for (const file of ["../escape.md", "nested/x.md", "x.txt", "X.md"]) { + expect(() => + skillReferencePath({ + skill: "hack-cli", + file, + title: "x", + summary: "x", + render: () => "", + }) + ).toThrow("Invalid skill reference file name"); + } +}); From 726cc27601bf64657314289820f456c1e77b1c89 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 04:01:11 -0400 Subject: [PATCH 22/32] test(e2e): add agent-surface-freeze negative control for guidance writes Seed an isolated HOME with sentinel Claude, Codex, Cursor and shared-skill config (including stale hack-cli copies) and two fixtures with customized AGENTS.md, CLAUDE.md, rules, skill copies, settings.local.json and .mcp.json. Run init --auto, init --auto --with codex, agent prime, agent prime --check, agent onboard, agent patterns, doctor --json, doctor --fix --no-interactive and setup sync --check twice each with HOME and HACK_HOME overridden, and require byte-identical hashes, no new client entries, git status changes only under .hack/, and each command's expected exit code. An explicit setup agents --agents-md is the positive control and must change exactly AGENTS.md. Docker is pointed at a missing socket and Mutagen at a stub so doctor --fix stays free of Docker and network effects; the hackd it starts is stopped. The scenario is registered in ALL_SCENARIOS and shard.ts places it in exactly one docker-e2e shard via the default duration hint. Refs: HACK-1124, HACK-1132, HACK-1155 Co-Authored-By: Claude Opus 5.5 --- tests/e2e/README.md | 34 +- tests/e2e/fixture.ts | 184 ++++++++++- tests/e2e/run.ts | 2 + tests/e2e/scenarios/agent-surface-freeze.ts | 338 ++++++++++++++++++++ 4 files changed, 552 insertions(+), 6 deletions(-) create mode 100644 tests/e2e/scenarios/agent-surface-freeze.ts diff --git a/tests/e2e/README.md b/tests/e2e/README.md index d88048602..6e20ee847 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -443,10 +443,11 @@ any e2e result. - **Tier 1 (`local`)** — runs everywhere; no docker needed: `automation-check`, `init`, `env-secrets`, `worktree-secrets`, `worktree-registry`, `worktree-branch-default`, `agent-docs-sync`, - `doctor` (doctor tolerates missing docker — it must report, not crash), and - `lifecycle-session-recovery`. The lifecycle recovery scenario skips when - tmux is unavailable unless `HACK_E2E_REQUIRE_TMUX=1` makes that capability - mandatory (as it is in the dedicated Docker/tmux CI job). + `agent-surface-freeze`, `doctor` (doctor tolerates missing docker — it must + report, not crash), and `lifecycle-session-recovery`. The lifecycle recovery + scenario skips when tmux is unavailable unless `HACK_E2E_REQUIRE_TMUX=1` + makes that capability mandatory (as it is in the dedicated Docker/tmux CI + job). - **Tier 2 (`docker`)** — opt-in via `HACK_E2E_DOCKER=1`; requires a running docker daemon, the machine's global `hack-dev` network (`hack global install`), and pulls `oven/bun:1`: `cache-prune`, `up-down`, @@ -455,6 +456,31 @@ any e2e result. processes only start via `hack up` (no standalone lifecycle command), which is why that scenario is tier 2. +## Agent surface freeze + +`agent-surface-freeze` is the negative control for surprise agent config +writes (HACK-1124, HACK-1132, HACK-1155). It seeds a scenario-owned HOME with +sentinel client config (`~/.claude/settings.json`, `~/.claude.json`, +`~/.claude/skills/`, `~/.codex/config.toml`, `~/.codex/skills/`, +`~/.cursor/mcp.json`, `~/.cursor/rules/`, `~/.ai/skills/`, including stale +`hack-cli` copies). It also commits customized `AGENTS.md`, `CLAUDE.md`, +`.cursor/rules/hack.mdc`, `.codex/skills/hack-cli/SKILL.md`, +`.claude/settings.local.json` and `.mcp.json` into two fixtures. It then runs +`init --auto`, `init --auto --with codex`, `agent prime`, +`agent prime --check`, `agent onboard`, `agent patterns`, `doctor --json`, +`doctor --fix --no-interactive` and `setup sync --check` twice each. After +every run it requires byte-identical hashes for those trees, no new entries +under them, and `git status` changes only under `.hack/`. Each command must +also exit with its expected code, so a usage error cannot pass as "no write". + +Every call sets `HOME` and `HACK_HOME` to scenario temp dirs, so the +contributor's real HOME is never read or written. `DOCKER_HOST` points at a +missing socket and `HACK_MUTAGEN_PATH` at a stub. This keeps +`doctor --fix` free of Docker and network effects even on a Docker CI shard. +The scenario stops the isolated hackd that `doctor --fix` starts. The positive +control, an explicit `setup agents --agents-md`, must change exactly +`AGENTS.md`. + ## Fixture `fixture.ts` scaffolds, per scenario, a fresh git repo with: diff --git a/tests/e2e/fixture.ts b/tests/e2e/fixture.ts index eeceb9355..d014f1ea1 100644 --- a/tests/e2e/fixture.ts +++ b/tests/e2e/fixture.ts @@ -1,5 +1,6 @@ -import { mkdir } from "node:fs/promises"; -import { join, resolve } from "node:path"; +import type { Stats } from "node:fs"; +import { lstat, mkdir, readdir, readlink } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; import { runCommand } from "./harness.ts"; @@ -495,3 +496,182 @@ async function git(opts: { ); } } + +/** Content hash per path; directories map to "dir" so new empty dirs count. */ +export type FileHashSnapshot = ReadonlyMap; + +export type FileHashDiff = { + readonly changed: readonly string[]; + readonly added: readonly string[]; + readonly removed: readonly string[]; +}; + +/** + * Agent client config a default Hack flow must never touch, relative to an + * isolated HOME. Stale Hack-owned guidance (the most tempting "repair" + * target) sits next to unrelated user config. + */ +const HOME_CLIENT_SENTINELS: Readonly> = { + ".claude/settings.json": `${JSON.stringify({ env: { HACK_E2E_SENTINEL: "home-claude-settings" } }, null, 2)}\n`, + ".claude.json": `${JSON.stringify({ mcpServers: { "e2e-sentinel": { command: "true" } } }, null, 2)}\n`, + ".claude/skills/e2e-sentinel/SKILL.md": + "---\nname: e2e-sentinel\n---\nhack-e2e sentinel: home claude skill\n", + ".claude/skills/hack-cli/SKILL.md": + "---\nname: hack-cli\n---\nhack-e2e sentinel: stale home claude hack skill\n", + ".codex/config.toml": + '# hack-e2e sentinel: home codex config\nmodel = "e2e-sentinel"\n', + ".codex/skills/e2e-sentinel/SKILL.md": + "---\nname: e2e-sentinel\n---\nhack-e2e sentinel: home codex skill\n", + ".codex/skills/hack-cli/SKILL.md": + "---\nname: hack-cli\n---\nhack-e2e sentinel: stale home codex hack skill\n", + ".cursor/mcp.json": `${JSON.stringify({ mcpServers: { "e2e-sentinel": { command: "true" } } }, null, 2)}\n`, + ".cursor/rules/e2e-sentinel.mdc": "hack-e2e sentinel: home cursor rule\n", + ".cursor/rules/hack.mdc": "hack-e2e sentinel: stale home cursor hack rule\n", + ".ai/skills/e2e-sentinel/SKILL.md": + "---\nname: e2e-sentinel\n---\nhack-e2e sentinel: home shared skill\n", + ".ai/skills/hack-cli/SKILL.md": + "---\nname: hack-cli\n---\nhack-e2e sentinel: stale home shared hack skill\n", +}; + +/** HOME-relative client roots whose whole tree must stay byte-identical. */ +export const HOME_CLIENT_ROOTS: readonly string[] = [ + ".claude", + ".claude.json", + ".codex", + ".cursor", + ".ai", + ".agents", +]; + +/** + * Customized repo guidance a default Hack flow must never rewrite, relative + * to a fixture root. None carries Hack's managed markers, so any write is a + * surprise rewrite rather than an in-place marker refresh. + */ +const REPO_GUIDANCE_SENTINELS: Readonly> = { + "AGENTS.md": + "# Team agent rules\n\nhack-e2e sentinel: customized AGENTS.md\n", + "CLAUDE.md": + "# Team Claude rules\n\nhack-e2e sentinel: customized CLAUDE.md\n", + ".cursor/rules/hack.mdc": "hack-e2e sentinel: customized repo cursor rule\n", + ".codex/skills/hack-cli/SKILL.md": + "---\nname: hack-cli\n---\nhack-e2e sentinel: customized repo codex skill\n", + ".claude/settings.local.json": `${JSON.stringify({ permissions: { allow: ["Bash(true)"] } }, null, 2)}\n`, + ".mcp.json": `${JSON.stringify({ mcpServers: { "e2e-sentinel": { command: "true" } } }, null, 2)}\n`, +}; + +/** Root-relative repo guidance roots whose whole tree must stay byte-identical. */ +export const REPO_GUIDANCE_ROOTS: readonly string[] = [ + "AGENTS.md", + "CLAUDE.md", + ".mcp.json", + ".claude", + ".codex", + ".cursor", + ".ai", + ".agents", +]; + +async function writeSentinels(opts: { + readonly root: string; + readonly files: Readonly>; +}): Promise { + const written: string[] = []; + for (const [relative, content] of Object.entries(opts.files)) { + const path = join(opts.root, relative); + await mkdir(dirname(path), { recursive: true }); + await Bun.write(path, content); + written.push(path); + } + return written; +} + +/** Seed sentinel client config under an isolated HOME; returns the files. */ +export async function seedHomeClientSentinels(opts: { + readonly home: string; +}): Promise { + return await writeSentinels({ + root: opts.home, + files: HOME_CLIENT_SENTINELS, + }); +} + +/** Seed and commit customized repo guidance; returns the files. */ +export async function seedRepoGuidanceSentinels(opts: { + readonly root: string; +}): Promise { + const written = await writeSentinels({ + root: opts.root, + files: REPO_GUIDANCE_SENTINELS, + }); + // Force-add: settings.local.json is commonly ignored, but must be tracked + // here so `git status` would surface any edit. + await git({ root: opts.root, args: ["add", "-f", "--", ...written] }); + await git({ + root: opts.root, + args: ["commit", "-q", "-m", "fixture: customized agent guidance"], + }); + return written; +} + +/** + * Hash every entry under `roots` (files or directories; missing roots add + * nothing). Symlinks record their target text and are never followed. + */ +export async function snapshotFileHashes(opts: { + readonly roots: readonly string[]; +}): Promise { + const snapshot = new Map(); + async function visit(path: string): Promise { + let info: Stats; + try { + info = await lstat(path); + } catch { + return; + } + if (info.isSymbolicLink()) { + snapshot.set(path, `link:${await readlink(path)}`); + } else if (info.isDirectory()) { + snapshot.set(path, "dir"); + for (const entry of await readdir(path)) { + await visit(join(path, entry)); + } + } else { + const hasher = new Bun.CryptoHasher("sha256"); + hasher.update(await Bun.file(path).arrayBuffer()); + snapshot.set(path, `sha256:${hasher.digest("hex")}`); + } + } + for (const root of opts.roots) { + await visit(root); + } + return snapshot; +} + +/** Paths whose hash changed, appeared, or disappeared between snapshots. */ +export function diffFileHashes(opts: { + readonly before: FileHashSnapshot; + readonly after: FileHashSnapshot; +}): FileHashDiff { + const changed: string[] = []; + const added: string[] = []; + const removed: string[] = []; + for (const [path, hash] of opts.after) { + const prior = opts.before.get(path); + if (prior === undefined) { + added.push(path); + } else if (prior !== hash) { + changed.push(path); + } + } + for (const path of opts.before.keys()) { + if (!opts.after.has(path)) { + removed.push(path); + } + } + return { + changed: changed.sort(), + added: added.sort(), + removed: removed.sort(), + }; +} diff --git a/tests/e2e/run.ts b/tests/e2e/run.ts index 7e9f06682..e7151477f 100644 --- a/tests/e2e/run.ts +++ b/tests/e2e/run.ts @@ -6,6 +6,7 @@ import { } from "./harness.ts"; import { nativeConfigNetworksScenario } from "./native-config-networks-acceptance.ts"; import { agentDocsSyncScenario } from "./scenarios/agent-docs-sync.ts"; +import { agentSurfaceFreezeScenario } from "./scenarios/agent-surface-freeze.ts"; import { automationCheckScenario } from "./scenarios/automation-check.ts"; import { cachePruneScenario } from "./scenarios/cache-prune.ts"; import { dependencyCacheScenario } from "./scenarios/dependency-cache.ts"; @@ -81,6 +82,7 @@ export const ALL_SCENARIOS: readonly Scenario[] = [ worktreeRegistryScenario, worktreeBranchDefaultScenario, agentDocsSyncScenario, + agentSurfaceFreezeScenario, doctorScenario, domainMigrationFilesScenario, lifecycleSessionRecoveryScenario, diff --git a/tests/e2e/scenarios/agent-surface-freeze.ts b/tests/e2e/scenarios/agent-surface-freeze.ts new file mode 100644 index 000000000..e9ae98c97 --- /dev/null +++ b/tests/e2e/scenarios/agent-surface-freeze.ts @@ -0,0 +1,338 @@ +import { mkdir } from "node:fs/promises"; +import { join, relative } from "node:path"; + +import { + createMonorepoFixture, + diffFileHashes, + type FileHashSnapshot, + HOME_CLIENT_ROOTS, + type MonorepoFixture, + REPO_GUIDANCE_ROOTS, + seedHomeClientSentinels, + seedRepoGuidanceSentinels, + snapshotFileHashes, +} from "../fixture.ts"; +import { + type CliResult, + expect, + expectExit, + runCommand, + type Scenario, + type ScenarioContext, +} from "../harness.ts"; + +const FREEZE_TIMEOUT_MS = 240_000; +const RUNS_PER_COMMAND = 2; + +/** + * Exit codes accepted for one ordinary command, per run. A wrong code means + * the command did not exercise its real path (usage error, crash), so a + * byte-identical tree would prove nothing. + */ +type FreezeCommand = { + readonly label: string; + readonly fixture: "project" | "handoff"; + readonly args: readonly string[]; + readonly codes: readonly [readonly number[], readonly number[]]; +}; + +/** + * Default flows covered by the freeze. `init --auto` refuses an existing + * .hack/ on its second run (exit 1); `--with codex` instead hands off the + * onboarding prompt for the existing setup (exit 0). `doctor` exits 1 for the + * unreachable Docker and missing global install, and `--fix` runs only its + * Docker-free remediations (see `freezeEnv`). `setup sync --check` exits 1 + * because the customized guidance differs from Hack's render; it reports the + * drift and must not repair it. + */ +const FREEZE_COMMANDS: readonly FreezeCommand[] = [ + { + label: "init --auto", + fixture: "project", + args: ["init", "--auto"], + codes: [[0], [1]], + }, + { + label: "init --auto --with codex", + fixture: "handoff", + args: ["init", "--auto", "--with", "codex"], + codes: [[0], [0]], + }, + { + label: "agent prime", + fixture: "project", + args: ["agent", "prime"], + codes: [[0], [0]], + }, + { + label: "agent prime --check", + fixture: "project", + args: ["agent", "prime", "--check"], + codes: [[0], [0]], + }, + { + label: "agent onboard", + fixture: "project", + args: ["agent", "onboard"], + codes: [[0], [0]], + }, + { + label: "agent patterns", + fixture: "project", + args: ["agent", "patterns"], + codes: [[0], [0]], + }, + { + label: "doctor --json", + fixture: "project", + args: ["doctor", "--json"], + codes: [ + [0, 1], + [0, 1], + ], + }, + { + label: "doctor --fix --no-interactive", + fixture: "project", + args: ["doctor", "--fix", "--no-interactive"], + codes: [ + [0, 1], + [0, 1], + ], + }, + { + label: "setup sync --check", + fixture: "project", + args: ["setup", "sync", "--check"], + codes: [[1], [1]], + }, +]; + +type FreezeState = { + readonly ctx: ScenarioContext; + readonly env: Readonly>; + readonly roots: readonly string[]; + readonly fixtures: Readonly< + Record + >; + readonly baseline: FileHashSnapshot; +}; + +/** + * Negative control for agent guidance writes (HACK-1124, HACK-1132, + * HACK-1155): default setup, onboarding, agent and doctor flows leave an + * isolated HOME's client config and a repo's customized guidance + * byte-identical, twice in a row. An explicit `setup agents` export is the + * positive control: it changes only its documented target. + * + * Every call overrides HOME and HACK_HOME with scenario temp dirs; the + * contributor's real HOME is never read or written. + */ +export const agentSurfaceFreezeScenario: Scenario = { + name: "agent-surface-freeze", + tier: "local", + summary: + "default flows leave isolated-HOME client config and repo guidance byte-identical", + run: async (ctx) => { + const home = join(ctx.tempRoot, "home"); + await mkdir(home, { recursive: true }); + const seededHome = await seedHomeClientSentinels({ home }); + const project = await createMonorepoFixture({ + parentDir: ctx.tempRoot, + withHackConfig: false, + }); + const handoff = await createMonorepoFixture({ + parentDir: ctx.tempRoot, + withHackConfig: false, + }); + const seededRepo = [ + ...(await seedRepoGuidanceSentinels({ root: project.root })), + ...(await seedRepoGuidanceSentinels({ root: handoff.root })), + ]; + + const roots = [ + ...HOME_CLIENT_ROOTS.map((root) => join(home, root)), + ...REPO_GUIDANCE_ROOTS.map((root) => join(project.root, root)), + ...REPO_GUIDANCE_ROOTS.map((root) => join(handoff.root, root)), + ]; + const baseline = await snapshotFileHashes({ roots }); + const missing = [...seededHome, ...seededRepo].filter( + (path) => !baseline.get(path)?.startsWith("sha256:") + ); + expect({ + that: missing.length === 0, + message: `baseline snapshot is missing seeded sentinels: ${missing.join(", ")}`, + }); + + const state: FreezeState = { + ctx, + env: freezeEnv({ ctx, home }), + roots, + fixtures: { project, handoff }, + baseline, + }; + + try { + for (const command of FREEZE_COMMANDS) { + for (let run = 0; run < RUNS_PER_COMMAND; run += 1) { + await runFrozenCommand({ state, command, run }); + } + } + await runPositiveControl({ state }); + } finally { + // `doctor --fix` starts hackd under the isolated HACK_HOME; never let it + // outlive the scenario. `daemon stop` exits 0 whether or not it ran. + const stop = await ctx.cli({ + args: ["daemon", "stop"], + cwd: project.root, + env: state.env, + }); + if (stop.timedOut || stop.exitCode !== 0) { + ctx.retainFixtures("isolated hackd stop failed"); + } + } + }, +}; + +/** + * Per-call env: isolated HOME + HACK_HOME, Docker pointed at a socket that + * does not exist (local tier: no Docker side effects even when the host runs + * Docker), and a stub Mutagen path so `doctor --fix` never downloads. + */ +function freezeEnv(opts: { + readonly ctx: ScenarioContext; + readonly home: string; +}): Readonly> { + return { + HOME: opts.home, + HACK_HOME: opts.ctx.hackHome, + DOCKER_HOST: `unix://${join(opts.ctx.tempRoot, "no-docker.sock")}`, + HACK_MUTAGEN_PATH: join(opts.ctx.tempRoot, "no-mutagen"), + }; +} + +async function runFrozenCommand(opts: { + readonly state: FreezeState; + readonly command: FreezeCommand; + readonly run: number; +}): Promise { + const { state, command } = opts; + const fixture = state.fixtures[command.fixture]; + const label = `${command.label} (run ${opts.run + 1})`; + const result = await state.ctx.cli({ + args: command.args, + cwd: fixture.root, + env: state.env, + timeoutMs: FREEZE_TIMEOUT_MS, + }); + expectExit({ + result, + codes: command.codes[opts.run] ?? [], + message: `${label} should exercise its real path`, + }); + await expectFrozen({ state, label, result }); + state.ctx.log(`${label}: exit ${result.exitCode}, guidance unchanged`); +} + +async function expectFrozen(opts: { + readonly state: FreezeState; + readonly label: string; + readonly result: CliResult; +}): Promise { + const diff = diffFileHashes({ + before: opts.state.baseline, + after: await snapshotFileHashes({ roots: opts.state.roots }), + }); + expect({ + that: + diff.changed.length === 0 && + diff.added.length === 0 && + diff.removed.length === 0, + message: `${opts.label} wrote agent client or guidance surfaces: changed=[${diff.changed.join(", ")}] added=[${diff.added.join(", ")}] removed=[${diff.removed.join(", ")}]`, + result: opts.result, + }); + for (const fixture of Object.values(opts.state.fixtures)) { + const unexpected = (await gitStatusPaths({ root: fixture.root })).filter( + (line) => !line.path.startsWith(".hack/") + ); + expect({ + that: unexpected.length === 0, + message: `${opts.label} left tracked or untracked changes outside .hack/ in ${fixture.name}: ${unexpected.map((line) => line.raw).join(", ")}`, + result: opts.result, + }); + } +} + +/** + * Positive control: the explicit AGENTS.md export must write exactly + * AGENTS.md, proving the hash and git-status probes can see a real write. + */ +async function runPositiveControl(opts: { + readonly state: FreezeState; +}): Promise { + const { state } = opts; + const project = state.fixtures.project; + const target = join(project.root, "AGENTS.md"); + const result = await state.ctx.cli({ + args: ["setup", "agents", "--agents-md"], + cwd: project.root, + env: state.env, + }); + expectExit({ + result, + codes: [0], + message: "explicit setup agents --agents-md should succeed", + }); + const diff = diffFileHashes({ + before: state.baseline, + after: await snapshotFileHashes({ roots: state.roots }), + }); + expect({ + that: + diff.changed.length === 1 && + diff.changed[0] === target && + diff.added.length === 0 && + diff.removed.length === 0, + message: `explicit setup agents --agents-md should change only ${target}: changed=[${diff.changed.join(", ")}] added=[${diff.added.join(", ")}] removed=[${diff.removed.join(", ")}]`, + result, + }); + const outside = (await gitStatusPaths({ root: project.root })).filter( + (line) => !line.path.startsWith(".hack/") + ); + expect({ + that: + outside.length === 1 && + outside[0]?.path === relative(project.root, target), + message: `explicit export should leave only AGENTS.md modified outside .hack/: ${outside.map((line) => line.raw).join(", ")}`, + result, + }); + state.ctx.log("positive control: setup agents --agents-md changed AGENTS.md"); +} + +type StatusLine = { readonly raw: string; readonly path: string }; + +/** Porcelain v1 entries, including every untracked file. */ +async function gitStatusPaths(opts: { + readonly root: string; +}): Promise { + const status = await runCommand({ + argv: [ + "git", + "-C", + opts.root, + "status", + "--porcelain=v1", + "--untracked-files=all", + ], + cwd: opts.root, + }); + expectExit({ + result: status, + codes: [0], + message: "git status should succeed in the fixture", + }); + return status.stdout + .split("\n") + .filter((line) => line.length > 3) + .map((raw) => ({ raw, path: raw.slice(3) })); +} From 9b971fa00e94e2a691d7855469aa2781dc006836 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 04:07:49 -0400 Subject: [PATCH 23/32] ci: add gated Bun crash experiment and read-only crash census HACK-1211 needs a matched, bounded way to reproduce the intermittent Bun crash and a repeatable record of crashes per hosted run. This adds the instruments only; it changes no test, timeout, assertion or pin. - bun-crash-experiment.yml runs only on workflow_dispatch or when the bun-crash-experiment label is added to a pull request. A labeled pull request runs the matched pair (1.3.9 baseline, 1.4.2 candidate) on its head; a dispatch runs one version at a chosen revision. It uses blacksmith-6vcpu-macos-15, contents: read, and timeout = budget + 15. The instrument comes from the workflow's commit, so older revisions can be measured. - scripts/bun-crash-loop.ts alternates the native HTTPS owner test with the full bun test. It never starts an iteration whose estimate (its initial guess or longest observed run) ends past the budget, and records a run still going at the deadline as overrun, not a crash. It writes a closed summary and keeps logs plus bun*.ips reports for a 7-day artifact. It never opens crash-report links. - scripts/ci-bun-crash-census.ts reads every attempt's test job through the caller's gh, downloads only failed-job logs into an owned temp dir that it removes, and prints per-version counts and crashing ids only. - Both use scripts/lib/bun-crash-classify.ts. Co-Authored-By: Claude Opus 5.5 --- .ai/skills/hack-repo-verify/SKILL.md | 3 +- .github/workflows/bun-crash-experiment.yml | 163 ++++++ .github/workflows/ci.yml | 2 +- package.json | 1 + scripts/bun-crash-loop.ts | 519 ++++++++++++++++++ scripts/ci-bun-crash-census.ts | 605 +++++++++++++++++++++ scripts/lib/bun-crash-classify.ts | 135 +++++ tests/bun-crash-loop.test.ts | 402 ++++++++++++++ tests/ci-bun-crash-census.test.ts | 446 +++++++++++++++ 9 files changed, 2274 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/bun-crash-experiment.yml create mode 100644 scripts/bun-crash-loop.ts create mode 100644 scripts/ci-bun-crash-census.ts create mode 100644 scripts/lib/bun-crash-classify.ts create mode 100644 tests/bun-crash-loop.test.ts create mode 100644 tests/ci-bun-crash-census.test.ts diff --git a/.ai/skills/hack-repo-verify/SKILL.md b/.ai/skills/hack-repo-verify/SKILL.md index 63634aa01..39a1657d3 100644 --- a/.ai/skills/hack-repo-verify/SKILL.md +++ b/.ai/skills/hack-repo-verify/SKILL.md @@ -48,7 +48,7 @@ source provenance. Raw `bun test ` remains available for focused tests; it bypasses the suite preflight and does not prepare compiler-dependent fixtures. For release workflow edits, run the pinned syntax/context check also enforced by CI: -`mise x actionlint@1.7.12 -- actionlint -shellcheck= -pyflakes= .github/workflows/ci.yml .github/workflows/release.yml .github/workflows/prerelease.yml`. +`mise x actionlint@1.7.12 -- actionlint -shellcheck= -pyflakes= .github/workflows/ci.yml .github/workflows/release.yml .github/workflows/prerelease.yml .github/workflows/bun-crash-experiment.yml`. This validates event globs and reusable workflow contexts; Bun YAML parsing alone does not. `bun run privacy:check` scans Git-tracked files. Stage reviewed new source/tests @@ -70,6 +70,7 @@ untracked candidate additions. Keep private artifacts excluded from the index. | Native authored command slices | `tests/native-authored-command.test.ts` (unsupported matrix), `tests/native-authored-command-dispatch.test.ts`, `tests/native-authored-project-{start,run,observe,logs,exec,recovery}.test.ts` and the [command contract](../../../docs/reference/native-authored-commands.md); the dispatch "source CLI" cases need the prepared compiler (`bun run build:config-compiler` or absolute `HACK_CONFIG_COMPILER_BINARY`) and fail without it; `native-authored-project-start` and `native-authored-project-exec` are nondeterministic on a local macOS host, so a single local pass or failure is not proof; live acceptance runs host-serialized on one prepared macOS provider slot | | Native VM/routing/reclamation | [candidate guide](../../../docs/guides/native-candidate.md), relevant acceptance gate and isolated host fixture; Linux checks cannot qualify macOS effects | | Resource/performance claims | [performance skill](../hack-repo-performance/SKILL.md), matched workload and measured boundaries | +| Bun runtime crashes | Read-only history: `bun run ci:bun-crash-census -- --since --until ` (failed `test` job logs; `--scan-passing` for all). Matched reproduction: the gated [experiment](../../../.github/workflows/bun-crash-experiment.yml), which spends hosted macOS minutes and needs an explicit go-ahead; never loop on a shared host. Both share [the classifier](../../../scripts/lib/bun-crash-classify.ts) | Read [CI](../../../.github/workflows/ci.yml) for current hosted gates. A local pass does not mean CI ran. Existing unsupported remote/gateway/native-app checks are diff --git a/.github/workflows/bun-crash-experiment.yml b/.github/workflows/bun-crash-experiment.yml new file mode 100644 index 000000000..d7c60965c --- /dev/null +++ b/.github/workflows/bun-crash-experiment.yml @@ -0,0 +1,163 @@ +name: Bun crash experiment + +# HACK-1211 instrument. Nothing here runs on push or on ordinary pull-request +# activity. A run starts only from a manual dispatch (which GitHub registers once +# this file is on the default branch) or when someone adds the +# `bun-crash-experiment` label to a pull request. Every run spends hosted macOS +# minutes (about budget + 15 per Bun version), so start one only with an explicit +# go-ahead. +# +# A labeled pull request runs the matched pair on the pull request's head: the +# 1.3.9 baseline and the 1.4.2 candidate pinned on next, with the same job and +# budget. A dispatch runs one Bun version at a chosen source revision. +on: + workflow_dispatch: + inputs: + bun_version: + description: Bun version to loop under (major.minor.patch) + type: string + default: "1.3.9" + source_revision: + description: Full 40-character commit to test; empty means the dispatched commit + type: string + default: "" + budget_minutes: + description: Loop budget in minutes (5-240); the job timeout is budget + 15 + type: number + default: 30 + pull_request: + types: [labeled] + +permissions: + contents: read + +concurrency: + group: bun-crash-experiment-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: false + +jobs: + plan: + if: ${{ github.event_name == 'workflow_dispatch' || github.event.label.name == 'bun-crash-experiment' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + outputs: + bun_versions: ${{ steps.plan.outputs.bun_versions }} + source_revision: ${{ steps.plan.outputs.source_revision }} + budget_seconds: ${{ steps.plan.outputs.budget_seconds }} + timeout_minutes: ${{ steps.plan.outputs.timeout_minutes }} + steps: + - name: Validate inputs and plan the runs + id: plan + env: + EVENT_NAME: ${{ github.event_name }} + INPUT_BUN_VERSION: ${{ inputs.bun_version }} + INPUT_SOURCE_REVISION: ${{ inputs.source_revision }} + INPUT_BUDGET_MINUTES: ${{ inputs.budget_minutes }} + PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + EVENT_SHA: ${{ github.sha }} + run: | + set -euo pipefail + if [ "$EVENT_NAME" = "workflow_dispatch" ]; then + if ! [[ "$INPUT_BUN_VERSION" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then + echo "Invalid bun_version" + exit 1 + fi + versions="[\"$INPUT_BUN_VERSION\"]" + revision="${INPUT_SOURCE_REVISION:-$EVENT_SHA}" + budget="$INPUT_BUDGET_MINUTES" + else + versions='["1.3.9","1.4.2"]' + revision="$PR_HEAD_SHA" + budget=30 + fi + if ! [[ "$revision" =~ ^[0-9a-f]{40}$ ]]; then + echo "Invalid source_revision" + exit 1 + fi + if ! [[ "$budget" =~ ^[1-9][0-9]{0,2}$ ]] || [ "$budget" -lt 5 ] || [ "$budget" -gt 240 ]; then + echo "Invalid budget_minutes" + exit 1 + fi + { + echo "bun_versions=$versions" + echo "source_revision=$revision" + echo "budget_seconds=$((budget * 60))" + echo "timeout_minutes=$((budget + 15))" + } >> "$GITHUB_OUTPUT" + + loop: + name: Bun crash loop (${{ matrix.bun_version }}) + needs: plan + strategy: + fail-fast: false + matrix: + bun_version: ${{ fromJSON(needs.plan.outputs.bun_versions) }} + runs-on: blacksmith-6vcpu-macos-15 + timeout-minutes: ${{ fromJSON(needs.plan.outputs.timeout_minutes) }} + permissions: + contents: read + steps: + # The instrument comes from this workflow's commit, so an older source + # revision without these scripts can still be measured. + - name: Checkout instrument + uses: actions/checkout@v4 + with: + path: .bun-crash-instrument + persist-credentials: false + sparse-checkout: | + scripts/bun-crash-loop.ts + scripts/lib/bun-crash-classify.ts + sparse-checkout-cone-mode: false + - name: Move the instrument outside the tested tree + run: | + mkdir -p "$RUNNER_TEMP/bun-crash-instrument" + mv .bun-crash-instrument/scripts "$RUNNER_TEMP/bun-crash-instrument/scripts" + rm -rf .bun-crash-instrument + - name: Checkout source revision + uses: actions/checkout@v4 + with: + ref: ${{ needs.plan.outputs.source_revision }} + persist-credentials: false + - name: Setup Bun + uses: oven-sh/setup-bun@v1 + with: + bun-version: ${{ matrix.bun_version }} + - name: Install dependencies + run: bun install --frozen-lockfile + - uses: dtolnay/rust-toolchain@1.97.1 + - name: Prepare matching compiler for private file owner tests + run: | + if grep -q '"build:config-compiler":' package.json; then + bun run build:config-compiler + else + echo "Source revision has no config compiler build" + fi + - name: Check test preconditions + run: | + if grep -q '"test:preflight":' package.json; then + bun run test:preflight + else + echo "Source revision has no test preflight" + fi + - name: Loop the owner test and the full suite within the budget + env: + BUN_ENABLE_CRASH_REPORTING: "0" + EXPECTED_BUN_VERSION: ${{ matrix.bun_version }} + SOURCE_REVISION: ${{ needs.plan.outputs.source_revision }} + BUDGET_SECONDS: ${{ needs.plan.outputs.budget_seconds }} + run: >- + bun "$RUNNER_TEMP/bun-crash-instrument/scripts/bun-crash-loop.ts" + --budget-seconds="$BUDGET_SECONDS" + --expect-bun-version="$EXPECTED_BUN_VERSION" + --source-revision="$SOURCE_REVISION" + --out-dir="$RUNNER_TEMP/bun-crash-loop" + - name: Preserve iteration logs and Bun crash reports + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: bun-crash-loop-${{ matrix.bun_version }} + path: ${{ runner.temp }}/bun-crash-loop + if-no-files-found: ignore + retention-days: 7 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 457db89f4..1bef84117 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -130,7 +130,7 @@ jobs: fi sleep "$((attempt * 5))" done - mise x actionlint@1.7.12 -- actionlint -shellcheck= -pyflakes= .github/workflows/ci.yml .github/workflows/release.yml .github/workflows/prerelease.yml + mise x actionlint@1.7.12 -- actionlint -shellcheck= -pyflakes= .github/workflows/ci.yml .github/workflows/release.yml .github/workflows/prerelease.yml .github/workflows/bun-crash-experiment.yml - name: Fetch pinned model checker run: curl --fail --location --retry 2 --max-time 60 https://github.com/tlaplus/tlaplus/releases/download/v1.7.4/tla2tools.jar --output "$RUNNER_TEMP/tla2tools.jar" - name: Check evidence validation and models diff --git a/package.json b/package.json index 73053cacc..7c3234d4d 100644 --- a/package.json +++ b/package.json @@ -71,6 +71,7 @@ "test:e2e:gateway": "bun run scripts/gateway-e2e.ts", "test:e2e:dispatch": "bun run scripts/dispatch-e2e.ts", "test:e2e:railway": "bun run scripts/railway-node-e2e.ts", + "ci:bun-crash-census": "bun scripts/ci-bun-crash-census.ts", "docs:cli-reference": "bun run scripts/generate-cli-reference.ts", "generate:agent-plugins": "bun run scripts/generate-agent-plugins.ts" }, diff --git a/scripts/bun-crash-loop.ts b/scripts/bun-crash-loop.ts new file mode 100644 index 000000000..220aa1d21 --- /dev/null +++ b/scripts/bun-crash-loop.ts @@ -0,0 +1,519 @@ +#!/usr/bin/env bun +/** + * Fixed-budget Bun crash loop for the gated `bun-crash-experiment` workflow. + * + * It alternates `bun test tests/native-https-owner.test.ts` with the full + * `bun test`, classifies every run with the shared classifier, keeps each run's + * log, and writes a closed, value-free summary. + * + * Budget invariant: an iteration starts only when its estimated duration (the + * larger of its initial estimate and its longest observed run) ends at or before + * the deadline. An iteration that is still running at the deadline is stopped and + * recorded as `overrun`, never as a crash, unless its log carries the crash banner. + * + * The loop never opens or fetches crash-report links. It sets + * `BUN_ENABLE_CRASH_REPORTING=0` so Bun does not upload a report itself; the logs + * and macOS `.ips` reports stay local for offline decoding. + * + * Do not run this on a shared host: it is meant for an isolated hosted runner. + */ +import { closeSync, openSync } from "node:fs"; +import { + appendFile, + copyFile, + mkdir, + readdir, + readFile, + stat, +} from "node:fs/promises"; +import { homedir, tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { + type BunRunOutcome, + type BunRunReason, + classifyBunRun, + closedBunVersion, + closedSignalName, + hasBunCrashBanner, +} from "./lib/bun-crash-classify.ts"; + +export type LoopCommandId = "focused" | "full"; + +export interface LoopCommand { + readonly id: LoopCommandId; + readonly argv: readonly string[]; + readonly initialEstimateMs: number; +} + +export type IterationOutcome = BunRunOutcome | "overrun"; +export type IterationReason = BunRunReason | "deadline"; + +export interface ExecutedRun { + readonly exitCode: number | null; + readonly signal: string | null; + readonly output: string; + /** The harness stopped the run at the deadline. */ + readonly stoppedAtDeadline: boolean; +} + +export interface IterationRecord { + readonly index: number; + readonly command: LoopCommandId; + readonly outcome: IterationOutcome; + readonly reason: IterationReason; + readonly exitCode: number | null; + readonly signal: string | null; + readonly durationMs: number; +} + +export interface LoopResult { + readonly budgetMs: number; + readonly elapsedMs: number; + readonly iterations: readonly IterationRecord[]; + readonly estimatesMs: Readonly>; +} + +export const DEFAULT_FOCUSED_ESTIMATE_SECONDS = 120; +export const DEFAULT_FULL_ESTIMATE_SECONDS = 600; +const MIN_BUDGET_SECONDS = 60; +const MAX_BUDGET_SECONDS = 6 * 60 * 60; +const KILL_GRACE_MS = 10_000; +const REVISION = /^[0-9a-f]{40}$/; + +/** A closed, value-free diagnostic the loop may print. */ +export class LoopError extends Error {} + +export function loopCommands(opts: { + readonly bunExecutable: string; + readonly focusedEstimateSeconds: number; + readonly fullEstimateSeconds: number; +}): readonly LoopCommand[] { + return [ + { + id: "focused", + argv: [opts.bunExecutable, "test", "tests/native-https-owner.test.ts"], + initialEstimateMs: opts.focusedEstimateSeconds * 1000, + }, + { + id: "full", + argv: [opts.bunExecutable, "test"], + initialEstimateMs: opts.fullEstimateSeconds * 1000, + }, + ]; +} + +/** + * Picks the next command, or null when none can finish by the deadline. + * The command at `cursor` is preferred; when it does not fit, the next command in + * sequence order that fits runs instead. Equality with the deadline fits. + */ +export function planNextCommand(opts: { + readonly nowMs: number; + readonly deadlineMs: number; + readonly commands: readonly LoopCommand[]; + readonly cursor: number; + readonly estimatesMs: Readonly>; +}): { readonly command: LoopCommand; readonly cursor: number } | null { + const count = opts.commands.length; + if (count === 0 || opts.nowMs >= opts.deadlineMs) { + return null; + } + const preferred = ((opts.cursor % count) + count) % count; + for (let offset = 0; offset < count; offset += 1) { + const command = opts.commands[(preferred + offset) % count]; + if ( + command !== undefined && + opts.nowMs + opts.estimatesMs[command.id] <= opts.deadlineMs + ) { + return { command, cursor: preferred + 1 }; + } + } + return null; +} + +function classifyIteration(run: ExecutedRun): { + readonly outcome: IterationOutcome; + readonly reason: IterationReason; +} { + if (run.stoppedAtDeadline) { + return hasBunCrashBanner(run.output) + ? { outcome: "crash", reason: "banner" } + : { outcome: "overrun", reason: "deadline" }; + } + return classifyBunRun(run); +} + +/** Runs commands until none can finish within the budget. */ +export async function runBunCrashLoop(opts: { + readonly budgetMs: number; + readonly commands: readonly LoopCommand[]; + readonly now: () => number; + readonly execute: ( + command: LoopCommand, + context: { readonly index: number; readonly deadlineMs: number } + ) => Promise; + readonly onIteration?: (record: IterationRecord) => void; +}): Promise { + const startMs = opts.now(); + const deadlineMs = startMs + opts.budgetMs; + const estimatesMs = {} as Record; + for (const command of opts.commands) { + estimatesMs[command.id] = command.initialEstimateMs; + } + const iterations: IterationRecord[] = []; + let cursor = 0; + for (;;) { + const next = planNextCommand({ + nowMs: opts.now(), + deadlineMs, + commands: opts.commands, + cursor, + estimatesMs, + }); + if (next === null) { + break; + } + cursor = next.cursor; + const index = iterations.length + 1; + const started = opts.now(); + const run = await opts.execute(next.command, { index, deadlineMs }); + const durationMs = Math.max(0, opts.now() - started); + estimatesMs[next.command.id] = Math.max( + estimatesMs[next.command.id], + durationMs + ); + const { outcome, reason } = classifyIteration(run); + const record: IterationRecord = { + index, + command: next.command.id, + outcome, + reason, + exitCode: + typeof run.exitCode === "number" && Number.isInteger(run.exitCode) + ? run.exitCode + : null, + signal: run.stoppedAtDeadline ? null : closedSignalName(run.signal), + durationMs, + }; + iterations.push(record); + opts.onIteration?.(record); + } + return { + budgetMs: opts.budgetMs, + elapsedMs: Math.max(0, opts.now() - startMs), + iterations, + estimatesMs, + }; +} + +const OUTCOMES: readonly IterationOutcome[] = [ + "pass", + "failure", + "crash", + "overrun", +]; + +export function summarizeIterations( + iterations: readonly IterationRecord[] +): Record> { + const empty = (): Record => ({ + pass: 0, + failure: 0, + crash: 0, + overrun: 0, + }); + const totals = { focused: empty(), full: empty() }; + for (const record of iterations) { + totals[record.command][record.outcome] += 1; + } + return totals; +} + +const seconds = (ms: number): string => (ms / 1000).toFixed(1); + +/** Markdown for `$GITHUB_STEP_SUMMARY`. Only closed tokens, numbers and validated identifiers. */ +export function renderLoopSummary(opts: { + readonly result: LoopResult; + readonly bunVersion: string | null; + readonly sourceRevision: string | null; + readonly diagnosticReports: number; +}): string { + const totals = summarizeIterations(opts.result.iterations); + const lines = [ + "## Bun crash experiment", + "", + "| Field | Value |", + "| --- | --- |", + `| Bun | ${opts.bunVersion ?? "unknown"} |`, + `| Source revision | ${opts.sourceRevision ?? "unknown"} |`, + `| Budget seconds | ${seconds(opts.result.budgetMs)} |`, + `| Loop seconds | ${seconds(opts.result.elapsedMs)} |`, + `| Iterations | ${opts.result.iterations.length} |`, + `| Crash reports (bun*.ips) | ${opts.diagnosticReports} |`, + "", + "| Command | Runs | Pass | Failure | Crash | Overrun | Estimate seconds |", + "| --- | --- | --- | --- | --- | --- | --- |", + ]; + for (const id of ["focused", "full"] as const) { + const row = totals[id]; + const runs = OUTCOMES.reduce((sum, outcome) => sum + row[outcome], 0); + lines.push( + `| ${id} | ${runs} | ${row.pass} | ${row.failure} | ${row.crash} | ${row.overrun} | ${seconds(opts.result.estimatesMs[id] ?? 0)} |` + ); + } + lines.push( + "", + "| # | Command | Outcome | Reason | Exit | Signal | Seconds |", + "| --- | --- | --- | --- | --- | --- | --- |" + ); + for (const record of opts.result.iterations) { + lines.push( + `| ${record.index} | ${record.command} | ${record.outcome} | ${record.reason} | ${record.exitCode ?? "-"} | ${record.signal ?? "-"} | ${seconds(record.durationMs)} |` + ); + } + lines.push( + "", + "Logs and crash reports are in the run artifact. Crash-report links are never opened.", + "" + ); + return lines.join("\n"); +} + +/** Copies `bun*.ips` reports modified at or after `sinceMs` into `destination`. */ +export async function collectDiagnosticReports(opts: { + readonly source: string; + readonly destination: string; + readonly sinceMs: number; +}): Promise { + let names: string[]; + try { + names = await readdir(opts.source); + } catch { + return 0; + } + let copied = 0; + for (const name of names) { + if (!/^bun.*\.ips$/i.test(name)) { + continue; + } + const source = join(opts.source, name); + const info = await stat(source).catch(() => null); + if (!info?.isFile() || info.mtimeMs < opts.sinceMs) { + continue; + } + await mkdir(opts.destination, { recursive: true }); + await copyFile(source, join(opts.destination, name)); + copied += 1; + } + return copied; +} + +export function executeWithLog(opts: { + readonly cwd: string; + readonly logDir: string; +}): ( + command: LoopCommand, + context: { readonly index: number; readonly deadlineMs: number } +) => Promise { + return async (command, context) => { + const logPath = join( + opts.logDir, + `iteration-${String(context.index).padStart(3, "0")}-${command.id}.log` + ); + // Write straight to a file, so a child that keeps the descriptor open cannot block the loop. + const fd = openSync(logPath, "w", 0o600); + let stoppedAtDeadline = false; + let proc: ReturnType; + try { + proc = Bun.spawn([...command.argv], { + cwd: opts.cwd, + env: { ...process.env, BUN_ENABLE_CRASH_REPORTING: "0" }, + stdin: "ignore", + stdout: fd, + stderr: fd, + }); + } catch (error) { + closeSync(fd); + throw error; + } + let killTimer: ReturnType | undefined; + const stopTimer = setTimeout( + () => { + stoppedAtDeadline = true; + proc.kill("SIGTERM"); + killTimer = setTimeout(() => proc.kill("SIGKILL"), KILL_GRACE_MS); + }, + Math.max(0, context.deadlineMs - Date.now()) + ); + try { + await proc.exited; + } finally { + clearTimeout(stopTimer); + if (killTimer !== undefined) { + clearTimeout(killTimer); + } + closeSync(fd); + } + return { + exitCode: proc.exitCode, + signal: proc.signalCode ?? null, + output: await readFile(logPath, "utf8"), + stoppedAtDeadline, + }; + }; +} + +export interface LoopArguments { + readonly budgetSeconds: number; + readonly outDir: string; + readonly expectBunVersion: string | null; + readonly sourceRevision: string | null; + readonly focusedEstimateSeconds: number; + readonly fullEstimateSeconds: number; +} + +function parseSeconds(name: string, value: string, min: number, max: number) { + if (!/^\d{1,6}$/.test(value)) { + throw new LoopError(`Invalid ${name}`); + } + const parsed = Number(value); + if (parsed < min || parsed > max) { + throw new LoopError(`Invalid ${name}`); + } + return parsed; +} + +export function parseLoopArguments(argv: readonly string[]): LoopArguments { + const values = new Map(); + for (const arg of argv) { + const match = /^--([a-z-]+)=(.*)$/.exec(arg); + if (!match?.[1] || match[2] === undefined || values.has(match[1])) { + throw new LoopError("Invalid argument"); + } + values.set(match[1], match[2]); + } + const known = new Set([ + "budget-seconds", + "out-dir", + "expect-bun-version", + "source-revision", + "focused-estimate-seconds", + "full-estimate-seconds", + ]); + for (const key of values.keys()) { + if (!known.has(key)) { + throw new LoopError("Invalid argument"); + } + } + const budget = values.get("budget-seconds"); + if (budget === undefined) { + throw new LoopError("Missing --budget-seconds"); + } + const expect = values.get("expect-bun-version"); + if (expect !== undefined && closedBunVersion(expect) === null) { + throw new LoopError("Invalid --expect-bun-version"); + } + const revision = values.get("source-revision"); + if (revision !== undefined && !REVISION.test(revision)) { + throw new LoopError("Invalid --source-revision"); + } + const outDir = values.get("out-dir"); + if (outDir !== undefined && outDir.length === 0) { + throw new LoopError("Invalid --out-dir"); + } + return { + budgetSeconds: parseSeconds( + "--budget-seconds", + budget, + MIN_BUDGET_SECONDS, + MAX_BUDGET_SECONDS + ), + outDir: resolve(outDir ?? join(tmpdir(), "bun-crash-loop")), + expectBunVersion: expect ?? null, + sourceRevision: revision ?? null, + focusedEstimateSeconds: parseSeconds( + "--focused-estimate-seconds", + values.get("focused-estimate-seconds") ?? + String(DEFAULT_FOCUSED_ESTIMATE_SECONDS), + 1, + MAX_BUDGET_SECONDS + ), + fullEstimateSeconds: parseSeconds( + "--full-estimate-seconds", + values.get("full-estimate-seconds") ?? + String(DEFAULT_FULL_ESTIMATE_SECONDS), + 1, + MAX_BUDGET_SECONDS + ), + }; +} + +async function main(): Promise { + const args = parseLoopArguments(process.argv.slice(2)); + const bunVersion = closedBunVersion(Bun.version); + if (args.expectBunVersion !== null && bunVersion !== args.expectBunVersion) { + throw new LoopError("Running Bun does not match --expect-bun-version"); + } + const logDir = join(args.outDir, "logs"); + await mkdir(logDir, { recursive: true }); + const loopStartMs = Date.now(); + const result = await runBunCrashLoop({ + budgetMs: args.budgetSeconds * 1000, + commands: loopCommands({ + bunExecutable: process.execPath, + focusedEstimateSeconds: args.focusedEstimateSeconds, + fullEstimateSeconds: args.fullEstimateSeconds, + }), + now: Date.now, + execute: executeWithLog({ cwd: process.cwd(), logDir }), + onIteration: (record) => { + process.stdout.write( + `iteration=${record.index} command=${record.command} outcome=${record.outcome} reason=${record.reason} seconds=${seconds(record.durationMs)}\n` + ); + }, + }); + const diagnosticReports = await collectDiagnosticReports({ + source: join(homedir(), "Library", "Logs", "DiagnosticReports"), + destination: join(args.outDir, "diagnostic-reports"), + sinceMs: loopStartMs, + }); + const summary = renderLoopSummary({ + result, + bunVersion, + sourceRevision: args.sourceRevision, + diagnosticReports, + }); + await Bun.write( + join(args.outDir, "summary.json"), + `${JSON.stringify( + { + bunVersion, + sourceRevision: args.sourceRevision, + diagnosticReports, + ...result, + }, + null, + 2 + )}\n` + ); + const stepSummary = process.env.GITHUB_STEP_SUMMARY; + if (stepSummary) { + await appendFile(stepSummary, summary); + } else { + process.stdout.write(summary); + } + if (result.iterations.length === 0) { + throw new LoopError("No iteration fits the budget"); + } +} + +if (import.meta.main) { + try { + await main(); + } catch (error) { + process.stderr.write( + `${error instanceof LoopError ? error.message : "Bun crash loop failed"}\n` + ); + process.exitCode = 1; + } +} diff --git a/scripts/ci-bun-crash-census.ts b/scripts/ci-bun-crash-census.ts new file mode 100644 index 000000000..c6b853687 --- /dev/null +++ b/scripts/ci-bun-crash-census.ts @@ -0,0 +1,605 @@ +#!/usr/bin/env bun +/** + * Read-only census of Bun crashes in hosted CI history. + * + * For a `--since`/`--until` window it lists every run of the CI workflow, reads + * the `test` job of every run attempt, downloads only failed-job logs (all + * completed logs with `--scan-passing`) into an owned temporary directory, and + * classifies each with the shared classifier used by `scripts/bun-crash-loop.ts`. + * + * It uses the caller's existing `gh` authentication and never handles a token. + * It is strictly read-only: no reruns, comments, labels or uploads. It prints + * counts, closed tokens, versions and numeric run/job ids only, never log bodies. + * + * Bun version per job comes from the `Setup Bun` input in the job log when a log + * is read, and otherwise from the workflow file at the run's head commit. For a + * pull-request run the job actually used the merge commit's workflow, so a stale + * pull-request head can attribute a job to the older pin; `versionSources` shows + * how many jobs each source covers. + */ +import { closeSync, openSync, rmSync } from "node:fs"; +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + type BunRunReason, + classifyHostedJobLog, + closedBunVersion, + hostedJobBunVersion, +} from "./lib/bun-crash-classify.ts"; + +/** A closed, value-free diagnostic the census may print. */ +export class CensusError extends Error {} + +/** The GitHub REST reads the census needs, through the caller's `gh` authentication. */ +export interface GhReader { + /** Paginated GET; one JSON value per output line of `jq`. */ + readonly lines: (path: string, jq: string) => Promise; + /** One GET returning JSON. */ + readonly json: (path: string) => Promise; + /** GET whose body is written to `destination`. */ + readonly download: (path: string, destination: string) => Promise; +} + +export type JobConclusion = + | "success" + | "failure" + | "timed_out" + | "cancelled" + | "other"; + +export type VersionSource = "log" | "workflow" | "unknown"; + +export interface JobRecord { + readonly runId: number; + readonly attempt: number; + readonly jobId: number; + readonly conclusion: JobConclusion; + readonly bunVersion: string; + readonly versionSource: VersionSource; + /** Null when no log was read. */ + readonly scan: "crash" | "failure" | "pass" | "unavailable" | null; + readonly reason: BunRunReason | null; +} + +export interface CrashingJob { + readonly runId: number; + readonly attempt: number; + readonly jobId: number; + readonly reason: BunRunReason; +} + +export interface VersionTotals { + readonly bunVersion: string; + executions: number; + successes: number; + failures: number; + cancelled: number; + crashes: number; + logsScanned: number; + logsUnavailable: number; + readonly versionSources: Record; + readonly crashing: CrashingJob[]; +} + +const ISO_INSTANT = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$/; +const REPOSITORY = /^[A-Za-z0-9_.-]{1,100}\/[A-Za-z0-9_.-]{1,100}$/; +const WORKFLOW_FILE = /^[A-Za-z0-9_.-]{1,100}\.ya?ml$/; +const JOB_NAME = /^[A-Za-z0-9_. -]{1,100}$/; +const SHA = /^[0-9a-f]{40}$/; +/** The runs endpoint returns at most 1000 results for one `created` query. */ +const MAX_WINDOW_RUNS = 1000; +const DEFAULT_CONCURRENCY = 6; + +const FAILED: ReadonlySet = new Set(["failure", "timed_out"]); + +function closedConclusion(value: unknown): JobConclusion { + return value === "success" || + value === "failure" || + value === "timed_out" || + value === "cancelled" + ? value + : "other"; +} + +const positiveInteger = (value: unknown): number | null => + typeof value === "number" && Number.isSafeInteger(value) && value > 0 + ? value + : null; + +/** + * The `bun-version` given to the named job in a workflow file, or null. The + * job block runs from ` :` to the next two-space-indented key. + */ +export function workflowJobBunVersion( + workflow: string, + job: string +): string | null { + const lines = workflow.split(/\r?\n/); + const start = lines.findIndex((line) => line.trimEnd() === ` ${job}:`); + if (start === -1) { + return null; + } + for (const line of lines.slice(start + 1)) { + if (/^ {2}\S/.test(line)) { + return null; + } + const match = /^\s+bun-version:\s*["']?([^"'\s]+)["']?\s*$/.exec(line); + if (match) { + return closedBunVersion(match[1]); + } + } + return null; +} + +export function emptyTotals(bunVersion: string): VersionTotals { + return { + bunVersion, + executions: 0, + successes: 0, + failures: 0, + cancelled: 0, + crashes: 0, + logsScanned: 0, + logsUnavailable: 0, + versionSources: { log: 0, workflow: 0, unknown: 0 }, + crashing: [], + }; +} + +/** Per-Bun-version totals, versions in ascending order with `unknown` last. */ +export function aggregateCensus( + records: readonly JobRecord[] +): VersionTotals[] { + const byVersion = new Map(); + for (const record of records) { + const totals = + byVersion.get(record.bunVersion) ?? emptyTotals(record.bunVersion); + byVersion.set(record.bunVersion, totals); + totals.executions += 1; + totals.versionSources[record.versionSource] += 1; + if (record.conclusion === "success") { + totals.successes += 1; + } else if (FAILED.has(record.conclusion)) { + totals.failures += 1; + } else if (record.conclusion === "cancelled") { + totals.cancelled += 1; + } + if (record.scan === "unavailable") { + totals.logsUnavailable += 1; + } else if (record.scan !== null) { + totals.logsScanned += 1; + } + if (record.scan === "crash" && record.reason !== null) { + totals.crashes += 1; + totals.crashing.push({ + runId: record.runId, + attempt: record.attempt, + jobId: record.jobId, + reason: record.reason, + }); + } + } + const order = (version: string): number[] => + version === "unknown" + ? [Number.POSITIVE_INFINITY] + : version.split(".").map(Number); + return [...byVersion.values()].sort((left, right) => { + const a = order(left.bunVersion); + const b = order(right.bunVersion); + for (let index = 0; index < Math.max(a.length, b.length); index += 1) { + const difference = (a[index] ?? 0) - (b[index] ?? 0); + if (difference !== 0) { + return difference; + } + } + return 0; + }); +} + +export interface CensusOptions { + readonly repository: string; + readonly workflow: string; + readonly job: string; + readonly since: string; + readonly until: string; + readonly scanPassing: boolean; + readonly concurrency?: number; +} + +export interface CensusResult { + readonly runs: number; + readonly records: readonly JobRecord[]; + readonly totals: readonly VersionTotals[]; +} + +async function mapConcurrent( + items: readonly T[], + limit: number, + task: (item: T) => Promise +): Promise { + const results = new Array(items.length); + let next = 0; + const worker = async () => { + while (next < items.length) { + const index = next; + next += 1; + results[index] = await task(items[index] as T); + } + }; + await Promise.all( + Array.from({ length: Math.max(1, Math.min(limit, items.length)) }, worker) + ); + return results; +} + +function parseLine(line: string): Record { + let value: unknown; + try { + value = JSON.parse(line); + } catch { + throw new CensusError("Unexpected GitHub response"); + } + if (typeof value !== "object" || value === null || Array.isArray(value)) { + throw new CensusError("Unexpected GitHub response"); + } + return value as Record; +} + +interface ScannedLog { + readonly scan: Exclude; + readonly reason: BunRunReason | null; + readonly bunVersion: string | null; +} + +/** Downloads one job log into `workDir`, classifies it and removes it. */ +async function scanJobLog(opts: { + readonly gh: GhReader; + readonly repo: string; + readonly jobId: number; + readonly workDir: string; +}): Promise { + const destination = join(opts.workDir, `job-${opts.jobId}.log`); + try { + await opts.gh.download( + `repos/${opts.repo}/actions/jobs/${opts.jobId}/logs`, + destination + ); + const log = await readFile(destination, "utf8"); + const classification = classifyHostedJobLog(log); + return { + scan: classification.outcome, + reason: classification.reason, + bunVersion: hostedJobBunVersion(log), + }; + } catch { + return { scan: "unavailable", reason: null, bunVersion: null }; + } finally { + await rm(destination, { force: true }); + } +} + +function jobVersion( + fromLog: string | null, + fromWorkflow: string | null +): { readonly bunVersion: string; readonly versionSource: VersionSource } { + if (fromLog !== null) { + return { bunVersion: fromLog, versionSource: "log" }; + } + if (fromWorkflow !== null) { + return { bunVersion: fromWorkflow, versionSource: "workflow" }; + } + return { bunVersion: "unknown", versionSource: "unknown" }; +} + +/** + * Runs the census. Logs are written only under `workDir`, one at a time per + * worker, and each is removed as soon as it is classified. + */ +export async function runCensus( + options: CensusOptions, + gh: GhReader, + workDir: string +): Promise { + const repo = options.repository; + const created = `${options.since}..${options.until}`; + const runsPath = `repos/${repo}/actions/workflows/${options.workflow}/runs?created=${created}`; + const head = await gh.json(`${runsPath}&per_page=1`); + const total = + typeof head === "object" && head !== null && "total_count" in head + ? head.total_count + : null; + if (typeof total !== "number" || !Number.isSafeInteger(total) || total < 0) { + throw new CensusError("Unexpected GitHub response"); + } + if (total > MAX_WINDOW_RUNS) { + throw new CensusError("Window has more than 1000 runs; narrow it"); + } + const runs: { readonly id: number; readonly headSha: string | null }[] = []; + for (const line of await gh.lines( + `${runsPath}&per_page=100`, + ".workflow_runs[] | {id, head_sha}" + )) { + const run = parseLine(line); + const id = positiveInteger(run.id); + if (id === null) { + throw new CensusError("Unexpected GitHub response"); + } + const sha = + typeof run.head_sha === "string" && SHA.test(run.head_sha) + ? run.head_sha + : null; + if (!runs.some((existing) => existing.id === id)) { + runs.push({ id, headSha: sha }); + } + } + + const workflowVersions = new Map>(); + const workflowVersion = (sha: string | null): Promise => { + if (sha === null) { + return Promise.resolve(null); + } + const cached = workflowVersions.get(sha); + if (cached) { + return cached; + } + const pending = gh + .json( + `repos/${repo}/contents/.github/workflows/${options.workflow}?ref=${sha}` + ) + .then((value) => { + if ( + typeof value !== "object" || + value === null || + !("content" in value) || + typeof value.content !== "string" + ) { + return null; + } + return workflowJobBunVersion( + Buffer.from(value.content, "base64").toString("utf8"), + options.job + ); + }) + .catch(() => null); + workflowVersions.set(sha, pending); + return pending; + }; + + const perRun = await mapConcurrent( + runs, + options.concurrency ?? DEFAULT_CONCURRENCY, + async (run) => { + const seen = new Set(); + const records: JobRecord[] = []; + const lines = await gh.lines( + `repos/${repo}/actions/runs/${run.id}/jobs?filter=all&per_page=100`, + ".jobs[] | {id, name, run_attempt, status, conclusion}" + ); + for (const line of lines) { + const job = parseLine(line); + const jobId = positiveInteger(job.id); + const attempt = positiveInteger(job.run_attempt); + if ( + job.name !== options.job || + job.status !== "completed" || + job.conclusion === "skipped" || + jobId === null || + attempt === null || + seen.has(jobId) + ) { + continue; + } + seen.add(jobId); + const conclusion = closedConclusion(job.conclusion); + const scanned = + FAILED.has(conclusion) || + (options.scanPassing && conclusion !== "other") + ? await scanJobLog({ gh, repo, jobId, workDir }) + : null; + const fromWorkflow = scanned?.bunVersion + ? null + : await workflowVersion(run.headSha); + records.push({ + runId: run.id, + attempt, + jobId, + conclusion, + ...jobVersion(scanned?.bunVersion ?? null, fromWorkflow), + scan: scanned?.scan ?? null, + reason: scanned?.reason ?? null, + }); + } + return records; + } + ); + const records = perRun.flat(); + return { runs: runs.length, records, totals: aggregateCensus(records) }; +} + +/** Closed, value-free text report. */ +export function renderCensus(opts: { + readonly options: CensusOptions; + readonly result: CensusResult; +}): string { + const { options, result } = opts; + const lines = [ + `Bun crash census repo=${options.repository} workflow=${options.workflow} job=${options.job} since=${options.since} until=${options.until} runs=${result.runs} jobs=${result.records.length} scan_passing=${options.scanPassing}`, + ]; + for (const totals of result.totals) { + const sources = Object.entries(totals.versionSources) + .map(([source, count]) => `${source}:${count}`) + .join(","); + lines.push( + `bun=${totals.bunVersion} executions=${totals.executions} successes=${totals.successes} failures=${totals.failures} cancelled=${totals.cancelled} crashes=${totals.crashes} logs_scanned=${totals.logsScanned} logs_unavailable=${totals.logsUnavailable} version_sources=${sources}` + ); + for (const crash of totals.crashing) { + lines.push( + ` crash run=${crash.runId} attempt=${crash.attempt} job=${crash.jobId} reason=${crash.reason}` + ); + } + } + return `${lines.join("\n")}\n`; +} + +const VALUED_ARGUMENTS = new Set(["since", "until", "repo", "workflow", "job"]); +const BOOLEAN_ARGUMENTS = new Set(["scan-passing", "json"]); + +function tokenizeCensusArguments(argv: readonly string[]): { + readonly values: Map; + readonly flags: Set; +} { + const values = new Map(); + const flags = new Set(); + for (let index = 0; index < argv.length; index += 1) { + const match = /^--([a-z-]+)(?:=(.*))?$/.exec(argv[index] ?? ""); + const key = match?.[1]; + let value = match?.[2]; + if (key && BOOLEAN_ARGUMENTS.has(key) && value === undefined) { + if (flags.has(key)) { + throw new CensusError("Invalid argument"); + } + flags.add(key); + continue; + } + if (!(key && VALUED_ARGUMENTS.has(key)) || values.has(key)) { + throw new CensusError("Invalid argument"); + } + if (value === undefined) { + index += 1; + value = argv[index]; + } + if (value === undefined) { + throw new CensusError("Invalid argument"); + } + values.set(key, value); + } + return { values, flags }; +} + +export function parseCensusArguments(argv: readonly string[]): CensusOptions & { + readonly json: boolean; +} { + const { values, flags } = tokenizeCensusArguments(argv); + const since = values.get("since"); + const until = values.get("until"); + if ( + !(since && until && ISO_INSTANT.test(since) && ISO_INSTANT.test(until)) || + Number.isNaN(Date.parse(since)) || + Number.isNaN(Date.parse(until)) || + Date.parse(since) >= Date.parse(until) + ) { + throw new CensusError( + "--since and --until must be UTC instants (YYYY-MM-DDTHH:MM:SSZ) with since before until" + ); + } + const repository = values.get("repo") ?? "hack-dance/hack"; + const workflow = values.get("workflow") ?? "ci.yml"; + const job = values.get("job") ?? "test"; + if ( + !( + REPOSITORY.test(repository) && + WORKFLOW_FILE.test(workflow) && + JOB_NAME.test(job) + ) + ) { + throw new CensusError("Invalid argument"); + } + return { + repository, + workflow, + job, + since, + until, + scanPassing: flags.has("scan-passing"), + json: flags.has("json"), + }; +} + +async function gh( + args: readonly string[], + stdout: "pipe" | number +): Promise { + const proc = Bun.spawn(["gh", ...args], { + stdin: "ignore", + stdout, + stderr: "ignore", + }); + const text = + stdout === "pipe" && proc.stdout instanceof ReadableStream + ? await new Response(proc.stdout).text() + : ""; + if ((await proc.exited) !== 0) { + throw new CensusError("gh api request failed"); + } + return text; +} + +/** `gh api` reader; GET only, with the caller's existing authentication. */ +export const ghCliReader: GhReader = { + lines: async (path, jq) => + ( + await gh( + ["api", "--method", "GET", "--paginate", "--jq", jq, path], + "pipe" + ) + ) + .split("\n") + .filter((line) => line.length > 0), + json: async (path) => { + try { + return JSON.parse(await gh(["api", "--method", "GET", path], "pipe")); + } catch (error) { + throw error instanceof CensusError + ? error + : new CensusError("Unexpected GitHub response"); + } + }, + download: async (path, destination) => { + const fd = openSync(destination, "wx", 0o600); + try { + // Logs carry terminal colors; the body goes to the owned file, never the terminal. + await gh( + ["api", "--method", "GET", "--allow-escape-sequences", path], + fd + ); + } finally { + closeSync(fd); + } + }, +}; + +async function main(): Promise { + const { json, ...options } = parseCensusArguments(process.argv.slice(2)); + const workDir = await mkdtemp(join(tmpdir(), "hack-bun-census-")); + const cleanup = () => rmSync(workDir, { recursive: true, force: true }); + const interrupt = () => { + cleanup(); + process.exit(130); + }; + process.once("SIGINT", interrupt); + process.once("SIGTERM", interrupt); + try { + const result = await runCensus(options, ghCliReader, workDir); + process.stdout.write( + json + ? `${JSON.stringify({ options, runs: result.runs, totals: result.totals }, null, 2)}\n` + : renderCensus({ options, result }) + ); + } finally { + process.off("SIGINT", interrupt); + process.off("SIGTERM", interrupt); + await rm(workDir, { recursive: true, force: true }); + } +} + +if (import.meta.main) { + try { + await main(); + } catch (error) { + process.stderr.write( + `${error instanceof CensusError ? error.message : "Bun crash census failed"}\n` + ); + process.exitCode = 1; + } +} diff --git a/scripts/lib/bun-crash-classify.ts b/scripts/lib/bun-crash-classify.ts new file mode 100644 index 000000000..443176811 --- /dev/null +++ b/scripts/lib/bun-crash-classify.ts @@ -0,0 +1,135 @@ +/** + * Shared, pure classifier for one Bun process run or one hosted job log. + * + * The bounded crash-loop harness (`scripts/bun-crash-loop.ts`) and the read-only + * hosted census (`scripts/ci-bun-crash-census.ts`) both classify with this module, + * so a crash means the same thing in an experiment and in hosted history. + * + * Results are closed tokens only. Nothing from the output (paths, test names, + * crash-report links or values) is ever returned. + */ + +/** Exit statuses a shell reports for Bun's crash signals: SIGTRAP (133) and SIGSEGV (139). */ +export const BUN_CRASH_EXIT_CODES: readonly number[] = [133, 139]; + +export type BunRunOutcome = "crash" | "failure" | "pass"; + +export type BunRunReason = + /** The output carries Bun's own crash banner. */ + | "banner" + /** The process was ended by a signal it did not get from the harness. */ + | "signal" + /** The exit status encodes a crash signal (133 or 139). */ + | "signal_exit" + /** Exit 0 and no crash banner. */ + | "exit_zero" + /** A non-zero exit with `bun test` failure markers. */ + | "test_failure" + /** Any other non-zero exit status, for example a typecheck exit 2. */ + | "exit_code" + /** No exit status and no signal were observed. */ + | "missing_exit"; + +export interface BunRunClassification { + readonly outcome: BunRunOutcome; + readonly reason: BunRunReason; +} + +export interface BunRunObservation { + readonly exitCode: number | null; + readonly signal: string | null; + readonly output: string; +} + +const CRASH_BANNERS = [/Bun has crashed/, /panic\(main thread\)/]; +// CSI (ESC [ ... final), OSC (ESC ] ... BEL or ST) and two-byte ESC sequences. +const ANSI_ESCAPES = + /\u001b\[[0-?]*[ -/]*[@-~]|\u001b\][^\u0007\u001b]*(?:\u0007|\u001b\\)|\u001b[@-Z\\-_]/g; +const TEST_FAILURE_LINE = /(?:^|\s)\(fail\)\s/m; +const TEST_FAILURE_SUMMARY = /(?:^|\s)[1-9]\d* fail\s*$/m; +const HOSTED_EXIT = /##\[error\]Process completed with exit code (\d{1,3})\./g; +const SIGNAL_NAME = /^SIG[A-Z0-9]{2,10}$/; +const BUN_VERSION = /^\d{1,3}\.\d{1,3}\.\d{1,3}$/; +const HOSTED_BUN_VERSION_INPUT = + /^(?:\S+Z )?\s*bun-version: "?(\d{1,3}\.\d{1,3}\.\d{1,3})"?\s*$/m; + +/** Removes terminal escape sequences so banners and markers match colored output. */ +export function stripAnsi(text: string): string { + return text.replace(ANSI_ESCAPES, ""); +} + +/** True when the output carries Bun's crash banner, colored or not. */ +export function hasBunCrashBanner(output: string): boolean { + const plain = stripAnsi(output); + return CRASH_BANNERS.some((banner) => banner.test(plain)); +} + +/** + * Classifies one run. A crash banner, a signal, or a crash exit status is a crash + * whatever else the run printed; exit 0 is a pass; every other result is a + * non-crash failure. + */ +export function classifyBunRun( + observation: BunRunObservation +): BunRunClassification { + const plain = stripAnsi(observation.output); + if (CRASH_BANNERS.some((banner) => banner.test(plain))) { + return { outcome: "crash", reason: "banner" }; + } + if (observation.signal !== null) { + return { outcome: "crash", reason: "signal" }; + } + if (observation.exitCode === null) { + return { outcome: "failure", reason: "missing_exit" }; + } + if (BUN_CRASH_EXIT_CODES.includes(observation.exitCode)) { + return { outcome: "crash", reason: "signal_exit" }; + } + if (observation.exitCode === 0) { + return { outcome: "pass", reason: "exit_zero" }; + } + if (TEST_FAILURE_LINE.test(plain) || TEST_FAILURE_SUMMARY.test(plain)) { + return { outcome: "failure", reason: "test_failure" }; + } + return { outcome: "failure", reason: "exit_code" }; +} + +/** + * The exit status of the step that failed a hosted job, from the runner's + * `Process completed with exit code N.` annotation, or null when there is none + * (for example a cancelled or timed-out job). + */ +export function hostedJobExitCode(log: string): number | null { + let last: number | null = null; + for (const match of stripAnsi(log).matchAll(HOSTED_EXIT)) { + last = Number(match[1]); + } + return last; +} + +/** The Bun version the hosted `Setup Bun` step was given, or null. */ +export function hostedJobBunVersion(log: string): string | null { + const match = HOSTED_BUN_VERSION_INPUT.exec(stripAnsi(log)); + return match?.[1] ?? null; +} + +/** Classifies a hosted job log with the same rules as a local run. */ +export function classifyHostedJobLog(log: string): BunRunClassification { + return classifyBunRun({ + exitCode: hostedJobExitCode(log), + signal: null, + output: log, + }); +} + +/** A closed signal name, or null for anything that is not one. */ +export function closedSignalName(signal: unknown): string | null { + return typeof signal === "string" && SIGNAL_NAME.test(signal) ? signal : null; +} + +/** A closed `major.minor.patch` Bun version, or null for anything else. */ +export function closedBunVersion(version: unknown): string | null { + return typeof version === "string" && BUN_VERSION.test(version) + ? version + : null; +} diff --git a/tests/bun-crash-loop.test.ts b/tests/bun-crash-loop.test.ts new file mode 100644 index 000000000..2f575c7d6 --- /dev/null +++ b/tests/bun-crash-loop.test.ts @@ -0,0 +1,402 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { mkdtemp, readdir, rm, utimes, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + collectDiagnosticReports, + type ExecutedRun, + executeWithLog, + type LoopCommand, + loopCommands, + parseLoopArguments, + planNextCommand, + renderLoopSummary, + runBunCrashLoop, +} from "../scripts/bun-crash-loop.ts"; +import { + classifyBunRun, + hasBunCrashBanner, + stripAnsi, +} from "../scripts/lib/bun-crash-classify.ts"; + +const CRASH_TRANSCRIPT = [ + "tests/native-https-owner.test.ts:", + "(pass) owner starts [12.00ms]", + "Bun v1.3.9 (cf6cdbbb) macOS Silicon", + "panic(main thread): Segmentation fault at address 0x80", + "oh no: Bun has crashed. This indicates a bug in Bun, not your code.", + " https://bun.report/1.3.9/Mt1cf6cdbbAAAA", +].join("\n"); + +const roots: string[] = []; +afterEach(async () => { + await Promise.all( + roots.splice(0).map((root) => rm(root, { recursive: true, force: true })) + ); +}); + +async function tempRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "hack-bun-crash-loop-")); + roots.push(root); + return root; +} + +describe("classifyBunRun", () => { + test("a crash-banner transcript is a crash even when the exit looks ordinary", () => { + expect( + classifyBunRun({ exitCode: 1, signal: null, output: CRASH_TRANSCRIPT }) + ).toEqual({ outcome: "crash", reason: "banner" }); + expect( + classifyBunRun({ exitCode: 0, signal: null, output: CRASH_TRANSCRIPT }) + ).toEqual({ outcome: "crash", reason: "banner" }); + }); + + test("exit 133 or 139 without a banner is a signal crash", () => { + for (const exitCode of [133, 139]) { + expect(classifyBunRun({ exitCode, signal: null, output: "" })).toEqual({ + outcome: "crash", + reason: "signal_exit", + }); + } + }); + + test("a signal is a crash", () => { + expect( + classifyBunRun({ exitCode: null, signal: "SIGSEGV", output: "" }) + ).toEqual({ outcome: "crash", reason: "signal" }); + }); + + test("(fail) with exit 1 is a non-crash test failure", () => { + expect( + classifyBunRun({ + exitCode: 1, + signal: null, + output: "(fail) owner starts [5.00ms]\n\n 3 pass\n 1 fail\n", + }) + ).toEqual({ outcome: "failure", reason: "test_failure" }); + }); + + test("a typecheck exit 2 is a non-crash failure", () => { + expect( + classifyBunRun({ + exitCode: 2, + signal: null, + output: "src/index.ts(1,1): error TS2304: Cannot find name 'x'.\n", + }) + ).toEqual({ outcome: "failure", reason: "exit_code" }); + }); + + test("exit 0 is a pass", () => { + expect( + classifyBunRun({ + exitCode: 0, + signal: null, + output: " 12 pass\n 0 fail\n", + }) + ).toEqual({ outcome: "pass", reason: "exit_zero" }); + }); + + test("no exit status and no signal is a failure, not a pass", () => { + expect( + classifyBunRun({ exitCode: null, signal: null, output: "" }) + ).toEqual({ outcome: "failure", reason: "missing_exit" }); + }); + + test("ANSI-escaped banners and failure markers still match", () => { + const colored = + "\u001b[31mpanic\u001b[0m(main thread): Segmentation fault\n\u001b]8;;https://example.invalid\u0007link\u001b]8;;\u0007\n"; + expect(stripAnsi(colored)).toContain("panic(main thread)"); + expect(hasBunCrashBanner(colored)).toBe(true); + expect( + classifyBunRun({ exitCode: 1, signal: null, output: colored }) + ).toEqual({ outcome: "crash", reason: "banner" }); + expect( + classifyBunRun({ + exitCode: 1, + signal: null, + output: "\u001b[0m\u001b[31m(fail)\u001b[0m owner starts\n", + }) + ).toEqual({ outcome: "failure", reason: "test_failure" }); + }); +}); + +const commands: readonly LoopCommand[] = [ + { id: "focused", argv: ["focused"], initialEstimateMs: 100_000 }, + { id: "full", argv: ["full"], initialEstimateMs: 300_000 }, +]; + +describe("planNextCommand deadline arithmetic", () => { + const estimatesMs = { focused: 100_000, full: 300_000 }; + + test("a command that ends exactly at the deadline may start", () => { + expect( + planNextCommand({ + nowMs: 700_000, + deadlineMs: 1_000_000, + commands, + cursor: 1, + estimatesMs, + })?.command.id + ).toBe("full"); + }); + + test("a command that would end past the deadline falls back to one that fits", () => { + const next = planNextCommand({ + nowMs: 700_001, + deadlineMs: 1_000_000, + commands, + cursor: 1, + estimatesMs, + }); + expect(next?.command.id).toBe("focused"); + expect(next?.cursor).toBe(2); + }); + + test("nothing starts when no command fits or the deadline has passed", () => { + expect( + planNextCommand({ + nowMs: 900_001, + deadlineMs: 1_000_000, + commands, + cursor: 0, + estimatesMs, + }) + ).toBeNull(); + expect( + planNextCommand({ + nowMs: 1_000_000, + deadlineMs: 1_000_000, + commands, + cursor: 0, + estimatesMs: { focused: 0, full: 0 }, + }) + ).toBeNull(); + }); +}); + +describe("runBunCrashLoop", () => { + test("no iteration starts past the budget and estimates grow to observed durations", async () => { + let clock = 0; + const durations = { focused: 90_000, full: 500_000 }; + const starts: { id: string; at: number }[] = []; + const result = await runBunCrashLoop({ + budgetMs: 1_000_000, + commands, + now: () => clock, + execute: async (command) => { + starts.push({ id: command.id, at: clock }); + clock += durations[command.id]; + return { + exitCode: 0, + signal: null, + output: "", + stoppedAtDeadline: false, + }; + }, + }); + // After the full suite takes 500 s, its estimate is 500 s and it no longer + // fits; focused runs (estimate 100 s) continue until 950 s + 100 s > 1000 s. + expect(starts).toEqual([ + { id: "focused", at: 0 }, + { id: "full", at: 90_000 }, + { id: "focused", at: 590_000 }, + { id: "focused", at: 680_000 }, + { id: "focused", at: 770_000 }, + { id: "focused", at: 860_000 }, + ]); + for (const start of starts) { + const estimate = start.id === "full" ? 300_000 : 100_000; + expect(start.at + estimate).toBeLessThanOrEqual(1_000_000); + } + expect(result.estimatesMs).toEqual({ focused: 100_000, full: 500_000 }); + expect(result.iterations.map((record) => record.outcome)).toEqual( + new Array(6).fill("pass") + ); + }); + + test("a run stopped at the deadline is an overrun unless it printed the crash banner", async () => { + let clock = 0; + const runs: ExecutedRun[] = [ + { + exitCode: null, + signal: "SIGTERM", + output: "", + stoppedAtDeadline: true, + }, + { + exitCode: null, + signal: "SIGTERM", + output: CRASH_TRANSCRIPT, + stoppedAtDeadline: true, + }, + ]; + const result = await runBunCrashLoop({ + budgetMs: 1_000_000, + commands: [commands[0] as LoopCommand], + now: () => clock, + execute: async () => { + clock += 100_000; + return ( + runs.shift() ?? { + exitCode: 0, + signal: null, + output: "", + stoppedAtDeadline: false, + } + ); + }, + }); + expect(result.iterations[0]).toMatchObject({ + outcome: "overrun", + reason: "deadline", + signal: null, + }); + expect(result.iterations[1]).toMatchObject({ + outcome: "crash", + reason: "banner", + }); + }); + + test("the summary is closed and value-free", async () => { + let clock = 0; + const result = await runBunCrashLoop({ + budgetMs: 250_000, + commands, + now: () => clock, + execute: async () => { + clock += 100_000; + return { + exitCode: 133, + signal: null, + output: `TOKEN=do-not-print ${CRASH_TRANSCRIPT}`, + stoppedAtDeadline: false, + }; + }, + }); + const summary = renderLoopSummary({ + result, + bunVersion: "1.3.9", + sourceRevision: "2b8057c1ae6f4311150b6e139a9a919bbc7788d4", + diagnosticReports: 1, + }); + expect(summary).toContain("| focused | 2 | 0 | 0 | 2 | 0 |"); + expect(summary).toContain("| 1 | focused | crash | banner | 133 | - |"); + expect(summary).not.toContain("do-not-print"); + expect(summary).not.toContain("bun.report"); + expect(summary).not.toContain("native-https-owner"); + }); +}); + +describe("executeWithLog", () => { + test("keeps each run's log, reports signals and stops a run at the deadline", async () => { + const root = await tempRoot(); + const execute = executeWithLog({ cwd: root, logDir: root }); + const far = Date.now() + 60_000; + const pass = await execute( + { + id: "focused", + argv: [process.execPath, "-e", "console.log('ok')"], + initialEstimateMs: 1000, + }, + { index: 1, deadlineMs: far } + ); + expect(pass).toMatchObject({ exitCode: 0, stoppedAtDeadline: false }); + expect(pass.output).toContain("ok"); + + const crashed = await execute( + { + id: "focused", + argv: [process.execPath, "-e", "process.kill(process.pid, 'SIGSEGV')"], + initialEstimateMs: 1000, + }, + { index: 2, deadlineMs: far } + ); + expect(crashed.signal).toBe("SIGSEGV"); + expect(classifyBunRun(crashed).outcome).toBe("crash"); + + const stopped = await execute( + { + id: "full", + argv: [process.execPath, "-e", "setTimeout(() => {}, 60_000)"], + initialEstimateMs: 1000, + }, + { index: 3, deadlineMs: Date.now() + 200 } + ); + expect(stopped.stoppedAtDeadline).toBe(true); + expect((await readdir(root)).sort()).toEqual([ + "iteration-001-focused.log", + "iteration-002-focused.log", + "iteration-003-full.log", + ]); + }); +}); + +test("loop commands run the focused owner test and then the full suite", () => { + expect( + loopCommands({ + bunExecutable: "/bin/bun", + focusedEstimateSeconds: 60, + fullEstimateSeconds: 600, + }).map((command) => [command.id, command.argv, command.initialEstimateMs]) + ).toEqual([ + [ + "focused", + ["/bin/bun", "test", "tests/native-https-owner.test.ts"], + 60_000, + ], + ["full", ["/bin/bun", "test"], 600_000], + ]); +}); + +test("loop arguments are validated and closed", () => { + expect( + parseLoopArguments([ + "--budget-seconds=1800", + "--out-dir=/tmp/x", + "--expect-bun-version=1.3.9", + "--source-revision=2b8057c1ae6f4311150b6e139a9a919bbc7788d4", + ]) + ).toMatchObject({ + budgetSeconds: 1800, + outDir: "/tmp/x", + expectBunVersion: "1.3.9", + focusedEstimateSeconds: 120, + fullEstimateSeconds: 600, + }); + for (const argv of [ + [], + ["--budget-seconds=10"], + ["--budget-seconds=abc"], + ["--budget-seconds=1800", "--budget-seconds=1800"], + ["--budget-seconds=1800", "--unknown=1"], + ["--budget-seconds=1800", "--expect-bun-version=latest"], + ["--budget-seconds=1800", "--source-revision=main"], + ]) { + expect(() => parseLoopArguments(argv)).toThrow(); + } +}); + +test("only bun crash reports written during the loop are collected", async () => { + const root = await tempRoot(); + const source = join(root, "DiagnosticReports"); + const destination = join(root, "out"); + await Bun.write(join(source, "bun-2026-10-12-old.ips"), "{}"); + await writeFile(join(source, "bun-2026-10-12-new.ips"), "{}"); + await writeFile(join(source, "node-2026-10-12.ips"), "{}"); + const old = new Date(Date.now() - 3_600_000); + await utimes(join(source, "bun-2026-10-12-old.ips"), old, old); + const copied = await collectDiagnosticReports({ + source, + destination, + sinceMs: Date.now() - 60_000, + }); + expect(copied).toBe(1); + expect(await readdir(destination)).toEqual(["bun-2026-10-12-new.ips"]); + expect( + await collectDiagnosticReports({ + source: join(root, "missing"), + destination, + sinceMs: 0, + }) + ).toBe(0); +}); diff --git a/tests/ci-bun-crash-census.test.ts b/tests/ci-bun-crash-census.test.ts new file mode 100644 index 000000000..ca037363d --- /dev/null +++ b/tests/ci-bun-crash-census.test.ts @@ -0,0 +1,446 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { mkdtemp, readdir, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + aggregateCensus, + type CensusOptions, + type GhReader, + type JobRecord, + parseCensusArguments, + renderCensus, + runCensus, + workflowJobBunVersion, +} from "../scripts/ci-bun-crash-census.ts"; +import { + classifyHostedJobLog, + hostedJobBunVersion, + hostedJobExitCode, +} from "../scripts/lib/bun-crash-classify.ts"; + +const stamp = (line: string) => `2026-09-30T21:01:25.1622680Z ${line}`; + +function hostedLog(opts: { + readonly bunVersion: string; + readonly body: readonly string[]; + readonly exitCode: number | null; +}): string { + return [ + stamp("##[group]Run oven-sh/setup-bun@v1"), + stamp("with:"), + stamp(` bun-version: ${opts.bunVersion}`), + stamp("##[endgroup]"), + ...opts.body.map(stamp), + ...(opts.exitCode === null + ? [] + : [stamp(`##[error]Process completed with exit code ${opts.exitCode}.`)]), + ].join("\n"); +} + +const CRASH_LOG = hostedLog({ + bunVersion: "1.3.9", + body: [ + "\u001b[0m\u001b[2mBun v1.3.9 (cf6cdbbb) macOS Silicon\u001b[0m", + "\u001b[31mpanic(main thread)\u001b[0m: Segmentation fault at address 0x80", + "oh no: Bun has crashed. This indicates a bug in Bun, not your code.", + " https://bun.report/1.3.9/Mt1cf6cdbbAAAA", + "SECRET_LOOKING_VALUE=do-not-print", + ], + exitCode: 133, +}); +const TEST_FAILURE_LOG = hostedLog({ + bunVersion: "1.4.2", + body: ["(fail) owner starts [5.00ms]", " 1 fail"], + exitCode: 1, +}); +const TYPECHECK_LOG = hostedLog({ + bunVersion: "1.4.2", + body: ["src/index.ts(1,1): error TS2304: Cannot find name 'x'."], + exitCode: 2, +}); +const SIGNAL_EXIT_LOG = hostedLog({ + bunVersion: "1.3.9", + body: ["/bin/bash: line 1: 1243 Trace/BPT trap: 5 bun test"], + exitCode: 133, +}); + +describe("hosted job logs", () => { + test("a hosted crash transcript with timestamps and ANSI escapes is a crash", () => { + expect(classifyHostedJobLog(CRASH_LOG)).toEqual({ + outcome: "crash", + reason: "banner", + }); + expect(hostedJobExitCode(CRASH_LOG)).toBe(133); + expect(hostedJobBunVersion(CRASH_LOG)).toBe("1.3.9"); + }); + + test("exit 133 without a banner is a signal crash", () => { + expect(classifyHostedJobLog(SIGNAL_EXIT_LOG)).toEqual({ + outcome: "crash", + reason: "signal_exit", + }); + }); + + test("test failures and typecheck exit 2 are non-crash failures", () => { + expect(classifyHostedJobLog(TEST_FAILURE_LOG)).toEqual({ + outcome: "failure", + reason: "test_failure", + }); + expect(classifyHostedJobLog(TYPECHECK_LOG)).toEqual({ + outcome: "failure", + reason: "exit_code", + }); + }); + + test("a log without an exit annotation is not a pass", () => { + expect( + classifyHostedJobLog( + hostedLog({ bunVersion: "1.4.2", body: [], exitCode: null }) + ) + ).toEqual({ outcome: "failure", reason: "missing_exit" }); + }); +}); + +test("the test job's Bun version is read from its own workflow block", () => { + const workflow = [ + "jobs:", + " state-models:", + " steps:", + " - uses: oven-sh/setup-bun@v1", + " with:", + ' bun-version: "1.0.0"', + " test:", + " runs-on: blacksmith-6vcpu-macos-15", + " steps:", + " - uses: oven-sh/setup-bun@v1", + " with:", + ' bun-version: "1.3.9"', + " linux-process-lifetime:", + ' bun-version: "1.4.2"', + ].join("\n"); + expect(workflowJobBunVersion(workflow, "test")).toBe("1.3.9"); + expect(workflowJobBunVersion(workflow, "missing")).toBeNull(); + expect( + workflowJobBunVersion( + "jobs:\n test:\n other:\n bun-version: 1.4.2", + "test" + ) + ).toBeNull(); +}); + +const record = (overrides: Partial): JobRecord => ({ + runId: 1, + attempt: 1, + jobId: 1, + conclusion: "success", + bunVersion: "1.4.2", + versionSource: "workflow", + scan: null, + reason: null, + ...overrides, +}); + +test("totals aggregate per Bun version with crashing run and job ids", () => { + const totals = aggregateCensus([ + record({ jobId: 1 }), + record({ + jobId: 2, + conclusion: "failure", + scan: "failure", + reason: "exit_code", + versionSource: "log", + }), + record({ jobId: 3, conclusion: "cancelled" }), + record({ + runId: 7, + attempt: 2, + jobId: 4, + bunVersion: "1.3.9", + conclusion: "failure", + scan: "crash", + reason: "banner", + versionSource: "log", + }), + record({ jobId: 5, bunVersion: "1.3.9" }), + record({ + jobId: 6, + bunVersion: "1.10.0", + conclusion: "timed_out", + scan: "unavailable", + }), + record({ jobId: 8, bunVersion: "unknown", versionSource: "unknown" }), + ]); + expect(totals.map((entry) => entry.bunVersion)).toEqual([ + "1.3.9", + "1.4.2", + "1.10.0", + "unknown", + ]); + expect(totals[0]).toMatchObject({ + executions: 2, + successes: 1, + failures: 1, + crashes: 1, + logsScanned: 1, + crashing: [{ runId: 7, attempt: 2, jobId: 4, reason: "banner" }], + }); + expect(totals[1]).toMatchObject({ + executions: 3, + successes: 1, + failures: 1, + cancelled: 1, + crashes: 0, + versionSources: { log: 1, workflow: 2, unknown: 0 }, + }); + expect(totals[2]).toMatchObject({ failures: 1, logsUnavailable: 1 }); +}); + +const SHA_OLD = "a".repeat(40); +const SHA_NEW = "b".repeat(40); +const workflowFor = (version: string) => + Buffer.from( + `jobs:\n test:\n steps:\n - with:\n bun-version: "${version}"\n` + ).toString("base64"); + +function fakeGh(opts: { readonly total?: number } = {}) { + const downloads: string[] = []; + const destinations: string[] = []; + const logs: Record = { + 11: CRASH_LOG, + 21: TEST_FAILURE_LOG, + 22: "passing log body", + }; + const reader: GhReader = { + json: async (path) => { + if (path.includes("/runs?created=")) { + return { total_count: opts.total ?? 2 }; + } + if (path.endsWith(`?ref=${SHA_OLD}`)) { + return { content: workflowFor("1.3.9") }; + } + if (path.endsWith(`?ref=${SHA_NEW}`)) { + return { content: workflowFor("1.4.2") }; + } + throw new Error("unexpected path"); + }, + lines: async (path) => { + if (path.includes("/runs?created=")) { + return [ + JSON.stringify({ id: 100, head_sha: SHA_OLD }), + JSON.stringify({ id: 200, head_sha: SHA_NEW }), + ]; + } + if (path.includes("/runs/100/jobs?filter=all")) { + return [ + { + id: 11, + name: "test", + run_attempt: 1, + status: "completed", + conclusion: "failure", + }, + { + id: 12, + name: "test", + run_attempt: 2, + status: "completed", + conclusion: "success", + }, + { + id: 13, + name: "runtime-images", + run_attempt: 1, + status: "completed", + conclusion: "failure", + }, + ].map((job) => JSON.stringify(job)); + } + if (path.includes("/runs/200/jobs?filter=all")) { + return [ + { + id: 21, + name: "test", + run_attempt: 1, + status: "completed", + conclusion: "failure", + }, + { + id: 21, + name: "test", + run_attempt: 1, + status: "completed", + conclusion: "failure", + }, + { + id: 22, + name: "test", + run_attempt: 2, + status: "completed", + conclusion: "success", + }, + { + id: 23, + name: "test", + run_attempt: 3, + status: "in_progress", + conclusion: null, + }, + { + id: 24, + name: "test", + run_attempt: 3, + status: "completed", + conclusion: "skipped", + }, + ].map((job) => JSON.stringify(job)); + } + throw new Error("unexpected path"); + }, + download: async (path, destination) => { + const jobId = Number(/\/jobs\/(\d+)\/logs$/.exec(path)?.[1]); + downloads.push(path); + destinations.push(destination); + await Bun.write(destination, logs[jobId] ?? ""); + }, + }; + return { reader, downloads, destinations }; +} + +const options: CensusOptions = { + repository: "hack-dance/hack", + workflow: "ci.yml", + job: "test", + since: "2026-09-29T00:00:00Z", + until: "2026-10-01T17:35:00Z", + scanPassing: false, + concurrency: 2, +}; + +const workDirs: string[] = []; +afterEach(async () => { + await Promise.all( + workDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true })) + ); +}); + +async function workDir(): Promise { + const dir = await mkdtemp(join(tmpdir(), "hack-bun-census-test-")); + workDirs.push(dir); + return dir; +} + +describe("runCensus", () => { + test("reads every attempt's test job, downloads only failed logs and removes them", async () => { + const dir = await workDir(); + const gh = fakeGh(); + const result = await runCensus(options, gh.reader, dir); + expect(gh.downloads).toEqual([ + "repos/hack-dance/hack/actions/jobs/11/logs", + "repos/hack-dance/hack/actions/jobs/21/logs", + ]); + expect(gh.destinations.every((path) => path.startsWith(dir))).toBe(true); + expect(await readdir(dir)).toEqual([]); + expect(result.runs).toBe(2); + expect(result.records.map((entry) => [entry.jobId, entry.attempt])).toEqual( + [ + [11, 1], + [12, 2], + [21, 1], + [22, 2], + ] + ); + expect(result.totals).toMatchObject([ + { + bunVersion: "1.3.9", + executions: 2, + failures: 1, + crashes: 1, + versionSources: { log: 1, workflow: 1, unknown: 0 }, + crashing: [{ runId: 100, attempt: 1, jobId: 11, reason: "banner" }], + }, + { bunVersion: "1.4.2", executions: 2, failures: 1, crashes: 0 }, + ]); + + const text = renderCensus({ options, result }); + expect(text).toContain( + "bun=1.3.9 executions=2 successes=1 failures=1 cancelled=0 crashes=1" + ); + expect(text).toContain(" crash run=100 attempt=1 job=11 reason=banner"); + expect(text).not.toContain("do-not-print"); + expect(text).not.toContain("bun.report"); + expect(text).not.toContain("Segmentation"); + }); + + test("--scan-passing also classifies passing logs", async () => { + const dir = await workDir(); + const gh = fakeGh(); + const result = await runCensus( + { ...options, scanPassing: true }, + gh.reader, + dir + ); + expect(gh.downloads).toHaveLength(4); + expect(await readdir(dir)).toEqual([]); + expect(result.totals[1]).toMatchObject({ + bunVersion: "1.4.2", + logsScanned: 2, + }); + }); + + test("a window beyond the runs endpoint limit is refused before any job read", async () => { + const gh = fakeGh({ total: 1001 }); + await expect( + runCensus(options, gh.reader, await workDir()) + ).rejects.toThrow("narrow it"); + expect(gh.downloads).toEqual([]); + }); +}); + +test("census arguments are validated and closed", () => { + expect( + parseCensusArguments([ + "--since", + "2026-10-01T17:35:00Z", + "--until=2026-10-11T23:43:30Z", + "--scan-passing", + ]) + ).toEqual({ + repository: "hack-dance/hack", + workflow: "ci.yml", + job: "test", + since: "2026-10-01T17:35:00Z", + until: "2026-10-11T23:43:30Z", + scanPassing: true, + json: false, + }); + for (const argv of [ + [], + ["--since", "2026-10-01", "--until", "2026-10-11T23:43:30Z"], + ["--since", "2026-10-11T23:43:30Z", "--until", "2026-10-01T17:35:00Z"], + [ + "--since", + "2026-10-01T17:35:00Z", + "--until", + "2026-10-11T23:43:30Z", + "--repo", + "x/y/z", + ], + [ + "--since", + "2026-10-01T17:35:00Z", + "--until", + "2026-10-11T23:43:30Z", + "--rerun", + ], + [ + "--since", + "2026-10-01T17:35:00Z", + "--until", + "2026-10-11T23:43:30Z", + "--json", + "--json", + ], + ]) { + expect(() => parseCensusArguments(argv)).toThrow(); + } +}); From 134287129db41c88acca308646c08772be9af058 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 04:43:38 -0400 Subject: [PATCH 24/32] perf(storage): read engine identity directly for docker context selections Storage-witness engine-ID admissions now resolve the Docker CLI's endpoint selection the way the CLI does: a nonempty DOCKER_HOST, then DOCKER_CONTEXT, then the client config's currentContext, then the default context at unix:///var/run/docker.sock. A named context is read from its contexts/meta//meta.json through a no-follow binding; only a plain unix:/// docker endpoint with no TLS option, no unknown or duplicate field and no entry at the context's TLS path is eligible. TLS material is checked for absence by name and never opened. Missing, malformed, linked, TCP, SSH, TLS or header-bearing contexts, and DOCKER_HOST with DOCKER_CONTEXT, keep the CLI. Admission is unchanged: the first fence under a binding is the CLI read and must equal one direct GET /info on the resolved socket. The binding now also pins the context store entry and the absent TLS path, so a changed config, context entry, socket or executable is admitted through the CLI again. The ingress and VM-file observers, trusted without a CLI comparison, still accept only an explicit Unix DOCKER_HOST. Tests that run commands against a synthetic docker now pin an isolated client config naming a missing context, so they never read the contributor's Docker config or dial a real engine socket. A new opt-in test (HACK_TEST_DOCKER_CLI=1, run on docker-e2e shard 4) runs the real Docker CLI against two fake Unix engines for every selection. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 4 + docs/reference/native-compose-ownership.md | 22 +- .../native-compose-storage-xattr-carrier.md | 9 +- src/lib/docker-context.ts | 144 ++++++ src/lib/native-compose-engine-config.ts | 264 +++++++--- src/lib/native-compose-engine-identity.ts | 130 +++-- ...compose-storage-witness-docker-identity.ts | 12 +- tests/docker-context.test.ts | 461 ++++++++++++++++++ tests/helpers/docker-client.ts | 29 ++ tests/helpers/native-compose-command.ts | 8 +- tests/native-compose-engine-identity.test.ts | 239 ++++++++- ...compose-storage-activation-command.test.ts | 2 + .../native-compose-storage-fence-cost.test.ts | 65 ++- 13 files changed, 1267 insertions(+), 122 deletions(-) create mode 100644 src/lib/docker-context.ts create mode 100644 tests/docker-context.test.ts create mode 100644 tests/helpers/docker-client.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 457db89f4..695b9a0d2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -300,6 +300,10 @@ jobs: - name: Verify owned bridge policy and aliases through real compiler and Compose normalization if: matrix.shard == 4 run: HACK_TEST_COMPOSE_CONFIG=1 bun test tests/native-compose-renderer-config.test.ts --test-name-pattern '^one authored bridge keeps internal policy and static aliases through compiler and Compose normalization$' + # Real Docker CLI against two fake Unix engines (isolated HOME and config; no real engine). + - name: Verify Docker context endpoint selection against the real Docker CLI + if: matrix.shard == 4 + run: HACK_TEST_DOCKER_CLI=1 bun test tests/docker-context.test.ts - name: Run local and Docker E2E shard env: HACK_E2E_PROCESS_POLICY_REFUSAL_CAPSULE: "1" diff --git a/docs/reference/native-compose-ownership.md b/docs/reference/native-compose-ownership.md index 23aa78198..452a252ef 100644 --- a/docs/reference/native-compose-ownership.md +++ b/docs/reference/native-compose-ownership.md @@ -137,7 +137,27 @@ binding for the rest of the command. A different or failed admission read keeps the CLI for that binding, a changed binding is admitted through the CLI again, and an admitted direct failure refuses without fallback. -Real Unix-server controls live in `tests/native-compose-engine-identity.test.ts`; +Because each admission is checked against the CLI's own read, storage fences +also accept Docker context selections, which is how Docker Desktop and OrbStack +select their engine. The endpoint is resolved in the Docker CLI's order: a +nonempty `DOCKER_HOST`, then `DOCKER_CONTEXT`, then the client config's +`currentContext` (field names matched without case, like the CLI's decoder), +then the `default` context at `unix:///var/run/docker.sock`. A named context is +read from `contexts/meta//meta.json` under the config +directory; its `docker` endpoint must be a plain `unix:///` host with no TLS +option, no unknown or duplicate field, and no entry at the context's +`contexts/tls/` path. That TLS path is only checked for absence by name; +key material is never opened or listed. Missing, malformed, linked, TCP, SSH, +TLS or header-bearing contexts, and `DOCKER_HOST` together with +`DOCKER_CONTEXT`, keep the CLI. The binding adds the context store entry's path +identities and content digest and the absent TLS path, so any change to the +config, that entry, the socket path or the CLI executable is admitted through +the CLI again. The ingress and VM-file observers, which are trusted without a +CLI comparison, still accept only an explicit Unix `DOCKER_HOST`. + +Real Unix-server controls live in `tests/native-compose-engine-identity.test.ts` +and `tests/docker-context.test.ts` (with `HACK_TEST_DOCKER_CLI=1`, it also runs +the installed Docker CLI against two fake Unix engines for every selection); existing CLI-transport and proxy ownership controls remain in `tests/native-compose-ingress.test.ts`. These tests establish the bounded transport and refusal behavior. Actual Docker compatibility and performance require separate diff --git a/docs/reference/native-compose-storage-xattr-carrier.md b/docs/reference/native-compose-storage-xattr-carrier.md index 059d03e74..e9a000efb 100644 --- a/docs/reference/native-compose-storage-xattr-carrier.md +++ b/docs/reference/native-compose-storage-xattr-carrier.md @@ -232,11 +232,14 @@ runs; three exact costs are removed. Each change is local to one CLI process, which runs exactly one native command. - Engine-ID fences (`docker info --format '{{json .ID}}'`) in the Docker - carrier use command-scoped admission for an explicit `DOCKER_HOST=unix:///...` - selection that the ingress transport also admits (see + carrier use command-scoped admission for a selection the Docker CLI resolves + to a plain Unix socket: an explicit `DOCKER_HOST=unix:///...`, + `DOCKER_CONTEXT`, the client config's current context, or the `default` + context (see [ownership](native-compose-ownership.md#ingress-daemon-identity-transport)). The first fence under one exact binding (selection values, `docker` - executable, client config and socket identities) is the Docker CLI read and + executable, client config, context store entry and socket identities) is the + Docker CLI read and must equal one direct `GET /info` on that socket. Only then does each later fence under the unchanged binding make one fresh direct request instead of a CLI process that also lists every client plugin. A different or failed diff --git a/src/lib/docker-context.ts b/src/lib/docker-context.ts new file mode 100644 index 000000000..fd51c3df1 --- /dev/null +++ b/src/lib/docker-context.ts @@ -0,0 +1,144 @@ +import { isAbsolute, join, normalize } from "node:path"; +import { isRecord } from "./guards.ts"; +import { + bindNativeComposeEngineAbsence, + bindNativeComposeEngineDocument, + foldNativeComposeEngineFields, + nativeComposeDockerConfigDirectory, +} from "./native-compose-engine-config.ts"; + +type Environment = Readonly>; + +/** The Docker CLI's built-in default endpoint on Unix (`client.DefaultDockerHost`). */ +export const DOCKER_DEFAULT_UNIX_SOCKET = "/var/run/docker.sock"; +/** The reserved name the CLI resolves from its flags and environment, never the store. */ +const DEFAULT_CONTEXT = "default"; +/** docker/cli `ValidateContextName`; other spellings keep the CLI. */ +const CONTEXT_NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.+-]+$/; +const AMBIGUOUS_PATH = /[%?#\0\r\n]/; + +/** + * The Unix socket the Docker CLI would dial for one selection, and the + * value-free facts that pin how it was selected. The facts are compared and + * digested only; they are never displayed. + */ +export type DockerContextEndpoint = { + readonly socketPath: string; + readonly binding: unknown; +}; + +/** + * Path of an unambiguous `unix:///absolute/path` host, else null. The CLI uses + * the text after `unix://` verbatim; spellings it would trim, decode or + * normalize are refused so that no other spelling can name a different socket. + */ +export function dockerUnixHostPath(host: string): string | null { + if (!host.startsWith("unix:///")) { + return null; + } + const path = host.slice("unix://".length); + return AMBIGUOUS_PATH.test(path) || + path.trim() !== path || + !isAbsolute(path) || + normalize(path) !== path + ? null + : path; +} + +/** sha256 of the context name: the store's directory name for that context. */ +function contextId(name: string): string { + return new Bun.CryptoHasher("sha256").update(name).digest("hex"); +} + +/** + * The docker endpoint of a context's `meta.json`, matched the way the CLI's Go + * decoder matches it: top-level and endpoint fields without case, endpoint map + * keys exactly. Unknown fields, duplicate spellings, TLS options and any host + * other than an unambiguous Unix socket keep the CLI. + */ +function inspectMeta(value: Record): string | null { + const fields = foldNativeComposeEngineFields(value, [ + "name", + "metadata", + "endpoints", + ]); + const name = fields?.get("name"); + const metadata = fields?.get("metadata"); + const endpoints = fields?.get("endpoints"); + if ( + fields === null || + (name !== undefined && typeof name !== "string") || + !(metadata === undefined || metadata === null || isRecord(metadata)) || + !isRecord(endpoints) || + !Object.hasOwn(endpoints, "docker") + ) { + return null; + } + const docker = endpoints.docker; + const endpoint = isRecord(docker) + ? foldNativeComposeEngineFields(docker, ["host", "skiptlsverify"]) + : null; + const host = endpoint?.get("host"); + const skip = endpoint?.get("skiptlsverify"); + return typeof host === "string" && (skip === undefined || skip === false) + ? dockerUnixHostPath(host) + : null; +} + +/** + * Resolve the endpoint exactly as the Docker CLI does for a flagless command, + * then return it only when it is a plain Unix socket this transport can read: + * + * 1. a non-empty DOCKER_HOST selects the default context with that host (with a + * non-empty DOCKER_CONTEXT too, the selection is treated as ambiguous); + * 2. else a non-empty DOCKER_CONTEXT names the context; + * 3. else the client config's `currentContext` names it; + * 4. else the `default` context, whose endpoint is `defaultSocketPath`. + * + * A named context is read from `/contexts/meta//meta.json` + * through a no-follow binding. Its TLS directory must be absent: it is proven + * absent by name and never opened, so no key material is read. A missing, + * malformed, TLS, TCP, SSH or otherwise ineligible context returns null, and + * the caller keeps the Docker CLI. The caller must have already refused TLS, + * certificate, header and API-version environment selections, and must + * re-resolve and compare the binding around every read. + */ +export function resolveDockerContextEndpoint(opts: { + readonly environment: Environment; + readonly currentContext: string | null; + readonly defaultSocketPath: string; +}): DockerContextEndpoint | null { + const { environment } = opts; + if (environment.DOCKER_HOST) { + const path = environment.DOCKER_CONTEXT + ? null + : dockerUnixHostPath(environment.DOCKER_HOST); + return path === null ? null : { socketPath: path, binding: ["host"] }; + } + const name = + environment.DOCKER_CONTEXT || opts.currentContext || DEFAULT_CONTEXT; + if (name === DEFAULT_CONTEXT) { + return { socketPath: opts.defaultSocketPath, binding: [DEFAULT_CONTEXT] }; + } + const directory = nativeComposeDockerConfigDirectory(environment); + if (directory === null || !CONTEXT_NAME.test(name)) { + return null; + } + const id = contextId(name); + const meta = bindNativeComposeEngineDocument({ + path: join(directory, "contexts", "meta", id, "meta.json"), + inspect: inspectMeta, + }); + if (meta === null || !meta.found) { + return null; + } + const tls = bindNativeComposeEngineAbsence( + join(directory, "contexts", "tls", id) + ); + return tls === null + ? null + : { + socketPath: meta.value, + binding: ["context", name, meta.binding, tls], + }; +} diff --git a/src/lib/native-compose-engine-config.ts b/src/lib/native-compose-engine-config.ts index 5a3e7f1df..58a0ba582 100644 --- a/src/lib/native-compose-engine-config.ts +++ b/src/lib/native-compose-engine-config.ts @@ -11,15 +11,21 @@ import { isAbsolute, join, normalize, parse } from "node:path"; import { isRecord } from "./guards.ts"; import { parseImportDocument } from "./native-config-import-parser.ts"; -const CONFIG_LIMIT = 1024 * 1024; +const DOCUMENT_LIMIT = 1024 * 1024; const ASCII_FIELD = /^[\x20-\x7e]+$/; +type Environment = Readonly>; type Entry = { readonly path: string; readonly identity: readonly number[] }; -type ConfigBinding = { +/** Named-path facts of one document: each component identity, then content. */ +type DocumentBinding = { readonly path: string; readonly entries: readonly Entry[]; readonly missing: string | null; readonly sha256: string | null; }; +type ConfigBinding = DocumentBinding & { + /** The client config's `currentContext`, or null when absent or empty. */ + readonly currentContext: string | null; +}; function identity(info: Stats): readonly number[] { return [ @@ -36,8 +42,60 @@ function identity(info: Stats): readonly number[] { function same(left: readonly number[], right: readonly number[]): boolean { return JSON.stringify(left) === JSON.stringify(right); } -function eligibleFile(path: string, named: Stats): string | null { - if (!(named.isFile() && named.size <= CONFIG_LIMIT)) { + +/** + * lstat every named component from the root without following any link. A + * missing component ends the walk with that component recorded; a link, a + * non-directory ancestor or any other error makes the path ineligible. + */ +function walk(path: string): { + readonly entries: readonly Entry[]; + readonly missing: string | null; + readonly leaf: Stats | null; +} | null { + if (!isAbsolute(path) || normalize(path) !== path) { + return null; + } + const root = parse(path).root; + let current = root; + const entries: Entry[] = []; + for (const component of ["", ...path.slice(root.length).split("/")]) { + current = component ? join(current, component) : current; + let info: Stats; + try { + info = lstatSync(current); + } catch (error: unknown) { + if (isRecord(error) && error.code === "ENOENT") { + return { entries, missing: current, leaf: null }; + } + return null; + } + if (info.isSymbolicLink()) { + return null; + } + entries.push({ path: current, identity: identity(info) }); + if (current === path) { + return { entries, missing: null, leaf: info }; + } + if (!info.isDirectory()) { + return null; + } + } + return null; +} + +/** + * Read one bounded strict JSON object through a no-follow descriptor whose + * identity equals the named entry before and after the read. The existing + * strict parser rejects decoded duplicate keys; every object key must be + * printable ASCII so that case-insensitive field matching is exact. + */ +function readDocument( + path: string, + named: Stats, + inspect: (value: Record) => T | null +): { readonly sha256: string; readonly value: T } | null { + if (!(named.isFile() && named.size <= DOCUMENT_LIMIT)) { return null; } const fd = openSync( @@ -66,31 +124,123 @@ function eligibleFile(path: string, named: Stats): string | null { } const data = bytes.subarray(0, size); const text = new TextDecoder("utf-8", { fatal: true }).decode(data); - // The existing bounded strict parser rejects decoded duplicate keys. Go's - // Docker decoder also matches field names without case, so inspect that set. const { value } = parseImportDocument({ text, document: "config" }); if (!value || Object.keys(value).some((key) => !ASCII_FIELD.test(key))) { return null; } - const headers = Object.entries(value).filter( - ([key]) => key.toLowerCase() === "httpheaders" - ); + const inspected = inspect(value); + return inspected === null + ? null + : { + sha256: new Bun.CryptoHasher("sha256").update(data).digest("hex"), + value: inspected, + }; + } finally { + closeSync(fd); + } +} + +/** + * Go's JSON decoder matches struct fields without case, and a later duplicate + * spelling overwrites (or merges into) an earlier one. Return the fields by + * folded name only when every key is printable ASCII and no folded name repeats. + * With `allowed`, any other folded name is ineligible too. + */ +export function foldNativeComposeEngineFields( + value: Record, + allowed?: readonly string[] +): ReadonlyMap | null { + const fields = new Map(); + for (const [key, field] of Object.entries(value)) { + const folded = key.toLowerCase(); if ( - headers.length > 1 || - headers.some( - ([, value]) => !isRecord(value) || Object.keys(value).length > 0 - ) + !ASCII_FIELD.test(key) || + fields.has(folded) || + (allowed !== undefined && !allowed.includes(folded)) ) { return null; } - return new Bun.CryptoHasher("sha256").update(data).digest("hex"); - } finally { - closeSync(fd); + fields.set(folded, field); + } + return fields; +} + +/** + * Bind one named JSON document without following links. Returns null when the + * path or document is ineligible (including when `inspect` returns null), + * `found: false` with the first missing component, or the inspected value. + * Never returns diagnostics or document text; the digest is compared only. + */ +export function bindNativeComposeEngineDocument(opts: { + readonly path: string; + readonly inspect: (value: Record) => T | null; +}): + | { readonly binding: DocumentBinding; readonly found: false } + | { + readonly binding: DocumentBinding; + readonly found: true; + readonly value: T; + } + | null { + try { + const named = walk(opts.path); + if (named === null) { + return null; + } + if (named.leaf === null) { + return { + binding: { + path: opts.path, + entries: named.entries, + missing: named.missing, + sha256: null, + }, + found: false, + }; + } + const document = readDocument(opts.path, named.leaf, opts.inspect); + return document === null + ? null + : { + binding: { + path: opts.path, + entries: named.entries, + missing: null, + sha256: document.sha256, + }, + found: true, + value: document.value, + }; + } catch { + return null; + } +} + +/** + * Prove a named path absent without following links or reading anything below + * it: the binding records each existing ancestor and the first missing + * component. Any existing entry at the path, a link or an error returns null. + */ +export function bindNativeComposeEngineAbsence(path: string): { + readonly entries: readonly Entry[]; + readonly missing: string; +} | null { + try { + const named = walk(path); + return named === null || named.missing === null + ? null + : { entries: named.entries, missing: named.missing }; + } catch { + return null; } } -function configPath( - environment: Readonly> +/** + * The Docker CLI's client config directory: DOCKER_CONFIG, else HOME/.docker. + * Relative or unnormalized spellings return null and keep the CLI. + */ +export function nativeComposeDockerConfigDirectory( + environment: Environment ): string | null { const directory = environment.DOCKER_CONFIG || @@ -98,54 +248,54 @@ function configPath( return directory && isAbsolute(directory) && normalize(directory) === directory - ? join(directory, "config.json") + ? directory : null; } +function inspectConfig( + value: Record +): { readonly currentContext: string | null } | null { + const fields = foldNativeComposeEngineFields(value); + if (fields === null) { + return null; + } + const headers = fields.get("httpheaders"); + if ( + headers !== undefined && + !(isRecord(headers) && Object.keys(headers).length === 0) + ) { + return null; + } + const current = fields.get("currentcontext"); + if (current !== undefined && typeof current !== "string") { + return null; + } + return { currentContext: current || null }; +} + /** * Eligibility only: configured headers, ambiguous paths and unreadable inputs - * keep Docker's CLI transport. Never return parsed configuration or diagnostics. - * Once admitted, compare this named-path/content binding before and after reads; - * a changed or newly ineligible config must refuse, never select another transport. + * keep Docker's CLI transport. Never return parsed configuration or diagnostics + * beyond the selected context name. Once admitted, compare this named-path and + * content binding before and after reads; a changed or newly ineligible config + * must refuse, never select another transport. */ export function bindNativeComposeEngineConfig( - environment: Readonly> + environment: Environment ): ConfigBinding | null { - try { - const path = configPath(environment); - if (path === null) { - return null; - } - const root = parse(path).root; - let current = root; - const entries: Entry[] = []; - for (const component of ["", ...path.slice(root.length).split("/")]) { - current = component ? join(current, component) : current; - let info: Stats; - try { - info = lstatSync(current); - } catch (error: unknown) { - if (isRecord(error) && error.code === "ENOENT") { - return { path, entries, missing: current, sha256: null }; - } - return null; - } - if (info.isSymbolicLink()) { - return null; - } - entries.push({ path: current, identity: identity(info) }); - if (current === path) { - const sha256 = eligibleFile(path, info); - return sha256 === null - ? null - : { path, entries, missing: null, sha256 }; - } - if (!info.isDirectory()) { - return null; - } - } + const directory = nativeComposeDockerConfigDirectory(environment); + if (directory === null) { return null; - } catch { + } + const bound = bindNativeComposeEngineDocument({ + path: join(directory, "config.json"), + inspect: inspectConfig, + }); + if (bound === null) { return null; } + return { + ...bound.binding, + currentContext: bound.found ? bound.value.currentContext : null, + }; } diff --git a/src/lib/native-compose-engine-identity.ts b/src/lib/native-compose-engine-identity.ts index 63618b2cc..d9b31f739 100644 --- a/src/lib/native-compose-engine-identity.ts +++ b/src/lib/native-compose-engine-identity.ts @@ -1,6 +1,11 @@ import { lstatSync, readlinkSync, type Stats } from "node:fs"; import { request } from "node:http"; import { dirname, isAbsolute, join, normalize, parse } from "node:path"; +import { + DOCKER_DEFAULT_UNIX_SOCKET, + dockerUnixHostPath, + resolveDockerContextEndpoint, +} from "./docker-context.ts"; import { isRecord } from "./guards.ts"; import { bindNativeComposeEngineConfig } from "./native-compose-engine-config.ts"; @@ -150,29 +155,47 @@ export function bindNativeComposeEngineSocketPath(path: string): { function check(signal: AbortSignal | undefined, deadline: number): void { requireValue(!signal?.aborted && Date.now() < deadline); } -function localEndpoint(environment: Environment): string | null { - if ( +/** TLS, certificate, header and API-version selections always keep the CLI. */ +function transportSelected(environment: Environment): boolean { + return Boolean( process.platform === "win32" || - environment.DOCKER_CONTEXT || - environment.DOCKER_TLS || - environment.DOCKER_TLS_VERIFY || - environment.DOCKER_CUSTOM_HEADERS || - environment.DOCKER_CERT_PATH || - environment.DOCKER_API_VERSION - ) { - return null; - } + environment.DOCKER_TLS || + environment.DOCKER_TLS_VERIFY || + environment.DOCKER_CUSTOM_HEADERS || + environment.DOCKER_CERT_PATH || + environment.DOCKER_API_VERSION + ); +} +/** An explicit Unix DOCKER_HOST with no context selection, else null. */ +function explicitEndpoint(environment: Environment): string | null { const host = environment.DOCKER_HOST; - if (!host?.startsWith("unix:///")) { - return null; - } - const path = host.slice("unix://".length); // Ambiguous URL spelling keeps the existing Docker CLI selection semantics. - return AMBIGUOUS_PATH.test(path) || - !isAbsolute(path) || - normalize(path) !== path + return transportSelected(environment) || environment.DOCKER_CONTEXT || !host ? null - : path; + : dockerUnixHostPath(host); +} +type Contexts = { readonly defaultSocketPath: string } | null; +/** + * Endpoint selection under one config binding. Explicit observers accept only an + * explicit Unix DOCKER_HOST; admissions resolve the CLI's full context + * selection. Both return null for every other selection. + */ +function selectEndpoint(opts: { + readonly environment: Environment; + readonly currentContext: string | null; + readonly contexts: Contexts; +}): { readonly socketPath: string; readonly binding: unknown } | null { + if (opts.contexts === null) { + const path = explicitEndpoint(opts.environment); + return path === null ? null : { socketPath: path, binding: ["host"] }; + } + return transportSelected(opts.environment) + ? null + : resolveDockerContextEndpoint({ + environment: opts.environment, + currentContext: opts.currentContext, + defaultSocketPath: opts.contexts.defaultSocketPath, + }); } function readId(opts: { @@ -282,12 +305,16 @@ function bindObserver(opts: { readonly signal?: AbortSignal; readonly timeoutMs?: number; readonly environment?: Environment; + readonly contexts: Contexts; }): BoundObserver | null { const environment = opts.environment ?? process.env; const signal = opts.signal; const selected = selection(environment); - const path = localEndpoint(environment); - if (path === null) { + if ( + opts.contexts === null + ? explicitEndpoint(environment) === null + : transportSelected(environment) + ) { return null; } try { @@ -301,6 +328,17 @@ function bindObserver(opts: { if (config === null) { return null; } + const select = () => + selectEndpoint({ + environment, + currentContext: config.currentContext, + contexts: opts.contexts, + }); + const endpoint = select(); + if (endpoint === null) { + return null; + } + const path = endpoint.socketPath; const binary = Bun.which("docker", { PATH: environment.PATH ?? "" }); requireValue(binary); const socket = bindPath(path, "socket"); @@ -319,13 +357,18 @@ function bindObserver(opts: { JSON.stringify(config) === JSON.stringify(bindNativeComposeEngineConfig(environment)) ); + // An unchanged config can still name another socket through a changed + // context store entry or a newly created context TLS directory. + requireValue(JSON.stringify(endpoint) === JSON.stringify(select())); requireValue(sameBinding(socket, bindPath(path, "socket"))); requireValue(sameBinding(executable, bindPath(binary, "executable"))); check(signal, deadline); }; recheck(); const key = new Bun.CryptoHasher("sha256") - .update(JSON.stringify([selected, binary, config, socket, executable])) + .update( + JSON.stringify([selected, binary, config, endpoint, socket, executable]) + ) .digest("hex"); return Object.freeze({ key, @@ -360,9 +403,11 @@ function bindObserver(opts: { /** * Replace only `docker info` transport for an explicit local Unix selection. * Every call makes a fresh bounded daemon-ID request; no result is cached. Other - * selections return null before admission and retain their existing CLI path. - * Admitted errors never fall back. Socket, executable and selection bindings are - * checked before and after each request under the caller's existing time budget. + * selections, including every Docker context selection, return null before + * admission and retain their existing CLI path: this observer is trusted + * without a CLI comparison. Admitted errors never fall back. Socket, executable + * and selection bindings are checked before and after each request under the + * caller's existing time budget. */ export function createNativeComposeEngineIdentityObserver( opts: { @@ -371,31 +416,43 @@ export function createNativeComposeEngineIdentityObserver( readonly environment?: Environment; } = {} ): (() => Promise) | null { - return bindObserver(opts)?.read ?? null; + return bindObserver({ ...opts, contexts: null })?.read ?? null; } /** * Command-scoped transport for repeated `docker info --format '{{json .ID}}'` * fences. Each fence still makes exactly one fresh engine-ID read; no ID is cached. * + * Eligible selections are those the Docker CLI resolves to a plain Unix socket: + * an explicit Unix DOCKER_HOST, DOCKER_CONTEXT, the client config's + * `currentContext`, or the `default` context (`resolveDockerContextEndpoint`). * Admission is per exact binding: the selection values, `docker` executable path - * and identity, client-config path/content and socket path identities. The first - * fence under an eligible binding returns the caller's own Docker CLI read and - * compares it with one direct `GET /info` on the same explicit Unix socket. Only - * an equal ID admits that binding; a direct failure or a different ID keeps the - * CLI for that binding for the rest of the command. Later fences under an admitted - * binding use one fresh direct request each, and its failures refuse without CLI - * fallback. A changed binding is a new binding and is admitted through the CLI - * again. Ineligible selections, and bindings that cannot be established, always - * use the CLI. The returned text has the CLI's `{{json .ID}}` line shape. + * and identity, client-config path/content, the selected context store entry + * and absent TLS directory, and socket path identities. The first fence under + * an eligible binding returns the caller's own Docker CLI read and compares it + * with one direct `GET /info` on the socket that selection names. Only an equal + * ID admits that binding; a direct failure or a different ID keeps the CLI for + * that binding for the rest of the command. Later fences under an admitted + * binding use one fresh direct request each, and its failures refuse without + * CLI fallback. A changed binding is a new binding and is admitted through the + * CLI again. Ineligible selections, and bindings that cannot be established, + * always use the CLI. The returned text has the CLI's `{{json .ID}}` line shape. + * + * `defaultSocketPath` is the `default` context's endpoint; only tests replace + * the CLI's built-in default. */ -export function createNativeComposeEngineIdentityAdmissions(): (opts: { +export function createNativeComposeEngineIdentityAdmissions( + options: { readonly defaultSocketPath?: string } = {} +): (opts: { readonly cli: () => Promise; readonly signal?: AbortSignal; readonly timeoutMs: number; readonly environment: Environment; }) => Promise { const admitted = new Map(); + const contexts = { + defaultSocketPath: options.defaultSocketPath ?? DOCKER_DEFAULT_UNIX_SOCKET, + }; return async (opts) => { let bound: BoundObserver | null; try { @@ -403,6 +460,7 @@ export function createNativeComposeEngineIdentityAdmissions(): (opts: { signal: opts.signal, timeoutMs: opts.timeoutMs, environment: opts.environment, + contexts, }); } catch { bound = null; diff --git a/src/lib/native-compose-storage-witness-docker-identity.ts b/src/lib/native-compose-storage-witness-docker-identity.ts index b77d53f29..4280d8a5a 100644 --- a/src/lib/native-compose-storage-witness-docker-identity.ts +++ b/src/lib/native-compose-storage-witness-docker-identity.ts @@ -17,11 +17,13 @@ function identityQuery(args: readonly string[]): boolean { /** * Storage-witness probe whose exact engine-ID query keeps one fresh read per * fence but may use the command's admitted direct Unix transport - * (`createNativeComposeEngineIdentityAdmissions`). Every other query, every - * ineligible selection and every unadmitted binding goes to the supplied CLI - * probe unchanged. The direct read shares this probe's signal and deadline, and - * the same environment snapshot the CLI probe captured. Recovery observation - * keeps its own CLI probes. + * (`createNativeComposeEngineIdentityAdmissions`): an explicit Unix DOCKER_HOST + * or a Docker context selection (DOCKER_CONTEXT, the client config's current + * context, or the default context) that the CLI resolves to a Unix socket. + * Every other query, every ineligible selection and every unadmitted binding + * goes to the supplied CLI probe unchanged. The direct read shares this probe's + * signal and deadline, and the same environment snapshot the CLI probe + * captured. Recovery observation keeps its own CLI probes. */ export function nativeComposeStorageIdentityProbe(opts: { readonly probe: Probe; diff --git a/tests/docker-context.test.ts b/tests/docker-context.test.ts new file mode 100644 index 000000000..c85a6f753 --- /dev/null +++ b/tests/docker-context.test.ts @@ -0,0 +1,461 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + mkdir, + mkdtemp, + realpath, + rename, + rm, + symlink, + writeFile, +} from "node:fs/promises"; +import { createServer, type Server } from "node:http"; +import type { Socket } from "node:net"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { + DOCKER_DEFAULT_UNIX_SOCKET, + resolveDockerContextEndpoint, +} from "../src/lib/docker-context.ts"; +import { bindNativeComposeEngineConfig } from "../src/lib/native-compose-engine-config.ts"; +import { createNativeComposeEngineIdentityAdmissions } from "../src/lib/native-compose-engine-identity.ts"; + +/** + * Two fake Unix engines answer both the Docker CLI (`HEAD /_ping`, then a + * versioned `GET /v1.N/info`) and the direct transport (unversioned + * `GET /info`). Each records which requests it served, so a test can prove + * which socket each transport dialed. + */ +type Engine = { readonly path: string; readonly served: string[] }; +type Environment = Record; +const CANARY = "docker-context-private-canary"; +let root: string; +let config: string; +let engines: Record<"a" | "b", Engine>; +let servers: Server[]; +let peers: Set; + +async function engine(name: string): Promise { + const path = join(root, `${name}.sock`); + const served: string[] = []; + const server = createServer((req, res) => { + served.push(`${req.method} ${req.url}`); + res.setHeader("Api-Version", "1.47"); + res.setHeader("Ostype", "linux"); + if (req.url?.endsWith("/_ping")) { + res.end("OK"); + } else if (req.url?.endsWith("/info")) { + res.setHeader("Content-Type", "application/json"); + res.end(JSON.stringify({ ID: `engine-${name}`, Name: CANARY })); + } else { + res.statusCode = 404; + res.end("{}"); + } + }); + server.on("connection", (peer) => peers.add(peer)); + servers.push(server); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(path, resolve); + }); + return { path, served }; +} +beforeEach(async () => { + root = await realpath(await mkdtemp(join(tmpdir(), "docker-context-"))); + config = join(root, "client"); + await mkdir(config, { mode: 0o700 }); + servers = []; + peers = new Set(); + engines = { a: await engine("a"), b: await engine("b") }; + // The shape `docker context create --docker host=unix://...` writes. + await context("ctx-a", { + Host: `unix://${engines.a.path}`, + SkipTLSVerify: false, + }); + await context("ctx-b", { + Host: `unix://${engines.b.path}`, + SkipTLSVerify: false, + }); +}); +afterEach(async () => { + for (const peer of peers) { + peer.destroy(); + } + for (const server of servers) { + await new Promise((resolve) => server.close(() => resolve())); + } + await rm(root, { recursive: true, force: true }); +}); + +function id(name: string): string { + return createHash("sha256").update(name).digest("hex"); +} +/** Write a context store entry the way `docker context create` lays it out. */ +async function context( + name: string, + docker: Record | null, + raw?: string +) { + const directory = join(config, "contexts", "meta", id(name)); + await mkdir(directory, { recursive: true, mode: 0o755 }); + await writeFile( + join(directory, "meta.json"), + raw ?? + JSON.stringify({ + Name: name, + Metadata: { Description: "synthetic" }, + Endpoints: { docker }, + }), + { mode: 0o644 } + ); +} +async function current(name: string | null, raw?: string) { + await writeFile( + join(config, "config.json"), + raw ?? + JSON.stringify({ + auths: { synthetic: { auth: CANARY } }, + ...(name === null ? {} : { currentContext: name }), + }), + { mode: 0o600 } + ); +} +function base(extra: Environment = {}): Environment { + return { HOME: root, DOCKER_CONFIG: config, ...extra }; +} +/** The socket this transport would read, or null when the CLI keeps the selection. */ +function resolved(environment: Environment): string | null { + const bound = bindNativeComposeEngineConfig(environment); + if (bound === null) { + return null; + } + return ( + resolveDockerContextEndpoint({ + environment, + currentContext: bound.currentContext, + defaultSocketPath: DOCKER_DEFAULT_UNIX_SOCKET, + })?.socketPath ?? null + ); +} + +// Each row is one selection and the socket the Docker CLI dials for it, as +// docker/cli resolveContextName orders them: DOCKER_HOST, DOCKER_CONTEXT, the +// config's currentContext, then `default`. Empty variables count as unset. +const SELECTIONS: readonly { + readonly name: string; + readonly current: string | null; + readonly env: (engines: Record<"a" | "b", Engine>) => Environment; + readonly cli: "a" | "b" | "default"; + /** False only where the CLI resolves but this transport keeps the CLI. */ + readonly eligible: boolean; +}[] = [ + { + name: "no selection", + current: null, + env: () => ({}), + cli: "default", + eligible: true, + }, + { + name: "currentContext", + current: "ctx-b", + env: () => ({}), + cli: "b", + eligible: true, + }, + { + name: "DOCKER_CONTEXT over currentContext", + current: "ctx-b", + env: () => ({ DOCKER_CONTEXT: "ctx-a" }), + cli: "a", + eligible: true, + }, + { + name: "DOCKER_HOST over currentContext", + current: "ctx-b", + env: (e) => ({ DOCKER_HOST: `unix://${e.a.path}` }), + cli: "a", + eligible: true, + }, + { + name: "empty DOCKER_HOST and DOCKER_CONTEXT are unset", + current: "ctx-b", + env: () => ({ DOCKER_HOST: "", DOCKER_CONTEXT: "" }), + cli: "b", + eligible: true, + }, + { + name: "DOCKER_CONTEXT=default over currentContext", + current: "ctx-b", + env: () => ({ DOCKER_CONTEXT: "default" }), + cli: "default", + eligible: true, + }, + { + name: "currentContext=default", + current: "default", + env: () => ({}), + cli: "default", + eligible: true, + }, + { + name: "DOCKER_HOST with DOCKER_CONTEXT", + current: "ctx-b", + env: (e) => ({ + DOCKER_HOST: `unix://${e.a.path}`, + DOCKER_CONTEXT: "ctx-b", + }), + cli: "a", + eligible: false, + }, +]; +function socketFor(target: "a" | "b" | "default"): string { + return target === "default" + ? DOCKER_DEFAULT_UNIX_SOCKET + : engines[target].path; +} + +test("selection precedence names the same socket the Docker CLI dials", async () => { + for (const selection of SELECTIONS) { + await current(selection.current); + expect({ + name: selection.name, + socket: resolved(base(selection.env(engines))), + }).toEqual({ + name: selection.name, + socket: selection.eligible ? socketFor(selection.cli) : null, + }); + } + // A missing config file is the CLI's empty config: the default context. + await rm(join(config, "config.json")); + expect(resolved(base())).toBe(DOCKER_DEFAULT_UNIX_SOCKET); + expect([...engines.a.served, ...engines.b.served]).toEqual([]); +}); + +test("currentContext is matched without case, like the CLI's config decoder", async () => { + for (const spelling of [ + "currentContext", + "CurrentContext", + "currentcontext", + ]) { + await current(null, JSON.stringify({ [spelling]: "ctx-a" })); + expect(resolved(base())).toBe(engines.a.path); + } + for (const raw of [ + '{"currentContext":"ctx-a","CurrentContext":"ctx-b"}', + '{"currentContext":"ctx-a","currentContext":"ctx-b"}', + '{"currentContext":7}', + '{"currentContext":null}', + '{"currentContexſ":"ctx-a"}', + ]) { + await current(null, raw); + expect(resolved(base())).toBeNull(); + } +}); + +test("a context store entry is read only as an unambiguous plain Unix endpoint", async () => { + await current("ctx-x"); + const eligible = (docker: Record | null, raw?: string) => + context("ctx-x", docker, raw).then(() => resolved(base())); + expect(await eligible({ Host: `unix://${engines.a.path}` })).toBe( + engines.a.path + ); + expect( + await eligible({ host: `unix://${engines.a.path}`, skiptlsverify: false }) + ).toBe(engines.a.path); + for (const docker of [ + { Host: "tcp://127.0.0.1:2375" }, + { Host: "ssh://synthetic@example.invalid" }, + { Host: "npipe:////./pipe/docker_engine" }, + { Host: "unix://relative.sock" }, + { Host: `unix://${engines.a.path}?x=1` }, + { Host: `unix://${root}/./a.sock` }, + { Host: `unix://${engines.a.path} ` }, + { Host: `unix://${engines.a.path}%00` }, + { Host: "" }, + { Host: null }, + {}, + { Host: `unix://${engines.a.path}`, SkipTLSVerify: true }, + { + Host: `unix://${engines.a.path}`, + AdditionalHeaders: { Private: CANARY }, + }, + { Host: `unix://${engines.a.path}`, host: `unix://${engines.b.path}` }, + ]) { + expect(await eligible(docker)).toBeNull(); + } + const host = `unix://${engines.a.path}`; + for (const raw of [ + JSON.stringify({ Name: "ctx-x", Endpoints: { Docker: { Host: host } } }), + JSON.stringify({ Name: "ctx-x", Endpoints: { docker: null } }), + JSON.stringify({ Name: "ctx-x", Endpoints: null }), + JSON.stringify({ + Name: "ctx-x", + Endpoints: { docker: { Host: host } }, + endpoints: {}, + }), + JSON.stringify({ + Name: "ctx-x", + Extra: true, + Endpoints: { docker: { Host: host } }, + }), + JSON.stringify({ Name: 1, Endpoints: { docker: { Host: host } } }), + JSON.stringify({ + Name: "ctx-x", + Metadata: [], + Endpoints: { docker: { Host: host } }, + }), + `{"Endpoints":{"docker":{"Host":"${host}"},"docker":{"Host":"${host}"}}}`, + "[]", + CANARY, + ]) { + expect(await eligible(null, raw)).toBeNull(); + } + // Other endpoints in the same context are ignored by the CLI's docker client. + expect( + await eligible( + null, + JSON.stringify({ + Name: "ctx-x", + Endpoints: { docker: { Host: host }, kubernetes: { Host: CANARY } }, + }) + ) + ).toBe(engines.a.path); +}); + +test("missing, unnamed or invalid contexts keep the CLI", async () => { + for (const name of ["missing", "x", "-leading", "a/b", "a b", "ctx-a\n"]) { + expect(resolved(base({ DOCKER_CONTEXT: name }))).toBeNull(); + } + // "Default" is a store name, not the reserved default context. + expect(resolved(base({ DOCKER_CONTEXT: "Default" }))).toBeNull(); + for (const environment of [ + { HOME: undefined, DOCKER_CONFIG: undefined, DOCKER_CONTEXT: "ctx-a" }, + { HOME: root, DOCKER_CONFIG: "client", DOCKER_CONTEXT: "ctx-a" }, + { HOME: root, DOCKER_CONFIG: `${root}/./client`, DOCKER_CONTEXT: "ctx-a" }, + ]) { + expect(resolved(environment)).toBeNull(); + } +}); + +test("context TLS material keeps the CLI without opening or listing it", async () => { + const tls = join(config, "contexts", "tls", id("ctx-a")); + await current("ctx-a"); + expect(resolved(base())).toBe(engines.a.path); + await mkdir(join(tls, "docker"), { recursive: true, mode: 0o700 }); + // A FIFO blocks any reader until a writer opens it: a read would hang here. + const fifo = Bun.spawnSync(["mkfifo", join(tls, "docker", "key.pem")]); + expect(fifo.exitCode).toBe(0); + expect(resolved(base())).toBeNull(); + await rm(tls, { recursive: true }); + // Any entry at the TLS path, even an empty file, keeps the CLI. + await mkdir(dirname(tls), { recursive: true }); + await writeFile(tls, ""); + expect(resolved(base())).toBeNull(); + await rm(tls); + expect(resolved(base())).toBe(engines.a.path); +}); + +test("linked store paths keep the CLI", async () => { + await current("ctx-a"); + const meta = join(config, "contexts", "meta", id("ctx-a"), "meta.json"); + await rename(meta, join(root, "meta.json")); + await symlink(join(root, "meta.json"), meta); + expect(resolved(base())).toBeNull(); + await rm(meta); + await rename(join(root, "meta.json"), meta); + expect(resolved(base())).toBe(engines.a.path); + await rename(join(config, "contexts"), join(root, "contexts")); + await symlink(join(root, "contexts"), join(config, "contexts")); + expect(resolved(base())).toBeNull(); +}); + +// Real Docker CLI agreement. Opt-in (HACK_TEST_DOCKER_CLI=1): it runs the +// installed `docker` against the two fake engines only, with an isolated HOME +// and DOCKER_CONFIG, and never contacts a real engine. +const dockerCliTest = + process.env.HACK_TEST_DOCKER_CLI === "1" ? test : test.skip; +function dockerEnvironment(extra: Environment): Environment { + const docker = Bun.which("docker"); + if (!docker) { + throw new Error("HACK_TEST_DOCKER_CLI=1 requires the docker CLI on PATH"); + } + return base({ PATH: dirname(docker), ...extra }); +} +async function docker(environment: Environment, args: readonly string[]) { + const child = Bun.spawn(["docker", ...args], { + env: Object.fromEntries( + Object.entries(environment).filter( + (entry): entry is [string, string] => entry[1] !== undefined + ) + ), + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + }); + const [stdout, code] = await Promise.all([ + new Response(child.stdout).text(), + child.exited, + ]); + if (code !== 0) { + throw new Error("docker CLI selection failed"); + } + return stdout; +} + +dockerCliTest( + "the direct read targets the same socket the real Docker CLI picks for each selection", + async () => { + for (const selection of SELECTIONS) { + await current(selection.current); + const environment = dockerEnvironment(selection.env(engines)); + // Client-only resolution: the CLI prints the endpoint without dialing it. + expect( + ( + await docker(environment, [ + "context", + "inspect", + "--format", + "{{.Endpoints.docker.Host}}", + ]) + ).trim() + ).toBe(`unix://${socketFor(selection.cli)}`); + if (selection.cli === "default") { + continue; + } + for (const fake of Object.values(engines)) { + fake.served.length = 0; + } + const admissions = createNativeComposeEngineIdentityAdmissions({ + defaultSocketPath: join(root, "unused-default.sock"), + }); + const fence = () => + admissions({ + cli: () => docker(environment, ["info", "--format", "{{json .ID}}"]), + timeoutMs: 30_000, + environment, + }); + const picked = engines[selection.cli]; + const other = engines[selection.cli === "a" ? "b" : "a"]; + // The first fence is the CLI's own read, on the CLI's own socket. + expect(await fence()).toBe(`"engine-${selection.cli}"\n`); + expect( + picked.served.some((line) => /^GET \/v[\d.]+\/info$/.test(line)) + ).toBe(true); + expect(await fence()).toBe(`"engine-${selection.cli}"\n`); + const direct = picked.served.filter((line) => line === "GET /info"); + expect({ name: selection.name, direct: direct.length }).toEqual({ + name: selection.name, + // Admission read plus one fresh read for the admitted second fence. + direct: selection.eligible ? 2 : 0, + }); + // The second fence never ran the CLI when admitted; the other engine was + // never dialed by either transport. + expect( + picked.served.filter((line) => line.endsWith("/_ping")).length + ).toBe(selection.eligible ? 1 : 2); + expect(other.served).toEqual([]); + } + }, + 120_000 +); diff --git a/tests/helpers/docker-client.ts b/tests/helpers/docker-client.ts new file mode 100644 index 000000000..0130ec585 --- /dev/null +++ b/tests/helpers/docker-client.ts @@ -0,0 +1,29 @@ +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +/** A context name with no store entry: the Docker CLI itself would refuse it. */ +export const ISOLATED_DOCKER_CONTEXT = "hack-test-unselected"; + +/** + * Docker client selection for commands that run against a synthetic `docker`: + * an isolated client config whose current context does not exist. The storage + * engine-ID admission then keeps the synthetic CLI. It never reads the + * contributor's Docker config or context store, and never dials the default + * engine socket or a real context's socket. An explicit `DOCKER_HOST` or + * `DOCKER_CONFIG` passed by a test still takes precedence. + */ +export function isolatedDockerClientEnv( + root: string +): Readonly> { + const directory = join(root, "docker-client"); + const config = join(directory, "config.json"); + if (!existsSync(config)) { + mkdirSync(directory, { recursive: true, mode: 0o700 }); + writeFileSync( + config, + `${JSON.stringify({ currentContext: ISOLATED_DOCKER_CONTEXT })}\n`, + { mode: 0o600 } + ); + } + return { DOCKER_CONFIG: directory }; +} diff --git a/tests/helpers/native-compose-command.ts b/tests/helpers/native-compose-command.ts index 33dc84cc7..b928ec72e 100644 --- a/tests/helpers/native-compose-command.ts +++ b/tests/helpers/native-compose-command.ts @@ -2,6 +2,7 @@ import { afterEach, expect } from "bun:test"; import { chmod, mkdir, mkdtemp, realpath, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; +import { isolatedDockerClientEnv } from "./docker-client.ts"; import { nativeComposeStorageDockerFixtureScript } from "./native-compose-storage-docker.ts"; const roots: string[] = []; @@ -227,7 +228,11 @@ export async function invoke( args = ["up", "--detach", "--json"], startupTimeoutMs = 1000, commandTimeoutMs = 15_000, - /** Docker client selection only (for example an explicit Unix DOCKER_HOST). */ + /** + * Docker client selection only (for example an explicit Unix DOCKER_HOST). + * Without one, the command uses an isolated client config that selects no + * real engine. + */ extraEnv: Readonly> = {} ) { const child = Bun.spawn( @@ -252,6 +257,7 @@ export async function invoke( HACK_COMPOSE_STARTUP_TIMEOUT_MS: String(startupTimeoutMs), CI: "1", HACK_EXECUTION_MODE: "non_interactive", + ...isolatedDockerClientEnv(root), ...extraEnv, }, stdin: "ignore", diff --git a/tests/native-compose-engine-identity.test.ts b/tests/native-compose-engine-identity.test.ts index 6b64461da..98d7b9338 100644 --- a/tests/native-compose-engine-identity.test.ts +++ b/tests/native-compose-engine-identity.test.ts @@ -1,4 +1,5 @@ import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { createHash } from "node:crypto"; import { chmod, mkdir, @@ -679,8 +680,15 @@ function fence( ) { return admissions({ cli: cli.read, timeoutMs: 5000, environment }); } +/** + * No test dials the CLI's built-in default socket: the `default` context's + * endpoint is a fixture path, absent unless a test listens on it. + */ +function createAdmissions(defaultSocketPath = join(root, "default.sock")) { + return createNativeComposeEngineIdentityAdmissions({ defaultSocketPath }); +} test("storage fences answer the first one with the CLI and admit only an equal direct ID", async () => { - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const cli = cliStandIn(); expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); expect([cli.calls(), requests.length]).toEqual([1, 1]); @@ -691,7 +699,7 @@ test("storage fences answer the first one with the CLI and admit only an equal d expect([cli.calls(), requests.length, connections.size]).toEqual([1, 4, 4]); }); test("a changed daemon ID between two admitted fences is returned, never cached", async () => { - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const cli = cliStandIn(); await fence(admissions, cli); expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); @@ -713,7 +721,7 @@ test("a different or failed admission read keeps the CLI for that binding", asyn ]) { handler = reply; requests = []; - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const cli = cliStandIn(); for (let index = 0; index < 3; index++) { expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); @@ -722,7 +730,7 @@ test("a different or failed admission read keeps the CLI for that binding", asyn } }); test("admitted direct failures refuse without CLI fallback", async () => { - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const cli = cliStandIn(); await fence(admissions, cli); handler = (_req, res) => { @@ -734,7 +742,7 @@ test("admitted direct failures refuse without CLI fallback", async () => { }); test("a moved config, socket or executable binding is admitted through the CLI again", async () => { const path = await config("{}"); - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const cli = cliStandIn(); await fence(admissions, cli); await fence(admissions, cli); @@ -757,11 +765,14 @@ test("a moved config, socket or executable binding is admitted through the CLI a }); test("ineligible or unbindable selections and CLI admission failures stay on the CLI", async () => { for (const selection of [ + // The default context, whose fixture socket is absent. { DOCKER_HOST: undefined }, + // DOCKER_HOST with DOCKER_CONTEXT is treated as ambiguous. { DOCKER_CONTEXT: "selected-context" }, + { DOCKER_HOST: undefined, DOCKER_CONTEXT: "selected-context" }, { DOCKER_HOST: `unix://${join(root, "missing.sock")}` }, ]) { - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const cli = cliStandIn(); for (let index = 0; index < 2; index++) { expect( @@ -775,7 +786,7 @@ test("ineligible or unbindable selections and CLI admission failures stay on the expect(cli.calls()).toBe(2); } expect(requests).toHaveLength(0); - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const failure = new Error("synthetic CLI fence failure"); await expect( admissions({ @@ -818,7 +829,7 @@ async function withSelectedProcessEnvironment(body: () => Promise) { } test("cancelling an admitted direct fence keeps the CLI probe's closed cancellation without a process", async () => { await withSelectedProcessEnvironment(async () => { - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); await fence(admissions, cliStandIn()); const controller = new AbortController(); const probe = nativeComposeStorageIdentityProbe({ @@ -846,7 +857,7 @@ test("cancelling an admitted direct fence keeps the CLI probe's closed cancellat test("storage identity probe routes only the exact engine-ID query", async () => { const routed: string[] = []; await withSelectedProcessEnvironment(async () => { - const admissions = createNativeComposeEngineIdentityAdmissions(); + const admissions = createAdmissions(); const probe = nativeComposeStorageIdentityProbe({ probe: (args) => { routed.push(args.join(" ")); @@ -871,3 +882,213 @@ test("storage identity probe routes only the exact engine-ID query", async () => expect(requests).toHaveLength(2); }); }); + +// Docker context selections (DOCKER_HOST unset): the default context, the +// client config's currentContext and DOCKER_CONTEXT. Only admissions resolve +// them; the explicit observer, trusted without a CLI comparison, never does. +const CLIENT_CONTEXT = "fixture-context"; +function contextId(name: string): string { + return createHash("sha256").update(name).digest("hex"); +} +async function storeContext(name: string, host: string): Promise { + const directory = join(root, "client", "contexts", "meta", contextId(name)); + await mkdir(directory, { recursive: true }); + const path = join(directory, "meta.json"); + await writeFile( + path, + JSON.stringify({ + Name: name, + Metadata: {}, + Endpoints: { docker: { Host: host, SkipTLSVerify: false } }, + }) + ); + return path; +} +/** Select the fixture engine through the client config's currentContext. */ +async function contextSelection(): Promise<{ + readonly config: string; + readonly meta: string; +}> { + environment.DOCKER_HOST = undefined; + const meta = await storeContext(CLIENT_CONTEXT, `unix://${socket}`); + const path = await config(JSON.stringify({ currentContext: CLIENT_CONTEXT })); + return { config: path, meta }; +} +/** A second engine with its own request count. */ +async function otherEngine(id: string) { + const path = join(root, "other.sock"); + let served = 0; + const server = createServer((_req, res) => { + served++; + res.end(JSON.stringify({ ID: id })); + }); + server.on("connection", (peer) => connections.add(peer)); + servers.push(server); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(path, resolve); + }); + return { path, served: () => served }; +} + +test("context selections admit through the CLI, then read that context's socket", async () => { + await contextSelection(); + for (const selection of [ + {}, + // DOCKER_CONTEXT names the same store entry over an unrelated currentContext. + { DOCKER_CONTEXT: CLIENT_CONTEXT }, + ]) { + if ("DOCKER_CONTEXT" in selection) { + await config(JSON.stringify({ currentContext: "unrelated-context" })); + } + requests = []; + const admissions = createAdmissions(); + const cli = cliStandIn(); + for (let index = 0; index < 4; index++) { + expect( + await admissions({ + cli: cli.read, + timeoutMs: 5000, + environment: { ...environment, ...selection }, + }) + ).toBe(`"${ENGINE}"\n`); + } + expect([cli.calls(), requests.length]).toEqual([1, 4]); + } +}); +test("the default context admits its own endpoint, never a store entry named default", async () => { + environment.DOCKER_HOST = undefined; + await config("{}"); + const admissions = createAdmissions(socket); + const cli = cliStandIn(); + for (let index = 0; index < 3; index++) { + await fence(admissions, cli); + } + expect([cli.calls(), requests.length]).toEqual([1, 3]); + const other = await otherEngine(ENGINE); + await storeContext("default", `unix://${other.path}`); + environment.DOCKER_CONTEXT = "default"; + const named = createAdmissions(socket); + await fence(named, cli); + await fence(named, cli); + expect([cli.calls(), requests.length, other.served()]).toEqual([2, 5, 0]); +}); +test("DOCKER_HOST takes precedence: a current context never redirects the direct read", async () => { + const other = await otherEngine("other-engine"); + await storeContext(CLIENT_CONTEXT, `unix://${other.path}`); + await config(JSON.stringify({ currentContext: CLIENT_CONTEXT })); + const admissions = createAdmissions(); + const cli = cliStandIn(); + for (let index = 0; index < 3; index++) { + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + } + expect([cli.calls(), requests.length, other.served()]).toEqual([1, 3, 0]); +}); +test("a context naming another engine than the CLI's read is never admitted", async () => { + const { meta } = await contextSelection(); + const other = await otherEngine("other-engine"); + await writeFile( + meta, + JSON.stringify({ + Name: CLIENT_CONTEXT, + Endpoints: { docker: { Host: `unix://${other.path}` } }, + }) + ); + const admissions = createAdmissions(); + const cli = cliStandIn(); + for (let index = 0; index < 3; index++) { + expect(await fence(admissions, cli)).toBe(`"${ENGINE}"\n`); + } + expect([cli.calls(), requests.length, other.served()]).toEqual([3, 0, 1]); +}); +test("a changed config, context entry, TLS directory, socket or executable admits through the CLI again", async () => { + const selected = await contextSelection(); + const admissions = createAdmissions(); + const cli = cliStandIn(); + const twice = async (counts: readonly [number, number]) => { + await fence(admissions, cli); + await fence(admissions, cli); + expect([cli.calls(), requests.length]).toEqual([...counts]); + }; + await twice([1, 2]); + // Same bytes under a new inode: the config, then the context store entry. + for (const [index, path] of [selected.config, selected.meta].entries()) { + const bytes = await readFile(path); + await rename(path, join(root, `previous-${index}.json`)); + await writeFile(path, bytes); + await twice([2 + index, 4 + 2 * index]); + } + // Published TLS material keeps the CLI while present; it is never opened. + const tls = join( + root, + "client", + "contexts", + "tls", + contextId(CLIENT_CONTEXT) + ); + await mkdir(join(tls, "docker"), { recursive: true }); + await writeFile(join(tls, "docker", "key.pem"), CANARY, { mode: 0o000 }); + await twice([5, 6]); + await rm(tls, { recursive: true }); + await twice([6, 8]); + await rename(socket, join(root, "original.sock")); + await listen(socket); + await twice([7, 10]); + await rename(docker, join(root, "original-docker")); + await writeFile(docker, "#!/bin/sh\nexit 98\n", { mode: 0o700 }); + await twice([8, 12]); +}); +test("a context store change during an admitted read refuses without fallback", async () => { + const { meta } = await contextSelection(); + const admissions = createAdmissions(); + const cli = cliStandIn(); + await fence(admissions, cli); + handler = (_req, res) => { + void (async () => { + await writeFile( + meta, + JSON.stringify({ + Name: CLIENT_CONTEXT, + Endpoints: { docker: { Host: `unix://${join(root, "other.sock")}` } }, + }) + ); + res.end(JSON.stringify({ ID: ENGINE })); + })(); + }; + await refusal(() => fence(admissions, cli)); + expect([cli.calls(), requests.length]).toEqual([1, 2]); +}); +test("TLS, header and API-version environment keeps context selections on the CLI", async () => { + await contextSelection(); + for (const selection of [ + { DOCKER_TLS: "1" }, + { DOCKER_TLS_VERIFY: "1" }, + { DOCKER_CERT_PATH: join(root, "certificates") }, + { DOCKER_CUSTOM_HEADERS: `Private=${CANARY}` }, + { DOCKER_API_VERSION: "1.44" }, + ]) { + for (const context of [{}, { DOCKER_CONTEXT: "default" }]) { + const admissions = createAdmissions(socket); + const cli = cliStandIn(); + for (let index = 0; index < 2; index++) { + await admissions({ + cli: cli.read, + timeoutMs: 5000, + environment: { ...environment, ...selection, ...context }, + }); + } + expect(cli.calls()).toBe(2); + } + } + expect(requests).toHaveLength(0); +}); +test("the explicit observer never resolves a context or the default context", async () => { + await contextSelection(); + expect(createNativeComposeEngineIdentityObserver({ environment })).toBeNull(); + environment.DOCKER_CONTEXT = CLIENT_CONTEXT; + expect(createNativeComposeEngineIdentityObserver({ environment })).toBeNull(); + await config("{}"); + environment.DOCKER_CONTEXT = undefined; + expect(createNativeComposeEngineIdentityObserver({ environment })).toBeNull(); + expect(requests).toHaveLength(0); +}); diff --git a/tests/native-compose-storage-activation-command.test.ts b/tests/native-compose-storage-activation-command.test.ts index 93267d2d7..e7c5fae6d 100644 --- a/tests/native-compose-storage-activation-command.test.ts +++ b/tests/native-compose-storage-activation-command.test.ts @@ -2,6 +2,7 @@ import { afterEach, expect, test } from "bun:test"; import { join, resolve } from "node:path"; import { openNativeComposeGenerationStore } from "../src/lib/native-compose-generation.ts"; import { captureCompletedJobFixtureCommand } from "./e2e/scenarios/native-compose-adoption-job-worktrees.ts"; +import { isolatedDockerClientEnv } from "./helpers/docker-client.ts"; import { fixture, invoke } from "./helpers/native-compose-command.ts"; const storage = { data: { kind: "persistent", scope: "worktree" } } as const; @@ -82,6 +83,7 @@ async function ownedColdCase( HACK_COMPOSE_STARTUP_TIMEOUT_MS: "30000", CI: "1", HACK_EXECUTION_MODE: "non_interactive", + ...isolatedDockerClientEnv(root), }, captures: join(root, `storage-capture-${capture++}`), timeoutMs: 120_000, diff --git a/tests/native-compose-storage-fence-cost.test.ts b/tests/native-compose-storage-fence-cost.test.ts index 3f096511b..c0322ce87 100644 --- a/tests/native-compose-storage-fence-cost.test.ts +++ b/tests/native-compose-storage-fence-cost.test.ts @@ -1,4 +1,5 @@ import { afterEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import { createServer, type Server } from "node:http"; import type { Socket } from "node:net"; @@ -33,15 +34,46 @@ afterEach(async () => { await new Promise((resolve) => server.close(() => resolve())); } }); -/** A real Unix `GET /info` endpoint for the same engine the synthetic CLI reports. */ +/** + * A real Unix `GET /info` endpoint for the same engine the synthetic CLI reports, + * selected by an explicit DOCKER_HOST or, with `context`, by the client config's + * currentContext naming a context store entry for that socket. + */ async function engine( root: string, - id: (request: number) => string = () => "fixture-engine" + id: (request: number) => string = () => "fixture-engine", + selection: "host" | "context" = "host" ) { - const socket = join(root, "engine.sock"); - const client = join(root, "client"); + const socket = join(root, `${selection}-engine.sock`); + const client = join(root, `${selection}-client`); await mkdir(client, { mode: 0o700 }); - await writeFile(join(client, "config.json"), "{}\n", { mode: 0o600 }); + await writeFile( + join(client, "config.json"), + selection === "host" + ? "{}\n" + : `${JSON.stringify({ currentContext: "fixture-context" })}\n`, + { mode: 0o600 } + ); + if (selection === "context") { + const meta = join( + client, + "contexts", + "meta", + createHash("sha256").update("fixture-context").digest("hex") + ); + await mkdir(meta, { recursive: true, mode: 0o755 }); + await writeFile( + join(meta, "meta.json"), + JSON.stringify({ + Name: "fixture-context", + Metadata: {}, + Endpoints: { + docker: { Host: `unix://${socket}`, SkipTLSVerify: false }, + }, + }), + { mode: 0o644 } + ); + } let served = 0; const server = createServer((req, res) => { const request = served++; @@ -58,10 +90,11 @@ async function engine( server.once("error", reject); server.listen(socket, resolve); }); - return { - env: { DOCKER_HOST: `unix://${socket}`, DOCKER_CONFIG: client }, - served: () => served, - }; + const env: Readonly> = + selection === "host" + ? { DOCKER_HOST: `unix://${socket}`, DOCKER_CONFIG: client } + : { DOCKER_CONFIG: client }; + return { env, served: () => served }; } async function lines(root: string, name: string): Promise { const file = Bun.file(join(root, name)); @@ -111,7 +144,8 @@ async function enrolled() { test("storage fence count per command: every proof kept, exact duplicates gone, one CLI admission", async () => { const root = await enrolled(); - // CLI transport (no explicit Unix selection): every fence is a `docker info`. + // CLI transport (the isolated client selects a context with no store entry): + // every fence is a `docker info`. expect(await counted(root, ["exec", "web", "--", "true"])).toEqual({ code: 0, cli: FILE_ENGINE_FENCES + 2 * PROOF_FENCES, @@ -139,6 +173,17 @@ test("storage fence count per command: every proof kept, exact duplicates gone, direct: 6 * PROOF_FENCES, observations: 6 * PROOF_OBSERVATIONS, }); + // Admitted context selection (DOCKER_HOST unset; the client config's current + // context names the socket): the same counts as an explicit Unix host. + const contextual = await engine(root, undefined, "context"); + expect( + await counted(root, ["exec", "web", "--", "true"], contextual) + ).toEqual({ + code: 0, + cli: FILE_ENGINE_FENCES + 1, + direct: 2 * PROOF_FENCES, + observations: 2 * PROOF_OBSERVATIONS, + }); }, 390_000); test("a refused admission keeps every storage fence on the Docker CLI for the command", async () => { From c2ecf9cb321d3865d1d5d0ab059a225e5b5904b8 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 04:47:12 -0400 Subject: [PATCH 25/32] chore(x04): temporary witness probe trace for hosted measurement Temporary measurement only; reverted before the final commit. Re-applies the X03 closed-label trace (6a4e118e) without its identity benchmark, labels each engine-ID admission with its selection kind (host, default or context), counts direct reads as trace leaves, and prints the runner's Docker client selection. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 25 ++ scripts/x01-trace-summary.ts | 285 ++++++++++++++++++ src/lib/native-compose-command-storage.ts | 14 +- src/lib/native-compose-command.ts | 9 +- src/lib/native-compose-engine-identity.ts | 18 +- src/lib/native-compose-ownership.ts | 5 +- src/lib/native-compose-phase-trace.ts | 8 +- ...tive-compose-storage-command-executable.ts | 4 + ...ompose-storage-witness-docker-inventory.ts | 8 +- .../native-compose-storage-witness-docker.ts | 45 ++- src/lib/native-compose-storage-witness.ts | 34 ++- src/lib/shell.ts | 17 +- src/lib/x01-cost-probe.ts | 176 +++++++++++ tests/e2e/harness.ts | 9 + .../e2e/native-config-networks-acceptance.ts | 4 + 15 files changed, 629 insertions(+), 32 deletions(-) create mode 100644 scripts/x01-trace-summary.ts create mode 100644 src/lib/x01-cost-probe.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 695b9a0d2..419fb1a1d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -304,13 +304,22 @@ jobs: - name: Verify Docker context endpoint selection against the real Docker CLI if: matrix.shard == 4 run: HACK_TEST_DOCKER_CLI=1 bun test tests/docker-context.test.ts + # TEMPORARY X04 measurement; reverted before the final commit. + - name: X04 runner Docker client selection + run: | + docker context inspect --format 'context={{.Name}} host={{.Endpoints.docker.Host}}' || true + if [ -f "$HOME/.docker/config.json" ]; then echo "client-config=present"; else echo "client-config=absent"; fi + echo "DOCKER_HOST set=${DOCKER_HOST:+yes} DOCKER_CONTEXT set=${DOCKER_CONTEXT:+yes} DOCKER_CONFIG set=${DOCKER_CONFIG:+yes}" - name: Run local and Docker E2E shard env: HACK_E2E_PROCESS_POLICY_REFUSAL_CAPSULE: "1" HACK_E2E_CAPSULE_DIR: ${{ runner.temp }}/process-policy-capsules HACK_E2E_CLI_BIN: ${{ github.workspace }}/dist/hack HACK_E2E_SHARD: ${{ matrix.shard }}/4 + # TEMPORARY X01 measurement; reverted before the final commit. + HACK_X01_TRACE_DIR: ${{ runner.temp }}/x01-trace run: | + mkdir -p "$HACK_X01_TRACE_DIR" HACK_E2E_DOCKER_HOST="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" HACK_E2E_NETWORK_SOURCE_REVISION="$(git rev-parse HEAD)" HACK_E2E_NETWORK_CLI_SHA256="$(sha256sum dist/hack | cut -d ' ' -f1)" @@ -319,6 +328,22 @@ jobs: HACK_E2E_COMPOSE_PLUGIN_SHA256="$(sha256sum "$HACK_E2E_COMPOSE_PLUGIN_PATH" | cut -d ' ' -f1)" export HACK_E2E_DOCKER_HOST HACK_E2E_NETWORK_SOURCE_REVISION HACK_E2E_NETWORK_CLI_SHA256 HACK_E2E_NETWORK_COMPILER_SHA256 HACK_E2E_COMPOSE_PLUGIN_PATH HACK_E2E_COMPOSE_PLUGIN_SHA256 HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker --shard="$HACK_E2E_SHARD" + # TEMPORARY X01 measurement; reverted before the final commit. + - name: X01 witness cost summary + if: always() + run: | + for f in "$RUNNER_TEMP"/x01-trace/*.jsonl; do + [ -f "$f" ] || continue + bun scripts/x01-trace-summary.ts "$f" "$(basename "$f" .jsonl)" || true + done + - name: X01 witness cost trace artifact + if: always() + uses: actions/upload-artifact@v4 + with: + name: x01-trace-shard-${{ matrix.shard }} + path: ${{ runner.temp }}/x01-trace + if-no-files-found: ignore + retention-days: 7 - name: Preserve process-policy capsule and startup diagnostic # Closed fixed-field JSON only (counts, enums, booleans); the gh log viewer truncates it inline. if: failure() diff --git a/scripts/x01-trace-summary.ts b/scripts/x01-trace-summary.ts new file mode 100644 index 000000000..62ace753c --- /dev/null +++ b/scripts/x01-trace-summary.ts @@ -0,0 +1,285 @@ +// TEMPORARY X01 measurement summary. Reverted before the final commit. +// Reads closed-label JSONL written by src/lib/x01-cost-probe.ts. +type Event = { + readonly p: number; + readonly v: string; + readonly k: string; + readonly s?: string; + readonly l?: string; + readonly ms?: number; + readonly t?: number; + readonly c?: number; + readonly line?: { + readonly phase?: string; + readonly boundary?: string; + readonly durationMs?: number | null; + }; +}; +const WITNESS = new Set([ + "storage-prepare", + "storage-verify", + "carrier-new", + "inspect", + "provision", + "finish", + "witness-verify", + "witness-prepare", + "witness-enroll", +]); +const isWitness = (label: string) => + WITNESS.has(label) || label.startsWith("invoke:"); +// TEMPORARY X04: direct engine-ID reads (`ident:direct`, `ident:admission`) are leaves too. +const LEAF = new Set(["probe", "run", "exec", "hash", "ident"]); +const PHASE = /^(oneoff|storage|guard|finish|finalize)\.[a-z-]+$/; + +const path = process.argv[2]; +const title = process.argv[3] ?? path; +if (!path) { + throw new Error("usage: x01-trace-summary [title]"); +} +const text = await Bun.file(path).text(); +const events: Event[] = []; +for (const line of text.split("\n")) { + if (line.trim() === "") { + continue; + } + try { + events.push(JSON.parse(line) as Event); + } catch { + // skip partial line + } +} +const byProcess = new Map(); +for (const event of events) { + const list = byProcess.get(event.p) ?? []; + list.push(event); + byProcess.set(event.p, list); +} +type Proc = { + readonly pid: number; + readonly verb: string; + readonly totalMs: number; + readonly witnessMs: number; + readonly events: Event[]; + readonly kind: string; +}; +function topWitnessMs(list: Event[]): number { + let total = 0; + for (const event of list) { + if (event.k !== "scope" || !event.l || !isWitness(event.l)) { + continue; + } + const parts = (event.s ?? "").split(">"); + parts.pop(); + if (parts.some(isWitness)) { + continue; + } + total += event.ms ?? 0; + } + return total; +} +const procs: Proc[] = []; +for (const [pid, list] of byProcess) { + const exit = list.find((event) => event.k === "exit"); + const verb = list[0]?.v ?? "other"; + const labels = new Set(list.map((event) => event.l ?? "")); + let kind = verb; + if (verb === "up" || verb === "restart" || verb === "run") { + if (labels.has("invoke:seed")) { + kind = `${verb}(cold)`; + } else if (labels.has("invoke:verify")) { + kind = `${verb}(warm)`; + } else { + kind = `${verb}(no-witness)`; + } + } else if (verb === "exec") { + kind = labels.has("invoke:verify") ? "exec(storage)" : "exec(image)"; + } + procs.push({ + pid, + verb, + totalMs: exit?.t ?? Math.max(...list.map((event) => event.t ?? 0)), + witnessMs: topWitnessMs(list), + events: list, + kind, + }); +} +const fmt = (ms: number) => (ms / 1000).toFixed(1); +const out: string[] = []; +out.push(`#### X01 trace: ${title}`); +out.push(`events=${events.length} processes=${procs.length}`); +out.push(""); +out.push( + "kind | n | avg total s | sum total s | avg witness s | sum witness s" +); +const kinds = [...new Set(procs.map((proc) => proc.kind))].sort(); +for (const kind of kinds) { + const selected = procs.filter((proc) => proc.kind === kind); + const total = selected.reduce((sum, proc) => sum + proc.totalMs, 0); + const witness = selected.reduce((sum, proc) => sum + proc.witnessMs, 0); + out.push( + `${kind} | ${selected.length} | ${fmt(total / selected.length)} | ${fmt(total)} | ${fmt(witness / selected.length)} | ${fmt(witness)}` + ); +} +const allTotal = procs.reduce((sum, proc) => sum + proc.totalMs, 0); +const allWitness = procs.reduce((sum, proc) => sum + proc.witnessMs, 0); +out.push( + `ALL | ${procs.length} | - | ${fmt(allTotal)} | - | ${fmt(allWitness)}` +); +out.push(""); +// TEMPORARY X04: engine-ID admissions by selection kind and outcome. +const admissions = new Map(); +for (const event of events) { + if (event.k === "ident-admit" && event.l) { + admissions.set(event.l, (admissions.get(event.l) ?? 0) + 1); + } +} +out.push( + `ident admissions: ${ + [...admissions.entries()] + .sort() + .map(([label, count]) => `${label}=${count}`) + .join(" ") || "none" + }` +); +out.push(""); + +function leafTable(selected: Proc[], heading: string, limit = 24) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (!(LEAF.has(event.k) && event.l)) { + continue; + } + const key = `${event.k}:${event.l}`; + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push(`${heading} (per process avg over n=${selected.length})`); + out.push("leaf | calls/proc | s/proc | ms/call"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function leafByScope(selected: Proc[], heading: string, limit = 40) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (!(LEAF.has(event.k) && event.l)) { + continue; + } + const parts = (event.s ?? "") + .split(">") + .filter( + (part) => + isWitness(part) || part.startsWith("cmd:") || PHASE.test(part) + ); + const inner = parts.slice(-2).join(">") || "(none)"; + const key = `${inner} :: ${event.k}:${event.l}`; + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push( + `${heading} leaves by scope (per process avg over n=${selected.length})` + ); + out.push("scope :: leaf | calls/proc | s/proc | ms/call"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function scopeTable(selected: Proc[], heading: string, limit = 30) { + const rows = new Map(); + for (const proc of selected) { + for (const event of proc.events) { + if (event.k !== "scope" || !event.l) { + continue; + } + const parts = (event.s ?? "").split(">"); + const key = parts + .filter( + (part) => + PHASE.test(part) || + isWitness(part) || + part.startsWith("xattr:") || + part.startsWith("cmd:") || + part.startsWith("exec-") + ) + .join(">"); + if (key === "") { + continue; + } + const row = rows.get(key) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.ms ?? 0; + rows.set(key, row); + } + } + out.push(`${heading} scopes (per process avg over n=${selected.length})`); + out.push("scope path | n/proc | s/proc | ms/each"); + const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); + for (const [key, row] of sorted.slice(0, limit)) { + out.push( + `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` + ); + } + out.push(""); +} +function phaseRows(proc: Proc) { + const rows = new Map(); + for (const event of proc.events) { + if ( + event.k === "phase" && + event.line?.boundary !== "begin" && + event.line?.phase + ) { + const row = rows.get(event.line.phase) ?? { n: 0, ms: 0 }; + row.n += 1; + row.ms += event.line.durationMs ?? 0; + rows.set(event.line.phase, row); + } + } + return rows; +} +for (const kind of [ + "up(cold)", + "up(warm)", + "exec(storage)", + "restart(warm)", + "run(warm)", + "run(cold)", + "down", +]) { + const selected = procs.filter((proc) => proc.kind === kind); + if (selected.length === 0) { + continue; + } + const sample = selected[0] as Proc; + out.push( + `### ${kind}: first sample total ${fmt(sample.totalMs)} s, witness ${fmt(sample.witnessMs)} s` + ); + out.push("phase-trace phase | spans | s"); + for (const [phase, row] of phaseRows(sample)) { + out.push(`${phase} | ${row.n} | ${fmt(row.ms)}`); + } + out.push(""); + scopeTable([sample], `${kind} first sample`, 40); + leafTable([sample], `${kind} first sample leaves`); + scopeTable(selected, `${kind} all`, 30); + leafTable(selected, `${kind} all leaves`); + leafByScope(selected, `${kind} all`); +} +console.log(out.join("\n")); diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index c9384e64c..d9b58c132 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -23,6 +23,7 @@ import { } from "./native-compose-storage-witness.ts"; import { createNativeComposeDockerStorageXattrCarrier } from "./native-compose-storage-witness-docker.ts"; import type { NativeComposeStorageXattrCarrier } from "./native-compose-storage-witness-xattr-carrier.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; type Selection = { readonly name: string; readonly storage: string }; type Document = Readonly>; @@ -134,7 +135,12 @@ export type NativeComposeCommandStorageVerifyOptions = { /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ -export async function prepareNativeComposeCommandStorage(opts: { +export function prepareNativeComposeCommandStorage( + opts: Parameters[0] +): ReturnType { + return x01Scope("storage-prepare", () => x01Prepare(opts)); +} +async function x01Prepare(opts: { readonly store: NativeComposeGenerationStore; readonly mutation: NativeComposeMutation; readonly operation: "up" | "restart" | "run" | "exec"; @@ -225,7 +231,11 @@ export async function prepareNativeComposeCommandStorage(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = async ( + const verify = ( + generation: NativeComposeGeneration, + options: NativeComposeCommandStorageVerifyOptions = {} + ) => x01Scope("storage-verify", () => x01Verify(generation, options)); + const x01Verify = async ( generation: NativeComposeGeneration, options: NativeComposeCommandStorageVerifyOptions = {} ) => { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index 8c5dcaf82..0aa92b81e 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -138,6 +138,7 @@ import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; export type NativeComposeCommandOptions = { @@ -943,7 +944,9 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = async ( + const assertSaved = (proof: NativeComposeCommandStorageVerifyOptions) => + x01Scope("exec-assert-saved", () => x01AssertSaved(proof)); + const x01AssertSaved = async ( proof: NativeComposeCommandStorageVerifyOptions ) => { const latest = await store.loadCurrent(); @@ -962,7 +965,9 @@ async function savedCommand(opts: { signal, assertSaved, run: () => - runSavedProcess({ options, generation, document, base, signal }), + x01Scope("exec-process", () => + runSavedProcess({ options, generation, document, base, signal }) + ), }); }); } diff --git a/src/lib/native-compose-engine-identity.ts b/src/lib/native-compose-engine-identity.ts index d9b31f739..d19ec5ed0 100644 --- a/src/lib/native-compose-engine-identity.ts +++ b/src/lib/native-compose-engine-identity.ts @@ -8,6 +8,7 @@ import { } from "./docker-context.ts"; import { isRecord } from "./guards.ts"; import { bindNativeComposeEngineConfig } from "./native-compose-engine-config.ts"; +import { x01Mark, x01Time } from "./x01-cost-probe.ts"; const ENGINE_ID = /^[A-Za-z0-9:-]{1,128}$/; const AMBIGUOUS_PATH = /[%?#\0\r\n]/; @@ -300,6 +301,8 @@ type BoundObserver = { /** Opaque digest of the exact selection, executable, config and socket binding. */ readonly key: string; readonly read: () => Promise; + /** TEMPORARY X04 measurement: closed selection kind label. */ + readonly kind: string; }; function bindObserver(opts: { readonly signal?: AbortSignal; @@ -370,8 +373,15 @@ function bindObserver(opts: { JSON.stringify([selected, binary, config, endpoint, socket, executable]) ) .digest("hex"); + const label = Array.isArray(endpoint.binding) + ? endpoint.binding[0] + : undefined; return Object.freeze({ key, + kind: + label === "host" || label === "default" || label === "context" + ? label + : "other", read: async () => { if (reading) { return refuse(); @@ -469,18 +479,22 @@ export function createNativeComposeEngineIdentityAdmissions( return await opts.cli(); } const state = admitted.get(bound.key); + const direct = bound.read; if (state === true) { - return `${JSON.stringify(await bound.read())}\n`; + return `${JSON.stringify(await x01Time("ident", "direct", direct))}\n`; } const text = await opts.cli(); if (state === undefined) { let equal = false; try { const value: unknown = JSON.parse(text); - equal = typeof value === "string" && value === (await bound.read()); + equal = + typeof value === "string" && + value === (await x01Time("ident", "admission", direct)); } catch { equal = false; } + x01Mark("ident-admit", `${bound.kind}:${equal ? "admitted" : "refused"}`); admitted.set(bound.key, equal); } return text; diff --git a/src/lib/native-compose-ownership.ts b/src/lib/native-compose-ownership.ts index 0019b3fc3..d8bc9dc8a 100644 --- a/src/lib/native-compose-ownership.ts +++ b/src/lib/native-compose-ownership.ts @@ -3,6 +3,7 @@ import { isRecord } from "./guards.ts"; import { nativeComposeVolumeCreatedAt } from "./native-compose-retained-storage.ts"; import { beginNativeCpuChild } from "./native-cpu-diagnostics.ts"; import { findExecutableInPath } from "./shell.ts"; +import { x01Label, x01Time } from "./x01-cost-probe.ts"; const PROJECT = /^[a-z0-9][a-z0-9_-]*$/; const SERVICE = /^[a-z0-9][a-z0-9._-]{0,62}$/; @@ -1232,7 +1233,7 @@ export function createNativeComposeProbe( probeRefused("E_NATIVE_COMPOSE_PROBE", "operation"); } const environment = { ...process.env }; - return async (args) => { + const x01Inner = async (args: readonly string[]): Promise => { checkInterruption(opts.signal, false); const remainingTime = deadline - Date.now(); if (remainingTime <= 0) { @@ -1310,6 +1311,8 @@ export function createNativeComposeProbe( observeCpu(probeExitCode(completion[2])); } }; + return (args) => + x01Time("probe", x01Label(args, false), () => x01Inner(args)); } function probeExitCode( diff --git a/src/lib/native-compose-phase-trace.ts b/src/lib/native-compose-phase-trace.ts index 540f833b0..943b79320 100644 --- a/src/lib/native-compose-phase-trace.ts +++ b/src/lib/native-compose-phase-trace.ts @@ -1,5 +1,6 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { writeSync } from "node:fs"; +import { x01Enabled, x01Phase, x01Scope } from "./x01-cost-probe.ts"; export const NATIVE_COMPOSE_PHASE_TRACE = "HACK_NATIVE_COMPOSE_PHASE_TRACE"; const PHASES = [ @@ -135,6 +136,8 @@ export function withNativeComposePhaseTrace(operation: () => Promise) { writeSync(2, line); }, }); + } else if (x01Enabled()) { + trace = createNativeComposePhaseTrace({ write: x01Phase }); } } catch { // Diagnostic setup never changes selection, results or effect authority. @@ -147,5 +150,8 @@ export function measureNativeComposePhase( phase: Phase, operation: () => Promise ): Promise { - return context.getStore()?.measure(phase, operation) ?? operation(); + return x01Scope( + phase, + () => context.getStore()?.measure(phase, operation) ?? operation() + ); } diff --git a/src/lib/native-compose-storage-command-executable.ts b/src/lib/native-compose-storage-command-executable.ts index a8817d685..780dae040 100644 --- a/src/lib/native-compose-storage-command-executable.ts +++ b/src/lib/native-compose-storage-command-executable.ts @@ -2,6 +2,7 @@ import { constants } from "node:fs"; import { type FileHandle, lstat, open, realpath } from "node:fs/promises"; import type { NativeComposeStorageCommandExecutable } from "./native-compose-storage-command-record.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Label, x01Mark } from "./x01-cost-probe.ts"; /** Coarser than any supported filesystem timestamp granularity (1 s on HFS+/ext3). */ const RACY_MS = 2000; @@ -83,6 +84,9 @@ export function createNativeComposeStorageCommandExecutableReader( const size = Number(before.size); const started = now(); const cached = settled.get(canonical); + if (cached?.identity === identity) { + x01Mark("hash-cached", x01Label([canonical])); + } const hash = cached?.identity === identity ? cached.hash : await digest(file, size); const after = await file.stat({ bigint: true }), diff --git a/src/lib/native-compose-storage-witness-docker-inventory.ts b/src/lib/native-compose-storage-witness-docker-inventory.ts index b5bdb7410..f292021ce 100644 --- a/src/lib/native-compose-storage-witness-docker-inventory.ts +++ b/src/lib/native-compose-storage-witness-docker-inventory.ts @@ -8,6 +8,7 @@ import { type NativeComposeStorageXattrTarget, } from "./native-compose-storage-witness-xattr-carrier.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$/; @@ -164,7 +165,12 @@ export async function assertNativeComposeStorageDockerCarrierVolume(opts: { /** Complete holder reads include foreign containers. The two inventories and selected * volume reread are observed fences; they do not claim an atomic daemon transaction. */ -export async function observeNativeComposeStorageDockerTarget(opts: { +export function observeNativeComposeStorageDockerTarget( + opts: Parameters[0] +): ReturnType { + return x01Scope("target", () => x01ObserveTarget(opts)); +} +async function x01ObserveTarget(opts: { readonly probe: Probe; readonly selection: Selection; readonly current: NativeComposeMaterialBinding; diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index c8e062188..d955faf06 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -98,6 +98,7 @@ import { refuseNativeComposeStorageXattr as refuse, } from "./native-compose-storage-witness-xattr-codec.ts"; import { findExecutableInPath, type RunExitEvent, run } from "./shell.ts"; +import { x01Label, x01Mark, x01Scope, x01Time } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const PROCESS = @@ -817,7 +818,10 @@ async function groupAbsent( } /** Every identity check runs per call; only an unchanged stat identity skips the * content read for the rest of this command. */ -const commandExecutable = readNativeComposeStorageCommandExecutable; +const commandExecutable = (path: string) => + x01Time("hash", x01Label([path]), () => + readNativeComposeStorageCommandExecutable(path) + ); async function boundedHostFile(path: string): Promise { const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); try { @@ -903,7 +907,14 @@ async function commandBoot( } /** No callback can reject before the shared owner awaits exit. A rejected owner or * unproven group retains the journal; daemon effects are never inferred from exit. */ -export async function runNativeComposeStorageDockerCommand(supplied: { +export function runNativeComposeStorageDockerCommand( + supplied: Parameters[0] +): ReturnType { + return x01Scope(`cmd:${x01Label(supplied.args)}`, () => + x01RunCommandInner(supplied) + ); +} +async function x01RunCommandInner(supplied: { readonly context: Pick; readonly files: Pick< Files, @@ -1248,7 +1259,12 @@ function matchesScope( * image; other daemons and uncached images refuse rather than pull or emulate. * PostgreSQL data roots and crash durability remain unqualified. No module * initialization executes Docker or the helper. */ -export async function createNativeComposeDockerStorageXattrCarrier( +export function createNativeComposeDockerStorageXattrCarrier( + opts: Context +): Promise { + return x01Scope("carrier-new", () => x01CreateCarrier(opts)); +} +async function x01CreateCarrier( opts: Context ): Promise { const context: Context = Object.freeze({ ...opts }); @@ -1370,6 +1386,7 @@ export async function createNativeComposeDockerStorageXattrCarrier( const binding = await current(context, selection); if (same(binding, closed.binding)) { lifetime(context); + x01Mark("target-reused", "closing"); return closed.target; } } catch (error) { @@ -2034,17 +2051,21 @@ export async function createNativeComposeDockerStorageXattrCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect: inspectPort, - provision, - invoke, + inspect: (selection, options) => + x01Scope("inspect", () => inspectPort(selection, options)), + provision: (selection) => + x01Scope("provision", () => provision(selection)), + invoke: (input) => + x01Scope(`invoke:${input.request.operation}`, () => invoke(input)), finish: context.originalCommandRecords === true - ? async ({ input, completion }) => { - closing = null; - const finish = finalizers.get(input) ?? refuse(); - finalizers.delete(input); - await finish(completion); - } + ? ({ input, completion }) => + x01Scope("finish", async () => { + closing = null; + const finish = finalizers.get(input) ?? refuse(); + finalizers.delete(input); + await finish(completion); + }) : undefined, }, signal: context.signal, diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index 7586eb73b..1fc68abf7 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -78,6 +78,7 @@ import { nativeComposeStorageXattrValueMatches, sameNativeComposeStorageXattrRoot, } from "./native-compose-storage-witness-xattr-codec.ts"; +import { x01Scope } from "./x01-cost-probe.ts"; export type { NativeComposeStorageWitnessReference } from "./native-compose-storage-witness-state.ts"; @@ -498,7 +499,16 @@ function requireXattrAdmission( refuse(); } } -async function invokeXattr(opts: { +function invokeXattr( + opts: Parameters[0] +): ReturnType { + const operation = + isRecord(opts.request) && typeof opts.request.operation === "string" + ? opts.request.operation + : "unknown"; + return x01Scope(`xattr:${operation}`, () => x01InvokeXattr(opts)); +} +async function x01InvokeXattr(opts: { readonly carrier: NativeComposeStorageXattrCarrier; readonly saved: XattrExpectation; readonly volume: NativeComposeRetainedVolume; @@ -1464,17 +1474,21 @@ export async function prepareNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - return await prepareNativeComposeStorageWitness({ - ...input, - xattrCarrier: carrier, - }); + return await x01Scope("witness-prepare", () => + prepareNativeComposeStorageWitness({ + ...input, + xattrCarrier: carrier, + }) + ); } export async function enrollNativeComposeStorageXattrWitness(opts: { readonly enrollment: NativeComposeStorageWitnessEnrollment; }): Promise { - return await enrollNativeComposeStorageWitness({ - enrollment: opts.enrollment, - }); + return await x01Scope("witness-enroll", () => + enrollNativeComposeStorageWitness({ + enrollment: opts.enrollment, + }) + ); } export async function verifyNativeComposeStorageXattrWitness( opts: Omit< @@ -1485,7 +1499,9 @@ export async function verifyNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - await verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }); + await x01Scope("witness-verify", () => + verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }) + ); } /** Private saved observation seam. It never performs a kernel read, repairs an diff --git a/src/lib/shell.ts b/src/lib/shell.ts index ef9faf1ba..ff27e6b85 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -4,6 +4,7 @@ import { } from "./native-cpu-diagnostics.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; +import { x01Label, x01Time } from "./x01-cost-probe.ts"; export interface ExecResult { readonly exitCode: number; @@ -40,7 +41,13 @@ function buildSpawnEnv( return override ? { ...base, ...override } : base; } -export async function exec( +export function exec( + cmd: readonly string[], + opts: ExecOptions = {} +): Promise { + return x01Time("exec", x01Label(cmd), () => x01ExecInner(cmd, opts)); +} +async function x01ExecInner( cmd: readonly string[], opts: ExecOptions = {} ): Promise { @@ -123,7 +130,13 @@ export type RunExitEvent = { readonly maxRssBytes: number | null; }; -export async function run( +export function run( + cmd: readonly string[], + opts: RunOptions = {} +): Promise { + return x01Time("run", x01Label(cmd), () => x01RunInner(cmd, opts)); +} +async function x01RunInner( cmd: readonly string[], opts: RunOptions = {} ): Promise { diff --git a/src/lib/x01-cost-probe.ts b/src/lib/x01-cost-probe.ts new file mode 100644 index 000000000..17b2f5552 --- /dev/null +++ b/src/lib/x01-cost-probe.ts @@ -0,0 +1,176 @@ +// TEMPORARY X01 measurement probe. Reverted before the final commit. +// Writes closed labels and durations only (no ids, paths or values). +import { AsyncLocalStorage } from "node:async_hooks"; +import { appendFileSync } from "node:fs"; + +const FILE = process.env.HACK_X01_TRACE_FILE; +const VERBS = new Set([ + "up", + "down", + "exec", + "run", + "restart", + "ps", + "logs", + "status", + "init", +]); +const SUBS = new Set([ + "info", + "image", + "container", + "volume", + "network", + "compose", + "create", + "start", + "rm", + "inspect", + "ls", + "ps", + "exec", + "run", + "logs", + "version", + "events", + "stop", + "kill", + "wait", + "up", + "down", + "config", + "pull", + "build", + "restart", + "port", + "cp", + "top", +]); +const verb = (() => { + for (const arg of process.argv.slice(2)) { + if (VERBS.has(arg)) { + return arg; + } + } + return "other"; +})(); +const scopes = new AsyncLocalStorage(); +let enabled = typeof FILE === "string" && FILE.startsWith("/"); +let wroteStart = false; + +function append(record: Record): void { + appendFileSync( + FILE as string, + `${JSON.stringify({ p: process.pid, v: verb, ...record })}\n` + ); +} + +function write(record: Record): void { + if (!enabled) { + return; + } + try { + if (!wroteStart) { + wroteStart = true; + append({ k: "start", t: Math.round(performance.now()) }); + process.on("exit", (code) => { + try { + append({ k: "exit", c: code, t: Math.round(performance.now()) }); + } catch { + // measurement only + } + }); + } + append({ + s: scopes.getStore() ?? "", + t: Math.round(performance.now()), + ...record, + }); + } catch { + enabled = false; + } +} + +export function x01Enabled(): boolean { + return enabled; +} + +/** Raw line from the phase trace, re-wrapped with process identity. */ +export function x01Phase(line: string): void { + try { + write({ k: "phase", line: JSON.parse(line) as unknown }); + } catch { + enabled = false; + } +} + +export function x01Label(argv: readonly string[], binary = true): string { + const parts: string[] = []; + let rest = argv; + if (binary) { + const name = (argv[0] ?? "").split("/").at(-1) ?? ""; + parts.push( + ["docker", "ps", "sysctl", "sh", "git", "bun", "tmux"].includes(name) + ? name + : "bin" + ); + rest = argv.slice(1); + } else { + parts.push("docker"); + } + for (const token of rest) { + if (parts.length >= 3) { + break; + } + if (token.startsWith("-")) { + continue; + } + if (SUBS.has(token)) { + parts.push(token); + } else { + break; + } + } + return parts.join(" "); +} + +export async function x01Time( + kind: string, + label: string, + operation: () => Promise +): Promise { + if (!enabled) { + return await operation(); + } + const started = performance.now(); + let ok = false; + try { + const result = await operation(); + ok = true; + return result; + } finally { + write({ + k: kind, + l: label, + ms: Math.round((performance.now() - started) * 10) / 10, + ok, + }); + } +} + +/** TEMPORARY X03 measurement: a closed-label point event. */ +export function x01Mark(kind: string, label: string): void { + write({ k: kind, l: label }); +} + +export async function x01Scope( + label: string, + operation: () => Promise +): Promise { + if (!enabled) { + return await operation(); + } + const parent = scopes.getStore(); + const path = parent ? `${parent}>${label}` : label; + return await scopes.run(path, () => x01Time("scope", label, operation)); +} diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index 2201457b1..e7599c528 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,6 +43,8 @@ const PASSTHROUGH_ENV_KEYS = [ "LC_ALL", "DOCKER_HOST", "DOCKER_CONFIG", + // TEMPORARY X01 measurement passthrough; reverted before the final commit. + "HACK_X01_TRACE_FILE", ] as const; export type CliResult = { @@ -768,6 +770,13 @@ export async function runScenarios( }; process.stdout.write(`-- ${scenario.name} (${scenario.tier}) --\n`); + // TEMPORARY X01 measurement: one trace file per scenario. + if (process.env.HACK_X01_TRACE_DIR?.startsWith("/")) { + process.env.HACK_X01_TRACE_FILE = join( + process.env.HACK_X01_TRACE_DIR, + `${scenario.name}.jsonl` + ); + } let outcome: ScenarioOutcome; let preserveFailedFixture = false; try { diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index 633cb0449..6c5194702 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -1154,6 +1154,10 @@ export const nativeConfigNetworksScenario: Scenario = { NO_COLOR: "1", HACK_RUNTIME_BACKEND: "compose", HACK_DAEMON_DISABLE_DOCKER_EVENTS: "1", + // TEMPORARY X01 measurement passthrough; reverted before the final commit. + ...(process.env.HACK_X01_TRACE_FILE + ? { HACK_X01_TRACE_FILE: process.env.HACK_X01_TRACE_FILE } + : {}), }; const engine = Bun.which("docker"); const git = Bun.which("git"); From 5cf9f0f3eef372541afac832ce4b30cada6c26fc Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 04:48:03 -0400 Subject: [PATCH 26/32] chore(x04): revert temporary witness probe trace Measurement is complete; the hosted trace run was 38225752202. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 25 -- scripts/x01-trace-summary.ts | 285 ------------------ src/lib/native-compose-command-storage.ts | 14 +- src/lib/native-compose-command.ts | 9 +- src/lib/native-compose-engine-identity.ts | 18 +- src/lib/native-compose-ownership.ts | 5 +- src/lib/native-compose-phase-trace.ts | 8 +- ...tive-compose-storage-command-executable.ts | 4 - ...ompose-storage-witness-docker-inventory.ts | 8 +- .../native-compose-storage-witness-docker.ts | 45 +-- src/lib/native-compose-storage-witness.ts | 34 +-- src/lib/shell.ts | 17 +- src/lib/x01-cost-probe.ts | 176 ----------- tests/e2e/harness.ts | 9 - .../e2e/native-config-networks-acceptance.ts | 4 - 15 files changed, 32 insertions(+), 629 deletions(-) delete mode 100644 scripts/x01-trace-summary.ts delete mode 100644 src/lib/x01-cost-probe.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 419fb1a1d..695b9a0d2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -304,22 +304,13 @@ jobs: - name: Verify Docker context endpoint selection against the real Docker CLI if: matrix.shard == 4 run: HACK_TEST_DOCKER_CLI=1 bun test tests/docker-context.test.ts - # TEMPORARY X04 measurement; reverted before the final commit. - - name: X04 runner Docker client selection - run: | - docker context inspect --format 'context={{.Name}} host={{.Endpoints.docker.Host}}' || true - if [ -f "$HOME/.docker/config.json" ]; then echo "client-config=present"; else echo "client-config=absent"; fi - echo "DOCKER_HOST set=${DOCKER_HOST:+yes} DOCKER_CONTEXT set=${DOCKER_CONTEXT:+yes} DOCKER_CONFIG set=${DOCKER_CONFIG:+yes}" - name: Run local and Docker E2E shard env: HACK_E2E_PROCESS_POLICY_REFUSAL_CAPSULE: "1" HACK_E2E_CAPSULE_DIR: ${{ runner.temp }}/process-policy-capsules HACK_E2E_CLI_BIN: ${{ github.workspace }}/dist/hack HACK_E2E_SHARD: ${{ matrix.shard }}/4 - # TEMPORARY X01 measurement; reverted before the final commit. - HACK_X01_TRACE_DIR: ${{ runner.temp }}/x01-trace run: | - mkdir -p "$HACK_X01_TRACE_DIR" HACK_E2E_DOCKER_HOST="${DOCKER_HOST:-$(docker context inspect --format '{{.Endpoints.docker.Host}}')}" HACK_E2E_NETWORK_SOURCE_REVISION="$(git rev-parse HEAD)" HACK_E2E_NETWORK_CLI_SHA256="$(sha256sum dist/hack | cut -d ' ' -f1)" @@ -328,22 +319,6 @@ jobs: HACK_E2E_COMPOSE_PLUGIN_SHA256="$(sha256sum "$HACK_E2E_COMPOSE_PLUGIN_PATH" | cut -d ' ' -f1)" export HACK_E2E_DOCKER_HOST HACK_E2E_NETWORK_SOURCE_REVISION HACK_E2E_NETWORK_CLI_SHA256 HACK_E2E_NETWORK_COMPILER_SHA256 HACK_E2E_COMPOSE_PLUGIN_PATH HACK_E2E_COMPOSE_PLUGIN_SHA256 HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker --shard="$HACK_E2E_SHARD" - # TEMPORARY X01 measurement; reverted before the final commit. - - name: X01 witness cost summary - if: always() - run: | - for f in "$RUNNER_TEMP"/x01-trace/*.jsonl; do - [ -f "$f" ] || continue - bun scripts/x01-trace-summary.ts "$f" "$(basename "$f" .jsonl)" || true - done - - name: X01 witness cost trace artifact - if: always() - uses: actions/upload-artifact@v4 - with: - name: x01-trace-shard-${{ matrix.shard }} - path: ${{ runner.temp }}/x01-trace - if-no-files-found: ignore - retention-days: 7 - name: Preserve process-policy capsule and startup diagnostic # Closed fixed-field JSON only (counts, enums, booleans); the gh log viewer truncates it inline. if: failure() diff --git a/scripts/x01-trace-summary.ts b/scripts/x01-trace-summary.ts deleted file mode 100644 index 62ace753c..000000000 --- a/scripts/x01-trace-summary.ts +++ /dev/null @@ -1,285 +0,0 @@ -// TEMPORARY X01 measurement summary. Reverted before the final commit. -// Reads closed-label JSONL written by src/lib/x01-cost-probe.ts. -type Event = { - readonly p: number; - readonly v: string; - readonly k: string; - readonly s?: string; - readonly l?: string; - readonly ms?: number; - readonly t?: number; - readonly c?: number; - readonly line?: { - readonly phase?: string; - readonly boundary?: string; - readonly durationMs?: number | null; - }; -}; -const WITNESS = new Set([ - "storage-prepare", - "storage-verify", - "carrier-new", - "inspect", - "provision", - "finish", - "witness-verify", - "witness-prepare", - "witness-enroll", -]); -const isWitness = (label: string) => - WITNESS.has(label) || label.startsWith("invoke:"); -// TEMPORARY X04: direct engine-ID reads (`ident:direct`, `ident:admission`) are leaves too. -const LEAF = new Set(["probe", "run", "exec", "hash", "ident"]); -const PHASE = /^(oneoff|storage|guard|finish|finalize)\.[a-z-]+$/; - -const path = process.argv[2]; -const title = process.argv[3] ?? path; -if (!path) { - throw new Error("usage: x01-trace-summary [title]"); -} -const text = await Bun.file(path).text(); -const events: Event[] = []; -for (const line of text.split("\n")) { - if (line.trim() === "") { - continue; - } - try { - events.push(JSON.parse(line) as Event); - } catch { - // skip partial line - } -} -const byProcess = new Map(); -for (const event of events) { - const list = byProcess.get(event.p) ?? []; - list.push(event); - byProcess.set(event.p, list); -} -type Proc = { - readonly pid: number; - readonly verb: string; - readonly totalMs: number; - readonly witnessMs: number; - readonly events: Event[]; - readonly kind: string; -}; -function topWitnessMs(list: Event[]): number { - let total = 0; - for (const event of list) { - if (event.k !== "scope" || !event.l || !isWitness(event.l)) { - continue; - } - const parts = (event.s ?? "").split(">"); - parts.pop(); - if (parts.some(isWitness)) { - continue; - } - total += event.ms ?? 0; - } - return total; -} -const procs: Proc[] = []; -for (const [pid, list] of byProcess) { - const exit = list.find((event) => event.k === "exit"); - const verb = list[0]?.v ?? "other"; - const labels = new Set(list.map((event) => event.l ?? "")); - let kind = verb; - if (verb === "up" || verb === "restart" || verb === "run") { - if (labels.has("invoke:seed")) { - kind = `${verb}(cold)`; - } else if (labels.has("invoke:verify")) { - kind = `${verb}(warm)`; - } else { - kind = `${verb}(no-witness)`; - } - } else if (verb === "exec") { - kind = labels.has("invoke:verify") ? "exec(storage)" : "exec(image)"; - } - procs.push({ - pid, - verb, - totalMs: exit?.t ?? Math.max(...list.map((event) => event.t ?? 0)), - witnessMs: topWitnessMs(list), - events: list, - kind, - }); -} -const fmt = (ms: number) => (ms / 1000).toFixed(1); -const out: string[] = []; -out.push(`#### X01 trace: ${title}`); -out.push(`events=${events.length} processes=${procs.length}`); -out.push(""); -out.push( - "kind | n | avg total s | sum total s | avg witness s | sum witness s" -); -const kinds = [...new Set(procs.map((proc) => proc.kind))].sort(); -for (const kind of kinds) { - const selected = procs.filter((proc) => proc.kind === kind); - const total = selected.reduce((sum, proc) => sum + proc.totalMs, 0); - const witness = selected.reduce((sum, proc) => sum + proc.witnessMs, 0); - out.push( - `${kind} | ${selected.length} | ${fmt(total / selected.length)} | ${fmt(total)} | ${fmt(witness / selected.length)} | ${fmt(witness)}` - ); -} -const allTotal = procs.reduce((sum, proc) => sum + proc.totalMs, 0); -const allWitness = procs.reduce((sum, proc) => sum + proc.witnessMs, 0); -out.push( - `ALL | ${procs.length} | - | ${fmt(allTotal)} | - | ${fmt(allWitness)}` -); -out.push(""); -// TEMPORARY X04: engine-ID admissions by selection kind and outcome. -const admissions = new Map(); -for (const event of events) { - if (event.k === "ident-admit" && event.l) { - admissions.set(event.l, (admissions.get(event.l) ?? 0) + 1); - } -} -out.push( - `ident admissions: ${ - [...admissions.entries()] - .sort() - .map(([label, count]) => `${label}=${count}`) - .join(" ") || "none" - }` -); -out.push(""); - -function leafTable(selected: Proc[], heading: string, limit = 24) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (!(LEAF.has(event.k) && event.l)) { - continue; - } - const key = `${event.k}:${event.l}`; - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push(`${heading} (per process avg over n=${selected.length})`); - out.push("leaf | calls/proc | s/proc | ms/call"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function leafByScope(selected: Proc[], heading: string, limit = 40) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (!(LEAF.has(event.k) && event.l)) { - continue; - } - const parts = (event.s ?? "") - .split(">") - .filter( - (part) => - isWitness(part) || part.startsWith("cmd:") || PHASE.test(part) - ); - const inner = parts.slice(-2).join(">") || "(none)"; - const key = `${inner} :: ${event.k}:${event.l}`; - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push( - `${heading} leaves by scope (per process avg over n=${selected.length})` - ); - out.push("scope :: leaf | calls/proc | s/proc | ms/call"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function scopeTable(selected: Proc[], heading: string, limit = 30) { - const rows = new Map(); - for (const proc of selected) { - for (const event of proc.events) { - if (event.k !== "scope" || !event.l) { - continue; - } - const parts = (event.s ?? "").split(">"); - const key = parts - .filter( - (part) => - PHASE.test(part) || - isWitness(part) || - part.startsWith("xattr:") || - part.startsWith("cmd:") || - part.startsWith("exec-") - ) - .join(">"); - if (key === "") { - continue; - } - const row = rows.get(key) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.ms ?? 0; - rows.set(key, row); - } - } - out.push(`${heading} scopes (per process avg over n=${selected.length})`); - out.push("scope path | n/proc | s/proc | ms/each"); - const sorted = [...rows.entries()].sort((a, b) => b[1].ms - a[1].ms); - for (const [key, row] of sorted.slice(0, limit)) { - out.push( - `${key} | ${(row.n / selected.length).toFixed(1)} | ${fmt(row.ms / selected.length)} | ${(row.ms / row.n).toFixed(0)}` - ); - } - out.push(""); -} -function phaseRows(proc: Proc) { - const rows = new Map(); - for (const event of proc.events) { - if ( - event.k === "phase" && - event.line?.boundary !== "begin" && - event.line?.phase - ) { - const row = rows.get(event.line.phase) ?? { n: 0, ms: 0 }; - row.n += 1; - row.ms += event.line.durationMs ?? 0; - rows.set(event.line.phase, row); - } - } - return rows; -} -for (const kind of [ - "up(cold)", - "up(warm)", - "exec(storage)", - "restart(warm)", - "run(warm)", - "run(cold)", - "down", -]) { - const selected = procs.filter((proc) => proc.kind === kind); - if (selected.length === 0) { - continue; - } - const sample = selected[0] as Proc; - out.push( - `### ${kind}: first sample total ${fmt(sample.totalMs)} s, witness ${fmt(sample.witnessMs)} s` - ); - out.push("phase-trace phase | spans | s"); - for (const [phase, row] of phaseRows(sample)) { - out.push(`${phase} | ${row.n} | ${fmt(row.ms)}`); - } - out.push(""); - scopeTable([sample], `${kind} first sample`, 40); - leafTable([sample], `${kind} first sample leaves`); - scopeTable(selected, `${kind} all`, 30); - leafTable(selected, `${kind} all leaves`); - leafByScope(selected, `${kind} all`); -} -console.log(out.join("\n")); diff --git a/src/lib/native-compose-command-storage.ts b/src/lib/native-compose-command-storage.ts index d9b58c132..c9384e64c 100644 --- a/src/lib/native-compose-command-storage.ts +++ b/src/lib/native-compose-command-storage.ts @@ -23,7 +23,6 @@ import { } from "./native-compose-storage-witness.ts"; import { createNativeComposeDockerStorageXattrCarrier } from "./native-compose-storage-witness-docker.ts"; import type { NativeComposeStorageXattrCarrier } from "./native-compose-storage-witness-xattr-carrier.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; type Selection = { readonly name: string; readonly storage: string }; type Document = Readonly>; @@ -135,12 +134,7 @@ export type NativeComposeCommandStorageVerifyOptions = { /** Command admission owns no helper process. Each proof gets a fresh finite carrier; * only the original startup effect can publish Expected and consume cold enrollment. */ -export function prepareNativeComposeCommandStorage( - opts: Parameters[0] -): ReturnType { - return x01Scope("storage-prepare", () => x01Prepare(opts)); -} -async function x01Prepare(opts: { +export async function prepareNativeComposeCommandStorage(opts: { readonly store: NativeComposeGenerationStore; readonly mutation: NativeComposeMutation; readonly operation: "up" | "restart" | "run" | "exec"; @@ -231,11 +225,7 @@ async function x01Prepare(opts: { await engine({ expected: engineId, signal }); await assertFresh?.(); }; - const verify = ( - generation: NativeComposeGeneration, - options: NativeComposeCommandStorageVerifyOptions = {} - ) => x01Scope("storage-verify", () => x01Verify(generation, options)); - const x01Verify = async ( + const verify = async ( generation: NativeComposeGeneration, options: NativeComposeCommandStorageVerifyOptions = {} ) => { diff --git a/src/lib/native-compose-command.ts b/src/lib/native-compose-command.ts index 0aa92b81e..8c5dcaf82 100644 --- a/src/lib/native-compose-command.ts +++ b/src/lib/native-compose-command.ts @@ -138,7 +138,6 @@ import { NativeConfigCompilerError } from "./native-config-compiler.ts"; import { nativeFilePlanningRequired } from "./native-file-plan-protocol.ts"; import { parseEnvConfigSelection } from "./project.ts"; import { run } from "./shell.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; type Operation = "up" | "restart" | "down" | "ps" | "logs" | "exec" | "run"; export type NativeComposeCommandOptions = { @@ -944,9 +943,7 @@ async function savedCommand(opts: { selected, signal, }); - const assertSaved = (proof: NativeComposeCommandStorageVerifyOptions) => - x01Scope("exec-assert-saved", () => x01AssertSaved(proof)); - const x01AssertSaved = async ( + const assertSaved = async ( proof: NativeComposeCommandStorageVerifyOptions ) => { const latest = await store.loadCurrent(); @@ -965,9 +962,7 @@ async function savedCommand(opts: { signal, assertSaved, run: () => - x01Scope("exec-process", () => - runSavedProcess({ options, generation, document, base, signal }) - ), + runSavedProcess({ options, generation, document, base, signal }), }); }); } diff --git a/src/lib/native-compose-engine-identity.ts b/src/lib/native-compose-engine-identity.ts index d19ec5ed0..d9b31f739 100644 --- a/src/lib/native-compose-engine-identity.ts +++ b/src/lib/native-compose-engine-identity.ts @@ -8,7 +8,6 @@ import { } from "./docker-context.ts"; import { isRecord } from "./guards.ts"; import { bindNativeComposeEngineConfig } from "./native-compose-engine-config.ts"; -import { x01Mark, x01Time } from "./x01-cost-probe.ts"; const ENGINE_ID = /^[A-Za-z0-9:-]{1,128}$/; const AMBIGUOUS_PATH = /[%?#\0\r\n]/; @@ -301,8 +300,6 @@ type BoundObserver = { /** Opaque digest of the exact selection, executable, config and socket binding. */ readonly key: string; readonly read: () => Promise; - /** TEMPORARY X04 measurement: closed selection kind label. */ - readonly kind: string; }; function bindObserver(opts: { readonly signal?: AbortSignal; @@ -373,15 +370,8 @@ function bindObserver(opts: { JSON.stringify([selected, binary, config, endpoint, socket, executable]) ) .digest("hex"); - const label = Array.isArray(endpoint.binding) - ? endpoint.binding[0] - : undefined; return Object.freeze({ key, - kind: - label === "host" || label === "default" || label === "context" - ? label - : "other", read: async () => { if (reading) { return refuse(); @@ -479,22 +469,18 @@ export function createNativeComposeEngineIdentityAdmissions( return await opts.cli(); } const state = admitted.get(bound.key); - const direct = bound.read; if (state === true) { - return `${JSON.stringify(await x01Time("ident", "direct", direct))}\n`; + return `${JSON.stringify(await bound.read())}\n`; } const text = await opts.cli(); if (state === undefined) { let equal = false; try { const value: unknown = JSON.parse(text); - equal = - typeof value === "string" && - value === (await x01Time("ident", "admission", direct)); + equal = typeof value === "string" && value === (await bound.read()); } catch { equal = false; } - x01Mark("ident-admit", `${bound.kind}:${equal ? "admitted" : "refused"}`); admitted.set(bound.key, equal); } return text; diff --git a/src/lib/native-compose-ownership.ts b/src/lib/native-compose-ownership.ts index d8bc9dc8a..0019b3fc3 100644 --- a/src/lib/native-compose-ownership.ts +++ b/src/lib/native-compose-ownership.ts @@ -3,7 +3,6 @@ import { isRecord } from "./guards.ts"; import { nativeComposeVolumeCreatedAt } from "./native-compose-retained-storage.ts"; import { beginNativeCpuChild } from "./native-cpu-diagnostics.ts"; import { findExecutableInPath } from "./shell.ts"; -import { x01Label, x01Time } from "./x01-cost-probe.ts"; const PROJECT = /^[a-z0-9][a-z0-9_-]*$/; const SERVICE = /^[a-z0-9][a-z0-9._-]{0,62}$/; @@ -1233,7 +1232,7 @@ export function createNativeComposeProbe( probeRefused("E_NATIVE_COMPOSE_PROBE", "operation"); } const environment = { ...process.env }; - const x01Inner = async (args: readonly string[]): Promise => { + return async (args) => { checkInterruption(opts.signal, false); const remainingTime = deadline - Date.now(); if (remainingTime <= 0) { @@ -1311,8 +1310,6 @@ export function createNativeComposeProbe( observeCpu(probeExitCode(completion[2])); } }; - return (args) => - x01Time("probe", x01Label(args, false), () => x01Inner(args)); } function probeExitCode( diff --git a/src/lib/native-compose-phase-trace.ts b/src/lib/native-compose-phase-trace.ts index 943b79320..540f833b0 100644 --- a/src/lib/native-compose-phase-trace.ts +++ b/src/lib/native-compose-phase-trace.ts @@ -1,6 +1,5 @@ import { AsyncLocalStorage } from "node:async_hooks"; import { writeSync } from "node:fs"; -import { x01Enabled, x01Phase, x01Scope } from "./x01-cost-probe.ts"; export const NATIVE_COMPOSE_PHASE_TRACE = "HACK_NATIVE_COMPOSE_PHASE_TRACE"; const PHASES = [ @@ -136,8 +135,6 @@ export function withNativeComposePhaseTrace(operation: () => Promise) { writeSync(2, line); }, }); - } else if (x01Enabled()) { - trace = createNativeComposePhaseTrace({ write: x01Phase }); } } catch { // Diagnostic setup never changes selection, results or effect authority. @@ -150,8 +147,5 @@ export function measureNativeComposePhase( phase: Phase, operation: () => Promise ): Promise { - return x01Scope( - phase, - () => context.getStore()?.measure(phase, operation) ?? operation() - ); + return context.getStore()?.measure(phase, operation) ?? operation(); } diff --git a/src/lib/native-compose-storage-command-executable.ts b/src/lib/native-compose-storage-command-executable.ts index 780dae040..a8817d685 100644 --- a/src/lib/native-compose-storage-command-executable.ts +++ b/src/lib/native-compose-storage-command-executable.ts @@ -2,7 +2,6 @@ import { constants } from "node:fs"; import { type FileHandle, lstat, open, realpath } from "node:fs/promises"; import type { NativeComposeStorageCommandExecutable } from "./native-compose-storage-command-record.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Label, x01Mark } from "./x01-cost-probe.ts"; /** Coarser than any supported filesystem timestamp granularity (1 s on HFS+/ext3). */ const RACY_MS = 2000; @@ -84,9 +83,6 @@ export function createNativeComposeStorageCommandExecutableReader( const size = Number(before.size); const started = now(); const cached = settled.get(canonical); - if (cached?.identity === identity) { - x01Mark("hash-cached", x01Label([canonical])); - } const hash = cached?.identity === identity ? cached.hash : await digest(file, size); const after = await file.stat({ bigint: true }), diff --git a/src/lib/native-compose-storage-witness-docker-inventory.ts b/src/lib/native-compose-storage-witness-docker-inventory.ts index f292021ce..b5bdb7410 100644 --- a/src/lib/native-compose-storage-witness-docker-inventory.ts +++ b/src/lib/native-compose-storage-witness-docker-inventory.ts @@ -8,7 +8,6 @@ import { type NativeComposeStorageXattrTarget, } from "./native-compose-storage-witness-xattr-carrier.ts"; import { refuseNativeComposeStorageXattr as refuse } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const NAME = /^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,254}$/; @@ -165,12 +164,7 @@ export async function assertNativeComposeStorageDockerCarrierVolume(opts: { /** Complete holder reads include foreign containers. The two inventories and selected * volume reread are observed fences; they do not claim an atomic daemon transaction. */ -export function observeNativeComposeStorageDockerTarget( - opts: Parameters[0] -): ReturnType { - return x01Scope("target", () => x01ObserveTarget(opts)); -} -async function x01ObserveTarget(opts: { +export async function observeNativeComposeStorageDockerTarget(opts: { readonly probe: Probe; readonly selection: Selection; readonly current: NativeComposeMaterialBinding; diff --git a/src/lib/native-compose-storage-witness-docker.ts b/src/lib/native-compose-storage-witness-docker.ts index d955faf06..c8e062188 100644 --- a/src/lib/native-compose-storage-witness-docker.ts +++ b/src/lib/native-compose-storage-witness-docker.ts @@ -98,7 +98,6 @@ import { refuseNativeComposeStorageXattr as refuse, } from "./native-compose-storage-witness-xattr-codec.ts"; import { findExecutableInPath, type RunExitEvent, run } from "./shell.ts"; -import { x01Label, x01Mark, x01Scope, x01Time } from "./x01-cost-probe.ts"; const ID = /^[a-f0-9]{64}$/; const PROCESS = @@ -818,10 +817,7 @@ async function groupAbsent( } /** Every identity check runs per call; only an unchanged stat identity skips the * content read for the rest of this command. */ -const commandExecutable = (path: string) => - x01Time("hash", x01Label([path]), () => - readNativeComposeStorageCommandExecutable(path) - ); +const commandExecutable = readNativeComposeStorageCommandExecutable; async function boundedHostFile(path: string): Promise { const file = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); try { @@ -907,14 +903,7 @@ async function commandBoot( } /** No callback can reject before the shared owner awaits exit. A rejected owner or * unproven group retains the journal; daemon effects are never inferred from exit. */ -export function runNativeComposeStorageDockerCommand( - supplied: Parameters[0] -): ReturnType { - return x01Scope(`cmd:${x01Label(supplied.args)}`, () => - x01RunCommandInner(supplied) - ); -} -async function x01RunCommandInner(supplied: { +export async function runNativeComposeStorageDockerCommand(supplied: { readonly context: Pick; readonly files: Pick< Files, @@ -1259,12 +1248,7 @@ function matchesScope( * image; other daemons and uncached images refuse rather than pull or emulate. * PostgreSQL data roots and crash durability remain unqualified. No module * initialization executes Docker or the helper. */ -export function createNativeComposeDockerStorageXattrCarrier( - opts: Context -): Promise { - return x01Scope("carrier-new", () => x01CreateCarrier(opts)); -} -async function x01CreateCarrier( +export async function createNativeComposeDockerStorageXattrCarrier( opts: Context ): Promise { const context: Context = Object.freeze({ ...opts }); @@ -1386,7 +1370,6 @@ async function x01CreateCarrier( const binding = await current(context, selection); if (same(binding, closed.binding)) { lifetime(context); - x01Mark("target-reused", "closing"); return closed.target; } } catch (error) { @@ -2051,21 +2034,17 @@ async function x01CreateCarrier( return captureNativeComposeStorageXattrCarrier({ artifact: selectedArtifact, ports: { - inspect: (selection, options) => - x01Scope("inspect", () => inspectPort(selection, options)), - provision: (selection) => - x01Scope("provision", () => provision(selection)), - invoke: (input) => - x01Scope(`invoke:${input.request.operation}`, () => invoke(input)), + inspect: inspectPort, + provision, + invoke, finish: context.originalCommandRecords === true - ? ({ input, completion }) => - x01Scope("finish", async () => { - closing = null; - const finish = finalizers.get(input) ?? refuse(); - finalizers.delete(input); - await finish(completion); - }) + ? async ({ input, completion }) => { + closing = null; + const finish = finalizers.get(input) ?? refuse(); + finalizers.delete(input); + await finish(completion); + } : undefined, }, signal: context.signal, diff --git a/src/lib/native-compose-storage-witness.ts b/src/lib/native-compose-storage-witness.ts index 1fc68abf7..7586eb73b 100644 --- a/src/lib/native-compose-storage-witness.ts +++ b/src/lib/native-compose-storage-witness.ts @@ -78,7 +78,6 @@ import { nativeComposeStorageXattrValueMatches, sameNativeComposeStorageXattrRoot, } from "./native-compose-storage-witness-xattr-codec.ts"; -import { x01Scope } from "./x01-cost-probe.ts"; export type { NativeComposeStorageWitnessReference } from "./native-compose-storage-witness-state.ts"; @@ -499,16 +498,7 @@ function requireXattrAdmission( refuse(); } } -function invokeXattr( - opts: Parameters[0] -): ReturnType { - const operation = - isRecord(opts.request) && typeof opts.request.operation === "string" - ? opts.request.operation - : "unknown"; - return x01Scope(`xattr:${operation}`, () => x01InvokeXattr(opts)); -} -async function x01InvokeXattr(opts: { +async function invokeXattr(opts: { readonly carrier: NativeComposeStorageXattrCarrier; readonly saved: XattrExpectation; readonly volume: NativeComposeRetainedVolume; @@ -1474,21 +1464,17 @@ export async function prepareNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - return await x01Scope("witness-prepare", () => - prepareNativeComposeStorageWitness({ - ...input, - xattrCarrier: carrier, - }) - ); + return await prepareNativeComposeStorageWitness({ + ...input, + xattrCarrier: carrier, + }); } export async function enrollNativeComposeStorageXattrWitness(opts: { readonly enrollment: NativeComposeStorageWitnessEnrollment; }): Promise { - return await x01Scope("witness-enroll", () => - enrollNativeComposeStorageWitness({ - enrollment: opts.enrollment, - }) - ); + return await enrollNativeComposeStorageWitness({ + enrollment: opts.enrollment, + }); } export async function verifyNativeComposeStorageXattrWitness( opts: Omit< @@ -1499,9 +1485,7 @@ export async function verifyNativeComposeStorageXattrWitness( } ): Promise { const { carrier, ...input } = opts; - await x01Scope("witness-verify", () => - verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }) - ); + await verifyNativeComposeStorageWitness({ ...input, xattrCarrier: carrier }); } /** Private saved observation seam. It never performs a kernel read, repairs an diff --git a/src/lib/shell.ts b/src/lib/shell.ts index ff27e6b85..ef9faf1ba 100644 --- a/src/lib/shell.ts +++ b/src/lib/shell.ts @@ -4,7 +4,6 @@ import { } from "./native-cpu-diagnostics.ts"; import { readSubprocessResourceUsage } from "./process-resource-usage.ts"; import { hasControllingTerminal } from "./tty-process-group.ts"; -import { x01Label, x01Time } from "./x01-cost-probe.ts"; export interface ExecResult { readonly exitCode: number; @@ -41,13 +40,7 @@ function buildSpawnEnv( return override ? { ...base, ...override } : base; } -export function exec( - cmd: readonly string[], - opts: ExecOptions = {} -): Promise { - return x01Time("exec", x01Label(cmd), () => x01ExecInner(cmd, opts)); -} -async function x01ExecInner( +export async function exec( cmd: readonly string[], opts: ExecOptions = {} ): Promise { @@ -130,13 +123,7 @@ export type RunExitEvent = { readonly maxRssBytes: number | null; }; -export function run( - cmd: readonly string[], - opts: RunOptions = {} -): Promise { - return x01Time("run", x01Label(cmd), () => x01RunInner(cmd, opts)); -} -async function x01RunInner( +export async function run( cmd: readonly string[], opts: RunOptions = {} ): Promise { diff --git a/src/lib/x01-cost-probe.ts b/src/lib/x01-cost-probe.ts deleted file mode 100644 index 17b2f5552..000000000 --- a/src/lib/x01-cost-probe.ts +++ /dev/null @@ -1,176 +0,0 @@ -// TEMPORARY X01 measurement probe. Reverted before the final commit. -// Writes closed labels and durations only (no ids, paths or values). -import { AsyncLocalStorage } from "node:async_hooks"; -import { appendFileSync } from "node:fs"; - -const FILE = process.env.HACK_X01_TRACE_FILE; -const VERBS = new Set([ - "up", - "down", - "exec", - "run", - "restart", - "ps", - "logs", - "status", - "init", -]); -const SUBS = new Set([ - "info", - "image", - "container", - "volume", - "network", - "compose", - "create", - "start", - "rm", - "inspect", - "ls", - "ps", - "exec", - "run", - "logs", - "version", - "events", - "stop", - "kill", - "wait", - "up", - "down", - "config", - "pull", - "build", - "restart", - "port", - "cp", - "top", -]); -const verb = (() => { - for (const arg of process.argv.slice(2)) { - if (VERBS.has(arg)) { - return arg; - } - } - return "other"; -})(); -const scopes = new AsyncLocalStorage(); -let enabled = typeof FILE === "string" && FILE.startsWith("/"); -let wroteStart = false; - -function append(record: Record): void { - appendFileSync( - FILE as string, - `${JSON.stringify({ p: process.pid, v: verb, ...record })}\n` - ); -} - -function write(record: Record): void { - if (!enabled) { - return; - } - try { - if (!wroteStart) { - wroteStart = true; - append({ k: "start", t: Math.round(performance.now()) }); - process.on("exit", (code) => { - try { - append({ k: "exit", c: code, t: Math.round(performance.now()) }); - } catch { - // measurement only - } - }); - } - append({ - s: scopes.getStore() ?? "", - t: Math.round(performance.now()), - ...record, - }); - } catch { - enabled = false; - } -} - -export function x01Enabled(): boolean { - return enabled; -} - -/** Raw line from the phase trace, re-wrapped with process identity. */ -export function x01Phase(line: string): void { - try { - write({ k: "phase", line: JSON.parse(line) as unknown }); - } catch { - enabled = false; - } -} - -export function x01Label(argv: readonly string[], binary = true): string { - const parts: string[] = []; - let rest = argv; - if (binary) { - const name = (argv[0] ?? "").split("/").at(-1) ?? ""; - parts.push( - ["docker", "ps", "sysctl", "sh", "git", "bun", "tmux"].includes(name) - ? name - : "bin" - ); - rest = argv.slice(1); - } else { - parts.push("docker"); - } - for (const token of rest) { - if (parts.length >= 3) { - break; - } - if (token.startsWith("-")) { - continue; - } - if (SUBS.has(token)) { - parts.push(token); - } else { - break; - } - } - return parts.join(" "); -} - -export async function x01Time( - kind: string, - label: string, - operation: () => Promise -): Promise { - if (!enabled) { - return await operation(); - } - const started = performance.now(); - let ok = false; - try { - const result = await operation(); - ok = true; - return result; - } finally { - write({ - k: kind, - l: label, - ms: Math.round((performance.now() - started) * 10) / 10, - ok, - }); - } -} - -/** TEMPORARY X03 measurement: a closed-label point event. */ -export function x01Mark(kind: string, label: string): void { - write({ k: kind, l: label }); -} - -export async function x01Scope( - label: string, - operation: () => Promise -): Promise { - if (!enabled) { - return await operation(); - } - const parent = scopes.getStore(); - const path = parent ? `${parent}>${label}` : label; - return await scopes.run(path, () => x01Time("scope", label, operation)); -} diff --git a/tests/e2e/harness.ts b/tests/e2e/harness.ts index e7599c528..2201457b1 100644 --- a/tests/e2e/harness.ts +++ b/tests/e2e/harness.ts @@ -43,8 +43,6 @@ const PASSTHROUGH_ENV_KEYS = [ "LC_ALL", "DOCKER_HOST", "DOCKER_CONFIG", - // TEMPORARY X01 measurement passthrough; reverted before the final commit. - "HACK_X01_TRACE_FILE", ] as const; export type CliResult = { @@ -770,13 +768,6 @@ export async function runScenarios( }; process.stdout.write(`-- ${scenario.name} (${scenario.tier}) --\n`); - // TEMPORARY X01 measurement: one trace file per scenario. - if (process.env.HACK_X01_TRACE_DIR?.startsWith("/")) { - process.env.HACK_X01_TRACE_FILE = join( - process.env.HACK_X01_TRACE_DIR, - `${scenario.name}.jsonl` - ); - } let outcome: ScenarioOutcome; let preserveFailedFixture = false; try { diff --git a/tests/e2e/native-config-networks-acceptance.ts b/tests/e2e/native-config-networks-acceptance.ts index 6c5194702..633cb0449 100644 --- a/tests/e2e/native-config-networks-acceptance.ts +++ b/tests/e2e/native-config-networks-acceptance.ts @@ -1154,10 +1154,6 @@ export const nativeConfigNetworksScenario: Scenario = { NO_COLOR: "1", HACK_RUNTIME_BACKEND: "compose", HACK_DAEMON_DISABLE_DOCKER_EVENTS: "1", - // TEMPORARY X01 measurement passthrough; reverted before the final commit. - ...(process.env.HACK_X01_TRACE_FILE - ? { HACK_X01_TRACE_FILE: process.env.HACK_X01_TRACE_FILE } - : {}), }; const engine = Bun.which("docker"); const git = Bun.which("git"); From ebe05399e018a59f394930b0dc16d2bd8099694a Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 06:14:23 -0400 Subject: [PATCH 27/32] test(agents): cover reference generation and reference-style and HTML skill links Add a temp-root regression test proving generateAgentPlugins writes every registered references/*.md byte for byte and prunes stale reference markdown while keeping non-markdown and out-of-directory files. Extend the offline skill link contract to check Markdown reference-style definitions and HTML href/src attributes, with fixtures proving links into repository docs/ or external hosts are refused and in-plugin targets pass. Co-Authored-By: Claude Opus 5.5 --- tests/agent-plugin-generation.test.ts | 66 +++++++++++++++++--- tests/helpers/skill-links.ts | 88 ++++++++++++++++++++------- tests/skill-references.test.ts | 58 ++++++++++++++++++ 3 files changed, 181 insertions(+), 31 deletions(-) diff --git a/tests/agent-plugin-generation.test.ts b/tests/agent-plugin-generation.test.ts index 891b70366..85b16ec6d 100644 --- a/tests/agent-plugin-generation.test.ts +++ b/tests/agent-plugin-generation.test.ts @@ -26,15 +26,14 @@ afterEach(async () => { } }); -test("release generation applies the requested version to every client manifest", async () => { - tempDir = await mkdtemp(join(tmpdir(), "hack-agent-plugins-")); - const pluginRoot = join(tempDir, "plugins", "hack"); - const manifestDirectories = [ - ".codex-plugin", - ".claude-plugin", - ".cursor-plugin", - ] as const; +const manifestDirectories = [ + ".codex-plugin", + ".claude-plugin", + ".cursor-plugin", +] as const; +/** Seed the client manifests the generator rewrites into a temp plugin root. */ +async function seedPluginManifests(pluginRoot: string): Promise { for (const manifestDirectory of manifestDirectories) { const directory = join(pluginRoot, manifestDirectory); await mkdir(directory, { recursive: true }); @@ -55,6 +54,12 @@ test("release generation applies the requested version to every client manifest" mcpServers: { hack: { command: "hack", args: ["mcp", "serve"] } }, }) ); +} + +test("release generation applies the requested version to every client manifest", async () => { + tempDir = await mkdtemp(join(tmpdir(), "hack-agent-plugins-")); + const pluginRoot = join(tempDir, "plugins", "hack"); + await seedPluginManifests(pluginRoot); await generateAgentPlugins({ repoRoot: tempDir, version: "9.8.7" }); @@ -69,6 +74,51 @@ test("release generation applies the requested version to every client manifest" ); }); +test("generation writes every registered reference and prunes only stale reference markdown", async () => { + tempDir = await mkdtemp(join(tmpdir(), "hack-agent-plugin-references-")); + const pluginRoot = join(tempDir, "plugins", "hack"); + await seedPluginManifests(pluginRoot); + const references = renderPluginReferences(); + expect(references.size).toBeGreaterThan(0); + const [firstReference] = references.keys(); + if (!firstReference) { + throw new Error("expected at least one registered reference"); + } + // Generated references are rewritten in place, not left with stale bytes. + await Bun.write(join(pluginRoot, firstReference), "outdated\n"); + const stale = [ + "skills/hack-cli/references/retired.md", + "skills/hack-init/references/retired.md", + ] as const; + const kept = { + "skills/hack-cli/references/notes.txt": "not a reference\n", + "skills/hack-cli/references/diagram.svg": "\n", + "skills/hack-cli/notes.md": "outside references\n", + } as const; + for (const path of stale) { + await Bun.write(join(pluginRoot, path), "# Retired\n"); + } + for (const [path, content] of Object.entries(kept)) { + await Bun.write(join(pluginRoot, path), content); + } + + await generateAgentPlugins({ repoRoot: tempDir, version: "9.8.7" }); + + for (const [path, content] of references) { + const file = Bun.file(join(pluginRoot, path)); + expect(await file.exists()).toBe(true); + expect(new Uint8Array(await file.arrayBuffer())).toEqual( + new TextEncoder().encode(content) + ); + } + for (const path of stale) { + expect(await Bun.file(join(pluginRoot, path)).exists()).toBe(false); + } + for (const [path, content] of Object.entries(kept)) { + expect(await Bun.file(join(pluginRoot, path)).text()).toBe(content); + } +}); + test("semantic-release commits all generated client manifests", async () => { const releaseConfig = await Bun.file( join(import.meta.dir, "..", ".releaserc.json") diff --git a/tests/helpers/skill-links.ts b/tests/helpers/skill-links.ts index fcd41f668..d8ec4fa1d 100644 --- a/tests/helpers/skill-links.ts +++ b/tests/helpers/skill-links.ts @@ -13,8 +13,11 @@ import { dirname, isAbsolute, relative, resolve, sep } from "node:path"; * - every `skills//references/*.md` file is linked from that skill's * SKILL.md. * - * Inline code and fenced blocks are literals (commands, schema ids), not - * links, and are ignored. Issues carry closed codes and plugin-relative paths. + * Links are inline Markdown links and images (`[x](target)`), reference-style + * definitions (`[x]: target`), HTML `href`/`src` attributes (``, + * ``), and bare URLs. Inline code and fenced blocks are literals + * (commands, schema ids), not links, and are ignored. Issues carry closed codes + * and plugin-relative paths. */ export type SkillLinkIssueCode = | "absolute-link" @@ -32,8 +35,14 @@ export type SkillLinkIssue = { const FENCED_BLOCK = /```[\s\S]*?```/g; const INLINE_CODE = /`[^`\n]*`/g; -const HEADING = /^#{1,6}\s+(.+?)\s*#*\s*$/; +const HEADING = /^#{1,6}[ \t]+(.+?)[ \t]*#*[ \t]*$/gm; const MARKDOWN_LINK = /\]\(\s*]+)>?(?:\s+"[^"]*")?\s*\)/g; +/** `[label]: target` or `[label]: `; `[^n]:` footnotes are not links. */ +const REFERENCE_DEFINITION = + /^[ \t>]*\[(?!\^)(?:[^\]\\\n]|\\.)+\]:[ \t]*\n?[ \t]*(?:<([^>\n]*)>|([^\s<]\S*))/gm; +const HTML_TAG = /<[a-z][a-z0-9-]*(?:\s[^<>]*)?>/gi; +const HTML_LINK_ATTRIBUTE = + /(?:^|\s)(?:href|src)\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+))/gi; const URL = /\b[a-z][a-z0-9+.-]*:\/\/[^\s)>\]]+/gi; const SCHEME = /^[a-z][a-z0-9+.-]*:/i; const FURTHER_READING = /^further reading$/i; @@ -143,30 +152,63 @@ function referenceLinkIssue(opts: { : "unlinked-reference"; } -/** Yield link targets and URLs with whether they sit under "Further reading". */ -function* scanLinks( - text: string -): Generator<{ readonly target: string; readonly furtherReading: boolean }> { +type ScannedLink = { + readonly target: string; + readonly furtherReading: boolean; +}; + +type Located = { readonly start: number; readonly end: number }; + +/** + * Yield link targets and URLs in document order, each with whether it sits + * under a "Further reading" heading. A bare URL inside an explicit link is not + * yielded twice. + */ +function* scanLinks(text: string): Generator { const prose = text .replace(FENCED_BLOCK, (block) => block.replace(/[^\n]/g, " ")) .replace(INLINE_CODE, (code) => " ".repeat(code.length)); - let furtherReading = false; - for (const line of prose.split("\n")) { - const heading = HEADING.exec(line); - if (heading) { - furtherReading = FURTHER_READING.test(heading[1] ?? ""); - continue; - } - const seen = new Set(); - for (const match of line.matchAll(MARKDOWN_LINK)) { - const target = match[1] ?? ""; - seen.add(target); - yield { target, furtherReading }; + const headings = [...prose.matchAll(HEADING)].map((match) => ({ + start: match.index, + furtherReading: FURTHER_READING.test(match[1] ?? ""), + })); + const furtherReadingAt = (offset: number): boolean => + headings.findLast((heading) => heading.start <= offset)?.furtherReading ?? + false; + + const explicit: (Located & { readonly target: string })[] = []; + const add = (match: RegExpMatchArray, target: string | undefined): void => { + const start = match.index ?? 0; + explicit.push({ + start, + end: start + match[0].length, + target: target ?? "", + }); + }; + for (const match of prose.matchAll(MARKDOWN_LINK)) { + add(match, match[1]); + } + for (const match of prose.matchAll(REFERENCE_DEFINITION)) { + add(match, match[1] ?? match[2]); + } + for (const tag of prose.matchAll(HTML_TAG)) { + for (const attribute of tag[0].matchAll(HTML_LINK_ATTRIBUTE)) { + add(tag, (attribute[1] ?? attribute[2] ?? attribute[3] ?? "").trim()); } - for (const match of line.matchAll(URL)) { - if (!seen.has(match[0])) { - yield { target: match[0], furtherReading }; - } + } + + const links: (Located & { readonly target: string })[] = [...explicit]; + for (const match of prose.matchAll(URL)) { + const start = match.index; + if (!explicit.some((link) => link.start <= start && start < link.end)) { + links.push({ start, end: start + match[0].length, target: match[0] }); } } + links.sort((a, b) => a.start - b.start); + for (const link of links) { + yield { + target: link.target, + furtherReading: furtherReadingAt(link.start), + }; + } } diff --git a/tests/skill-references.test.ts b/tests/skill-references.test.ts index c650199e6..35622f3f7 100644 --- a/tests/skill-references.test.ts +++ b/tests/skill-references.test.ts @@ -115,6 +115,64 @@ test("negative control: absolute, missing, external and unlinked references fail ]); }); +test("negative control: reference-style and HTML links into repository docs or external hosts fail", async () => { + const root = await fixturePlugin({ + "skills/x/SKILL.md": [ + "# X", + "", + "See [env][env-doc], [guide][guide] and [up][up].", + "", + "[env-doc]: ../../docs/env.md", + "[guide]: https://example.com/guide", + '[up]: <../../../../docs/env.md> "Repository env"', + "", + 'env', + "a", + 'logo', + "', + "", + "[^1]: A footnote definition is prose, not a link target.", + "", + ].join("\n"), + }); + expect(await findSkillLinkIssues({ pluginRoot: root })).toEqual([ + { file: "skills/x/SKILL.md", code: "repo-docs-link" }, + { file: "skills/x/SKILL.md", code: "external-outside-further-reading" }, + { file: "skills/x/SKILL.md", code: "escapes-plugin" }, + { file: "skills/x/SKILL.md", code: "repo-docs-link" }, + { file: "skills/x/SKILL.md", code: "external-outside-further-reading" }, + { file: "skills/x/SKILL.md", code: "escapes-plugin" }, + { file: "skills/x/SKILL.md", code: "external-outside-further-reading" }, + ]); +}); + +test("reference-style and HTML links to in-plugin targets pass and count as reference links", async () => { + const root = await fixturePlugin({ + "skills/y/SKILL.md": [ + "# Y", + "", + 'Read the [patterns][patterns] and the lifecycle notes.', + "", + "[patterns]: references/patterns.md", + "", + 'diagram', + "", + "## Further reading", + "", + "- [Site][site] or the site.", + "", + "[site]: ", + "", + ].join("\n"), + "skills/y/references/patterns.md": "# Patterns\n", + "skills/y/references/lifecycle.md": "# Lifecycle\n", + "skills/y/assets/diagram.svg": "\n", + }); + expect(await findSkillLinkIssues({ pluginRoot: root })).toEqual([]); +}); + test("reference file names are constrained to plain markdown names", () => { for (const file of ["../escape.md", "nested/x.md", "x.txt", "X.md"]) { expect(() => From 6b5498034642663a0eb89f2f03e8f063e9433c64 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 06:15:56 -0400 Subject: [PATCH 28/32] test: keep the Bun crash loop tests free of real crash reports The executeWithLog test made a real Bun child kill itself with SIGSEGV, so every run wrote a bun-*.ips into the user's DiagnosticReports. In labeled mode the experiment's own full-suite iterations would also have counted those self-induced reports. - The real child now ends itself with SIGTERM, a non-crash signal that still exercises the "signal the harness did not send" path. - A pure classifier test feeds the exact Bun.spawn signal shapes (SIGSEGV, SIGABRT, SIGBUS, SIGILL, SIGTRAP, SIGTERM, SIGKILL). - A fake-executor loop test checks that signal deaths are recorded as crash/signal with a closed signal name. - The real-child test records the bun*.ips listing before and after and asserts it is unchanged. The experiment workflow's concurrency group now contends only for experiment events: a dispatch or a bun-crash-experiment label shares a group, and any other label gets a group keyed by its run id, so it can no longer replace or cancel a pending experiment run. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/bun-crash-experiment.yml | 5 +- tests/bun-crash-loop.test.ts | 113 ++++++++++++++++++++- 2 files changed, 112 insertions(+), 6 deletions(-) diff --git a/.github/workflows/bun-crash-experiment.yml b/.github/workflows/bun-crash-experiment.yml index d7c60965c..170bd2e11 100644 --- a/.github/workflows/bun-crash-experiment.yml +++ b/.github/workflows/bun-crash-experiment.yml @@ -31,8 +31,11 @@ on: permissions: contents: read +# Only experiment events contend. A label other than `bun-crash-experiment` gets a +# group of its own (keyed by run id), so its no-op run can never queue behind, +# replace or cancel a pending experiment run on the same pull request. concurrency: - group: bun-crash-experiment-${{ github.event.pull_request.number || github.ref }} + group: bun-crash-experiment-${{ github.event_name == 'workflow_dispatch' && format('dispatch-{0}', github.ref) || github.event.label.name == 'bun-crash-experiment' && format('pr-{0}', github.event.pull_request.number) || format('ignored-label-{0}', github.run_id) }} cancel-in-progress: false jobs: diff --git a/tests/bun-crash-loop.test.ts b/tests/bun-crash-loop.test.ts index 2f575c7d6..e7a3ff683 100644 --- a/tests/bun-crash-loop.test.ts +++ b/tests/bun-crash-loop.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, test } from "bun:test"; import { mkdtemp, readdir, rm, utimes, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; +import { homedir, tmpdir } from "node:os"; import { join } from "node:path"; import { collectDiagnosticReports, @@ -66,6 +66,29 @@ describe("classifyBunRun", () => { ).toEqual({ outcome: "crash", reason: "signal" }); }); + test("every signal shape Bun.spawn reports is a crash, whatever the signal", () => { + // Bun.spawn reports a signal death as exitCode null plus the signal name. + // These shapes are fed directly, so no test process ever receives a crash + // signal (which would make macOS write a real crash report). + for (const signal of [ + "SIGSEGV", + "SIGABRT", + "SIGBUS", + "SIGILL", + "SIGTRAP", + "SIGTERM", + "SIGKILL", + ]) { + expect(classifyBunRun({ exitCode: null, signal, output: "" })).toEqual({ + outcome: "crash", + reason: "signal", + }); + expect( + classifyBunRun({ exitCode: null, signal, output: " 3 pass\n 0 fail\n" }) + ).toEqual({ outcome: "crash", reason: "signal" }); + } + }); + test("(fail) with exit 1 is a non-crash test failure", () => { expect( classifyBunRun({ @@ -257,6 +280,58 @@ describe("runBunCrashLoop", () => { }); }); + test("a signal death from the executor is recorded as a crash with its closed signal name", async () => { + let clock = 0; + const runs: ExecutedRun[] = [ + { + exitCode: null, + signal: "SIGSEGV", + output: "", + stoppedAtDeadline: false, + }, + { + exitCode: null, + signal: "SIGABRT", + output: "", + stoppedAtDeadline: false, + }, + { + exitCode: null, + signal: "not a signal name", + output: "", + stoppedAtDeadline: false, + }, + ]; + const result = await runBunCrashLoop({ + budgetMs: 300_000, + commands: [commands[0] as LoopCommand], + now: () => clock, + execute: async () => { + clock += 100_000; + return ( + runs.shift() ?? { + exitCode: 0, + signal: null, + output: "", + stoppedAtDeadline: false, + } + ); + }, + }); + expect( + result.iterations.map((record) => [ + record.outcome, + record.reason, + record.exitCode, + record.signal, + ]) + ).toEqual([ + ["crash", "signal", null, "SIGSEGV"], + ["crash", "signal", null, "SIGABRT"], + ["crash", "signal", null, null], + ]); + }); + test("the summary is closed and value-free", async () => { let clock = 0; const result = await runBunCrashLoop({ @@ -287,8 +362,26 @@ describe("runBunCrashLoop", () => { }); }); +const BUN_CRASH_REPORT = /^bun.*\.ips$/i; + +/** The `bun*.ips` names in the user's crash-report directory (read-only; empty when absent). */ +async function bunCrashReportNames(): Promise { + try { + const names = await readdir( + join(homedir(), "Library", "Logs", "DiagnosticReports") + ); + return names.filter((name) => BUN_CRASH_REPORT.test(name)).sort(); + } catch { + return []; + } +} + describe("executeWithLog", () => { test("keeps each run's log, reports signals and stops a run at the deadline", async () => { + // Real children here must never die from a crash signal (SIGSEGV, SIGABRT, + // SIGBUS, SIGILL, SIGTRAP): macOS ReportCrash would write a bun-*.ips into + // the user's real DiagnosticReports, whatever HOME is set to. + const reportsBefore = await bunCrashReportNames(); const root = await tempRoot(); const execute = executeWithLog({ cwd: root, logDir: root }); const far = Date.now() + 60_000; @@ -303,16 +396,25 @@ describe("executeWithLog", () => { expect(pass).toMatchObject({ exitCode: 0, stoppedAtDeadline: false }); expect(pass.output).toContain("ok"); - const crashed = await execute( + // SIGTERM is a non-crash signal, so no crash report is written; it still + // exercises the path where a run dies from a signal the harness did not send. + const signaled = await execute( { id: "focused", - argv: [process.execPath, "-e", "process.kill(process.pid, 'SIGSEGV')"], + argv: [process.execPath, "-e", "process.kill(process.pid, 'SIGTERM')"], initialEstimateMs: 1000, }, { index: 2, deadlineMs: far } ); - expect(crashed.signal).toBe("SIGSEGV"); - expect(classifyBunRun(crashed).outcome).toBe("crash"); + expect(signaled).toMatchObject({ + exitCode: null, + signal: "SIGTERM", + stoppedAtDeadline: false, + }); + expect(classifyBunRun(signaled)).toEqual({ + outcome: "crash", + reason: "signal", + }); const stopped = await execute( { @@ -328,6 +430,7 @@ describe("executeWithLog", () => { "iteration-002-focused.log", "iteration-003-full.log", ]); + expect(await bunCrashReportNames()).toEqual(reportsBefore); }); }); From 9f203664dfbaf4aa3e5fd486c6416a7079b92b3c Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 06:41:12 -0400 Subject: [PATCH 29/32] fix(env): write .hack/.env and the encrypted store atomically with 0600 .hack/.env and .hack/.env. (materialize, legacy env set/unset and the daemon env route) are now read without following a symlink and replaced atomically: a 0600 O_EXCL|O_NOFOLLOW sibling in .hack/ is written, fsynced and renamed over the target, then the directory is synced. A symlinked (including dangling) or non-regular env file, or a symlinked .hack directory, is refused with a fixed message that names no file, overlay or value. An unchanged file with group/other bits is tightened. The encrypted_file store (and the cloud shim that shares it) is replaced the same way with mode 0600; a symlinked store path is refused for writes. set/delete now serialize read-modify-write with a per-path in-process queue plus the cooperative hard-link owner lock used by the projects registry at .lock, so concurrent writers, including separate processes, no longer lose updates. Reads stay lock-free. Adds v1 AES-GCM context-binding characterization tests for the store envelope and legacy v1: values: SHA-256 key derivation, fixed field set, no authenticated associated data, no path/scope/variable binding beyond the scoped plaintext key. No envelope format change. Refs HACK-1127 Co-Authored-By: Claude Opus 5.5 --- docs/env.md | 18 +- src/lib/hack-env.ts | 24 +- src/lib/private-env-file.ts | 74 ++++ src/lib/private-file.ts | 166 +++++++- src/lib/project-env-config.ts | 6 +- src/lib/projects-registry-lock.ts | 5 +- src/lib/secret-store.ts | 133 +++++-- tests/aes-gcm-v1-context-binding.test.ts | 482 +++++++++++++++++++++++ tests/private-env-file.test.ts | 358 +++++++++++++++++ tests/secret-store-file-hygiene.test.ts | 248 ++++++++++++ 10 files changed, 1467 insertions(+), 47 deletions(-) create mode 100644 src/lib/private-env-file.ts create mode 100644 tests/aes-gcm-v1-context-binding.test.ts create mode 100644 tests/private-env-file.test.ts create mode 100644 tests/secret-store-file-hygiene.test.ts diff --git a/docs/env.md b/docs/env.md index a9e1eed56..ccd5e690c 100644 --- a/docs/env.md +++ b/docs/env.md @@ -195,6 +195,15 @@ That means `.hack/.env` is no longer the primary runtime source of truth. `hack env materialize` is manual by design. Use it only when you need a compatibility file for an external tool that expects `.env` on disk. +Hack writes `.hack/.env` and `.hack/.env.` (from `materialize`, legacy +`hack env set`, and the daemon env route) with mode `0600` by atomic replace: a +private temporary file in `.hack/` is written, fsynced and renamed over the target, +so readers never see a partial file. An existing file with group or other access is +tightened to `0600` even when its contents are unchanged. Hack refuses, with a fixed +message that names no file or value, to read or write an env file that is a symlink +(including a dangling one) or not a regular file, or one whose `.hack` directory is a +symlink. Replace the link with a regular file or directory to continue. + For host execution, `hack host exec`, `hack host shell`, and lifecycle hooks/processes use a host-local view. That means Hack prefers host-usable values when it can: @@ -529,7 +538,14 @@ The modern YAML env model is the canonical repo format today. Some older surfaces are still compatibility-oriented: -- `hack env backend ...` remains relevant for legacy repos and older secret-store configurations +- `hack env backend ...` remains relevant for legacy repos and older secret-store configurations. + The `encrypted_file` store (also used by the `cloud` shim) is replaced atomically + with mode `0600`, and set/delete hold an in-process queue plus a cooperative + `.lock` so concurrent writers, including separate processes, do not lose + updates. A store path that is a symlink is refused for writes. The v1 store + envelope and legacy `v1:` values authenticate no associated data: they are not + bound to a path, project or variable name beyond the scoped key inside the + encrypted store plaintext - the daemon's env HTTP route (used by the macOS app) reads and writes the legacy dotenv-shaped contract (`source: 'plain_env'|'keychain'`, `resolvedFrom: 'dotenv'/'process'/'keychain'/'portable_backend'`), not the modern YAML `hack env list --json` shape; there is no web dashboard surface anymore diff --git a/src/lib/hack-env.ts b/src/lib/hack-env.ts index 6a0277aeb..77ba543cf 100644 --- a/src/lib/hack-env.ts +++ b/src/lib/hack-env.ts @@ -6,8 +6,12 @@ import { } from "../constants.ts"; import { readControlPlaneConfig } from "../control-plane/sdk/config.ts"; import { parseDotEnv, serializeDotEnv } from "./env.ts"; -import { readTextFile, writeTextFileIfChanged } from "./fs.ts"; +import { readTextFile } from "./fs.ts"; import { getString, isRecord } from "./guards.ts"; +import { + readPrivateEnvFile, + writePrivateEnvFileIfChanged, +} from "./private-env-file.ts"; import { readProjectDefaultEnvConfig } from "./project.ts"; import type { SecretStoreDescriptor } from "./secret-store.ts"; import { @@ -502,19 +506,23 @@ export async function upsertDotEnvValue(opts: { readonly key: string; readonly value: string; }): Promise<{ readonly changed: boolean }> { - const existingText = (await readTextFile(opts.envFile)) ?? ""; + const existingText = + (await readPrivateEnvFile({ envFile: opts.envFile })) ?? ""; const env = parseDotEnv(existingText); const nextEnv: Record = { ...env, [opts.key]: opts.value }; const nextText = serializeDotEnvStable(nextEnv); - const result = await writeTextFileIfChanged(opts.envFile, nextText); - return { changed: result.changed }; + return await writePrivateEnvFileIfChanged({ + envFile: opts.envFile, + text: nextText, + }); } export async function removeDotEnvKey(opts: { readonly envFile: string; readonly key: string; }): Promise<{ readonly changed: boolean }> { - const existingText = (await readTextFile(opts.envFile)) ?? ""; + const existingText = + (await readPrivateEnvFile({ envFile: opts.envFile })) ?? ""; const env = parseDotEnv(existingText); if (!(opts.key in env)) { return { changed: false }; @@ -527,8 +535,10 @@ export async function removeDotEnvKey(opts: { nextEnv[key] = value; } const nextText = serializeDotEnvStable(nextEnv); - const result = await writeTextFileIfChanged(opts.envFile, nextText); - return { changed: result.changed }; + return await writePrivateEnvFileIfChanged({ + envFile: opts.envFile, + text: nextText, + }); } export function resolveKeychainServiceName(opts: { diff --git a/src/lib/private-env-file.ts b/src/lib/private-env-file.ts new file mode 100644 index 000000000..48e58a46c --- /dev/null +++ b/src/lib/private-env-file.ts @@ -0,0 +1,74 @@ +import { lstat, mkdir } from "node:fs/promises"; +import { dirname } from "node:path"; +import { isRecord } from "./guards.ts"; +import { + type ReplaceablePrivateFileOptions, + readReplaceablePrivateFile, + replacePrivateFileAtomic, +} from "./private-file.ts"; + +/** + * Plaintext env files under `.hack/` (`.hack/.env` and `.hack/.env.`). + * + * Ownership: Hack owns these files. They are read without following a symlink + * and replaced atomically with mode 0600. A symlink or non-regular entry at the + * file, or a symlinked `.hack` directory, is refused with + * {@link PRIVATE_ENV_FILE_REFUSAL}; the refusal never names the file, overlay + * or any value. Ancestors above `.hack` are not inspected. + */ +export const PRIVATE_ENV_FILE_REFUSAL = + "Refused .hack env file access: the env file or its .hack directory is a symlink or not a regular file. Replace it with a regular file or directory and retry; values omitted."; + +function privateEnvFileRefusal(): Error { + return new Error(PRIVATE_ENV_FILE_REFUSAL); +} + +function envFileOptions(envFile: string): ReplaceablePrivateFileOptions { + return { + path: envFile, + directories: [dirname(envFile)], + refusal: privateEnvFileRefusal, + }; +} + +/** Current text, or null when the env file is absent. */ +export async function readPrivateEnvFile(opts: { + readonly envFile: string; +}): Promise { + const current = await readReplaceablePrivateFile( + envFileOptions(opts.envFile) + ); + return current?.text ?? null; +} + +/** + * Writes `text` when it differs from the current contents, or when the existing + * file grants any group/other permission. `changed` reports a content change. + */ +export async function writePrivateEnvFileIfChanged(opts: { + readonly envFile: string; + readonly text: string; +}): Promise<{ readonly changed: boolean }> { + await ensureEnvFileDirectory({ directory: dirname(opts.envFile) }); + const options = envFileOptions(opts.envFile); + const existing = await readReplaceablePrivateFile(options); + const changed = existing?.text !== opts.text; + if (changed || ((existing?.mode ?? 0) & 0o077) !== 0) { + await replacePrivateFileAtomic({ ...options, text: opts.text }); + } + return { changed }; +} + +/** Creates a missing `.hack` directory; an existing entry is left to the refusal checks. */ +async function ensureEnvFileDirectory(opts: { + readonly directory: string; +}): Promise { + try { + await lstat(opts.directory); + } catch (error: unknown) { + if (!(isRecord(error) && error.code === "ENOENT")) { + throw error; + } + await mkdir(opts.directory, { recursive: true }); + } +} diff --git a/src/lib/private-file.ts b/src/lib/private-file.ts index fd7f02d6e..3776ef235 100644 --- a/src/lib/private-file.ts +++ b/src/lib/private-file.ts @@ -1,12 +1,14 @@ +import { randomUUID } from "node:crypto"; import { constants, type Stats } from "node:fs"; import { type FileHandle, lstat, open, realpath, + rename, unlink, } from "node:fs/promises"; -import { dirname } from "node:path"; +import { basename, dirname, join } from "node:path"; import { isRecord } from "./guards.ts"; /** @@ -199,6 +201,168 @@ export async function createPrivateFileExclusive(opts: { await syncDirectory({ path: dirname(opts.path) }); } +/** Options shared by the no-follow read and atomic replace of a private file. */ +export type ReplaceablePrivateFileOptions = { + readonly path: string; + /** + * Directories (normally the file's parent inside the project) that must be + * real directories rather than symlinks, with a stable identity across the + * operation. Ancestors outside this list are not inspected. + */ + readonly directories: readonly string[]; + /** Builds the caller's fixed, value-free refusal. */ + readonly refusal: () => Error; +}; + +/** Current contents and permission bits of a replaceable private file. */ +export type ReplaceablePrivateFile = { + readonly text: string; + readonly mode: number; +}; + +type DirectoryIdentity = { readonly path: string; readonly stats: Stats }; + +async function captureRealDirectories( + opts: ReplaceablePrivateFileOptions +): Promise { + return await Promise.all( + opts.directories.map(async (path) => { + const stats = await lstat(path).catch(() => null); + if (!stats?.isDirectory()) { + throw opts.refusal(); + } + return { path, stats }; + }) + ); +} + +async function recheckRealDirectories(opts: { + readonly directories: readonly DirectoryIdentity[]; + readonly refusal: () => Error; +}): Promise { + for (const { path, stats } of opts.directories) { + const current = await lstat(path).catch(() => null); + if ( + !current?.isDirectory() || + current.dev !== stats.dev || + current.ino !== stats.ino + ) { + throw opts.refusal(); + } + } +} + +/** Null when absent; refuses a symlink (dangling or not) or non-regular entry. */ +async function observeReplaceableTarget(opts: { + readonly path: string; + readonly refusal: () => Error; +}): Promise { + const observed = await lstat(opts.path).catch((error: unknown) => { + if (isMissingEntryError(error)) { + return null; + } + throw error; + }); + if (observed && !observed.isFile()) { + throw opts.refusal(); + } + return observed; +} + +/** + * Reads a private file that is later replaced with + * {@link replacePrivateFileAtomic}. Returns null when the entry is absent. A + * symlink at the path, a non-regular entry, a symlinked or replaced directory + * from `directories`, or a file that changes identity during the read throws + * `refusal()`; other filesystem errors (for example EACCES) propagate, so an + * unreadable file is never mistaken for an empty one and overwritten. + */ +export async function readReplaceablePrivateFile( + opts: ReplaceablePrivateFileOptions +): Promise { + const directories = await captureRealDirectories(opts); + const observed = await observeReplaceableTarget(opts); + if (!observed) { + await recheckRealDirectories({ directories, refusal: opts.refusal }); + return null; + } + let file: FileHandle; + try { + file = await open( + opts.path, + constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK + ); + } catch (error: unknown) { + if (isRecord(error) && error.code === "ELOOP") { + throw opts.refusal(); + } + throw error; + } + try { + const opened = await file.stat(); + if ( + !opened.isFile() || + opened.dev !== observed.dev || + opened.ino !== observed.ino + ) { + throw opts.refusal(); + } + const text = await file.readFile({ encoding: "utf8" }); + await recheckRealDirectories({ directories, refusal: opts.refusal }); + return { text, mode: opened.mode & 0o7777 }; + } finally { + await file.close(); + } +} + +/** + * Replaces `path` with `text` atomically and durably, leaving mode 0600. + * + * A uniquely named sibling is created with O_CREAT|O_EXCL|O_NOFOLLOW and mode + * 0600, fchmod 0600, written and fsynced, then renamed over `path`; readers see + * either the previous or the complete new contents, never a partial file. A + * symlink (dangling or not) or non-regular entry at `path`, or a symlinked or + * replaced directory from `directories`, throws `refusal()` before any write + * reaches the target. rename(2) never follows a final-component symlink, so + * even a link planted after the check is replaced rather than written through. + * The staging file is removed on every failure path. + */ +export async function replacePrivateFileAtomic( + opts: ReplaceablePrivateFileOptions & { readonly text: string } +): Promise { + const directories = await captureRealDirectories(opts); + await observeReplaceableTarget(opts); + const staging = join( + dirname(opts.path), + `.${basename(opts.path)}.${randomUUID()}.tmp` + ); + const file = await open( + staging, + constants.O_CREAT | + constants.O_EXCL | + constants.O_NOFOLLOW | + constants.O_WRONLY, + 0o600 + ); + let committed = false; + try { + // The umask can only remove bits; restore owner read/write explicitly. + await file.chmod(0o600); + await file.writeFile(opts.text, "utf8"); + await file.sync(); + await recheckRealDirectories({ directories, refusal: opts.refusal }); + await observeReplaceableTarget(opts); + await rename(staging, opts.path); + committed = true; + } finally { + await file.close(); + if (!committed) { + await unlink(staging).catch(() => undefined); + } + } + await syncDirectory({ path: dirname(opts.path) }); +} + /** Best effort: persist the directory entry where the platform allows it. */ async function syncDirectory(opts: { readonly path: string }): Promise { let directory: FileHandle | null = null; diff --git a/src/lib/project-env-config.ts b/src/lib/project-env-config.ts index 1d756962a..b4d0166e1 100644 --- a/src/lib/project-env-config.ts +++ b/src/lib/project-env-config.ts @@ -54,6 +54,7 @@ import { acquireNativeManagedEnvFile, assertNativeProjectInputRoot, } from "./native-project-inputs.ts"; +import { writePrivateEnvFileIfChanged } from "./private-env-file.ts"; import { createPrivateFileExclusive, decodeCanonicalBase64, @@ -2398,7 +2399,10 @@ export async function materializeProjectEnv(opts: { const envPath = resolve(opts.projectDir, PROJECT_ENV_FILENAME); const text = serializeDotEnv(selectedEnv); await assertLegacyProjectDirectory({ projectDir: opts.projectDir }); - const changed = (await writeTextFileIfChanged(envPath, text)).changed; + // Mode 0600, atomic replace; a symlinked .env or .hack is refused. + const changed = ( + await writePrivateEnvFileIfChanged({ envFile: envPath, text }) + ).changed; await ensureDir( dirname(resolveProjectEnvStatePath({ projectDir: opts.projectDir })) ); diff --git a/src/lib/projects-registry-lock.ts b/src/lib/projects-registry-lock.ts index 84f03bb4c..4448879fe 100644 --- a/src/lib/projects-registry-lock.ts +++ b/src/lib/projects-registry-lock.ts @@ -39,6 +39,8 @@ export async function withProjectsRegistryLock(opts: { readonly waitForLock?: boolean; readonly signal?: AbortSignal; readonly timeoutMs?: number; + /** Fixed resource name for the timeout message; defaults to the registry. */ + readonly label?: string; }): Promise { opts.signal?.throwIfAborted(); if (opts.waitForLock === false) { @@ -102,6 +104,7 @@ async function acquire(opts: { readonly waitForLock?: boolean; readonly signal?: AbortSignal; readonly timeoutMs?: number; + readonly label?: string; }): Promise { const started = performance.now(); const timeout = opts.timeoutMs ?? WAIT_MS; @@ -121,7 +124,7 @@ async function acquire(opts: { ? "; recovery guard is busy (interrupted recovery requires offline inspection)" : ""; throw new Error( - `Timed out waiting for projects registry lock${recovery}` + `Timed out waiting for ${opts.label ?? "projects registry"} lock${recovery}` ); } // Jitter prevents a burst of CLI clients from retrying in lockstep. diff --git a/src/lib/secret-store.ts b/src/lib/secret-store.ts index 847492244..5d56cdbef 100644 --- a/src/lib/secret-store.ts +++ b/src/lib/secret-store.ts @@ -15,7 +15,11 @@ import type { import { readControlPlaneConfig } from "../control-plane/sdk/config.ts"; import { resolveGlobalHackDir } from "./config-paths.ts"; import { ensureDir, readTextFile } from "./fs.ts"; -import { decodeCanonicalBase64 } from "./private-file.ts"; +import { + decodeCanonicalBase64, + replacePrivateFileAtomic, +} from "./private-file.ts"; +import { withProjectsRegistryLock } from "./projects-registry-lock.ts"; const ENCRYPTED_FILE_KEY_ENV = "HACK_SECRETS_FILE_KEY"; const DISABLE_ENCRYPTED_FILE_KEYCHAIN_FALLBACK_ENV = @@ -31,7 +35,11 @@ const ENCRYPTED_IV_BYTES = 12; const ENCRYPTED_TAG_BYTES = 16; /** Decoded ciphertext bound for the whole encrypted store. */ const ENCRYPTED_CIPHERTEXT_LIMIT = 16 * 1024 * 1024; +const ENCRYPTED_STORE_WRITE_REFUSAL = + "Refused encrypted secret store write: the store path is a symlink or not a regular file. Replace it with a regular file and retry; values omitted."; let cachedEncryptedFileKey: string | null = null; +/** Per-path tail of the in-process mutation queue; see withEncryptedStoreMutation. */ +const encryptedStoreMutationTails = new Map>(); let encryptedFileKeyFailureCooldownUntilMs = 0; type EncryptedStorePayload = { @@ -388,45 +396,91 @@ function createEncryptedScopedStore(input: { }); return entries[scopedKey] ?? null; }, - set: async ({ key, value }) => { - const entries = await readEncryptedEntries({ - filePath: input.filePath, - encryptionKey: input.encryptionKey, - }); - const scopedKey = buildScopedSecretKey({ - scope: input.scope, - key, - }); - entries[scopedKey] = value; - await writeEncryptedEntries({ - filePath: input.filePath, - encryptionKey: input.encryptionKey, - entries, - }); - }, - delete: async ({ key }) => { - const entries = await readEncryptedEntries({ + set: async ({ key, value }) => + await withEncryptedStoreMutation({ filePath: input.filePath, - encryptionKey: input.encryptionKey, - }); - const scopedKey = buildScopedSecretKey({ - scope: input.scope, - key, - }); - if (!(scopedKey in entries)) { - return false; - } - delete entries[scopedKey]; - await writeEncryptedEntries({ + run: async () => { + const entries = await readEncryptedEntries({ + filePath: input.filePath, + encryptionKey: input.encryptionKey, + }); + const scopedKey = buildScopedSecretKey({ + scope: input.scope, + key, + }); + entries[scopedKey] = value; + await writeEncryptedEntries({ + filePath: input.filePath, + encryptionKey: input.encryptionKey, + entries, + }); + }, + }), + delete: async ({ key }) => + await withEncryptedStoreMutation({ filePath: input.filePath, - encryptionKey: input.encryptionKey, - entries, - }); - return true; - }, + run: async () => { + const entries = await readEncryptedEntries({ + filePath: input.filePath, + encryptionKey: input.encryptionKey, + }); + const scopedKey = buildScopedSecretKey({ + scope: input.scope, + key, + }); + if (!(scopedKey in entries)) { + return false; + } + delete entries[scopedKey]; + await writeEncryptedEntries({ + filePath: input.filePath, + encryptionKey: input.encryptionKey, + entries, + }); + return true; + }, + }), }; } +/** + * Serializes read-modify-write of one encrypted store file. + * + * Writers in this process queue per configured path, so they never poll each + * other; across processes the cooperative `.lock` (the same exclusive + * hard-link owner protocol as the projects registry, with dead-owner recovery) + * admits one writer at a time. Every set/delete re-reads the store inside the + * lock, so a concurrent writer's update is never lost. Reads take no lock: + * writes replace the file atomically, so a reader sees a whole envelope. + */ +async function withEncryptedStoreMutation(input: { + readonly filePath: string; + readonly run: () => Promise; +}): Promise { + const previous = + encryptedStoreMutationTails.get(input.filePath) ?? Promise.resolve(); + let release = (): void => undefined; + const turn = new Promise((resolveTurn) => { + release = resolveTurn; + }); + const tail = previous.then(() => turn); + encryptedStoreMutationTails.set(input.filePath, tail); + await previous; + try { + await ensureDir(dirname(input.filePath)); + return await withProjectsRegistryLock({ + lockPath: `${input.filePath}.lock`, + label: "encrypted secret store", + run: input.run, + }); + } finally { + release(); + if (encryptedStoreMutationTails.get(input.filePath) === tail) { + encryptedStoreMutationTails.delete(input.filePath); + } + } +} + function buildScopedSecretKey(input: { readonly scope: string; readonly key: string; @@ -751,7 +805,14 @@ async function writeEncryptedEntries(input: { encryptionKey: input.encryptionKey, }); await ensureDir(dirname(input.filePath)); - await Bun.write(input.filePath, `${JSON.stringify(envelope, null, 2)}\n`); + // Atomic same-directory replace with mode 0600. The configured store + // location's ancestors are the user's choice and are not inspected. + await replacePrivateFileAtomic({ + path: input.filePath, + directories: [], + text: `${JSON.stringify(envelope, null, 2)}\n`, + refusal: () => new Error(ENCRYPTED_STORE_WRITE_REFUSAL), + }); } /** diff --git a/tests/aes-gcm-v1-context-binding.test.ts b/tests/aes-gcm-v1-context-binding.test.ts new file mode 100644 index 000000000..3f29993bb --- /dev/null +++ b/tests/aes-gcm-v1-context-binding.test.ts @@ -0,0 +1,482 @@ +/** + * Characterization of v1 AES-GCM context binding (HACK-1127). + * + * These tests pin existing behavior; they do not change or endorse it. Both v1 + * formats (the encrypted_file store envelope and the legacy `v1:` secure env + * value) derive the AES-256 key as SHA-256(key text), use a 12-byte IV and a + * 16-byte tag, and authenticate NO associated data. Consequently a v1 envelope + * is not bound to its file path, project, scope, overlay or variable name: + * + * - the store binds an entry to its project only through the scoped key inside + * the authenticated plaintext (`encodeURIComponent(scope)::encodeURIComponent(key)`); + * - a legacy `v1:` value moved to another variable or scope still decrypts. + * + * Adding context binding requires a new envelope version; a v1 change that made + * any of these tests fail would break existing v1 stores and values. + */ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import * as nodeCrypto from "node:crypto"; +import { + copyFile, + mkdir, + mkdtemp, + readFile, + rm, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { YAML } from "bun"; + +import { + PROJECT_COMPOSE_FILENAME, + PROJECT_CONFIG_FILENAME, + PROJECT_ENV_KEY_FILENAME, +} from "../src/constants.ts"; +import { + resolveProjectEnvConfig, + setProjectEnvValue, +} from "../src/lib/project-env-config.ts"; +import { resolveSecretStore } from "../src/lib/secret-store.ts"; +import { restoreEnv } from "./helpers/env.ts"; + +const STORE_KEY = "synthetic-v1-binding-store-key"; +const ENV_KEY = "synthetic-v1-binding-env-key"; +const SENTINEL = "synthetic-v1-binding-value-never-in-errors"; +const OTHER_SENTINEL = "synthetic-v1-binding-other-value"; +/** Fixed test vector IV; production always draws a random IV. */ +const FIXED_IV = Buffer.from("000102030405060708090a0b", "hex"); +const ISOLATED_ENV_KEYS = [ + "HOME", + "HACK_HOME", + "HACK_GLOBAL_CONFIG_PATH", + "HACK_SECRETS_FILE_KEY", + "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK", + "HACK_ENV_SECRET_KEY", +] as const; + +let root: string | null = null; +const previousEnv = new Map(); + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "hack-v1-binding-")); + for (const key of ISOLATED_ENV_KEYS) { + previousEnv.set(key, process.env[key]); + } + process.env.HOME = join(root, "home"); + process.env.HACK_HOME = join(root, "home", ".hack"); + process.env.HACK_GLOBAL_CONFIG_PATH = join(root, "home", "hack.config.json"); + process.env.HACK_SECRETS_FILE_KEY = STORE_KEY; + process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; + Reflect.deleteProperty(process.env, "HACK_ENV_SECRET_KEY"); + await mkdir(join(root, "home"), { recursive: true }); +}); + +afterEach(async () => { + if (root) { + await rm(root, { recursive: true, force: true }); + root = null; + } + for (const key of ISOLATED_ENV_KEYS) { + restoreEnv(key, previousEnv.get(key)); + } + previousEnv.clear(); +}); + +function requireRoot(): string { + if (!root) { + throw new Error("Missing temp root"); + } + return root; +} + +function deriveKey(keyText: string): Buffer { + return nodeCrypto.createHash("sha256").update(keyText, "utf8").digest(); +} + +/** AES-256-GCM seal with an optional AAD, mirroring the v1 parameters. */ +function seal(opts: { + readonly keyText: string; + readonly plaintext: string; + readonly aad?: Buffer; +}): { readonly iv: Buffer; readonly tag: Buffer; readonly ciphertext: Buffer } { + const cipher = nodeCrypto.createCipheriv( + "aes-256-gcm", + deriveKey(opts.keyText), + FIXED_IV, + { authTagLength: 16 } + ); + if (opts.aad) { + cipher.setAAD(opts.aad); + } + const ciphertext = Buffer.concat([ + cipher.update(opts.plaintext, "utf8"), + cipher.final(), + ]); + return { iv: FIXED_IV, tag: cipher.getAuthTag(), ciphertext }; +} + +function open(opts: { + readonly keyText: string; + readonly iv: string; + readonly tag: string; + readonly ciphertext: string; +}): string { + const decipher = nodeCrypto.createDecipheriv( + "aes-256-gcm", + deriveKey(opts.keyText), + Buffer.from(opts.iv, "base64"), + { authTagLength: 16 } + ); + decipher.setAuthTag(Buffer.from(opts.tag, "base64")); + return Buffer.concat([ + decipher.update(Buffer.from(opts.ciphertext, "base64")), + decipher.final(), + ]).toString("utf8"); +} + +// --------------------------------------------------------------------------- +// encrypted_file store envelope +// --------------------------------------------------------------------------- + +async function configureStore(opts: { + readonly storePath: string; + readonly cloud?: boolean; +}): Promise { + const configPath = process.env.HACK_GLOBAL_CONFIG_PATH; + if (!configPath) { + throw new Error("Missing HACK_GLOBAL_CONFIG_PATH"); + } + const secrets = opts.cloud + ? { + backend: "cloud", + encryptedFile: { path: opts.storePath }, + cloud: { + provider: "aws", + project: "dev-account", + secretPrefix: "hack-cli", + }, + } + : { backend: "encrypted_file", encryptedFile: { path: opts.storePath } }; + await writeFile( + configPath, + `${JSON.stringify({ controlPlane: { secrets } })}\n` + ); +} + +function storeEnvelopeText(opts: { + readonly plaintext: string; + readonly aad?: Buffer; +}): string { + const sealed = seal({ keyText: STORE_KEY, ...opts }); + return `${JSON.stringify({ + version: 1, + algorithm: "aes-256-gcm", + iv: sealed.iv.toString("base64"), + tag: sealed.tag.toString("base64"), + ciphertext: sealed.ciphertext.toString("base64"), + })}\n`; +} + +const STORE_PLAINTEXT = JSON.stringify({ + version: 1, + entries: { "app::TOKEN": SENTINEL }, +}); + +test("v1 store: written envelope has the pinned fields and a scoped-key plaintext", async () => { + const storePath = resolve(requireRoot(), "secrets.enc.json"); + await configureStore({ storePath }); + const store = await resolveSecretStore({ projectName: "app" }); + await store.set({ key: "TOKEN", value: SENTINEL }); + + const envelope = JSON.parse(await readFile(storePath, "utf8")) as Record< + string, + string | number + >; + expect(Object.keys(envelope)).toEqual([ + "version", + "algorithm", + "iv", + "tag", + "ciphertext", + ]); + expect(envelope.version).toBe(1); + expect(envelope.algorithm).toBe("aes-256-gcm"); + // Opening with SHA-256(key), no AAD, reproduces the exact v1 plaintext. + expect( + open({ + keyText: STORE_KEY, + iv: String(envelope.iv), + tag: String(envelope.tag), + ciphertext: String(envelope.ciphertext), + }) + ).toBe(STORE_PLAINTEXT); +}); + +test("v1 store: a test-vector envelope sealed without AAD opens", async () => { + const storePath = resolve(requireRoot(), "secrets.enc.json"); + await configureStore({ storePath }); + await writeFile(storePath, storeEnvelopeText({ plaintext: STORE_PLAINTEXT })); + const store = await resolveSecretStore({ projectName: "app" }); + expect(await store.get({ key: "TOKEN" })).toBe(SENTINEL); +}); + +for (const [label, aad] of [ + ["the store path", (storePath: string) => Buffer.from(storePath, "utf8")], + ["the project scope", () => Buffer.from("app", "utf8")], + ["the envelope version", () => Buffer.from("v1", "utf8")], +] as const) { + test(`v1 store: an envelope sealed with ${label} as AAD is refused (no AAD is authenticated)`, async () => { + const storePath = resolve(requireRoot(), "secrets.enc.json"); + await configureStore({ storePath }); + await writeFile( + storePath, + storeEnvelopeText({ plaintext: STORE_PLAINTEXT, aad: aad(storePath) }) + ); + const store = await resolveSecretStore({ projectName: "app" }); + const error = await store.get({ key: "TOKEN" }).then( + () => null, + (caught: unknown) => caught + ); + expect(error).toBeInstanceOf(Error); + expect(String(error)).toContain( + "Cannot decrypt encrypted secret store at " + ); + for (const privateText of [SENTINEL, STORE_KEY]) { + expect(String(error)).not.toContain(privateText); + } + }); +} + +test("v1 store: an envelope is not bound to its file path", async () => { + const firstPath = resolve(requireRoot(), "first.enc.json"); + await configureStore({ storePath: firstPath }); + const first = await resolveSecretStore({ projectName: "app" }); + await first.set({ key: "TOKEN", value: SENTINEL }); + + const movedPath = resolve(requireRoot(), "moved", "second.enc.json"); + await mkdir(resolve(requireRoot(), "moved")); + await copyFile(firstPath, movedPath); + await configureStore({ storePath: movedPath }); + const moved = await resolveSecretStore({ projectName: "app" }); + expect(moved.descriptor.location).toBe(movedPath); + expect(await moved.get({ key: "TOKEN" })).toBe(SENTINEL); +}); + +test("v1 store: project binding lives only in the scoped plaintext key", async () => { + const storePath = resolve(requireRoot(), "secrets.enc.json"); + await configureStore({ storePath }); + // A test-vector entry under another project's scoped key is visible to that + // project and invisible to this one; nothing outside the plaintext binds it. + await writeFile( + storePath, + storeEnvelopeText({ + plaintext: JSON.stringify({ + version: 1, + entries: { + "app::TOKEN": SENTINEL, + "my%20app::TOKEN": OTHER_SENTINEL, + }, + }), + }) + ); + expect( + await (await resolveSecretStore({ projectName: "app" })).get({ + key: "TOKEN", + }) + ).toBe(SENTINEL); + expect( + await (await resolveSecretStore({ projectName: "other" })).get({ + key: "TOKEN", + }) + ).toBeNull(); + expect( + await (await resolveSecretStore({ projectName: "my app" })).get({ + key: "TOKEN", + }) + ).toBe(OTHER_SENTINEL); +}); + +test("v1 store: the cloud shim scope is a plaintext key prefix in the same envelope", async () => { + const storePath = resolve(requireRoot(), "cloud.enc.json"); + await configureStore({ storePath, cloud: true }); + const store = await resolveSecretStore({ projectName: "app" }); + await store.set({ key: "API_TOKEN", value: SENTINEL }); + const envelope = JSON.parse(await readFile(storePath, "utf8")) as Record< + string, + string + >; + const plaintext = JSON.parse( + open({ + keyText: STORE_KEY, + iv: String(envelope.iv), + tag: String(envelope.tag), + ciphertext: String(envelope.ciphertext), + }) + ) as { readonly entries: Record }; + expect(plaintext.entries).toEqual({ + "cloud%3Aaws%3Adev-account%3Ahack-cli%3Aapp::API_TOKEN": SENTINEL, + }); +}); + +// --------------------------------------------------------------------------- +// legacy `v1:` secure env values +// --------------------------------------------------------------------------- + +async function createLegacyProject(): Promise<{ + readonly projectRoot: string; + readonly projectDir: string; + readonly configPath: string; +}> { + const projectRoot = resolve(requireRoot(), "repo"); + const projectDir = resolve(projectRoot, ".hack"); + await mkdir(projectDir, { recursive: true }); + await writeFile( + resolve(projectDir, PROJECT_COMPOSE_FILENAME), + "services:\n api: {}\n" + ); + await writeFile( + resolve(projectDir, PROJECT_CONFIG_FILENAME), + `${JSON.stringify({ name: "v1-binding", dev_host: "v1-binding.hack" })}\n` + ); + await writeFile( + resolve(projectRoot, PROJECT_ENV_KEY_FILENAME), + `${ENV_KEY}\n` + ); + return { + projectRoot, + projectDir, + configPath: resolve(projectDir, "hack.env.default.yaml"), + }; +} + +type LegacyConfig = { + readonly version: number; + readonly environment: string; + readonly secretsprovider: string; + readonly values: Record>; +}; + +async function readLegacyConfig(path: string): Promise { + return YAML.parse(await readFile(path, "utf8")) as LegacyConfig; +} + +function secureOf(config: LegacyConfig, scope: string, key: string): string { + const secure = config.values[scope]?.[key]?.secure; + if (typeof secure !== "string") { + throw new Error("missing secure value"); + } + return secure; +} + +async function resolveLegacy(project: { + readonly projectRoot: string; + readonly projectDir: string; +}) { + const resolved = await resolveProjectEnvConfig({ + projectRoot: project.projectRoot, + projectDir: project.projectDir, + envName: null, + serviceNames: ["api"], + }); + if (!resolved) { + throw new Error("missing resolved config"); + } + return resolved; +} + +test("legacy v1: a secure value swapped across variable and scope still decrypts", async () => { + const project = await createLegacyProject(); + for (const [scope, key, value] of [ + ["global", "TOKEN", SENTINEL], + ["api", "OTHER", OTHER_SENTINEL], + ] as const) { + await setProjectEnvValue({ + projectRoot: project.projectRoot, + projectDir: project.projectDir, + envName: null, + scope, + key, + value, + secret: true, + }); + } + const config = await readLegacyConfig(project.configPath); + const token = secureOf(config, "global", "TOKEN"); + const other = secureOf(config, "api", "OTHER"); + expect(token.split(":")[0]).toBe("v1"); + expect(Buffer.from(token.split(":")[1] ?? "", "base64").length).toBe(12); + + await writeFile( + project.configPath, + JSON.stringify({ + ...config, + values: { + global: { TOKEN: { secure: other } }, + api: { OTHER: { secure: token } }, + }, + }) + ); + const resolved = await resolveLegacy(project); + expect(resolved.globalEnv.TOKEN).toBe(OTHER_SENTINEL); + expect(resolved.serviceEnv.api?.OTHER).toBe(SENTINEL); +}); + +function legacyValue(aad?: Buffer): string { + const sealed = seal({ keyText: ENV_KEY, plaintext: SENTINEL, aad }); + return [ + "v1", + sealed.iv.toString("base64"), + sealed.tag.toString("base64"), + sealed.ciphertext.toString("base64"), + ].join(":"); +} + +async function writeLegacyValue(opts: { + readonly configPath: string; + readonly secure: string; +}): Promise { + await writeFile( + opts.configPath, + JSON.stringify({ + version: 1, + environment: "default", + secretsprovider: "project_key", + values: { global: { TOKEN: { secure: opts.secure } } }, + }) + ); +} + +test("legacy v1: a test-vector value sealed without AAD decrypts", async () => { + const project = await createLegacyProject(); + await writeLegacyValue({ + configPath: project.configPath, + secure: legacyValue(), + }); + expect((await resolveLegacy(project)).globalEnv.TOKEN).toBe(SENTINEL); +}); + +for (const [label, aad] of [ + ["the variable name", Buffer.from("TOKEN", "utf8")], + ["the scope", Buffer.from("global", "utf8")], + ["the version prefix", Buffer.from("v1", "utf8")], +] as const) { + test(`legacy v1: a value sealed with ${label} as AAD is refused (no AAD is authenticated)`, async () => { + const project = await createLegacyProject(); + await writeLegacyValue({ + configPath: project.configPath, + secure: legacyValue(aad), + }); + const error = await resolveLegacy(project).then( + () => null, + (caught: unknown) => caught + ); + expect(error).toBeInstanceOf(Error); + expect(error instanceof Error ? error.message : "").toBe( + "Invalid secure env value." + ); + for (const privateText of [SENTINEL, ENV_KEY]) { + expect(String(error)).not.toContain(privateText); + } + }); +} diff --git a/tests/private-env-file.test.ts b/tests/private-env-file.test.ts new file mode 100644 index 000000000..6d099d276 --- /dev/null +++ b/tests/private-env-file.test.ts @@ -0,0 +1,358 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + chmod, + link, + lstat, + mkdir, + mkdtemp, + readdir, + readFile, + rm, + stat, + symlink, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { + PROJECT_COMPOSE_FILENAME, + PROJECT_CONFIG_FILENAME, + PROJECT_ENV_FILENAME, +} from "../src/constants.ts"; +import { removeDotEnvKey, upsertDotEnvValue } from "../src/lib/hack-env.ts"; +import { PRIVATE_ENV_FILE_REFUSAL } from "../src/lib/private-env-file.ts"; +import { + materializeProjectEnv, + setProjectEnvValue, +} from "../src/lib/project-env-config.ts"; +import { restoreEnv } from "./helpers/env.ts"; + +// Synthetic values; none of them may appear in a refusal. +const SENTINEL_KEY = "SYNTHETIC_PRIVATE_ENV_KEY"; +const SENTINEL_VALUE = "synthetic-private-env-value-never-in-errors"; +const OUTSIDE_TEXT = "OUTSIDE_SENTINEL=synthetic-outside-file-never-read\n"; + +let root: string | null = null; +let previousHome: string | undefined; +let previousGlobalConfigPath: string | undefined; + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "hack-private-env-file-")); + previousHome = process.env.HOME; + previousGlobalConfigPath = process.env.HACK_GLOBAL_CONFIG_PATH; + // No real HOME or global config is read or written by these tests. + process.env.HOME = join(root, "home"); + process.env.HACK_GLOBAL_CONFIG_PATH = join(root, "home", "hack.config.json"); + await mkdir(join(root, "home"), { recursive: true }); +}); + +afterEach(async () => { + if (root) { + await rm(root, { recursive: true, force: true }); + root = null; + } + restoreEnv("HOME", previousHome); + restoreEnv("HACK_GLOBAL_CONFIG_PATH", previousGlobalConfigPath); +}); + +async function createProject(): Promise<{ + readonly projectRoot: string; + readonly projectDir: string; + readonly envFile: string; + readonly outsideFile: string; +}> { + if (!root) { + throw new Error("Missing temp root"); + } + const projectRoot = resolve(root, "repo"); + const projectDir = resolve(projectRoot, ".hack"); + await mkdir(projectDir, { recursive: true }); + await writeFile( + resolve(projectDir, PROJECT_COMPOSE_FILENAME), + "services:\n api: {}\n" + ); + await writeFile( + resolve(projectDir, PROJECT_CONFIG_FILENAME), + `${JSON.stringify({ name: "private-env-test", dev_host: "private-env.hack" })}\n` + ); + const outsideFile = resolve(root, "outside.env"); + await writeFile(outsideFile, OUTSIDE_TEXT); + return { + projectRoot, + projectDir, + envFile: resolve(projectDir, PROJECT_ENV_FILENAME), + outsideFile, + }; +} + +function expectClosedRefusal(error: unknown, paths: readonly string[]): void { + expect(error).toBeInstanceOf(Error); + expect(error instanceof Error ? error.message : "").toBe( + PRIVATE_ENV_FILE_REFUSAL + ); + for (const privateText of [ + SENTINEL_KEY, + SENTINEL_VALUE, + "OUTSIDE_SENTINEL", + ...paths, + ]) { + expect(String(error)).not.toContain(privateText); + } +} + +async function caught(promise: Promise): Promise { + return await promise.then( + () => null, + (error: unknown) => error + ); +} + +async function stagingLeftovers(directory: string): Promise { + return (await readdir(directory)).filter((name) => name.endsWith(".tmp")); +} + +test("upsertDotEnvValue creates .hack/.env with mode 0600", async () => { + const project = await createProject(); + const result = await upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }); + expect(result.changed).toBe(true); + expect((await stat(project.envFile)).mode & 0o777).toBe(0o600); + expect(await readFile(project.envFile, "utf8")).toContain( + `${SENTINEL_KEY}=${SENTINEL_VALUE}` + ); + expect(await stagingLeftovers(project.projectDir)).toEqual([]); +}); + +test("upsertDotEnvValue replaces .hack/.env atomically rather than writing in place", async () => { + const project = await createProject(); + await upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: "first", + }); + // A second name for the old inode keeps the old bytes only if the writer + // renamed a new file over the path instead of truncating and rewriting it. + const previousName = resolve(project.projectDir, "previous-inode"); + await link(project.envFile, previousName); + const before = await stat(project.envFile); + + await upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: "second", + }); + + const after = await stat(project.envFile); + expect(after.ino).not.toBe(before.ino); + expect(after.mode & 0o777).toBe(0o600); + expect(await readFile(previousName, "utf8")).toContain( + `${SENTINEL_KEY}=first` + ); + expect(await readFile(project.envFile, "utf8")).toContain( + `${SENTINEL_KEY}=second` + ); + expect(await stagingLeftovers(project.projectDir)).toEqual([]); +}); + +test("an unchanged .hack/.env with group or other bits is tightened to 0600", async () => { + const project = await createProject(); + await upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }); + const text = await readFile(project.envFile, "utf8"); + await chmod(project.envFile, 0o644); + + const result = await upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }); + expect(result.changed).toBe(false); + expect((await stat(project.envFile)).mode & 0o777).toBe(0o600); + expect(await readFile(project.envFile, "utf8")).toBe(text); +}); + +test("removeDotEnvKey keeps mode 0600 on the rewritten file", async () => { + const project = await createProject(); + await upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }); + await upsertDotEnvValue({ + envFile: project.envFile, + key: "KEEP", + value: "1", + }); + const result = await removeDotEnvKey({ + envFile: project.envFile, + key: SENTINEL_KEY, + }); + expect(result.changed).toBe(true); + expect((await stat(project.envFile)).mode & 0o777).toBe(0o600); + expect(await readFile(project.envFile, "utf8")).toBe("KEEP=1\n"); +}); + +test("negative control: a symlinked .hack/.env is refused without reading or writing its target", async () => { + const project = await createProject(); + await symlink(project.outsideFile, project.envFile); + + for (const attempt of [ + upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }), + removeDotEnvKey({ envFile: project.envFile, key: "OUTSIDE_SENTINEL" }), + ]) { + expectClosedRefusal(await caught(attempt), [ + project.envFile, + project.outsideFile, + ]); + } + + expect(await readFile(project.outsideFile, "utf8")).toBe(OUTSIDE_TEXT); + expect((await lstat(project.envFile)).isSymbolicLink()).toBe(true); + expect(await stagingLeftovers(project.projectDir)).toEqual([]); +}); + +test("negative control: a dangling .hack/.env symlink is refused and its target is never created", async () => { + const project = await createProject(); + const missingTarget = resolve(project.projectRoot, "..", "never-created.env"); + await symlink(missingTarget, project.envFile); + + expectClosedRefusal( + await caught( + upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }) + ), + [project.envFile, missingTarget] + ); + await expect(lstat(missingTarget)).rejects.toThrow(); + expect(await stagingLeftovers(project.projectDir)).toEqual([]); +}); + +test("negative control: an env overlay file that is a symlink is refused", async () => { + const project = await createProject(); + const overlayFile = `${project.envFile}.staging`; + await symlink(project.outsideFile, overlayFile); + + expectClosedRefusal( + await caught( + upsertDotEnvValue({ + envFile: overlayFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }) + ), + [overlayFile, project.outsideFile, "staging"] + ); + expect(await readFile(project.outsideFile, "utf8")).toBe(OUTSIDE_TEXT); +}); + +test("negative control: a symlinked .hack directory is refused and the linked directory is untouched", async () => { + if (!root) { + throw new Error("Missing temp root"); + } + const outsideDir = resolve(root, "outside-hack"); + await mkdir(outsideDir); + const projectRoot = resolve(root, "linked-repo"); + await mkdir(projectRoot); + const projectDir = resolve(projectRoot, ".hack"); + await symlink(outsideDir, projectDir); + const envFile = resolve(projectDir, PROJECT_ENV_FILENAME); + + expectClosedRefusal( + await caught( + upsertDotEnvValue({ envFile, key: SENTINEL_KEY, value: SENTINEL_VALUE }) + ), + [envFile, outsideDir] + ); + expect(await readdir(outsideDir)).toEqual([]); +}); + +test("a directory at the .hack/.env path is refused with the same closed diagnostic", async () => { + const project = await createProject(); + await mkdir(project.envFile); + expectClosedRefusal( + await caught( + upsertDotEnvValue({ + envFile: project.envFile, + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + }) + ), + [project.envFile] + ); +}); + +test("materializeProjectEnv writes .hack/.env with mode 0600 by atomic replace", async () => { + const project = await createProject(); + await setProjectEnvValue({ + projectRoot: project.projectRoot, + projectDir: project.projectDir, + envName: null, + scope: "global", + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + secret: false, + }); + await writeFile(project.envFile, "STALE=1\n", { mode: 0o644 }); + await chmod(project.envFile, 0o644); + const previousName = resolve(project.projectDir, "previous-inode"); + await link(project.envFile, previousName); + + const result = await materializeProjectEnv({ + projectRoot: project.projectRoot, + projectDir: project.projectDir, + envName: null, + serviceName: null, + serviceNames: ["api"], + }); + + expect(result.changed).toBe(true); + expect((await stat(project.envFile)).mode & 0o777).toBe(0o600); + expect(await readFile(project.envFile, "utf8")).toContain( + `${SENTINEL_KEY}=${SENTINEL_VALUE}` + ); + expect(await readFile(previousName, "utf8")).toBe("STALE=1\n"); + expect(await stagingLeftovers(project.projectDir)).toEqual([]); +}); + +test("negative control: materializeProjectEnv refuses a symlinked .hack/.env target", async () => { + const project = await createProject(); + await setProjectEnvValue({ + projectRoot: project.projectRoot, + projectDir: project.projectDir, + envName: null, + scope: "global", + key: SENTINEL_KEY, + value: SENTINEL_VALUE, + secret: false, + }); + await symlink(project.outsideFile, project.envFile); + + expectClosedRefusal( + await caught( + materializeProjectEnv({ + projectRoot: project.projectRoot, + projectDir: project.projectDir, + envName: null, + serviceName: null, + serviceNames: ["api"], + }) + ), + [project.envFile, project.outsideFile] + ); + expect(await readFile(project.outsideFile, "utf8")).toBe(OUTSIDE_TEXT); + expect((await lstat(project.envFile)).isSymbolicLink()).toBe(true); +}); diff --git a/tests/secret-store-file-hygiene.test.ts b/tests/secret-store-file-hygiene.test.ts new file mode 100644 index 000000000..3be32e5c7 --- /dev/null +++ b/tests/secret-store-file-hygiene.test.ts @@ -0,0 +1,248 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + link, + lstat, + mkdir, + mkdtemp, + readdir, + readFile, + rm, + stat, + symlink, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { resolveSecretStore } from "../src/lib/secret-store.ts"; +import { restoreEnv } from "./helpers/env.ts"; + +const STORE_KEY = "synthetic-store-hygiene-key-for-tests-only"; +const STORE_SENTINEL = "synthetic-store-hygiene-value-never-in-errors"; +const SECRET_STORE_MODULE = resolve( + import.meta.dir, + "..", + "src", + "lib", + "secret-store.ts" +); +const WRITES_PER_PROCESS = 20; +const ISOLATED_ENV_KEYS = [ + "HOME", + "HACK_HOME", + "HACK_GLOBAL_CONFIG_PATH", + "HACK_SECRETS_FILE_KEY", + "HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK", +] as const; + +let root: string | null = null; +const previousEnv = new Map(); + +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "hack-secret-store-hygiene-")); + for (const key of ISOLATED_ENV_KEYS) { + previousEnv.set(key, process.env[key]); + } + // Temp HOME and global config; the env key wins, and the keychain fallback is + // disabled so no test can reach the native keychain. + process.env.HOME = join(root, "home"); + process.env.HACK_HOME = join(root, "home", ".hack"); + process.env.HACK_GLOBAL_CONFIG_PATH = join(root, "home", "hack.config.json"); + process.env.HACK_SECRETS_FILE_KEY = STORE_KEY; + process.env.HACK_SECRETS_DISABLE_KEYCHAIN_FALLBACK = "1"; + await mkdir(join(root, "home"), { recursive: true }); +}); + +afterEach(async () => { + if (root) { + await rm(root, { recursive: true, force: true }); + root = null; + } + for (const key of ISOLATED_ENV_KEYS) { + restoreEnv(key, previousEnv.get(key)); + } + previousEnv.clear(); +}); + +async function configureEncryptedFileStore(): Promise<{ + readonly storeDir: string; + readonly storePath: string; +}> { + if (!root) { + throw new Error("Missing temp root"); + } + const storeDir = resolve(root, "store"); + await mkdir(storeDir); + const storePath = resolve(storeDir, "secrets.enc.json"); + const configPath = process.env.HACK_GLOBAL_CONFIG_PATH; + if (!configPath) { + throw new Error("Missing HACK_GLOBAL_CONFIG_PATH"); + } + await writeFile( + configPath, + `${JSON.stringify({ + controlPlane: { + secrets: { + backend: "encrypted_file", + encryptedFile: { path: storePath }, + }, + }, + })}\n` + ); + return { storeDir, storePath }; +} + +async function nonEnvelopeEntries(directory: string): Promise { + return (await readdir(directory)).filter( + (name) => name !== "secrets.enc.json" + ); +} + +test("encrypted_file store writes mode 0600 by atomic replace", async () => { + const { storeDir, storePath } = await configureEncryptedFileStore(); + const store = await resolveSecretStore({ projectName: "app" }); + await store.set({ key: "TOKEN", value: "first" }); + expect((await stat(storePath)).mode & 0o777).toBe(0o600); + + // The old inode keeps its bytes only when the writer renames a new file over + // the path instead of truncating and rewriting the existing one. + const previousName = resolve(storeDir, "previous-inode"); + await link(storePath, previousName); + const previousText = await readFile(storePath, "utf8"); + const before = await stat(storePath); + + await store.set({ key: "TOKEN", value: STORE_SENTINEL }); + + const after = await stat(storePath); + expect(after.ino).not.toBe(before.ino); + expect(after.mode & 0o777).toBe(0o600); + expect(await readFile(previousName, "utf8")).toBe(previousText); + expect(await store.get({ key: "TOKEN" })).toBe(STORE_SENTINEL); + expect(await readFile(storePath, "utf8")).not.toContain(STORE_SENTINEL); + + expect(await store.delete({ key: "TOKEN" })).toBe(true); + expect((await stat(storePath)).mode & 0o777).toBe(0o600); + // No staging file or lock receipt is left behind. + expect(await nonEnvelopeEntries(storeDir)).toEqual(["previous-inode"]); +}); + +test("encrypted_file store refuses to write through a symlinked store path", async () => { + if (!root) { + throw new Error("Missing temp root"); + } + const { storeDir, storePath } = await configureEncryptedFileStore(); + const outsideFile = resolve(root, "outside-store.json"); + // An empty target reads as an empty store, so only the write guard can stop + // the set below. + await writeFile(outsideFile, ""); + await symlink(outsideFile, storePath); + + const store = await resolveSecretStore({ projectName: "app" }); + const error = await store.set({ key: "TOKEN", value: STORE_SENTINEL }).then( + () => null, + (caught: unknown) => caught + ); + + expect(error).toBeInstanceOf(Error); + expect(error instanceof Error ? error.message : "").toBe( + "Refused encrypted secret store write: the store path is a symlink or not a regular file. Replace it with a regular file and retry; values omitted." + ); + for (const privateText of [STORE_SENTINEL, STORE_KEY, "TOKEN", outsideFile]) { + expect(String(error)).not.toContain(privateText); + } + expect(await readFile(outsideFile, "utf8")).toBe(""); + expect((await lstat(storePath)).isSymbolicLink()).toBe(true); + expect(await nonEnvelopeEntries(storeDir)).toEqual([]); +}); + +test("negative control: concurrent in-process writers do not lose an update", async () => { + const { storeDir } = await configureEncryptedFileStore(); + const writers = await Promise.all( + ["app", "app", "other", "other"].map((projectName) => + resolveSecretStore({ projectName }) + ) + ); + const writes: Promise[] = []; + for (let index = 0; index < 24; index += 1) { + const writer = writers[index % writers.length]; + if (!writer) { + throw new Error("missing writer"); + } + writes.push(writer.set({ key: `KEY_${index}`, value: `value-${index}` })); + } + await Promise.all(writes); + + for (const projectName of ["app", "other"]) { + const store = await resolveSecretStore({ projectName }); + for (let index = 0; index < 24; index += 1) { + const writtenBy = index % 4 < 2 ? "app" : "other"; + expect(await store.get({ key: `KEY_${index}` })).toBe( + writtenBy === projectName ? `value-${index}` : null + ); + } + } + expect(await nonEnvelopeEntries(storeDir)).toEqual([]); +}); + +test("negative control: concurrent writer processes do not lose an update", async () => { + if (!root) { + throw new Error("Missing temp root"); + } + const { storeDir } = await configureEncryptedFileStore(); + const script = resolve(root, "store-writer.ts"); + await writeFile( + script, + [ + `import { resolveSecretStore } from ${JSON.stringify(SECRET_STORE_MODULE)};`, + "const prefix = process.argv[2] ?? 'P';", + "const startAt = Number(process.argv[3]);", + "const store = await resolveSecretStore({ projectName: 'app' });", + "// Start together so the writers' read-modify-write cycles overlap.", + "await Bun.sleep(Math.max(0, startAt - Date.now()));", + `for (let index = 0; index < ${WRITES_PER_PROCESS}; index += 1) {`, + " await store.set({ key: `${prefix}_${index}`, value: `${prefix}-${index}` });", + "}", + "", + ].join("\n") + ); + const childEnv: Record = { + PATH: process.env.PATH ?? "", + TMPDIR: process.env.TMPDIR ?? tmpdir(), + }; + for (const key of ISOLATED_ENV_KEYS) { + const value = process.env[key]; + if (value !== undefined) { + childEnv[key] = value; + } + } + const prefixes = ["A", "B", "C"]; + const startAt = String(Date.now() + 750); + const children = prefixes.map((prefix) => + Bun.spawn({ + cmd: [process.execPath, script, prefix, startAt], + cwd: root ?? undefined, + env: childEnv, + stdout: "ignore", + stderr: "pipe", + }) + ); + const exits = await Promise.all( + children.map(async (child) => ({ + code: await child.exited, + stderr: await new Response(child.stderr).text(), + })) + ); + for (const exit of exits) { + expect(exit).toEqual({ code: 0, stderr: "" }); + } + + const store = await resolveSecretStore({ projectName: "app" }); + for (const prefix of prefixes) { + for (let index = 0; index < WRITES_PER_PROCESS; index += 1) { + expect(await store.get({ key: `${prefix}_${index}` })).toBe( + `${prefix}-${index}` + ); + } + } + expect(await nonEnvelopeEntries(storeDir)).toEqual([]); +}); From 10b6d73cfa8fab7460bff1299a9c328e43f4bf35 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 07:30:32 -0400 Subject: [PATCH 30/32] fix(agents): point the env matrix bullet at the offline env guide The env-composition section linked the command and delivery matrix by a bare GitHub URL. Packaged skills must work offline (#295), and that test fails once both changes land. Point at docs/env.md in words instead, bump the content revision, re-render the checked-in surfaces, and append the new hack-cli-skill and cursor-rules render fingerprints. Co-Authored-By: Claude Opus 5.5 --- .codex/skills/hack-cli/SKILL.md | 4 ++-- .cursor/rules/hack.mdc | 4 ++-- examples/basic/AGENTS.md | 4 ++-- examples/basic/CLAUDE.md | 4 ++-- plugins/hack/rules/hack.mdc | 4 ++-- plugins/hack/skills/hack-cli/SKILL.md | 4 ++-- src/agents/instruction-source.ts | 2 +- src/agents/integration-ownership.ts | 2 ++ src/agents/integration-revision.ts | 2 +- 9 files changed, 16 insertions(+), 14 deletions(-) diff --git a/.codex/skills/hack-cli/SKILL.md b/.codex/skills/hack-cli/SKILL.md index a46be0dbf..28776a4d0 100644 --- a/.codex/skills/hack-cli/SKILL.md +++ b/.codex/skills/hack-cli/SKILL.md @@ -14,7 +14,7 @@ Use `hack` as the primary interface for local-first development. - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `3cb1b0eabdea` (version alone is not a freshness guarantee). +- Content revision: `51163ae32e56` (version alone is not a freshness guarantee). ## Product boundary @@ -152,7 +152,7 @@ Use `hack` as the primary interface for local-first development. - Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. - Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. - Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. -- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix +- Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository). ## Branch instances (parallel envs) diff --git a/.cursor/rules/hack.mdc b/.cursor/rules/hack.mdc index 1adb401c3..2dcc4e8f2 100644 --- a/.cursor/rules/hack.mdc +++ b/.cursor/rules/hack.mdc @@ -9,7 +9,7 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `3cb1b0eabdea` (version alone is not a freshness guarantee). +- Content revision: `51163ae32e56` (version alone is not a freshness guarantee). ## Product boundary @@ -71,7 +71,7 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. - Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. - Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. -- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix +- Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository). ## Agent integration maintenance diff --git a/examples/basic/AGENTS.md b/examples/basic/AGENTS.md index 8c448dc10..c5fb011c0 100644 --- a/examples/basic/AGENTS.md +++ b/examples/basic/AGENTS.md @@ -6,7 +6,7 @@ Use `hack` as the single interface for local-first runtime orchestration (compos Integration freshness: - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `3cb1b0eabdea` (version alone is not a freshness guarantee). +- Content revision: `51163ae32e56` (version alone is not a freshness guarantee). Product boundary: - Supported CLI surface: project init, up/down/restart, open, logs, env, host exec/shell, sessions, doctor, and daemon. @@ -129,7 +129,7 @@ Env composition: - Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. - Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. - Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. -- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix +- Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository). Branch instances (parallel envs): - Use a branch instance when you need two versions running at once (PR review, experiments, migrations) or want to keep a stable environment while testing another branch. diff --git a/examples/basic/CLAUDE.md b/examples/basic/CLAUDE.md index 8c448dc10..c5fb011c0 100644 --- a/examples/basic/CLAUDE.md +++ b/examples/basic/CLAUDE.md @@ -6,7 +6,7 @@ Use `hack` as the single interface for local-first runtime orchestration (compos Integration freshness: - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `3cb1b0eabdea` (version alone is not a freshness guarantee). +- Content revision: `51163ae32e56` (version alone is not a freshness guarantee). Product boundary: - Supported CLI surface: project init, up/down/restart, open, logs, env, host exec/shell, sessions, doctor, and daemon. @@ -129,7 +129,7 @@ Env composition: - Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. - Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. - Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. -- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix +- Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository). Branch instances (parallel envs): - Use a branch instance when you need two versions running at once (PR review, experiments, migrations) or want to keep a stable environment while testing another branch. diff --git a/plugins/hack/rules/hack.mdc b/plugins/hack/rules/hack.mdc index 77e495ecf..e184f0ef9 100644 --- a/plugins/hack/rules/hack.mdc +++ b/plugins/hack/rules/hack.mdc @@ -14,7 +14,7 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `3cb1b0eabdea` (version alone is not a freshness guarantee). +- Content revision: `51163ae32e56` (version alone is not a freshness guarantee). ## Product boundary @@ -76,7 +76,7 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is - Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. - Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. - Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. -- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix +- Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository). ## Agent integration maintenance diff --git a/plugins/hack/skills/hack-cli/SKILL.md b/plugins/hack/skills/hack-cli/SKILL.md index a46be0dbf..28776a4d0 100644 --- a/plugins/hack/skills/hack-cli/SKILL.md +++ b/plugins/hack/skills/hack-cli/SKILL.md @@ -14,7 +14,7 @@ Use `hack` as the primary interface for local-first development. - These instructions were generated by hack CLI v4.2.1; use the content revision and command behavior to assess freshness. - Check guidance only when Hack is relevant and there is evidence of drift; use targeted `hack setup` checks for a read-only inventory. Routine tasks do not require a startup audit. -- Content revision: `3cb1b0eabdea` (version alone is not a freshness guarantee). +- Content revision: `51163ae32e56` (version alone is not a freshness guarantee). ## Product boundary @@ -152,7 +152,7 @@ Use `hack` as the primary interface for local-first development. - Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation. - Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret. - Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`. -- Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix +- Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository). ## Branch instances (parallel envs) diff --git a/src/agents/instruction-source.ts b/src/agents/instruction-source.ts index 153f83ebe..25a1cb569 100644 --- a/src/agents/instruction-source.ts +++ b/src/agents/instruction-source.ts @@ -241,7 +241,7 @@ export const INSTRUCTION_SECTIONS: readonly InstructionSection[] = [ "Omit `--env` to use the configured default overlay; `--env base` (or `default`/`none`) selects base layers only, the same on every command. An overlay name grants no production authority and no network or instance isolation.", "Use `hack env explain --env --service ` for provenance without the value; use `hack env get ` only when a plaintext value is explicitly needed, and treat its output as secret.", "Running containers keep the env they were created with; after an env change, refresh with `hack env apply --service ` or `hack restart`.", - "Accepted flags, injection timing and refresh path per command: https://github.com/hack-dance/hack/blob/next/docs/env.md#command-and-delivery-matrix", + "Accepted flags, injection timing and refresh path per command: the command and delivery matrix in Hack's env guide (`docs/env.md` in the Hack repository).", ], }, diff --git a/src/agents/integration-ownership.ts b/src/agents/integration-ownership.ts index 0e1cbb076..4156a6251 100644 --- a/src/agents/integration-ownership.ts +++ b/src/agents/integration-ownership.ts @@ -105,6 +105,7 @@ const KNOWN_RENDER_FINGERPRINTS: Readonly< "b6465c18c6d4f41b30a600b8", // v5.0.0-next.1..v5.0.0-next.2 "121d8f26374d24f2a09890a3", // v5.0.0-next.3..next "6449442a8f230412d4e5f9e7", // v5.0.0-next.4..next + "d83da046cf7541ba30950880", // v5.0.0-next.4..next (offline env-matrix pointer) ]), "hack-init-skill": new Set([ "cd15fb6d8ed8e25071d05e75", // v3.1.0..v4.0.3 @@ -131,6 +132,7 @@ const KNOWN_RENDER_FINGERPRINTS: Readonly< "01d7431b9a5caaba3ffd3463", // v5.0.0-next.1..v5.0.0-next.2 "06cf6d0920afc1780b9946ba", // v5.0.0-next.3..next "cf0a31e197f9034485413674", // v5.0.0-next.4..next + "3a99c9097855f93808e482cb", // v5.0.0-next.4..next (offline env-matrix pointer) ]), }; diff --git a/src/agents/integration-revision.ts b/src/agents/integration-revision.ts index 6f3a624e3..d418edef4 100644 --- a/src/agents/integration-revision.ts +++ b/src/agents/integration-revision.ts @@ -3,4 +3,4 @@ * source test recomputes this value and fails whenever guidance changes * without a revision update. */ -export const HACK_AGENT_INTEGRATION_CONTENT_REVISION = "3cb1b0eabdea"; +export const HACK_AGENT_INTEGRATION_CONTENT_REVISION = "51163ae32e56"; From cb665d5efc66c96560d748ba6f1afcff8028d7d1 Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 07:32:42 -0400 Subject: [PATCH 31/32] fix(env): scope the env-file symlink refusal doc to write paths docs/env.md said Hack refuses to read or write a symlinked env file, but only the write and update paths (env set/unset, materialize, the daemon env route) use the no-follow reader; code that only reads an env file still follows a symlink. Narrow the sentence accordingly. Add a test that calls replacePrivateFileAtomic directly with a live and a dangling symlink, so its own target check is covered instead of being masked by the earlier O_NOFOLLOW read in the env-file callers. Refs HACK-1127 Co-Authored-By: Claude Opus 5.5 --- docs/env.md | 7 ++++--- tests/private-env-file.test.ts | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/docs/env.md b/docs/env.md index ccd5e690c..339205d00 100644 --- a/docs/env.md +++ b/docs/env.md @@ -199,10 +199,11 @@ Hack writes `.hack/.env` and `.hack/.env.` (from `materialize`, legacy `hack env set`, and the daemon env route) with mode `0600` by atomic replace: a private temporary file in `.hack/` is written, fsynced and renamed over the target, so readers never see a partial file. An existing file with group or other access is -tightened to `0600` even when its contents are unchanged. Hack refuses, with a fixed -message that names no file or value, to read or write an env file that is a symlink +tightened to `0600` even when its contents are unchanged. These write and update paths +refuse, with a fixed message that names no file or value, an env file that is a symlink (including a dangling one) or not a regular file, or one whose `.hack` directory is a -symlink. Replace the link with a regular file or directory to continue. +symlink. Replace the link with a regular file or directory to continue. Code paths +that only read an env file do not apply this check and still follow a symlink. For host execution, `hack host exec`, `hack host shell`, and lifecycle hooks/processes use a host-local view. That means Hack prefers host-usable values when it can: diff --git a/tests/private-env-file.test.ts b/tests/private-env-file.test.ts index 6d099d276..798792cae 100644 --- a/tests/private-env-file.test.ts +++ b/tests/private-env-file.test.ts @@ -22,6 +22,7 @@ import { } from "../src/constants.ts"; import { removeDotEnvKey, upsertDotEnvValue } from "../src/lib/hack-env.ts"; import { PRIVATE_ENV_FILE_REFUSAL } from "../src/lib/private-env-file.ts"; +import { replacePrivateFileAtomic } from "../src/lib/private-file.ts"; import { materializeProjectEnv, setProjectEnvValue, @@ -356,3 +357,35 @@ test("negative control: materializeProjectEnv refuses a symlinked .hack/.env tar expect(await readFile(project.outsideFile, "utf8")).toBe(OUTSIDE_TEXT); expect((await lstat(project.envFile)).isSymbolicLink()).toBe(true); }); + +test("negative control: replacePrivateFileAtomic itself refuses a symlinked target without the read guard", async () => { + const project = await createProject(); + const danglingTarget = resolve( + project.projectRoot, + "..", + "never-created.env" + ); + const danglingLink = `${project.envFile}.dangling`; + await symlink(project.outsideFile, project.envFile); + await symlink(danglingTarget, danglingLink); + + // Called directly, so the earlier O_NOFOLLOW read in the env-file callers + // cannot mask a missing check in the replace step. + for (const path of [project.envFile, danglingLink]) { + expectClosedRefusal( + await caught( + replacePrivateFileAtomic({ + path, + directories: [project.projectDir], + text: `${SENTINEL_KEY}=${SENTINEL_VALUE}\n`, + refusal: () => new Error(PRIVATE_ENV_FILE_REFUSAL), + }) + ), + [path, project.outsideFile, danglingTarget] + ); + expect((await lstat(path)).isSymbolicLink()).toBe(true); + } + expect(await readFile(project.outsideFile, "utf8")).toBe(OUTSIDE_TEXT); + await expect(lstat(danglingTarget)).rejects.toThrow(); + expect(await stagingLeftovers(project.projectDir)).toEqual([]); +}); From 5f8cb79c06a1b28935e9b59c60de1b121dbc4a2f Mon Sep 17 00:00:00 2001 From: Dimitri Kennedy Date: Mon, 12 Oct 2026 09:07:05 -0400 Subject: [PATCH 32/32] fix(logger): pass messages to gum after an end-of-options marker The gum-backed logger passed the log message as a bare argument, so a message starting with "-" or "--" (for example "--value-stdin requires ...") was parsed by gum's kong CLI as an unknown flag. gum printed its usage and "gum: error: unknown flag" to stderr before the logger fell back to clack. Every gum call site in src/ui/gum.ts now places free text (log message and structured fields, style/join text, confirm prompt, choose/filter options, file path, pager content, format template, spin command, version-check constraint) after "--", and binds valued flags such as --header, --placeholder, --prompt, --value, --title and --affirmative in single-token --flag=value form so a dash-leading value cannot be read as an option. gum invocations now go through a small GumRunner boundary so tests can assert argv without gum installed or any process being spawned. Co-Authored-By: Claude Opus 5.5 --- src/ui/gum.ts | 406 +++++++++++++++++-------------- tests/fixtures/gum | 3 + tests/gum-end-of-options.test.ts | 301 +++++++++++++++++++++++ tests/gum.test.ts | 4 +- 4 files changed, 529 insertions(+), 185 deletions(-) create mode 100644 tests/gum-end-of-options.test.ts diff --git a/src/ui/gum.ts b/src/ui/gum.ts index ccfa78765..914f011c0 100644 --- a/src/ui/gum.ts +++ b/src/ui/gum.ts @@ -142,6 +142,76 @@ export async function ensureBundledGumInstalled(): Promise>; + readonly stdin: "inherit" | "ignore" | { readonly text: string }; + readonly stdout: "inherit" | "pipe" | "ignore"; +} + +export interface GumRunResult { + readonly exitCode: number; + /** Captured stdout when the request used `stdout: "pipe"`, otherwise "". */ + readonly stdout: string; +} + +/** Process boundary for every gum invocation in this module. */ +export interface GumRunner { + /** Synchronous run, used by the logger so log lines keep their order. */ + runSync(request: GumRunRequest): number; + run(request: GumRunRequest): Promise; +} + +const bunGumRunner: GumRunner = { + runSync({ cmd, env, stdin, stdout }) { + const res = Bun.spawnSync({ + cmd: [...cmd], + env: { ...env }, + stdin: + typeof stdin === "string" + ? stdin + : new TextEncoder().encode(stdin.text), + stdout, + stderr: "inherit", + }); + return res.exitCode; + }, + async run({ cmd, env, stdin, stdout }) { + const proc = Bun.spawn([...cmd], { + env: { ...env }, + stdin: typeof stdin === "string" ? stdin : streamFromText(stdin.text), + stdout, + stderr: "inherit", + }); + const piped = proc.stdout; + const [text, exitCode] = await Promise.all([ + piped instanceof ReadableStream + ? new Response(piped).text() + : Promise.resolve(""), + proc.exited, + ]); + return { exitCode, stdout: text }; + }, +}; + +let gumRunner: GumRunner = bunGumRunner; + +/** + * Test seam: route gum invocations through `runner` instead of spawning. + * Pass `null` to restore the real Bun process runner. + */ +export function setGumRunnerForTests({ + runner, +}: { + readonly runner: GumRunner | null; +}): void { + gumRunner = runner ?? bunGumRunner; +} + export function tryGumLog({ level, message, fields }: GumLogInput): boolean { const gum = getGumPath(); if (!gum) { @@ -162,22 +232,19 @@ export function tryGumLog({ level, message, fields }: GumLogInput): boolean { const cmd = [ gum, "log", - "--level", - level, + ...valueFlag({ flag: "level", value: level }), ...(kv.length > 0 ? ["--structured"] : []), - message, - ...kv, + ...positionals([message, ...kv]), ]; - const res = Bun.spawnSync({ + const exitCode = gumRunner.runSync({ cmd, env: buildGumRenderEnv(), stdin: "ignore", stdout: "inherit", - stderr: "inherit", }); - return res.exitCode === 0; + return exitCode === 0; } function getBundledGumInstallPath(): string { @@ -268,23 +335,21 @@ export async function gumConfirm({ const cmd = [ gum, "confirm", - ...(prompt ? [prompt] : []), ...(defaultChoice ? ["--default"] : []), ...(showOutput ? ["--show-output"] : []), - ...(affirmative ? ["--affirmative", affirmative] : []), - ...(negative ? ["--negative", negative] : []), + ...valueFlag({ flag: "affirmative", value: affirmative }), + ...valueFlag({ flag: "negative", value: negative }), ...booleanFlag({ value: showHelp, flag: "show-help" }), - ...(timeout ? ["--timeout", timeout] : []), + ...valueFlag({ flag: "timeout", value: timeout }), + ...positionals(prompt ? [prompt] : []), ]; - const proc = Bun.spawn(cmd, { + const { exitCode } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "inherit", - stderr: "inherit", }); - - const exitCode = await proc.exited; if (exitCode === 0) { return { ok: true, value: true }; } @@ -330,29 +395,24 @@ export async function gumInput({ const cmd = [ gum, "input", - ...(placeholder ? ["--placeholder", placeholder] : []), - ...(prompt ? ["--prompt", prompt] : []), - ...(header ? ["--header", header] : []), - ...(value ? ["--value", value] : []), - ...(width === undefined ? [] : ["--width", String(width)]), - ...(charLimit === undefined ? [] : ["--char-limit", String(charLimit)]), + ...valueFlag({ flag: "placeholder", value: placeholder }), + ...valueFlag({ flag: "prompt", value: prompt }), + ...valueFlag({ flag: "header", value: header }), + ...valueFlag({ flag: "value", value }), + ...valueFlag({ flag: "width", value: width }), + ...valueFlag({ flag: "char-limit", value: charLimit }), ...(password ? ["--password"] : []), ...booleanFlag({ value: showHelp, flag: "show-help" }), - ...(timeout ? ["--timeout", timeout] : []), + ...valueFlag({ flag: "timeout", value: timeout }), ...booleanFlag({ value: stripAnsi, flag: "strip-ansi" }), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout.trimEnd() }; } @@ -401,32 +461,27 @@ export async function gumWrite({ const cmd = [ gum, "write", - ...(width === undefined ? [] : ["--width", String(width)]), - ...(height === undefined ? [] : ["--height", String(height)]), - ...(header ? ["--header", header] : []), - ...(placeholder ? ["--placeholder", placeholder] : []), - ...(prompt ? ["--prompt", prompt] : []), + ...valueFlag({ flag: "width", value: width }), + ...valueFlag({ flag: "height", value: height }), + ...valueFlag({ flag: "header", value: header }), + ...valueFlag({ flag: "placeholder", value: placeholder }), + ...valueFlag({ flag: "prompt", value: prompt }), ...(showCursorLine ? ["--show-cursor-line"] : []), ...(showLineNumbers ? ["--show-line-numbers"] : []), - ...(value ? ["--value", value] : []), - ...(charLimit === undefined ? [] : ["--char-limit", String(charLimit)]), - ...(maxLines === undefined ? [] : ["--max-lines", String(maxLines)]), + ...valueFlag({ flag: "value", value }), + ...valueFlag({ flag: "char-limit", value: charLimit }), + ...valueFlag({ flag: "max-lines", value: maxLines }), ...booleanFlag({ value: showHelp, flag: "show-help" }), - ...(timeout ? ["--timeout", timeout] : []), + ...valueFlag({ flag: "timeout", value: timeout }), ...booleanFlag({ value: stripAnsi, flag: "strip-ansi" }), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout.trimEnd() }; } @@ -481,40 +536,29 @@ export async function gumChooseMany( const cmd = [ gum, "choose", - ...input.options, ...(input.ordered ? ["--ordered"] : []), - ...(input.height === undefined ? [] : ["--height", String(input.height)]), - ...(input.cursor ? ["--cursor", input.cursor] : []), + ...valueFlag({ flag: "height", value: input.height }), + ...valueFlag({ flag: "cursor", value: input.cursor }), ...booleanFlag({ value: input.showHelp, flag: "show-help" }), - ...(input.timeout ? ["--timeout", input.timeout] : []), - ...(input.header ? ["--header", input.header] : []), + ...valueFlag({ flag: "timeout", value: input.timeout }), + ...valueFlag({ flag: "header", value: input.header }), ...buildSelectedArg(input.selected), - ...(input.inputDelimiter - ? ["--input-delimiter", input.inputDelimiter] - : []), - ...(input.outputDelimiter - ? ["--output-delimiter", input.outputDelimiter] - : []), - ...(input.labelDelimiter - ? ["--label-delimiter", input.labelDelimiter] - : []), + ...valueFlag({ flag: "input-delimiter", value: input.inputDelimiter }), + ...valueFlag({ flag: "output-delimiter", value: input.outputDelimiter }), + ...valueFlag({ flag: "label-delimiter", value: input.labelDelimiter }), ...booleanFlag({ value: input.stripAnsi, flag: "strip-ansi" }), ...(input.noLimit ? ["--no-limit"] : []), - ...(input.limit === undefined ? [] : ["--limit", String(input.limit)]), + ...valueFlag({ flag: "limit", value: input.limit }), ...(input.selectIfOne ? ["--select-if-one"] : []), + ...positionals(input.options), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { const out = stdout.trimEnd(); const parts = delimiter === "\n" ? out.split("\n") : out.split(delimiter); @@ -576,43 +620,34 @@ export async function gumFilterMany( const cmd = [ gum, "filter", - ...input.options, ...buildSelectedArg(input.selected), ...booleanFlag({ value: input.showHelp, flag: "show-help" }), - ...(input.header ? ["--header", input.header] : []), - ...(input.placeholder ? ["--placeholder", input.placeholder] : []), - ...(input.prompt ? ["--prompt", input.prompt] : []), - ...(input.width === undefined ? [] : ["--width", String(input.width)]), - ...(input.height === undefined ? [] : ["--height", String(input.height)]), - ...(input.value ? ["--value", input.value] : []), + ...valueFlag({ flag: "header", value: input.header }), + ...valueFlag({ flag: "placeholder", value: input.placeholder }), + ...valueFlag({ flag: "prompt", value: input.prompt }), + ...valueFlag({ flag: "width", value: input.width }), + ...valueFlag({ flag: "height", value: input.height }), + ...valueFlag({ flag: "value", value: input.value }), ...(input.reverse ? ["--reverse"] : []), ...booleanFlag({ value: input.fuzzy, flag: "fuzzy" }), ...booleanFlag({ value: input.fuzzySort, flag: "fuzzy-sort" }), - ...(input.timeout ? ["--timeout", input.timeout] : []), - ...(input.inputDelimiter - ? ["--input-delimiter", input.inputDelimiter] - : []), - ...(input.outputDelimiter - ? ["--output-delimiter", input.outputDelimiter] - : []), + ...valueFlag({ flag: "timeout", value: input.timeout }), + ...valueFlag({ flag: "input-delimiter", value: input.inputDelimiter }), + ...valueFlag({ flag: "output-delimiter", value: input.outputDelimiter }), ...booleanFlag({ value: input.stripAnsi, flag: "strip-ansi" }), ...(input.noLimit ? ["--no-limit"] : []), - ...(input.limit === undefined ? [] : ["--limit", String(input.limit)]), + ...valueFlag({ flag: "limit", value: input.limit }), ...(input.selectIfOne ? ["--select-if-one"] : []), ...booleanFlag({ value: input.strict, flag: "strict" }), + ...positionals(input.options), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { const out = stdout.trimEnd(); const parts = delimiter === "\n" ? out.split("\n") : out.split(delimiter); @@ -659,30 +694,25 @@ export async function gumFile({ const cmd = [ gum, "file", - ...(path ? [path] : []), - ...(cursor ? ["--cursor", cursor] : []), + ...valueFlag({ flag: "cursor", value: cursor }), ...(all ? ["--all"] : []), ...booleanFlag({ value: permissions, flag: "permissions" }), ...booleanFlag({ value: size, flag: "size" }), ...(file ? ["--file"] : []), ...(directory ? ["--directory"] : []), ...booleanFlag({ value: showHelp, flag: "show-help" }), - ...(timeout ? ["--timeout", timeout] : []), - ...(header ? ["--header", header] : []), - ...(height === undefined ? [] : ["--height", String(height)]), + ...valueFlag({ flag: "timeout", value: timeout }), + ...valueFlag({ flag: "header", value: header }), + ...valueFlag({ flag: "height", value: height }), + ...positionals(path ? [path] : []), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout.trimEnd() }; } @@ -713,20 +743,18 @@ export async function gumPager({ const cmd = [ gum, "pager", - ...(content ? [content] : []), ...(showLineNumbers ? ["--show-line-numbers"] : []), ...booleanFlag({ value: softWrap, flag: "soft-wrap" }), - ...(timeout ? ["--timeout", timeout] : []), + ...valueFlag({ flag: "timeout", value: timeout }), + ...positionals(content ? [content] : []), ]; - const proc = Bun.spawn(cmd, { + const { exitCode } = await gumRunner.run({ + cmd, env: process.env, stdin: "inherit", stdout: "inherit", - stderr: "inherit", }); - - const exitCode = await proc.exited; if (exitCode === 0) { return { ok: true, value: undefined }; } @@ -771,21 +799,20 @@ export async function gumSpin({ ...(showError ? ["--show-error"] : []), ...(showStdout ? ["--show-stdout"] : []), ...(showStderr ? ["--show-stderr"] : []), - ...(spinner ? ["--spinner", spinner] : []), - ...(title ? ["--title", title] : []), - ...(align ? ["--align", align] : []), - ...(timeout ? ["--timeout", timeout] : []), - ...cmd, + ...valueFlag({ flag: "spinner", value: spinner }), + ...valueFlag({ flag: "title", value: title }), + ...valueFlag({ flag: "align", value: align }), + ...valueFlag({ flag: "timeout", value: timeout }), + // gum consumes this marker; the wrapped command keeps its own flags. + ...positionals(cmd), ]; - const proc = Bun.spawn(gumCmd, { + const { exitCode } = await gumRunner.run({ + cmd: gumCmd, env: process.env, stdin: "inherit", stdout: "inherit", - stderr: "inherit", }); - - const exitCode = await proc.exited; if (exitCode === 0) { return { ok: true, value: 0 }; } @@ -844,37 +871,32 @@ export async function gumStyle({ const cmd = [ gum, "style", - ...text, ...(trim ? ["--trim"] : []), ...booleanFlag({ value: stripAnsi, flag: "strip-ansi" }), - ...(foreground ? ["--foreground", foreground] : []), - ...(background ? ["--background", background] : []), - ...(border ? ["--border", border] : []), - ...(borderBackground ? ["--border-background", borderBackground] : []), - ...(borderForeground ? ["--border-foreground", borderForeground] : []), - ...(align ? ["--align", align] : []), - ...(height === undefined ? [] : ["--height", String(height)]), - ...(width === undefined ? [] : ["--width", String(width)]), - ...(margin ? ["--margin", margin] : []), - ...(padding ? ["--padding", padding] : []), + ...valueFlag({ flag: "foreground", value: foreground }), + ...valueFlag({ flag: "background", value: background }), + ...valueFlag({ flag: "border", value: border }), + ...valueFlag({ flag: "border-background", value: borderBackground }), + ...valueFlag({ flag: "border-foreground", value: borderForeground }), + ...valueFlag({ flag: "align", value: align }), + ...valueFlag({ flag: "height", value: height }), + ...valueFlag({ flag: "width", value: width }), + ...valueFlag({ flag: "margin", value: margin }), + ...valueFlag({ flag: "padding", value: padding }), ...(bold ? ["--bold"] : []), ...(faint ? ["--faint"] : []), ...(italic ? ["--italic"] : []), ...(strikethrough ? ["--strikethrough"] : []), ...(underline ? ["--underline"] : []), + ...positionals(text), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: buildGumRenderEnv(), stdin: "ignore", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout }; } @@ -902,23 +924,18 @@ export async function gumJoin({ const cmd = [ gum, "join", - ...text, - ...(align ? ["--align", align] : []), + ...valueFlag({ flag: "align", value: align }), ...(horizontal ? ["--horizontal"] : []), ...(vertical ? ["--vertical"] : []), + ...positionals(text), ]; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: buildGumRenderEnv(), stdin: "ignore", stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout }; } @@ -950,25 +967,19 @@ export async function gumFormat({ const cmd = [ gum, "format", - ...(template ?? []), - ...(theme ? ["--theme", theme] : []), - ...(language ? ["--language", language] : []), + ...valueFlag({ flag: "theme", value: theme }), + ...valueFlag({ flag: "language", value: language }), ...booleanFlag({ value: stripAnsi, flag: "strip-ansi" }), - ...(type ? ["--type", type] : []), + ...valueFlag({ flag: "type", value: type }), + ...positionals(template ?? []), ]; - const stdin = input !== undefined ? streamFromText(input) : "inherit"; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: buildGumRenderEnv(), - stdin, + stdin: input === undefined ? "inherit" : { text: input }, stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout }; } @@ -1019,37 +1030,27 @@ export async function gumTable({ const cmd = [ gum, "table", - ...(separator ? ["--separator", separator] : []), - ...(columns ? ["--columns", columns.join(",")] : []), - ...(widths ? ["--widths", widths.join(",")] : []), - ...(height === undefined ? [] : ["--height", String(height)]), + ...valueFlag({ flag: "separator", value: separator }), + ...valueFlag({ flag: "columns", value: columns?.join(",") }), + ...valueFlag({ flag: "widths", value: widths?.join(",") }), + ...valueFlag({ flag: "height", value: height }), ...(print ? ["--print"] : []), - ...(file ? ["--file", file] : []), - ...(border ? ["--border", border] : []), + ...valueFlag({ flag: "file", value: file }), + ...valueFlag({ flag: "border", value: border }), ...booleanFlag({ value: showHelp, flag: "show-help" }), ...booleanFlag({ value: hideCount, flag: "hide-count" }), ...(lazyQuotes ? ["--lazy-quotes"] : []), - ...(fieldsPerRecord === undefined - ? [] - : ["--fields-per-record", String(fieldsPerRecord)]), - ...(returnColumn === undefined - ? [] - : ["--return-column", String(returnColumn)]), - ...(timeout ? ["--timeout", timeout] : []), + ...valueFlag({ flag: "fields-per-record", value: fieldsPerRecord }), + ...valueFlag({ flag: "return-column", value: returnColumn }), + ...valueFlag({ flag: "timeout", value: timeout }), ]; - const stdin = input !== undefined ? streamFromText(input) : "inherit"; - const proc = Bun.spawn(cmd, { + const { exitCode, stdout } = await gumRunner.run({ + cmd, env: buildGumRenderEnv(), - stdin, + stdin: input === undefined ? "inherit" : { text: input }, stdout: "pipe", - stderr: "inherit", }); - - const [stdout, exitCode] = await Promise.all([ - new Response(proc.stdout).text(), - proc.exited, - ]); if (exitCode === 0) { return { ok: true, value: stdout.trimEnd() }; } @@ -1071,14 +1072,12 @@ export async function gumVersionCheck({ return { ok: false, reason: "unavailable" }; } - const proc = Bun.spawn([gum, "version-check", constraint], { + const { exitCode } = await gumRunner.run({ + cmd: [gum, "version-check", ...positionals([constraint])], env: process.env, stdin: "ignore", stdout: "ignore", - stderr: "inherit", }); - - const exitCode = await proc.exited; if (exitCode === 0) { return { ok: true, value: true }; } @@ -1138,7 +1137,46 @@ function buildSelectedArg( return []; } if (selected === "*") { - return ["--selected", "*"]; + return ["--selected=*"]; + } + return valueFlag({ flag: "selected", value: selected.join(",") }); +} + +/** + * gum (kong) ends option parsing at this token; every later argument is a + * positional, even when it begins with "-" or "--". + */ +export const GUM_END_OF_OPTIONS = "--"; + +/** + * Builds a valued flag in single-token `--flag=value` form. + * + * Invariant: free text must never be parsed as a gum option. In the + * two-token form kong rejects a value that begins with "-" (it reads it as a + * flag); the joined form always binds the value to the flag. Empty strings and + * `undefined` omit the flag, matching the previous truthiness checks. + */ +function valueFlag({ + flag, + value, +}: { + readonly flag: string; + readonly value: string | number | undefined; +}): readonly string[] { + if (value === undefined || value === "") { + return []; + } + return [`--${flag}=${String(value)}`]; +} + +/** + * Places positional text after the end-of-options marker. Callers must spread + * this last, after every flag, so text such as "--value-stdin requires ..." + * is printed instead of being parsed as an unknown flag. + */ +function positionals(args: readonly string[]): readonly string[] { + if (args.length === 0) { + return []; } - return ["--selected", selected.join(",")]; + return [GUM_END_OF_OPTIONS, ...args]; } diff --git a/tests/fixtures/gum b/tests/fixtures/gum index fadf0cf46..6c790c9a8 100755 --- a/tests/fixtures/gum +++ b/tests/fixtures/gum @@ -58,6 +58,9 @@ case "$sub" in exit "${GUM_FIXTURE_CONFIRM_EXIT:-0}" ;; version-check) + if [[ "${1:-}" == "--" ]]; then + shift + fi if [[ "${1:-}" == "fail" ]]; then exit 1 fi diff --git a/tests/gum-end-of-options.test.ts b/tests/gum-end-of-options.test.ts new file mode 100644 index 000000000..24d5cc2ab --- /dev/null +++ b/tests/gum-end-of-options.test.ts @@ -0,0 +1,301 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; + +import { + GUM_END_OF_OPTIONS, + type GumRunRequest, + gumChooseMany, + gumConfirm, + gumFile, + gumFilterMany, + gumFormat, + gumInput, + gumJoin, + gumPager, + gumSpin, + gumStyle, + gumTable, + gumVersionCheck, + gumWrite, + resetGumPathCacheForTests, + setGumRunnerForTests, + tryGumLog, +} from "../src/ui/gum.ts"; +import { logger, setLoggerBackendOverride } from "../src/ui/logger.ts"; + +// Never spawned: every invocation goes through the recording runner below. +const FAKE_GUM = "/nonexistent/hack-test/gum"; +const DASH_TEXT = "--value-stdin requires --env"; +const SHORT_DASH_TEXT = "-n is not a number"; + +let calls: GumRunRequest[] = []; +let previousGumPath: string | undefined; + +beforeEach(() => { + calls = []; + previousGumPath = process.env.HACK_GUM_PATH; + process.env.HACK_GUM_PATH = FAKE_GUM; + resetGumPathCacheForTests(); + setGumRunnerForTests({ + runner: { + runSync(request) { + calls.push(request); + return 0; + }, + run(request) { + calls.push(request); + return Promise.resolve({ exitCode: 0, stdout: "picked\n" }); + }, + }, + }); +}); + +afterEach(() => { + setGumRunnerForTests({ runner: null }); + setLoggerBackendOverride({ backend: null }); + if (previousGumPath === undefined) { + Reflect.deleteProperty(process.env, "HACK_GUM_PATH"); + } else { + process.env.HACK_GUM_PATH = previousGumPath; + } + resetGumPathCacheForTests(); +}); + +function onlyCall(): readonly string[] { + expect(calls).toHaveLength(1); + const call = calls[0]; + if (!call) { + throw new Error("expected one gum invocation"); + } + return call.cmd; +} + +/** + * Asserts every listed text is a positional after the end-of-options marker + * and that nothing before the marker is free text (only `--flag[=value]`). + */ +function expectTextAfterMarker({ + cmd, + subcommand, + text, +}: { + readonly cmd: readonly string[]; + readonly subcommand: string; + readonly text: readonly string[]; +}): void { + expect(cmd.slice(0, 2)).toEqual([FAKE_GUM, subcommand]); + const marker = cmd.indexOf(GUM_END_OF_OPTIONS); + expect(marker).toBeGreaterThan(1); + expect(cmd.slice(marker + 1)).toEqual([...text]); + for (const arg of cmd.slice(2, marker)) { + expect(arg.startsWith("--")).toBe(true); + } +} + +test("tryGumLog puts a dash-leading message and fields after the marker", () => { + expect( + tryGumLog({ + level: "warn", + message: DASH_TEXT, + fields: { flag: "--force", b: SHORT_DASH_TEXT }, + }) + ).toBe(true); + + const cmd = onlyCall(); + expect(cmd).toEqual([ + FAKE_GUM, + "log", + "--level=warn", + "--structured", + GUM_END_OF_OPTIONS, + DASH_TEXT, + "b", + SHORT_DASH_TEXT, + "flag", + "--force", + ]); + expect(calls[0]?.stdin).toBe("ignore"); +}); + +test("tryGumLog without fields still ends options before the message", () => { + tryGumLog({ level: "info", message: DASH_TEXT }); + expect(onlyCall()).toEqual([ + FAKE_GUM, + "log", + "--level=info", + GUM_END_OF_OPTIONS, + DASH_TEXT, + ]); +}); + +test("logger selected as gum passes a dash-leading message after the marker", () => { + setLoggerBackendOverride({ backend: "gum" }); + logger.error({ message: DASH_TEXT }); + logger.success({ message: SHORT_DASH_TEXT }); + + expect(calls).toHaveLength(2); + expect(calls[0]?.cmd).toEqual([ + FAKE_GUM, + "log", + "--level=error", + GUM_END_OF_OPTIONS, + DASH_TEXT, + ]); + expect(calls[1]?.cmd).toEqual([ + FAKE_GUM, + "log", + "--level=info", + "--structured", + GUM_END_OF_OPTIONS, + SHORT_DASH_TEXT, + "status", + "success", + ]); +}); + +test("gumStyle and gumJoin pass text blocks after the marker", async () => { + await gumStyle({ text: [DASH_TEXT, "-"], bold: true, padding: "0 1" }); + expectTextAfterMarker({ + cmd: onlyCall(), + subcommand: "style", + text: [DASH_TEXT, "-"], + }); + expect(calls[0]?.cmd).toContain("--padding=0 1"); + + calls = []; + await gumJoin({ text: [SHORT_DASH_TEXT, DASH_TEXT], horizontal: true }); + expectTextAfterMarker({ + cmd: onlyCall(), + subcommand: "join", + text: [SHORT_DASH_TEXT, DASH_TEXT], + }); +}); + +test("gumConfirm passes the prompt after the marker and binds button labels", async () => { + const res = await gumConfirm({ + prompt: DASH_TEXT, + affirmative: "--yes", + negative: "-no", + timeout: "5s", + }); + expect(res).toEqual({ ok: true, value: true }); + + const cmd = onlyCall(); + expectTextAfterMarker({ cmd, subcommand: "confirm", text: [DASH_TEXT] }); + expect(cmd).toContain("--affirmative=--yes"); + expect(cmd).toContain("--negative=-no"); +}); + +test("gumConfirm without a prompt adds no marker", async () => { + await gumConfirm({ default: true }); + expect(onlyCall()).toEqual([FAKE_GUM, "confirm", "--default"]); +}); + +test("gumChooseMany and gumFilterMany pass options after the marker", async () => { + const chosen = await gumChooseMany({ + options: [DASH_TEXT, "-a"], + header: "--pick one", + selected: [DASH_TEXT], + }); + expect(chosen).toEqual({ ok: true, value: ["picked"] }); + let cmd = onlyCall(); + expectTextAfterMarker({ cmd, subcommand: "choose", text: [DASH_TEXT, "-a"] }); + expect(cmd).toContain("--header=--pick one"); + expect(cmd).toContain(`--selected=${DASH_TEXT}`); + + calls = []; + await gumFilterMany({ + options: ["-x", DASH_TEXT], + placeholder: "-search", + prompt: "--> ", + value: "-", + }); + cmd = onlyCall(); + expectTextAfterMarker({ cmd, subcommand: "filter", text: ["-x", DASH_TEXT] }); + expect(cmd).toContain("--placeholder=-search"); + expect(cmd).toContain("--prompt=--> "); + expect(cmd).toContain("--value=-"); +}); + +test("text prompts bind dash-leading values to their flags", async () => { + await gumInput({ header: DASH_TEXT, placeholder: "-p", value: "--v" }); + let cmd = onlyCall(); + expect(cmd.slice(0, 2)).toEqual([FAKE_GUM, "input"]); + expect(cmd).toEqual([ + FAKE_GUM, + "input", + "--placeholder=-p", + `--header=${DASH_TEXT}`, + "--value=--v", + ]); + + calls = []; + await gumWrite({ header: DASH_TEXT, prompt: "-> ", width: 40 }); + cmd = onlyCall(); + expect(cmd).toEqual([ + FAKE_GUM, + "write", + "--width=40", + `--header=${DASH_TEXT}`, + "--prompt=-> ", + ]); +}); + +test("gumFile, gumPager and gumFormat pass content after the marker", async () => { + await gumFile({ path: "-dir", header: "--h" }); + expectTextAfterMarker({ + cmd: onlyCall(), + subcommand: "file", + text: ["-dir"], + }); + + calls = []; + await gumPager({ content: DASH_TEXT, softWrap: true }); + expectTextAfterMarker({ + cmd: onlyCall(), + subcommand: "pager", + text: [DASH_TEXT], + }); + + calls = []; + await gumFormat({ template: [DASH_TEXT], type: "template" }); + expectTextAfterMarker({ + cmd: onlyCall(), + subcommand: "format", + text: [DASH_TEXT], + }); + + calls = []; + await gumFormat({ input: DASH_TEXT, type: "markdown" }); + expect(onlyCall()).toEqual([FAKE_GUM, "format", "--type=markdown"]); + expect(calls[0]?.stdin).toEqual({ text: DASH_TEXT }); +}); + +test("gumSpin ends gum options before the wrapped command", async () => { + await gumSpin({ cmd: ["bun", "--version"], title: "-loading" }); + const cmd = onlyCall(); + expectTextAfterMarker({ + cmd, + subcommand: "spin", + text: ["bun", "--version"], + }); + expect(cmd).toContain("--title=-loading"); +}); + +test("gumTable feeds rows on stdin and binds flag values", async () => { + await gumTable({ + input: `${DASH_TEXT}\n`, + columns: ["-a", "b"], + print: true, + }); + expect(onlyCall()).toEqual([FAKE_GUM, "table", "--columns=-a,b", "--print"]); + expect(calls[0]?.stdin).toEqual({ text: `${DASH_TEXT}\n` }); +}); + +test("gumVersionCheck passes the constraint after the marker", async () => { + await gumVersionCheck({ constraint: "-1" }); + expectTextAfterMarker({ + cmd: onlyCall(), + subcommand: "version-check", + text: ["-1"], + }); +}); diff --git a/tests/gum.test.ts b/tests/gum.test.ts index 162e501ea..02b9c6d76 100644 --- a/tests/gum.test.ts +++ b/tests/gum.test.ts @@ -104,5 +104,7 @@ test("tryGumLog returns true and writes structured args (fixture)", async () => expect(ok).toBe(true); const text = await Bun.file(logFile).text(); - expect(text).toContain("gum log --level info --structured hello a 1 ok true"); + expect(text).toContain( + "gum log --level=info --structured -- hello a 1 ok true" + ); });