From b0c1016b016a75f578410b3504425763ce62363e Mon Sep 17 00:00:00 2001 From: abrichr Date: Sun, 18 Oct 2026 17:08:09 -0400 Subject: [PATCH 1/4] docs(capabilities): code as action for the run-1 agent Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index fcdfd4d28..5556e511c 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -19,7 +19,8 @@ being merged now. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles the verified run into a program (`openadapt_flow.jit`, released; - prompt-to-program `compile-agent`, landing). Agents can drive the recording + prompt-to-program `compile-agent`, landing; the agent can act through code + instead of screenshots, [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)). Agents can drive the recording on web, native Windows, RDP and Citrix. 2. **Compiles to the fastest layer the app allows.** Page structure first (DOM, Windows UI Automation, macOS Accessibility), direct API calls where the app's @@ -80,6 +81,7 @@ being merged now. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](packages/flow/docs/JIT.md) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. On the local OpenEMR referral it was correct 8 of 8 at $0.055 a run, against 6 of 8 at $0.33 for the screenshot loop. | [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](README.md) | @@ -187,6 +189,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](packages/flow/docs/JIT.md) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](packages/agent/README.md) | | Live agent loop end to end on a real app, plus a launch demo (agent once, OpenAdapt 100 times) | In progress | Untested | none yet | From eabaa175bfcfa647003c0c683f4c974dba2a7930 Mon Sep 17 00:00:00 2001 From: abrichr Date: Sun, 18 Oct 2026 17:50:08 -0400 Subject: [PATCH 2/4] docs(capabilities): quote arm 7 from the scoreboard Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 5556e511c..1dc74ce6b 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -81,7 +81,7 @@ being merged now. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](packages/flow/docs/JIT.md) | -| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. On the local OpenEMR referral it was correct 8 of 8 at $0.055 a run, against 6 of 8 at $0.33 for the screenshot loop. | [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. Scoreboard arm 7: the local OpenEMR referral 4 of 5 correct (the fifth a safe stop) at $0.041 a run, against $0.10 for the screenshot agent's first run (arm 2). | [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](README.md) | From 846d696c72852f5791d925161f4d3025dfe7f3c7 Mon Sep 17 00:00:00 2001 From: abrichr Date: Sun, 18 Oct 2026 17:52:02 -0400 Subject: [PATCH 3/4] docs(capabilities): quote arm 7 only once the scoreboard has it Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 1dc74ce6b..d483245ef 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -81,7 +81,7 @@ being merged now. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](packages/flow/docs/JIT.md) | -| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. Scoreboard arm 7: the local OpenEMR referral 4 of 5 correct (the fifth a safe stop) at $0.041 a run, against $0.10 for the screenshot agent's first run (arm 2). | [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](README.md) | From 2f115f5443e25852eff24d041423e44cef1276a0 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 08:32:46 -0400 Subject: [PATCH 4/4] docs(capabilities): link #1305 instead of a file only on that branch Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index d483245ef..d10533783 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -81,7 +81,7 @@ being merged now. | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](packages/flow/docs/JIT.md) | -| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](README.md) | @@ -189,7 +189,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](packages/flow/docs/JIT.md) | -| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, [CODE_AS_ACTION.md](packages/flow/docs/CODE_AS_ACTION.md) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](packages/agent/README.md) | | Live agent loop end to end on a real app, plus a launch demo (agent once, OpenAdapt 100 times) | In progress | Untested | none yet |