Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ improvising or retrying when something is off.
| Teach over MCP: an agent does a new task once through OpenAdapt; the run is read back, compiled and admitted (a person confirms the record check; the agent admits only under a person-signed policy), and becomes a tool it calls with no model calls. Web, and Windows, macOS, Linux, RDP and Citrix with a person approving each save | On main ([#1335](https://github.com/OpenAdaptAI/OpenAdapt/pull/1335)) | Synthetic (MockMed over MCP, and through the real RDP and Citrix backends) | `openadapt-agent serve --teach`, [packages/agent](packages/agent/README.md#teach-a-new-workflow-from-your-agent) |
| Guard an agent you already run: attach to its own Chromium over CDP (one added line), hold each save until the record on screen is the one the task names, block it and tell the agent why when it isn't, read the save back; a run that reads back correctly becomes a draft program with a record check from scaffold-verifier, which a person confirms and admits, then a `taught_*` tool with no model calls. `--observe-only` never blocks | On main ([#1357](https://github.com/OpenAdaptAI/OpenAdapt/pull/1357), stacked on [#1335](https://github.com/OpenAdaptAI/OpenAdapt/pull/1335)) | Synthetic (MockMed, scripted Playwright agent, `effect_e2e` faults) | `openadapt-flow guard`, [GUARD_AN_AGENT.md](packages/flow/docs/GUARD_AN_AGENT.md) |
| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | On main ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](packages/agent/README.md#track-a-long-job) |
| Limit a teaching agent to the declared task: typed text must be a declared value, links and browser requests to other origins are refused, writes are checked before they are sent, and the record is read back before a draft exists. Injected instructions in record notes, banners, tooltips, alt text and server responses are blocked or caught on MockMed (scripted agent; a live-model mode is off by default). Self-repair refuses a control that leaves the app | On main ([#1359](https://github.com/OpenAdaptAI/OpenAdapt/pull/1359)) | Synthetic | `openadapt_flow.teach_scope`, [THREAT_MODEL.md](packages/flow/docs/THREAT_MODEL.md) |
| Limit a teaching agent to the declared task on every teach path (`compile_agent`, `compile-agent`, teach over MCP, the airbag guard): typed text must be a declared value, links and browser requests to other origins are refused, writes are checked before they are sent, and the record is read back before a draft exists. Injected instructions in record notes, banners, tooltips, alt text and server responses are blocked or caught on MockMed through each path (scripted agent; a live-model mode is off by default). Self-repair refuses a control that leaves the app | On main ([#1359](https://github.com/OpenAdaptAI/OpenAdapt/pull/1359), [#1386](https://github.com/OpenAdaptAI/OpenAdapt/pull/1386)) | Synthetic | `openadapt_flow.teach_scope`, [THREAT_MODEL.md](packages/flow/docs/THREAT_MODEL.md) |
| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | On main ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` |
| Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` |
| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | On main ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` |
Expand Down Expand Up @@ -424,11 +424,6 @@ A test fails when a command has no file or this list is out of date.
OpenEMR stops none reached repair (each came after "Create New Patient",
which counts as a save). On openIMIS, with #1337 and a risk review, the
relabelled Save reached repair 3 of 3, but no fix passed the probe.
- Prompt-injection limits (the teach scope) are on `openadapt_flow.jit` only.
`compile-agent`, teach over MCP and the airbag guard do not pass a scope. No
live model has been run against the injection fixtures. A steered agent can still read other
records in the same app. See
[THREAT_MODEL.md](packages/flow/docs/THREAT_MODEL.md).
- Windows installers are unsigned, and macOS builds are ad-hoc signed.
- Several capabilities have run only on OpenAdapt's synthetic apps. The
Evidence column says which.
26 changes: 13 additions & 13 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -629,7 +629,7 @@ improvising or retrying when something is off.
| Teach over MCP: an agent does a new task once through OpenAdapt; the run is read back, compiled and admitted (a person confirms the record check; the agent admits only under a person-signed policy), and becomes a tool it calls with no model calls. Web, and Windows, macOS, Linux, RDP and Citrix with a person approving each save | On main ([#1335](https://github.com/OpenAdaptAI/OpenAdapt/pull/1335)) | Synthetic (MockMed over MCP, and through the real RDP and Citrix backends) | `openadapt-agent serve --teach`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#teach-a-new-workflow-from-your-agent) |
| Guard an agent you already run: attach to its own Chromium over CDP (one added line), hold each save until the record on screen is the one the task names, block it and tell the agent why when it isn't, read the save back; a run that reads back correctly becomes a draft program with a record check from scaffold-verifier, which a person confirms and admits, then a `taught_*` tool with no model calls. `--observe-only` never blocks | On main ([#1357](https://github.com/OpenAdaptAI/OpenAdapt/pull/1357), stacked on [#1335](https://github.com/OpenAdaptAI/OpenAdapt/pull/1335)) | Synthetic (MockMed, scripted Playwright agent, `effect_e2e` faults) | `openadapt-flow guard`, [GUARD_AN_AGENT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/GUARD_AN_AGENT.md) |
| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | On main ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) |
| Limit a teaching agent to the declared task: typed text must be a declared value, links and browser requests to other origins are refused, writes are checked before they are sent, and the record is read back before a draft exists. Injected instructions in record notes, banners, tooltips, alt text and server responses are blocked or caught on MockMed (scripted agent; a live-model mode is off by default). Self-repair refuses a control that leaves the app | On main ([#1359](https://github.com/OpenAdaptAI/OpenAdapt/pull/1359)) | Synthetic | `openadapt_flow.teach_scope`, [THREAT_MODEL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THREAT_MODEL.md) |
| Limit a teaching agent to the declared task on every teach path (`compile_agent`, `compile-agent`, teach over MCP, the airbag guard): typed text must be a declared value, links and browser requests to other origins are refused, writes are checked before they are sent, and the record is read back before a draft exists. Injected instructions in record notes, banners, tooltips, alt text and server responses are blocked or caught on MockMed through each path (scripted agent; a live-model mode is off by default). Self-repair refuses a control that leaves the app | On main ([#1359](https://github.com/OpenAdaptAI/OpenAdapt/pull/1359), [#1386](https://github.com/OpenAdaptAI/OpenAdapt/pull/1386)) | Synthetic | `openadapt_flow.teach_scope`, [THREAT_MODEL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THREAT_MODEL.md) |
| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | On main ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` |
| Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` |
| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | On main ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` |
Expand Down Expand Up @@ -820,11 +820,6 @@ A test fails when a command has no file or this list is out of date.
OpenEMR stops none reached repair (each came after "Create New Patient",
which counts as a save). On openIMIS, with #1337 and a risk review, the
relabelled Save reached repair 3 of 3, but no fix passed the probe.
- Prompt-injection limits (the teach scope) are on `openadapt_flow.jit` only.
`compile-agent`, teach over MCP and the airbag guard do not pass a scope. No
live model has been run against the injection fixtures. A steered agent can still read other
records in the same app. See
[THREAT_MODEL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THREAT_MODEL.md).
- Windows installers are unsigned, and macOS builds are ad-hoc signed.
- Several capabilities have run only on OpenAdapt's synthetic apps. The
Evidence column says which.
Expand Down Expand Up @@ -1809,25 +1804,30 @@ rely on the agent ignoring it. The full analysis is in

What is enforced:

- With a declared teach scope (`compile_agent(..., teach_scope=...)`), the
agent can type only declared values, can't follow a link or submit a form to
- Every teach path runs under a teach scope declared before the agent reads
the app: `compile_agent`, `compile-agent`, teach over MCP and the airbag
guard. The agent can type only declared values, can't follow a link or submit a form to
an undeclared origin, and can't make the browser reach an undeclared origin.
With a write adapter for the app, a write to another record, an undeclared
field or an undeclared value is aborted before it is sent. After the run,
Flow reads the system of record and refuses the draft unless it shows
exactly the declared write. On MockMed, every injected instruction in the
test suite (5 places by 4 goals) is blocked or caught this way, with a
scripted agent that obeys all of them.
test suite (5 places by 4 goals, and each goal through each path) is
blocked or caught this way, with a scripted agent that obeys all of them.
The airbag guard can't refuse keystrokes in the agent's own browser; it
blocks requests and saves instead.
- Self-repair refuses a proposed control that leads to another origin.
- A save approval shows who asked, the agent's own reason (as a quote), the
screen text it saw, and any of that text that reads like an instruction to
an agent.

What is not:

- The scope is wired into `openadapt_flow.jit` only. `compile-agent` from a
prompt (on main) does not pass one yet; teach over MCP and the airbag guard
are separate pull requests that must pass a scope when they land.
- Over MCP without `--teach-tasks`, the agent declares the record at
`teach_start`; after reading the app, a hijacked agent can start again with
another record. The person confirming the record check sees it.
- The airbag guard sees requests, not keystrokes; without a write adapter, a
save's values are judged by the read-back after it lands.
- A self-repair bridge may type text that the screen shows, so injected text
can become a typed value in a proposed repair. The person approving sees it.
- A steered agent can still read other records in the same app and send what
Expand Down
3 changes: 3 additions & 0 deletions packages/agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,9 @@ It works on a web app (`--url`) and on the app a deployment file's `backend:` se
| `--teach-key PEM`, `--teach-policy JSON` | The registered agent's encrypted key (passphrase in `OPENADAPT_JIT_KEY_PASSPHRASE`) and the signed admission policy. Only an agent key under a policy; an admitter key admits from CI with `openadapt-flow jit admit`. |
| `--key-dir DIR` | The JIT key directory (default `OPENADAPT_JIT_KEY_DIR` or `~/.openadapt/jit`). |
| `--teach-hold` | On the web, a person also approves every save in the first run. Without it, your agent names the identity the save gate checks (`on_screen`); a value every form shows proves nothing, and then only the record check after the run and the person confirming it catch a wrong record. |
| `--teach-tasks DIR` | Run 1's record and inputs come from the operator's task files (`openadapt.guard-task/v1`); `teach_start` names one with `task_file`. Without it, your agent declares them at `teach_start`, before it reads the app. |
| `--teach-allow-origin URL` | Another origin run 1's browser may reach. Every origin but the app's is refused otherwise. |
| `--teach-write-adapter MODULE:ATTR` | The app's write adapter: each write in run 1 is read before it is sent, and one to another record, field or value is refused. |
| `--teach-screens scrubbed` | Adds `teach_screen`: the app's screen with identifiers scrubbed, and nothing shared if scrubbing fails. `unscrubbed` skips scrubbing. Off by default. |
| `--mode production --bundles DIR` | Serves your other workflows next to the taught ones. |

Expand Down
11 changes: 10 additions & 1 deletion packages/flow/docs/COMPILE_AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,14 @@ no key. `--agent steps.json` runs a scripted agent with no model calls.

1. **Run 1, in the harness.** The agent does the task while
`openadapt_flow.jit` records every action, exactly as for any other JIT
capture ([JIT.md](JIT.md)).
capture ([JIT.md](JIT.md)). Run 1 runs under the task's teach scope
([THREAT_MODEL.md](THREAT_MODEL.md)), built from what the command
declares before the agent reads a page: the agent may type only the
inputs, the `--expect` values, the `--on-screen` text and `--allow-text`;
its browser reaches only the app's origin and `--allow-origin`; with
`--write-adapter`, each write is read before it is sent; and the system of
record is read back after the run. Anything outside it ends run 1 with no
draft.
2. **The record check.** The system of record is read when run 1 starts and
after it ends. Run 1 passes only when exactly one record appeared, no other
record changed or disappeared, the new record has every `--expect` field
Expand Down Expand Up @@ -254,6 +261,8 @@ object with `messages.create`; the tests pass a fake.
| `refused_by_hold` | The hold said no; nothing was saved. |
| `record_check_failed` | The system of record did not show exactly the record the task named. |
| `canary_failed` | A canary did not end VERIFIED, or its receipt did not. |
| `scope_refused` | Run 1 went outside the task's teach scope (typed text, a link or request to another origin, an undeclared write); nothing past that point was done, and there is no draft. |
| `readback_refused` | After run 1, the system of record did not show exactly the declared write. |
| `capture_failed`, `unsupported` | The agent failed or used an action the recording can't hold. |

## Files
Expand Down
26 changes: 22 additions & 4 deletions packages/flow/docs/GUARD_AN_AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ Nothing leaves your computer, and the guard makes no model calls.
not_saves: # optional: writes that aren't the save
- {method: POST, path: /chart/signin}
approval: not_gated # each_item: a person approves each save
allow_origins: [] # optional: other origins the tab may reach
write_adapter: openadapt_flow.mockmed.teach_scope:EncounterWrites # optional
allow_text: [Triage] # optional: values a save may carry besides the task's
teach: # optional: make run 2 a program
dir: .openadapt-teach
name: triage_note
Expand Down Expand Up @@ -76,17 +79,28 @@ Nothing leaves your computer, and the guard makes no model calls.

## What the guard does with each request

The guard pauses the tab's requests with the CDP `Fetch` domain: documents
(form posts), `fetch`, XHR, beacons and others. Images, event streams,
scripts, styles, fonts and media aren't paused.
The guard pauses every request of the tab with the CDP `Fetch` domain:
documents (form posts and navigations), `fetch`, XHR, beacons, images,
scripts, styles, fonts and media.

The task's teach scope ([THREAT_MODEL.md](THREAT_MODEL.md)) decides first.
The tab may reach the app's origin, the origins in `allow_origins` and the
origins of `not_saves` rules, and nothing else, so text on the page that
steers the agent to another site, or an image beacon the page loads by
itself, is blocked. With `write_adapter`, the guard also reads each save
before the identity check: a save to another record, or with a field value
that is neither the task's nor in `allow_text`, is blocked.

| Request | What happens |
|---|---|
| Any request to an origin outside the scope | Blocked (`egress_origin_not_declared`). |
| `GET`, `HEAD`, `OPTIONS`, a GraphQL `query` | Goes on at once. |
| A write listed in `not_saves` | Goes on. |
| `POST`, `PUT`, `PATCH`, `DELETE`, another method, or a GraphQL `mutation` to the app's origin | Held. This is a save. |
| `POST`, `PUT`, `PATCH`, `DELETE`, another method, or a GraphQL `mutation` to the app's origin | Held. This is a save. With `write_adapter`, a save the task doesn't declare is blocked (`write_wrong_record`, `write_field_not_declared`, `write_value_not_declared`, `write_unattributed`). |
| A write to another origin | Blocked, unless it is in `not_saves`. |

A run in which anything went outside the scope is never handed to run 2.

A GraphQL body that Flow can't read, such as a persisted query with no text,
counts as a save.

Expand Down Expand Up @@ -233,6 +247,10 @@ GUI steps, and the run-2 times are GUI times.
- One tab: the tab on the app's origin. A save from another tab or a popup
isn't guarded.
- Saves sent over a WebSocket aren't seen. CDP `Fetch` doesn't pause them.
- The guard sees requests, not keystrokes: the agent drives its own browser,
so the guard can't refuse typed text the way the compile-agent harness
does. Without `write_adapter`, a save's values are judged by the read-back
after it lands, not before.
- The record check is text on the screen. If the screen shows the expected
values for two records (a list), the check can pass on the wrong row. Set
`screen_selector` to the element that shows the open record.
Expand Down
Loading
Loading