From e25c4476076ec909a710b3c4e3a46a50464e219f Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 09:53:11 -0400 Subject: [PATCH 1/5] docs: an agent's claims versus the read-back, 30 of 30 bad saves Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 7 +++++++ README.md | 10 ++++++++++ 2 files changed, 17 insertions(+) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 8259b43fe..4a2700804 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -246,6 +246,13 @@ fails when a command is added without being listed here. complete database read path caught all 72. This is a coverage test over a small, hand-authored set of faults, not a real-world error rate: [EFFECT_E2E.md](packages/flow/benchmark/effect_e2e/EFFECT_E2E.md). +- **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, + working alone on a test app whose server broke the save in 6 ways (5 items + each), reported all 30 bad saves as done. `verify-run` flagged all 30 and + confirmed all 10 clean saves, with no false alarms. The app shows "saved" + before the server answers, so this measures what the screen can't show, not + the agent's care. Not yet run on OpenEMR: + [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). ## What is not done yet diff --git a/README.md b/README.md index d9f005687..373844aad 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,16 @@ This is a coverage test over a small, hand-written set of faults, not a real-world error rate. [EFFECT_E2E.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md) has the method and raw results. +An agent working alone can't see these faults either. Claude Sonnet 5.5, using +computer use, did a task on a test app whose server broke the save in 6 ways, +5 times each. The agent reported all 30 bad saves as done. Given the agent's +run, `openadapt-flow verify-run` read each record back: it flagged all 30 and +confirmed all 10 clean saves, with no false alarms. The test app shows "saved" +before the server answers, so the screen gave the agent nothing to notice. That +gap is what the read-back closes. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back) +has the method, the raw runs and the cost. + ## Try it in one command ```bash From 2bdbd2a60f6de060308c8d5dc89d62cdba8b8980 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 13:33:33 -0400 Subject: [PATCH 2/5] docs: README leads with the measured OpenEMR times and the 30 of 30 claims result Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 9 +++ README.md | 26 +++++++++ llms-full.txt | 148 ++++++++++++++++++++++++++++++++++++++++-------- 3 files changed, 160 insertions(+), 23 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index d6e720877..c10f3e04f 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -258,6 +258,15 @@ fails when a command is added without being listed here. complete database read path caught all 72. This is a coverage test over a small, hand-authored set of faults, not a real-world error rate: [EFFECT_E2E.md](packages/flow/benchmark/effect_e2e/EFFECT_E2E.md). +- **OpenEMR referral: agent, screen program and API step.** Median wall time + per referral, from a fresh browser through sign-in, the entry and the read-back: + agent alone 36.5 s (62 of 63 correct, the 14 to 15 Oct study); compiled program + on the screen 22.5 s (18 of 18); the same program with the API step 3.3 s + (18 of 18). The program rows ran on 19 Oct, interleaved on one copy, without + the approval gate or resumable runs, and the API step was admitted by the + benchmark harness. The approval-gated production profile still runs the + screen steps; making the API step work under it is in progress: + [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). - **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, working alone on a test app whose server broke the save in 6 ways (5 items each), reported all 30 bad saves as done. `verify-run` flagged all 30 and diff --git a/README.md b/README.md index 7b7c7f25c..2faaba9cc 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,32 @@ each save, reads the save back from the system of record after it, and returns a plain outcome. When anything doesn't match, it stops and tells your agent instead of guessing. +## Measured on OpenEMR + +One new-patient referral in a test copy of OpenEMR, timed from a fresh +browser through sign-in, the entry and the saved record read back (median): + +| Who does the entry | Time per referral | Saved correctly | Model cost | +|---|---|---|---| +| Agent alone (Claude Sonnet 5.5 computer use) | `██████████████████` 36.5 s | 62 of 63 | about $0.08 | +| OpenAdapt program, screen steps | `███████████` 22.5 s | 18 of 18 | $0 | +| OpenAdapt program, API step | `██` 3.3 s | 18 of 18 | $0 | + +The API step replaces the program's 41 screen steps with the one save request +the app itself sends, then reads the record back. In both program rows a +possible-duplicate referral stopped without saving, as it should. Both program +rows ran on 19 Oct on one copy of the app, interleaved, without the approval +gate and without resumable runs; the API step was admitted by the benchmark +harness, not by a person. The agent rows are the 14 to 15 Oct timing study, on +its own copy of the app. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step) +has every run. + +A screen can't show a save that went wrong behind it. On a test app that +shows a save as done before its server answers, with the server breaking the +save in 6 ways, 5 times each, the agent reported all 30 bad saves as done. Reading each record back caught all 30, with no false +alarms on 10 clean saves ([details below](#why-the-read-back-matters)). + ## One harness, three modes | Mode | What happens | Model calls | diff --git a/llms-full.txt b/llms-full.txt index 623dfd5b6..b176a0a78 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -31,6 +31,32 @@ each save, reads the save back from the system of record after it, and returns a plain outcome. When anything doesn't match, it stops and tells your agent instead of guessing. +## Measured on OpenEMR + +One new-patient referral in a test copy of OpenEMR, timed from a fresh +browser through sign-in, the entry and the saved record read back (median): + +| Who does the entry | Time per referral | Saved correctly | Model cost | +|---|---|---|---| +| Agent alone (Claude Sonnet 5.5 computer use) | `██████████████████` 36.5 s | 62 of 63 | about $0.08 | +| OpenAdapt program, screen steps | `███████████` 22.5 s | 18 of 18 | $0 | +| OpenAdapt program, API step | `██` 3.3 s | 18 of 18 | $0 | + +The API step replaces the program's 41 screen steps with the one save request +the app itself sends, then reads the record back. In both program rows a +possible-duplicate referral stopped without saving, as it should. Both program +rows ran on 19 Oct on one copy of the app, interleaved, without the approval +gate and without resumable runs; the API step was admitted by the benchmark +harness, not by a person. The agent rows are the 14 to 15 Oct timing study, on +its own copy of the app. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step) +has every run. + +A screen can't show a save that went wrong behind it. On a test app that +shows a save as done before its server answers, with the server breaking the +save in 6 ways, 5 times each, the agent reported all 30 bad saves as done. Reading each record back caught all 30, with no false +alarms on 10 clean saves ([details below](#why-the-read-back-matters)). + ## One harness, three modes | Mode | What happens | Model calls | @@ -56,6 +82,16 @@ This is a coverage test over a small, hand-written set of faults, not a real-world error rate. [EFFECT_E2E.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md) has the method and raw results. +An agent working alone can't see these faults either. Claude Sonnet 5.5, using +computer use, did a task on a test app whose server broke the save in 6 ways, +5 times each. The agent reported all 30 bad saves as done. Given the agent's +run, `openadapt-flow verify-run` read each record back: it flagged all 30 and +confirmed all 10 clean saves, with no false alarms. The test app shows "saved" +before the server answers, so the screen gave the agent nothing to notice. That +gap is what the read-back closes. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back) +has the method, the raw runs and the cost. + ## Try it in one command ```bash @@ -263,6 +299,7 @@ including the few still being merged. | Stopped before saving | `HALTED_BEFORE_EFFECT` | `needs_review` | Fix the cause, or show the program the fix, and run again. | | Check the record | `RECONCILIATION_REQUIRED` | `not_sure_if_saved` | A save may have gone through. A person checks the record before anything is retried. | | Finished, not checked | `COMPLETED_UNVERIFIED` | `not_sure_if_saved` | Fine for testing; not a production success. | +| Saved, waiting for its check | `SAVED_AWAITING_CHECK` | `not_sure_if_saved` | The save's check is a later report (a claim acknowledgment, an export). Don't rerun it. `openadapt-flow deferred settle` counts it as done only when the report confirms it. | | Didn't run | `REJECTED_POLICY`, `CANCELED`, `FAILED_PLATFORM` | `did_not_run` | Fix the gate or the environment and run again. | The console's last line can read `HALTED` for both the second and third rows. @@ -375,7 +412,8 @@ shipping in the next release; a pull request number means it is being merged. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles the verified run into a program (`openadapt_flow.jit`, released; - prompt-to-program `compile-agent`, landing). Agents can drive the recording + prompt-to-program `compile-agent`, landing; the agent can act through code + instead of screenshots, [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)). Agents can drive the recording on web, native Windows, RDP and Citrix. 2. **Compiles to the fastest layer the app allows.** Page structure first (DOM, Windows UI Automation, macOS Accessibility), direct API calls where the app's @@ -414,7 +452,7 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is landing. + without saving, is on main; fixes of up to three new steps are in review. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` @@ -433,12 +471,18 @@ shipping in the next release; a pull request number means it is being merged. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) | +| Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) | +| Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | +| Lists every change made before a stop, and reverses them with undo steps a person approves. | `undo`, [AFTER_A_STOP.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/AFTER_A_STOP.md) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | +| Repairs a change that needs new steps, such as a new required dialog or a control moved into a menu: up to three new steps, each with its own record check, probed up to the save before one person approves. | [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md), [#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| The agent that does a task once can act through code, not a screenshot-and-click loop: it reads the page or accessibility tree, acts by role and name, keeps notes with a sandboxed read-only shell, and asks for a screenshot only when it needs one. It is arm 7 of the scoreboard; its figures are quoted from there once its rows are in. | `CODE_AS_ACTION.md` in [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | -| Ships as one monorepo with separately installable packages (flow, agent, types, capture, and privacy and desktop moving in). | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | +| Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy and desktop), each released from the monorepo. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | | Makes no model calls on healthy runs. Model training (openadapt-ml) is separate research. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | ## The full map @@ -474,6 +518,7 @@ improvising or retrying when something is off. | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/README.md) | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md), `runtime/effects/` | +| Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | | Stop before the effect instead of improvising (`HALTED_BEFORE_EFFECT`) | Released | Real app and synthetic | `runtime/replayer.py` | | Never retry an uncertain save (`RECONCILIATION_REQUIRED`) | Released | Synthetic (fault server) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md) | | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` | @@ -508,7 +553,7 @@ improvising or retrying when something is off. | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | | Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | -| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | In review ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276)) | Synthetic (MockMed) | `induce --probe-on` | +| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | ## Surfaces @@ -519,7 +564,8 @@ improvising or retrying when something is off. | Native Windows (UI Automation) | Released, early | Synthetic (WinForms, 3 of 3 trials) | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | | Native macOS (Accessibility) | Released, early | Synthetic (TextEdit, 3 of 3 trials) | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | | RDP (pixels; Aardwolf or headless FreeRDP with a session pool) | Released, early | Real Windows 11 host, 3 of 3, plus a FreeRDP batch | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md), `backends/xfreerdp.py` | -| Citrix | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md) | +| Citrix and other virtual desktops, with OpenAdapt installed inside the session (`in_session`): the same structure-first path as a local desktop | Released, early | Same evidence as the Windows path; no live Citrix session yet | [SURFACES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SURFACES.md) | +| Citrix from outside the session, pixels only (`remote_display`), the fallback when nothing can be installed inside | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md) | | Chain programs across surfaces, passing confirmed values | Released | Synthetic | `openadapt-flow compose`, `process`; [PROCESS_CONTRACT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PROCESS_CONTRACT.md) | | One program that switches surfaces, tabs and popups mid-run | On main ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Halt when a workflow qualified with page structure runs on a pixels-only surface | Released | Synthetic | `openadapt-flow qualify` | @@ -533,7 +579,10 @@ improvising or retrying when something is off. | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | | Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | | Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Run a worklist on several workers at once, one signed-in session each, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--workers` | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -542,10 +591,10 @@ improvising or retrying when something is off. | Governed repair: propose, review, campaigns, approval, staging, canary, rollback | Released | Synthetic | `openadapt-flow repair`, [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | -| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | -| Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | -| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | +| Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | +| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) | +| Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | +| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents @@ -553,9 +602,11 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Synthetic | `compile-agent` | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -565,6 +616,7 @@ improvising or retrying when something is off. | Record mouse, keyboard, screen video and accessibility trees on one clock (Windows, macOS, Linux X11) | Released (capture 1.3.0) | Real OS | [packages/capture](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/README.md); Wayland is not supported | | Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [packages/privacy](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/privacy/README.md), [PRIVACY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PRIVACY.md) | | Scrub by default before anything leaves the machine | On main ([#1262](https://github.com/OpenAdaptAI/OpenAdapt/pull/1262)) | Unit tests | `openadapt-privacy` | +| Each package installs on its own within a size budget, checked in CI (privacy without desktop, capture without plotting) | On main ([#1270](https://github.com/OpenAdaptAI/OpenAdapt/pull/1270), [#1271](https://github.com/OpenAdaptAI/OpenAdapt/pull/1271), [#1279](https://github.com/OpenAdaptAI/OpenAdapt/pull/1279)) | CI | `.github/workflows/packages.yml` | | Signed receipts (Seals), countersigned and timestamped, offline by default | Released | Unit tests, real TSA tokens | `openadapt-flow seal`, `countersign`, `local-seal` | | Strict no-outbound mode and an unattended on-prem service | Released | Synthetic | `openadapt-flow no-outbound`, `service`; [ON_PREM.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ON_PREM.md) | @@ -573,9 +625,9 @@ improvising or retrying when something is off. Each `openadapt-flow` command, grouped by the capability it serves. A test fails when a command is added without being listed here. -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`. +- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. +- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. +- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. - **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). - **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). - **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. @@ -599,11 +651,29 @@ fails when a command is added without being listed here. complete database read path caught all 72. This is a coverage test over a small, hand-authored set of faults, not a real-world error rate: [EFFECT_E2E.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md). +- **OpenEMR referral: agent, screen program and API step.** Median wall time + per referral, from a fresh browser through sign-in, the entry and the read-back: + agent alone 36.5 s (62 of 63 correct, the 14 to 15 Oct study); compiled program + on the screen 22.5 s (18 of 18); the same program with the API step 3.3 s + (18 of 18). The program rows ran on 19 Oct, interleaved on one copy, without + the approval gate or resumable runs, and the API step was admitted by the + benchmark harness. The approval-gated production profile still runs the + screen steps; making the API step work under it is in progress: + [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). +- **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, + working alone on a test app whose server broke the save in 6 ways (5 items + each), reported all 30 bad saves as done. `verify-run` flagged all 30 and + confirmed all 10 clean saves, with no false alarms. The app shows "saved" + before the server answers, so this measures what the screen can't show, not + the agent's care. Not yet run on OpenEMR: + [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). ## What is not done yet - No commercial EMR, claims system or real Citrix session has been tested. -- In progress: self-repair across several new steps, and the scoreboard. +- In review: self-repair across several new steps + ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In + progress: the scoreboard. - Windows installers are unsigned until free code signing through SignPath Foundation is approved; macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The @@ -962,6 +1032,31 @@ class named with `--allow-halt`. A handoff copies an effect-bound parameter from a confirmed receipt. Missing evidence HALTs. Subflows and worklists stay inside each child, on that child's surface. +## Parallel worklists + +`--workers N` runs a worklist's items on N signed-in sessions at once +([`THROUGHPUT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md)). Every item keeps every check. The run +refuses N above the deployment's `worklist.max_workers`, fewer accounts than +workers, two accounts for the same user, two workers sharing one session, a +program with steps outside its loop, and rows not declared +`items_independent`. It can't tell on its own whether two rows touch the same +record; that declaration is yours. A halt on any worker stops new items on +all of them, and an item claimed before a crash is never run again +automatically: it is marked `uncertain` for a person to check. The pipelined +read-back window applies only to an item's last save and only to read-backs +that don't use the screen; durable runs, which include the Standard and +Regulated profiles, always read back inline. Parallel RDP runs use the +FreeRDP session pool and have unit tests only. Throughput figures come from +one machine and a local OpenEMR test copy, not from a production EMR. + +## Short parameter values + +The right-record check binds a typed value into its expected text only when +the value has at least 4 characters. An item whose bound value is shorter, +such as the first name "Bo", halts at the identity check instead of saving. +This is deliberate: a short value matches too much unrelated text to identify +a record. + ## Save triage `openadapt-flow triage` orders verified saves for review under @@ -1296,18 +1391,25 @@ grade is `screen_readback`, never `independent_readback`. **Without a read that is independent of the acting session, the run cannot be `VERIFIED`.** That read is either a system-of-record read (REST, FHIR, SQL, -file; tier 1) or the application's record page read in a separate signed-in -session (`ui_readback` for web, `fresh_session` for desktop and RDP, -tested on mocks only; tier 2). The second goes through the same -application, so it is not a system-of-record read and has its own limits -([`EFFECT_KIT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-ui-read-back-ui_readback)). -A pixel-only Citrix or RDP session with neither is that case. Reopening the +file; tier 1) or the application's own screens read in a separate signed-in +session (tier 2): a record page in a separate browser (`ui_readback`), a +relaunched desktop app or a second RDP session (`fresh_session`, tested on +mocks only), or a look-up in a separate desktop, RDP or Citrix session +(`desktop_readback`, tier 2 with its own RDP logon). None of them goes around +the application, so none is a system-of-record read, and each has its own +limits ([`EFFECT_KIT.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-ui-read-back-ui_readback), +[`desktop_readback`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback)). +If a person can look the record up, the check can too. A pixel-only Citrix or +RDP deployment that declares no such check is the remaining case. Reopening the record in the acting session (tier 3) is reported as its own check level, `reopened` ("Reopened in the same session"), and is not `VERIFIED`: a confirmed reopen ends `COMPLETED_UNVERIFIED`, and the agent result says `not_sure_if_saved` with `checked_below_minimum` ([`READBACK_LADDER.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/READBACK_LADDER.md)). A workflow's `minimum_readback` -can raise the floor to `system_of_record`, never lower it. The on-screen +can raise the floor to `system_of_record`, never lower it. +A save that only a later report can confirm (a claim acknowledgment, a +nightly export) ends `SAVED_AWAITING_CHECK` and becomes `VERIFIED` only when +that report accepts it ([`deferred_report`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#checked-later-deferred_report)). The on-screen oracle can still halt when the region is unreadable or wrong. If a consequential write may already have been delivered and the check did not confirm it, the production `transaction_outcome` is `RECONCILIATION_REQUIRED`, From 0de133dfd22b2077ea2398b229732a5617726121 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 14:02:44 -0400 Subject: [PATCH 3/5] docs: scope the API step to single HTML form submissions Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 2 +- README.md | 6 ++++-- llms-full.txt | 8 +++++--- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c10f3e04f..7eec73659 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -78,7 +78,7 @@ shipping in the next release; a pull request number means it is being merged. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. JSON or GraphQL requests, multi-step wizards and governed runs still use the screen steps. | `api`, [API_TIER.md](packages/flow/docs/API_TIER.md) | | Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) | | Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) | | Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) | diff --git a/README.md b/README.md index 2faaba9cc..bf66238de 100644 --- a/README.md +++ b/README.md @@ -32,8 +32,10 @@ browser through sign-in, the entry and the saved record read back (median): | OpenAdapt program, screen steps | `███████████` 22.5 s | 18 of 18 | $0 | | OpenAdapt program, API step | `██` 3.3 s | 18 of 18 | $0 | -The API step replaces the program's 41 screen steps with the one save request -the app itself sends, then reads the record back. In both program rows a +The API step replaces the program's 41 screen steps with the one form +submission the app itself sends, then reads the record back. It covers single +HTML form saves like this one; other apps and request types are not yet +measured. In both program rows a possible-duplicate referral stopped without saving, as it should. Both program rows ran on 19 Oct on one copy of the app, interleaved, without the approval gate and without resumable runs; the API step was admitted by the benchmark diff --git a/llms-full.txt b/llms-full.txt index b176a0a78..f51d59863 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -42,8 +42,10 @@ browser through sign-in, the entry and the saved record read back (median): | OpenAdapt program, screen steps | `███████████` 22.5 s | 18 of 18 | $0 | | OpenAdapt program, API step | `██` 3.3 s | 18 of 18 | $0 | -The API step replaces the program's 41 screen steps with the one save request -the app itself sends, then reads the record back. In both program rows a +The API step replaces the program's 41 screen steps with the one form +submission the app itself sends, then reads the record back. It covers single +HTML form saves like this one; other apps and request types are not yet +measured. In both program rows a possible-duplicate referral stopped without saving, as it should. Both program rows ran on 19 Oct on one copy of the app, interleaved, without the approval gate and without resumable runs; the API step was admitted by the benchmark @@ -471,7 +473,7 @@ shipping in the next release; a pull request number means it is being merged. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. JSON or GraphQL requests, multi-step wizards and governed runs still use the screen steps. | `api`, [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md) | | Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) | | Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) | | Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | From 1e1d83e9e00711aebd258dced8d5545d45374c70 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 14:26:34 -0400 Subject: [PATCH 4/5] docs: the OpenEMR claims result, 25 of 30 caught, and its collateral-edit gap Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 8 ++++++++ README.md | 13 +++++++++---- llms-full.txt | 21 +++++++++++++++++---- 3 files changed, 34 insertions(+), 8 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 7eec73659..a9d41dad2 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -267,6 +267,14 @@ fails when a command is added without being listed here. benchmark harness. The approval-gated production profile still runs the screen steps; making the API step work under it is in progress: [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). +- **An agent's claims versus the read-back, on OpenEMR.** With faults injected + in the fixture's database (partial, wrong value, duplicate, not persisted, + another chart deleted, another chart edited; 5 items each), Claude Sonnet 5.5 + computer use claimed success on all 30 bad saves. `verify-run` caught 25 and + confirmed 5 of 5 clean saves with no false alarms. It missed all 5 edits to a + different chart: its collateral check finds a removed record, not a changed + one. That check is being added: + [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). - **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, working alone on a test app whose server broke the save in 6 ways (5 items each), reported all 30 bad saves as done. `verify-run` flagged all 30 and diff --git a/README.md b/README.md index bf66238de..f213cb45b 100644 --- a/README.md +++ b/README.md @@ -44,10 +44,15 @@ its own copy of the app. [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step) has every run. -A screen can't show a save that went wrong behind it. On a test app that -shows a save as done before its server answers, with the server breaking the -save in 6 ways, 5 times each, the agent reported all 30 bad saves as done. Reading each record back caught all 30, with no false -alarms on 10 clean saves ([details below](#why-the-read-back-matters)). +A screen can't show a save that went wrong behind it. On a test copy of +OpenEMR, with faults injected in its database (a field blanked, a value changed, a +duplicate chart, a chart removed, another chart deleted or edited), the agent +claimed success on all 30 bad saves. Reading the record back caught 25 of the +30 and confirmed all 5 clean saves, with no false alarms. It missed the 5 where +a different chart was edited; that check is being added. On MockMed, a test app +that shows a save as done before its server answers, the agent also claimed +all 30 bad saves, and the read-back caught all 30 +([details below](#why-the-read-back-matters)). ## One harness, three modes diff --git a/llms-full.txt b/llms-full.txt index f51d59863..bf6e06e35 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -54,10 +54,15 @@ its own copy of the app. [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step) has every run. -A screen can't show a save that went wrong behind it. On a test app that -shows a save as done before its server answers, with the server breaking the -save in 6 ways, 5 times each, the agent reported all 30 bad saves as done. Reading each record back caught all 30, with no false -alarms on 10 clean saves ([details below](#why-the-read-back-matters)). +A screen can't show a save that went wrong behind it. On a test copy of +OpenEMR, with faults injected in its database (a field blanked, a value changed, a +duplicate chart, a chart removed, another chart deleted or edited), the agent +claimed success on all 30 bad saves. Reading the record back caught 25 of the +30 and confirmed all 5 clean saves, with no false alarms. It missed the 5 where +a different chart was edited; that check is being added. On MockMed, a test app +that shows a save as done before its server answers, the agent also claimed +all 30 bad saves, and the read-back caught all 30 +([details below](#why-the-read-back-matters)). ## One harness, three modes @@ -662,6 +667,14 @@ fails when a command is added without being listed here. benchmark harness. The approval-gated production profile still runs the screen steps; making the API step work under it is in progress: [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). +- **An agent's claims versus the read-back, on OpenEMR.** With faults injected + in the fixture's database (partial, wrong value, duplicate, not persisted, + another chart deleted, another chart edited; 5 items each), Claude Sonnet 5.5 + computer use claimed success on all 30 bad saves. `verify-run` caught 25 and + confirmed 5 of 5 clean saves with no false alarms. It missed all 5 edits to a + different chart: its collateral check finds a removed record, not a changed + one. That check is being added: + [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). - **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, working alone on a test app whose server broke the save in 6 ways (5 items each), reported all 30 bad saves as done. `verify-run` flagged all 30 and From ee0f348aacce26a5e1a433ba5ba7379afe5e85b2 Mon Sep 17 00:00:00 2001 From: abrichr Date: Mon, 19 Oct 2026 19:25:02 -0400 Subject: [PATCH 5/5] docs: public copy lists only shipped work; test for status words Removes 'in review', 'being merged', 'not yet released', 'being built' and the 'What is not done yet' section from README, CAPABILITIES, llms.txt and the agent README. Rows for merged PRs read 'On main'; rows for unmerged ones are removed. Co-Authored-By: Claude Opus 5.5 --- CAPABILITIES.md | 57 +++++++++++-------------- README.md | 18 +++----- llms-full.txt | 75 ++++++++++++++------------------- llms.txt | 8 ++-- packages/agent/README.md | 2 +- tests/test_shipped_only_copy.py | 39 +++++++++++++++++ 6 files changed, 106 insertions(+), 93 deletions(-) create mode 100644 tests/test_shipped_only_copy.py diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 1d6dabec3..c01dbc8ea 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -14,7 +14,7 @@ an older version or an older page: check here, then the linked code. Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" means it ships in the current PyPI packages; "on main" means merged and -shipping in the next release; a pull request number means it is being merged. +shipping in the next release. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -101,8 +101,6 @@ Status: - **Released**: in the current PyPI release (openadapt-flow 2.2.0). - **On main**: merged; ships in the next release. -- **In review**: an open pull request. -- **Planned**: designed, not built. Evidence: @@ -134,10 +132,9 @@ improvising or retrying when something is off. | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | | Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | -| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | In review ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | -| Learn the record check from the recording, when the person looks the record up after saving | In review ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | -| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | In review ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | -| Keep the server's own answer to a save as lower-level evidence on the receipt | In review ([#1298](https://github.com/OpenAdaptAI/OpenAdapt/pull/1298)) | Synthetic | receipts | +| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | On main ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | +| Learn the record check from the recording, when the person looks the record up after saving | On main ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | +| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | On main ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | | A ledger of what changed before a stop, a write-ordering lint, and undo steps a person approves | On main ([#1263](https://github.com/OpenAdaptAI/OpenAdapt/pull/1263)) | Synthetic (MockMed) | change ledger | ## Approvals @@ -160,7 +157,6 @@ improvising or retrying when something is off. | Typed business decisions inside a run | Released | Synthetic | `business_decision` states, `openadapt-flow business-decisions` | | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | -| Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | | Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | @@ -185,16 +181,16 @@ improvising or retrying when something is off. | --- | --- | --- | --- | | Page structure (DOM, UI Automation, Accessibility) first, then template, OCR and geometry | Released | Real app | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md) | | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | -| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | -| Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | -| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | -| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | -| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | -| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | -| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | -| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | -| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | +| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | +| Fall back from the API step to page structure when the API is unavailable, before anything is sent | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | On main ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | On main ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | +| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | On main ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | +| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | On main ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -204,8 +200,8 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | -| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | On main ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | On main ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md) | | Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | @@ -215,13 +211,13 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](packages/flow/docs/JIT.md) | -| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | On main ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](packages/agent/README.md) | -| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](packages/agent/README.md#track-a-long-job) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | On main ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](packages/agent/README.md#track-a-long-job) | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | On main ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | -| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | +| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | On main ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -364,8 +360,7 @@ A test fails when a command has no file or this list is out of date. on the screen 22.5 s (18 of 18); the same program with the API step 3.3 s (18 of 18). The program rows ran on 19 Oct, interleaved on one copy, without the approval gate or resumable runs, and the API step was admitted by the - benchmark harness. The approval-gated production profile still runs the - screen steps; making the API step work under it is in progress: + benchmark harness: [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). - **An agent's claims versus the read-back, on OpenEMR.** With faults injected in the fixture's database (partial, wrong value, duplicate, not persisted, @@ -373,7 +368,7 @@ A test fails when a command has no file or this list is out of date. computer use claimed success on all 30 bad saves. `verify-run` caught 25 and confirmed 5 of 5 clean saves with no false alarms. It missed all 5 edits to a different chart: its collateral check finds a removed record, not a changed - one. That check is being added: + one.: [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). - **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, working alone on a test app whose server broke the save in 6 ways (5 items @@ -383,13 +378,9 @@ A test fails when a command has no file or this list is out of date. the agent's care. Not yet run on OpenEMR: [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). -## What is not done yet +## Limits - No commercial EMR, claims system or real Citrix session has been tested. -- In review: self-repair across several new steps - ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In - progress: the scoreboard. -- Windows installers are unsigned until free code signing through SignPath - Foundation is approved; macOS builds are ad-hoc signed. +- Windows installers are unsigned, and macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The Evidence column says which. diff --git a/README.md b/README.md index f213cb45b..83b7f694f 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ OpenEMR, with faults injected in its database (a field blanked, a value changed, duplicate chart, a chart removed, another chart deleted or edited), the agent claimed success on all 30 bad saves. Reading the record back caught 25 of the 30 and confirmed all 5 clean saves, with no false alarms. It missed the 5 where -a different chart was edited; that check is being added. On MockMed, a test app +a different chart was edited. On MockMed, a test app that shows a save as done before its server answers, the agent also claimed all 30 bad saves, and the read-back caught all 30 ([details below](#why-the-read-back-matters)). @@ -149,10 +149,8 @@ jit.run(note="Recheck knee in ten days") # the program runs: 0 model calls, sav ``` [Compile a computer-use agent's run](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) has the full -example. This path runs web apps today. On Windows, RDP and Citrix, an agent can -already drive a recording through the MCP server's recording tools (on `main`, -not yet released), and the same compile path for those surfaces is being -built. +example. The same path compiles an agent's run in web apps and in Windows, +macOS, Linux, RDP and Citrix apps. ### Call programs from your agent @@ -219,10 +217,9 @@ database read-back: Both were equally correct here, and the agent matched or beat OpenAdapt in three other tests. [Every result](https://openadapt.ai/compare/computer-use-agents) -is published. The program was only about a fifth faster because it still drives -every page, as the agent does; direct API steps compiled from the recording's -network traffic are in review -([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)). OpenAdapt isn't +is published. That program was only about a fifth faster because it drove every +page, as the agent does; with the API step compiled from the recording's network +traffic, the same referral takes 3.3 s ([Measured on OpenEMR](#measured-on-openemr)). OpenAdapt isn't affiliated with the OpenEMR project. ## Every backend @@ -285,8 +282,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. - **Runs locally**: no model calls on healthy runs, patient data redacted, signed receipts. -[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the status, evidence and code for each, -including the few still being merged. +[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. ## What your agent gets back diff --git a/llms-full.txt b/llms-full.txt index f8dafc0c4..e7f545cb5 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -59,7 +59,7 @@ OpenEMR, with faults injected in its database (a field blanked, a value changed, duplicate chart, a chart removed, another chart deleted or edited), the agent claimed success on all 30 bad saves. Reading the record back caught 25 of the 30 and confirmed all 5 clean saves, with no false alarms. It missed the 5 where -a different chart was edited; that check is being added. On MockMed, a test app +a different chart was edited. On MockMed, a test app that shows a save as done before its server answers, the agent also claimed all 30 bad saves, and the read-back caught all 30 ([details below](#why-the-read-back-matters)). @@ -159,10 +159,8 @@ jit.run(note="Recheck knee in ten days") # the program runs: 0 model calls, sav ``` [Compile a computer-use agent's run](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) has the full -example. This path runs web apps today. On Windows, RDP and Citrix, an agent can -already drive a recording through the MCP server's recording tools (on `main`, -not yet released), and the same compile path for those surfaces is being -built. +example. The same path compiles an agent's run in web apps and in Windows, +macOS, Linux, RDP and Citrix apps. ### Call programs from your agent @@ -229,10 +227,9 @@ database read-back: Both were equally correct here, and the agent matched or beat OpenAdapt in three other tests. [Every result](https://openadapt.ai/compare/computer-use-agents) -is published. The program was only about a fifth faster because it still drives -every page, as the agent does; direct API steps compiled from the recording's -network traffic are in review -([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)). OpenAdapt isn't +is published. That program was only about a fifth faster because it drove every +page, as the agent does; with the API step compiled from the recording's network +traffic, the same referral takes 3.3 s ([Measured on OpenEMR](#measured-on-openemr)). OpenAdapt isn't affiliated with the OpenEMR project. ## Every backend @@ -295,8 +292,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. - **Runs locally**: no model calls on healthy runs, patient data redacted, signed receipts. -[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the status, evidence and code for each, -including the few still being merged. +[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. ## What your agent gets back @@ -414,7 +410,7 @@ an older version or an older page: check here, then the linked code. Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" means it ships in the current PyPI packages; "on main" means merged and -shipping in the next release; a pull request number means it is being merged. +shipping in the next release. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -501,8 +497,6 @@ Status: - **Released**: in the current PyPI release (openadapt-flow 2.2.0). - **On main**: merged; ships in the next release. -- **In review**: an open pull request. -- **Planned**: designed, not built. Evidence: @@ -534,10 +528,9 @@ improvising or retrying when something is off. | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | | Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | -| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | In review ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | -| Learn the record check from the recording, when the person looks the record up after saving | In review ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | -| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | In review ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | -| Keep the server's own answer to a save as lower-level evidence on the receipt | In review ([#1298](https://github.com/OpenAdaptAI/OpenAdapt/pull/1298)) | Synthetic | receipts | +| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | On main ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | +| Learn the record check from the recording, when the person looks the record up after saving | On main ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | +| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | On main ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | | A ledger of what changed before a stop, a write-ordering lint, and undo steps a person approves | On main ([#1263](https://github.com/OpenAdaptAI/OpenAdapt/pull/1263)) | Synthetic (MockMed) | change ledger | ## Approvals @@ -560,7 +553,6 @@ improvising or retrying when something is off. | Typed business decisions inside a run | Released | Synthetic | `business_decision` states, `openadapt-flow business-decisions` | | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | -| Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | | Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | @@ -585,16 +577,16 @@ improvising or retrying when something is off. | --- | --- | --- | --- | | Page structure (DOM, UI Automation, Accessibility) first, then template, OCR and geometry | Released | Real app | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | -| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | -| Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | -| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | -| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | -| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | -| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | -| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | -| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | -| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | +| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | +| Fall back from the API step to page structure when the API is unavailable, before anything is sent | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | On main ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | On main ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | +| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | On main ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | +| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | On main ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -604,8 +596,8 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | -| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | On main ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | On main ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) | | Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | @@ -615,13 +607,13 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | -| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | On main ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | -| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | On main ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | On main ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | -| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | +| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | On main ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -764,8 +756,7 @@ A test fails when a command has no file or this list is out of date. on the screen 22.5 s (18 of 18); the same program with the API step 3.3 s (18 of 18). The program rows ran on 19 Oct, interleaved on one copy, without the approval gate or resumable runs, and the API step was admitted by the - benchmark harness. The approval-gated production profile still runs the - screen steps; making the API step work under it is in progress: + benchmark harness: [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). - **An agent's claims versus the read-back, on OpenEMR.** With faults injected in the fixture's database (partial, wrong value, duplicate, not persisted, @@ -773,7 +764,7 @@ A test fails when a command has no file or this list is out of date. computer use claimed success on all 30 bad saves. `verify-run` caught 25 and confirmed 5 of 5 clean saves with no false alarms. It missed all 5 edits to a different chart: its collateral check finds a removed record, not a changed - one. That check is being added: + one.: [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). - **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, working alone on a test app whose server broke the save in 6 ways (5 items @@ -783,14 +774,10 @@ A test fails when a command has no file or this list is out of date. the agent's care. Not yet run on OpenEMR: [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). -## What is not done yet +## Limits - No commercial EMR, claims system or real Citrix session has been tested. -- In review: self-repair across several new steps - ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In - progress: the scoreboard. -- Windows installers are unsigned until free code signing through SignPath - Foundation is approved; macOS builds are ad-hoc signed. +- Windows installers are unsigned, and macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The Evidence column says which. diff --git a/llms.txt b/llms.txt index ad3bd008a..bc7d61c50 100644 --- a/llms.txt +++ b/llms.txt @@ -2,14 +2,14 @@ > A person, or a computer-use agent once, shows a task in an application with no usable API. OpenAdapt compiles it into a program that makes the same entry every time with no model calls, confirms the right record before acting, reads the saved record back from the system of record, and stops instead of improvising or retrying. -This is the llms.txt for the OpenAdapt monorepo (github.com/OpenAdaptAI/OpenAdapt). Start with the capability map: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md. It lists every capability with its status (released, in review, planned), its evidence (real app, synthetic, untested) and where it lives. Check it before concluding that OpenAdapt lacks something. +This is the llms.txt for the OpenAdapt monorepo (github.com/OpenAdaptAI/OpenAdapt). Start with the capability map: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md. It lists every capability with its release (released, or on main for the next release), its evidence (real app, synthetic, untested) and where it lives. Check it before concluding that OpenAdapt lacks something. ## OpenAdapt in ten capabilities Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" -means it ships in the current PyPI packages; a pull request number means it is -being merged now. +means it ships in the current PyPI packages; "on main" means it ships in the +next release. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -72,7 +72,7 @@ being merged now. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. | `api`, [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | diff --git a/packages/agent/README.md b/packages/agent/README.md index 800c75c27..e09dd2acc 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -284,7 +284,7 @@ Flow's [execution profiles](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/p | `--timeout SECONDS` | How long one run may work (default 600). Time a person spends approving a save doesn't count. A run stopped at the limit is `not_sure_if_saved`. | | `--approval-mode MODE` | Whether each save waits for approval. Without it, Flow decides, and a run stops before each save until it's approved (`each_item`). `not_gated` turns that off for every run this server starts; each result records it under `technical.approval_mode`. | | `--delegate-approvals AGENT_ID` | Lets your agent answer a waiting save as `AGENT_ID` (an opaque id such as `agent:intake-bot`), with `list_approval_requests` and `answer_approval_request`. Flow accepts the answer only where the run's approver policy allows an agent for that step, and records the agent as the approver. By default only a person may answer. Needs openadapt-flow 2.2.0 or later. | -| `--admissions DIR` | Adds `prepare_admission`, `list_admissions` and `decide_admission`, the same steps as `openadapt-flow admit`, over recordings in `DIR/recordings` and shadow candidates in `DIR/shadow-candidates`. A person decides in your client's form. With `--delegate-approvals` and `--approver-grant`, your agent decides as that approver instead, under a grant a person signed that names it. It never decides an admission it prepared or one prepared on a server without `--delegate-approvals`, and never under the `regulated` policy pack. Accepting sets the workflow's runs to `problems_only`. The record names who decided, and for your agent the grant and who signed it. Not released yet. | +| `--admissions DIR` | Adds `prepare_admission`, `list_admissions` and `decide_admission`, the same steps as `openadapt-flow admit`, over recordings in `DIR/recordings` and shadow candidates in `DIR/shadow-candidates`. A person decides in your client's form. With `--delegate-approvals` and `--approver-grant`, your agent decides as that approver instead, under a grant a person signed that names it. It never decides an admission it prepared or one prepared on a server without `--delegate-approvals`, and never under the `regulated` policy pack. Accepting sets the workflow's runs to `problems_only`. The record names who decided, and for your agent the grant and who signed it. | | `--approver-grant FILE` | With `--admissions` and `--delegate-approvals`: the approver grant a person signed (`openadapt-flow approver-grant sign`) that lets your agent decide admissions. Flow checks it on every decision. Without it only a person decides. | | `--headed` | Shows the browser. A person needs it to continue or skip a stopped run in a web app. | | `--extra-run-arg ARG`, `--flow-cli CMD` | Change how Flow is called. Either one can relax Flow's checks, for example `--extra-run-arg=--allow-unencrypted`. | diff --git a/tests/test_shipped_only_copy.py b/tests/test_shipped_only_copy.py new file mode 100644 index 000000000..a9d5252a2 --- /dev/null +++ b/tests/test_shipped_only_copy.py @@ -0,0 +1,39 @@ +"""Public copy describes the product as it ships. + +Never publish unreleased or in-progress work on a public surface: no "being +built", "not released", "in review", "coming soon", "being merged" or roadmap +lists. Track gaps internally (issues, private notes) and build them instead. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + +PUBLIC = [ + ROOT / "README.md", + ROOT / "CAPABILITIES.md", + ROOT / "llms.txt", + *sorted((ROOT / "packages").glob("*/README.md")), +] + +NEVER = re.compile( + r"\bbeing built\b|\bnot (?:yet )?released\b|\bnot released yet\b" + r"|\bcoming soon\b|\bin review\b|\bin progress\b|\bbeing merged\b" + r"|\bbeing added\b|\bwhen it ships\b|\bnot done yet\b|\bon (?:our|the) roadmap\b", + re.IGNORECASE, +) + + +def test_public_copy_lists_only_shipped_work() -> None: + hits = [] + for path in PUBLIC: + if not path.exists(): + continue + for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + match = NEVER.search(line) + if match: + hits.append(f"{path.relative_to(ROOT)}:{number}: {match.group(0)!r}") + assert hits == [], "unreleased-work copy on a public surface:\n" + "\n".join(hits)