diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c42ee9af9..c01dbc8ea 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -14,7 +14,7 @@ an older version or an older page: check here, then the linked code. Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" means it ships in the current PyPI packages; "on main" means merged and -shipping in the next release; a pull request number means it is being merged. +shipping in the next release. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -79,7 +79,7 @@ shipping in the next release; a pull request number means it is being merged. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. JSON or GraphQL requests, multi-step wizards and governed runs still use the screen steps. | `api`, [API_TIER.md](packages/flow/docs/API_TIER.md) | | Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) | | Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) | | Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) | @@ -101,8 +101,6 @@ Status: - **Released**: in the current PyPI release (openadapt-flow 2.2.0). - **On main**: merged; ships in the next release. -- **In review**: an open pull request. -- **Planned**: designed, not built. Evidence: @@ -134,10 +132,9 @@ improvising or retrying when something is off. | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | | Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | -| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | In review ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | -| Learn the record check from the recording, when the person looks the record up after saving | In review ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | -| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | In review ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | -| Keep the server's own answer to a save as lower-level evidence on the receipt | In review ([#1298](https://github.com/OpenAdaptAI/OpenAdapt/pull/1298)) | Synthetic | receipts | +| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | On main ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | +| Learn the record check from the recording, when the person looks the record up after saving | On main ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | +| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | On main ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | | A ledger of what changed before a stop, a write-ordering lint, and undo steps a person approves | On main ([#1263](https://github.com/OpenAdaptAI/OpenAdapt/pull/1263)) | Synthetic (MockMed) | change ledger | ## Approvals @@ -160,7 +157,6 @@ improvising or retrying when something is off. | Typed business decisions inside a run | Released | Synthetic | `business_decision` states, `openadapt-flow business-decisions` | | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | -| Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | | Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | @@ -185,16 +181,16 @@ improvising or retrying when something is off. | --- | --- | --- | --- | | Page structure (DOM, UI Automation, Accessibility) first, then template, OCR and geometry | Released | Real app | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md) | | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | -| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | -| Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | -| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | -| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | -| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | -| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | -| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | -| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | -| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | +| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | +| Fall back from the API step to page structure when the API is unavailable, before anything is sent | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | On main ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | On main ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#in-a-durable-run) | +| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | On main ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#under-governance) | +| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | On main ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -204,8 +200,8 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | -| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | On main ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | On main ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md) | | Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | @@ -215,13 +211,13 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](packages/flow/docs/JIT.md) | -| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | On main ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](packages/agent/README.md) | -| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](packages/agent/README.md#track-a-long-job) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | On main ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](packages/agent/README.md#track-a-long-job) | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | On main ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | -| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | +| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | On main ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -358,14 +354,33 @@ A test fails when a command has no file or this list is out of date. complete database read path caught all 72. This is a coverage test over a small, hand-authored set of faults, not a real-world error rate: [EFFECT_E2E.md](packages/flow/benchmark/effect_e2e/EFFECT_E2E.md). - -## What is not done yet +- **OpenEMR referral: agent, screen program and API step.** Median wall time + per referral, from a fresh browser through sign-in, the entry and the read-back: + agent alone 36.5 s (62 of 63 correct, the 14 to 15 Oct study); compiled program + on the screen 22.5 s (18 of 18); the same program with the API step 3.3 s + (18 of 18). The program rows ran on 19 Oct, interleaved on one copy, without + the approval gate or resumable runs, and the API step was admitted by the + benchmark harness: + [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). +- **An agent's claims versus the read-back, on OpenEMR.** With faults injected + in the fixture's database (partial, wrong value, duplicate, not persisted, + another chart deleted, another chart edited; 5 items each), Claude Sonnet 5.5 + computer use claimed success on all 30 bad saves. `verify-run` caught 25 and + confirmed 5 of 5 clean saves with no false alarms. It missed all 5 edits to a + different chart: its collateral check finds a removed record, not a changed + one.: + [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). +- **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, + working alone on a test app whose server broke the save in 6 ways (5 items + each), reported all 30 bad saves as done. `verify-run` flagged all 30 and + confirmed all 10 clean saves, with no false alarms. The app shows "saved" + before the server answers, so this measures what the screen can't show, not + the agent's care. Not yet run on OpenEMR: + [SCOREBOARD.md](packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). + +## Limits - No commercial EMR, claims system or real Citrix session has been tested. -- In review: self-repair across several new steps - ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In - progress: the scoreboard. -- Windows installers are unsigned until free code signing through SignPath - Foundation is approved; macOS builds are ad-hoc signed. +- Windows installers are unsigned, and macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The Evidence column says which. diff --git a/README.md b/README.md index c5b5685bd..83b7f694f 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,39 @@ each save, reads the save back from the system of record after it, and returns a plain outcome. When anything doesn't match, it stops and tells your agent instead of guessing. +## Measured on OpenEMR + +One new-patient referral in a test copy of OpenEMR, timed from a fresh +browser through sign-in, the entry and the saved record read back (median): + +| Who does the entry | Time per referral | Saved correctly | Model cost | +|---|---|---|---| +| Agent alone (Claude Sonnet 5.5 computer use) | `██████████████████` 36.5 s | 62 of 63 | about $0.08 | +| OpenAdapt program, screen steps | `███████████` 22.5 s | 18 of 18 | $0 | +| OpenAdapt program, API step | `██` 3.3 s | 18 of 18 | $0 | + +The API step replaces the program's 41 screen steps with the one form +submission the app itself sends, then reads the record back. It covers single +HTML form saves like this one; other apps and request types are not yet +measured. In both program rows a +possible-duplicate referral stopped without saving, as it should. Both program +rows ran on 19 Oct on one copy of the app, interleaved, without the approval +gate and without resumable runs; the API step was admitted by the benchmark +harness, not by a person. The agent rows are the 14 to 15 Oct timing study, on +its own copy of the app. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step) +has every run. + +A screen can't show a save that went wrong behind it. On a test copy of +OpenEMR, with faults injected in its database (a field blanked, a value changed, a +duplicate chart, a chart removed, another chart deleted or edited), the agent +claimed success on all 30 bad saves. Reading the record back caught 25 of the +30 and confirmed all 5 clean saves, with no false alarms. It missed the 5 where +a different chart was edited. On MockMed, a test app +that shows a save as done before its server answers, the agent also claimed +all 30 bad saves, and the read-back caught all 30 +([details below](#why-the-read-back-matters)). + ## One harness, three modes | Mode | What happens | Model calls | @@ -46,6 +79,16 @@ This is a coverage test over a small, hand-written set of faults, not a real-world error rate. [EFFECT_E2E.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md) has the method and raw results. +An agent working alone can't see these faults either. Claude Sonnet 5.5, using +computer use, did a task on a test app whose server broke the save in 6 ways, +5 times each. The agent reported all 30 bad saves as done. Given the agent's +run, `openadapt-flow verify-run` read each record back: it flagged all 30 and +confirmed all 10 clean saves, with no false alarms. The test app shows "saved" +before the server answers, so the screen gave the agent nothing to notice. That +gap is what the read-back closes. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back) +has the method, the raw runs and the cost. + ## Try it in one command ```bash @@ -106,10 +149,8 @@ jit.run(note="Recheck knee in ten days") # the program runs: 0 model calls, sav ``` [Compile a computer-use agent's run](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) has the full -example. This path runs web apps today. On Windows, RDP and Citrix, an agent can -already drive a recording through the MCP server's recording tools (on `main`, -not yet released), and the same compile path for those surfaces is being -built. +example. The same path compiles an agent's run in web apps and in Windows, +macOS, Linux, RDP and Citrix apps. ### Call programs from your agent @@ -176,10 +217,9 @@ database read-back: Both were equally correct here, and the agent matched or beat OpenAdapt in three other tests. [Every result](https://openadapt.ai/compare/computer-use-agents) -is published. The program was only about a fifth faster because it still drives -every page, as the agent does; direct API steps compiled from the recording's -network traffic are in review -([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)). OpenAdapt isn't +is published. That program was only about a fifth faster because it drove every +page, as the agent does; with the API step compiled from the recording's network +traffic, the same referral takes 3.3 s ([Measured on OpenEMR](#measured-on-openemr)). OpenAdapt isn't affiliated with the OpenEMR project. ## Every backend @@ -242,8 +282,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. - **Runs locally**: no model calls on healthy runs, patient data redacted, signed receipts. -[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the status, evidence and code for each, -including the few still being merged. +[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. ## What your agent gets back diff --git a/llms-full.txt b/llms-full.txt index 7f4752fa6..e7f545cb5 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -31,6 +31,39 @@ each save, reads the save back from the system of record after it, and returns a plain outcome. When anything doesn't match, it stops and tells your agent instead of guessing. +## Measured on OpenEMR + +One new-patient referral in a test copy of OpenEMR, timed from a fresh +browser through sign-in, the entry and the saved record read back (median): + +| Who does the entry | Time per referral | Saved correctly | Model cost | +|---|---|---|---| +| Agent alone (Claude Sonnet 5.5 computer use) | `██████████████████` 36.5 s | 62 of 63 | about $0.08 | +| OpenAdapt program, screen steps | `███████████` 22.5 s | 18 of 18 | $0 | +| OpenAdapt program, API step | `██` 3.3 s | 18 of 18 | $0 | + +The API step replaces the program's 41 screen steps with the one form +submission the app itself sends, then reads the record back. It covers single +HTML form saves like this one; other apps and request types are not yet +measured. In both program rows a +possible-duplicate referral stopped without saving, as it should. Both program +rows ran on 19 Oct on one copy of the app, interleaved, without the approval +gate and without resumable runs; the API step was admitted by the benchmark +harness, not by a person. The agent rows are the 14 to 15 Oct timing study, on +its own copy of the app. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step) +has every run. + +A screen can't show a save that went wrong behind it. On a test copy of +OpenEMR, with faults injected in its database (a field blanked, a value changed, a +duplicate chart, a chart removed, another chart deleted or edited), the agent +claimed success on all 30 bad saves. Reading the record back caught 25 of the +30 and confirmed all 5 clean saves, with no false alarms. It missed the 5 where +a different chart was edited. On MockMed, a test app +that shows a save as done before its server answers, the agent also claimed +all 30 bad saves, and the read-back caught all 30 +([details below](#why-the-read-back-matters)). + ## One harness, three modes | Mode | What happens | Model calls | @@ -56,6 +89,16 @@ This is a coverage test over a small, hand-written set of faults, not a real-world error rate. [EFFECT_E2E.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md) has the method and raw results. +An agent working alone can't see these faults either. Claude Sonnet 5.5, using +computer use, did a task on a test app whose server broke the save in 6 ways, +5 times each. The agent reported all 30 bad saves as done. Given the agent's +run, `openadapt-flow verify-run` read each record back: it flagged all 30 and +confirmed all 10 clean saves, with no false alarms. The test app shows "saved" +before the server answers, so the screen gave the agent nothing to notice. That +gap is what the read-back closes. +[SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back) +has the method, the raw runs and the cost. + ## Try it in one command ```bash @@ -116,10 +159,8 @@ jit.run(note="Recheck knee in ten days") # the program runs: 0 model calls, sav ``` [Compile a computer-use agent's run](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) has the full -example. This path runs web apps today. On Windows, RDP and Citrix, an agent can -already drive a recording through the MCP server's recording tools (on `main`, -not yet released), and the same compile path for those surfaces is being -built. +example. The same path compiles an agent's run in web apps and in Windows, +macOS, Linux, RDP and Citrix apps. ### Call programs from your agent @@ -186,10 +227,9 @@ database read-back: Both were equally correct here, and the agent matched or beat OpenAdapt in three other tests. [Every result](https://openadapt.ai/compare/computer-use-agents) -is published. The program was only about a fifth faster because it still drives -every page, as the agent does; direct API steps compiled from the recording's -network traffic are in review -([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)). OpenAdapt isn't +is published. That program was only about a fifth faster because it drove every +page, as the agent does; with the API step compiled from the recording's network +traffic, the same referral takes 3.3 s ([Measured on OpenEMR](#measured-on-openemr)). OpenAdapt isn't affiliated with the OpenEMR project. ## Every backend @@ -252,8 +292,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. - **Runs locally**: no model calls on healthy runs, patient data redacted, signed receipts. -[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the status, evidence and code for each, -including the few still being merged. +[CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. ## What your agent gets back @@ -371,7 +410,7 @@ an older version or an older page: check here, then the linked code. Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" means it ships in the current PyPI packages; "on main" means merged and -shipping in the next release; a pull request number means it is being merged. +shipping in the next release. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -416,9 +455,10 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is on main; fixes of up to three new steps are in review. -9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded - save around any agent's write + without saving, is on main, and so are fixes of up to three new steps. +9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task + ledger that keeps a long job's rules, items, questions and checked saves on + the server ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)), a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. @@ -435,7 +475,7 @@ shipping in the next release; a pull request number means it is being merged. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls, compiled from the network traffic captured while recording. | `api`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. JSON or GraphQL requests, multi-step wizards and governed runs still use the screen steps. | `api`, [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md) | | Builds the record check itself. After a save, the app usually reads the record back to show it; `scaffold-verifier` finds that read in the captured traffic or a read-only database, builds the check from it and tries to break it before a person reviews it. No API or SQL needs to be written by hand. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#a-record-check-built-from-the-apps-own-reads-scaffold-verifier) | | Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) | | Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | @@ -457,8 +497,6 @@ Status: - **Released**: in the current PyPI release (openadapt-flow 2.2.0). - **On main**: merged; ships in the next release. -- **In review**: an open pull request. -- **Planned**: designed, not built. Evidence: @@ -490,10 +528,9 @@ improvising or retrying when something is off. | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | | Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | -| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | In review ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | -| Learn the record check from the recording, when the person looks the record up after saving | In review ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | -| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | In review ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | -| Keep the server's own answer to a save as lower-level evidence on the receipt | In review ([#1298](https://github.com/OpenAdaptAI/OpenAdapt/pull/1298)) | Synthetic | receipts | +| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | On main ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | +| Learn the record check from the recording, when the person looks the record up after saving | On main ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | +| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | On main ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | | A ledger of what changed before a stop, a write-ordering lint, and undo steps a person approves | On main ([#1263](https://github.com/OpenAdaptAI/OpenAdapt/pull/1263)) | Synthetic (MockMed) | change ledger | ## Approvals @@ -516,7 +553,6 @@ improvising or retrying when something is off. | Typed business decisions inside a run | Released | Synthetic | `business_decision` states, `openadapt-flow business-decisions` | | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | -| Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | | Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | On main ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276), [#1289](https://github.com/OpenAdaptAI/OpenAdapt/pull/1289)) | Synthetic (MockMed; desktop and remote test copies) | `induce --probe-on` | | Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | @@ -541,16 +577,16 @@ improvising or retrying when something is off. | --- | --- | --- | --- | | Page structure (DOM, UI Automation, Accessibility) first, then template, OCR and geometry | Released | Real app | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | -| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | -| Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | -| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | In review ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | -| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | In review ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | -| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | In review ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | -| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | In review ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | -| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | -| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | -| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | -| Record where each step's time goes in every run report | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | +| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | +| Fall back from the API step to page structure when the API is unavailable, before anything is sent | On main ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Approve an API step's exact request under per-item approval: one digest binds the request bytes, the record and the record check, and a changed request is not sent | On main ([#1355](https://github.com/OpenAdaptAI/OpenAdapt/pull/1355)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-an-approval-mode) | +| Run an API step in a durable run: checkpoint the span after the save is sent, never resend or redo a save that may have landed, and reconcile it by reading the record back | On main ([#1358](https://github.com/OpenAdaptAI/OpenAdapt/pull/1358)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#in-a-durable-run) | +| Run an admitted API step under a governed run authorization and the Standard or Regulated profile: the authorization names the API step by digest, the person who admits it names the right-record parameter, and the outcome reads `VERIFIED` | On main ([#1361](https://github.com/OpenAdaptAI/OpenAdapt/pull/1361)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#under-governance) | +| Defer an API save's read-back with pipelined read-back, as for a GUI save, and keep self-repair's probes and resumed run on the repaired GUI step | On main ([#1362](https://github.com/OpenAdaptAI/OpenAdapt/pull/1362)) | Synthetic | [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md#with-parallel-workers-pipelined-read-back-and-self-repair) | +| Run a worklist on several workers at once, one signed-in session each, each item claimed once, with a stop on any worker stopping all | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic patients on a local OpenEMR test copy | `--workers`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Read a save back while the next item runs (pipelined read-back, REST, FHIR, GraphQL or SQL record checks) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--readback-window`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| A faster speed profile that keeps every check (one insert per plain text field, shorter page-quiet wait) | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `runtime.speed: fast`, [THROUGHPUT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THROUGHPUT.md) | +| Record where each step's time goes in every run report | On main ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Real app and synthetic | `timings` in `report.json` | ## When the app changes @@ -560,8 +596,8 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | On main ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | In review ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | -| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | In review ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) | +| A scroll that never finds the next click's target halts on that click, so self-repair can take a relabelled control behind a scroll; the refusal for unreviewed-risk steps names them | On main ([#1337](https://github.com/OpenAdaptAI/OpenAdapt/pull/1337)) | Synthetic; openIMIS drift row checked offline on its recorded frames, not re-run | `runtime/replayer.py`, `qualify set-risk` | +| Self-repair across several new steps: a bridge of up to three actions for a new required dialog or a control moved into a menu, each new step armed with a record check and probed up to the save, one person approving | On main ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)) | Synthetic (MockMed menu and provider drifts) | `run --self-repair`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md) | | Keep pixel-path runs working across display scaling and font smoothing changes | On main ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (in-process RDP and canvas renderings: 17 of 24 healthy runs across 100–175%, up from 3; 0 wrong saves) | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) | | Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | On main ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | @@ -571,12 +607,13 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | -| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | +| Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | On main ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | -| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | On main ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | On main ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | -| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | +| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | On main ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof @@ -591,19 +628,112 @@ improvising or retrying when something is off. ## Every command, by capability -Each `openadapt-flow` command, grouped by the capability it serves. A test -fails when a command is added without being listed here. - -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. -- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). -- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). -- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. -- **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`. -- **Run on premises:** `service`, `no-outbound`. -- **Share safely with OpenAdapt Cloud:** `sanitize`, `review-sanitized`, `approve-sanitized`, `materialize-approved`, `push`, `validate-hosted`, `report-break`, `report-run`, `connect`, `login`, `connector`. -- **Try it and measure it:** `demo`, `tutorial`, `bench`, `benchmark`. +Each `openadapt-flow` command, grouped by the capability it serves. Each +command has one file under +[`packages/flow/docs/commands/`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/commands/): to add a +command, add `/.md` and run `python scripts/build_command_map.py`. +A test fails when a command has no file or this list is out of date. + + + +**Record and build a program:** + +- `compile-agent` +- `compile` +- `compose` +- `demo-record` +- `disambiguate` +- `for-each` +- `induce` +- `intake` +- `process` +- `pull` +- `record` +- `route` +- `variants` +- `visualize` + +**Check and admit:** + +- `admit` +- `api` +- `approver-grant` +- `bind-record-checks` +- `certify` +- `check` +- `jit` +- `known-ids` +- `lint` +- `propose-qualification` +- `qualify` +- `scaffold-verifier` + +**Run and answer:** + +- `approve` +- `business-decisions` +- `console` +- `deferred` +- `explain` +- `item-approval` +- `replay` +- `resume` +- `run` +- `triage` + +**Judge any executor's write:** + +- `serve-reward`: Scores agent training episodes by reading the system of record. +- `verify-run` + +**Repair and learn from halts:** + +- `fleet` +- `repair` +- `teach-branch` +- `teach` +- `undo`: The undo steps a person approves after a stop. + +**Give agents the program:** + +- `emit-mcp` +- `emit-skill` +- `serve-execute` + +**Proof:** + +- `attach-seal` +- `countersign` +- `local-seal` +- `seal` + +**Run on premises:** + +- `no-outbound` +- `service` + +**Share safely with OpenAdapt Cloud:** + +- `approve-sanitized` +- `connect` +- `connector` +- `login` +- `materialize-approved` +- `push` +- `report-break` +- `report-run` +- `review-sanitized` +- `sanitize` +- `validate-hosted` + +**Try it and measure it:** + +- `bench` +- `benchmark` +- `demo` +- `tutorial` + + ## Measured results @@ -620,15 +750,34 @@ fails when a command is added without being listed here. complete database read path caught all 72. This is a coverage test over a small, hand-authored set of faults, not a real-world error rate: [EFFECT_E2E.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md). - -## What is not done yet +- **OpenEMR referral: agent, screen program and API step.** Median wall time + per referral, from a fresh browser through sign-in, the entry and the read-back: + agent alone 36.5 s (62 of 63 correct, the 14 to 15 Oct study); compiled program + on the screen 22.5 s (18 of 18); the same program with the API step 3.3 s + (18 of 18). The program rows ran on 19 Oct, interleaved on one copy, without + the approval gate or resumable runs, and the API step was admitted by the + benchmark harness: + [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#openemr-agent-screen-program-and-api-step). +- **An agent's claims versus the read-back, on OpenEMR.** With faults injected + in the fixture's database (partial, wrong value, duplicate, not persisted, + another chart deleted, another chart edited; 5 items each), Claude Sonnet 5.5 + computer use claimed success on all 30 bad saves. `verify-run` caught 25 and + confirmed 5 of 5 clean saves with no false alarms. It missed all 5 edits to a + different chart: its collateral check finds a removed record, not a changed + one.: + [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). +- **An agent's claims versus the read-back.** Claude Sonnet 5.5 computer use, + working alone on a test app whose server broke the save in 6 ways (5 items + each), reported all 30 bad saves as done. `verify-run` flagged all 30 and + confirmed all 10 clean saves, with no false alarms. The app shows "saved" + before the server answers, so this measures what the screen can't show, not + the agent's care. Not yet run on OpenEMR: + [SCOREBOARD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/scoreboard/SCOREBOARD.md#the-agents-claims-versus-the-read-back). + +## Limits - No commercial EMR, claims system or real Citrix session has been tested. -- In review: self-repair across several new steps - ([#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325)). In - progress: the scoreboard. -- Windows installers are unsigned until free code signing through SignPath - Foundation is approved; macOS builds are ad-hoc signed. +- Windows installers are unsigned, and macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The Evidence column says which. @@ -1128,6 +1277,30 @@ halts with a named reason when the confirmation is missing before and after numbers are in [`benchmark/known_ids/KNOWN_IDS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../benchmark/known_ids/KNOWN_IDS.md). +Before a step stops on a look-alike identifier, the replayer takes extra looks +that change nothing +([`openadapt_flow/runtime/identity_looks.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/identity_looks.py)): +the same read on two fresh screenshots, the 2x re-read, the known-ID list when +one is configured, and, where the surface allows it, a select-and-copy of the +identifier's text box (`runtime.identity_copy_look`). OCR looks can't settle +the record on their own, because a look-alike twin can read the same on every +frame and at every size. Only two looks can: the known-ID list, and a copied +identifier that equals a run parameter in the identity band. A look that names +another record stops the run. Each look is recorded on the step's identity +check. + +The copy look has limits. It needs a surface where selecting the identifier +and copying it changes nothing, and a clipboard that reaches the machine Flow +runs on (RDP or Citrix clipboard redirection). It drags across only the one +text box that holds the identifier, and the run stops if the screen changes +anywhere else during the look. A canvas with no selectable text has no copy +look. A recording's own identifier is OCR text and can be a misread (the RDP +kiosk recorded `A1001` as `Alo01`), so in a workflow that always opens the +recorded record, the copy can stop a run but never confirm one. It confirms +only a record chosen by a run parameter. Governed and qualification runs never +take it. The check just before the click takes no input; it reuses the copy +only when the identifier's pixels are unchanged since the look. + Permissive `replay` may proceed on an unreadable reversible target and reports that condition. Governed `run` requires an affirmative live verdict for every identity-required admitted step; mismatch, unreadable, or abstain halts before @@ -1156,6 +1329,39 @@ irreversible, and qualification requires its record check. An icon-only click that only navigates after data entry is therefore over-classified until an operator reclassifies it. +On a pixel surface an icon-only click carries no text that names its control. +When the recorder can move the pointer without clicking +(`Recorder(capture_hover=True)` on a backend with `hover`), it rests the +pointer on each click target first and keeps that frame. For a click with no +other identity evidence, the compiler stores what changed near the target, such +as the control's highlight or tooltip, as the step's hover evidence +([`openadapt_flow/runtime/hover_evidence.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/hover_evidence.py)). +At replay, on the pixel rungs, the runtime rests the pointer on the resolved +target, waits the same 0.8 s, and compares. A different response, or a +response where the recording saw none, stops the run before the click. A +missing response, or one that can't be compared (another render with no +readable text), is recorded on the step and changes nothing. A tooltip that +opens later than the wait looks like a different response and stops the run. +Controls that don't answer the pointer, such as most text fields, get no +evidence this way. The browser, remote-display (Citrix and RDP client window) +and FreeRDP backends can hover. The Aardwolf RDP transport and the native +Windows, macOS and Linux backends can't move the pointer without a button +yet, so they take no hover look. + +A pixel step is checked on one frame and clicked a moment later. Consequential +identity-gated clicks were already bound to the exact checked frame. Every +other pixel click now gets a check-to-click guard (`Replayer._pixel_recheck`): +right before the click, the runtime takes a fresh frame and compares the +target region, the identity region and the hover response with the frame the +step was checked on. If one changed (a dialog opened, the list re-sorted), it +doesn't click. It re-resolves the target and re-runs the identity check on +the fresh frame, with no input, and clicks only when the next frame matches +that one. After two re-observations that still see a change, or when the +target or its identity can't be confirmed again, it refuses the click. Each +change is recorded on the step (`StepResult.check_to_click`). A change smaller +than 4 pixels, or under 24 levels per channel, doesn't count. Structured +targets are bound by element tokens instead and aren't rechecked. + A consequential click must be tied to a record before Flow delivers it. Inside a table row, the row is that record. An icon-only Save that floats beside a form (openIMIS's claim Save has no text, accessible name, or tooltip) has no diff --git a/llms.txt b/llms.txt index ad3bd008a..bc7d61c50 100644 --- a/llms.txt +++ b/llms.txt @@ -2,14 +2,14 @@ > A person, or a computer-use agent once, shows a task in an application with no usable API. OpenAdapt compiles it into a program that makes the same entry every time with no model calls, confirms the right record before acting, reads the saved record back from the system of record, and stops instead of improvising or retrying. -This is the llms.txt for the OpenAdapt monorepo (github.com/OpenAdaptAI/OpenAdapt). Start with the capability map: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md. It lists every capability with its status (released, in review, planned), its evidence (real app, synthetic, untested) and where it lives. Check it before concluding that OpenAdapt lacks something. +This is the llms.txt for the OpenAdapt monorepo (github.com/OpenAdaptAI/OpenAdapt). Start with the capability map: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md. It lists every capability with its release (released, or on main for the next release), its evidence (real app, synthetic, untested) and where it lives. Check it before concluding that OpenAdapt lacks something. ## OpenAdapt in ten capabilities Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" -means it ships in the current PyPI packages; a pull request number means it is -being merged now. +means it ships in the current PyPI packages; "on main" means it ships in the +next release. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -72,7 +72,7 @@ being merged now. | Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md) | -| Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Replaces a single HTML form submission with the request the app itself sends, compiled from the network traffic captured while recording, then reads the record back. | `api`, [API_TIER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/API_TIER.md) | | Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) | | Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) | | Binds each approval to the exact screenshot and form state, and re-reads the form before saving. | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md) | diff --git a/packages/agent/README.md b/packages/agent/README.md index 800c75c27..e09dd2acc 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -284,7 +284,7 @@ Flow's [execution profiles](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/p | `--timeout SECONDS` | How long one run may work (default 600). Time a person spends approving a save doesn't count. A run stopped at the limit is `not_sure_if_saved`. | | `--approval-mode MODE` | Whether each save waits for approval. Without it, Flow decides, and a run stops before each save until it's approved (`each_item`). `not_gated` turns that off for every run this server starts; each result records it under `technical.approval_mode`. | | `--delegate-approvals AGENT_ID` | Lets your agent answer a waiting save as `AGENT_ID` (an opaque id such as `agent:intake-bot`), with `list_approval_requests` and `answer_approval_request`. Flow accepts the answer only where the run's approver policy allows an agent for that step, and records the agent as the approver. By default only a person may answer. Needs openadapt-flow 2.2.0 or later. | -| `--admissions DIR` | Adds `prepare_admission`, `list_admissions` and `decide_admission`, the same steps as `openadapt-flow admit`, over recordings in `DIR/recordings` and shadow candidates in `DIR/shadow-candidates`. A person decides in your client's form. With `--delegate-approvals` and `--approver-grant`, your agent decides as that approver instead, under a grant a person signed that names it. It never decides an admission it prepared or one prepared on a server without `--delegate-approvals`, and never under the `regulated` policy pack. Accepting sets the workflow's runs to `problems_only`. The record names who decided, and for your agent the grant and who signed it. Not released yet. | +| `--admissions DIR` | Adds `prepare_admission`, `list_admissions` and `decide_admission`, the same steps as `openadapt-flow admit`, over recordings in `DIR/recordings` and shadow candidates in `DIR/shadow-candidates`. A person decides in your client's form. With `--delegate-approvals` and `--approver-grant`, your agent decides as that approver instead, under a grant a person signed that names it. It never decides an admission it prepared or one prepared on a server without `--delegate-approvals`, and never under the `regulated` policy pack. Accepting sets the workflow's runs to `problems_only`. The record names who decided, and for your agent the grant and who signed it. | | `--approver-grant FILE` | With `--admissions` and `--delegate-approvals`: the approver grant a person signed (`openadapt-flow approver-grant sign`) that lets your agent decide admissions. Flow checks it on every decision. Without it only a person decides. | | `--headed` | Shows the browser. A person needs it to continue or skip a stopped run in a web app. | | `--extra-run-arg ARG`, `--flow-cli CMD` | Change how Flow is called. Either one can relax Flow's checks, for example `--extra-run-arg=--allow-unencrypted`. | diff --git a/tests/test_shipped_only_copy.py b/tests/test_shipped_only_copy.py new file mode 100644 index 000000000..a9d5252a2 --- /dev/null +++ b/tests/test_shipped_only_copy.py @@ -0,0 +1,39 @@ +"""Public copy describes the product as it ships. + +Never publish unreleased or in-progress work on a public surface: no "being +built", "not released", "in review", "coming soon", "being merged" or roadmap +lists. Track gaps internally (issues, private notes) and build them instead. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + +PUBLIC = [ + ROOT / "README.md", + ROOT / "CAPABILITIES.md", + ROOT / "llms.txt", + *sorted((ROOT / "packages").glob("*/README.md")), +] + +NEVER = re.compile( + r"\bbeing built\b|\bnot (?:yet )?released\b|\bnot released yet\b" + r"|\bcoming soon\b|\bin review\b|\bin progress\b|\bbeing merged\b" + r"|\bbeing added\b|\bwhen it ships\b|\bnot done yet\b|\bon (?:our|the) roadmap\b", + re.IGNORECASE, +) + + +def test_public_copy_lists_only_shipped_work() -> None: + hits = [] + for path in PUBLIC: + if not path.exists(): + continue + for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + match = NEVER.search(line) + if match: + hits.append(f"{path.relative_to(ROOT)}:{number}: {match.group(0)!r}") + assert hits == [], "unreleased-work copy on a public surface:\n" + "\n".join(hits)