diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 2be355e42..94b152ca3 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -13,8 +13,8 @@ an older version or an older page: check here, then the linked code. Each capability is implemented in the [OpenAdapt monorepo](https://github.com/OpenAdaptAI/OpenAdapt). "Released" -means it ships in the current PyPI packages; a pull request number means it is -being merged now. +means it ships in the current PyPI packages; "on main" means merged and +shipping in the next release; a pull request number means it is being merged. 1. **Learns from any successful run, a person's or an agent's.** A person records a task, or a computer-use agent does it once and OpenAdapt compiles @@ -33,8 +33,8 @@ being merged now. halts fixed the same way (`teach-branch`), candidate branches mined from many real runs in the background (shadow mode, [#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)), and a router - that sends each item to the right verified variant (landing). Released - except where a PR is named. + that sends each item to the right verified variant. Shadow mining and the + router are on main; the rest is released. 4. **Works across surfaces in one workflow.** Web, native Windows and macOS, RDP and Citrix. Programs chain across surfaces passing confirmed values (`compose`, `process`, released), and one program can switch surfaces, tabs @@ -47,8 +47,8 @@ being merged now. 6. **Stops instead of improvising, and never retries an uncertain save.** A run that can't prove a save ends `HALTED_BEFORE_EFFECT` or `RECONCILIATION_REQUIRED`, never "success". A ledger of what changed before - a stop, with undo steps a person approves, is landing. Released except the - ledger. + a stop, with undo steps a person approves, is on main. The rest is + released. 7. **Approvals are bound to exactly what will be saved.** Each save waits for a person by default and can be answered on a paired phone with one card per decision. The approval is bound to the exact screenshot and form state, and @@ -67,14 +67,14 @@ being merged now. 10. **Runs locally and keeps data private.** Healthy runs make no model calls and send nothing out. Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the - machine is landing); a strict no-outbound mode, an unattended on-prem + machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. ## Often missed | What OpenAdapt does | Where | | --- | --- | -| Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md) | +| Learns branching programs from several demonstrations, from repeated halts, and from many background runs. | `induce`, `teach-branch`, [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Chains programs across surfaces, and one program can switch surfaces, tabs and popups mid-run. | `compose`, `process`, [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Uses page structure first on web, Windows, macOS and Linux, with pixels as the fallback for remote desktops. Benchmarks labelled "vision-only" ran a deliberately pixel-only arm. | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md) | | Runs steps as direct API calls; compiling them from captured network traffic is being merged. | `runtime/actuators/api.py`, [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | @@ -92,6 +92,7 @@ Every capability, with its status, its evidence and where it lives. Status: - **Released**: in the current PyPI release (openadapt-flow 2.2.0). +- **On main**: merged; ships in the next release. - **In review**: an open pull request. - **Planned**: designed, not built. @@ -121,9 +122,14 @@ improvising or retrying when something is off. | Never retry an uncertain save (`RECONCILIATION_REQUIRED`) | Released | Synthetic (fault server) | [VERIFICATION.md](packages/flow/docs/VERIFICATION.md) | | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` | | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md) | -| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | In review ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | +| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | -| Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | Planned | Untested | none yet | +| Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | +| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | In review ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | +| Learn the record check from the recording, when the person looks the record up after saving | In review ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | +| Checked later: a save confirmed by a later report or acknowledgment, never counted as done until then | In review ([#1297](https://github.com/OpenAdaptAI/OpenAdapt/pull/1297)) | Synthetic | `SAVED_AWAITING_CHECK` | +| Keep the server's own answer to a save as lower-level evidence on the receipt | In review ([#1298](https://github.com/OpenAdaptAI/OpenAdapt/pull/1298)) | Synthetic | receipts | +| A ledger of what changed before a stop, a write-ordering lint, and undo steps a person approves | On main ([#1263](https://github.com/OpenAdaptAI/OpenAdapt/pull/1263)) | Synthetic (MockMed) | change ledger | ## Approvals @@ -132,8 +138,8 @@ improvising or retrying when something is off. | Each save waits for a person by default | Released | Synthetic | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | | Approve or decline a save on a paired phone, with one card per decision and swipe or tap | Released (Desktop) | Synthetic | openadapt-desktop `portal/shell` | | Person-signed grants let named agents or systems answer saves | Released | Unit tests | `openadapt-flow approver-grant` | -| Admit once, then stop only on problems | Released, behind a readiness gate | Synthetic | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | -| Show the field values a save writes on the phone | In review ([#1257](https://github.com/OpenAdaptAI/OpenAdapt/pull/1257)) | Unit tests | `console/save_approvals.py` | +| Admit once, then stop only on problems | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)), behind a readiness gate | Synthetic | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md) | +| Show the field values a save writes on the phone | On main ([#1257](https://github.com/OpenAdaptAI/OpenAdapt/pull/1257)) | Unit tests | `console/save_approvals.py` | ## Branching work, not one linear path @@ -143,10 +149,11 @@ improvising or retrying when something is off. | Learn a guarded branch from several halts fixed the same way | Released | Synthetic | `openadapt-flow teach-branch`, `learning/branch_teach.py` | | Run a program over a worklist | Released | Real app | `openadapt-flow for-each` | | Typed business decisions inside a run | Released | Synthetic | `business_decision` states, `openadapt-flow business-decisions` | -| Watch many real runs in the background and mine candidate branches (shadow mode) | Capture released; drafts in review ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md) | +| Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | -| Use spoken narration while recording to learn why a branch was taken | Planned | Untested | Capture records audio (`--audio`, local Whisper). The compiler doesn't use it yet. | -| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | Planned | Untested | none yet (today `induce` names the recording that is missing) | +| Use spoken narration while recording to propose the rule behind a branch | In review ([#1291](https://github.com/OpenAdaptAI/OpenAdapt/pull/1291)) | Synthetic (generated audio) | `openadapt-capture[audio,privacy]` | +| Settle a disagreement between recordings by trying both paths on a test copy of the app and reading the record back | In review ([#1276](https://github.com/OpenAdaptAI/OpenAdapt/pull/1276)) | Synthetic (MockMed) | `induce --probe-on` | +| Route each item to the right verified variant of a program | On main ([#1264](https://github.com/OpenAdaptAI/OpenAdapt/pull/1264)) | Synthetic | `variants`, `route` | ## Surfaces @@ -158,7 +165,7 @@ improvising or retrying when something is off. | RDP (pixels; Aardwolf or headless FreeRDP with a session pool) | Released, early | Real Windows 11 host, 3 of 3, plus a FreeRDP batch | [SURFACES.md](packages/flow/docs/SURFACES.md), `backends/xfreerdp.py` | | Citrix | Exploratory | A no-DOM stand-in only; no real ICA/HDX session yet | [LIMITS.md](packages/flow/docs/LIMITS.md) | | Chain programs across surfaces, passing confirmed values | Released | Synthetic | `openadapt-flow compose`, `process`; [PROCESS_CONTRACT.md](packages/flow/docs/PROCESS_CONTRACT.md) | -| One program that switches surfaces, tabs and popups mid-run | In review ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | +| One program that switches surfaces, tabs and popups mid-run | On main ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) | | Halt when a workflow qualified with page structure runs on a pixels-only surface | Released | Synthetic | `openadapt-flow qualify` | | Lift look-alike identity stops with a complete known-ID list | Released | Synthetic (rendered record list) | `openadapt-flow known-ids` | @@ -168,8 +175,9 @@ improvising or retrying when something is off. | --- | --- | --- | --- | | Page structure (DOM, UI Automation, Accessibility) first, then template, OCR and geometry | Released | Real app | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md) | | Run a step as a direct API call (`ApiActuator`, `ApiBinding`) | Released, hand-authored bindings | Synthetic benchmarks | `runtime/actuators/api.py` | -| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1261](https://github.com/OpenAdaptAI/OpenAdapt/pull/1261)) | Synthetic patients on a local OpenEMR test copy | [#1261](https://github.com/OpenAdaptAI/OpenAdapt/pull/1261) | -| Fall back from the API step to page structure when the API is unavailable, before anything is sent | Planned | Untested | none yet | +| Capture a recording's network traffic and compile a clean save into an API step, with the same record check | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic patients on a local OpenEMR test copy | network capture, `ApiBinding` | +| Fall back from the API step to page structure when the API is unavailable, before anything is sent | In review ([#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269)) | Synthetic | [#1269](https://github.com/OpenAdaptAI/OpenAdapt/pull/1269) | +| Run a worklist on several workers at once, one signed-in session each, with a stop on any worker stopping all | In review ([#1296](https://github.com/OpenAdaptAI/OpenAdapt/pull/1296)) | Synthetic | `--workers` | ## When the app changes @@ -179,7 +187,9 @@ improvising or retrying when something is off. | A calling agent demonstrates the fix for a halted run; the result is a repair candidate | Released (2.1.0) | Synthetic | `run --hold-on-halt`, `repair/agent_teach.py` | | Font and layout drift halts with a named reason | Released | Synthetic | `runtime/layout_drift.py` | | Self-repair: an agent proposes a fix for one halted click, Flow probes it up to the save without saving, a person approves | In review ([#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265)) | Synthetic (MockMed redesign) | `run --self-repair`, [#1265](https://github.com/OpenAdaptAI/OpenAdapt/pull/1265) | -| Self-repair across several new steps (a new dialog, a control moved into a menu) | Planned | Untested | none yet | +| Self-repair across several new steps (a new dialog, a control moved into a menu) | In progress | Synthetic (MockMed drift modes) | bridge branch, stacked on #1265 | +| Keep pixel-path runs working across display scaling and font smoothing changes | In review ([#1286](https://github.com/OpenAdaptAI/OpenAdapt/pull/1286)) | Synthetic (RDP and Citrix-style renderings) | `vision/match.py` | +| Pre-approved repairs: a signed policy lists low-risk changes that apply without a person, a test-record run before each queue, and one approval for every item halted the same way | In review ([#1294](https://github.com/OpenAdaptAI/OpenAdapt/pull/1294)) | Synthetic | standard changes | | Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents @@ -189,15 +199,16 @@ improvising or retrying when something is off. | Compile a computer-use agent's successful run into a program (JIT); later runs make no model calls | Released | Synthetic (a recorded Claude run on MockMed) | `openadapt_flow.jit`, [JIT.md](packages/flow/docs/JIT.md) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](packages/agent/README.md) | -| Live agent loop end to end on a real app, plus a launch demo (agent once, OpenAdapt 100 times) | In progress | Untested | none yet | +| Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Synthetic | `compile-agent` | +| One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | ## Recording, privacy and proof | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Record mouse, keyboard, screen video and accessibility trees on one clock (Windows, macOS, Linux X11) | Released (capture 1.3.0) | Real OS | [packages/capture](packages/capture/README.md); Wayland is not supported | -| Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [openadapt-privacy](https://github.com/OpenAdaptAI/openadapt-privacy), [PRIVACY.md](packages/flow/docs/PRIVACY.md) | -| Scrub by default before anything leaves the machine | Planned | Untested | Today scrubbing is opt-in (`OPENADAPT_FLOW_SCRUB=on`), and `capture share` does not scrub | +| Find and redact personal and health data in text, screenshots and structure | Released (privacy 1.1.3) | Unit tests | [packages/privacy](packages/privacy/README.md), [PRIVACY.md](packages/flow/docs/PRIVACY.md) | +| Scrub by default before anything leaves the machine | On main ([#1262](https://github.com/OpenAdaptAI/OpenAdapt/pull/1262)) | Unit tests | `openadapt-privacy` | | Signed receipts (Seals), countersigned and timestamped, offline by default | Released | Unit tests, real TSA tokens | `openadapt-flow seal`, `countersign`, `local-seal` | | Strict no-outbound mode and an unattended on-prem service | Released | Synthetic | `openadapt-flow no-outbound`, `service`; [ON_PREM.md](packages/flow/docs/ON_PREM.md) | @@ -210,7 +221,7 @@ fails when a command is added without being listed here. - **Check and admit:** `lint`, `scaffold-verifier`, `check`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. - **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`. - **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). -- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`. +- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). - **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. - **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`. - **Run on premises:** `service`, `no-outbound`. @@ -226,8 +237,8 @@ fails when a command is added without being listed here. took 38.6 s, slower than the agent. Raw data and figures: [#1251](https://github.com/OpenAdaptAI/OpenAdapt/pull/1251) (`benchmark/openemr_referral_timing/` once merged). -- **Fault test.** In 90 runs with injected faults, 72 left a wrong record - behind. A check that trusts the screen's "Saved" banner reported 54 of those +- **Fault test.** We injected 8 kinds of bad save, 9 runs each, alongside + clean runs: 72 runs left a wrong record behind. A check that trusts the screen's "Saved" banner reported 54 of those as success; reading the record back over the app's REST API missed 9; the complete database read path caught all 72. This is a coverage test over a small, hand-authored set of faults, not a real-world error rate: @@ -236,9 +247,8 @@ fails when a command is added without being listed here. ## What is not done yet - No commercial EMR, claims system or real Citrix session has been tested. -- Not built yet: record checks built from the app's own reads, self-repair - across several new steps, settling a disagreement between recordings on a - test copy, and narration-driven rules. Network capture to API calls and - scrub-by-default are in review. +- In progress: self-repair across several new steps, and the scoreboard. +- Windows installers are unsigned until free code signing through SignPath + Foundation is approved; macOS builds are ad-hoc signed. - Several capabilities have run only on OpenAdapt's synthetic apps. The Evidence column says which. diff --git a/llms.txt b/llms.txt index 8c041ca0d..ad3bd008a 100644 --- a/llms.txt +++ b/llms.txt @@ -28,8 +28,8 @@ being merged now. halts fixed the same way (`teach-branch`), candidate branches mined from many real runs in the background (shadow mode, [#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)), and a router - that sends each item to the right verified variant (landing). Released - except where a PR is named. + that sends each item to the right verified variant. Shadow mining and the + router are on main; the rest is released. 4. **Works across surfaces in one workflow.** Web, native Windows and macOS, RDP and Citrix. Programs chain across surfaces passing confirmed values (`compose`, `process`, released), and one program can switch surfaces, tabs @@ -42,8 +42,8 @@ being merged now. 6. **Stops instead of improvising, and never retries an uncertain save.** A run that can't prove a save ends `HALTED_BEFORE_EFFECT` or `RECONCILIATION_REQUIRED`, never "success". A ledger of what changed before - a stop, with undo steps a person approves, is landing. Released except the - ledger. + a stop, with undo steps a person approves, is on main. The rest is + released. 7. **Approvals are bound to exactly what will be saved.** Each save waits for a person by default and can be answered on a paired phone with one card per decision. The approval is bound to the exact screenshot and form state, and @@ -62,7 +62,7 @@ being merged now. 10. **Runs locally and keeps data private.** Healthy runs make no model calls and send nothing out. Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the - machine is landing); a strict no-outbound mode, an unattended on-prem + machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. ## Often missed