diff --git a/CAPABILITIES.md b/CAPABILITIES.md index c42ee9af9..415000e81 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -124,6 +124,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](packages/flow/README.md) | +| Classify common local edits without a person (select-all, Enter in a field or dropdown, an in-form tab, an informational dialog's dismiss), only when the recording's network capture shows no write and a runtime check confirms only the expected change | In review ([#1366](https://github.com/OpenAdaptAI/OpenAdapt/pull/1366)) | Synthetic; OpenEMR referral recording re-admitted offline (violations 12 to 1 under `permissive`) | [LOCAL_EDIT_CLASSIFICATION.md](packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md), `compiler/local_edit.py` | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](packages/flow/docs/VERIFICATION.md), `runtime/effects/` | | Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | diff --git a/llms-full.txt b/llms-full.txt index 7f4752fa6..5a0686033 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -416,9 +416,10 @@ shipping in the next release; a pull request number means it is being merged. review, campaign, canary, rollback), agent-demonstrated fixes for a halted run, and fleet drift patches shared with no patient data are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save - without saving, is on main; fixes of up to three new steps are in review. -9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded - save around any agent's write + without saving, is on main, and so are fixes of up to three new steps. +9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task + ledger that keeps a long job's rules, items, questions and checked saves on + the server ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)), a guarded save around any agent's write ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. @@ -480,6 +481,7 @@ improvising or retrying when something is off. | Capability | Status | Evidence | Where | | --- | --- | --- | --- | | Compile one or more demonstrations into a deterministic program | Released | Real app (OpenEMR referral, 62 timed runs) | `openadapt-flow record`, `compile`; [packages/flow](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/README.md) | +| Classify common local edits without a person (select-all, Enter in a field or dropdown, an in-form tab, an informational dialog's dismiss), only when the recording's network capture shows no write and a runtime check confirms only the expected change | In review ([#1366](https://github.com/OpenAdaptAI/OpenAdapt/pull/1366)) | Synthetic; OpenEMR referral recording re-admitted offline (violations 12 to 1 under `permissive`) | [LOCAL_EDIT_CLASSIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md), `compiler/local_edit.py` | | Right-record check before any write (structure, then OCR, then pixels) | Released | Real app and synthetic | [RESOLUTION_LADDER.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/RESOLUTION_LADDER.md), `runtime/identity.py` | | Read the saved record back from the system of record (API, database, FHIR, separate session) | Released | Real app (OpenEMR 12 of 12 fields) | [VERIFICATION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION.md), `runtime/effects/` | | Read values out of an app as a workflow step (READ), with a records format, a signed review queue and export to CSV or a database | On main ([#1229](https://github.com/OpenAdaptAI/OpenAdapt/pull/1229), [#1215](https://github.com/OpenAdaptAI/OpenAdapt/pull/1215), [#1218](https://github.com/OpenAdaptAI/OpenAdapt/pull/1218)–[#1224](https://github.com/OpenAdaptAI/OpenAdapt/pull/1224)) | Synthetic (mock desktop apps) | `openadapt_flow/interchange/` | @@ -574,6 +576,7 @@ improvising or retrying when something is off. | Code as action: the run-1 agent acts through one terminal tool (structure first, actions by role and name, a sandboxed read-only shell, screenshots on request); every action is a structured step checked against the compiled program | In review ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | Real app (local OpenEMR) and synthetic | `ClaudeAgent(vision="code")`, `CODE_AS_ACTION.md` ([#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)) | | Admit a draft by a person, an admitter key, or an agent under a person-signed policy, including self-admission when the policy allows it | Released | Unit tests | [JIT_ADMISSION.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT_ADMISSION.md) | | MCP server: your agent decides what to enter, OpenAdapt makes the entry and checks it | Released | Synthetic sandbox | `openadapt-agent`, [packages/agent](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md) | +| Task ledger per job: rules locked when the job opens and enforced on every run, item status set only from OpenAdapt's own results, and yes-or-no questions a person answers on the phone or in Desktop, including when a run stops | In review ([#1332](https://github.com/OpenAdaptAI/OpenAdapt/pull/1332)) | Synthetic (30-item MockMed worklist, scripted agent, no model calls) | MCP tools `open_job`, `get_job`, `add_items`, `record_evidence`, `ask_user`, `get_answer`, `change_constraint`, `close_job`; [Track a long job](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/agent/README.md#track-a-long-job) | | Compile an agent's first verified run from a prompt (`compile-agent`), and a code-as-action authoring agent | In review ([#1277](https://github.com/OpenAdaptAI/OpenAdapt/pull/1277), [#1305](https://github.com/OpenAdaptAI/OpenAdapt/pull/1305)); the agent harness behind it runs on every backend, on main ([#1283](https://github.com/OpenAdaptAI/OpenAdapt/pull/1283)) | Synthetic | `compile-agent` | | Score harness optimizers (Harness², Meta-Harness) on GUI tasks with the record read-back as the reward, with full trace export | On main ([#1299](https://github.com/OpenAdaptAI/OpenAdapt/pull/1299), [#1300](https://github.com/OpenAdaptAI/OpenAdapt/pull/1300), [#1301](https://github.com/OpenAdaptAI/OpenAdapt/pull/1301)) | Synthetic (scripted proposer) | `integrations/harness2/` | | One scoreboard comparing the agent alone with OpenAdapt across tasks (correctness by read-back, cost, time, items per hour) | In progress ([#1303](https://github.com/OpenAdaptAI/OpenAdapt/pull/1303)) | Synthetic and local OpenEMR and openIMIS test copies | `benchmark/scoreboard/` | @@ -591,19 +594,112 @@ improvising or retrying when something is off. ## Every command, by capability -Each `openadapt-flow` command, grouped by the capability it serves. A test -fails when a command is added without being listed here. - -- **Record and build a program:** `record`, `demo-record`, `compile`, `induce`, `disambiguate`, `for-each`, `compose`, `process`, `intake`, `visualize`, `pull`, `variants`, `route`, `compile-agent`. -- **Check and admit:** `lint`, `scaffold-verifier`, `check`, `api`, `bind-record-checks`, `certify`, `qualify`, `propose-qualification`, `known-ids`, `jit`, `admit`, `approver-grant`. -- **Run and answer:** `run`, `replay`, `resume`, `approve`, `item-approval`, `triage`, `business-decisions`, `explain`, `console`, `deferred`. -- **Judge any executor's write:** `verify-run`, `serve-reward` (scores agent training episodes by reading the system of record). -- **Repair and learn from halts:** `teach`, `teach-branch`, `repair`, `fleet`, `undo` (the undo steps a person approves after a stop). -- **Give agents the program:** `emit-skill`, `emit-mcp`, `serve-execute`. -- **Proof:** `seal`, `local-seal`, `countersign`, `attach-seal`. -- **Run on premises:** `service`, `no-outbound`. -- **Share safely with OpenAdapt Cloud:** `sanitize`, `review-sanitized`, `approve-sanitized`, `materialize-approved`, `push`, `validate-hosted`, `report-break`, `report-run`, `connect`, `login`, `connector`. -- **Try it and measure it:** `demo`, `tutorial`, `bench`, `benchmark`. +Each `openadapt-flow` command, grouped by the capability it serves. Each +command has one file under +[`packages/flow/docs/commands/`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/commands/): to add a +command, add `/.md` and run `python scripts/build_command_map.py`. +A test fails when a command has no file or this list is out of date. + + + +**Record and build a program:** + +- `compile-agent` +- `compile` +- `compose` +- `demo-record` +- `disambiguate` +- `for-each` +- `induce` +- `intake` +- `process` +- `pull` +- `record` +- `route` +- `variants` +- `visualize` + +**Check and admit:** + +- `admit` +- `api` +- `approver-grant` +- `bind-record-checks` +- `certify` +- `check` +- `jit` +- `known-ids` +- `lint` +- `propose-qualification` +- `qualify` +- `scaffold-verifier` + +**Run and answer:** + +- `approve` +- `business-decisions` +- `console` +- `deferred` +- `explain` +- `item-approval` +- `replay` +- `resume` +- `run` +- `triage` + +**Judge any executor's write:** + +- `serve-reward`: Scores agent training episodes by reading the system of record. +- `verify-run` + +**Repair and learn from halts:** + +- `fleet` +- `repair` +- `teach-branch` +- `teach` +- `undo`: The undo steps a person approves after a stop. + +**Give agents the program:** + +- `emit-mcp` +- `emit-skill` +- `serve-execute` + +**Proof:** + +- `attach-seal` +- `countersign` +- `local-seal` +- `seal` + +**Run on premises:** + +- `no-outbound` +- `service` + +**Share safely with OpenAdapt Cloud:** + +- `approve-sanitized` +- `connect` +- `connector` +- `login` +- `materialize-approved` +- `push` +- `report-break` +- `report-run` +- `review-sanitized` +- `sanitize` +- `validate-hosted` + +**Try it and measure it:** + +- `bench` +- `benchmark` +- `demo` +- `tutorial` + + ## Measured results @@ -1128,6 +1224,30 @@ halts with a named reason when the confirmation is missing before and after numbers are in [`benchmark/known_ids/KNOWN_IDS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../benchmark/known_ids/KNOWN_IDS.md). +Before a step stops on a look-alike identifier, the replayer takes extra looks +that change nothing +([`openadapt_flow/runtime/identity_looks.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/identity_looks.py)): +the same read on two fresh screenshots, the 2x re-read, the known-ID list when +one is configured, and, where the surface allows it, a select-and-copy of the +identifier's text box (`runtime.identity_copy_look`). OCR looks can't settle +the record on their own, because a look-alike twin can read the same on every +frame and at every size. Only two looks can: the known-ID list, and a copied +identifier that equals a run parameter in the identity band. A look that names +another record stops the run. Each look is recorded on the step's identity +check. + +The copy look has limits. It needs a surface where selecting the identifier +and copying it changes nothing, and a clipboard that reaches the machine Flow +runs on (RDP or Citrix clipboard redirection). It drags across only the one +text box that holds the identifier, and the run stops if the screen changes +anywhere else during the look. A canvas with no selectable text has no copy +look. A recording's own identifier is OCR text and can be a misread (the RDP +kiosk recorded `A1001` as `Alo01`), so in a workflow that always opens the +recorded record, the copy can stop a run but never confirm one. It confirms +only a record chosen by a run parameter. Governed and qualification runs never +take it. The check just before the click takes no input; it reuses the copy +only when the identifier's pixels are unchanged since the look. + Permissive `replay` may proceed on an unreadable reversible target and reports that condition. Governed `run` requires an affirmative live verdict for every identity-required admitted step; mismatch, unreadable, or abstain halts before @@ -1156,6 +1276,39 @@ irreversible, and qualification requires its record check. An icon-only click that only navigates after data entry is therefore over-classified until an operator reclassifies it. +On a pixel surface an icon-only click carries no text that names its control. +When the recorder can move the pointer without clicking +(`Recorder(capture_hover=True)` on a backend with `hover`), it rests the +pointer on each click target first and keeps that frame. For a click with no +other identity evidence, the compiler stores what changed near the target, such +as the control's highlight or tooltip, as the step's hover evidence +([`openadapt_flow/runtime/hover_evidence.py`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/../openadapt_flow/runtime/hover_evidence.py)). +At replay, on the pixel rungs, the runtime rests the pointer on the resolved +target, waits the same 0.8 s, and compares. A different response, or a +response where the recording saw none, stops the run before the click. A +missing response, or one that can't be compared (another render with no +readable text), is recorded on the step and changes nothing. A tooltip that +opens later than the wait looks like a different response and stops the run. +Controls that don't answer the pointer, such as most text fields, get no +evidence this way. The browser, remote-display (Citrix and RDP client window) +and FreeRDP backends can hover. The Aardwolf RDP transport and the native +Windows, macOS and Linux backends can't move the pointer without a button +yet, so they take no hover look. + +A pixel step is checked on one frame and clicked a moment later. Consequential +identity-gated clicks were already bound to the exact checked frame. Every +other pixel click now gets a check-to-click guard (`Replayer._pixel_recheck`): +right before the click, the runtime takes a fresh frame and compares the +target region, the identity region and the hover response with the frame the +step was checked on. If one changed (a dialog opened, the list re-sorted), it +doesn't click. It re-resolves the target and re-runs the identity check on +the fresh frame, with no input, and clicks only when the next frame matches +that one. After two re-observations that still see a change, or when the +target or its identity can't be confirmed again, it refuses the click. Each +change is recorded on the step (`StepResult.check_to_click`). A change smaller +than 4 pixels, or under 24 levels per channel, doesn't count. Structured +targets are bound by element tokens instead and aren't rechecked. + A consequential click must be tied to a record before Flow delivers it. Inside a table row, the row is that record. An icon-only Save that floats beside a form (openIMIS's claim Save has no text, accessible name, or tooltip) has no diff --git a/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md new file mode 100644 index 000000000..4b94b33fc --- /dev/null +++ b/packages/flow/docs/design/LOCAL_EDIT_CLASSIFICATION.md @@ -0,0 +1,149 @@ +# Classifying local edits without a person + +Status: approved by the founder on 2026-10-19, with a strict rule: the +compiler classifies a step only when the recording shows no mutating request +sent for it **and** a runtime check confirms only the expected local change. +Anything else goes to a person. + +## The problem + +A compiled program flags every special key and every unlabelled or +navigation-shaped click for review (`risk_review_required`), because its +risk can't be proven from the step alone. The run gate then refuses the +bundle until a person classifies each one. On the OpenEMR referral that is 12 +steps (blocker section A in the referral admission list): + +- 8 `Control+a` presses, each in a text field right before typing into it; +- 4 `Enter` presses, each right after typing an option's label into a + dropdown; + +plus, under `clinical-write`, the same keys fail `prohibit_vacuous_postconditions` +because a key has no outcome check (section B), and the Contact tab click is +unarmed (section C). + +These are the same few patterns in every form-filling program, on every app. +A person approving each one, per bundle, is busywork that trains people to +approve without reading. + +## The rule + +The compiler classifies five local-edit patterns itself. Every pattern needs +all of this evidence from the recording: + +- network capture (`record --capture-network`), with no request other than + `GET`, `HEAD` or `OPTIONS` between the previous event and the next one; +- URL, title and page count recorded on both sides of the action, and + unchanged; +- when the recorder snapshotted the record store around the step, the two + snapshots equal. + +A recording without network capture classifies nothing. + +| Pattern | Step evidence | Outcome check added to the step | +| --- | --- | --- | +| `select_all`: `Control+a` / `Meta+a` | The previous step clicked a text field (role `textbox`, `searchbox` or `spinbutton`, with a selector); the next step types into it | `local_edit`: focus is still on that field; URL, title and page count unchanged. The next step's typed-input read-back then checks the field's whole value. | +| `confirm_option`: `Enter` in a dropdown | The steps before clicked a dropdown (role `combobox` or `listbox`) and typed a parameter's value | `local_edit` on the dropdown, plus `field_value`: the focused dropdown shows this run's value | +| `confirm_entry`: `Enter` in a text field | The steps before clicked a text field and typed a parameter's value | `local_edit` on the field, plus `field_value`: the focused field's value equals this run's value exactly | +| `in_form_tab`: a tab click | Role `tab`, or a `button` whose only mined check is a region change; the next acting step is in the same frame | `local_edit` (URL, title and page count unchanged; focus may move) | +| `dismiss_dialog`: an informational dialog's dismiss | A `button` or `link` labelled only "Continue", "Continue to ...", "Next", "Close", "OK", "Got it", "Dismiss", "Not now", "Maybe later" or "Skip for now" | `local_edit`, plus `text_absent` of the label: the dialog is gone | + +`continue` and `next` are write words for the risk inference, so a dismiss +labelled with them starts irreversible. It is reconsidered only when those +are its only write words and its risk is still the per-step keyword +inference. A click that a later rule made irreversible (the click that +confirms a dialog, an icon-only click presumed to save) is never +reconsidered. Because a classified dismiss is reversible, effect mining no +longer attaches a placeholder `record_written` to it (#1377). + +The runtime checks fail closed: a backend that can't read focus, the field +value, the URL, the title or the page count fails the step. + +What stays the same: + +- **The risk rule.** A step whose evidence is missing or mixed (a URL change, + a write request in the window, no network capture for a submission key, a + key that isn't one of these, a field role the compiler can't read) keeps + `risk_review_required` and goes to a person, exactly as today. +- **Write-shaped context still wins.** A key or click whose retained text + names a consequential write (`submit`, `save`, `delete`, ...) classifies + irreversible before any of this runs. The one exception is a dismiss whose + only write words are `continue` or `next`, with the full evidence above. +- **Explicit `risk_overrides` still win.** +- **D, the duplicate-prevention (idempotency) key on the save, is untouched.** + That is a business decision. + +What the run gate sees: the classified step is reversible, reviewed by the +compiler with recorded evidence (`risk_explanation` names the pattern and the +evidence), and has an outcome check, so `require_reviewed_action_risk`, +`prohibit_vacuous_postconditions` and `max_unverified_steps` pass for it. A +tab click classified this way is in-form navigation that writes nothing, so it +isn't consequential and isn't entity navigation; `prohibit_unarmed_clicks` +under `clinical-write` still applies to it, so it still needs an identity +check (decision 2 below). + +## Why this is safe enough + +- The classification is backed by what the demonstration did, not by the + label. A step that sent a write request in the recording, changed the URL, + title or page count, or changed the record store is never classified. +- The outcome check runs every time. If the same key does something else at + run time (the app changed, a different field had focus), the step fails its + postcondition and the run halts at that step. The halt comes after the key + was pressed, which is the residual risk: a key that, at run time only, + commits something. For these patterns that requires the app to have + changed the field's behavior since the recording, and the first write it + makes is still caught by the save's record check or by the next step's + checks. +- Evidence is sealed in the bundle (the step's explanation and postconditions + are part of the content digest), so a later edit to a classification + changes the digest and needs a new admission. + +## Decisions + +1. The residual risk above is accepted for these patterns (founder, + 2026-10-19). +2. `clinical-write`'s `prohibit_unarmed_clicks` still applies to a classified + tab click. On the OpenEMR referral, the Contact tab (`step_015`) still needs + an identity check or a person. +3. Network capture is required for every pattern, `Control+a` included. + +## Implementation + +- `openadapt_flow.compiler.local_edit.local_edits(steps, events, network)` + returns the classified steps with their explanation and checks. +- `compile_recording` applies them after the per-step risk inference and the + confirmation-dialog pairing, and before `risk_overrides`, which still win. + Effect mining runs after both. +- `openadapt_flow.risk.is_local_edit` and `is_sealed_dismiss` let the run + gate's risk re-inference read a classified step from what the bundle seals: + the explanation and its runtime checks. +- Postcondition kinds `local_edit` and `field_value`, evaluated by the + replayer (`_local_edit_held`, `_field_value_shown`). +- Tests: `tests/test_local_edit_classification.py`. Each pattern has a + positive case and a case where a write request was sent, which must + refuse. + +## OpenEMR referral, re-admitted on this branch + +Run gate on the recorded referral with the API step admitted +(`--identity subject_name=lname`), compared with `main`: + +| Policy | Violations on `main` | With local edits | +| --- | --- | --- | +| `permissive` | 12 | 1 | +| `clinical-write` | 32 | 9 | + +What remains: + +- `step_007` (`Control+a`): OpenEMR's background + `POST /library/ajax/execute_background_services.php` landed in its window, + so the strict rule leaves it for a person. Under `clinical-write` it also + has no outcome check. +- `step_015` (the Contact tab): classified, but unarmed under + `clinical-write`'s `prohibit_unarmed_clicks`. +- `step_039` (Create New Patient): unarmed, no identity contract and no + record check on its GUI path. +- `step_039` and `step_040`: no duplicate-prevention (idempotency) key. That + stays a person's choice. +- Encryption at rest, under the default and regulated profiles: seal the + bundle. diff --git a/packages/flow/openadapt_flow/backends/playwright_backend.py b/packages/flow/openadapt_flow/backends/playwright_backend.py index d30d4598f..d68b05d85 100644 --- a/packages/flow/openadapt_flow/backends/playwright_backend.py +++ b/packages/flow/openadapt_flow/backends/playwright_backend.py @@ -1665,6 +1665,28 @@ def focused_text_value(self) -> Optional[str]: return self._focused_element_eval(_EDITABLE_VALUE_JS) + def focused_element_matches(self, selector: str) -> Optional[bool]: + """Whether the focused element (through frames) matches ``selector``. + + None when it can't be read. Used by the ``local_edit`` postcondition. + """ + + result = self._focused_element_call( + "(el, sel) => { try { return el.matches(sel); } catch (e) { return null; } }", + arg=selector, + with_arg=True, + ) + return result if isinstance(result, bool) else None + + def focused_select_label(self) -> Optional[str]: + """The focused ``