diff --git a/packages/desktop/CHANGELOG.md b/packages/desktop/CHANGELOG.md index c6851892c..06c4668fe 100644 --- a/packages/desktop/CHANGELOG.md +++ b/packages/desktop/CHANGELOG.md @@ -1,6 +1,20 @@ # CHANGELOG +## v0.22.0 (2026-10-19) + +0.21.1 was never published: its release stopped before PyPI, because the release check accepted only Python packages and refused the Desktop installers. This release ships the 0.21.1 and 0.21.0 changes below together with these. + +### Features + +- **desktop**: Check it asks three questions; the rest comes from the recording + +### Bug Fixes + +- **desktop**: check and approve the one sealed version that runs + + + ## v0.21.1 (2026-10-19) 0.21.0 was never published, because main's macOS, Windows and Linux builds of it failed. This release ships the 0.21.0 changes below together with these fixes. diff --git a/packages/desktop/docs/OPERATOR_SCREENS.md b/packages/desktop/docs/OPERATOR_SCREENS.md index 0d1864a92..95a7f110f 100644 --- a/packages/desktop/docs/OPERATOR_SCREENS.md +++ b/packages/desktop/docs/OPERATOR_SCREENS.md @@ -12,7 +12,7 @@ calls, the safety rules it keeps, and what the engine can't do for it yet. | Sample | Watches Flow's built-in tutorial run twice on a synthetic clinic app: once normally, once with a backend that drops the save after the screen says "saved". No account and no operating-system permission. | `run_sample` | | Tasks | Sees one card per recorded task with one status (Not checked, Checking, Ready, Needs you, Needs a fix, Check the record) and one next step: Check it, Run it, Show it, or Open Needs you. Versions of a task are grouped by recording. | `get_workflows`, `get_results_summary` | | Show it | Records a task in its app. The app type is checked just in time, so browser tasks need no native permission. | `start_recording`, `stop_recording`, `compile_recording` | -| Check it | Confirms what the task changes, the right-record check, and the record check; runs one practice item; runs the safety drills; approves and locks the version. | `get_qualification`, `propose_qualification_from_demo`, `initialize_qualification`, `set_qualification_risk`, `arm_qualification_identity`, `bind_qualification_effect`, `add_qualification_case`, `run_qualification_case`, `certify_qualification`, `seal_qualification_workflow` | +| Check it | Answers three questions: is this the right record check, which values change per item, and approve. Everything else the recording shows is filled in and shown once, folded. A person's agent can answer the first two (see [SETUP_INPUTS.md](SETUP_INPUTS.md)). Without a record check learned from the recording, or after Change, it asks the parts: what the task changes, the right-record check, and the record check. | `get_check_setup`, `answer_check_setup`, `get_qualification`, `propose_qualification_from_demo`, `initialize_qualification`, `set_qualification_risk`, `arm_qualification_identity`, `bind_qualification_effect`, `seal_qualification_workflow`, `add_qualification_case`, `run_qualification_case`, `certify_qualification` | | Run it | Pastes a list, opens a CSV, or types one item, then watches per-item progress. | `get_task_inputs`, `run_item` | | Needs you | Answers each stopped item: the question, what was prevented, the stopped screen with the point ringed, and only the choices the engine allows. | `list_attention_items`, `get_attention_screenshot`, `resolve_attention_item` | | Results | Sees items done and checked, items that needed a person, wrong-record saves prevented, and a time-saved estimate whose per-item baseline is shown and editable. | `get_results_summary`, `get_workflows` | @@ -77,11 +77,14 @@ Each gap below is stated precisely so the next change can close it. `run_qualification_case` with a fault target. Where the installed Flow has no fault driver for the target type, the drill reports the engine's refusal and the exact checks page is the fallback. -4. **The record check needs a connected system of record.** When the - recording shows no system-of-record change, Check it asks plain questions - that become an effect binding. Reading the record back still needs a - verifier in the deployment settings (REST, FHIR, SQL, file, or document - hash), which IT configures once. +4. **The record check needs a read the recording holds, or a person.** When + the recording's network capture holds the read the app made after the + save, Check it proposes the record check from it, and a yes writes that + read as the record-check settings every run uses. Desktop's recorder + doesn't keep network traffic yet, and Desktop binds only REST reads, so + otherwise Check it asks plain questions that become an effect binding, + and reading the record back needs a verifier in the deployment settings + (REST, FHIR, SQL, file, or document hash), which IT configures once. 5. **The time baseline is the recording length.** Desktop doesn't measure how long the task takes by hand. Results use the person's recording as the default minutes per item and let the reader type their own number. The @@ -110,8 +113,8 @@ runs under Standard); a drill never asks. | --- | --- | --- | | Any screen, while a run waits | A dialog: the item's own values next to Flow's screenshot of the filled form, the step it's about to take, and when the run stops waiting (10 minutes after it asks). Approve and save, or Don't save with one of three reasons (wrong values, wrong record, something wrong on the screen). Approve isn't offered when the screenshot is missing or doesn't match the digest in Flow's request. | `get_item_approval`, `answer_item_approval`; event `item_approval` | | Run it | The list stops at an item nobody approved in time. A declined item goes to Needs you with the reason the person gave. | `run_item` | -| Check it, part 4 | The practice item stops before its save for the same approval. A practice save nobody approved, or one the person declined, records no result. The deployment settings file for the record check goes under Change where it runs. | `run_qualification_case` | -| Check it, part 5 | Each drill runs with the practice item's values. Flow changes one thing right before the save; the drill passes only when Flow's own check refuses it before saving. | `run_qualification_case` | +| Check it, question 2 | Desktop seals the version first, so the cases bind the version that runs. The practice item then stops before its save for the same approval. A practice save nobody approved, or one the person declined, records no result. A deployment settings file from IT, which replaces the learned record check, goes under Change the test values or where it runs. | `seal_qualification_workflow`, `run_qualification_case` | +| Check it, question 2 (drills) | After the practice item passes, the drills run on their own with its values. Flow changes one thing right before the save; the drill passes only when Flow's own check refuses it before saving. | `run_qualification_case` | How it works: diff --git a/packages/desktop/docs/SETUP_INPUTS.md b/packages/desktop/docs/SETUP_INPUTS.md new file mode 100644 index 000000000..0303b72b1 --- /dev/null +++ b/packages/desktop/docs/SETUP_INPUTS.md @@ -0,0 +1,238 @@ +# From "I recorded something" to "it runs checked": what a person gives + +This page counts every input a person gives OpenAdapt Desktop between a +finished recording and the first real item. It compares +the screens before this change (Calm UI, PR #1310) with the screens after it, +on the same recording of the same app, and links a screenshot of each input. + +## How it was measured + +`scripts/setup_walk/walk.py` drives the Desktop UI (from `npm run dev`) in a +headless browser against the real Desktop engine, real Flow, and MockMed's +transactional server (`openadapt_flow.mockmed.fault_server`, `?fault=ok`). +Nothing is mocked: every answer the screens show comes from the engine, and +the practice item, the drills and the first item are real Flow runs that save +to MockMed's store. All data is synthetic. + +The recording is the MockMed triage task Flow's tutorial records: open the +first referral, start a triage encounter, type a note, save. Desktop's +recorder doesn't keep the app's network traffic yet, so the walk writes the +recording's `network.jsonl` itself, in the `openadapt.network-capture/1` +format: the app's real save request and one read of `GET /api/db` right after +the save. That read stands in for the screen an app shows after saving; Flow's +own record-check tests build the same MockMed recording +(`packages/flow/tests/_record_check_fixtures.py`). + +What counts as an input: + +| Kind | Meaning | +| --- | --- | +| decision | A button that commits an answer. | +| choice | A radio button, select or checkbox the person has to set or check. | +| free text | A field the person types into. | +| confirmation | A statement the person confirms, including approving a save. | +| navigation | Opening a screen or a folded section. Counted apart. | +| item | The first item's own value, its Run button and its save approval. Running any item needs these, before and after. Counted apart. | + +## The counts + +| | Before (#1310) | After | +| --- | --- | --- | +| Setup inputs, recording to approval | 10, then blocked | 4: three decisions and one save approval | +| of which free text | 2 (a field name, a settings file path) | 0 | +| Navigation | 4 | 2 | +| Certified, sealed and approved | No | Yes | +| First item ran | No | Yes, gated by the person's approval; Flow's record check confirmed the save | +| First item shown as "done and checked" | No | No: it needs a signed qualification admission (see below) | + +After this change the person makes three decisions and approves one save: + +1. **Is this the right record check?** Yes. +2. **Which values change per item?** Yes, practice once with these test values. + The practice item then stops before its save until the person approves it + (the confirmation below). +3. **Approve?** Approve this workflow. + +The practice save's approval stays a person's: it is the one real save +before the workflow is approved, and Flow's each-item gate holds it. + +With the person's agent answering over MCP (the `agent` walk), the person +gives two inputs: the practice save's approval and Approve. + +## Before: where it stopped + +Ten inputs in, Flow refused the practice item before it started. Check it +showed "The practice run didn't pass" with "Qualification run did not retain +a valid report" as the detail. Flow's actual reasons were: + +- the clinical-write policy needs an idempotency key on the save's record + check, and Check it's form has no field for one (MockMed sends a fixed key, + not a typed value, and the exact checks only take a key from a typed value); +- five steps stayed unclassified until a person classifies each in Advanced: + exact checks, a page about 10,000 pixels tall + ([screenshot](setup-walk/before/17-exact-checks.png)). + +Under those were four more problems, each of which also stopped a Check it +run on a web app: + +- Desktop started the check without an environment observer a web app with + no OpenAdapt markers or session cookie can satisfy; +- the project got no identity policy for the save when the compiled step was + already armed, so part 2 showed "Confirmed" while Flow refused; +- Desktop stopped writing `run-report-receipt.json` in July but still required + it, so no practice item or drill result could be recorded; +- Desktop retained only the run report, not the frames it names, so Flow + couldn't credit a drill. + +And one in the approval itself: approving certified the plaintext version and +then sealed a copy, but Flow binds case results to the exact executable +workflow, so the sealed copy was never certified and couldn't run. + +| # | Kind | Input | Screenshot | +| --- | --- | --- | --- | +| | navigation | Check it | [01-workflows.png](setup-walk/before/01-workflows.png) | +| 1 | decision | Start the check with the app from your recording | [02-start-check.png](setup-walk/before/02-start-check.png) | +| 2 | choice | What does this step save? (It adds a new entry) | [03-effect-kind.png](setup-walk/before/03-effect-kind.png) | +| 3 | choice | Which value identifies the record? (Note) | [04-effect-match-param.png](setup-walk/before/04-effect-match-param.png) | +| 4 | free text | What is that ID called in your system's records? (note) | [05-effect-match-field.png](setup-walk/before/05-effect-match-field.png) | +| 5 | choice | How should OpenAdapt read the record back? (system's records) | [06-effect-tier.png](setup-walk/before/06-effect-tier.png) | +| 6 | decision | Save how it confirms the save | [07-effect-save.png](setup-walk/before/07-effect-save.png) | +| | navigation | Show part 3 (its confirm button is inside the fold) | [08-unfold-part-3.png](setup-walk/before/08-unfold-part-3.png) | +| 7 | decision | These look right | [09-these-look-right.png](setup-walk/before/09-these-look-right.png) | +| | navigation | Change where it runs | [11-open-where.png](setup-walk/before/11-open-where.png) | +| 8 | free text | Deployment settings file (for IT) | [12-practice-settings-file.png](setup-walk/before/12-practice-settings-file.png) | +| 9 | confirmation | These are test values, and I'll keep the app in view | [13-practice-attest.png](setup-walk/before/13-practice-attest.png) | +| 10 | decision | Run the practice item | [14-practice-run.png](setup-walk/before/14-practice-run.png) | +| | navigation | Advanced: exact checks (the 5 unclassified steps) | [17-exact-checks.png](setup-walk/before/17-exact-checks.png) | + +## After + +The walk took 10 minutes, most of it the practice item and five drills. + +| # | Kind | Input | Screenshot | +| --- | --- | --- | --- | +| | navigation | Check it | [01-workflows.png](setup-walk/after/01-workflows.png) | +| 1 | decision | Is this the right record check? Yes, use this check | [02-q1-record-check.png](setup-walk/after/02-q1-record-check.png) | +| 2 | decision | Which values change per item? Yes, practice once with these test values | [03-q2-values.png](setup-walk/after/03-q2-values.png) | +| 3 | confirmation | Approve and save (the practice item's save) | [04-practice-approve-save.png](setup-walk/after/04-practice-approve-save.png) | +| 4 | decision | Approve? Approve this workflow | [05-q3-approve.png](setup-walk/after/05-q3-approve.png) | +| | navigation | Run it | [07-run-it.png](setup-walk/after/07-run-it.png) | +| | item | Note (the first item's own value) | [08-item-note.png](setup-walk/after/08-item-note.png) | +| | item | Run 1 item | [09-item-run.png](setup-walk/after/09-item-run.png) | +| | item | Approve and save (the first item's save) | [10-item-approve-save.png](setup-walk/after/10-item-approve-save.png) | + +## The first item, and the step that's still missing + +After approval the walk runs one real item from Run it: it types a new note, +clicks Run 1 item, and approves the save. Flow's report for that run shows the +right-record check verified on screen and the save confirmed by MockMed's +store at the strongest tier. Desktop still shows it as **Finished, not +checked** and sends it to Needs you. That is correct, and it is a decision +for the founder, not something Check it should infer: + +- A run gets Flow's transaction outcome ("done and checked") only under the + Standard or Regulated profile. +- Under those profiles Flow refuses to act without a **signed qualification + admission**: "Standard and Regulated actuation requires a signed + qualification admission. Nothing was executed." Today that admission comes + from OpenAdapt Cloud or the organization's signer. Approving in Check it + doesn't produce one. +- So the learned record-check settings don't select a profile, the same as an + IT settings file without one, and the item runs gated per save but + unclassified. + +The open question is whether a person's Approve in Check it should sign a +local admission for this computer, or whether admission stays with Cloud. + +## With the person's agent + +The `agent` walk runs the same recording with an agent doing the setup. Its +calls go through `openadapt_desktop.agent_mcp` to the engine's local socket, +exactly as an MCP client's would: + +1. `list_workflows` +2. `get_check_setup` with `{"workflow_id": "24ad4eda"}` +3. `answer_record_check` with `{"workflow_id": "24ad4eda"}` +4. `answer_values` with `{"workflow_id": "24ad4eda", "changes": ["note"]}` +5. `run_checks` with `{"workflow_id": "24ad4eda"}` + +The person gives two inputs, both in the Desktop window: approving the +practice item's save, and Approve. The window says the agent answered and +lists what the person is approving +([screenshot](setup-walk/agent/03-q3-approve.png)). + +| # | Kind | Input | Screenshot | +| --- | --- | --- | --- | +| | navigation | Check it | [01-workflows.png](setup-walk/agent/01-workflows.png) | +| 1 | confirmation | Approve and save (the practice item's save) | [02-practice-approve-save.png](setup-walk/agent/02-practice-approve-save.png) | +| 2 | decision | Approve? Approve this workflow | [03-q3-approve.png](setup-walk/agent/03-q3-approve.png) | + +## What changed + +- **Pre-filled from the recording** (`openadapt_desktop/check_setup.py`, + engine command `get_check_setup`): + - the record check, from the read after the save (Flow's record-check + synthesis, #1284), bound as the same effect a person would type in the + form, with the app's own anti-double-save key and the fields the save + sent. Its fault test runs on exactly that binding, and Check it says what + it caught and what it can't catch; + - which steps only read: the network capture shows the app sent data only + when the person saved; + - the right-record check from the recording, the app, and its environment; + - which values change per item, from one recording or from several + recordings of the same workflow (a value the same in all of them is + proposed as fixed). +- **Only three questions.** A yes sends the same qualification commands the + long form sends (`answer_check_setup`). "Change" opens the parts. Without a + learned record check, the parts ask as before. +- **Values that don't change** are filled in by `run_item` with the recorded + value, and Run it doesn't ask for them. +- **The version that runs is the version checked.** Check it seals the + workflow before the practice item, so the cases, the approval and every run + bind one sealed version. +- **Agents.** `python -m openadapt_desktop.agent_mcp` is a stdio MCP server + that forwards to the running Desktop's local socket. An agent can read the + setup, answer the first two questions and run the checks. Its answers are + recorded as the agent's, and the window says so. It can't approve a + workflow or a save. + +Nothing here certifies or approves on its own, and no check is weaker: the +learned binding is the form's binding with more fields matched, the record +check runs under the Standard profile, and Flow still compares every +environment, identity and evidence digest. + +## Limits + +- Desktop's recorder doesn't keep network traffic yet. Until it does, a real + Desktop recording has no read to learn from, and Check it asks the parts. +- Desktop binds a learned check only when it is a REST read on one saving + step. A check across several tables, or a FHIR or SQL read, goes through + the form or a settings file from IT. +- The learned check here misses three kinds of bad save on MockMed (listed on + the screen). The right-record check before the save covers a save on + another patient; a stray write to another table isn't covered. +- MockMed's demo key is the same for every save on one patient, and its + server keeps only the first save with a key. The walk empties MockMed's + test store before the practice item and the first item, as Flow's tutorial + does before each run. +- Flow's generic web observer binds the page load, so an app that reloads the + whole page mid-run stops at the next input. That fails closed; such an app + needs the signed-in browser path, which binds the session instead. +- While the agent's practice item runs, it holds the engine's command queue, + like any run. The approval dialog still opens (approvals bypass the queue), + but the window shows the agent's answers only once the run ends. +- When Flow refuses a run before it starts (exit 2, "Nothing was executed"), + Desktop reports that the save may have gone through. That is safe but + wrong, and it is not fixed here. + +## Reproduce + +```bash +cd packages/desktop +npm run dev +OPENADAPT_DESKTOP_FLOW_BIN= python scripts/setup_walk/walk.py --variant after --out docs/setup-walk/after +``` + +`--variant before` runs the steps for the screens on commit 926d6c940. +`--variant agent` has an agent answer over MCP while the person approves. +`python scripts/setup_walk/shrink.py ` shrinks the screenshots. diff --git a/packages/desktop/docs/setup-walk/after/01-workflows.png b/packages/desktop/docs/setup-walk/after/01-workflows.png new file mode 100644 index 000000000..ec83146b2 Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/01-workflows.png differ diff --git a/packages/desktop/docs/setup-walk/after/02-q1-record-check.png b/packages/desktop/docs/setup-walk/after/02-q1-record-check.png new file mode 100644 index 000000000..3349e9bdf Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/02-q1-record-check.png differ diff --git a/packages/desktop/docs/setup-walk/after/03-q2-values.png b/packages/desktop/docs/setup-walk/after/03-q2-values.png new file mode 100644 index 000000000..dbddf3427 Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/03-q2-values.png differ diff --git a/packages/desktop/docs/setup-walk/after/04-practice-approve-save.png b/packages/desktop/docs/setup-walk/after/04-practice-approve-save.png new file mode 100644 index 000000000..d61c8eae5 Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/04-practice-approve-save.png differ diff --git a/packages/desktop/docs/setup-walk/after/05-q3-approve.png b/packages/desktop/docs/setup-walk/after/05-q3-approve.png new file mode 100644 index 000000000..4273397a2 Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/05-q3-approve.png differ diff --git a/packages/desktop/docs/setup-walk/after/06-approved.png b/packages/desktop/docs/setup-walk/after/06-approved.png new file mode 100644 index 000000000..556c7abda Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/06-approved.png differ diff --git a/packages/desktop/docs/setup-walk/after/07-run-it.png b/packages/desktop/docs/setup-walk/after/07-run-it.png new file mode 100644 index 000000000..735969d8a Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/07-run-it.png differ diff --git a/packages/desktop/docs/setup-walk/after/08-item-note.png b/packages/desktop/docs/setup-walk/after/08-item-note.png new file mode 100644 index 000000000..aebd9038f Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/08-item-note.png differ diff --git a/packages/desktop/docs/setup-walk/after/09-item-run.png b/packages/desktop/docs/setup-walk/after/09-item-run.png new file mode 100644 index 000000000..f6d9e4b90 Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/09-item-run.png differ diff --git a/packages/desktop/docs/setup-walk/after/10-item-approve-save.png b/packages/desktop/docs/setup-walk/after/10-item-approve-save.png new file mode 100644 index 000000000..849fd058b Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/10-item-approve-save.png differ diff --git a/packages/desktop/docs/setup-walk/after/11-first-item-done.png b/packages/desktop/docs/setup-walk/after/11-first-item-done.png new file mode 100644 index 000000000..bf0d22925 Binary files /dev/null and b/packages/desktop/docs/setup-walk/after/11-first-item-done.png differ diff --git a/packages/desktop/docs/setup-walk/after/inputs.json b/packages/desktop/docs/setup-walk/after/inputs.json new file mode 100644 index 000000000..851748f26 --- /dev/null +++ b/packages/desktop/docs/setup-walk/after/inputs.json @@ -0,0 +1,214 @@ +{ + "variant": "after", + "app": "MockMed (openadapt_flow.mockmed.fault_server, ?fault=ok)", + "workflow": "Add a triage note", + "seconds": 620, + "counts": { + "navigation": 2, + "decision": 3, + "confirmation": 1, + "item_value": 1, + "item_run": 1, + "item_approval": 1 + }, + "total": 9, + "setup_inputs": 4, + "first_item_inputs": 3, + "navigation": 2, + "reached_checked_run": true, + "blocked": null, + "inputs": [ + { + "screen": "Workflows", + "kind": "navigation", + "label": "Check it", + "shot": "01-workflows.png", + "note": "" + }, + { + "screen": "Check it: question 1", + "kind": "decision", + "label": "Is this the right record check? Yes, use this check", + "shot": "02-q1-record-check.png", + "note": "" + }, + { + "screen": "Check it: question 2", + "kind": "decision", + "label": "Which values change per item? Yes, practice once with these test values", + "shot": "03-q2-values.png", + "note": "" + }, + { + "screen": "Approval dialog", + "kind": "confirmation", + "label": "Approve and save (the practice item's save)", + "shot": "04-practice-approve-save.png", + "note": "" + }, + { + "screen": "Check it: question 3", + "kind": "decision", + "label": "Approve? Approve this workflow", + "shot": "05-q3-approve.png", + "note": "" + }, + { + "screen": "Check it", + "kind": "navigation", + "label": "Run it", + "shot": "07-run-it.png", + "note": "" + }, + { + "screen": "Run it", + "kind": "item_value", + "label": "Note (the first item's own value)", + "shot": "08-item-note.png", + "note": "" + }, + { + "screen": "Run it", + "kind": "item_run", + "label": "Run 1 item", + "shot": "09-item-run.png", + "note": "" + }, + { + "screen": "Approval dialog", + "kind": "item_approval", + "label": "Approve and save (the first item's save)", + "shot": "10-item-approve-save.png", + "note": "" + } + ], + "engine_commands": [ + "compile_recording", + "shadow_status", + "shadow_status", + "get_auth_status", + "get_auth_status", + "get_first_workflow_state", + "get_workflows", + "get_first_workflow_state", + "get_workflows", + "get_first_workflow_state", + "get_needs_attention", + "get_needs_attention", + "get_sync_state", + "get_sync_state", + "authoring_status", + "authoring_status", + "get_workflows", + "get_results_summary", + "get_workflows", + "get_results_summary", + "get_workflows", + "get_task_inputs", + "get_workflows", + "get_qualification", + "get_check_setup", + "get_task_inputs", + "get_qualification", + "get_check_setup", + "propose_qualification_from_demo", + "get_capabilities", + "get_capabilities", + "propose_qualification_from_demo", + "answer_check_setup", + "get_qualification", + "answer_check_setup", + "seal_qualification_workflow", + "get_qualification", + "add_qualification_case", + "run_qualification_case", + "shadow_status", + "shadow_status", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "shadow_status", + "get_item_approval", + "answer_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "shadow_status", + "run_qualification_case", + "shadow_status", + "shadow_status", + "run_qualification_case", + "shadow_status", + "shadow_status", + "shadow_status", + "run_qualification_case", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "run_qualification_case", + "shadow_status", + "shadow_status", + "shadow_status", + "run_qualification_case", + "shadow_status", + "shadow_status", + "shadow_status", + "certify_qualification", + "get_workflows", + "get_task_inputs", + "get_approval_mode", + "get_workflows", + "get_task_inputs", + "get_approval_mode", + "get_capabilities", + "get_capabilities", + "run_item", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "shadow_status", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "answer_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval" + ], + "agent_calls": [] +} diff --git a/packages/desktop/docs/setup-walk/agent/01-workflows.png b/packages/desktop/docs/setup-walk/agent/01-workflows.png new file mode 100644 index 000000000..ec83146b2 Binary files /dev/null and b/packages/desktop/docs/setup-walk/agent/01-workflows.png differ diff --git a/packages/desktop/docs/setup-walk/agent/02-practice-approve-save.png b/packages/desktop/docs/setup-walk/agent/02-practice-approve-save.png new file mode 100644 index 000000000..d77d9e562 Binary files /dev/null and b/packages/desktop/docs/setup-walk/agent/02-practice-approve-save.png differ diff --git a/packages/desktop/docs/setup-walk/agent/03-q3-approve.png b/packages/desktop/docs/setup-walk/agent/03-q3-approve.png new file mode 100644 index 000000000..4d6c3605a Binary files /dev/null and b/packages/desktop/docs/setup-walk/agent/03-q3-approve.png differ diff --git a/packages/desktop/docs/setup-walk/agent/04-approved.png b/packages/desktop/docs/setup-walk/agent/04-approved.png new file mode 100644 index 000000000..abcd6f4de Binary files /dev/null and b/packages/desktop/docs/setup-walk/agent/04-approved.png differ diff --git a/packages/desktop/docs/setup-walk/agent/inputs.json b/packages/desktop/docs/setup-walk/agent/inputs.json new file mode 100644 index 000000000..ca4980658 --- /dev/null +++ b/packages/desktop/docs/setup-walk/agent/inputs.json @@ -0,0 +1,179 @@ +{ + "variant": "agent", + "app": "MockMed (openadapt_flow.mockmed.fault_server, ?fault=ok)", + "workflow": "Add a triage note", + "seconds": 664, + "counts": { + "navigation": 1, + "confirmation": 1, + "decision": 1 + }, + "total": 3, + "setup_inputs": 2, + "first_item_inputs": 0, + "navigation": 1, + "reached_checked_run": true, + "blocked": null, + "inputs": [ + { + "screen": "Workflows", + "kind": "navigation", + "label": "Check it", + "shot": "01-workflows.png", + "note": "" + }, + { + "screen": "Approval dialog", + "kind": "confirmation", + "label": "Approve and save (the practice item's save)", + "shot": "02-practice-approve-save.png", + "note": "" + }, + { + "screen": "Check it: question 3", + "kind": "decision", + "label": "Approve? Approve this workflow", + "shot": "03-q3-approve.png", + "note": "" + } + ], + "engine_commands": [ + "compile_recording", + "shadow_status", + "get_auth_status", + "get_auth_status", + "shadow_status", + "get_first_workflow_state", + "get_workflows", + "get_first_workflow_state", + "get_first_workflow_state", + "get_workflows", + "get_needs_attention", + "get_needs_attention", + "get_sync_state", + "get_sync_state", + "authoring_status", + "authoring_status", + "get_workflows", + "get_workflows", + "get_results_summary", + "get_results_summary", + "get_workflows", + "get_task_inputs", + "get_qualification", + "get_check_setup", + "get_workflows", + "get_task_inputs", + "get_qualification", + "get_check_setup", + "propose_qualification_from_demo", + "get_capabilities", + "get_capabilities", + "propose_qualification_from_demo", + "get_workflows", + "get_task_inputs", + "get_qualification", + "get_check_setup", + "get_workflows", + "get_qualification", + "get_task_inputs", + "get_check_setup", + "get_workflows", + "get_task_inputs", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "answer_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_item_approval", + "get_qualification", + "get_check_setup", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "shadow_status", + "get_workflows", + "get_task_inputs", + "get_qualification", + "get_check_setup", + "shadow_status", + "certify_qualification", + "shadow_status" + ], + "agent_calls": [ + { + "tool": "list_workflows", + "arguments": {} + }, + { + "tool": "get_check_setup", + "arguments": { + "workflow_id": "24ad4eda" + } + }, + { + "tool": "answer_record_check", + "arguments": { + "workflow_id": "24ad4eda" + } + }, + { + "tool": "answer_values", + "arguments": { + "workflow_id": "24ad4eda", + "changes": [ + "note" + ] + } + }, + { + "tool": "run_checks", + "arguments": { + "workflow_id": "24ad4eda" + } + } + ] +} diff --git a/packages/desktop/docs/setup-walk/before/01-workflows.png b/packages/desktop/docs/setup-walk/before/01-workflows.png new file mode 100644 index 000000000..ec83146b2 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/01-workflows.png differ diff --git a/packages/desktop/docs/setup-walk/before/02-start-check.png b/packages/desktop/docs/setup-walk/before/02-start-check.png new file mode 100644 index 000000000..1ccffd42a Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/02-start-check.png differ diff --git a/packages/desktop/docs/setup-walk/before/03-effect-kind.png b/packages/desktop/docs/setup-walk/before/03-effect-kind.png new file mode 100644 index 000000000..2a4f0f2fa Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/03-effect-kind.png differ diff --git a/packages/desktop/docs/setup-walk/before/04-effect-match-param.png b/packages/desktop/docs/setup-walk/before/04-effect-match-param.png new file mode 100644 index 000000000..5b35e2a94 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/04-effect-match-param.png differ diff --git a/packages/desktop/docs/setup-walk/before/05-effect-match-field.png b/packages/desktop/docs/setup-walk/before/05-effect-match-field.png new file mode 100644 index 000000000..5b35e2a94 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/05-effect-match-field.png differ diff --git a/packages/desktop/docs/setup-walk/before/06-effect-tier.png b/packages/desktop/docs/setup-walk/before/06-effect-tier.png new file mode 100644 index 000000000..21abc11a9 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/06-effect-tier.png differ diff --git a/packages/desktop/docs/setup-walk/before/07-effect-save.png b/packages/desktop/docs/setup-walk/before/07-effect-save.png new file mode 100644 index 000000000..21abc11a9 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/07-effect-save.png differ diff --git a/packages/desktop/docs/setup-walk/before/08-unfold-part-3.png b/packages/desktop/docs/setup-walk/before/08-unfold-part-3.png new file mode 100644 index 000000000..0dc9fa72a Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/08-unfold-part-3.png differ diff --git a/packages/desktop/docs/setup-walk/before/09-these-look-right.png b/packages/desktop/docs/setup-walk/before/09-these-look-right.png new file mode 100644 index 000000000..4a408b5be Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/09-these-look-right.png differ diff --git a/packages/desktop/docs/setup-walk/before/10-after-confirm.png b/packages/desktop/docs/setup-walk/before/10-after-confirm.png new file mode 100644 index 000000000..46644775c Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/10-after-confirm.png differ diff --git a/packages/desktop/docs/setup-walk/before/11-open-where.png b/packages/desktop/docs/setup-walk/before/11-open-where.png new file mode 100644 index 000000000..46644775c Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/11-open-where.png differ diff --git a/packages/desktop/docs/setup-walk/before/12-practice-settings-file.png b/packages/desktop/docs/setup-walk/before/12-practice-settings-file.png new file mode 100644 index 000000000..78bcdc469 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/12-practice-settings-file.png differ diff --git a/packages/desktop/docs/setup-walk/before/13-practice-attest.png b/packages/desktop/docs/setup-walk/before/13-practice-attest.png new file mode 100644 index 000000000..756c58acd Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/13-practice-attest.png differ diff --git a/packages/desktop/docs/setup-walk/before/14-practice-run.png b/packages/desktop/docs/setup-walk/before/14-practice-run.png new file mode 100644 index 000000000..fec66ca10 Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/14-practice-run.png differ diff --git a/packages/desktop/docs/setup-walk/before/15-practice-refused.png b/packages/desktop/docs/setup-walk/before/15-practice-refused.png new file mode 100644 index 000000000..5f1418aed Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/15-practice-refused.png differ diff --git a/packages/desktop/docs/setup-walk/before/17-exact-checks.png b/packages/desktop/docs/setup-walk/before/17-exact-checks.png new file mode 100644 index 000000000..60f38c37c Binary files /dev/null and b/packages/desktop/docs/setup-walk/before/17-exact-checks.png differ diff --git a/packages/desktop/docs/setup-walk/before/inputs.json b/packages/desktop/docs/setup-walk/before/inputs.json new file mode 100644 index 000000000..0dfbedf95 --- /dev/null +++ b/packages/desktop/docs/setup-walk/before/inputs.json @@ -0,0 +1,175 @@ +{ + "variant": "before", + "app": "MockMed (openadapt_flow.mockmed.fault_server, ?fault=ok)", + "workflow": "Add a triage note", + "seconds": 49, + "counts": { + "navigation": 4, + "decision": 4, + "choice": 3, + "free_text": 2, + "confirmation": 1 + }, + "total": 14, + "reached_checked_run": false, + "blocked": { + "where": "Check it: practice run", + "flow_refusal": [ + "Certification: require_idempotency_key_for: the save's record check carries no idempotency key", + "Approval fallback: qualified action path has no effect contract to assign a tier" + ], + "shown_to_the_person": "The practice run didn't pass. (technical details: Qualification run did not retain a valid report)", + "why_check_it_cannot_fix_it": [ + "The plain record-check form has no idempotency key. MockMed sends a fixed key (not a typed value), and exact checks only take a key from a typed value.", + "Five steps stay unclassified until a person classifies each in exact checks; unclassified steps count as saves without a record check." + ] + }, + "inputs": [ + { + "screen": "Workflows", + "kind": "navigation", + "label": "Check it", + "shot": "01-workflows.png", + "note": "" + }, + { + "screen": "Check it", + "kind": "decision", + "label": "Start the check with the app from your recording", + "shot": "02-start-check.png", + "note": "" + }, + { + "screen": "Check it: how it confirms the save", + "kind": "choice", + "label": "What does this step save? (It adds a new entry)", + "shot": "03-effect-kind.png", + "note": "" + }, + { + "screen": "Check it: how it confirms the save", + "kind": "choice", + "label": "Which value identifies the record? (Note)", + "shot": "04-effect-match-param.png", + "note": "Already Note; the person still has to check it" + }, + { + "screen": "Check it: how it confirms the save", + "kind": "free_text", + "label": "What is that ID called in your system's records? (note)", + "shot": "05-effect-match-field.png", + "note": "" + }, + { + "screen": "Check it: how it confirms the save", + "kind": "choice", + "label": "How should OpenAdapt read the record back? (system's records)", + "shot": "06-effect-tier.png", + "note": "Already selected; the person still has to check it" + }, + { + "screen": "Check it: how it confirms the save", + "kind": "decision", + "label": "Save how it confirms the save", + "shot": "07-effect-save.png", + "note": "" + }, + { + "screen": "Check it", + "kind": "navigation", + "label": "Show part 3 (its confirm button is inside the fold)", + "shot": "08-unfold-part-3.png", + "note": "" + }, + { + "screen": "Check it", + "kind": "decision", + "label": "These look right", + "shot": "09-these-look-right.png", + "note": "" + }, + { + "screen": "Check it: practice run", + "kind": "navigation", + "label": "Change where it runs", + "shot": "11-open-where.png", + "note": "" + }, + { + "screen": "Check it: practice run", + "kind": "free_text", + "label": "Deployment settings file (for IT)", + "shot": "12-practice-settings-file.png", + "note": "The record check reads MockMed's store; without this file the practice run has no record check to run" + }, + { + "screen": "Check it: practice run", + "kind": "confirmation", + "label": "These are test values, and I'll keep the app in view", + "shot": "13-practice-attest.png", + "note": "" + }, + { + "screen": "Check it: practice run", + "kind": "decision", + "label": "Run the practice item", + "shot": "14-practice-run.png", + "note": "" + }, + { + "screen": "Check it", + "kind": "navigation", + "label": "Advanced: exact checks (the 5 unclassified steps)", + "shot": "17-exact-checks.png", + "note": "" + } + ], + "engine_commands": [ + "compile_recording", + "shadow_status", + "shadow_status", + "get_auth_status", + "get_auth_status", + "get_first_workflow_state", + "get_workflows", + "get_first_workflow_state", + "get_workflows", + "get_first_workflow_state", + "get_needs_attention", + "get_needs_attention", + "get_sync_state", + "get_sync_state", + "authoring_status", + "authoring_status", + "get_workflows", + "get_results_summary", + "get_workflows", + "get_results_summary", + "get_workflows", + "get_task_inputs", + "get_qualification", + "get_workflows", + "get_task_inputs", + "get_qualification", + "propose_qualification_from_demo", + "get_capabilities", + "get_capabilities", + "propose_qualification_from_demo", + "initialize_qualification", + "bind_qualification_effect", + "set_qualification_risk", + "shadow_status", + "add_qualification_case", + "run_qualification_case", + "get_item_approval", + "get_item_approval", + "get_qualification", + "get_qualification", + "get_capabilities", + "get_capabilities" + ], + "setup_inputs": 10, + "first_item_inputs": 0, + "navigation": 4, + "screens": "Calm UI (#1310), commit 926d6c940" +} diff --git a/packages/desktop/openadapt_desktop/__init__.py b/packages/desktop/openadapt_desktop/__init__.py index bd0497bcd..1cdce97ed 100644 --- a/packages/desktop/openadapt_desktop/__init__.py +++ b/packages/desktop/openadapt_desktop/__init__.py @@ -21,4 +21,4 @@ +-- backends/ Storage backend plugins """ -__version__ = "0.21.1" +__version__ = "0.22.0" diff --git a/packages/desktop/openadapt_desktop/agent_mcp.py b/packages/desktop/openadapt_desktop/agent_mcp.py new file mode 100644 index 000000000..9cc9250fc --- /dev/null +++ b/packages/desktop/openadapt_desktop/agent_mcp.py @@ -0,0 +1,252 @@ +"""Let the person's own agent set up Check it over MCP (stdio). + +Run it from the agent's MCP settings while OpenAdapt Desktop is open:: + + python -m openadapt_desktop.agent_mcp + +It forwards to the running Desktop engine over its loopback socket +(``~/.openadapt/desktop_ipc.json``, owner-only, with a per-session token), so +the window and the agent see the same workflows and answers. The agent can: + +- read what OpenAdapt filled in from a recording (``get_check_setup``); +- answer "Is this the right record check?" with yes, and "Which values change + per item?"; +- run the practice item and the safety drills (``run_checks``). The practice + item saves for real, so Flow stops before its save until a person approves + it in the Desktop window. + +It can't approve a workflow, change a check, or approve a save. Answers it +gives are recorded as the agent's, and the window shows them as such until a +person approves. This process opens no network listener. +""" + +from __future__ import annotations + +import json +import os +import socket +from pathlib import Path +from typing import Any + +DISCOVERY_ENV = "OPENADAPT_DESKTOP_IPC" +DEFAULT_DISCOVERY = Path.home() / ".openadapt" / "desktop_ipc.json" +SERVER_NAME = "openadapt-desktop-setup" + +INSTRUCTIONS = ( + "OpenAdapt Desktop turns a recorded workflow into one that runs on its own " + "and checks every save. Before it runs, Check it asks three things. You can " + "answer the first two for the person: call list_workflows, then " + "get_check_setup to read what OpenAdapt learned from the recording. If the " + "record check it learned matches what the workflow saves, call " + "answer_record_check. Then answer_values with the names of the values that " + "change from one item to the next. run_checks runs one practice item (a " + "person approves its save in the Desktop window) and the safety drills. The " + "third question, approving the workflow, is the person's: tell them to " + "approve it in OpenAdapt Desktop. Never say the workflow is ready until they " + "have." +) + + +class DesktopUnavailable(RuntimeError): + """OpenAdapt Desktop isn't running, or refused the call.""" + + +def _discovery() -> dict[str, Any]: + path = Path(os.environ.get(DISCOVERY_ENV) or DEFAULT_DISCOVERY) + try: + data = json.loads(path.read_text("utf-8")) + except (OSError, ValueError) as exc: + raise DesktopUnavailable( + "OpenAdapt Desktop isn't running on this computer. Open it and try again." + ) from exc + if not isinstance(data, dict) or not data.get("port") or not data.get("token"): + raise DesktopUnavailable("OpenAdapt Desktop's local address file is incomplete.") + if str(data.get("host") or "127.0.0.1") not in {"127.0.0.1", "localhost", "::1"}: + raise DesktopUnavailable("OpenAdapt Desktop's local address isn't on this computer.") + return data + + +def call_desktop(cmd: str, params: dict[str, Any], *, timeout: float = 60.0) -> Any: + """Send one setup command to the running Desktop and wait for its answer.""" + + found = _discovery() + frame = json.dumps({"type": cmd, "data": params, "token": found["token"]}) + "\n" + with socket.create_connection( + (str(found.get("host") or "127.0.0.1"), int(found["port"])), timeout=timeout + ) as conn: + conn.sendall(frame.encode("utf-8")) + buffer = "" + while True: + chunk = conn.recv(65536) + if not chunk: + raise DesktopUnavailable("OpenAdapt Desktop closed the connection.") + buffer += chunk.decode("utf-8", errors="replace") + while "\n" in buffer: + line, buffer = buffer.split("\n", 1) + if not line.strip(): + continue + message = json.loads(line) + kind = message.get("type") + data = message.get("data") or {} + if kind == "check_setup" and data.get("cmd") == cmd: + return data.get("result") + if kind == "recording_error": + raise DesktopUnavailable(str(data.get("error") or "Desktop refused the call.")) + + +def _plain_setup(setup: dict[str, Any]) -> dict[str, Any]: + """The setup without the binding internals an agent doesn't need.""" + + if not setup or not setup.get("ok"): + return setup + learned = dict(setup.get("record_check") or {}) + learned.pop("binding", None) + learned.pop("effects", None) + return { + "workflow_id": setup.get("workflow_id"), + "record_check": learned, + "steps_that_only_read": (setup.get("quiet_steps") or {}).get("evidence"), + "values": setup.get("values"), + "answers": setup.get("answers"), + "approval": setup.get("approval"), + } + + +TOOLS: dict[str, dict[str, Any]] = { + "list_workflows": { + "description": "The person's recorded workflows in OpenAdapt Desktop, with their ids.", + "schema": {"type": "object", "properties": {}, "additionalProperties": False}, + }, + "get_check_setup": { + "description": ( + "What OpenAdapt filled in for one workflow from its recording: the record " + "check it learned (which read, what it checks, what its fault test caught " + "and missed), which steps only read, the values and whether each seems to " + "change per item, and what was already answered and by whom." + ), + "schema": { + "type": "object", + "properties": {"workflow_id": {"type": "string"}}, + "required": ["workflow_id"], + "additionalProperties": False, + }, + }, + "answer_record_check": { + "description": ( + "Answer yes to 'Is this the right record check?' for a workflow. Only when " + "the learned read is where the app keeps what this workflow saves. Recorded " + "as your answer; the person sees it before approving." + ), + "schema": { + "type": "object", + "properties": {"workflow_id": {"type": "string"}}, + "required": ["workflow_id"], + "additionalProperties": False, + }, + }, + "answer_values": { + "description": ( + "Answer 'Which values change per item?' with the names of the values that " + "change. Any other value is filled in with its recorded value on every item." + ), + "schema": { + "type": "object", + "properties": { + "workflow_id": {"type": "string"}, + "changes": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["workflow_id", "changes"], + "additionalProperties": False, + }, + }, + "run_checks": { + "description": ( + "Run one practice item with the recorded values, then the safety drills. " + "The practice item saves for real, so it waits for the person to approve " + "its save in the Desktop window. Can take several minutes." + ), + "schema": { + "type": "object", + "properties": {"workflow_id": {"type": "string"}}, + "required": ["workflow_id"], + "additionalProperties": False, + }, + }, +} + + +def handle(name: str, arguments: dict[str, Any]) -> Any: + """Run one tool against the running Desktop.""" + + if name == "list_workflows": + workflows = call_desktop("get_workflows", {}) or [] + return [ + {"workflow_id": item.get("id"), "name": item.get("name"), "status": item.get("status")} + for item in workflows + if isinstance(item, dict) + ] + workflow_id = str(arguments.get("workflow_id") or "") + if name == "get_check_setup": + return _plain_setup(call_desktop("get_check_setup", {"workflow_id": workflow_id})) + if name == "answer_record_check": + return _plain_setup( + call_desktop( + "answer_check_setup", + {"workflow_id": workflow_id, "question": "record_check", "answer": "yes"}, + ) + ) + if name == "answer_values": + changes = [str(item) for item in arguments.get("changes") or []] + return _plain_setup( + call_desktop( + "answer_check_setup", + {"workflow_id": workflow_id, "question": "values", "changes": changes}, + ) + ) + if name == "run_checks": + return call_desktop("run_check_cases", {"workflow_id": workflow_id}, timeout=3600.0) + raise ValueError(f"unknown tool {name}") + + +def build_server() -> Any: + """The MCP server (no I/O started). Needs the ``mcp`` package.""" + + import anyio + import mcp.types as types + from mcp.server.lowlevel import Server + + server: Any = Server(SERVER_NAME, instructions=INSTRUCTIONS) + + @server.list_tools() + async def _list_tools() -> list[types.Tool]: + return [ + types.Tool(name=name, description=spec["description"], inputSchema=spec["schema"]) + for name, spec in TOOLS.items() + ] + + @server.call_tool() + async def _call_tool(name: str, arguments: dict[str, Any] | None) -> list[types.TextContent]: + try: + result = await anyio.to_thread.run_sync(handle, name, dict(arguments or {})) + except (DesktopUnavailable, OSError, ValueError) as exc: + result = {"ok": False, "error": str(exc)} + return [types.TextContent(type="text", text=json.dumps(result, default=str))] + + return server + + +def main() -> None: + import anyio + from mcp.server.stdio import stdio_server + + server = build_server() + + async def run() -> None: + async with stdio_server() as (read, write): + await server.run(read, write, server.create_initialization_options()) + + anyio.run(run) + + +if __name__ == "__main__": + main() diff --git a/packages/desktop/openadapt_desktop/check_setup.py b/packages/desktop/openadapt_desktop/check_setup.py new file mode 100644 index 000000000..0244031ec --- /dev/null +++ b/packages/desktop/openadapt_desktop/check_setup.py @@ -0,0 +1,861 @@ +"""Check it, pre-filled: what the recording already shows, and three questions. + +A person used to answer every part of Check it by hand: which step saves, +which steps only move around, how to read the saved record back, which +values change per item. Most of that is in the recording. This module reads +it and proposes each answer; the person (or their agent, over the local +socket) answers three questions: + +1. **Is this the right record check?** The read the app made after the save + (Flow's record-check synthesis, ``openadapt_flow.record_check_synthesis``) + becomes the same effect binding a person would type in the plain + question form, with the read itself as the record-check settings file. + Its fault test runs on exactly that binding, so the evidence shown is + for what will be saved, not for a stronger check it doesn't bind. +2. **Which values change per item?** Every value typed during the + recording is an input. A value the person marks as fixed is filled in + with its recorded value on every item. +3. **Approve?** Unchanged: certify and seal, in the window, by a person. + +Nothing here certifies, approves or loosens a check. A "yes" calls the same +qualification commands the forms call; a proposal that isn't certain says +so and leaves the form open. +""" + +from __future__ import annotations + +import json +import os +from collections.abc import Mapping +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import yaml + +from openadapt_desktop.private_fs import ensure_private_dir, make_private_file + +SETUP_SCHEMA = "openadapt.desktop.check-setup/v1" + +#: The verifier kinds a learned check can bind today. A REST read is what +#: Flow's synthesis builds from a recorded network capture. +_BINDABLE_READS = frozenset({"rest"}) + +#: Who answered a question. The window passes "person"; the local agent +#: socket passes "agent". Approval is never answered by an agent. +ANSWERERS = frozenset({"person", "agent"}) + + +class CheckSetupError(ValueError): + """A setup answer the engine refuses.""" + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + +# --------------------------------------------------------------------------- +# What the recording shows +# --------------------------------------------------------------------------- + + +def _read_label(config: Mapping[str, Any]) -> str: + path = str(config.get("records_path") or "") + return f"GET {path}" if path else "a read of the app's records" + + +def _plain_fault(result: Any) -> str: + detail = str(getattr(result, "detail", "") or "") + # Flow names the surface first ("api_db: nothing saved"); a person + # doesn't need the surface name. + return detail.split(": ", 1)[1] if ": " in detail else detail + + +def _is_idempotency_field(name: str) -> bool: + lowered = name.lower().replace("-", "_") + return lowered in {"key", "idempotency_key", "idempotencykey"} or "idempot" in lowered + + +def learn_record_check( + recording_dir: Path, + *, + inputs: Mapping[str, Any], + secret_inputs: frozenset[str] = frozenset(), +) -> dict[str, Any]: + """Propose the record check from the reads the recording holds. + + Returns ``{"found": False, "reason": ...}`` when there is nothing to + propose. Otherwise ``found`` is True and ``binding`` holds the exact + arguments for ``bind_qualification_effect``, ``effects`` the record-check + settings (a deployment ``effects:`` section), and ``fault_test`` the + result of judging that binding against the recorded records with each + bad save injected. + """ + + try: + from openadapt_flow.record_check_synthesis import ( + SynthesisError, + prove_against_faults, + recorded_params, + synthesize, + ) + from openadapt_flow.runtime.effects import Effect, EffectKind, ValueExpr + from openadapt_flow.runtime.effects.surfaces import surface_probe + except ImportError: + return {"found": False, "reason": "This Flow can't read a recording's network capture."} + + recording_dir = Path(recording_dir) + if not (recording_dir / "network.jsonl").is_file(): + return { + "found": False, + "reason": ( + "The recording holds no network capture, so OpenAdapt didn't see a read " + "after the save." + ), + } + params = { + name: value + for name, value in recorded_params(recording_dir).items() + if name in inputs and name not in secret_inputs + } + try: + proposal = synthesize(recording_dir, params=params) + except SynthesisError as exc: + return {"found": False, "reason": str(exc)} + targets = [surface for surface in proposal.search.surfaces if surface.is_target] + if len(targets) != 1: + return { + "found": False, + "reason": ( + "OpenAdapt didn't find the saved record in any read after the save." + if not targets + else "The saved record showed up in more than one read, so OpenAdapt won't " + "pick one." + ), + } + surface = targets[0] + config = {key: value for key, value in surface.config.items() if not key.startswith("_")} + kind = str(config.get("kind")) + if kind not in _BINDABLE_READS: + return { + "found": False, + "reason": f"OpenAdapt found the read, but Desktop can't run a {kind} read yet.", + } + bound = {field: param for field, param in sorted(surface.bound.items()) if param in params} + if not bound: + return { + "found": False, + "reason": "None of the values you typed shows up in the saved record.", + } + # Select the record by an id-like value when there is one, else by the + # first value the record holds. Every other recorded value is matched too. + match_field, match_param = next( + ((field, param) for field, param in bound.items() if " " not in str(params.get(param, ""))), + next(iter(bound.items())), + ) + extra_match: dict[str, dict[str, str]] = { + field: {"param": param} for field, param in bound.items() if field != match_field + } + constants = dict(sorted(surface.constants.items())) + # A fixed key the app sends with the save to stop a double save. Flow's + # clinical-write policy wants it on the record check; it is the app's + # own value, so it is checked as sent, never invented. + key_field = next((field for field in constants if _is_idempotency_field(field)), None) + idempotency_literal = constants.pop(key_field) if key_field else None + extra_match.update({field: {"literal": str(value)} for field, value in constants.items()}) + binding: dict[str, Any] = { + "kind": "record_written", + "match_field": match_field, + "match_param": match_param, + "extra_match": extra_match, + "expected_count": 1, + "count_new_only": True, + "forbid_collateral_loss": True, + "verification_tier": 1, + } + if key_field: + binding["idempotency_literal"] = idempotency_literal + binding["key_field"] = key_field + + # Judge exactly this binding, not the fuller check synthesis proposes. + match = {match_field: ValueExpr(param=match_param)} + for field, expr in extra_match.items(): + match[field] = ( + ValueExpr(param=expr["param"]) + if "param" in expr + else ValueExpr(literal=expr["literal"]) + ) + effect = Effect( + kind=EffectKind.RECORD_WRITTEN, + match=match, + expected_count=1, + count_new_only=True, + forbid_collateral_loss=True, + idempotency_key=ValueExpr(literal=idempotency_literal) if key_field else None, + key_field=key_field or "key", + probe=surface_probe(surface.name), + ) + faults = prove_against_faults( + [surface], [effect], params, closed_world=proposal.search.closed_world + ) + clean = [result for result in faults if result.correct] + bad = [result for result in faults if not result.correct] + caught = [result for result in bad if result.ok] + missed = [result for result in bad if not result.ok] + clean_pass = bool(clean) and all(result.ok for result in clean) + if not clean_pass: + return { + "found": False, + "reason": ( + "The read OpenAdapt found would stop the recorded save itself, so it isn't " + "proposed." + ), + } + return { + "found": True, + "source": "recorded_reads", + "read": _read_label(config), + "checked": sorted({match_field, *extra_match}), + "values_checked": sorted(set(bound.values())), + "fixed_fields": constants, + "idempotency_field": key_field, + "binding": binding, + "effects": config, + "fault_test": { + "bad_saves": len(bad), + "caught": len(caught), + "caught_kinds": [_plain_fault(result) for result in caught], + "missed_kinds": [_plain_fault(result) for result in missed], + }, + } + + +def quiet_steps(recording_dir: Path) -> dict[str, Any] | None: + """Whether every write the recording sent belongs to the save. + + When the network capture shows the app sent a write request only during + the save, every other step only opened screens or filled in the form. + Returns ``None`` when that can't be shown. + """ + + try: + from openadapt_flow.record_check_synthesis import ( + SynthesisError, + load_network, + save_window, + ) + except ImportError: + return None + try: + network = load_network(Path(recording_dir)) + except SynthesisError: + return None + if not network: + return None + writes = [ + line + for line in network + if str(line.get("method") or "").upper() in {"POST", "PUT", "PATCH", "DELETE"} + ] + _first, _last, save_writes = save_window(Path(recording_dir), network) + if not save_writes or len(save_writes) != len(writes): + return None + return { + "write_requests": len(writes), + "evidence": ( + "In your recording the app sent data to its server only when you saved." + if len(writes) == 1 + else f"In your recording the app sent its {len(writes)} save requests only when " + "you saved." + ), + } + + +def _recording_params(recording_dir: Path) -> dict[str, str]: + try: + meta = json.loads((Path(recording_dir) / "meta.json").read_text("utf-8")) + except (OSError, ValueError): + return {} + params = meta.get("params") if isinstance(meta, dict) else None + if not isinstance(params, dict): + return {} + return {str(name): str(value) for name, value in params.items() if value is not None} + + +def value_proposals(inputs: list[dict[str, Any]], recordings: list[Path]) -> list[dict[str, Any]]: + """Which inputs change per item, from one recording or several. + + With one recording every typed value is proposed as changing per item: + the person typed it, and nothing shows it stays the same. With several + recordings of the same workflow, a value that was the same in every one + is proposed as fixed, and one that differed as changing. + """ + + seen: dict[str, set[str]] = {} + counted = 0 + for recording in recordings: + values = _recording_params(recording) + if not values: + continue + counted += 1 + for name, value in values.items(): + seen.setdefault(name, set()).add(value) + proposals: list[dict[str, Any]] = [] + for item in inputs: + name = str(item.get("name") or "") + if not name or item.get("secret"): + continue + distinct = seen.get(name, set()) + if counted > 1 and len(distinct) == 1: + changes = False + why = f"The same in all {counted} recordings." + elif counted > 1 and len(distinct) > 1: + changes = True + why = f"Different in your {counted} recordings." + else: + changes = True + why = "You typed it while recording." + proposals.append( + { + "name": name, + "example": item.get("example"), + "changes": changes, + "why": why, + } + ) + return proposals + + +# --------------------------------------------------------------------------- +# What was answered +# --------------------------------------------------------------------------- + + +class SetupStore: + """Answers per workflow, owner-only, next to the engine's other state.""" + + def __init__(self, data_dir: Path) -> None: + self.root = Path(data_dir) / "check_setup" + + def _path(self, workflow_id: str) -> Path: + safe = "".join(ch for ch in workflow_id if ch.isalnum() or ch in "-_") + if not safe or safe != workflow_id: + raise CheckSetupError("Unknown workflow") + return self.root / f"{safe}.json" + + def load(self, workflow_id: str) -> dict[str, Any]: + path = self._path(workflow_id) + try: + data = json.loads(path.read_text("utf-8")) + except (OSError, ValueError): + return {"schema": SETUP_SCHEMA, "answers": {}, "fixed_values": {}} + if not isinstance(data, dict) or data.get("schema") != SETUP_SCHEMA: + return {"schema": SETUP_SCHEMA, "answers": {}, "fixed_values": {}} + data.setdefault("answers", {}) + data.setdefault("fixed_values", {}) + return data + + def save(self, workflow_id: str, data: dict[str, Any]) -> None: + ensure_private_dir(self.root) + path = self._path(workflow_id) + tmp = path.with_suffix(".tmp") + tmp.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + make_private_file(tmp) + os.replace(tmp, path) + + def record_check_config(self, workflow_id: str) -> Path | None: + """The record-check settings a "yes" saved, if the file is intact.""" + + data = self.load(workflow_id) + raw = data.get("record_check_config") + if not raw: + return None + path = Path(str(raw)) + if not path.is_file() or path.parent != self.root: + return None + return path + + def write_record_check_config(self, workflow_id: str, effects: Mapping[str, Any]) -> Path: + """Write the read as a deployment settings file Flow loads for runs.""" + + ensure_private_dir(self.root) + path = self.root / f"{self._path(workflow_id).stem}.record-check.yaml" + tmp = path.with_suffix(".tmp") + tmp.write_text( + "# Written by OpenAdapt Desktop when a person (or their agent) said\n" + "# yes to the record check Check it learned from the recording.\n" + + yaml.safe_dump({"effects": dict(effects)}, sort_keys=False), + encoding="utf-8", + ) + make_private_file(tmp) + os.replace(tmp, path) + return path + + +def answer_entry(answered_by: str, **extra: Any) -> dict[str, Any]: + if answered_by not in ANSWERERS: + raise CheckSetupError("answered_by must be person or agent") + return {"by": answered_by, "at": _now(), **extra} + + +# --------------------------------------------------------------------------- +# The engine commands (thin wrappers in dispatch.py call these) +# --------------------------------------------------------------------------- + +_WRITE_CLASSES = frozenset({"state_changing", "consequential", "irreversible"}) +POLICY = "clinical-write" + + +def _plan(qualification: Mapping[str, Any]) -> dict[str, list[dict[str, Any]]]: + """Saving steps and unproven steps, the way Check it's plan sorts them.""" + + controls = (qualification.get("controls") or {}).get("actions") or {} + writes: list[dict[str, Any]] = [] + unproven: list[dict[str, Any]] = [] + for node in (qualification.get("graph") or {}).get("nodes") or []: + if node.get("kind") != "action": + continue + control = controls.get(node["id"]) or {} + classification = (control.get("classification") or {}).get("classification") + confirmed = bool((control.get("classification") or {}).get("operator_confirmed")) + is_write = ( + node.get("risk") == "irreversible" + or bool(node.get("effects")) + or bool(control.get("effects")) + or classification in _WRITE_CLASSES + ) + entry = {"id": node["id"], "node": node, "control": control} + if is_write: + writes.append(entry) + elif not (classification == "read_only" and confirmed): + unproven.append(entry) + return {"writes": writes, "unproven": unproven} + + +def _recording_dir(dispatcher: Any, workflow_id: str) -> Path | None: + row = dispatcher.services.db.get_bundle(workflow_id) or {} + capture_id = row.get("capture_id") + capture = dispatcher.services.db.get_capture(str(capture_id)) if capture_id else None + raw = capture and (capture.get("capture_path") or capture.get("capture_dir")) + return Path(str(raw)) if raw else None + + +def _sibling_recordings(dispatcher: Any, workflow_id: str) -> list[Path]: + """Recordings of the same workflow: same description as this one's.""" + + own = _recording_dir(dispatcher, workflow_id) + row = dispatcher.services.db.get_bundle(workflow_id) or {} + capture = ( + dispatcher.services.db.get_capture(str(row["capture_id"])) + if row.get("capture_id") + else None + ) + description = str((capture or {}).get("task_description") or "").strip().lower() + found = [own] if own else [] + if not description: + return found + for other in dispatcher.services.db.list_captures(limit=200): + if str(other.get("task_description") or "").strip().lower() != description: + continue + raw = other.get("capture_path") or other.get("capture_dir") + if raw and Path(str(raw)) not in found and Path(str(raw)).is_dir(): + found.append(Path(str(raw))) + return found + + +def get_check_setup(dispatcher: Any, workflow_id: str) -> dict[str, Any]: + """Everything Check it can fill in, and what is still a question.""" + + store = SetupStore(dispatcher.config.data_dir) + saved = store.load(workflow_id) + inputs = dispatcher._task_inputs(workflow_id) + secrets = frozenset(str(item["name"]) for item in inputs if item.get("secret")) + recording = _recording_dir(dispatcher, workflow_id) + qualification = dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY) + plan = _plan(qualification) if qualification.get("ok") else {"writes": [], "unproven": []} + + if recording is None: + learned: dict[str, Any] = { + "found": False, + "reason": "OpenAdapt can't find this workflow's recording.", + } + else: + learned = learn_record_check( + recording, + inputs={str(item["name"]): item for item in inputs}, + secret_inputs=secrets, + ) + if learned.get("found") and len(plan["writes"]) != 1: + learned = { + "found": False, + "reason": ( + "This workflow saves in more than one step, so OpenAdapt won't guess which " + "step the read belongs to." + ), + } + quiet = quiet_steps(recording) if recording is not None else None + recordings = _sibling_recordings(dispatcher, workflow_id) + values = value_proposals(inputs, recordings) + answered_values = saved["answers"].get("values") + if answered_values: + changing = set(answered_values.get("changes") or []) + for item in values: + item["changes"] = item["name"] in changing + return { + "ok": True, + "schema": SETUP_SCHEMA, + "workflow_id": workflow_id, + "record_check": learned, + "saving_steps": [entry["id"] for entry in plan["writes"]], + "quiet_steps": ( + {**quiet, "step_ids": [entry["id"] for entry in plan["unproven"]]} + if quiet and plan["unproven"] + else None + ), + "unproven_steps": [entry["id"] for entry in plan["unproven"]], + "values": values, + "recordings": len(recordings), + "answers": saved["answers"], + "fixed_values": saved["fixed_values"], + # Said here so an agent reading this knows its limit. + "approval": "A person approves in OpenAdapt Desktop. An agent can't.", + } + + +def _ok(result: Mapping[str, Any], what: str) -> Mapping[str, Any]: + if not result.get("ok"): + raise CheckSetupError(f"{what}: {result.get('error') or 'refused'}") + return result + + +def _start_project(dispatcher: Any, workflow_id: str) -> None: + """Start the check with the app and environment from the recording.""" + + draft = dispatcher.propose_qualification_from_demo( + workflow_id=workflow_id, policy_pack="community", accept=False + ) + pins = {pin.get("kind"): pin for pin in ((draft.get("proposal") or {}).get("pins") or [])} + application = pins.get("application") or {} + environment = pins.get("environment") or {} + if application.get("status") == "missing" or environment.get("status") == "missing": + raise CheckSetupError( + "OpenAdapt couldn't tell which app this is from the recording. " + "Say which app in Check it." + ) + app = application.get("payload") or {} + env = environment.get("payload") or {} + target_kind = str(app.get("target_kind") or "web") + version = str(app.get("application_version") or "recorded") + digest = str(env.get("environment_digest") or "") + observer: dict[str, str] | None = None + target = dispatcher._task_target(workflow_id) or {} + recorded_origin = str(app.get("application_identity") or "") + if target_kind == "web" and target.get("browser_cdp_endpoint") and recorded_origin: + # The person's signed-in browser: the app is known by its address + # and signed-in session, at the version recorded. A sign-out or a + # move to another address stops the run before the next input. + from openadapt_flow.qualification_web_session import ( + SIGNED_IN_WEB_SESSION_OBSERVER_ID, + signed_in_web_session_contract_sha256, + signed_in_web_session_environment_digest, + ) + + digest = signed_in_web_session_environment_digest(recorded_origin, version) + observer = { + "environment_observer_id": SIGNED_IN_WEB_SESSION_OBSERVER_ID, + "environment_observer_contract_sha256": signed_in_web_session_contract_sha256(version), + "application_identity": recorded_origin, + } + elif target_kind == "web" and target.get("url"): + # A browser OpenAdapt opens, on an app without OpenAdapt's markers: + # bind Flow's generic web observer (origin, browser, viewport, + # platform, runtime, declared version), read once from the recorded + # address. Its session part is the page load, so an app that reloads + # the whole page mid-run stops at the next input; that fails closed. + from openadapt_flow.qualify.environment import observe_web_environment + + observer = observe_web_environment( + str(target["url"]), + application_version=version, + headless=not bool(target.get("headed")), + ) + digest = observer["environment_digest"] + _ok( + dispatcher.initialize_qualification( + workflow_id=workflow_id, + policy=POLICY, + target_kind=target_kind, + application=str(app.get("application") or ""), + application_version=version, + environment_digest=digest, + required_capabilities=list(env.get("required_capabilities") or []), + **({"environment_observer": observer} if observer else {}), + ), + "Starting the check", + ) + + +def accept_record_check(dispatcher: Any, workflow_id: str, *, answered_by: str) -> dict[str, Any]: + """Yes to "Is this the right record check?". + + Each part is the command the matching form sends: start the check with + the recorded app, mark the steps the capture shows sent nothing as only + reading, use the record details seen before the save, confirm what the + save step changes, and bind the learned record check. + """ + + setup = get_check_setup(dispatcher, workflow_id) + learned = setup["record_check"] + if not learned.get("found"): + raise CheckSetupError( + "OpenAdapt has no record check to propose for this workflow. " + "Answer the questions in Check it." + ) + current = dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY) + if not current.get("project") or current.get("draft_environment"): + _start_project(dispatcher, workflow_id) + current = dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY) + plan = _plan(current) + quiet = setup.get("quiet_steps") + if quiet: + for entry in plan["unproven"]: + _ok( + dispatcher.set_qualification_risk( + workflow_id=workflow_id, + policy=POLICY, + step_id=entry["id"], + risk="read_only", + explanation=( + f"{quiet['evidence']} Proposed from the recording's network capture; " + f"confirmed in Check it by the {answered_by}." + ), + ), + "Marking a step as only reading", + ) + for entry in plan["writes"]: + control = entry["control"] + identity = control.get("identity") or {} + # The project needs its own identity policy for the save, even when + # the compiled step already carries the recorded identity gate. + if not identity.get("policy"): + if identity.get("can_arm"): + _ok( + dispatcher.arm_qualification_identity( + workflow_id=workflow_id, policy=POLICY, step_id=entry["id"] + ), + "Using the record details from the recording", + ) + current_class = (control.get("classification") or {}).get("classification") + risk = ( + "irreversible" + if entry["node"].get("risk") == "irreversible" + else current_class + if current_class in _WRITE_CLASSES + else "consequential" + ) + _ok( + dispatcher.set_qualification_risk( + workflow_id=workflow_id, + policy=POLICY, + step_id=entry["id"], + risk=risk, + explanation=f"Confirmed in Check it by the {answered_by}.", + ), + "Confirming what the save changes", + ) + has_checks = bool(control.get("effects")) and all( + isinstance(effect.get("verification_tier"), int) for effect in control["effects"] + ) + if not has_checks: + _ok( + dispatcher.bind_qualification_effect( + workflow_id=workflow_id, + policy=POLICY, + step_id=entry["id"], + **learned["binding"], + ), + "Saving the record check", + ) + store = SetupStore(dispatcher.config.data_dir) + config = store.write_record_check_config(workflow_id, learned["effects"]) + saved = store.load(workflow_id) + saved["record_check_config"] = str(config) + saved["answers"]["record_check"] = answer_entry( + answered_by, + answer="yes", + read=learned.get("read"), + quiet_steps=len(plan["unproven"]) if quiet else 0, + ) + store.save(workflow_id, saved) + return get_check_setup(dispatcher, workflow_id) + + +def answer_values( + dispatcher: Any, workflow_id: str, changes: list[str], *, answered_by: str +) -> dict[str, Any]: + """Answer "Which values change per item?". + + A value that doesn't change is filled in with its recorded value on + every item. Nothing about the workflow itself changes. + """ + + inputs = [item for item in dispatcher._task_inputs(workflow_id) if not item.get("secret")] + names = {str(item["name"]) for item in inputs} + unknown = sorted(set(changes) - names) + if unknown: + raise CheckSetupError("Unknown value(s): " + ", ".join(unknown)) + fixed: dict[str, str] = {} + for item in inputs: + name = str(item["name"]) + if name in changes: + continue + if item.get("example") is None or str(item["example"]) == "": + raise CheckSetupError( + f"OpenAdapt has no recorded value for {name}, so it must change per item." + ) + fixed[name] = str(item["example"]) + store = SetupStore(dispatcher.config.data_dir) + saved = store.load(workflow_id) + saved["fixed_values"] = fixed + saved["answers"]["values"] = answer_entry(answered_by, changes=sorted(changes)) + store.save(workflow_id, saved) + return get_check_setup(dispatcher, workflow_id) + + +def carry_to_sealed(data_dir: Path, workflow_id: str, sealed_id: str) -> None: + """Approving seals a new version; its answers come with it.""" + + store = SetupStore(data_dir) + saved = store.load(workflow_id) + if not saved["answers"] and not saved["fixed_values"]: + return + config = store.record_check_config(workflow_id) + if config is not None: + effects = (yaml.safe_load(config.read_text("utf-8")) or {}).get("effects") or {} + saved["record_check_config"] = str(store.write_record_check_config(sealed_id, effects)) + saved["sealed_from"] = workflow_id + store.save(sealed_id, saved) + + +def run_checks(dispatcher: Any, workflow_id: str) -> dict[str, Any]: + """The practice item on the recorded values, then the safety drills. + + The practice item saves for real: Flow stops before its save until a + person approves it in the window (or it times out and saves nothing). A + drill never saves. Nothing here approves the workflow. + """ + + setup = get_check_setup(dispatcher, workflow_id) + current = dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY) + if "record_check" not in setup["answers"] and not current.get("project"): + raise CheckSetupError('Answer "Is this the right record check?" first.') + if not ((current.get("graph") or {}).get("bundle") or {}).get("encrypted"): + # Flow binds each case result to the exact executable workflow, so + # the cases run on the sealed version that will be approved and run. + sealed = dispatcher.seal_qualification_workflow(workflow_id=workflow_id) + if not sealed.get("ok") or not sealed.get("workflow_id"): + raise CheckSetupError(f"Locking this version: {sealed.get('error') or 'refused'}") + dispatcher.emit( + "check_setup", + { + "workflow_id": workflow_id, + "sealed_id": sealed["workflow_id"], + "answered_by": "agent", + }, + ) + workflow_id = str(sealed["workflow_id"]) + values = { + str(item["name"]): str(item["example"]) + for item in dispatcher._task_inputs(workflow_id) + if not item.get("secret") and item.get("example") is not None + } + parameters = json.dumps(values, sort_keys=True) + target = dispatcher._task_target(workflow_id) + common: dict[str, Any] = { + "workflow_id": workflow_id, + "policy": POLICY, + "parameters_json": parameters, + } + if target: + common["target"] = target + + def cases() -> list[dict[str, Any]]: + current = dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY) + return list((current.get("project") or {}).get("cases") or []) + + def passed(case: Mapping[str, Any], revision: Any) -> bool: + return any( + result.get("status") == "passed" and result.get("project_revision") == revision + for result in case.get("results") or [] + ) + + practice_id = "practice-1" + existing = [case for case in cases() if case.get("kind") in {"representative", "nominal"}] + if existing: + practice_id = str(existing[0]["id"]) + else: + _ok( + dispatcher.add_qualification_case( + **common, + case_id=practice_id, + kind="representative", + description="Practice run on a test item", + ), + "Setting up the practice item", + ) + revision = ( + dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY).get("project") or {} + ).get("revision") + practice = next(case for case in cases() if case["id"] == practice_id) + summary: dict[str, Any] = {"practice": None, "drills": []} + if not passed(practice, revision): + result = dispatcher.run_qualification_case(**common, case_id=practice_id) + if result.get("not_approved"): + summary["practice"] = "not approved: nothing was saved" + return { + "ok": False, + "workflow_id": workflow_id, + **summary, + "error": result.get("error"), + } + if not result.get("ok"): + summary["practice"] = "did not pass" + return { + "ok": False, + "workflow_id": workflow_id, + **summary, + "error": result.get("error"), + } + summary["practice"] = "passed" + for case in cases(): + if case.get("kind") in {"representative", "nominal"}: + continue + revision = ( + dispatcher.get_qualification(workflow_id=workflow_id, policy=POLICY).get("project") + or {} + ).get("revision") + if passed(case, revision): + summary["drills"].append({"kind": case["kind"], "result": "passed"}) + continue + step = (case.get("fault_target") or {}).get("step_id") + result = dispatcher.run_qualification_case( + **common, + case_id=case["id"], + **({"fault_target": {"step_id": step, "actuation_path": "gui"}} if step else {}), + ) + summary["drills"].append( + {"kind": case["kind"], "result": "passed" if result.get("ok") else "did not pass"} + ) + if not result.get("ok"): + return { + "ok": False, + "workflow_id": workflow_id, + **summary, + "error": result.get("error"), + } + return { + "ok": True, + "workflow_id": workflow_id, + **summary, + "next": "A person approves the workflow in OpenAdapt Desktop.", + } diff --git a/packages/desktop/openadapt_desktop/dispatch.py b/packages/desktop/openadapt_desktop/dispatch.py index 03e8b3289..2150cb1da 100644 --- a/packages/desktop/openadapt_desktop/dispatch.py +++ b/packages/desktop/openadapt_desktop/dispatch.py @@ -287,6 +287,11 @@ def _register(self) -> None: "retry_run_persistence": self.retry_run_persistence, # simple operator surface: inputs, items, the inbox, results "get_task_inputs": self.get_task_inputs, + # Check it, pre-filled from the recording: three questions. The + # local agent socket serves these too; approval stays a person's. + "get_check_setup": self.get_check_setup, + "answer_check_setup": self.answer_check_setup, + "run_check_cases": self.run_check_cases, "run_item": self.run_item, "list_attention_items": self.list_attention_items, "resolve_attention_item": self.resolve_attention_item, @@ -1396,6 +1401,9 @@ def _replay_or_run( target, deployment_config = self._execution_target(params) except ValueError as exc: return self._pre_action_refusal(str(exc)) + if deployment_config is None: + # The record check a person said yes to in Check it. + deployment_config = self._learned_record_check_config(str(workflow_id)) if target is not None: from openadapt_desktop.capabilities import CapabilityError, ensure_backend_capability @@ -2734,6 +2742,11 @@ def initialize_qualification(self, **params: Any) -> dict: policy_source=policy, bundle_key=bundle_key, replace_first_replay_draft=replace_first_replay_draft, + environment_observer=( + {str(k): str(v) for k, v in params["environment_observer"].items()} + if isinstance(params.get("environment_observer"), dict) + else None + ), ) result["draft_environment"] = False self.services.db.update_bundle(workflow_id, status="qualification_pending") @@ -2876,6 +2889,17 @@ def bind_qualification_effect(self, **params: Any) -> dict: verification_tier=int(params.get("verification_tier", 3)), policy_source=policy, bundle_key=self._qualification_bundle_key(workflow_id), + extra_match=( + dict(params["extra_match"]) + if isinstance(params.get("extra_match"), dict) + else None + ), + idempotency_literal=( + str(params["idempotency_literal"]) + if params.get("idempotency_literal") is not None + else None + ), + forbid_collateral_loss=bool(params.get("forbid_collateral_loss", False)), ) self.services.db.update_bundle( workflow_id, @@ -3669,6 +3693,9 @@ def seal_qualification_workflow(self, **params: Any) -> dict: else "qualification_pending" ), ) + from openadapt_desktop.check_setup import carry_to_sealed + + carry_to_sealed(self.config.data_dir, workflow_id, new_id) return { "ok": True, "workflow_id": new_id, @@ -3968,6 +3995,98 @@ def _task_inputs(self, workflow_id: str) -> list[dict]: bundle_key=self._qualification_bundle_key(workflow_id), ) + def _fixed_values(self, workflow_id: str) -> dict[str, str]: + from openadapt_desktop.check_setup import CheckSetupError, SetupStore + + try: + return dict(SetupStore(self.config.data_dir).load(workflow_id)["fixed_values"]) + except CheckSetupError: + return {} + + def _learned_record_check_config(self, workflow_id: str) -> Path | None: + from openadapt_desktop.check_setup import CheckSetupError, SetupStore + + try: + return SetupStore(self.config.data_dir).record_check_config(workflow_id) + except CheckSetupError: + return None + + def get_check_setup(self, **params: Any) -> dict: + """What Check it can fill in from the recording, and what's still a question.""" + + from openadapt_desktop.check_setup import get_check_setup + + workflow_id = str(params.get("workflow_id") or "") + try: + return get_check_setup(self, workflow_id) + except Exception as exc: + return {"ok": False, "workflow_id": workflow_id, "error": str(exc)} + + def answer_check_setup(self, **params: Any) -> dict: + """Answer one Check it question: ``record_check`` (yes) or ``values``. + + ``answered_by`` is ``person`` (the window) or ``agent`` (the local + agent socket). There is no answer for approval here: a person + approves in the window. + """ + + from openadapt_desktop.check_setup import ( + CheckSetupError, + accept_record_check, + answer_values, + ) + + workflow_id = str(params.get("workflow_id") or "") + question = str(params.get("question") or "") + answered_by = str(params.get("answered_by") or "person") + try: + if question == "record_check": + if params.get("answer") != "yes": + raise CheckSetupError( + "Only yes is answered here. To change the record check, use the " + "form in Check it." + ) + result = accept_record_check(self, workflow_id, answered_by=answered_by) + elif question == "values": + changes = params.get("changes") + if not isinstance(changes, list) or not all(isinstance(n, str) for n in changes): + raise CheckSetupError("changes must be a list of value names") + result = answer_values(self, workflow_id, changes, answered_by=answered_by) + else: + raise CheckSetupError("question must be record_check or values") + except Exception as exc: + return {"ok": False, "workflow_id": workflow_id, "error": str(exc)} + self.emit( + "check_setup", + {"workflow_id": workflow_id, "question": question, "answered_by": answered_by}, + ) + return result + + def run_check_cases(self, **params: Any) -> dict: + """The practice item on the recorded values, then the drills. + + The practice save still waits for a person in the window. Used by + the person's agent; the window runs the same cases itself. + """ + + from openadapt_desktop.check_setup import run_checks + + workflow_id = str(params.get("workflow_id") or "") + try: + result = run_checks(self, workflow_id) + except Exception as exc: + result = {"ok": False, "workflow_id": workflow_id, "error": str(exc)} + # The window shows the results without a click. + self.emit( + "check_setup", + { + "workflow_id": result.get("workflow_id") or workflow_id, + "question": "checks", + "answered_by": "agent", + }, + ) + return result + def get_task_inputs(self, **params: Any) -> dict: """Return the inputs one run of a task needs and where it was recorded. @@ -3982,6 +4101,8 @@ def get_task_inputs(self, **params: Any) -> dict: "workflow_id": workflow_id, "inputs": self._task_inputs(workflow_id), "target": self._task_target(workflow_id), + # Values Check it said stay the same; Run it doesn't ask. + "fixed_values": self._fixed_values(workflow_id), } except Exception as exc: return {"ok": False, "workflow_id": workflow_id, "error": str(exc)} @@ -4255,8 +4376,14 @@ def run_item(self, **params: Any) -> dict: return self._pre_action_refusal( "Desktop couldn't read the inputs this task needs." ) + raw_inputs = params.get("inputs") + fixed = self._fixed_values(workflow_id) + if fixed and isinstance(raw_inputs, dict): + # A value Check it marked as the same on every item is filled in + # with its recorded value; a value the caller gives still wins. + raw_inputs = {**fixed, **raw_inputs} try: - values = validate_inputs(params.get("inputs"), declared) + values = validate_inputs(raw_inputs, declared) except WorkItemError as exc: return self._pre_action_refusal(str(exc)) diff --git a/packages/desktop/openadapt_desktop/qualification.py b/packages/desktop/openadapt_desktop/qualification.py index 6c4132594..67a7693df 100644 --- a/packages/desktop/openadapt_desktop/qualification.py +++ b/packages/desktop/openadapt_desktop/qualification.py @@ -1012,8 +1012,16 @@ def initialize_qualification( policy_source: str = DEFAULT_QUALIFICATION_POLICY, bundle_key: str | None = None, replace_first_replay_draft: bool = False, + environment_observer: dict[str, str] | None = None, ) -> dict: - """Attach the canonical v1 project to an existing compiled workflow.""" + """Attach the canonical v1 project to an existing compiled workflow. + + ``environment_observer`` binds the project to an observer Flow builds at + run time (``environment_observer_id``, ``..._contract_sha256``, and the + ``application_identity`` it observed). Without it the backend's own + marker observer applies, which a web app without OpenAdapt markers can't + satisfy. + """ if target_kind not in {"web", "windows", "macos", "linux", "rdp", "citrix"}: raise QualificationError("target_kind is not a supported execution surface") @@ -1038,6 +1046,7 @@ def initialize_qualification( "This workflow already has a qualification project; reopen it to continue." ) try: + observer = dict(environment_observer or {}) environment = api["EnvironmentBoundary"]( target_kind=target_kind, application=application, @@ -1045,6 +1054,15 @@ def initialize_qualification( environment_digest=environment_digest, runtime_version=_runtime_version(), required_capabilities=required_capabilities or [], + **{ + key: observer[key] + for key in ( + "environment_observer_id", + "environment_observer_contract_sha256", + "application_identity", + ) + if observer.get(key) + }, ) api["init_project"]( workflow, @@ -1596,8 +1614,19 @@ def bind_action_effect( verification_tier: int = 3, policy_source: str = DEFAULT_QUALIFICATION_POLICY, bundle_key: str | None = None, + extra_match: dict[str, dict[str, str]] | None = None, + idempotency_literal: str | None = None, + forbid_collateral_loss: bool = False, ) -> dict: - """Bind an executable Flow effect and its canonical evidence-strength policy.""" + """Bind an executable Flow effect and its canonical evidence-strength policy. + + ``extra_match`` adds selector fields beyond ``match_field`` (each + ``{"param": name}`` or ``{"literal": value}``), ``idempotency_literal`` is + a fixed key the app itself sends with the save, and + ``forbid_collateral_loss`` refuses a save that made another record + disappear. Each one only narrows what the check accepts. The record is + still selected through ``match_param``, a run parameter. + """ if kind not in ("record_written", "field_equals"): raise QualificationError("kind must be record_written or field_equals") @@ -1617,7 +1646,24 @@ def bind_action_effect( step = _resolve_action(workflow, step_id) secrets = set(workflow.secret_params) parameters = set(workflow.params) | set(workflow.param_specs) | secrets - requested = {name for name in (match_param, value_param, idempotency_param) if name is not None} + extra_match = dict(extra_match or {}) + extra_params: set[str] = set() + for field_name, expr in extra_match.items(): + if not str(field_name).strip() or str(field_name).strip() == match_field.strip(): + raise QualificationError("extra_match fields must be named and differ from match_field") + if not isinstance(expr, dict) or len(expr) != 1 or not ( + set(expr) <= {"param", "literal"} + ): + raise QualificationError( + "each extra_match value must be {param: ...} or {literal: ...}" + ) + if "param" in expr: + extra_params.add(str(expr["param"])) + if idempotency_param is not None and idempotency_literal is not None: + raise QualificationError("an idempotency key is a parameter or a literal, not both") + requested = { + name for name in (match_param, value_param, idempotency_param) if name is not None + } | extra_params unknown = sorted(requested - parameters) if unknown: raise QualificationError( @@ -1636,15 +1682,29 @@ def bind_action_effect( if count_new_only and kind != "record_written": raise QualificationError("count_new_only applies only to record_written") + match = {match_field.strip(): ValueExpr(param=match_param)} + for field_name, expr in sorted(extra_match.items()): + match[str(field_name).strip()] = ( + ValueExpr(param=str(expr["param"])) + if "param" in expr + else ValueExpr(literal=str(expr["literal"])) + ) + if idempotency_param: + idempotency_key = ValueExpr(param=idempotency_param) + elif idempotency_literal is not None and str(idempotency_literal).strip(): + idempotency_key = ValueExpr(literal=str(idempotency_literal)) + else: + idempotency_key = None effect = Effect( kind=EffectKind(kind), - match={match_field.strip(): ValueExpr(param=match_param)}, + match=match, field=field.strip() if field else None, value=ValueExpr(param=value_param) if value_param else None, expected_count=expected_count, - idempotency_key=(ValueExpr(param=idempotency_param) if idempotency_param else None), + idempotency_key=idempotency_key, key_field=key_field.strip(), count_new_only=count_new_only, + forbid_collateral_loss=forbid_collateral_loss, risk=step.risk, needs_operator_confirmation=False, ) @@ -1827,10 +1887,30 @@ def set_local_qualification_case_scope( not in executable_actuation_paths(fault_step) ): raise QualificationError("Fault target is outside the executable case scope") + from openadapt_flow.qualification import qualification_action_requirements + + # The case proves Flow's required actions: every step not confirmed as + # only reading (an unreviewed step stays required), plus the fault target. + # A step a person confirmed read-only isn't a qualified action, as in + # Flow's own qualify campaign. + required_actions, _required_identity = qualification_action_requirements(workflow) + scoped = { + step_id + for step_id, step in steps.items() + if executable_actuation_paths(step) + and ( + step_id in required_actions + or (selected_fault_target is not None and selected_fault_target.step_id == step_id) + ) + } + if not scoped: + # Nothing is required (every step only reads): the case still runs + # every executable step. + scoped = {step_id for step_id, step in steps.items() if executable_actuation_paths(step)} targets = [] for step_id, step in sorted(steps.items()): paths = executable_actuation_paths(step) - if not paths: + if not paths or step_id not in scoped: continue if selected_fault_target is not None and selected_fault_target.step_id == step_id: path = selected_fault_target.actuation_path @@ -2065,6 +2145,11 @@ def seal_qualification_bundle( provenance.certified = False provenance.certification_status = "expired" provenance.expires_at = datetime.now(timezone.utc).isoformat() + if provenance is not None and getattr(provenance, "governed_authorization_template", None): + # Derived from the certification cleared above. Left in place, the + # sealed copy can't be loaded: Flow rebuilds the template from the + # certification on load and refuses when it can't. + provenance.governed_authorization_template = None try: copy_bundle_version(source, destination) diff --git a/packages/desktop/openadapt_desktop/qualification_capabilities.py b/packages/desktop/openadapt_desktop/qualification_capabilities.py index 613f4a2bc..4dcd31555 100644 --- a/packages/desktop/openadapt_desktop/qualification_capabilities.py +++ b/packages/desktop/openadapt_desktop/qualification_capabilities.py @@ -10,6 +10,7 @@ from __future__ import annotations import base64 +import hashlib import json from collections.abc import Mapping from datetime import datetime, timezone @@ -175,6 +176,28 @@ def verify_qualification_capability_observation( return True +def _retained_run_receipt(observation_path: Path, run_id: str) -> dict | None: + """The run report this observation is bound to, as a receipt. + + ``retain_run_evidence`` keeps the exact ``report.json`` next to the + observation; its digest is the binding. Evidence retained before that + change kept a ``run-report-receipt.json`` instead, which is still read. + """ + + report_path = observation_path.with_name("report.json") + if report_path.is_file() and not report_path.is_symlink(): + return { + "schema": "openadapt.qualification-run-receipt/v1", + "run_id": run_id, + "report_sha256": hashlib.sha256(report_path.read_bytes()).hexdigest(), + } + receipt_path = observation_path.with_name("run-report-receipt.json") + if receipt_path.is_symlink() or not receipt_path.is_file(): + return None + receipt = json.loads(receipt_path.read_text(encoding="utf-8")) + return receipt if isinstance(receipt, dict) else None + + def current_signed_capability_observations( bundle_dir: Path, *, @@ -203,10 +226,9 @@ def current_signed_capability_observations( resolved.read_text(encoding="utf-8") ) public_key = project.trusted_runner_keys.get(observation.attestation_key_id) - run_receipt_path = resolved.with_name("run-report-receipt.json") - if run_receipt_path.is_symlink() or not run_receipt_path.is_file(): + run_receipt = _retained_run_receipt(resolved, observation.run_id) + if run_receipt is None: continue - run_receipt = json.loads(run_receipt_path.read_text(encoding="utf-8")) except (OSError, ValueError, json.JSONDecodeError): continue if ( diff --git a/packages/desktop/openadapt_desktop/qualification_lifecycle.py b/packages/desktop/openadapt_desktop/qualification_lifecycle.py index 499627eb1..08e5af25c 100644 --- a/packages/desktop/openadapt_desktop/qualification_lifecycle.py +++ b/packages/desktop/openadapt_desktop/qualification_lifecycle.py @@ -186,9 +186,84 @@ def retain_run_evidence( "sha256": input_digest, "relative_path": input_relative.as_posix(), }, + *_retain_report_artifacts( + bundle_dir, report=report, run_dir=run_dir, case_id=case_id, run_id=run_id + ), ] +#: The largest single frame or proof Desktop keeps with a case result, and +#: the most it keeps for one run. +_ARTIFACT_MAX_BYTES = 16 * 1024 * 1024 +_ARTIFACTS_MAX_TOTAL_BYTES = 256 * 1024 * 1024 + + +def _report_paths(node: object) -> set[str]: + """Every run-relative file path a run report names (frames, proofs).""" + + found: set[str] = set() + if isinstance(node, dict): + for value in node.values(): + found |= _report_paths(value) + elif isinstance(node, list): + for value in node: + found |= _report_paths(value) + elif ( + isinstance(node, str) + and "/" in node + and "\\" not in node + and not node.startswith(("/", "http:", "https:", "sha256:")) + and node.rsplit(".", 1)[-1].lower() in {"png", "json"} + ): + found.add(node) + return found + + +def _retain_report_artifacts( + bundle_dir: Path, *, report: dict, run_dir: Path, case_id: str, run_id: str +) -> list[dict[str, str]]: + """Keep the step frames and proofs the report points at, hash-bound. + + Flow's case check reads each action's pre-action frame and resolution + proof from the retained evidence, at the path the report names. Without + them a drill can't be credited. Only regular files inside the run folder + are kept; they stay inside the local qualification-evidence folder. + """ + + root = run_dir.resolve() + kept: list[dict[str, str]] = [] + total = 0 + for name in sorted(_report_paths(report.get("results") or [])): + relative = Path(name) + if relative.is_absolute() or ".." in relative.parts: + continue + source = run_dir / relative + try: + if source.is_symlink() or not source.is_file(): + continue + if not source.resolve().is_relative_to(root): + continue + size = source.stat().st_size + except OSError: + continue + if size > _ARTIFACT_MAX_BYTES or total + size > _ARTIFACTS_MAX_TOTAL_BYTES: + continue + payload = source.read_bytes() + total += size + evidence_relative = Path(case_id) / run_id / relative + destination = bundle_dir / "qualification-evidence" / evidence_relative + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes(payload) + kept.append( + { + "kind": "other", + "sha256": hashlib.sha256(payload).hexdigest(), + "relative_path": evidence_relative.as_posix(), + } + ) + return kept + + #: The largest drill receipt or mutation record Desktop keeps. _DRILL_EVIDENCE_MAX_BYTES = 1_000_000 diff --git a/packages/desktop/openadapt_desktop/socket_server.py b/packages/desktop/openadapt_desktop/socket_server.py index f0da0f317..12bab7dd3 100644 --- a/packages/desktop/openadapt_desktop/socket_server.py +++ b/packages/desktop/openadapt_desktop/socket_server.py @@ -57,7 +57,16 @@ # Local coding-agent coach. Same loopback socket as the tray, different # vocabulary: coach events are never forwarded to tray clients. _LOCAL_AGENT_COMMANDS = frozenset({"set_coach", "get_coach", "clear_coach"}) -_SOCKET_COMMANDS = _TRAY_COMMANDS | _LOCAL_AGENT_COMMANDS + +# The person's own agent can set up Check it: read what OpenAdapt filled in +# from the recording, answer the record-check and values questions, and run +# the practice item and drills (the practice save still waits for a person in +# the window). Approving is not here: only a person approves, in the window. +# Every answer that arrives here is recorded as the agent's. +_SETUP_AGENT_COMMANDS = frozenset( + {"get_workflows", "get_check_setup", "answer_check_setup", "run_check_cases"} +) +_SOCKET_COMMANDS = _TRAY_COMMANDS | _LOCAL_AGENT_COMMANDS | _SETUP_AGENT_COMMANDS # Events the tray's IPCMessageType enum can decode. Anything else is dropped # before forwarding so the tray never hits a from_json ValueError. @@ -233,7 +242,10 @@ def _handle_frame(self, conn: socket.socket, line: str) -> None: if cmd not in _SOCKET_COMMANDS: logger.warning("Tray sent unsupported command: {c}", c=cmd) return - params = frame.get("data") or {} + params = dict(frame.get("data") or {}) + if cmd == "answer_check_setup": + # A socket caller is never the person at the window. + params["answered_by"] = "agent" try: with self._dispatch_lock: result = self.dispatcher.dispatch(cmd, params) @@ -249,6 +261,8 @@ def _handle_frame(self, conn: socket.socket, line: str) -> None: # to a tray client that cannot decode the type. if cmd in _LOCAL_AGENT_COMMANDS and result is not None: self._send(conn, "coach", result) + if cmd in _SETUP_AGENT_COMMANDS: + self._send(conn, "check_setup", {"cmd": cmd, "result": result}) # ------------------------------------------------------------- send/emit diff --git a/packages/desktop/package-lock.json b/packages/desktop/package-lock.json index 6a0a57428..2ec775247 100644 --- a/packages/desktop/package-lock.json +++ b/packages/desktop/package-lock.json @@ -1,12 +1,12 @@ { "name": "openadapt-desktop", - "version": "0.21.1", + "version": "0.22.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "openadapt-desktop", - "version": "0.21.1", + "version": "0.22.0", "dependencies": { "@tauri-apps/api": "^2", "@tauri-apps/plugin-notification": "^2", diff --git a/packages/desktop/package.json b/packages/desktop/package.json index 32a33a5d7..df6ea3b11 100644 --- a/packages/desktop/package.json +++ b/packages/desktop/package.json @@ -1,6 +1,6 @@ { "name": "openadapt-desktop", - "version": "0.21.1", + "version": "0.22.0", "description": "Installed companion for OpenAdapt authoring, teaching, and local pairing", "private": true, "type": "module", diff --git a/packages/desktop/pyproject.toml b/packages/desktop/pyproject.toml index e06f4b378..2ca86cc93 100644 --- a/packages/desktop/pyproject.toml +++ b/packages/desktop/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "openadapt-desktop" -version = "0.21.1" +version = "0.22.0" description = "The OpenAdapt desktop app and local engine: record a data-entry task, run it on a list, and check each saved record" readme = "README.md" requires-python = ">=3.11" diff --git a/packages/desktop/scripts/setup_walk/bridge.py b/packages/desktop/scripts/setup_walk/bridge.py new file mode 100644 index 000000000..4bf6b1fd9 --- /dev/null +++ b/packages/desktop/scripts/setup_walk/bridge.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""A loopback HTTP bridge from a browser page to a real Desktop engine. + +The setup walk (``walk.py``) drives the Desktop UI from ``npm run dev`` in a +headless browser. The page reaches the engine through Tauri's ``invoke``; the +walk's init script answers ``engine_invoke`` by posting here instead, and this +process hands the command to a real :class:`EngineDispatcher` over a private +data directory. Engine events queue here and the page polls for them, so the +approval dialog and progress events behave as they do in the app. + +Development and audit use only. It binds 127.0.0.1, accepts only the walk's +per-run token, and never starts with a user's data directory: ``--data-dir`` +must be a fresh folder the walk created. +""" + +from __future__ import annotations + +import argparse +import json +import threading +import traceback +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +from openadapt_desktop.config import EngineConfig +from openadapt_desktop.db import IndexDB +from openadapt_desktop.dispatch import EngineDispatcher, EngineServices +from openadapt_desktop.ipc import CONCURRENT_COMMANDS + + +class Bridge: + """One dispatcher, one event queue, one log of every call the page made.""" + + def __init__(self, data_dir: Path, token: str) -> None: + self.token = token + self.events: list[dict[str, Any]] = [] + self.calls: list[dict[str, Any]] = [] + self._lock = threading.Lock() + # Commands run one at a time, as on the app's stdin wire (ipc.py), + # except the two that answer a run waiting for a person. + self._dispatch_lock = threading.RLock() + config = EngineConfig(data_dir=data_dir / ".openadapt", log_level="WARNING") + self.db = IndexDB(data_dir / "index.db") + self.db.initialize() + self.dispatcher = EngineDispatcher( + config, services=EngineServices(config, db=self.db), emit=self._emit + ) + + def _emit(self, event: str, data: dict) -> None: + with self._lock: + self.events.append({"event": event, "payload": data}) + + def drain(self) -> list[dict[str, Any]]: + with self._lock: + out, self.events = self.events, [] + return out + + def invoke(self, cmd: str, params: dict[str, Any]) -> Any: + with self._lock: + self.calls.append({"cmd": cmd, "params": sorted(params)}) + if cmd in CONCURRENT_COMMANDS: + return self.dispatcher.dispatch(cmd, params) + with self._dispatch_lock: + return self.dispatcher.dispatch(cmd, params) + + +def serve(bridge: Bridge, port: int) -> ThreadingHTTPServer: + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args: Any) -> None: # quiet + return + + def _cors(self) -> None: + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Access-Control-Allow-Headers", "content-type, x-walk-token") + self.send_header("Access-Control-Allow-Methods", "POST, GET, OPTIONS") + + def do_OPTIONS(self) -> None: # noqa: N802 + self.send_response(204) + self._cors() + self.end_headers() + + def _reply(self, status: int, body: Any) -> None: + data = json.dumps(body, default=str).encode("utf-8") + self.send_response(status) + self._cors() + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_POST(self) -> None: # noqa: N802 + if self.headers.get("x-walk-token") != bridge.token: + self._reply(403, {"error": "bad token"}) + return + length = int(self.headers.get("Content-Length") or 0) + request = json.loads(self.rfile.read(length) or b"{}") + if self.path == "/events": + self._reply(200, bridge.drain()) + return + if self.path == "/calls": + self._reply(200, bridge.calls) + return + try: + result = bridge.invoke(str(request["cmd"]), dict(request.get("params") or {})) + self._reply(200, {"ok": True, "result": result}) + except KeyError as exc: + self._reply(200, {"ok": False, "error": f"unknown command {exc}"}) + except Exception as exc: # the page sees the engine's failure + traceback.print_exc() + self._reply(200, {"ok": False, "error": str(exc)}) + + server = ThreadingHTTPServer(("127.0.0.1", port), Handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + return server + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--data-dir", type=Path, required=True) + parser.add_argument("--port", type=int, default=0) + parser.add_argument("--token", required=True) + args = parser.parse_args() + bridge = Bridge(args.data_dir, args.token) + server = serve(bridge, args.port) + print(json.dumps({"port": server.server_address[1]}), flush=True) + threading.Event().wait() + + +if __name__ == "__main__": + main() diff --git a/packages/desktop/scripts/setup_walk/seed.py b/packages/desktop/scripts/setup_walk/seed.py new file mode 100644 index 000000000..15838a844 --- /dev/null +++ b/packages/desktop/scripts/setup_walk/seed.py @@ -0,0 +1,141 @@ +"""Record the MockMed triage demonstration and give it to a fresh engine. + +The recording is the one Flow's tutorial makes (``openadapt_flow.tutorial``): +MockMed served through its transactional fault server with ``?fault=ok``, so +a save reaches a backend store. The person's part (open the referral, start a +triage encounter, type the note, save) goes through Flow's ordinary +``Recorder``. + +Desktop's recorder does not yet keep the app's network traffic +(``record --capture-network`` is a separate change), and MockMed's page never +reads its store back after a save. So the walk writes ``network.jsonl`` in +the ``openadapt.network-capture/1`` format itself: the app's real save +request, captured from the browser, followed by one read of ``GET /api/db`` +made right after the save. That read stands in for the screen an app shows +after saving; Flow's own record-check tests build the same MockMed recording +(``tests/_record_check_fixtures.py::mockmed_recording``). Nothing else is +added. All data is synthetic. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any +from urllib.request import Request, urlopen + +SCHEMA = "openadapt.network-capture/1" +NOTE = "Synthetic follow-up in two weeks" +ENTRY_QUERY = "?fault=ok&idempotency=demo#tasks" +TASK = "Add a triage note" + + +def _get(url: str) -> tuple[int, str, str]: + with urlopen(Request(url), timeout=10) as response: # noqa: S310 - loopback + return response.status, response.headers.get("content-type", ""), response.read().decode() + + +def _post(url: str, body: dict) -> None: + data = json.dumps(body).encode() + request = Request(url, data=data, method="POST", headers={"Content-Type": "application/json"}) + with urlopen(request, timeout=10): # noqa: S310 - loopback + pass + + +def record(base_url: str, recording: Path) -> Path: + """Record the demonstration with the app's save request captured.""" + + from openadapt_flow.backends.playwright_backend import PlaywrightBackend + from openadapt_flow.recorder import Recorder + + base = base_url.rstrip("/") + _post(f"{base}/api/reset", {}) + entry = f"{base}/{ENTRY_QUERY}" + backend, close = PlaywrightBackend.launch(entry, headless=True) + started = time.monotonic() + lines: list[dict[str, Any]] = [] + + def on_finished(request: Any) -> None: + if "/api/" not in request.url: + return + response = request.response() + line = { + "method": request.method, + "url": request.url, + "resource_type": request.resource_type, + "frame_url": request.frame.url, + "status": response.status if response else 0, + "response_type": ( + (response.headers.get("content-type") if response else "") or "" + ).split(";")[0], + "response_text": response.text() if response else "", + "seq": len(lines), + "t": round(time.monotonic() - started, 3), + } + if request.post_data: + line["request_type"] = "application/json" + line["body"] = request.post_data + lines.append(line) + + try: + page = backend.page + page.on("requestfinished", on_finished) + recorder = Recorder(backend, recording, app_url=entry) + # Stamp requests on the recorder's own clock, as the recorder's + # network capture does, so a request lines up with its step. + started = getattr(recorder, "_t0", started) + + def center(selector: str) -> tuple[int, int]: + locator = page.locator(selector).first + locator.wait_for(state="visible") + box = locator.bounding_box() + assert box is not None + return int(box["x"] + box["width"] / 2), int(box["y"] + box["height"] / 2) + + recorder.click(*center(".open-btn")) + recorder.click(*center("#new-encounter")) + recorder.click(*center("#type-triage")) + recorder.click(*center("#note")) + recorder.type_text(NOTE, param="note") + recorder.click(*center("#save-encounter")) + page.wait_for_selector("#saved-banner", state="visible") + page.wait_for_timeout(300) + recorder.finish() + finally: + close() + + # The read after the save (see the module docstring). + status, content_type, text = _get(f"{base}/api/db") + lines.append( + { + "method": "GET", + "url": f"{base}/api/db", + "resource_type": "xhr", + "frame_url": entry, + "status": status, + "response_type": content_type.split(";")[0], + "response_text": text, + "seq": len(lines), + "t": round(time.monotonic() - started, 3), + } + ) + with (recording / "network.jsonl").open("w", encoding="utf-8") as fh: + fh.write(json.dumps({"schema": SCHEMA, "exchanges": len(lines)}) + "\n") + for line in lines: + fh.write(json.dumps(line, sort_keys=True) + "\n") + return recording + + +def seed(bridge: Any, recording: Path, base_url: str, capture_id: str = "triage-note") -> str: + """Register the recording as a finished capture and compile it.""" + + db = bridge.db + db.insert_capture(capture_id, str(recording), "2026-10-19T00:00:00Z", task_description=TASK) + db.set_task_target( + capture_id, {"backend": "web", "url": f"{base_url.rstrip('/')}/{ENTRY_QUERY}"} + ) + result = bridge.invoke("compile_recording", {"capture_id": capture_id}) + if not result.get("ok"): + raise SystemExit(f"compile failed: {result}") + return str(result["workflow_id"]) diff --git a/packages/desktop/scripts/setup_walk/shrink.py b/packages/desktop/scripts/setup_walk/shrink.py new file mode 100644 index 000000000..5ca61fbe6 --- /dev/null +++ b/packages/desktop/scripts/setup_walk/shrink.py @@ -0,0 +1,14 @@ +"""Shrink the walk's screenshots for the repository: 256-colour PNG, same pixels size.""" + +import sys +from pathlib import Path + +from PIL import Image + +for folder in sys.argv[1:]: + for path in sorted(Path(folder).glob("*.png")): + with Image.open(path) as image: + small = image.convert("RGB").quantize( + colors=256, method=Image.Quantize.MEDIANCUT, dither=Image.Dither.NONE + ) + small.save(path, optimize=True) diff --git a/packages/desktop/scripts/setup_walk/steps_after.py b/packages/desktop/scripts/setup_walk/steps_after.py new file mode 100644 index 000000000..45b78a571 --- /dev/null +++ b/packages/desktop/scripts/setup_walk/steps_after.py @@ -0,0 +1,107 @@ +"""Inputs on the screens after this change: three questions. + +The same recording and the same MockMed as the "before" walk. MockMed's demo +key is the same for every save on one patient, and its server keeps only the +first save with a key, so the walk empties MockMed's test store before the +practice item and before the first item, as Flow's tutorial does before each +of its runs. Nothing else is set up by hand: no settings file, no form. +""" + +from walk import Walk, button + +#: The first real item: a new note, typed in Run it. +FIRST_ITEM_NOTE = "Synthetic second visit: booked a call" + + +def run(walk: Walk, *, reset_store, **_: object) -> None: + page = walk.page + walk.give( + "Workflows", + "navigation", + "Check it", + lambda: button(page, "Check it").click(), + name="workflows", + ) + walk.wait_text("Is this the right record check?", timeout=120_000) + page.wait_for_timeout(1500) + + walk.give( + "Check it: question 1", + "decision", + "Is this the right record check? Yes, use this check", + lambda: button(page, "Yes, use this check").click(), + name="q1-record-check", + ) + page.wait_for_function( + "() => [...document.querySelectorAll('button')].some((b) => b.textContent.trim() === " + "'Yes, practice once with these test values' && !b.disabled)", + timeout=300_000, + ) + + reset_store() + walk.give( + "Check it: question 2", + "decision", + "Which values change per item? Yes, practice once with these test values", + lambda: button(page, "Yes, practice once with these test values").click(), + name="q2-values", + ) + page.get_by_role("button", name="Approve and save").wait_for(timeout=600_000) + walk.give( + "Approval dialog", + "confirmation", + "Approve and save (the practice item's save)", + lambda: button(page, "Approve and save").click(), + name="practice-approve-save", + ) + # The drills run on their own after the practice item. + approve = page.get_by_role("button", name="Approve this workflow") + page.wait_for_function( + "() => [...document.querySelectorAll('button')].some((b) => b.textContent.trim() === " + "'Approve this workflow' && !b.disabled)", + timeout=1_500_000, + ) + walk.give( + "Check it: question 3", + "decision", + "Approve? Approve this workflow", + lambda: approve.click(), + name="q3-approve", + ) + walk.wait_text("Approved and locked", timeout=300_000) + walk.shot("approved") + + walk.give( + "Check it", "navigation", "Run it", lambda: button(page, "Run it").click(), name="run-it" + ) + page.wait_for_timeout(2000) + page.get_by_role("radio", name="Type one").or_( + page.get_by_role("button", name="Type one") + ).first.click() + page.wait_for_timeout(500) + walk.give( + "Run it", + "item_value", + "Note (the first item's own value)", + lambda: page.get_by_label("Note").fill(FIRST_ITEM_NOTE), + name="item-note", + ) + reset_store() + walk.give( + "Run it", + "item_run", + "Run 1 item", + lambda: button(page, "Run 1 item").click(), + name="item-run", + ) + page.get_by_role("button", name="Approve and save").wait_for(timeout=600_000) + walk.give( + "Approval dialog", + "item_approval", + "Approve and save (the first item's save)", + lambda: button(page, "Approve and save").click(), + name="item-approve-save", + ) + walk.wait_text("See results", timeout=600_000) + page.wait_for_timeout(1500) + walk.shot("first-item-done") diff --git a/packages/desktop/scripts/setup_walk/steps_agent.py b/packages/desktop/scripts/setup_walk/steps_agent.py new file mode 100644 index 000000000..7f1fcaff8 --- /dev/null +++ b/packages/desktop/scripts/setup_walk/steps_agent.py @@ -0,0 +1,72 @@ +"""The person's agent sets the workflow up over MCP; the person only approves. + +The agent's calls go through openadapt_desktop.agent_mcp to the engine's +local socket, exactly as an MCP client's would. The window shows what the +agent answered. The person approves the practice item's save and the +workflow, in the window. +""" + +import threading + +from walk import Walk, button + +FIRST_ITEM_NOTE = "Synthetic second visit: booked a call" + + +def run(walk: Walk, *, reset_store, agent_log: list, **_: object) -> None: + from openadapt_desktop import agent_mcp + + page = walk.page + walk.give( + "Workflows", + "navigation", + "Check it", + lambda: button(page, "Check it").click(), + name="workflows", + ) + walk.wait_text("Is this the right record check?", timeout=120_000) + + def agent() -> None: + def call(name: str, **arguments: object) -> object: + result = agent_mcp.handle(name, dict(arguments)) + agent_log.append({"tool": name, "arguments": arguments, "result": result}) + return result + + workflows = call("list_workflows") + workflow_id = workflows[0]["workflow_id"] + call("get_check_setup", workflow_id=workflow_id) + call("answer_record_check", workflow_id=workflow_id) + call("answer_values", workflow_id=workflow_id, changes=["note"]) + reset_store() + call("run_checks", workflow_id=workflow_id) + + thread = threading.Thread(target=agent, daemon=True) + thread.start() + # The agent's practice item holds the engine's command queue until it + # ends, as any run does; only the approval commands bypass it. So the + # approval dialog comes first, and the window catches up afterwards. + page.get_by_role("button", name="Approve and save").wait_for(timeout=900_000) + walk.give( + "Approval dialog", + "confirmation", + "Approve and save (the practice item's save)", + lambda: button(page, "Approve and save").click(), + name="practice-approve-save", + ) + thread.join(timeout=1_800) + walk.wait_text("Your agent answered for you", timeout=300_000) + page.wait_for_function( + "() => [...document.querySelectorAll('button')].some((b) => b.textContent.trim() === " + "'Approve this workflow' && !b.disabled)", + timeout=300_000, + ) + page.locator("summary", has_text="What you").first.click() + walk.give( + "Check it: question 3", + "decision", + "Approve? Approve this workflow", + lambda: button(page, "Approve this workflow").click(), + name="q3-approve", + ) + walk.wait_text("Approved and locked", timeout=300_000) + walk.shot("approved") diff --git a/packages/desktop/scripts/setup_walk/steps_before.py b/packages/desktop/scripts/setup_walk/steps_before.py new file mode 100644 index 000000000..d3160bfdb --- /dev/null +++ b/packages/desktop/scripts/setup_walk/steps_before.py @@ -0,0 +1,165 @@ +"""Inputs on the screens before this change (Calm UI, #1310). + +The answers are the ones a person who knows MockMed would give. The record +check file is the one IT would hand over: MockMed's store read at /api/db. +""" + +from pathlib import Path + +from walk import Walk, button + + +def it_settings_file(work: Path, mockmed_url: str) -> Path: + path = work / "it-deployment.yaml" + path.write_text( + "effects:\n" + " kind: rest\n" + f" base_url: {mockmed_url.rstrip('/')}\n" + " records_path: /api/db\n" + " records_key: records\n" + ) + return path + + +def run(walk: Walk, *, mockmed_url: str, work: Path, **_: object) -> None: + page = walk.page + settings = str(it_settings_file(work, mockmed_url)) + walk.give( + "Workflows", + "navigation", + "Check it", + lambda: button(page, "Check it").click(), + name="workflows", + ) + page.wait_for_timeout(4000) + walk.give( + "Check it", + "decision", + "Start the check with the app from your recording", + lambda: button(page, "Start the check with the app from your recording").click(), + name="start-check", + ) + page.wait_for_timeout(3000) + walk.give( + "Check it: how it confirms the save", + "choice", + "What does this step save? (It adds a new entry)", + lambda: page.get_by_label("It adds a new entry to a record").check(), + name="effect-kind", + ) + walk.give( + "Check it: how it confirms the save", + "choice", + "Which value identifies the record? (Note)", + lambda: page.get_by_label("Which value identifies the record?").select_option("note"), + name="effect-match-param", + note="Already Note; the person still has to check it", + ) + walk.give( + "Check it: how it confirms the save", + "free_text", + "What is that ID called in your system's records? (note)", + lambda: page.get_by_label("What is that ID called in your system's records?").fill("note"), + name="effect-match-field", + ) + walk.give( + "Check it: how it confirms the save", + "choice", + "How should OpenAdapt read the record back? (system's records)", + lambda: page.get_by_label("Read it back from your system's records").check(), + name="effect-tier", + note="Already selected; the person still has to check it", + ) + walk.give( + "Check it: how it confirms the save", + "decision", + "Save how it confirms the save", + lambda: button(page, "Save how it confirms the save").click(), + name="effect-save", + ) + page.wait_for_timeout(2500) + walk.give( + "Check it", + "navigation", + "Show part 3 (its confirm button is inside the fold)", + lambda: page.locator("summary", has_text="How it confirms the save").click(), + name="unfold-part-3", + ) + walk.give( + "Check it", + "decision", + "These look right", + lambda: button(page, "These look right").click(), + name="these-look-right", + ) + page.wait_for_timeout(2500) + walk.shot("after-confirm") + walk.give( + "Check it: practice run", + "navigation", + "Change where it runs", + lambda: page.locator("summary", has_text="Change where it runs").click(), + name="open-where", + ) + walk.give( + "Check it: practice run", + "free_text", + "Deployment settings file (for IT)", + lambda: page.get_by_label("Deployment settings file (for IT)").fill(settings), + name="practice-settings-file", + note=( + "The record check reads MockMed's store; without this file the practice " + "run has no record check to run" + ), + ) + walk.give( + "Check it: practice run", + "confirmation", + "These are test values, and I'll keep the app in view", + lambda: page.get_by_label( + "These are test values, and I'll keep the app in view while it runs." + ).check(), + name="practice-attest", + ) + walk.give( + "Check it: practice run", + "decision", + "Run the practice item", + lambda: button(page, "Run the practice item").click(), + name="practice-run", + ) + # On MockMed Flow refuses this run before it starts (see BLOCKED below); + # the approval dialog never opens. + walk.wait_text("The practice run didn't pass.") + page.locator("summary", has_text="Technical details").first.click() + walk.shot("practice-refused") + walk.give( + "Check it", + "navigation", + "Advanced: exact checks (the 5 unclassified steps)", + lambda: button(page, "Advanced: exact checks").click(), + name="exact-checks", + ) + page.wait_for_timeout(3000) + walk.shot("exact-checks") + + +#: What Flow said, and what Check it can't do about it (read from the run). +BLOCKED = { + "where": "Check it: practice run", + "flow_refusal": [ + "Certification: require_idempotency_key_for: the save's record check carries no " + "idempotency key", + "Approval fallback: qualified action path has no effect contract to assign a tier", + ], + "shown_to_the_person": ( + "The practice run didn't pass. (technical details: Qualification run did not " + "retain a valid report)" + ), + "why_check_it_cannot_fix_it": [ + "The plain record-check form has no idempotency key. MockMed sends a fixed key " + "(not a typed value), and exact checks only take a key from a typed value.", + "Five steps stay unclassified until a person classifies each in exact checks; " + "unclassified steps count as saves without a record check.", + ], +} diff --git a/packages/desktop/scripts/setup_walk/walk.py b/packages/desktop/scripts/setup_walk/walk.py new file mode 100644 index 000000000..f575a75f9 --- /dev/null +++ b/packages/desktop/scripts/setup_walk/walk.py @@ -0,0 +1,285 @@ +#!/usr/bin/env python3 +"""Walk Desktop from "I recorded something" to the first checked run. + +Counts every input a person gives on the way and screenshots each one. It +runs the real Desktop engine (``bridge.py``) over a fresh data folder, the +real Flow underneath it, and MockMed's transactional server, with the UI from +``npm run dev`` in a headless browser. Nothing is mocked: every answer the +screens show comes from the engine, and the practice item, the drills and the +first item are real Flow runs against MockMed. All data is synthetic. + +Usage (from packages/desktop, with ``npm run dev`` running):: + + OPENADAPT_DESKTOP_FLOW_BIN= \\ + PYTHONPATH=:: \\ + python scripts/setup_walk/walk.py --variant after --out docs/setup-walk/after + +``--variant`` picks the step list for the screens on this branch (``after``) +or for the screens before it (``before``, run on the earlier commit). The +output folder gets one PNG per input and ``inputs.json``. +""" + +from __future__ import annotations + +import argparse +import json +import os +import secrets +import shutil +import socket +import sys +import tempfile +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) + +from bridge import Bridge, serve # noqa: E402 +from seed import record, seed # noqa: E402 + +INIT_SCRIPT = """ +(({ token, bridge }) => { + const callbacks = new Map(); + const listeners = new Map(); + let nextId = 1; + try { localStorage.setItem('openadapt.desktop.local-session.v1', 'enabled'); } catch {} + // Tauri's IPC has no connection limit; a browser allows six per host. The + // commands a waiting run needs (approvals) and the event poll go to a + // second host name for the same bridge, so calls parked behind a long run + // can't starve them, as they can't in the app. + const side = bridge.replace('127.0.0.1', 'localhost'); + const quick = new Set(['get_item_approval', 'answer_item_approval']); + const post = (path, body) => fetch( + (path === '/events' || quick.has(body && body.cmd) ? side : bridge) + path, { + method: 'POST', + headers: { 'content-type': 'application/json', 'x-walk-token': token }, + body: JSON.stringify(body || {}), + }).then((r) => r.json()); + window.__TAURI_INTERNALS__ = { + transformCallback(callback) { const id = nextId++; callbacks.set(id, callback); return id; }, + unregisterCallback(id) { callbacks.delete(id); }, + convertFileSrc(p) { return p; }, + async invoke(cmd, args) { + if (cmd === 'plugin:event|listen') { + const list = listeners.get(args.event) || []; + list.push(args.handler); + listeners.set(args.event, list); + return args.handler; + } + if (cmd === 'plugin:event|unlisten') { + for (const [name, list] of listeners) { + listeners.set(name, list.filter((id) => id !== args.eventId)); + } + return undefined; + } + if (cmd === 'engine_invoke') { + const reply = await post('/invoke', { cmd: args.cmd, params: args.params || {} }); + if (!reply.ok) throw new Error(reply.error); + return reply.result; + } + return undefined; + }, + }; + window.__TAURI_EVENT_PLUGIN_INTERNALS__ = { unregisterListener() {} }; + setInterval(async () => { + let events = []; + try { events = await post('/events'); } catch { return; } + for (const item of events) { + const name = 'engine://' + item.event; + for (const id of listeners.get(name) || []) { + const callback = callbacks.get(id); + if (callback) callback({ event: name, id, payload: item.payload }); + } + } + }, 250); +}) +""" + + +#: Inputs that set a workflow up. ``item_*`` inputs belong to running an +#: item (its own values and its save approval), ``navigation`` to moving +#: between screens. +SETUP_KINDS = frozenset({"free_text", "choice", "confirmation", "decision"}) + + +@dataclass +class Input: + """One thing a person gives: what kind, where, and what it was.""" + + screen: str + kind: str # SETUP_KINDS | item_value | item_run | item_approval | navigation + label: str + shot: str = "" + note: str = "" + + +@dataclass +class Walk: + page: Any + out: Path + inputs: list[Input] = field(default_factory=list) + shots: int = 0 + + def shot(self, name: str, *, full: bool = True) -> str: + self.shots += 1 + file = f"{self.shots:02d}-{name}.png" + self.page.wait_for_timeout(400) + if full: + # The app scrolls inside its own panes, so a full-page shot of the + # window misses the rest. Grow the window to the tallest pane, + # shoot, and put it back. + height = self.page.evaluate( + "() => Math.max(...[document.documentElement," + " ...document.querySelectorAll('main, .content')]" + ".map((el) => el.scrollHeight + el.getBoundingClientRect().top))" + ) + size = self.page.viewport_size + self.page.set_viewport_size( + {"width": size["width"], "height": max(size["height"], int(height) + 40)} + ) + self.page.wait_for_timeout(200) + self.page.screenshot(path=str(self.out / file)) + self.page.set_viewport_size(size) + else: + self.page.screenshot(path=str(self.out / file)) + return file + + def give( + self, + screen: str, + kind: str, + label: str, + act: Callable[[], None], + *, + name: str, + note: str = "", + ) -> None: + """Screenshot the input as the person sees it, then give it.""" + + file = self.shot(name) + self.inputs.append(Input(screen, kind, label, file, note)) + act() + self.page.wait_for_timeout(600) + + def wait_text(self, text: str, timeout: float = 240_000) -> None: + self.page.get_by_text(text, exact=False).first.wait_for(timeout=timeout) + + +def free_port() -> int: + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return int(sock.getsockname()[1]) + + +def button(page: Any, name: str) -> Any: + return page.get_by_role("button", name=name, exact=True).first + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--variant", choices=["before", "after", "agent"], required=True) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument("--base", default="http://localhost:1420") + parser.add_argument("--work", type=Path, default=None) + parser.add_argument("--headed", action="store_true") + args = parser.parse_args() + + from openadapt_flow.mockmed.fault_server import serve as serve_mockmed + from playwright.sync_api import sync_playwright + + out = args.out.resolve() + if out.exists(): + shutil.rmtree(out) + out.mkdir(parents=True) + work = Path(args.work or tempfile.mkdtemp(prefix="setup-walk-")).resolve() + work.mkdir(parents=True, exist_ok=True) + + # A fixed port per run: the recording, the program and the runs must agree. + mockmed_url, store, stop_mockmed = serve_mockmed(port=free_port()) + recording = record(mockmed_url, work / "captures" / "triage") + token = secrets.token_hex(16) + bridge = Bridge(work, token) + server = serve(bridge, 0) + bridge_url = f"http://127.0.0.1:{server.server_address[1]}" + workflow_id = seed(bridge, recording, mockmed_url) + # The engine's local socket, as the app runs it: the agent variant's MCP + # calls reach the same dispatcher, under the same lock, through it. + from openadapt_desktop.socket_server import DesktopSocketServer + + discovery = work / "desktop_ipc.json" + socket_server = DesktopSocketServer( + bridge.dispatcher.config, + discovery_path=discovery, + dispatcher=bridge.dispatcher, + dispatch_lock=bridge._dispatch_lock, + ) + socket_server.start() + os.environ["OPENADAPT_DESKTOP_IPC"] = str(discovery) + agent_log: list[dict] = [] + + steps = __import__(f"steps_{args.variant}") + started = time.monotonic() + with sync_playwright() as p: + browser = p.chromium.launch(headless=not args.headed) + context = browser.new_context( + viewport={"width": 1280, "height": 900}, device_scale_factor=1, color_scheme="light" + ) + context.add_init_script( + f"({INIT_SCRIPT})({json.dumps({'token': token, 'bridge': bridge_url})})" + ) + page = context.new_page() + page.on("pageerror", lambda error: print(f" ! page error: {error}", flush=True)) + page.goto(args.base, wait_until="domcontentloaded") + page.wait_for_timeout(1500) + walk = Walk(page, out) + try: + steps.run( + walk, + workflow_id=workflow_id, + mockmed_url=mockmed_url, + work=work, + reset_store=store.reset, + agent_log=agent_log, + ) + finally: + walk.shot("end") + browser.close() + stop_mockmed() + socket_server.stop() + server.shutdown() + + counts: dict[str, int] = {} + for item in walk.inputs: + counts[item.kind] = counts.get(item.kind, 0) + 1 + summary = { + "variant": args.variant, + "app": "MockMed (openadapt_flow.mockmed.fault_server, ?fault=ok)", + "workflow": "Add a triage note", + "seconds": round(time.monotonic() - started), + "counts": counts, + "total": len(walk.inputs), + # Setup: everything a person gives from "recorded" to "approved". + "setup_inputs": sum(v for k, v in counts.items() if k in SETUP_KINDS), + "first_item_inputs": sum(v for k, v in counts.items() if k.startswith("item_")), + "navigation": counts.get("navigation", 0), + "reached_checked_run": getattr(steps, "BLOCKED", None) is None, + "blocked": getattr(steps, "BLOCKED", None), + "inputs": [item.__dict__ for item in walk.inputs], + "engine_commands": [call["cmd"] for call in bridge.calls], + "agent_calls": [ + {"tool": call["tool"], "arguments": call["arguments"]} for call in agent_log + ], + } + (out / "inputs.json").write_text(json.dumps(summary, indent=2) + "\n") + print( + json.dumps( + {k: summary[k] for k in ("counts", "setup_inputs", "first_item_inputs", "navigation")} + ) + ) + + +if __name__ == "__main__": + main() diff --git a/packages/desktop/src-tauri/Cargo.lock b/packages/desktop/src-tauri/Cargo.lock index b7e27df90..9a497233a 100644 --- a/packages/desktop/src-tauri/Cargo.lock +++ b/packages/desktop/src-tauri/Cargo.lock @@ -2502,7 +2502,7 @@ dependencies = [ [[package]] name = "openadapt-desktop" -version = "0.21.1" +version = "0.22.0" dependencies = [ "hex", "reqwest", diff --git a/packages/desktop/src-tauri/Cargo.toml b/packages/desktop/src-tauri/Cargo.toml index fe9612c3c..8a7aaca69 100644 --- a/packages/desktop/src-tauri/Cargo.toml +++ b/packages/desktop/src-tauri/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "openadapt-desktop" -version = "0.21.1" +version = "0.22.0" description = "Installed companion for OpenAdapt authoring, teaching, and local pairing" authors = ["OpenAdapt AI"] license = "MIT" diff --git a/packages/desktop/src-tauri/tauri.conf.json b/packages/desktop/src-tauri/tauri.conf.json index fad2fc57b..ba5dbee49 100644 --- a/packages/desktop/src-tauri/tauri.conf.json +++ b/packages/desktop/src-tauri/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://raw.githubusercontent.com/nicoverbruggen/tauri-v2-schema/main/schema.json", "productName": "OpenAdapt Desktop", - "version": "0.21.1", + "version": "0.22.0", "identifier": "ai.openadapt.desktop", "build": { "frontendDist": "../dist", diff --git a/packages/desktop/src/lib/engine.ts b/packages/desktop/src/lib/engine.ts index b259d58ba..b2e642e7e 100644 --- a/packages/desktop/src/lib/engine.ts +++ b/packages/desktop/src/lib/engine.ts @@ -55,6 +55,9 @@ export const CMD = { TURN_OFF_APPROVAL_MODE: "turn_off_approval_mode", RUN_SAMPLE: "run_sample", GET_QUALIFICATION: "get_qualification", + // Check it pre-filled from the recording; answered by a person or their agent. + GET_CHECK_SETUP: "get_check_setup", + ANSWER_CHECK_SETUP: "answer_check_setup", INITIALIZE_QUALIFICATION: "initialize_qualification", PROPOSE_QUALIFICATION_FROM_DEMO: "propose_qualification_from_demo", SET_QUALIFICATION_RISK: "set_qualification_risk", @@ -169,6 +172,9 @@ export const EVT = { // {workflow_id, state: "watching" | "closed"}: a run that may stop before a // save for a person's approval started or ended. Never carries a value. ITEM_APPROVAL: "item_approval", + // {workflow_id, question, answered_by}: a Check it question was answered, + // in this window or by the person's agent over the local socket. + CHECK_SETUP: "check_setup", // Shadow mode status (see src/lib/shadowMode.ts). Never carries an event. SHADOW_STATE: "shadow_state", // {motif_id, count, title, body, actions}: generic text only. diff --git a/packages/desktop/src/lib/types.ts b/packages/desktop/src/lib/types.ts index deead620c..5c76d99e6 100644 --- a/packages/desktop/src/lib/types.ts +++ b/packages/desktop/src/lib/types.ts @@ -388,9 +388,55 @@ export interface TaskInputs { workflow_id: string; inputs?: QualificationParameter[]; target?: ExecutionTarget | null; + /** Values Check it said stay the same; Run it fills them in. */ + fixed_values?: Record; error?: string; } +/** One answer to a Check it question, and who gave it. */ +export interface CheckSetupAnswer { + by: "person" | "agent"; + at: string; + answer?: string; + changes?: string[]; + read?: string; + quiet_steps?: number; +} + +/** The record check Check it learned from the reads in the recording. */ +export interface LearnedRecordCheck { + found: boolean; + reason?: string; + read?: string; + checked?: string[]; + values_checked?: string[]; + fixed_fields?: Record; + idempotency_field?: string | null; + binding?: Record; + effects?: Record; + fault_test?: { + bad_saves: number; + caught: number; + caught_kinds: string[]; + missed_kinds: string[]; + }; +} + +/** Everything Check it can fill in from the recording (get_check_setup). */ +export interface CheckSetup { + ok: boolean; + workflow_id: string; + error?: string; + record_check: LearnedRecordCheck; + saving_steps: string[]; + quiet_steps: { write_requests: number; evidence: string; step_ids: string[] } | null; + unproven_steps: string[]; + values: { name: string; example: unknown; changes: boolean; why: string }[]; + recordings: number; + answers: { record_check?: CheckSetupAnswer; values?: CheckSetupAnswer }; + fixed_values: Record; +} + export interface RunItemResponse { ok: boolean; item?: WorkItem; diff --git a/packages/desktop/src/screens/CheckIt.test.tsx b/packages/desktop/src/screens/CheckIt.test.tsx index 9bdbb716a..5d948eeb2 100644 --- a/packages/desktop/src/screens/CheckIt.test.tsx +++ b/packages/desktop/src/screens/CheckIt.test.tsx @@ -194,7 +194,7 @@ it("never calls a workflow read-only while a step's effect is unknown", async () expect(screen.getAllByText("Needs an answer").length).toBeGreaterThanOrEqual(3); expect(screen.queryByText("Not needed")).toBeNull(); expect(screen.queryByRole("button", { name: "These look right" })).toBeNull(); - const run = screen.getByRole("button", { name: "Run the practice item" }); + const run = screen.getByRole("button", { name: "Yes, practice once with these test values" }); expect((run as HTMLButtonElement).disabled).toBe(true); }); @@ -262,6 +262,7 @@ it("runs the practice item on test values from the recording", async () => { engine(project()); const after = project({ cases: [{ id: "practice-1", kind: "representative", status: "passed" }] }); vi.mocked(engineInvoke).mockImplementation(async (command, params) => { + if (command === CMD.SEAL_QUALIFICATION_WORKFLOW) return { ok: true, workflow_id: "wf-1-v2-a1" }; if (command === CMD.GET_QUALIFICATION) return project(); if (command === CMD.ADD_QUALIFICATION_CASE) { expect(params).toEqual( @@ -283,13 +284,7 @@ it("runs the practice item on test values from the recording", async () => { }); render( {}} onRun={() => {}} />); - const run = await screen.findByRole("button", { name: "Run the practice item" }); - expect((run as HTMLButtonElement).disabled).toBe(true); - fireEvent.click( - screen.getByRole("checkbox", { - name: "These are test values, and I'll keep the app in view while it runs.", - }), - ); + const run = await screen.findByRole("button", { name: "Yes, practice once with these test values" }); fireEvent.click(run); expect(await screen.findByText(/Done and checked. It saved the entry/)).toBeTruthy(); @@ -400,7 +395,7 @@ it("runs one drill at a time in the signed-in browser", async () => { render( {}} onRun={() => {}} />); expect(await screen.findByText(/in your signed-in browser/)).toBeTruthy(); - expect(screen.getByText(/Open the screen where the workflow starts before you run it/)).toBeTruthy(); + expect(screen.getByText(/Open the\s+screen where the workflow starts first/)).toBeTruthy(); fireEvent.click(screen.getByRole("button", { name: "Run the next drill" })); expect(await screen.findByText(/then run the next drill/)).toBeTruthy(); @@ -411,6 +406,7 @@ it("counts nothing when the practice save wasn't approved, and passes IT's setti engine(project()); const calls: Record[] = []; vi.mocked(engineInvoke).mockImplementation(async (command, params) => { + if (command === CMD.SEAL_QUALIFICATION_WORKFLOW) return { ok: true, workflow_id: "wf-1-v2-a1" }; if (command === CMD.GET_QUALIFICATION) return project(); if (command === CMD.ADD_QUALIFICATION_CASE) { return project({ cases: [{ id: "practice-1", kind: "representative", status: null }] }); @@ -430,17 +426,12 @@ it("counts nothing when the practice save wasn't approved, and passes IT's setti }); render( {}} onRun={() => {}} />); - const run = await screen.findByRole("button", { name: "Run the practice item" }); - expect(screen.getByText(/Before it saves, it shows you the filled form and waits for you to approve/)).toBeTruthy(); + const run = await screen.findByRole("button", { name: "Yes, practice once with these test values" }); + expect(screen.getByText(/shows you the filled form and waits for\s+your OK first/)).toBeTruthy(); fireEvent.change(screen.getByLabelText("Deployment settings file (for IT)"), { target: { value: "/tmp/deployment.claims.yaml" }, }); expect(screen.getByText(/with your deployment settings file/)).toBeTruthy(); - fireEvent.click( - screen.getByRole("checkbox", { - name: "These are test values, and I'll keep the app in view while it runs.", - }), - ); fireEvent.click(run); expect(await screen.findByText(/Nobody approved the save in time/)).toBeTruthy(); @@ -462,7 +453,7 @@ it("describes each drill as the change Flow makes before the save", async () => engine(ready); render( {}} onRun={() => {}} />); - expect(await screen.findByText(/no drill asks you to approve/)).toBeTruthy(); + expect(await screen.findByText(/a drill never saves/)).toBeTruthy(); for (const label of [ "Wrong record", "Two possible matches", @@ -483,7 +474,7 @@ it("before the check starts, says drills come with the practice item and the rec engine(fresh); render( {}} onRun={() => {}} />); - expect(await screen.findByText(/sets up five drills from your practice item/)).toBeTruthy(); + expect(await screen.findByText(/runs 5 safety drills\s+on its own/)).toBeTruthy(); expect(screen.queryByText("Not needed")).toBeNull(); expect(screen.getByText(/will read the saved record back to check the\s+entries listed in part 1/)).toBeTruthy(); expect(screen.queryByText(/didn't see where this save lands/)).toBeNull(); @@ -493,6 +484,7 @@ it("says where and why a practice run that ran didn't pass", async () => { engine(project()); const failed = project({ cases: [{ id: "practice-1", kind: "representative", status: "failed" as never }] }); vi.mocked(engineInvoke).mockImplementation(async (command) => { + if (command === CMD.SEAL_QUALIFICATION_WORKFLOW) return { ok: true, workflow_id: "wf-1-v2-a1" }; if (command === CMD.GET_QUALIFICATION) return project(); if (command === CMD.ADD_QUALIFICATION_CASE) { return project({ cases: [{ id: "practice-1", kind: "representative", status: null }] }); @@ -512,12 +504,7 @@ it("says where and why a practice run that ran didn't pass", async () => { }); render( {}} onRun={() => {}} />); - const run = await screen.findByRole("button", { name: "Run the practice item" }); - fireEvent.click( - screen.getByRole("checkbox", { - name: "These are test values, and I'll keep the app in view while it runs.", - }), - ); + const run = await screen.findByRole("button", { name: "Yes, practice once with these test values" }); fireEvent.click(run); expect(await screen.findByText("The practice run didn't pass.")).toBeTruthy(); @@ -525,3 +512,154 @@ it("says where and why a practice run that ran didn't pass", async () => { screen.getByText(/observe exact qualification environment: qualification environment observer does not match the project/), ).toBeTruthy(); }); + +function learnedSetup(answers: Record = {}) { + return { + ok: true, + workflow_id: "wf-1", + record_check: { + found: true, + read: "GET /api/db", + checked: ["note", "patient_id"], + values_checked: ["note"], + fixed_fields: { patient_id: "p1" }, + idempotency_field: "key", + fault_test: { + bad_saves: 8, + caught: 5, + caught_kinds: ["nothing saved"], + missed_kinds: ["saved on another record"], + }, + }, + saving_steps: ["save"], + quiet_steps: { write_requests: 1, evidence: "In your recording the app sent data to its server only when you saved.", step_ids: ["search"] }, + unproven_steps: ["search"], + values: [ + { name: "member_id", example: "M-1", changes: true, why: "You typed it while recording." }, + { name: "note", example: "Reviewed", changes: true, why: "You typed it while recording." }, + ], + recordings: 1, + answers, + fixed_values: {}, + }; +} + +function guidedEngine(initial: ReturnType, setup: ReturnType) { + engine(initial); + const base = vi.mocked(engineTry).getMockImplementation()!; + vi.mocked(engineTry).mockImplementation(async (command, params, fallback) => { + if (command === CMD.GET_CHECK_SETUP) return setup; + return base(command, params, fallback); + }); +} + +it("asks three things when OpenAdapt learned the record check, and shows the rest once", async () => { + guidedEngine(project({ confirmed: false, armed: false, tier: null }), learnedSetup()); + render( {}} onRun={() => {}} />); + + expect(await screen.findByText("Is this the right record check?")).toBeTruthy(); + expect(screen.getByText("Which values change per item?")).toBeTruthy(); + expect(screen.getByText("Approve?")).toBeTruthy(); + expect(screen.getByText("GET /api/db")).toBeTruthy(); + expect(screen.getByText(/the note you give it and patient ID p1/)).toBeTruthy(); + // What it can't catch is said, not hidden. + expect(screen.getByText(/caught 5 of 8 kinds of bad save\. Not caught: saved on another record\./)).toBeTruthy(); + expect(screen.getByText(/only when you saved/)).toBeTruthy(); + // The long parts aren't asked. + expect(screen.queryByText("What it changes")).toBeNull(); + expect(screen.queryByText("How it confirms the save")).toBeNull(); +}); + +it("a yes sends one answer, from the person, and Change opens the parts", async () => { + guidedEngine(project({ confirmed: false, armed: false, tier: null }), learnedSetup()); + const sent: Record[] = []; + vi.mocked(engineInvoke).mockImplementation(async (command, params) => { + if (command === CMD.GET_QUALIFICATION) { + return sent.length ? project() : project({ confirmed: false, armed: false, tier: null }); + } + if (command === CMD.ANSWER_CHECK_SETUP) { + sent.push(params as Record); + return learnedSetup({ record_check: { by: "person", at: "2026-10-19T00:00:00Z", answer: "yes" } }); + } + throw new Error(`unexpected ${command}`); + }); + render( {}} onRun={() => {}} />); + + fireEvent.click(await screen.findByRole("button", { name: "Yes, use this check" })); + expect(await screen.findByText("You answered yes.")).toBeTruthy(); + expect(sent).toEqual([ + { workflow_id: "wf-1", question: "record_check", answer: "yes", answered_by: "person" }, + ]); +}); + +it("Change shows the plain parts instead of the learned check", async () => { + guidedEngine(project({ confirmed: false, armed: false, tier: null }), learnedSetup()); + render( {}} onRun={() => {}} />); + + fireEvent.click(await screen.findByRole("button", { name: "Change" })); + expect(await screen.findByText("What it changes")).toBeTruthy(); + expect(screen.getByText("How it confirms the save")).toBeTruthy(); + expect(screen.queryByText("Is this the right record check?")).toBeNull(); +}); + +it("answers which values change, then runs the practice item", async () => { + guidedEngine(project(), learnedSetup({ record_check: { by: "person", at: "x", answer: "yes" } })); + const order: string[] = []; + const ids: string[] = []; + let changes: unknown; + vi.mocked(engineInvoke).mockImplementation(async (command, params) => { + order.push(command); + ids.push((params as { workflow_id?: string } | undefined)?.workflow_id ?? ""); + if (command === CMD.SEAL_QUALIFICATION_WORKFLOW) return { ok: true, workflow_id: "wf-1-v2-a1" }; + if (command === CMD.GET_QUALIFICATION) return project(); + if (command === CMD.ANSWER_CHECK_SETUP) { + changes = (params as { changes: unknown }).changes; + return learnedSetup(); + } + if (command === CMD.ADD_QUALIFICATION_CASE) { + return project({ cases: [{ id: "practice-1", kind: "representative", status: null }] }); + } + if (command === CMD.RUN_QUALIFICATION_CASE) { + return { + ...project({ cases: [{ id: "practice-1", kind: "representative", status: "passed" }] }), + case_run: { outcome: "VERIFIED", transaction_outcome: "VERIFIED" }, + }; + } + throw new Error(`unexpected ${command}`); + }); + render( {}} onRun={() => {}} />); + + // The member ID is the same on every item. + fireEvent.click(await screen.findByRole("checkbox", { name: /Member ID/ })); + fireEvent.click(screen.getByRole("button", { name: "Yes, practice once with these test values" })); + + expect(await screen.findByText(/Done and checked/)).toBeTruthy(); + expect(changes).toEqual(["note"]); + const steps = order + .map((command, index) => [command, ids[index]]) + .filter(([command]) => command !== CMD.GET_QUALIFICATION); + // The version is sealed before the practice item, and the practice item + // runs on the sealed version: that is the version approved and run. + expect(steps.slice(0, 4)).toEqual([ + [CMD.ANSWER_CHECK_SETUP, "wf-1"], + [CMD.SEAL_QUALIFICATION_WORKFLOW, "wf-1"], + [CMD.ADD_QUALIFICATION_CASE, "wf-1-v2-a1"], + [CMD.RUN_QUALIFICATION_CASE, "wf-1-v2-a1"], + ]); +}); + +it("says when the person's agent answered, and leaves approval to the person", async () => { + guidedEngine( + project(), + learnedSetup({ + record_check: { by: "agent", at: "x", answer: "yes" }, + values: { by: "agent", at: "x", changes: ["note"] }, + }), + ); + render( {}} onRun={() => {}} />); + + expect(await screen.findByText("Your agent answered for you")).toBeTruthy(); + expect(screen.getByText("Your agent answered yes.")).toBeTruthy(); + const approve = screen.getByRole("button", { name: "Approve this workflow" }); + expect((approve as HTMLButtonElement).disabled).toBe(true); +}); diff --git a/packages/desktop/src/screens/CheckIt.tsx b/packages/desktop/src/screens/CheckIt.tsx index 86dfaed55..7506eb8da 100644 --- a/packages/desktop/src/screens/CheckIt.tsx +++ b/packages/desktop/src/screens/CheckIt.tsx @@ -1,15 +1,23 @@ // Check it: the short readiness test that replaces the nine-step page as the // default path. // -// 1. What it changes. 2. How it makes sure it's the right record. 3. How it -// confirms the save. 4. A practice run on a test item. 5. Safety drills. -// 6. Approve. Each answer is pre-filled from the recording by Flow's -// qualification proposal; the person confirms. Anything the engine can't -// pre-fill is asked in plain words. The full contract stays reachable under -// "Advanced: exact checks", and every command here is the same engine command -// that page uses, so neither path weakens the other. -import { useEffect, useId, useMemo, useState } from "react"; -import { CMD, engineInvoke, engineTry } from "../lib/engine"; +// When the recording shows the read the app made after the save, Check it +// asks three things: 1. Is this the right record check? 2. Which values change +// per item? (then one practice item, whose save a person approves, and the +// safety drills on their own) 3. Approve? Everything else the recording +// shows is filled in and shown once, folded: which step saves, that the other +// steps sent nothing, how it recognizes the right record, where it runs. The +// engine (get_check_setup / answer_check_setup) proposes the answers and a +// "yes" sends the same qualification commands the long form sends, so neither +// path weakens the other. The person's agent can answer the first two over the +// local socket; only a person approves. +// +// Without a learned record check, or after "Change", the parts ask each +// answer in plain words: what it changes, how it makes sure it's the right +// record, how it confirms the save. The full contract stays reachable under +// "Advanced: exact checks". +import { useEffect, useId, useMemo, useRef, useState } from "react"; +import { CMD, EVT, engineInvoke, engineTry, onEngineEvent } from "../lib/engine"; import { TIER_CHOICES, checkPlan, @@ -22,7 +30,9 @@ import { } from "../lib/checkPlan"; import { humanizeInput, plainResultForReport, resultCopy, sentenceCase } from "../lib/outcomes"; import type { + CheckSetup, ExecutionTarget, + LearnedRecordCheck, QualificationParameter, QualificationProject, QualificationResponse, @@ -100,12 +110,15 @@ function CheckSection({ title, state, doneWord, + fold = true, children, }: { index: number; title: string; state: StepState; doneWord?: string; + /** A finished part folds to its title unless it still holds the next action. */ + fold?: boolean; children: React.ReactNode; }) { // Green marks a check that ran and passed (the practice run, the drills). @@ -120,7 +133,7 @@ function CheckSection({
{/* A finished part folds to its title so the open question stands out; the last part keeps its Run button in view. */} - {state === "done" && index < 6 ? ( + {state === "done" && fold ? (

{title}

@@ -327,6 +340,90 @@ function armedIdentityGroups ({ text, count })); } +/** "the note you give it", "patient ID p1", joined in plain words. */ +function checkedThings(learned: LearnedRecordCheck): string[] { + const things = (learned.values_checked ?? []).map( + (name) => `the ${humanizeInput(name)} you give it`, + ); + for (const [field, value] of Object.entries(learned.fixed_fields ?? {})) { + things.push(`${humanizeInput(field)} ${value}`); + } + return things; +} + +function joinPlain(items: string[]): string { + if (items.length <= 1) return items.join(""); + return `${items.slice(0, -1).join(", ")} and ${items[items.length - 1]}`; +} + +/** + * The record check OpenAdapt learned from the reads in the recording, and + * everything else one "yes" confirms, shown once. The parts behind "Change" + * are the same answers in the long form. + */ +function LearnedCheck({ + learned, + plan, + quiet, + where, +}: { + learned: LearnedRecordCheck; + plan: CheckPlan; + quiet: CheckSetup["quiet_steps"]; + where: string; +}) { + const test = learned.fault_test; + const others = plan.unproven.length; + return ( +
+

+ After each save, OpenAdapt reads {learned.read} from + the app and looks for exactly one new record with{" "} + {joinPlain(checkedThings(learned))}. It stops if the record isn’t + there, is there twice, or a record that was there before is gone. + {learned.idempotency_field + ? " It also checks the key the app sends with each save, so a double save stops." + : ""} +

+ {test && ( +

+ Learned from the reads in your recording. On the recorded records it + caught {test.caught} of {test.bad_saves} kinds of bad save + {test.missed_kinds.length > 0 + ? `. Not caught: ${test.missed_kinds.join("; ")}.` + : "."} +

+ )} +
+ What else this answer confirms +
    +
  • + It saves only at{" "} + {joinPlain(plan.writes.map((step) => step.caption))}. +
    • + {others > 0 && + (quiet ? ( +
  • + The other {others} step{others === 1 ? "" : "s"} only open + screens or fill in the form. {quiet.evidence} +
    • + ) : ( +
  • + OpenAdapt can’t tell what {others} other step + {others === 1 ? " does" : "s do"}. Say what each does in + Advanced: exact checks before you approve. +
    • + ))} + {plan.writes.map((step) => ( +
  • {step.identityText}
    • + ))} +
  • Runs in: {where}.
    • +
+
+
+ ); +} + export function CheckIt({ workflowId, onExactChecks, @@ -340,6 +437,13 @@ export function CheckIt({ onReplaced?: (workflowId: string) => void; }) { const idPrefix = useId(); + // The version Check it works on. It seals the workflow before the practice + // item, so the cases, the approval and the runs all bind the one sealed + // version that runs. Engine calls read the ref; the sealed id replaces it. + const activeId = useRef(workflowId); + if (activeId.current !== workflowId && !activeId.current.startsWith(`${workflowId}-v`)) { + activeId.current = workflowId; + } const [name, setName] = useState(""); const [project, setProject] = useState(null); const [proposal, setProposal] = useState(null); @@ -351,7 +455,12 @@ export function CheckIt({ const [values, setValues] = useState>({}); const [target, setTarget] = useState(null); const [deploymentConfig, setDeploymentConfig] = useState(""); - const [attested, setAttested] = useState(false); + // What OpenAdapt filled in from the recording (get_check_setup). + const [setup, setSetup] = useState(null); + // The person chose "Change" on the learned record check: show the parts. + const [manual, setManual] = useState(false); + // Which values change per item, as ticked on screen (null: as proposed). + const [changing, setChanging] = useState | null>(null); const [practiceNote, setPracticeNote] = useState(""); const [drillNote, setDrillNote] = useState(""); const [approvedId, setApprovedId] = useState(null); @@ -372,17 +481,19 @@ export function CheckIt({ setBusy("loading"); setLoadError(""); try { - const [list, details, response] = await Promise.all([ + const [list, details, response, filled] = await Promise.all([ engineTry(CMD.GET_WORKFLOWS, {}, []), - engineTry(CMD.GET_TASK_INPUTS, { workflow_id: workflowId }, { + engineTry(CMD.GET_TASK_INPUTS, { workflow_id: activeId.current }, { ok: false, - workflow_id: workflowId, + workflow_id: activeId.current, }), engineInvoke(CMD.GET_QUALIFICATION, { - workflow_id: workflowId, + workflow_id: activeId.current, policy: POLICY, }), + engineTry(CMD.GET_CHECK_SETUP, { workflow_id: activeId.current }, null), ]); + setSetup(filled && filled.ok ? filled : null); const workflow = (Array.isArray(list) ? list : []).find((item) => item.id === workflowId); setName(workflow?.name ?? ""); const declared = (details.inputs ?? []).filter((item) => !item.secret); @@ -405,7 +516,7 @@ export function CheckIt({ if (!response.project || response.draft_environment) { const draft = await engineTry<{ ok: boolean; proposal?: Proposal; error?: string }>( CMD.PROPOSE_QUALIFICATION_FROM_DEMO, - { workflow_id: workflowId, policy_pack: "community", accept: false }, + { workflow_id: activeId.current, policy_pack: "community", accept: false }, { ok: false }, ); setProposal(draft.proposal ?? null); @@ -423,9 +534,26 @@ export function CheckIt({ // eslint-disable-next-line react-hooks/exhaustive-deps }, [workflowId]); + // The person's agent can answer the first two questions over the local + // socket. Show its answers as soon as it gives them. + useEffect(() => { + const unlisten = onEngineEvent<{ workflow_id?: string; answered_by?: string; sealed_id?: string }>( + EVT.CHECK_SETUP, + (payload) => { + if (payload?.workflow_id !== workflowId && payload?.workflow_id !== activeId.current) return; + if (payload.sealed_id) activeId.current = payload.sealed_id; + if (payload.answered_by === "agent") void load(); + }, + ).catch(() => null); + return () => { + void unlisten.then((fn) => fn?.()); + }; + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [workflowId]); + async function call(cmd: string, params: Record): Promise> { return engineInvoke>(cmd, { - workflow_id: workflowId, + workflow_id: activeId.current, policy: POLICY, ...params, }); @@ -590,14 +718,15 @@ export function CheckIt({ return path ? { deployment_config: path } : {}; } - async function runPractice() { - if (!plan || !target) return; + /** Runs the practice item; returns the project when it passed. */ + async function runPractice(from: CheckPlan | null = plan): Promise { + if (!from || !target) return null; setBusy("practice"); fail(""); setPracticeNote(""); try { const parameters = caseParameters(); - let caseId = plan.practice?.id ?? null; + let caseId = from.practice?.id ?? null; if (!caseId) { const added = await call(CMD.ADD_QUALIFICATION_CASE, { case_id: PRACTICE_CASE, @@ -607,7 +736,7 @@ export function CheckIt({ }); if (!added.ok) { fail("OpenAdapt couldn't set up the practice run.", String(added.error || "")); - return; + return null; } setProject(added as QualificationProject); caseId = PRACTICE_CASE; @@ -621,36 +750,42 @@ export function CheckIt({ if (response.not_approved) { // Nothing was saved and nothing was recorded: say why in plain words. setPracticeNote(String(response.error || "The save wasn't approved, so nothing was saved.")); - return; + return null; } const copy = caseResult(response); if (copy) setPracticeNote(`${copy.label}. ${copy.sentence}`); if (!response.ok) { fail("The practice run didn't pass.", String(response.error || "")); - return; + return null; } setProject(response as QualificationProject); const ran = checkPlan(response as QualificationProject).practice; if (ran && ran.status !== "passed") { // Say where and why it stopped: the result word alone doesn't. fail("The practice run didn't pass.", stopReason(response.case_run as RunReport | undefined, response.stop_reason)); + return null; } + return response as QualificationProject; } catch (error) { fail("The practice run didn't finish.", String(error)); + return null; } finally { setBusy(null); } } - async function runDrills() { - if (!plan || !target) return; + /** Runs the drills still to pass, from the given (latest) project. */ + async function runDrills(from: QualificationProject | null = project): Promise { + const current = from ? checkPlan(from) : plan; + if (!current || !target) return false; + let passed = false; setBusy("drills"); fail(""); setDrillNote(""); let latest: QualificationProject | null = null; try { const parameters = caseParameters(); - const pending = plan.drills.filter((item) => item.status !== "passed"); + const pending = current.drills.filter((item) => item.status !== "passed"); // In the signed-in browser every run starts where the last one ended, // so run one drill and let the person reopen the first screen. const batch = attachedBrowser(target) ? pending.slice(0, 1) : pending; @@ -677,6 +812,7 @@ export function CheckIt({ } if (latest) { setProject(latest); + passed = checkPlan(latest).drillsState === "done"; setDrillNote( batch.length < pending.length ? "That drill ended the way its description says. Open the screen where the workflow starts, then run the next drill." @@ -688,6 +824,101 @@ export function CheckIt({ } finally { setBusy(null); } + return passed; + } + + /** "Is this the right record check?" Yes. The engine sends the same commands the parts below send. */ + async function acceptRecordCheck() { + setBusy("confirm"); + fail(""); + try { + const answered = await engineInvoke(CMD.ANSWER_CHECK_SETUP, { + workflow_id: activeId.current, + question: "record_check", + answer: "yes", + answered_by: "person", + }); + if (!answered.ok) { + fail("OpenAdapt couldn't save that answer.", String(answered.error || "")); + return; + } + setSetup(answered); + const response = await call(CMD.GET_QUALIFICATION, {}); + if (response.ok) setProject(response as QualificationProject); + } catch (error) { + fail("OpenAdapt couldn't save that answer.", String(error)); + } finally { + setBusy(null); + } + } + + /** + * "Which values change per item?" The answer, then one practice item (a + * person approves its save in the dialog), then the drills on their own. + */ + async function practiceWithValues() { + setBusy("practice"); + fail(""); + try { + if (!setup) throw new Error("no setup"); + const changes = valueRows + .filter((item) => isChanging(item.name, item.changes)) + .map((item) => item.name); + const answered = await engineInvoke(CMD.ANSWER_CHECK_SETUP, { + workflow_id: activeId.current, + question: "values", + changes, + answered_by: "person", + }); + if (!answered.ok) { + fail("OpenAdapt couldn't save which values change.", String(answered.error || "")); + return; + } + setSetup(answered); + } catch (error) { + // An engine without Check it's setup still runs the practice item; + // every value then changes per item, as before. + if (setup) { + fail("OpenAdapt couldn't save which values change.", String(error)); + return; + } + } finally { + setBusy(null); + } + // Seal first: Flow binds each case result to the exact executable + // workflow, so the practice item and drills run on the version that + // will be approved and run. + let current = project; + if (current && !current.graph.bundle.encrypted && plan?.practiceState !== "done") { + setBusy("practice"); + try { + const sealed = await engineInvoke<{ ok: boolean; workflow_id?: string; error?: string }>( + CMD.SEAL_QUALIFICATION_WORKFLOW, + { workflow_id: activeId.current }, + ); + if (!sealed.ok || !sealed.workflow_id) { + fail("OpenAdapt couldn't lock this version for the practice item.", String(sealed.error || "")); + return; + } + activeId.current = sealed.workflow_id; + onReplaced?.(sealed.workflow_id); + const reloaded = await call(CMD.GET_QUALIFICATION, {}); + if (!reloaded.ok) { + fail("OpenAdapt couldn't read the locked version.", String(reloaded.error || "")); + return; + } + current = reloaded as QualificationProject; + setProject(current); + } catch (error) { + fail("OpenAdapt couldn't lock this version for the practice item.", String(error)); + return; + } finally { + setBusy(null); + } + } + const practiced = + plan?.practiceState === "done" ? project : await runPractice(current ? checkPlan(current) : null); + if (practiced && !attachedBrowser(target)) await runDrills(practiced); } async function approve() { @@ -712,7 +943,7 @@ export function CheckIt({ if (current && !current.graph.bundle.encrypted) { const sealed = await engineInvoke<{ ok: boolean; workflow_id?: string; error?: string }>( CMD.SEAL_QUALIFICATION_WORKFLOW, - { workflow_id: workflowId }, + { workflow_id: activeId.current }, ); if (!sealed.ok || !sealed.workflow_id) { fail("OpenAdapt approved the workflow but couldn't lock this version.", String(sealed.error || "")); @@ -722,7 +953,7 @@ export function CheckIt({ onReplaced?.(sealed.workflow_id); return; } - setApprovedId(workflowId); + setApprovedId(activeId.current); } catch (error) { fail("OpenAdapt couldn't approve this workflow.", String(error)); } finally { @@ -746,102 +977,96 @@ export function CheckIt({ ); const approved = approvedId !== null || (plan?.approved && plan.sealed); - const practiceForm = plan ? ( - <> - {inputs.length > 0 && ( -
- {inputs.map((input) => ( - - - setValues({ ...values, [input.name]: event.target.value }) - } - /> - - ))} -
- )} -

- Runs in: {targetSummary(target)} - {deploymentConfig.trim() ? ", with your deployment settings file" : ""} -

- {attachedBrowser(target) && ( -

- It starts from the screen your signed-in browser shows. Open the - screen where the workflow starts before you run it. -

- )} -
- Change where it runs -
- {target && ( - - )} - - setDeploymentConfig(event.target.value)} - /> - -
-
- - {practiceNote &&

{practiceNote}

} -
- - {!firstThreeDone && ( - Confirm the first three parts first. + // Test values and where the practice item runs: filled in from the + // recording, folded, and changeable. + const testSettings = plan ? ( +
+ Change the test values or where it runs +
+ {inputs.length > 0 && ( +
+ {inputs.map((input) => ( + + + setValues({ ...values, [input.name]: event.target.value }) + } + /> + + ))} +
)} + {target && ( + + )} + + setDeploymentConfig(event.target.value)} + /> +
- +
) : null; + const learned = setup?.record_check.found ? setup.record_check : null; + // The three questions replace the parts when OpenAdapt learned the record + // check. Without one, or after "Change", the parts ask it instead. + const guided = learned !== null && !manual; + const agentAnswered = Boolean( + setup && + (setup.answers.record_check?.by === "agent" || setup.answers.values?.by === "agent"), + ); + const isChanging = (name: string, proposed: boolean) => + changing ? changing.has(name) : proposed; + // Without the engine's proposals, every typed value changes per item. + const valueRows: CheckSetup["values"] = + setup?.values ?? + inputs.map((input) => ({ + name: input.name, + example: input.example, + changes: true, + why: "You typed it while recording.", + })); + const practicePassed = plan?.practiceState === "done"; + const checksPassed = practicePassed && plan?.drillsState === "done"; + const q2State: StepState = checksPassed ? "done" : "todo"; + const answeredBy = (answer?: { by: "person" | "agent" }) => + answer ? (answer.by === "agent" ? "Your agent answered yes." : "You answered yes.") : ""; + return (
onExactChecks(workflowId)}> + } @@ -873,248 +1098,319 @@ export function CheckIt({
)} + {plan && agentAnswered && !approved && ( + + Check its answers below before you approve. Only you can approve. + + )} + {plan && (
- - {plan.writes.length === 0 && plan.unproven.length === 0 ? ( -

- This workflow only reads. None of its steps saves anything, so there's - no record to check. -

- ) : plan.writes.length === 0 ? ( -

- OpenAdapt can't tell yet whether this workflow saves anything, so it - won't treat it as a workflow that only reads. -

- ) : ( - <> -
    - {plan.writes.map((step) => ( -
  • - {step.caption} - {step.effects.map((effect) => ( - {effect} - ))} -
    • - ))} -
- {plan.readCount > 0 && ( -

- {plan.readCount} other step{plan.readCount === 1 ? " only reads or moves" : "s only read or move"}{" "} - around in the app. -

- )} - - )} - {plan.unproven.length > 0 && ( -
-

- {plan.writes.length === 0 - ? `${plan.unproven.length} step${plan.unproven.length === 1 ? "" : "s"} might change something.` - : `OpenAdapt can't tell yet whether ${plan.unproven.length} other step${plan.unproven.length === 1 ? "" : "s"} change${plan.unproven.length === 1 ? "s" : ""} anything.`}{" "} - Before you approve the workflow, say what each one does in Advanced: - exact checks. -

-
- - Show {plan.unproven.length === 1 ? "it" : `the ${plan.unproven.length} steps`} - -
    - {plan.unproven.map((step) => ( -
  • {step.caption}
    • - ))} -
-
-
- )} - - - - {plan.writes.length === 0 && plan.unproven.length > 0 && ( -

- This depends on part 1. Once OpenAdapt knows which step saves, - it asks how to recognize the right record. -

- )} - {/* Saving steps that check the record the same way say it once. */} - {armedIdentityGroups(plan.writes).map((group) => ( -
-

{group.text}

- {group.count > 1 && ( -

For each of its {group.count} saving steps.

- )} -
- ))} - {plan.writes.filter((step) => !step.identityArmed).map((step) => ( -
-

{step.identityText}

- {!step.identityArmed && step.canArm && plan.hasProject && ( - - )} - {!step.identityArmed && !step.canArm && ( -

- OpenAdapt didn't see the record's name or ID on the screen - where it saves. Show the workflow again with the record's name - or ID visible before you save, or set this up in the exact - checks. + +

+ )} + + ) : ( + <> + + {plan.writes.length === 0 && plan.unproven.length === 0 ? ( +

+ This workflow only reads. None of its steps saves anything, so there's + no record to check.

+ ) : plan.writes.length === 0 ? ( +

+ OpenAdapt can't tell yet whether this workflow saves anything, so it + won't treat it as a workflow that only reads. +

+ ) : ( + <> +
    + {plan.writes.map((step) => ( +
  • + {step.caption} + {step.effects.map((effect) => ( + {effect} + ))} +
    • + ))} +
+ {plan.readCount > 0 && ( +

+ {plan.readCount} other step{plan.readCount === 1 ? " only reads or moves" : "s only read or move"}{" "} + around in the app. +

+ )} + )} -
- ))} - - - - {plan.writes.length === 0 && plan.unproven.length > 0 && ( -

- This depends on part 1. Once OpenAdapt knows which step saves, - it asks how to confirm the save. -

- )} - {plan.writes.map((step) => ( -
-

{step.checkText}

- {!step.checked && plan.hasProject && ( - void bindEffect(step, answer)} - /> + {plan.unproven.length > 0 && ( +
+

+ {plan.writes.length === 0 + ? `${plan.unproven.length} step${plan.unproven.length === 1 ? "" : "s"} might change something.` + : `OpenAdapt can't tell yet whether ${plan.unproven.length} other step${plan.unproven.length === 1 ? "" : "s"} change${plan.unproven.length === 1 ? "s" : ""} anything.`}{" "} + Before you approve the workflow, say what each one does in Advanced: + exact checks. +

+
+ + Show {plan.unproven.length === 1 ? "it" : `the ${plan.unproven.length} steps`} + +
    + {plan.unproven.map((step) => ( +
  • {step.caption}
    • + ))} +
+
+
)} - {!step.checked && !plan.hasProject && step.hasRecordChecks && ( -

- OpenAdapt will read the saved record back to check the - entries listed in part 1. Confirm the parts above to set - up how it reads the record. + + + + {plan.writes.length === 0 && plan.unproven.length > 0 && ( +

+ This depends on part 1. Once OpenAdapt knows which step saves, + it asks how to recognize the right record.

)} - {!step.checked && !plan.hasProject && !step.hasRecordChecks && ( -

- OpenAdapt didn't see where this save lands in your system, - so it will ask you a few questions about it. Start the check - first. + {/* Saving steps that check the record the same way say it once. */} + {armedIdentityGroups(plan.writes).map((group) => ( +

+

{group.text}

+ {group.count > 1 && ( +

For each of its {group.count} saving steps.

+ )} +
+ ))} + {plan.writes.filter((step) => !step.identityArmed).map((step) => ( +
+

{step.identityText}

+ {!step.identityArmed && step.canArm && plan.hasProject && ( + + )} + {!step.identityArmed && !step.canArm && ( +

+ OpenAdapt didn't see the record's name or ID on the screen + where it saves. Show the workflow again with the record's name + or ID visible before you save, or set this up in the exact + checks. +

+ )} +
+ ))} + + + + {plan.writes.length === 0 && plan.unproven.length > 0 && ( +

+ This depends on part 1. Once OpenAdapt knows which step saves, + it asks how to confirm the save.

)} -
- ))} - {/* Nobody can vouch for "what it changes" while OpenAdapt can't - say which step saves; that answer goes in the exact checks. */} - {(!plan.hasProject || plan.changes === "todo") && - !(plan.writes.length === 0 && plan.unproven.length > 0) && ( -
- -
- )} - {missingPins.length > 0 && ( -
-

OpenAdapt needs a few answers

-
    - {missingPins.map((pin) => ( -
  • {missingPinQuestion(pin)}
    • - ))} -
- {!appMissing && !plan.hasProject && ( + {plan.writes.map((step) => ( +
+

{step.checkText}

+ {!step.checked && plan.hasProject && ( + void bindEffect(step, answer)} + /> + )} + {!step.checked && !plan.hasProject && step.hasRecordChecks && ( +

+ OpenAdapt will read the saved record back to check the + entries listed in part 1. Confirm the parts above to set + up how it reads the record. +

+ )} + {!step.checked && !plan.hasProject && !step.hasRecordChecks && ( +

+ OpenAdapt didn't see where this save lands in your system, + so it will ask you a few questions about it. Start the check + first. +

+ )} +
+ ))} + {/* Nobody can vouch for "what it changes" while OpenAdapt can't + say which step saves; that answer goes in the exact checks. */} + {(!plan.hasProject || plan.changes === "todo") && + !(plan.writes.length === 0 && plan.unproven.length > 0) && (
- - Then answer the questions above for each save. -
)} - {appMissing && ( -
- - setAppName(event.target.value)} - /> - - - setAppVersion(event.target.value)} - /> - - + {missingPins.length > 0 && ( +
+

OpenAdapt needs a few answers

+
    + {missingPins.map((pin) => ( +
  • {missingPinQuestion(pin)}
    • + ))} +
+ {!appMissing && !plan.hasProject && ( +
+ + + Then answer the questions above for each save. + +
+ )} + {appMissing && ( +
+ + setAppName(event.target.value)} + /> + + + setAppVersion(event.target.value)} + /> + + +
+ )}
)} -
- )} - + + + + )} -

- OpenAdapt does the workflow once, start to finish, on a test record you - choose, with every check on. Before it saves, it shows you the - filled form and waits for you to approve. Then it saves for real - in the app, so use a test record. Keep the app where you can see - it. -

- {plan.practiceState === "done" && practiceNote === "" ? ( -
- Run the practice item again -
{practiceForm}
-
- ) : ( - practiceForm - )} - - - - {plan.drills.length === 0 ? ( -

- {plan.drillsState === "not_needed" - ? "No drills are needed, because no step saves anything." - : "OpenAdapt sets up five drills from your practice item when you run it. Each one makes the workflow go wrong on purpose right before the save, and must stop before saving."} -

- ) : ( + {valueRows.length > 0 ? ( <>

- Each drill runs the workflow with your test values and changes one - thing on purpose right before the save. A drill passes only - when OpenAdapt's own check catches the change and stops before - saving. Nothing is saved, and no drill asks you to approve. + Every value you typed while recording is an input. Untick one + that is the same on every item: OpenAdapt then fills in the + recorded value each time.

-
    +
      + {valueRows.map((item) => ( +
    • + +
      • + ))} +
    + + ) : ( +

    This workflow takes no values: every item is the same.

    + )} +

    + Then OpenAdapt does the workflow once with these test values in{" "} + {targetSummary(target)} + {deploymentConfig.trim() ? ", with your deployment settings file" : ""}. + It saves for real, so it shows you the filled form and waits for + your OK first. Then it runs {plan.drills.length || 5} safety drills + on its own; a drill never saves. +

    + {attachedBrowser(target) && ( +

    + It starts from the screen your signed-in browser shows. Open the + screen where the workflow starts first. +

    + )} + {!checksPassed && testSettings} + {practiceNote &&

    {practiceNote}

    } + {plan.drills.length > 0 && ( +
    + + {plan.drills.every((drill) => drill.status === "not_run") + ? `The ${plan.drills.length} safety drills` + : `Safety drills: ${plan.drills.filter((drill) => drill.status === "passed").length} of ${plan.drills.length} passed`} + +
      {plan.drills.map((drill: CaseView) => (
    • ))}
    - {drillNote &&

    {drillNote}

    } - {plan.drillsState !== "done" && ( -
    +
    + )} + {drillNote &&

    {drillNote}

    } + {!checksPassed && ( +
    + {!practicePassed ? ( + ) : ( + - {plan.practiceState !== "done" && ( - Run the practice item first. - )} -
    )} - + {!firstThreeDone && ( + + {guided ? "Answer the first question first." : "Confirm the first three parts first."} + + )} +
)} {approved ? (
@@ -1165,7 +1475,7 @@ export function CheckIt({ computer. It’s ready to run on real items, and it stops to ask you only when something doesn’t match.

-
@@ -1178,6 +1488,32 @@ export function CheckIt({ OpenAdapt lets you approve only when every check above has passed.

+ {checksPassed && ( +
+ What you’re approving +
    + {learned && ( +
  • + Record check: {learned.read}, as learned from your recording.{" "} + {answeredBy(setup?.answers.record_check)} +
    • + )} + {setup && setup.values.length > 0 && ( +
  • + Changes per item:{" "} + {setup.values + .filter((item) => item.changes) + .map((item) => humanizeInput(item.name)) + .join(", ") || "nothing"} + .{" "} + {setup.answers.values?.by === "agent" ? "Your agent answered." : ""} +
    • + )} +
  • Practice item: passed.
    • +
  • Safety drills: all {plan.drills.length} passed.
    • +
+
+ )} {plan.blockers.length > 0 && (

Still open in the exact checks:

@@ -1191,11 +1527,7 @@ export function CheckIt({
+ {Object.keys(fixed).length > 0 && ( +

+ Same on every item: + {Object.entries(fixed).map(([name, value]) => ( + + {sentenceCase(humanizeInput(name))}: {value} + + ))} +

+ )} {declared.length > 0 && (

Each item needs: diff --git a/packages/desktop/src/styles/app.css b/packages/desktop/src/styles/app.css index 925144efb..1b6a5437c 100644 --- a/packages/desktop/src/styles/app.css +++ b/packages/desktop/src/styles/app.css @@ -2977,6 +2977,45 @@ select.input:focus { font-size: var(--fs-caption); } +/* Check it's three questions: the learned record check and the values. */ +.learned-check p { + max-width: 64ch; + margin: var(--space-2) 0 0; +} +.value-choices, +.learned-check + .simple-sub { + margin-top: var(--space-3); +} +.learned-check strong { + font-family: var(--font-mono); + font-weight: var(--fw-medium); +} +.value-choices { + display: flex; + flex-direction: column; + gap: var(--space-2); + margin: 0 0 var(--space-3); + padding: 0; + list-style: none; +} +.value-choices label { + display: flex; + align-items: flex-start; + gap: var(--space-2); +} +.value-choices input { + margin-top: 4px; + accent-color: var(--action); +} +.value-choices .mono { + color: var(--text-muted); +} +.value-choices small { + display: block; + color: var(--text-muted); + font-size: var(--fs-caption); +} + /* ----------------------------------------------------------------------------- Approve each save: a run waits before a save for a person (ItemApproval.tsx) ----------------------------------------------------------------------------- */ diff --git a/packages/desktop/tests/test_engine/test_agent_setup.py b/packages/desktop/tests/test_engine/test_agent_setup.py new file mode 100644 index 000000000..2fc1796f4 --- /dev/null +++ b/packages/desktop/tests/test_engine/test_agent_setup.py @@ -0,0 +1,110 @@ +"""The person's agent sets up Check it over the local socket (and MCP). + +The agent may read the setup, answer the record-check and values questions +and run the checks. It can't approve, and every answer it sends is recorded +as the agent's, whatever it claims. +""" + +from __future__ import annotations + +import threading +from pathlib import Path +from typing import Any + +import pytest + +from openadapt_desktop import agent_mcp +from openadapt_desktop.config import EngineConfig +from openadapt_desktop.socket_server import _SOCKET_COMMANDS, DesktopSocketServer + + +class _Dispatcher: + def __init__(self) -> None: + self.calls: list[tuple[str, dict]] = [] + self.emit = lambda event, data: None + + def dispatch(self, cmd: str, params: dict | None = None) -> Any: + self.calls.append((cmd, dict(params or {}))) + if cmd == "get_workflows": + return [{"id": "wf1", "name": "Add a triage note", "status": "not_checked"}] + return { + "ok": True, + "workflow_id": "wf1", + "record_check": { + "found": True, + "read": "GET /api/db", + "binding": {"x": 1}, + "effects": {}, + }, + "quiet_steps": {"evidence": "sent nothing"}, + "values": [{"name": "note", "changes": True}], + "answers": {}, + "approval": "A person approves in OpenAdapt Desktop. An agent can't.", + } + + +@pytest.fixture +def desktop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + dispatcher = _Dispatcher() + config = EngineConfig(data_dir=tmp_path / ".openadapt", log_level="WARNING") + discovery = tmp_path / "desktop_ipc.json" + server = DesktopSocketServer( + config, discovery_path=discovery, dispatcher=dispatcher, dispatch_lock=threading.RLock() + ) + server.start() + monkeypatch.setenv(agent_mcp.DISCOVERY_ENV, str(discovery)) + yield dispatcher + server.stop() + + +def test_agent_reads_and_answers_through_the_running_desktop(desktop: _Dispatcher) -> None: + assert agent_mcp.handle("list_workflows", {}) == [ + {"workflow_id": "wf1", "name": "Add a triage note", "status": "not_checked"} + ] + setup = agent_mcp.handle("get_check_setup", {"workflow_id": "wf1"}) + # The binding internals stay in the engine. + assert "binding" not in setup["record_check"] and setup["record_check"]["read"] == "GET /api/db" + assert "An agent can't" in setup["approval"] + + agent_mcp.handle("answer_record_check", {"workflow_id": "wf1"}) + agent_mcp.handle("answer_values", {"workflow_id": "wf1", "changes": ["note"]}) + answers = [params for cmd, params in desktop.calls if cmd == "answer_check_setup"] + assert [params["question"] for params in answers] == ["record_check", "values"] + assert all(params["answered_by"] == "agent" for params in answers) + + +def test_an_agent_claiming_to_be_the_person_is_recorded_as_the_agent(desktop: _Dispatcher) -> None: + agent_mcp.call_desktop( + "answer_check_setup", + { + "workflow_id": "wf1", + "question": "record_check", + "answer": "yes", + "answered_by": "person", + }, + ) + (params,) = [params for cmd, params in desktop.calls if cmd == "answer_check_setup"] + assert params["answered_by"] == "agent" + + +def test_the_socket_never_approves_or_answers_a_save() -> None: + for command in ( + "certify_qualification", + "seal_qualification_workflow", + "answer_item_approval", + "run_qualification_case", + "bind_qualification_effect", + "set_qualification_risk", + "run_item", + ): + assert command not in _SOCKET_COMMANDS + for tool in agent_mcp.TOOLS: + assert "approve" not in tool + + +def test_no_desktop_running_is_a_plain_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(agent_mcp.DISCOVERY_ENV, str(tmp_path / "missing.json")) + with pytest.raises(agent_mcp.DesktopUnavailable, match="isn't running"): + agent_mcp.handle("list_workflows", {}) diff --git a/packages/desktop/tests/test_engine/test_check_setup.py b/packages/desktop/tests/test_engine/test_check_setup.py new file mode 100644 index 000000000..12ca9ea5a --- /dev/null +++ b/packages/desktop/tests/test_engine/test_check_setup.py @@ -0,0 +1,293 @@ +"""Check it pre-filled from the recording: proposals and answers. + +Recordings here are written by hand in the ``openadapt.network-capture/1`` +shape the recorder's network capture writes. All values are made up. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("openadapt_flow.record_check_synthesis") + +from openadapt_desktop.check_setup import ( # noqa: E402 + CheckSetupError, + SetupStore, + accept_record_check, + answer_values, + carry_to_sealed, + learn_record_check, + quiet_steps, + value_proposals, +) + +NOTE = "Synthetic follow-up in two weeks" +BASE = "http://127.0.0.1:9" + + +def _line(seq: int, method: str, path: str, t: float, body: Any = None, text: str = "") -> dict: + line: dict[str, Any] = { + "method": method, + "url": BASE + path, + "resource_type": "xhr", + "frame_url": BASE + "/", + "status": 200, + "response_type": "application/json", + "response_text": text, + "seq": seq, + "t": t, + } + if body is not None: + line["request_type"] = "application/json" + line["body"] = json.dumps(body) + return line + + +def _recording( + root: Path, + *, + params: dict[str, str] | None = None, + stray_write: bool = False, + network: bool = True, +) -> Path: + """Open a record (t=1.0), type the note, save (t=2.0); the app reads after.""" + + root.mkdir(parents=True) + events = [ + { + "i": 0, + "kind": "click", + "x": 10, + "y": 10, + "t": 1.0, + "structural": {"role": "button", "name": "Open"}, + }, + {"i": 1, "kind": "type", "text": NOTE, "param": "note", "t": 1.5}, + { + "i": 2, + "kind": "click", + "x": 300, + "y": 400, + "t": 2.0, + "structural": {"role": "button", "name": "Save encounter"}, + }, + ] + (root / "events.jsonl").write_text("".join(json.dumps(e) + "\n" for e in events)) + (root / "meta.json").write_text(json.dumps({"params": params or {"note": NOTE}})) + if not network: + return root + saved = {"id": 1, "patient_id": "p1", "type": "Triage", "note": NOTE, "key": "demo-key-p1"} + lines = [ + _line(0, "GET", "/api/db", 0.5, text=json.dumps({"records": []})), + ] + if stray_write: + lines.append(_line(len(lines), "POST", "/api/audit", 1.2, body={"opened": "p1"}, text="{}")) + lines += [ + _line( + len(lines), + "POST", + "/api/encounter", + 1.9, + body={"patient_id": "p1", "type": "Triage", "note": NOTE, "key": "demo-key-p1"}, + text=json.dumps({"ok": True, "id": 1}), + ), + _line(len(lines) + 1, "GET", "/api/db", 2.4, text=json.dumps({"records": [saved]})), + ] + with (root / "network.jsonl").open("w") as fh: + fh.write( + json.dumps({"schema": "openadapt.network-capture/1", "exchanges": len(lines)}) + "\n" + ) + for line in lines: + fh.write(json.dumps(line) + "\n") + return root + + +def test_learns_the_binding_a_person_would_type_from_the_read_after_the_save( + tmp_path: Path, +) -> None: + learned = learn_record_check(_recording(tmp_path / "rec"), inputs={"note": {}}) + + assert learned["found"] is True + assert learned["read"] == "GET /api/db" + binding = learned["binding"] + # Selected through a run parameter, never a literal alone. + assert binding["match_field"] == "note" and binding["match_param"] == "note" + assert binding["kind"] == "record_written" and binding["count_new_only"] is True + # What the save sent besides the typed value narrows the match. + assert binding["extra_match"] == { + "patient_id": {"literal": "p1"}, + "type": {"literal": "Triage"}, + } + # The app's own anti-double-save key is checked as sent. + assert binding["idempotency_literal"] == "demo-key-p1" + assert binding["key_field"] == "key" + assert binding["forbid_collateral_loss"] is True + assert learned["effects"] == { + "kind": "rest", + "base_url": BASE, + "records_path": "/api/db", + "records_key": "records", + } + # The evidence is for this binding, including what it can't catch. + test = learned["fault_test"] + assert test["caught"] + len(test["missed_kinds"]) == test["bad_saves"] + assert test["caught"] >= 3 + assert "nothing saved in api_db" in test["caught_kinds"] + + +def test_no_network_capture_proposes_nothing(tmp_path: Path) -> None: + learned = learn_record_check(_recording(tmp_path / "rec", network=False), inputs={"note": {}}) + + assert learned["found"] is False + assert "no network capture" in learned["reason"] + + +def test_a_secret_input_is_never_used_to_find_the_record(tmp_path: Path) -> None: + learned = learn_record_check( + _recording(tmp_path / "rec"), inputs={"note": {}}, secret_inputs=frozenset({"note"}) + ) + + assert learned["found"] is False + + +def test_steps_are_quiet_only_when_every_write_was_the_save(tmp_path: Path) -> None: + quiet = quiet_steps(_recording(tmp_path / "one")) + assert quiet is not None and quiet["write_requests"] == 1 + + # A write while the record was being opened: no claim about the others. + assert quiet_steps(_recording(tmp_path / "two", stray_write=True)) is None + assert quiet_steps(_recording(tmp_path / "none", network=False)) is None + + +def test_values_from_one_or_several_recordings(tmp_path: Path) -> None: + inputs = [ + {"name": "note", "example": NOTE}, + {"name": "clinic", "example": "North"}, + {"name": "pin", "example": "0000", "secret": True}, + ] + one = _recording(tmp_path / "a", params={"note": NOTE, "clinic": "North"}) + proposals = {item["name"]: item for item in value_proposals(inputs, [one])} + assert set(proposals) == {"note", "clinic"} # a secret is never offered + assert proposals["note"]["changes"] and proposals["clinic"]["changes"] + + two = _recording(tmp_path / "b", params={"note": "Another note", "clinic": "North"}) + proposals = {item["name"]: item for item in value_proposals(inputs, [one, two])} + assert proposals["note"]["changes"] is True + assert proposals["clinic"]["changes"] is False + assert proposals["clinic"]["why"] == "The same in all 2 recordings." + + +class _Dispatcher: + """Records the qualification commands a "yes" sends, in order.""" + + def __init__(self, data_dir: Path, recording: Path, *, project: bool = True) -> None: + self.config = type("C", (), {"data_dir": data_dir})() + self.recording = recording + self.calls: list[tuple[str, dict]] = [] + self.has_project = project + db = type("DB", (), {})() + db.get_bundle = lambda _id: {"capture_id": "cap"} + db.get_capture = lambda _id: { + "capture_path": str(recording), + "task_description": "Add a note", + } + db.list_captures = lambda limit=200: [] + self.services = type("S", (), {"db": db})() + + def _task_inputs(self, workflow_id: str) -> list[dict]: + return [{"name": "note", "example": NOTE, "secret": False}] + + def _task_target(self, workflow_id: str) -> dict: + return {"backend": "web", "url": BASE + "/"} + + def get_qualification(self, **params: Any) -> dict: + nodes = [ + {"id": "step_000", "kind": "action", "risk": "reversible", "effects": []}, + {"id": "step_001", "kind": "action", "risk": "reversible", "effects": []}, + {"id": "step_002", "kind": "action", "risk": "irreversible", "effects": []}, + ] + return { + "ok": True, + "project": {"cases": []} if self.has_project else None, + "graph": {"nodes": nodes}, + "controls": { + "actions": { + "step_002": { + "identity": {"can_arm": True, "armed": True, "policy": None}, + "effects": [], + } + } + }, + } + + def _record(self, name: str, params: dict) -> dict: + self.calls.append((name, params)) + return {"ok": True} + + def set_qualification_risk(self, **params: Any) -> dict: + return self._record("set_qualification_risk", params) + + def arm_qualification_identity(self, **params: Any) -> dict: + return self._record("arm_qualification_identity", params) + + def bind_qualification_effect(self, **params: Any) -> dict: + return self._record("bind_qualification_effect", params) + + +def test_yes_sends_the_same_commands_the_forms_send(tmp_path: Path) -> None: + dispatcher = _Dispatcher(tmp_path / "data", _recording(tmp_path / "rec")) + + result = accept_record_check(dispatcher, "wf1", answered_by="agent") + + names = [name for name, _ in dispatcher.calls] + assert names == [ + "set_qualification_risk", # step_000: sent nothing + "set_qualification_risk", # step_001: sent nothing + "arm_qualification_identity", + "set_qualification_risk", # the save, confirmed as irreversible + "bind_qualification_effect", + ] + quiet = [params for name, params in dispatcher.calls[:2]] + assert all(params["risk"] == "read_only" for params in quiet) + assert "by the agent" in quiet[0]["explanation"] + assert dispatcher.calls[3][1]["risk"] == "irreversible" + assert dispatcher.calls[4][1]["match_param"] == "note" + assert result["answers"]["record_check"]["by"] == "agent" + # The read becomes the record-check settings every later run uses. + config = SetupStore(tmp_path / "data").record_check_config("wf1") + assert config is not None and "records_path: /api/db" in config.read_text() + + +def test_no_learned_check_means_no_yes(tmp_path: Path) -> None: + dispatcher = _Dispatcher(tmp_path / "data", _recording(tmp_path / "rec", network=False)) + + with pytest.raises(CheckSetupError, match="no record check to propose"): + accept_record_check(dispatcher, "wf1", answered_by="person") + assert dispatcher.calls == [] + + +def test_fixed_values_are_the_recorded_ones_and_follow_the_sealed_version(tmp_path: Path) -> None: + dispatcher = _Dispatcher(tmp_path / "data", _recording(tmp_path / "rec")) + answer_values(dispatcher, "wf1", [], answered_by="person") + store = SetupStore(tmp_path / "data") + assert store.load("wf1")["fixed_values"] == {"note": NOTE} + + with pytest.raises(CheckSetupError, match="Unknown value"): + answer_values(dispatcher, "wf1", ["member_id"], answered_by="person") + + store.write_record_check_config("wf1", {"kind": "rest", "base_url": BASE}) + saved = store.load("wf1") + saved["record_check_config"] = str(store.root / "wf1.record-check.yaml") + store.save("wf1", saved) + carry_to_sealed(tmp_path / "data", "wf1", "wf2") + assert store.load("wf2")["fixed_values"] == {"note": NOTE} + assert store.record_check_config("wf2") is not None + + +def test_the_store_refuses_a_workflow_id_that_is_a_path(tmp_path: Path) -> None: + with pytest.raises(CheckSetupError): + SetupStore(tmp_path).load("../other") diff --git a/packages/desktop/tests/test_engine/test_qualification.py b/packages/desktop/tests/test_engine/test_qualification.py index 339659a0b..e9750a769 100644 --- a/packages/desktop/tests/test_engine/test_qualification.py +++ b/packages/desktop/tests/test_engine/test_qualification.py @@ -378,7 +378,9 @@ def test_local_case_result_refuses_unbound_retained_evidence( observation=signed_observation.model_dump(mode="json"), ) ) - with pytest.raises(QualificationError, match="current signed receipt"): + # The retained report binds the signed observation now, so the refusal + # comes from the case itself: no approved input or scope binds this run. + with pytest.raises(QualificationError, match="does not bind this exact input and run"): record_local_qualification_result( bundle, workflow_id="wf-1", diff --git a/packages/desktop/tests/test_engine/test_qualification_capabilities.py b/packages/desktop/tests/test_engine/test_qualification_capabilities.py index f48c1541f..58ccbc180 100644 --- a/packages/desktop/tests/test_engine/test_qualification_capabilities.py +++ b/packages/desktop/tests/test_engine/test_qualification_capabilities.py @@ -436,3 +436,79 @@ def test_untyped_step_evidence_cannot_mint_capabilities() -> None: report_sha256=_sha(), action_kinds={"submit": "click"}, ) + + +def test_current_receipt_is_bound_by_the_retained_report(tmp_path: Path) -> None: + """``retain_run_evidence`` keeps the exact report.json, not a receipt. + + The observation is current only while that report's digest is the one the + observation signed; Desktop never wrote run-report-receipt.json after it + started retaining the report, so every practice and drill was refused. + """ + + private = Ed25519PrivateKey.generate() + private_raw = private.private_bytes( + encoding=serialization.Encoding.Raw, + format=serialization.PrivateFormat.Raw, + encryption_algorithm=serialization.NoEncryption(), + ) + public_key = b64encode( + private.public_key().public_bytes( + encoding=serialization.Encoding.Raw, + format=serialization.PublicFormat.Raw, + ) + ).decode("ascii") + environment = SimpleNamespace( + environment_digest="4" * 64, + runtime_version="1.24.0", + target_kind="web", + required_capabilities=[], + contract_sha256=lambda: "3" * 64, + ) + project = SimpleNamespace( + project_id="project-1", + revision=1, + environment=environment, + trusted_runner_keys={"desktop-local-v1": public_key}, + contract_sha256=lambda: "1" * 64, + ) + observation = collect_qualification_capabilities( + _report(results=[]), + expected_target_kind="web", + runtime_version="1.24.0", + report_sha256=_sha(), + action_kinds={}, + ) + signed = sign_qualification_capability_observation( + observation, + project_id="project-1", + project_revision=1, + project_contract_sha256="1" * 64, + workflow_contract_sha256="2" * 64, + environment_contract_sha256="3" * 64, + environment_digest="4" * 64, + case_id="representative", + run_id="run-1", + attestation_key_id="desktop-local-v1", + private_key=private_raw, + ) + reference = retain_capability_observation( + tmp_path, + case_id="representative", + run_id="run-1", + observation=signed.model_dump(mode="json"), + ) + report = (tmp_path / "qualification-evidence" / reference["relative_path"]).with_name( + "report.json" + ) + + def current() -> dict: + return current_signed_capability_observations( + tmp_path, workflow_contract_sha256="2" * 64, project=project + ) + + assert current() == {} # no report retained: nothing is current + report.write_bytes(b"exact run report") + assert current() == {"representative": signed} + report.write_bytes(b"another run report") + assert current() == {} diff --git a/packages/desktop/uv.lock b/packages/desktop/uv.lock index 61e69db20..ed7c267c0 100644 --- a/packages/desktop/uv.lock +++ b/packages/desktop/uv.lock @@ -1159,7 +1159,7 @@ wheels = [ [[package]] name = "openadapt-desktop" -version = "0.21.1" +version = "0.22.0" source = { editable = "." } dependencies = [ { name = "cryptography" }, diff --git a/packages/flow/openadapt_flow/backends/playwright_backend.py b/packages/flow/openadapt_flow/backends/playwright_backend.py index d2250c3ad..2d8b784c7 100644 --- a/packages/flow/openadapt_flow/backends/playwright_backend.py +++ b/packages/flow/openadapt_flow/backends/playwright_backend.py @@ -3593,7 +3593,11 @@ def close() -> None: close() raise page.goto(url) - return cls(page), close + backend = cls(page) + # Read by the generic web environment observer: the environment + # digest of a headless and a headed browser differ. + backend.headless = headless + return backend, close from openadapt_flow.backends._browser_process import ( # noqa: E402 diff --git a/packages/flow/openadapt_flow/qualify/environment.py b/packages/flow/openadapt_flow/qualify/environment.py index b4299b028..cd7c504e4 100644 --- a/packages/flow/openadapt_flow/qualify/environment.py +++ b/packages/flow/openadapt_flow/qualify/environment.py @@ -100,10 +100,39 @@ def observe( ) +def observe_web_environment( + url: str, *, application_version: str, headless: bool = True +) -> dict[str, str]: + """Open ``url`` once, read its environment, and close it. + + For binding a qualification project to the generic web observer before + the first case runs: the digest is the one a governed run of the same + browser on this machine observes. Nothing on the page is clicked or typed. + """ + + from openadapt_flow.backends.playwright_backend import PlaywrightBackend + + backend, close = PlaywrightBackend.launch(url, headless=headless) + try: + observed = WebEnvironmentObserver( + application_version=application_version, headless=headless + ).observe(backend, "web") + finally: + close() + return { + "environment_observer_id": WEB_OBSERVER_ID, + "environment_observer_contract_sha256": WEB_OBSERVER_CONTRACT_SHA256, + "application_identity": observed.application_identity, + "application_version": observed.application_version, + "environment_digest": observed.environment_digest, + } + + __all__ = [ "WEB_OBSERVER_CONTRACT_SHA256", "WEB_OBSERVER_ID", "WebEnvironmentObserver", + "observe_web_environment", "web_environment_digest", "web_environment_payload", ] diff --git a/packages/flow/openadapt_flow/runtime/replayer.py b/packages/flow/openadapt_flow/runtime/replayer.py index f9ad6a1dc..49681fc5d 100644 --- a/packages/flow/openadapt_flow/runtime/replayer.py +++ b/packages/flow/openadapt_flow/runtime/replayer.py @@ -13799,13 +13799,36 @@ def _bind_qualification_boundary( # A run has one configured observer. A surface whose boundary # names another observer uses that observer's own route below. configured_observer = None - if configured_observer is None: - from openadapt_flow.qualification_web_session import ( - SIGNED_IN_WEB_SESSION_OBSERVER_ID, - SignedInWebSessionObserver, - ) + from openadapt_flow.qualification_web_session import ( + SIGNED_IN_WEB_SESSION_OBSERVER_ID, + SignedInWebSessionObserver, + ) + from openadapt_flow.qualify.environment import ( + WEB_OBSERVER_ID, + WebEnvironmentObserver, + ) - if boundary.environment_observer_id == SIGNED_IN_WEB_SESSION_OBSERVER_ID: + if isinstance( + configured_observer, BackendQualificationEnvironmentObserver + ) and boundary.environment_observer_id in ( + SIGNED_IN_WEB_SESSION_OBSERVER_ID, + WEB_OBSERVER_ID, + ): + # The deployment's default is the backend's own marker observer. + # A project bound to a page-owned web observer uses that observer + # instead; its id, contract and environment digest below still + # have to match the project exactly. + configured_observer = None + if configured_observer is None: + if boundary.environment_observer_id == WEB_OBSERVER_ID: + # A web app with neither OpenAdapt markers nor a session + # cookie: its origin, browser, viewport, platform, runtime + # and the version the operator confirmed (qualify's observer). + configured_observer = WebEnvironmentObserver( + application_version=boundary.application_version, + headless=bool(getattr(self.backend, "headless", True)), + ) + elif boundary.environment_observer_id == SIGNED_IN_WEB_SESSION_OBSERVER_ID: # Real web apps carry no OpenAdapt markers: identify them by # origin and signed-in session, with the version the operator # confirmed in Check it. The contract and environment digests diff --git a/packages/flow/tests/test_web_environment_binding.py b/packages/flow/tests/test_web_environment_binding.py new file mode 100644 index 000000000..eb2909164 --- /dev/null +++ b/packages/flow/tests/test_web_environment_binding.py @@ -0,0 +1,145 @@ +"""A project bound to the generic web observer uses it at run time. + +A web app with no OpenAdapt markers can't be observed by the backend's own +marker observer, which is what a deployment wires by default for a +qualification case. A project whose boundary names Flow's generic web +observer (or the signed-in session observer) gets that observer instead, and +every id, contract and digest is still compared with the project. +""" + +from __future__ import annotations + +from typing import Any + +from openadapt_flow.qualification import EnvironmentBoundary +from openadapt_flow.qualification_environment import ( + BackendQualificationEnvironmentObserver, +) +from openadapt_flow.qualify.environment import ( + WEB_OBSERVER_CONTRACT_SHA256, + WEB_OBSERVER_ID, + WebEnvironmentObserver, +) +from openadapt_flow.runtime.replayer import Replayer + +ORIGIN = "http://127.0.0.1:18917" + + +class _Browser: + version = "131.0.0.0" + + +class _Context: + browser = _Browser() + + +class _Page: + context = _Context() + + def __init__(self) -> None: + self.time_origin = 1000.0 + + def evaluate(self, script: str) -> Any: + if script == "location.origin": + return ORIGIN + if script == "navigator.userAgent": + return "Mozilla/5.0 HeadlessChrome/131.0.0.0" + if script == "window.devicePixelRatio": + return 1 + if "timeOrigin" in script: + return '{"origin":"%s","timeOrigin":%s,"userAgent":"x"}' % ( + ORIGIN, + self.time_origin, + ) + raise AssertionError(script) + + +class _Backend: + viewport = (1280, 800) + headless = True + + def __init__(self) -> None: + self.page = _Page() + self.guard = None + + def set_qualification_input_guard(self, guard: Any) -> None: + self.guard = guard + + +def _boundary(backend: _Backend, **change: Any) -> EnvironmentBoundary: + observed = WebEnvironmentObserver(application_version="recorded").observe( + backend, "web" + ) + fields = dict( + target_kind="web", + application="127.0.0.1", + application_identity=ORIGIN, + application_version="recorded", + environment_observer_id=WEB_OBSERVER_ID, + environment_observer_contract_sha256=WEB_OBSERVER_CONTRACT_SHA256, + environment_digest=observed.environment_digest, + runtime_version="test", + ) + fields.update(change) + return EnvironmentBoundary(**fields) + + +def _replayer(backend: _Backend) -> Replayer: + replayer = Replayer(backend, vision=object()) + # What build_replayer wires for every qualification case. + replayer.qualification_environment_observer = ( + BackendQualificationEnvironmentObserver(backend) + ) + return replayer + + +def test_a_web_observer_project_binds_with_the_web_observer() -> None: + backend = _Backend() + refusal, observed, ids = _replayer(backend)._bind_qualification_boundary( + _boundary(backend) + ) + + assert refusal is None + assert ids == (WEB_OBSERVER_ID, WEB_OBSERVER_CONTRACT_SHA256) + assert observed is not None and observed.application_identity == ORIGIN + assert backend.guard is not None + + +def test_every_digest_is_still_compared() -> None: + backend = _Backend() + other = _boundary(backend, environment_digest="0" * 64) + refusal, _observed, _ids = _replayer(backend)._bind_qualification_boundary(other) + assert ( + refusal + == "observed qualification environment digest does not match the project" + ) + + version = _boundary(backend, application_version="9.9") + refusal, _observed, _ids = _replayer(backend)._bind_qualification_boundary(version) + assert refusal is not None + + +def test_a_reload_mid_run_stops_before_the_next_input() -> None: + backend = _Backend() + replayer = _replayer(backend) + refusal, _observed, _ids = replayer._bind_qualification_boundary(_boundary(backend)) + assert refusal is None + + backend.page.time_origin = 2000.0 # a new page load + try: + backend.guard() + except Exception as exc: # StructuralResolutionRefused + assert "changed before input" in str(exc) + else: # pragma: no cover - must refuse + raise AssertionError("a reloaded page was accepted") + + +def test_a_project_without_an_observer_keeps_the_backend_observer() -> None: + backend = _Backend() + plain = _boundary( + backend, + environment_observer_id=None, + environment_observer_contract_sha256=None, + ) + refusal, _observed, _ids = _replayer(backend)._bind_qualification_boundary(plain) + assert refusal == "qualification environment observer does not match the project"