Skip to content

feat: narration proposes branch rules; recordings, a probe or a person certify them - #1291

Merged
abrichr merged 15 commits into
mainfrom
feat/narration-rules
Oct 20, 2026
Merged

abrichr merged 15 commits into
mainfrom
feat/narration-rules

Conversation

@abrichr

@abrichr abrichr commented Oct 18, 2026

Copy link
Copy Markdown
Member

Builds the map row "Use spoken narration while recording to learn why a branch was taken."

Stacked on #1276 (and #1260 through it). Merge those first; this diff then shrinks to the narration work. The install line uses the audio extra from #1271.

What it does

  • Capture scrubs the transcript with openadapt-privacy before storing it, and refuses --audio before the microphone opens when the scrubber is missing. Scrubbed sentences go to narration.json, timed on the recording clock. Per-word text is no longer stored. capture transcribe scrubs too.
  • Flow carries the sentences into the recording and lines each one up with a step (narration.json). openadapt-flow narration attach does the same for any recording from a WAV or a transcript, so web, native, RDP and Citrix all work the same way.
  • induce reads a sentence near a branch ("If the insurance is Medicare, skip the copay screen") and proposes a guard on a recorded field (param_equals) or on screen text (text_present). Works for optional steps and for two-path divergences.
  • Certification. A proposal becomes a guard only when every recording agrees, --probe-on confirms it (new narrated reading), or a person accepts it (--accept-narrated LOC --accepted-by NAME). A contradicting recording keeps the point refused, and a person can't accept it.
  • teach-branch uses a sentence to pick the narrowing parameter when every example and normal run agrees.
  • Explain back. Certified guards keep their sentence in Workflow.guard_sources (omitted when empty, so existing bundles are byte-identical). visualize (HTML, Mermaid, JSON) and repair show show it. Non-local visualize profiles drop the sentence.
  • Model. --narration-model is opt-in, compile time only, on the customer's key, scrubbed outbound, and its answer is validated. Runs make no model calls.

Tests

Synthetic data only. packages/capture/tests/fixtures/narration/medicare_copay.wav is speech generated locally with macOS say (invented name).

  • narrated branch becomes a proposed guard; certifies when recordings agree; a contradicting recording keeps it refused; transcript scrubbed (capture and flow); healthy runs make no model calls
  • probe confirms a guard the recordings can't check; person acceptance; divergence branch; every backend; visualize; teach-branch; CLI
  • End to end with real local Whisper (tiny.en) on the fixture: passes locally; skipped in CI unless the model is cached (never downloads)

Local: tests/test_narration_rules.py 29 passed, test_teach_branch_narration.py 4, capture test_narration.py 7 + test_audio_privacy.py 15; a 37-file Flow slice (induction, probe, teach, visualize, repair, privacy, capture adapter) 690 passed.

Docs: packages/flow/docs/NARRATION.md, capture README/COMPLIANCE/DESIGN, root CAPABILITIES.md row (build_llms_full.py re-run).

🤖 Generated with Claude Code

abrichr and others added 13 commits October 18, 2026 14:29
The transcript is scrubbed with openadapt-privacy before it is stored, and
recording refuses before the microphone opens when no scrubber is installed.
Scrubbed sentences go to narration.json on the recording clock; per-word text
is no longer stored. capture transcribe scrubs too. Synthetic WAV fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…certify them

A sentence lined up near a branch point (an optional step or a two-path
divergence) is parsed into a guard on a recorded field or a screen element.
It certifies only when every recording agrees or a person accepts it; a
contradicting recording keeps the point refused. Certified guards carry their
sentence in Workflow.guard_sources. Narration sidecar, alignment, attach, and
capture-adapter carry-over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion CLI

induce --probe-on also replays the narrated reading where narration proposed
a guard nobody contradicted; it certifies when that reading produces every
recorded effect and dropping the step does not. induce --accept-narrated
LOCATION --accepted-by NAME, --narration-model (opt-in, own key). New
narration attach/show command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion in teach-branch

visualize (HTML, Mermaid, JSON) shows each guard with the sentence that
suggested it and what certified it; non-local profiles drop the sentence.
teach-branch uses a narrated sentence to pick the narrowing parameter only
when every example and normal run agrees. repair show lists the sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- check a param guard against the recordings with the runtime's exact compare
- leave a divergence refused when another step sits inside it, and insert
  branches at the same position in column order
- pin acceptances and probe results to the guard id (LOCATION@GUARD_ID)
- accept only a JSON boolean from the model, and screen text the person said
- parser: stop at 'otherwise', flip 'it's not X', read 'isn't needed' as skip,
  drop double negatives and unplaced negations
- skip a malformed narration.json instead of failing
- link the capability row to #1291

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	CAPABILITIES.md
#	packages/capture/README.md
#	packages/flow/openadapt_flow/ir.py
Regenerate the public artifact inventory for the changed visualize files,
apply ruff format, and fix three mypy errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	CAPABILITIES.md
#	packages/flow/public-artifacts.json
@abrichr

abrichr commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

Included in batch #1384, which runs CI once for several reviewed PRs. This PR is marked merged when #1384 merges. Please don't push here unless asked; a new commit leaves it out of the batch.

@abrichr
abrichr merged commit e968b2d into main Oct 20, 2026
25 of 42 checks passed
abrichr added a commit that referenced this pull request Oct 20, 2026
@abrichr
abrichr deleted the feat/narration-rules branch October 20, 2026 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant