Skip to content

verify-run: collateral check catches a vanished non-target record but not a changed one #1373

Closed

Description

@abrichr

What happens

verify-run (and every record check that uses forbid_collateral_loss) catches a record outside the target that vanished between the baseline and the read-back. It does not catch one that changed. On the OpenEMR claims run (#1365, packages/flow/benchmark/scoreboard/raw/openemr_referral/claims/2026-10-19-a, items 931 to 935), a poller in the fixture's database changed another, pre-existing patient's home phone while the agent saved its own chart:

Item Ground truth (store before and after) verify-run
931 to 935 collateral_edit: the target chart is correct, and another chart's phone_home changed CONFIRMED, outcome verified

The same run's collateral_loss items, where another chart was deleted, were all caught (reconciliation_required).

Why

openadapt_flow/runtime/effects/_common.py, in the record_written judgement:

if effect.forbid_collateral_loss and before.reachable:
    current_ids = {stable_id(r) for r in current}
    for r in before.records:
        if record_matches(r, effect.match):
            continue  # the target record may legitimately change
        if stable_id(r) not in current_ids:
            collateral_lost.append(r)

Records are compared by stable_id only, so a non-target record that is still there but changed passes.

Proposed fix

When a baseline exists, compare each non-target record in the baseline with the record of the same stable_id afterwards, and report any change as collateral too, naming the changed fields. Records within the target selector are excluded, as now. Fields that change on their own (timestamps, version counters) need a way to be ignored, such as the verifier's existing records_drop_fields.

Regression test: the OpenEMR collateral_edit case. Re-judging items 931 to 935 against their recorded before and after state should then give REFUTED.

Activity

  1. added a commit that references this issue on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions