Skip to content

feat(orca): share execution plans in supervised cycles (#330) - #384

Open
datacas wants to merge 1 commit into
mainfrom
task-330
Open

datacas wants to merge 1 commit into
mainfrom
task-330

Conversation

@datacas

@datacas datacas commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Use the shared execution-plan operations across native and supervised Orca paths, and block planned Orca launches unless the existing Task's plan can be confirmed. Launch completion remains transport evidence; it cannot advance the functional cycle.

Refs #330. The issue remains open until the required exact-HEAD host Orca campaign passes.

Run: schema:2 profile:deep_coder requested:openai/gpt-6.1-sol effort:medium resolved:?

Diagnosis

Feature request; decision implement. The native PlanRuntime owned persistence and HEAD invalidation, OrcaDispatchContext had no plan, and the native route did not build that context. This confirms the issue's mechanism: transport-specific interpretation could diverge from the shared obligations. Defect reproduction was not applicable; fixtures exercise the new behavior. The invariant is that every transport consumes the same effective plan, canonical receipt, policy gates and HEAD-bound evidence rules, while Orca retains Run/Task/worker ownership.

Related work was searched by execution-plan/Orca terms and the implicated shared symbols: #329 is closed; merged #361, #362, #366 and #371 establish the shared contracts. #382 tracks host-attested evidence separately and explicitly is not a strict dependency of #330.

Work units

Unit Behaviour Files Tests Verification Rollback
WU-1 Shared plan persistence, invalidation and delegation; validated Task binding; pending launch evidence; fail-closed resume/cancellation; conditional rereview in all five modes scripts/plan_service.py, scripts/orca_plan.py, native driver/adapter modules, runtime/test manifests, skills/cc-orca-orchestrator/SKILL.md, docs/execution-plan.md, docs/orca-execution-plan.md tests/test_orca_plan.py; existing driver, installation, fast-path and workflow suites Agent-reported exact-HEAD verification is recorded in the stage summary comment independent: revert this commit; appended local telemetry history is retained and is not a publication or migration

Behaviour and documentation

  • Both transports share canonical receipt generation, CycleHistoryStore.save_plan, HEAD invalidation and security requirements. Frozen check identities/history survive recovery; stale results cannot satisfy another HEAD.
  • Managed Orca dispatches carry the effective profile, plan ID, revision, digest and complete Task spec. A coordinator-supplied reader must confirm the actual existing Task contents before worker-start. Existing workspace and publication gates remain active.
  • Native CLI flags consume existing coordinator/Run/Task identities. A confirmed launch records dispatch identity and plan receipt, then stops IN_PROGRESS with result_pending evidence. Missing capabilities report BLOCKED without inventing a coordinator or relaunching.
  • Resume/cancellation reconcile persisted plans and launch identities. Authoritative-outcome recovery validates schema, plan receipt, scope, HEAD, terminal status and evidence validity; it is not approval. Cancellation acknowledgement also remains pending.
  • The Orca skill consumes the installed helper and uses the shared conditional rereview decision, including Full. Unchanged open findings retain their blocks.

The new behaviour was not already documented. docs/orca-execution-plan.md now owns the helper/CLI flow, all five mode examples and terminal obligations, unsupported interfaces and required live evidence; docs/execution-plan.md links it. Test registration covers Windows shards and documentation fast-path dependencies.

Verification limits

Agent-reported local verification on c7f03906d2b7e76ddf3acb4b691f4ab229e705d6:

  • python3 -m unittest discover -s tests: exit 0, 2,143 tests, one skipped (270.107 seconds).
  • python3 -m unittest discover -s tests -p test_orca_plan.py: exit 0, 14 tests, including the helper subprocess boundary.
  • python3 scripts/validate-package.py: exit 0; 16 skills and runtime manifests validate.
  • git diff --check: exit 0.

The first full run was on an uncommitted worktree and failed (2,137 tests, 13 failures and one error). It exposed missing Windows shard and fast-path registrations, which were fixed before this commit. Its remaining driver case passed on a focused rerun and in the final full run. That earlier dirty-tree run is retained as history, not evidence for this committed HEAD.

The stock native CLI has no verified live Task-spec reader, authoritative-outcome or cancellation binding. It deliberately blocks these capabilities. API fixtures supply an existing coordinator's capability; no undocumented Orca CLI option or endpoint is assumed. The required manual check orca-live remains unverified until a host campaign on this exact commit records the actual installed version, commands, environment, Task contents, identities, plan receipt and authoritative functional outcomes for launch/resume/partial/stale/cancel cases. Orca 1.4.209 is planning context only, not observed evidence. No Orca CLI was run in the implementer sandbox.

Local fixtures use temporary repositories, fake process boundaries and explicit test identities; they do not require provider credentials or user-global Git identity. Prepared dependencies were inspected and are not counted as tests. BasedPyright reports 132 existing errors on both the base snapshot and the touched final runtime files, with no additional error messages; the raw type check still exits 1. No type-check settings were changed.

Security audit ran over the accumulated change: plan/Task/result identity boundaries, persisted commands, publication gates, workspace isolation and helper input handling. No confirmed vulnerability was found. Live coordinator authentication/capabilities were not exercised; the required host campaign remains the verification boundary. This is not review approval or merge readiness.

@datacas

datacas commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Implementation for #330 published as c7f0390.

Run: schema:2 profile:deep_coder requested:openai/gpt-6.1-sol effort:medium resolved:?

All required local executable checks pass: configured suite 2,143 tests (one skipped), Orca fixtures and helper boundary 14 tests, and package validation. The required host campaign remains unverified; this is not merge readiness. The stock CLI blocks planned Orca launch/resume/cancel capabilities whose live bindings have not been verified.

Security audit ran: no confirmed changed-surface vulnerability. Live coordinator authentication/capabilities remain unverified. BasedPyright CLI reports 132 baseline errors and no additional error messages on the touched runtime files. No settings were weakened.

Verification (agent-reported), tests.version=2. The frozen marker manual:orca-live is not a shell command: it names the required host campaign in docs/orca-execution-plan.md, which must supply actual version, commands, environment, exact HEAD and authoritative outcomes. No Orca CLI was run here.

{
  "version": 2,
  "ran": true,
  "passed": false,
  "conclusion": "not_verified",
  "boundary": "required",
  "head_sha": "c7f03906d2b7e76ddf3acb4b691f4ab229e705d6",
  "checks": [
    {
      "id": "configured-test",
      "command": "python3 -m unittest discover -s tests",
      "executor": "codex",
      "profile": "deep_coder",
      "context": "project",
      "required": true,
      "criteria": [],
      "level": "test",
      "provenance": "project",
      "reason": "configured test command",
      "head_sha": "c7f03906d2b7e76ddf3acb4b691f4ab229e705d6",
      "result": "pass",
      "evidence": [
        {
          "exit_code": 0,
          "executed": 2143
        }
      ],
      "cause": "",
      "attempts": [],
      "history": [
        {
          "head_sha": "7c0f28b5384b37a43daa6ef5a793f90a18006007",
          "result": "fail",
          "evidence": [
            {
              "exit_code": 1,
              "executed": 2137,
              "dirty_worktree": true
            }
          ],
          "cause": "defect",
          "attempts": [
            {
              "note": "Uncommitted implementation changed during this first run; fixed missing test registrations before final commit. This history does not verify the baseline commit."
            }
          ]
        }
      ]
    },
    {
      "id": "orca-parity",
      "command": "python3 -m unittest discover -s tests -p test_orca_plan.py",
      "executor": "codex",
      "profile": "deep_coder",
      "context": "project",
      "required": true,
      "criteria": [
        "AC-1",
        "AC-2",
        "AC-3",
        "AC-4",
        "AC-5",
        "AC-6",
        "AC-7",
        "AC-8"
      ],
      "level": "boundary",
      "provenance": "agent",
      "reason": "Orca parity fixtures and installed-helper subprocess boundary",
      "head_sha": "c7f03906d2b7e76ddf3acb4b691f4ab229e705d6",
      "result": "pass",
      "evidence": [
        {
          "exit_code": 0,
          "executed": 14
        }
      ],
      "cause": "",
      "attempts": []
    },
    {
      "id": "package-validation",
      "command": "python3 scripts/validate-package.py",
      "executor": "codex",
      "profile": "deep_coder",
      "context": "project",
      "required": true,
      "criteria": [
        "AC-2",
        "AC-4",
        "AC-6"
      ],
      "level": "static",
      "provenance": "project",
      "reason": "runtime manifests and skill/package validation",
      "head_sha": "c7f03906d2b7e76ddf3acb4b691f4ab229e705d6",
      "result": "pass",
      "evidence": [
        {
          "exit_code": 0,
          "executed": null
        }
      ],
      "cause": "",
      "attempts": []
    },
    {
      "id": "orca-live",
      "command": "manual:orca-live",
      "executor": "host",
      "profile": "deep_coder",
      "context": "host-orca",
      "required": true,
      "criteria": [
        "AC-1",
        "AC-2",
        "AC-3",
        "AC-4",
        "AC-5",
        "AC-7",
        "AC-8"
      ],
      "level": "boundary",
      "provenance": "project",
      "reason": "Required exact-HEAD controlled host Orca campaign; non-executable manual marker documented in docs/orca-execution-plan.md",
      "head_sha": "c7f03906d2b7e76ddf3acb4b691f4ab229e705d6",
      "result": "unverified",
      "evidence": [],
      "cause": "Per issue decision 10, Orca is not run in the implementer sandbox. No exact-HEAD host campaign result or observed installed version was supplied.",
      "attempts": []
    }
  ]
}

Checks of the pushed head (single read, parallel CI policy):

{
  "head_sha": "c7f03906d2b7e76ddf3acb4b691f4ab229e705d6",
  "passed": 0,
  "failed": 0,
  "pending": 5,
  "items": [
    {
      "name": "Skills, manifests, and Bash installer",
      "state": "pending"
    },
    {
      "name": "Windows test shard (windows-1)",
      "state": "pending"
    },
    {
      "name": "Windows test shard (windows-2)",
      "state": "pending"
    },
    {
      "name": "Windows test shard (windows-3)",
      "state": "pending"
    },
    {
      "name": "Windows test shard (windows-4)",
      "state": "pending"
    }
  ]
}

@datacas

datacas commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Initial review — PR #384 (work item #330)

[CCR-20261102-001] · senior_reviewer · anthropic/claude-opus-5-5→? · high · schema:2

Verdict: REVIEW_PASSED_AWAITING_CI for head c7f03906d2b7e76ddf3acb4b691f4ab229e705d6. No open finding blocks approval. Three required Windows shards are still pending, so this is not an approval; the cycle's CI gate decides. The required manual check orca-live (AC-7) is still unverified.

Scope

I read the full accumulated diff, origin/main...c7f0390 (merge base 7c0f28b, 1131 lines, matching the runtime's diff file). It covers:

  • the new scripts/plan_service.py and scripts/orca_plan.py;
  • changes to cycle_plan.py, cycle_session.py, cycle_dispatch.py, cycle_cli.py, executors.py and run_cycle.py;
  • the runtime manifest, the Windows test shards and the fast-path test registrations;
  • skills/cc-orca-orchestrator/SKILL.md, docs/orca-execution-plan.md and docs/execution-plan.md;
  • tests/test_orca_plan.py.

The documentation rule in AGENTS.md is met. docs/orca-execution-plan.md is the page for the new behaviour, docs/execution-plan.md links to it, and the README is unchanged, which is correct because no skill or entry point was added.

Sharing the plan logic is behaviour-preserving for the native driver:

  • save_plan still adds one to the revision, now in persist_plan.
  • invalidate_head and require_security are moved unchanged. invalidate_head now also sets head_sha, and metadata.verification_conclusion=not_verified, which save_plan overwrites in the native path.
  • The Orca launch now stops at IN_PROGRESS with result_pending evidence, and is never treated as a functional result.
  • The new resume and cancel paths fail closed.

Findings

[REV-001] · medium · open · - · blocks:no — Native run_cycle.py Orca launch can never pass the Task-spec binding, and the driver tests use a reader that only echoes the expected spec

Paths: scripts/cycle_dispatch.py:58, scripts/cycle_plan.py:330, scripts/executors.py:3243, tests/test_orca_plan.py:1073

Design decision 9 of #330 says the run_cycle.py Orca route "builds OrcaDispatchContext …, persists the launch receipt, and ends in a result-pending state". In this implementation that route cannot be reached outside the fixtures:

  1. The stock CLI has no Task-spec reader. cycle_cli.py builds OrcaDispatchContext(coordinator, run, task) without a task_spec_reader. So every planned Orca stage stops at executors.py:3246 with "no verified Orca Task-spec read interface". The PR description acknowledges this.
  2. A reader would not help either. StageDispatch calls select_plan_stage(role), which runs save_plan() and so persist_plan. That mints a new revision and digest in the same process, immediately before run() builds the context with that plan (cycle_dispatch.py:416 → :58). A Task the coordinator created earlier cannot already contain a spec with that revision. The bound != task_spec check at executors.py:3251 therefore always fails unless the reader writes the spec rather than reads it.
  3. The driver tests do not cover this. In tests/test_orca_plan.py:1073 the reader is lambda **_: context.task_spec(), which returns exactly what the adapter compares against. test_driver_constructs_context_and_persists_pending_launch proves only that the comparison runs, not that a real Task binding can match.

The helper path used by cc-orca-orchestrator is consistent: spec returns the persisted revision without adding one, the coordinator installs that spec, and the adapter reads it back. The gap is only in the native run_cycle.py route.

Impact: there is no regression, and the change is safe because it fails closed. Before this PR the native route had no context and was also blocked. But issue #330 cannot be completed through this route: the orca-live campaign will block at launch on the native path whatever the installed Orca version supports.

Suggested direction, for the resolver to decide:

  • either let the driver consume a revision the coordinator has already persisted with orca_plan.py spec (no new revision minted for the Orca stage);
  • or document the native Orca route as launch-blocked, and narrow decision 9 in the issue.

In both cases, replace or add to the echo reader with a fixture whose Task spec was installed before the driver ran.

Sensitivity triage and security audit

  • Why the audit ran: the runtime plan requires it (security_required=true). My own reading also flags the change as sensitive. It changes agent/toolkit execution boundaries: the Orca dispatch context, publication kwargs in executors.dispatch, how command bindings are rebound from history, and the instructions in skills/cc-orca-orchestrator/SKILL.md.
  • Deterministic rule: no changed path matches the mandatory minimum (CI, secrets, .code-cycle.yml, AGENTS.md, CLAUDE.md, .claude/**, .codex/**, .agents/**, opencode.json*). .code-cycle.yml declares no security_review.always_when.
  • Result: cc-security-review ran over the whole diff for this head. It found no confirmed vulnerability. Surfaces checked:
    • orca_plan.py CLI input: --head must be a 40-character hex SHA; the plan JSON goes through ExecutionPlan.from_dict and validation, with a repo/work-item scope check; initialize refuses to replace an existing plan.
    • SQLite access goes only through the existing parameterized CycleHistoryStore.
    • Command bindings: restore_plan rebinds only exact-digest matches, and history grants no execution permission.
    • reconcile_outcome validates schema, Run/Task, repo, stage, HEAD, receipt, terminal and evidence_valid, and treats a launch or worker_done as non-terminal.
    • In executors.dispatch, publishes is read and the publication_access gate is evaluated (line 3503) before those kwargs are dropped for Orca (line 3604).
    • The native resume path matches persisted launch identities against the requested context before doing anything else.
  • Potential risks, informational:
    1. A task_spec_reader that raises something other than ValueError, OSError, SubprocessError or ExecutorError propagates out of OrcaAdapter.dispatch. It does not return BLOCKED, but no worker is launched.
    2. The delegation spec carries plan.to_dict() with the rebound plain-text check commands into the Orca Task. These are coordinator-supplied commands, not secrets.
  • Not audited: live Orca coordinator authentication and capability behaviour, which needs the host orca-live campaign.

Verification (run in the review clone on c7f0390)

Checks were frozen before they ran.

Check Command Required Criteria Result
configured-test python3 -m unittest discover -s tests yes AC-1, AC-2, AC-3, AC-4, AC-5, AC-6, AC-8 pass: exit 0, Ran 2143 tests in 217.344s, OK (skipped=1)
validate-package python3 scripts/validate-package.py yes — pass: exit 0, VALID: 16 skills (7 cycle, 9 supporting), shared sections in sync, cross-agent manifests aligned
orca-plan-focused python3 -m unittest discover -s tests -p test_orca_plan.py -v no AC-1, AC-2, AC-3, AC-4, AC-5, AC-6, AC-8 pass: exit 0, 14 tests OK
unittest-default python3 -m unittest no — inconclusive: exit 0 but Ran 0 tests. Default discovery from the repository root finds no tests, so this configured command proves nothing.
orca-live host-side Orca campaign on the exact SHA (issue decision 10) yes AC-7 unverified: not runnable in this review stage

How each acceptance criterion is covered:

  • AC-1 (equivalent obligations): plan_service is shared, and test_equivalent_inputs_and_task_specs_in_every_mode checks every mode.
  • AC-2 (profile and revision travel with the delegation): delegation_spec and OrcaDispatchContext.task_spec().
  • AC-3 (a launch is not a result): IN_PROGRESS + result_pending, and reconcile_outcome rejects worker_done. Native-route reachability is limited; see REV-001.
  • AC-4 (review workspace and publication): test_launch_is_only_pending_and_review_workspace_is_preserved, and publication permissions are kept in the spec.
  • AC-5 (HEAD changes and partial results): the shared invalidate_head, plus frozen-check recovery tests.
  • AC-6 (documented modes): docs/orca-execution-plan.md covers the five modes and their terminal obligations, and a test checks them.
  • AC-7 (Orca version and limitations recorded): open. No observed Orca version or verified limitation is recorded yet, and 1.4.209 is planning context only.
  • AC-8 (conditional rereview): test_rereview_is_conditional_in_every_mode_and_retains_blocks, plus the skill update.

The behaviour of REV-001 was confirmed by reading the code: select_plan_stage → save_plan → persist_plan runs immediately before run() builds the context. I did not reproduce it with a non-echo reader.

CI (parallel policy, read once, not waited on)

  • Skills, manifests, and Bash installer: pass
  • Windows test shard (windows-2): pass
  • Windows test shard (windows-1): pending
  • Windows test shard (windows-3): pending
  • Windows test shard (windows-4): pending

The new test_orca_plan module is registered in the windows-1 and windows-3 shard lists.

Unverified items

  • orca-live and AC-7 need the host Orca campaign on this exact SHA. The implementer and reviewer sandboxes do not run Orca.
  • python3 -m unittest (default discovery) runs 0 tests, so it is not evidence.

Residual risks

  • The native Orca route stays launch-blocked (REV-001), and resume and cancel stay BLOCKED until a coordinator binding exists.
  • orca_plan.py initialize keeps the deterministic dry-run plan_id (run:<repo>#<task>). The native driver uses plan-<cycle_id>. Plan identity therefore does not distinguish Orca cycles of the same work item and PR scope; --cycle is recorded separately.
  • BasedPyright baseline errors are pre-existing (implementer report); not rechecked here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant