Skip to content

fix(experiment): preserve complete agent output - #644

Open
bingdongni wants to merge 1 commit into
EvoMap:mainfrom
bingdongni:fix/experiment-agent-output
Open

bingdongni wants to merge 1 commit into
EvoMap:mainfrom
bingdongni:fix/experiment-agent-output

Conversation

@bingdongni

Copy link
Copy Markdown

Summary

Experiment comparisons can corrupt multibyte agent output when a UTF-8 character spans stream chunks, or parse an incomplete response when the process exits before its output streams close. Decode the streams incrementally and parse after they close.

What changed

  • Preserve UTF-8 characters across stdout and stderr chunks.
  • Wait for the child process output streams to close before parsing the JSON response.
  • Cover split characters and output written after the child process exits with subprocess tests.

How to test

  • Linux: node --test test/experimentAgentRunner.test.js test/experimentComparison.test.js (32 passed) and npm test (3,308 passed), CI job.
  • Windows: the focused tests passed (31 passed, one inherited-pipe test skipped).
  • node --check index.js passed. I did not run bare node index.js because it starts an evolution run.
  • The template's full test selection had seven failures in proxyTracePlatformInstall.test.js and spawnReplacementProcess.test.js, both excluded by npm test. The same seven failures occur on the upstream base: platform templates are missing and some source assertions target obfuscated files.

Risk

Medium: an agent that leaves its output pipes open after exiting will now wait for the pipes to close or the existing per-arm timeout. The result shape and command invocation are unchanged.

Harness/evaluator governance

Upstream governance surface: src/experiment/agentRunner.js subprocess output handling for comparison arms.
Downstream EvoX impact: Experiment responses preserve complete UTF-8 output; the AgentResult shape and comparison contract are unchanged.
Rollout-local scope: Applies only when an experiment arm invokes its agent CLI; no live rollout is initiated by this change.
Promotion boundary: The experiment runner behavior changes by default only after maintainer review and merge.
Evaluator mismatch sets: Observation and evidence can now include previously corrupted or late output; action, repair, verification, and belief policies are unchanged.
Non-regression evidence: Real subprocess tests cover split stdout, split stderr, and delayed pipe closure; 3,308 project tests passed on Linux. The full template test selection has seven independently reproduced baseline failures as noted above.
Fix-severity review: medium
Owner approval: Experiment runner maintainer review is requested and remains pending.
Security boundary: The existing command, arguments, environment, host, network, and credential access are unchanged.
Rollback: Revert this commit to restore the previous output handling.
Live promotion: no
Autonomous evaluator self-editing: no

Self-check

  • No new runtime dependencies added.
  • Tests added for the new behavior; full template suite passes locally (node --test test/*.test.js). The seven baseline failures and CI results are described above.

Related

None.

@evomap-aiqa-bot

evomap-aiqa-bot Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Sentinel 已审核此变更,未发现需要报告的问题。

开发 agent 查询详情: 调用 MCP 工具 sentinel_pr_review_context,传入 owner_repo="evomap/evolver"、pr_number=644、head_sha="078408daf9f776d8b490dbac5cf2af17b3faf91c",并按条目追加 finding_id 或 finding_key。

由 Sentinel 审核 · 目标提交 078408daf9f7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant