Skip to content

Add GPU device selection, upgrade harnesses, and standardize task docs - #29

Merged
hanhainebula merged 2 commits into
mainfrom
update-gpu-harness-task-docs
Oct 23, 2026
Merged

hanhainebula merged 2 commits into
mainfrom
update-gpu-harness-task-docs

Conversation

@hanhainebula

@hanhainebula hanhainebula commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Make GPU selection available through the common evaluation command, refresh the coding-agent runtimes, and give all 49 existing task packages a consistent README and version baseline. Update the evaluation guides, contribution guidance, task scaffolding, and maintainer checks to match.

GPU selection

  • Add optional --gpu-device <host-index-or-GPU-UUID> for all 23 GPU tasks. Resolve the device once and use the same UUID for the agent and separate verifier.
  • Generate per-run Compose configurations that replace the GPU reservation while preserving mounts, network isolation, and cleanup behavior. Omitting the option retains the default single-GPU request.
  • Move query-encoder-alignment onto the common interface and document selection in the English/Chinese READMEs, evaluation guides, and GPU task READMEs.

For example, select host GPU 2 using the existing evaluation command:

bash scripts/run_task.sh --task code-embedding-finetuning --agent codex \
  --gpu-device 2 --output jobs/code-embedding-finetuning-gpu2

Harness updates

  • Pin Codex to 0.164.0, Claude Code to 2.1.309, and Pi to 1.4.0 across the launcher, adapters, all 49 agent images, documentation, and tests. Centralize the launcher pins in scripts/harness_versions.py and support Pi's max thinking level.
  • Upgrade the Codex runtime used by all 44 RewardKit judge integrations to 0.164.0, including verifier installation and audit-user cache paths.
  • Keep Harbor at 0.22.0 and RewardKit at 0.2.0. The RewardKit wrapper uses a version-guarded override of its internal Codex pin; future RewardKit upgrades require revalidating that integration.

Task documentation

  • Standardize every task README around its title and five sections: Task Metadata, Change Log, Difficulty Explanation, Verification Explanation, and References.
  • Set all task versions to 1.0.0 and record 2026-10-23 | 1.0.0 | Added initial task. Remove duplicate legacy version declarations from the README prose. Preserve historical validation reports and their original versions, scores, and dates.
  • Retain task-specific constraints, scoring details, usage instructions, and existing references. Correct stale README status descriptions using existing validation records.
  • Update the scaffolder, authoring/validation guidance, contribution documentation, and maintainer audit checklist so future tasks follow the same format.

Validation evidence

  • python scripts/check_tasks.py: 0 errors.
  • python -m unittest discover -s scripts/tests -p 'test_*.py': 181 tests, 179 passed, 2 skipped because RewardKit is absent from the host environment. The RewardKit integration was separately checked in an existing verifier container.
  • python .agents/skills/create-searchswe-task/scripts/test_scaffold_task.py: 5 passed. Both task skills passed skill validation.
  • All 49 task README structures, versions, and initial changelog entries passed consistency checks; 562 local Markdown links/anchors were checked. git diff --check passed.
  • Rendered and compared all 46 GPU phase configurations. Device selection preserved other Compose settings. During implementation, author-run minimal agent/verifier containers on GPUs 6 and 7 received the selected UUIDs; CUDA/NVML enumeration and a scalar CUDA operation succeeded. Complete successful stdout and docker inspect captures were not retained in the review evidence bundle, so these observations are distinct from the independently reproduced Compose checks.
  • Each upgraded coding CLI completed a minimal call against a local mock API and exited successfully. Pi output converted successfully to Harbor's trajectory format. RewardKit with the upgraded Codex parsed a structured result and used the expected versioned cache.

Validation was intentionally limited to static checks, automated tests, and minimal runtime/protocol checks. Full task evaluations were not run. Real external model API connectivity remains unverified because credentials were unavailable.

The documentation follow-up removes two stale inline version declarations in decision-model-training and lightweight-query-retrieval. All 49 task README versions, section structures, and initial changelogs were rechecked against task.toml; the follow-up diff contains only those two removals and passes git diff --check. Historical trial text and reports remain byte-for-byte unchanged. Runtime tests were not repeated for this documentation-only correction.

Contributor checklist

  • Shared infrastructure changes and all affected existing task packages are included.
  • Task authors, instructions, fixed inputs, asset manifests, and original validation reports are preserved.
  • Changes exclude secrets, downloaded assets, and job outputs.
  • Validation evidence and its limits are recorded.

Maintainer merge checklist

  • Review GPU selection across both phases and the RewardKit version guard.
  • Confirm required checks against the final PR head and current main.
  • Merge with a merge commit to preserve contributor history.

Select a shared GPU UUID for agent and verifier phases. Upgrade Codex, Claude Code, Pi, and the RewardKit Codex runtime with compatibility checks. Normalize all task READMEs and versions, and align scaffolding and contributor guidance.
@hanhainebula
hanhainebula merged commit b17b701 into main Oct 23, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant