Skip to content

Apply task-specific piecewise score normalization - #22

Merged
hanhainebula merged 1 commit into
VectorSpaceLab:mainfrom
hr-jin:score-normalization
Oct 21, 2026
Merged

hanhainebula merged 1 commit into
VectorSpaceLab:mainfrom
hr-jin:score-normalization

Conversation

@hr-jin

@hr-jin hr-jin commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Task names and package paths: the 13 existing tasks/<name> packages listed below.
Goal, starting system, and CPU/GPU requirements: update final-score normalization; raw quality metrics, starting resources, hardware, submission interfaces and execution limits are unchanged.
Authors and coauthors: Haoran Jin; existing authors and contributor history are preserved.

khmer-embedding-training, financial-report-qa, evolving-personal-memory, document-page-search, layout-page-search, visual-document-qa, composed-image-search, sketch-text-search, same-place-image-search, video-moment-search, narrative-clue-search, spoken-query-search, icsi-meeting-memory.

Each task now uses two linear segments with a fixed reference quality and an integer score anchor. Raw quality 0, the reference quality, and 1 map to normalized scores 0, the declared anchor, and 100. Harbor retains its 0–1 reward precision; displayed scores use one decimal place.

Raw metrics remain available separately, and executable, semantic-judge and trajectory-audit gates retain their existing behavior. Khmer uses raw Recall@100 instead of normalizing an already-normalized value. README scoring descriptions and task versions are updated. Twelve instruction.md files are byte-identical; only ICSI's obsolete scoring sentence is aligned with the new conversion.

Validation evidence

  • python scripts/check_release.py: passed.
  • python scripts/check_tasks.py --merge-ready --base 04bdc08548473d3661c67724ff514db7ab6717aa: passed.
  • git diff --check: passed.
  • Task-local unittest discover in cached verifier images with networking disabled: 512 tests passed, no skips.
  • Independent production-finalizer fixtures: 296 cases passed, including the actual isolated-Python ICSI entry, valid-zero submission failures, raw metric preservation, and audit/semantic failure gates.
  • Shared contract suite: 174 tests completed (172 passed, two host-dependency skips); all five RewardKit tests additionally passed without skips in the cached offline verifier image. Pi trajectory suite: 21 tests passed.
  • Three independent reviews of the final source passed.
  • Coding-model trials, GPU inference and paid judge calls were not repeated. Validation reuses existing raw measurements and synthetic finalizer inputs; this PR changes their score conversion, not their raw metric calculation or task inputs.

Contributor checklist

  • Existing descriptive task names and tasks/<task-name> paths are preserved; no new tasks are introduced.
  • Actual authors in task.toml; original author commits preserved.
  • Allow edits from maintainers.
  • No secrets, downloaded inputs/models, job outputs, trajectories or author-only solutions.
  • Static checks and applicable offline runtime evidence recorded.
  • Asset manifests and provenance files are unchanged; this PR uploads no data.

Maintainer merge checklist

  • Design, verifier isolation, runtime evidence and any validation exceptions reviewed.
  • Names checked against the current base and queued PRs.
  • Upstream or official asset downloads verified against immutable pins, sizes and SHA-256 hashes.
  • If official publication is needed: new assets published under tasks/<task-name>/, previous assets preserved and merged official SHA pinned.
  • Task-local docs/configuration complete; no shared task inventory updates needed.
  • Required judges, credential isolation and failure cases validated; checks and merge-ready pass.
  • Merge commit; original contributor history retained.

@hanhainebula
hanhainebula merged commit 11a170b into VectorSpaceLab:main Oct 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants