Repository navigation
Apply task-specific piecewise score normalization - #22
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Task names and package paths: the 13 existing
tasks/<name>packages listed below.Goal, starting system, and CPU/GPU requirements: update final-score normalization; raw quality metrics, starting resources, hardware, submission interfaces and execution limits are unchanged.
Authors and coauthors: Haoran Jin; existing authors and contributor history are preserved.
khmer-embedding-training,financial-report-qa,evolving-personal-memory,document-page-search,layout-page-search,visual-document-qa,composed-image-search,sketch-text-search,same-place-image-search,video-moment-search,narrative-clue-search,spoken-query-search,icsi-meeting-memory.Each task now uses two linear segments with a fixed reference quality and an integer score anchor. Raw quality 0, the reference quality, and 1 map to normalized scores 0, the declared anchor, and 100. Harbor retains its 0–1 reward precision; displayed scores use one decimal place.
Raw metrics remain available separately, and executable, semantic-judge and trajectory-audit gates retain their existing behavior. Khmer uses raw Recall@100 instead of normalizing an already-normalized value. README scoring descriptions and task versions are updated. Twelve
instruction.mdfiles are byte-identical; only ICSI's obsolete scoring sentence is aligned with the new conversion.Validation evidence
python scripts/check_release.py: passed.python scripts/check_tasks.py --merge-ready --base 04bdc08548473d3661c67724ff514db7ab6717aa: passed.git diff --check: passed.unittest discoverin cached verifier images with networking disabled: 512 tests passed, no skips.Contributor checklist
tasks/<task-name>paths are preserved; no new tasks are introduced.task.toml; original author commits preserved.Maintainer merge checklist
tasks/<task-name>/, previous assets preserved and merged official SHA pinned.