Repository navigation
feat(evals): freeze independently reviewed held-out qualification - #4797
ScriptedAlchemy wants to merge 15 commits into
Conversation
|
Refs #4535 Count wrong extra suggestions on positive cases with frozen unwanted-hint budgets; disclose coverage, abstention and missed positives. Refuse a passing report for missing or nonzero server truncation telemetry. Also prevent cross-project reuse of a session between train and holdout.
Refs #4535 Return nonzero after publishing a failed frozen qualification report and name failed checks on stderr. Preserve machine-readable measurements and correct the held-out refusal command to include the original dataset. Add Bazel-wired CLI regressions for every qualification check, replacement of previous passing reports, invalid evidence, and the passing control. Preserve the concurrent frozen split-group fix in parent 711a18f. Tests require Bazel execution; no local test pass or Kev inference claimed. Keep this decision-model work DRAFT and NO MERGE.
|
Published direct source/test fix Root cause: The command now publishes the full JSON measurements as before, names failed checks on stderr, and exits 1 when qualification fails. Passing qualification still exits 0. Also corrected the unqualified-mode refusal to include the required DATASET argument. Added Concurrency: the first expected-head update did not land because the branch advanced to Verification limitation: original score.py bytes were verified against the Git blob before editing; changed Python files were statically parsed. No Bazel test execution or actual Kev inference is claimed. This runtime has no Bazel/full checkout and GitHub DNS fails from the shell. Current hosted CI is queued with Required nonzero Bazel execution on this exact head: bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errorsFor sweep/PR-maintenance owners: retain the same issue branch and this commit; keep #4797 and mixed #4776 DRAFT, do not merge/auto-merge or route the changes through another branch. #4535 remains open, and release #4763 was untouched. |
Refs #4535 Require an explicit null-or-positive client calibration in the frozen qualification manifest. Refuse a changed --noul-temperature before any network work, and reject an otherwise rehashed result with a different experiment calibration before scoring. Cover malformed declarations, post-hoc rehashing, no-network refusal and the matching calibrated HTTP path with behavior-based regressions. Historical nine-question evaluation remains unqualified for runtime parity. DRAFT ONLY; NO MERGE.
Refs #4535 Only a contiguous prefix of the frozen HTTP case order can be resumed. Previously a partial log containing a later case ahead of an earlier one could skip and reorder inference while preserving apparently valid hashes, misrepresenting the sequence used for cold/warm measurements. Add a falsifiable HTTP-runner regression for skipped and reversed rows, no network or output mutation on refusal, and genuine-prefix recovery. Document the resume invariant. This is partial evaluation integrity only; DRAFT / NEVER MERGE. No Kev qualification is claimed.
Refs #4535 Holdout qualification now precommits the selected state renderer, optional model selector, endpoint digest, and entire serving declaration digest. Reject alternative renderers, model switches and server/endpoint drift before inference, and independently recheck result identities when scoring. Add falsifiable regression cases for malformed arms, zero-network refusal and rehashed result forgery. Preserve the nine-question historical status: this does not attest server identity or establish Scout runtime parity. Keep this Kev/decision-model PR DRAFT and NEVER MERGE.
|
Refs #4535 — published direct issue-owned fix Found: frozen holdouts still allowed post-hoc cherry-picking across the historical HTTP runner's three request renderers, requested model selectors, endpoint URLs and serving declarations. Individually self-consistent hashes did not bind the arm selected before inspecting the held-out predictions. Fixed: a required This prevents one class of evaluator-side holdout selection leakage, not independently verified labels, authenticated checkpoint identity, canonical Scout transcript-window rehydration, ten-question production renderer parity, actual Kev inference, task-success trials or resource contention measurement. No acceptance completion or Kev performance claim. Verification pending: new hosted CI run #42455050174 was queued with Existing sweep/PR maintenance: keep #4797 and mixed #4776 DRAFT and never merge/auto-merge or bypass via an integration branch. #4535 remains open, release #4763 remains read-only, no Cargo, credentials, infrastructure, release or production actions. |
Refs #4535 Precommit a per-request byte ceiling before any inference or output, record exact serialized input sizes and reconstruct them during qualification. Refuse forged or omitted counts and oversized frozen inputs, including bad resumable rows. Report min/p50/max/total request bytes without conflating them with token counts or claiming runtime parity. Add nonzero behavioral regressions for invalid budgets, pre-network denial, rehashed oversize results and tampered byte counts. Decision-model DRAFT, NO MERGE; real Kev inference and native Bazel verification remain pending.
Refs #4535 Assert the runner records exact transmitted JSON byte counts and refuses a tampered resumable partial before issuing another request or changing that evidence. Preserve successful interrupted-run recovery and the existing frozen-manifest refusal checks. DRAFT / NO MERGE; hosted native Bazel verification remains outstanding.
Refs #4535 Record the model name returned by the HTTP endpoint per evaluated case. Refuse a contradictory explicit model selector before recording a row; fail frozen qualification if a model name is absent, changes between holdout cases, or disagrees with its precommitted requested selector. Add behavioral regression tests for missing, malformed, switched and contradictory reported models, plus the passing control. Self-reported names are not authenticated checkpoint attestations. Decision-model work remains DRAFT and must never merge.
|
Refs #4535 — published Root cause fixed: the frozen HTTP arm pinned the requested model and operator-supplied server configuration but never recorded/checked the model reported by each response. The service could silently switch its default between held-out requests, or contradict an explicit Scope/limitations: remote model names are server self-reports, not cryptographic checkpoint attestation; this does not establish real Kev execution, actual ten-question Scout renderer parity, canonical transcript/digest export, independent labels, cold/warm contention or task-effectiveness. No new model download, credentials or production changes. Verification: prior CI on bazel test --config=ci --test_output=errors //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_testIndependent review and model/runtime experiment evidence remain required. Issue stays OPEN; release #4763 untouched. |
Refs #4535 Freeze a separate input-token ceiling and fail qualification when server usage is missing, zero, or exceeds the predeclared bound. JSON request bytes are not interchangeable with tokenizer-dependent context usage. Report measurement coverage and over-limit cases without inventing zero telemetry. Extend evaluator and CLI regressions with passing and negative controls; document server-reported token identity limitations. This is partial historical evaluation integrity, not Kev inference or runtime parity. Keep PR DRAFT / NO MERGE.
|
Refs #4535 — published Root cause fixed: qualification could pass when a server declared zero dropped tokens but omitted The manifest now freezes a separate positive Added falsifiable evaluator tests for invalid frozen limits, missing/zero/over-budget usage, forged post-hoc smaller limits, a passing measured control, and CLI nonzero exits for failures. Changed only Verification: these new tests have NOT run through Bazel. The exact-head hosted CI run #42465020702 is queued; draft-native jobs are skipped by policy. No independently reviewed canonical Scout transcript, real Kev inference, ten-question runtime parity, actual cold/warm resource measurements, or keep/replace/defer decision is claimed. Focused nonzero test command remains: bazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errors#4797 and mixed #4776: DRAFT, NEVER MERGE/AUTO-MERGE, including via integration branches. #4535 stays OPEN. Release #4763 remains read-only. Other sweep/PR owners should preserve this expected-head branch and not treat generic green CI as native qualification evidence. |
Refs #4535 — partial evidence/qualification integrity only; NOT a Kev replacement or completion.
Root cause
The historical intent HTTP runner scores nine hand-maintained categories and permits case-level threshold exploration. Its old leave-one-out rates do not prove independent session/project generalization, and there was no fail-closed path binding separately reviewed labels, project-isolated holdout cases and precommitted error tolerances to the precise HTTP experiment. Existing scorer reports p90 rather than the p95 requested for qualification. A historical report must not silently become a current-runtime replacement recommendation.
Changes
score.py --qualified MANIFEST DATASET RESULT: refuse mismatched experiment, annotation and holdout order; report exclusively frozen-threshold false-hint/positive-case recall and measured p50/p95 HTTP latency with pass/fail against frozen tolerances; keep historical output untouched.//evals/intent_engine:qualification_testwith falsifiable positive/negative, group leakage, threshold drift, tampering, exact-identity HTTP subset and no-network-refusal tests. Update usage/limitations documentation.Additional provenance fix (2026-11-07,
09915be9)run_http.py --dataset ORIGINAL.jsonnow accepts a separately frozen original dataset.score.py --qualified MANIFEST ORIGINAL.json RESULTrequires the same original bytes, checks every annotation against the complete file, and reconstructs each held-out request using the runner's own question/state renderer. Refuse dataset drift or a forged-but-rehashed request fingerprint before publishing a qualified report.Additional qualification-integrity fix (2026-11-07,
c7b6e0ee)max_unwanted_hint_rate, report all-case unwanted suggestions, missed positive cases, coverage and abstention.state_tokens_droppedor report actual truncation while a frozen-threshold score passed. Now measured zero drops on every holdout case is required to report a passing qualification; unknown truncation is never treated as zero.Additional group-provenance repair (2026-11-07,
711a18f8)project_idandsession_idclaims to define train/holdout isolation without checking the original dataset's source group identifiers. A rehashed manifest could therefore invent an independent group even when the underlying case belonged to another project/session.Frozen client-calibration integrity (2026-11-08,
1259eef2)--noul-temperaturefor client-side probability calibration. Re-running the holdout with different calibration could select an effective decision boundary post hoc despite an unchanged frozen threshold/holdout manifest.noul_temperature(nullor a positive finite value); the runner rejects a mismatch before HTTP work, and qualification refuses a recomputed result/manifest pair with inconsistent calibration. Dedicated regression cases cover invalid/missing declarations, mismatched no-network refusal, a matching calibrated run and post-hoc rehashing. Still historical nine-question evaluation, not actual Kev/runtime parity.Contiguous resume-order validation (2026-11-08,
1e27a2c6).partialcontaining the second case before the first, or only the second. It then synthesized a final result in canonical order even though the actual inference/warm-up sequence was different.Frozen HTTP arm integrity (2026-11-08,
f4494312)armnow precommits one renderer, nullable model selector, endpoint SHA-256 and complete sorted serving-config digest. The runner refuses mismatched or multiple arms before making HTTP requests; independently supplied results are checked against the same frozen arm at scoring. Rehashed endpoint/server changes cannot certify an alternative arm.Frozen HTTP request-size evidence (2026-11-08,
3f1b3322+990a88bc)max_request_bytesnow rejects oversized held-out requests before creating result files or sending model traffic. Each result records the actual serialized JSON byte count. Scoring reconstructs the exact request, rejects absent/forged counts and over-limit bodies even under self-consistently rehashed result/manifest metadata, and reports min/p50/max/total request bytes plus the frozen limit.Verification
Started from
master91c72aba7750a398dd28e83d64bf0c6d6fef4172; published tip990a88bc2299107f51a1e0905ce34039bf6246fbhas 8 eval-only files, 13 commits ahead / 0 behind (exact-head nonforce updates, no branch rewrites). No Cargo was used. A complete local checkout and Bazel are not installed in this runtime, so no local test pass or Kev inference is claimed. Earlier hosted CI completed green but native build/test jobs were skipped, so that is not proof of the new qualification tests. Previous CI completed green with native jobs skipped; previous CI has repository gates green, with native jobs skipped; previous CI was queued with native jobs skipped; previous CI passed repository gates and benchmark harness but native tests were skipped; latest CI is queued asnative=none. Hosted native CI must execute nonzerobazel test //evals/intent_engine:http_runner_test //evals/intent_engine:scoring_test //evals/intent_engine:qualification_test //evals/intent_engine:qualification_cli_test --test_output=errorsand full applicable gates. Independent review is outstanding.Deliberate limitations — do not merge
This still uses the historical nine-question renderer. It does not implement canonical Scout transcript-window rehydration/digest verification, actual current ten-question runtime parity, negative/no-candidate Scout receipts, independently verified label truth, inference on a pinned Kev host, task-success trials, cold/warm resource contention, or keep/replace/defer decisions. Reviewer identity/reference fields are declarations, not cryptographic proof. This PR is a safety-critical but partial evaluation slice, must stay DRAFT, and must never be merged or auto-merged under current Kev/decision-model hold. Issue remains open.
Separate from mixed DRAFT #4776 (which also contains a #4535 partial HTTP transport fix and is explicitly nonmergeable), and release #4763 remains read-only. No release/tag/production/infrastructure/dependency changes.