Skip to content

fix(judging): accept every valid answer and regenerate the excluded test cases - #26

Merged
RyPope merged 12 commits into
mainfrom
fix/judge-multi-answer
Oct 31, 2026
Merged

RyPope merged 12 commits into
mainfrom
fix/judge-multi-answer

Conversation

@RyPope

@RyPope RyPope commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Stacked on #25.

Problem

Correct solutions were marked wrong in two ways, and the list meant to hide bad test data was hiding one of them:

  1. Problems with several valid answers were judged against one. About 75 problems say "return any valid answer" (course orders, Gray codes, reorganised strings, balanced BSTs, CIDR covers, …), and "in any order" only applied to the outer list. For example find-the-difference-of-two-arrays rejected [[3,7],[101,8,9]].
  2. Invalid test inputs slipped through. The generated dataset has inputs that break their problem's constraints. A case was excluded only when both the Java and C++ references failed it, and C++ often doesn't crash on out-of-bounds input.

Judging

  • New comparison modes: any_order_deep (order free at every level), each_any_order (fixed slots, unordered contents), and checker:<slug>. judge() now receives the case's inputs from all four callers.
  • 75 answer checkers in judging/checkers/. Each validates an answer against the inputs, and uses the recorded answer only for optimal values such as a minimum length.
    • Unit tests: every checker accepts the statement's example and a different valid answer, and rejects invalid ones.
    • make check-checkers: every checker accepts the dataset's recorded answer on every non-excluded case (75/75).
  • Curated modes: judging/answer_modes.py holds the problems whose any-order rule isn't stated in the usual words.
  • Runner: Java submissions now import java.text. The reference for minimize-rounding-error-to-meet-target failed to compile on every case, and so would any user's DecimalFormat.

Excluded test cases

validate_cases now excludes a case in any of these situations:

  • every trusted reference (passing ≥50% of the suite) fails it, as before;
  • one confident reference (≥90%) fails it;
  • all references fail exactly the same cases and at least 3 cases survive;
  • its recorded answer fails the problem's checker.

It also reports two lists for a person to act on:

  • order-only differences: these need a compare mode;
  • exclusions resting only on agreeing references: a person should confirm these.

--from-sources builds an unfiltered database first, so make validate-cases regenerates from scratch. Before, it validated the already-filtered data and could never rediscover existing exclusions.

Result

  • Before: 1,314 cases in 220 problems. After: 2,086 cases in 308 problems, out of about 275k.
  • No longer excluded: 135 cases in 13 problems, because they were correct answers the old judge rejected. For example, all 42 in find-the-difference-of-two-arrays.
  • Checked by hand: the 9 problems where only reference agreement drove exclusions. They are regions-cut-by-slashes, recover-a-tree-from-preorder-traversal, count-of-integers, count-unhappy-friends, mean-of-array-after-removing-some-elements and four checker problems. In every one, the inputs really break the constraints: non-square grids, 4-child binary-tree nodes, 31-digit numbers where 22 are allowed, and similar.

Verification

  • Tests and lint: make lint passes, and make test passes (backend 373, web 17, runner 62).
  • Full rebuild: make validate-cases regenerated the list in 46 minutes with three runners.
  • On a live stack after re-import: make smoke passes. For course-schedule-ii, a different valid order ([0,2,1,3] against [0,1,2,3]) is Accepted and an invalid order gets Wrong Answer.
  • Independent review: a fresh reviewer read the whole judging change. I fixed its findings, including float tolerance in unordered comparison and non-canonical CIDR blocks, and added the review report for agreement-only exclusions.

Follow-ups (not in this PR)

  • Expected-answer display: the Run panel still shows the recorded answer as "expected" when a different valid answer passes. A short note there would avoid confusion.
  • find-a-corresponding-node-of-a-binary-tree-in-a-clone-of-that-tree: its target parameter is a node reference, which the harness can't represent in C++.
  • Transient failures: a reference that fails transiently under load (seen once for Java) scores 0, so it can't wrongly exclude cases. Timeouts are still treated as failures.

RyPope added 12 commits October 31, 2026 10:57
A case was excluded only when every trusted typed reference failed it, so invalid inputs that
crash Java but happen to pass in C++ (out-of-bounds reads) stayed in and rejected correct
solutions. A reference that passes at least 90% of a suite now excludes on its own; less
confident references still have to agree.
Adds any_order_deep (order free at every level) and each_any_order (fixed slots, unordered
contents) comparison modes, and per-problem answer checkers that validate an answer against
the case's inputs. The importer assigns modes from the checker registry and a curated table;
judge now receives the inputs from every caller. check_checkers verifies each checker accepts
the dataset's own expected answers.
These are judge-side problems (an unstated 'any order'), not invalid inputs: the cases count
as passes so they are not excluded, and the problems are listed in <out>.order.json for a
curated compare mode.
…wers

Each checker validates an answer against the case's inputs (and the reference answer for
optima such as a minimum length) instead of requiring the reference's exact output: course
schedules, gray codes, reorganised strings, balanced BSTs, CIDR covers, grid layouts and more.
Every checker accepts the statement's example and an alternative answer and rejects invalid
ones in unit tests, and accepts the dataset's expected answers on every valid case
(check_checkers; the remaining rejections are inputs that break their problem's constraints).
make validate-cases validated the already-filtered content.db and resumed from the old output,
so it could never rediscover existing exclusions. --from-sources builds an unfiltered database
first. Adds make check-checkers and documents answer checking.
An invalid input can look like a reordering, and a checker's rejection is never about order,
so order-only differences now fail like any other and are only reported for review (a re-run
replaces a problem's entry). Reviewing the first report adds a checker for
find-k-pairs-with-smallest-sums (ties allow other pairs) and any-order modes for
concatenated-words and high-access-employees.
…t fresh from sources

In suites where most inputs break the constraints every reference falls below the trusted bar,
so nothing was excluded. When at least two references fail exactly the same cases, those are
excluded as long as three cases survive. --from-sources now starts a fresh list (resuming against
a rebuilt database is meaningless), and make validate-cases rebuilds the image first.
…l CIDR blocks

Unordered lists were sorted by their exact JSON text, so floats within the tolerance could sort
apart and fail. ip-to-cidr accepted a block whose base address had host bits set.
References are often translations of one algorithm and can share a bug or limit, so when cases are
excluded although no reference passes half the suite, the problem goes into <out>.review.json for a
person to confirm. make validate-cases prints both review lists.
Java solutions may use DecimalFormat and friends without an import, as on other judges; the
reference for minimize-rounding-error-to-meet-target failed to compile on every case.
…hecker

Such cases have no valid answer (or a wrong record), so no submission can pass them. This also
covers problems with no runnable typed reference, such as valid-arrangement-of-pairs.
Rebuilt from unfiltered suites (make validate-cases) with answer checkers and the new exclusion
rules: 2,086 cases in 308 problems, against 1,314 in 220 before. 135 cases in 13 problems are
no longer excluded because they were correct answers the old judge rejected (for example all 42
for find-the-difference-of-two-arrays). The rest of the change is inputs that break their
problem's constraints and were previously kept; exclusions resting only on agreeing references
(9 problems) were checked by hand.
@RyPope
RyPope changed the base branch from chore/judge-versions to main October 31, 2026 15:57
@RyPope
RyPope force-pushed the fix/judge-multi-answer branch from fe265cb to 117c5a5 Compare October 31, 2026 15:57
@RyPope
RyPope merged commit 46b65e8 into main Oct 31, 2026
5 checks passed
@RyPope
RyPope deleted the fix/judge-multi-answer branch October 31, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant