Repository navigation
fix(judging): accept every valid answer and regenerate the excluded test cases - #26
Merged
Merged
Conversation
A case was excluded only when every trusted typed reference failed it, so invalid inputs that crash Java but happen to pass in C++ (out-of-bounds reads) stayed in and rejected correct solutions. A reference that passes at least 90% of a suite now excludes on its own; less confident references still have to agree.
Adds any_order_deep (order free at every level) and each_any_order (fixed slots, unordered contents) comparison modes, and per-problem answer checkers that validate an answer against the case's inputs. The importer assigns modes from the checker registry and a curated table; judge now receives the inputs from every caller. check_checkers verifies each checker accepts the dataset's own expected answers.
These are judge-side problems (an unstated 'any order'), not invalid inputs: the cases count as passes so they are not excluded, and the problems are listed in <out>.order.json for a curated compare mode.
…wers Each checker validates an answer against the case's inputs (and the reference answer for optima such as a minimum length) instead of requiring the reference's exact output: course schedules, gray codes, reorganised strings, balanced BSTs, CIDR covers, grid layouts and more. Every checker accepts the statement's example and an alternative answer and rejects invalid ones in unit tests, and accepts the dataset's expected answers on every valid case (check_checkers; the remaining rejections are inputs that break their problem's constraints).
make validate-cases validated the already-filtered content.db and resumed from the old output, so it could never rediscover existing exclusions. --from-sources builds an unfiltered database first. Adds make check-checkers and documents answer checking.
An invalid input can look like a reordering, and a checker's rejection is never about order, so order-only differences now fail like any other and are only reported for review (a re-run replaces a problem's entry). Reviewing the first report adds a checker for find-k-pairs-with-smallest-sums (ties allow other pairs) and any-order modes for concatenated-words and high-access-employees.
…t fresh from sources In suites where most inputs break the constraints every reference falls below the trusted bar, so nothing was excluded. When at least two references fail exactly the same cases, those are excluded as long as three cases survive. --from-sources now starts a fresh list (resuming against a rebuilt database is meaningless), and make validate-cases rebuilds the image first.
…l CIDR blocks Unordered lists were sorted by their exact JSON text, so floats within the tolerance could sort apart and fail. ip-to-cidr accepted a block whose base address had host bits set.
References are often translations of one algorithm and can share a bug or limit, so when cases are excluded although no reference passes half the suite, the problem goes into <out>.review.json for a person to confirm. make validate-cases prints both review lists.
Java solutions may use DecimalFormat and friends without an import, as on other judges; the reference for minimize-rounding-error-to-meet-target failed to compile on every case.
…hecker Such cases have no valid answer (or a wrong record), so no submission can pass them. This also covers problems with no runnable typed reference, such as valid-arrangement-of-pairs.
Rebuilt from unfiltered suites (make validate-cases) with answer checkers and the new exclusion rules: 2,086 cases in 308 problems, against 1,314 in 220 before. 135 cases in 13 problems are no longer excluded because they were correct answers the old judge rejected (for example all 42 for find-the-difference-of-two-arrays). The rest of the change is inputs that break their problem's constraints and were previously kept; exclusions resting only on agreeing references (9 problems) were checked by hand.
RyPope
force-pushed
the
fix/judge-multi-answer
branch
from
October 31, 2026 15:57
fe265cb to
117c5a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #25.
Problem
Correct solutions were marked wrong in two ways, and the list meant to hide bad test data was hiding one of them:
find-the-difference-of-two-arraysrejected[[3,7],[101,8,9]].Judging
any_order_deep(order free at every level),each_any_order(fixed slots, unordered contents), andchecker:<slug>.judge()now receives the case's inputs from all four callers.judging/checkers/. Each validates an answer against the inputs, and uses the recorded answer only for optimal values such as a minimum length.make check-checkers: every checker accepts the dataset's recorded answer on every non-excluded case (75/75).judging/answer_modes.pyholds the problems whose any-order rule isn't stated in the usual words.java.text. The reference forminimize-rounding-error-to-meet-targetfailed to compile on every case, and so would any user'sDecimalFormat.Excluded test cases
validate_casesnow excludes a case in any of these situations:It also reports two lists for a person to act on:
--from-sourcesbuilds an unfiltered database first, somake validate-casesregenerates from scratch. Before, it validated the already-filtered data and could never rediscover existing exclusions.Result
find-the-difference-of-two-arrays.regions-cut-by-slashes,recover-a-tree-from-preorder-traversal,count-of-integers,count-unhappy-friends,mean-of-array-after-removing-some-elementsand four checker problems. In every one, the inputs really break the constraints: non-square grids, 4-child binary-tree nodes, 31-digit numbers where 22 are allowed, and similar.Verification
make lintpasses, andmake testpasses (backend 373, web 17, runner 62).make validate-casesregenerated the list in 46 minutes with three runners.make smokepasses. For course-schedule-ii, a different valid order ([0,2,1,3]against[0,1,2,3]) is Accepted and an invalid order gets Wrong Answer.Follow-ups (not in this PR)
find-a-corresponding-node-of-a-binary-tree-in-a-clone-of-that-tree: itstargetparameter is a node reference, which the harness can't represent in C++.