fix(judging): accept every valid answer and regenerate the excluded test cases

CONTRIBUTING.md

@@ -61,7 +61,7 @@ New problems, extra solutions and corrected test cases go in [`content/`](conten
 - **Write solutions** for problems the upstream data has few of (Kotlin especially), in `content/`.
 - **Triage wrong verdicts.** Reproduce a reported case and, if the test case itself is invalid, add it to the exclusion list.
 - **Kotlin reference solutions.** The upstream dataset has very few, so the Solutions tab falls back to Java. Contributing Kotlin solutions upstream to [doocs/leetcode](https://github.com/doocs/leetcode) helps everyone.
-- **Special judges** for problems that accept several valid answers (for example "return any valid arrangement").
+- **Answer checkers** for problems that accept several valid answers and don't have one yet. A checker is a small function in `backend/localcode/judging/checkers/` that decides whether an answer is valid for the inputs; see the existing ones and their tests, and run `make check-checkers`.
 - **Accessibility and keyboard navigation** improvements in the web app.
 - **Docs**: troubleshooting tips for platforms we haven't tested (Windows, Linux distributions, low-memory machines).
 
@@ -75,6 +75,7 @@ New problems, extra solutions and corrected test cases go in [`content/`](conten
 
 ## Maintainer tasks
 
-- **Regenerate the test-case exclusion list** after a dataset update: `make validate-cases` (about 45 minutes with three runner replicas), then commit `backend/localcode/data/excluded_cases.json`.
+- **Regenerate the test-case exclusion list** after a dataset update or a judging change: `make validate-cases` (about an hour with three runner replicas), then commit `backend/localcode/data/excluded_cases.json`. It also prints two lists to act on: problems whose references differ from the expected answers only in list order (give those a mode in `backend/localcode/judging/answer_modes.py` and re-run them with `--only`), and problems whose exclusions rest on the references agreeing although none passes half the suite (check that those inputs really break the problem's constraints).
+- **Check the answer checkers** after a dataset update: `make check-checkers`. Every checker must accept the dataset's expected answers on every non-excluded case.
 - **Check reference pass rates** against a scratch instance: `backend/.venv/bin/python -m localcode.tools.sample_refs -n 100`.
 - **Release:** move the `Unreleased` section of [CHANGELOG.md](CHANGELOG.md) under a version heading, tag `vX.Y.Z`, and publish a GitHub release.

Makefile

@@ -63,10 +63,18 @@ update: ## Re-download sources and rebuild problem data (your progress is kept)
 validate-content: ## Check content/ against the running stack (FULL=1 also runs local solutions on upstream tests)
 	docker compose exec -T -u localcode app python -m localcode.tools.validate_content $(if $(FULL),--full)
 
-validate-cases: ## Maintainers: regenerate the excluded test-case list (slow)
-	docker compose up -d --scale runner=3
-	docker compose exec -T -u localcode app python -m localcode.tools.validate_cases /data/excluded_cases.json
-	docker compose cp app:/data/excluded_cases.json backend/localcode/data/excluded_cases.json
+validate-cases: ## Maintainers: regenerate the excluded test-case list from the unfiltered sources (slow)
+	docker compose up -d --build --scale runner=3
+	docker compose exec -T -u localcode app python -m localcode.tools.validate_cases \
+		--from-sources /data/validate /data/validate/excluded_cases.json
+	docker compose cp app:/data/validate/excluded_cases.json backend/localcode/data/excluded_cases.json
+	@echo "Order-only differences (give these a compare mode):"
+	@docker compose exec -T -u localcode app cat /data/validate/excluded_cases.order.json
+	@echo "Exclusions to confirm by hand (no reference passed half the suite):"
+	@docker compose exec -T -u localcode app cat /data/validate/excluded_cases.review.json
 	docker compose up -d --scale runner=1
 
-.PHONY: help up down logs setup test test-backend test-runner test-web lint format dev-web api-types screenshots smoke update validate-content validate-cases
+check-checkers: ## Maintainers: every answer checker must accept the dataset's expected answers
+	docker compose exec -T -u localcode app python -m localcode.tools.check_checkers
+
+.PHONY: help up down logs setup test test-backend test-runner test-web lint format dev-web api-types screenshots smoke update validate-content validate-cases check-checkers

README.md

@@ -129,7 +129,7 @@ flowchart LR
     I --> F["filter invalid test cases"] --> DB[(content.db)]
 ```
 
-The upstream test cases were generated by running Python solutions on synthetic inputs, and a few of them break the problems' own constraints (for example 64-bit values where a problem promises 32-bit ones). The importer removes cases the typed languages can't represent. It also applies an [exclusion list](backend/localcode/data/excluded_cases.json) of about 1,300 cases that the Java and C++ reference solutions reject, produced by running them over every suite.
+The upstream test cases were generated by running Python solutions on synthetic inputs, and a few of them break the problems' own constraints (for example 64-bit values where a problem promises 32-bit ones). The importer removes cases the typed languages can't represent. It also applies an [exclusion list](backend/localcode/data/excluded_cases.json) of cases that the Java and C++ reference solutions reject, produced by running them over every suite with the same answer checking users get.
 
 See [NOTICE.md](NOTICE.md) for attribution.
 
@@ -160,7 +160,7 @@ docker compose cp app:/data/user.db ./localcode-user-backup.db
 ## Known limitations
 
 - **Kotlin reference solutions are sparse upstream**, so the Solutions tab shows Java for most Kotlin problems. Kotlin submissions are judged normally.
-- **Problems with several valid answers** ("return any valid arrangement") are compared against a single expected answer, except where the statement says "in any order", which is compared ignoring order.
+- **Problems with several valid answers** are judged by checking the answer, not by matching one expected output: 74 problems ("return any valid arrangement", course orders, Gray codes, CIDR covers, …) have a checker that validates the answer against the input, and "in any order" answers are compared ignoring order (at every level where the problem allows it). A multi-answer problem without a checker yet is still compared against one expected answer; please report it.
 - **Design, SQL, shell and concurrency problems** can be read but not run.
 - **C++ is compiled with Clang 19 and `-std=c++23 -O2`**, as on LeetCode, but without AddressSanitizer, so some out-of-bounds bugs that LeetCode would report may go unnoticed.
 - **Cold-start compile times** are about 1 to 2 s for Java and C++ and about 4 s for Kotlin.