feat(api): cohort filters can exclude or isolate analysis results

[fedorov]

Summary

Closes #43. Cohort filters can now keep original data only, or analysis results only, so "NLST's original images" no longer needs SQL.

An analysis result's series carry the collection_id of the images they analyze. In NLST that is 388,254 of 592,600 series (IDC v25), and the original data's analysis_result_id is NULL, so no term value could select it.

analysis_results on CohortFilters

"include" (default), "exclude" (original data only) or "only" (analysis results only), named and valued as in idc-index's download_from_selection / idc download-from-selection --analysis-results.

{"filters": {"terms": {"collection_id": ["nlst"]}, "analysis_results": "exclude"}}

gives NLST's 204,346 original series, and the manifest endpoints take it too.

• Compiles to analysis_result_id IS NULL / IS NOT NULL on top of terms and ranges, and is echoed in filters_applied.
• Works on every filter surface: cohort/counts, cohort/manifest, cohort/manifest.txt, citations, licenses, and the MCP build_cohort, get_cohort_urls, get_citations and get_licenses. The MCP argument is a Literal, so its tool schema carries the enum.
• Not a predicate on its own. require_filter still refuses a manifest with no terms/ranges, because either side of the split is about half of IDC. The refusal says so.
• exclude + an analysis_result_id term is a 400. That combination is always empty, and a zero would read as "no such data".
• Download payload. For a collection-only filter, the payload used to suggest idc download nlst, which fetches the analysis results too. With exclude/only it now suggests idc download-from-selection --collection-id nlst --analysis-results exclude.

Download command for a whole analysis result

A filter of only analysis_result_id values now gets idc download <analysis_result_id> (resolved by idc-index 0.13.2+ across every collection the analysis result spans), as a filter of only collection_id values already got idc download <collection>. Before, it got the manifest route alone, worded "Or save…". The shape is read from the applied filter, and values are shlex-quoted because they land in a shell command.

null_count in get_attribute_values

Series with no value are not in values. Now null_count reports them, so analysis_result_id no longer reads as fully enumerated: 570,108 NULLs next to truncated: false. The analysis_result_id attribute note points from the term filter to analysis_results.

Why not source_type

The issue preferred a source_type term (original_data / analysis_result). A mode field does the same job and matches the name idc-index users already know. It also needs no synthetic column in the attribute allow-list, since source_type is not an index column. The issue's third acceptance item, the analysis_result_id description, came in with idc-index-data 25.0.4 (#66).

Text manifest pinned as an idc-index input format

This is a separate commit and doesn't depend on the rest. idc download and idc download-from-manifest read each line of manifest.txt / get_cohort_urls as a single CSV column, so one added field crashes them with ValueError: Length mismatch. Until now nothing recorded this. #64's dropped commit widened the JSON rows, which are safe, but nothing would have stopped the same change landing on the text lines.

• Architecture invariant ci: multi-tier (dev/test/prod) Cloud Run deploys via GitHub Environments #8 in CLAUDE.md and dev/developer_guide.md, plus a docstring where the URLs are produced.
• tests/test_manifest_contract.py. It uses the two smallest series from each AWS bucket, so source=gcs remaps some of them and leaves the rest unchanged. For aws and gcs it checks two things:
  • every manifest.txt line is exactly s3://<bucket>/<crdc_series_uuid>/*, and get_cohort_urls returns the same lines;
  • IDCClient.download_from_manifest, the path both CLI commands take, resolves every line to an IDC series, the right endpoint and the expected total size. Only _s5cmd_run is stubbed, as idc-index's own tests do.
• Checked that the tests fail. Appending ,5 to each line makes all four fail: two on the format check, two with idc-index's own Length mismatch.

Changes

• core/models.py: the AnalysisResultsMode literal, CohortFilters.analysis_results and AttributeValues.null_count
• core/filters.py: compiling the new field, the contradiction check, and the require_filter wording
• core/services/discovery.py: the null_count query
• core/services/manifest.py: the split download command, the analysis-result command, quoting
• core/schema.py: the analysis_result_id note
• mcp/server.py: the new tool arguments and docstrings, and idc://guide
• rest/app.py: docstrings, a counts example, and a top-level analysis_results is now named as a stray key
• docs/user-guide.md, CHANGELOG.md

INSTRUCTIONS is unchanged: the guidance is in the build_cohort description and idc://guide.

Test plan

• tests/test_analysis_results_filter.py (new):
  • in rider_pilot, include = exclude + only (774 = 194 + 580), and exclude matches the SQL IS NULL count
  • REST and MCP agree, and manifest.txt line count equals the exclude count
  • the download command form
  • the contradiction and lone-mode refusals on REST and MCP
  • an unknown mode is rejected
  • a top-level key names the fix
  • null_count plus the listed values equals all series
  • the one-line analysis-result command, and shell quoting
• tests/test_manifest_contract.py (see above)
• uv run pytest tests -q: 126 passed. ruff check, ruff format --check and bandit are clean.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

Static review identified no blocking issues in the shared filtering logic, adapter integration, or regression coverage; tests were not executed.

No open issues 🎉

What changed in this PR

Adds original-data-only and analysis-results-only cohort filtering across REST and MCP, addressing #43 without requiring SQL.

Changes:

• Adds analysis_results modes, validation, and matching download commands.
• Exposes null_count in attribute discovery.
• Updates documentation and regression tests.

File	Description
tests/​test_filter_shape.py	Updates the default filter echo expectation.
tests/​test_analysis_results_filter.py	Tests filtering, validation, parity, downloads, and NULL counts.
src/​idc_api/​rest/​app.py	Updates request validation, examples, and descriptions.
src/​idc_api/​mcp/​server.py	Exposes filter modes and updates agent guidance.
src/​idc_api/​core/​services/​manifest.py	Generates mode-aware download commands.
src/​idc_api/​core/​services/​discovery.py	Counts unset attribute values.
src/​idc_api/​core/​schema.py	Explains how to select original data.
src/​idc_api/​core/​models.py	Defines filter modes and null_count.
src/​idc_api/​core/​filters.py	Compiles modes and enforces filter safeguards.
docs/​user-guide.md	Documents filtering and download behavior.
CHANGELOG.md	Records user-visible additions.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Static review identified no blocking issues, and regression coverage addresses filtering, adapter parity, and manifest compatibility; tests were not executed here.

No open issues 🎉

_{Review effort: Balanced}