[solarsys]

Stacked on #1320, then #1319, #1318, #1317 and #1316. Review the commits from autoimprove: literature knowledge on.

Why

The feature space is large: operators × code groups × windows × thresholds. Published predictors are the best guide to what to try first. A clinician also wants each feature's rationale backed by evidence, not by the LLM's memory. LLMs invent references, though, so every piece of evidence has to be checked.

What changes

New pyhealth/autoimprove/knowledge.py:

• Sources:

  • Curated tables (add_table_evidence): one row per predictor and effect. Effects are parsed from text such as RR 1.94 (1.61–2.32) or OR 2.0–4.1.
  • Documents: tables of abstracts, text files, and PDFs when pypdf is installed.
  • PubMed through E-utilities: off unless called, cached, with queries built from the task card's population and outcome only. Every query is kept in kb.queries for audit.

• extract(). The LLM returns records with the exact sentence that states each effect. Each record's source is set from the document, never by the LLM.

• verify() keeps a record only if:

  • its source was gathered;
  • its quote is in that source;
  • its effect size and CI appear in the quote.

  An invented citation, a misquote or a made-up number is rejected by construction. The direction of a ratio effect is set from its value.

• map(fs, card). The LLM sees only the task card, the feature menu and column names. Each predictor becomes one of:

  • have: an existing column;
  • build: a spec, validated by the FeatureSpace, carrying the literature's citations and direction;
  • not in data: a data request.

  Each gets a prior of strength × population match × data support. The prior only orders candidates; the feature search decides what is kept.

• Outputs:

  • directions(), which is ambiguous when sources disagree (e.g. controller use);
  • ranked_specs(), for QueueProposer or the LLM;
  • gap_report();
  • benchmark_range();
  • save() / load().

• AutoImprover(knowledge=kb). Feature cycles gain an "evidence" strategy, which shows the LLM the untried published predictors with their citations and priors. The report gains a Literature section (the citation-check counts, the gap report and the published performance range) and a citations column in Features.

Fixes from the first real run (ARCHES pediatric asthma)

The ARCHES experiment (v2 proposal PR 5) built a knowledge base from 16 curated rows, 22 review abstracts and 64 reference abstracts, then ran four feature-search arms. Running it turned up these issues, each fixed in its own commit with a test:

• fetch_pubmed(pmids) adds a review's references by id; only the ids are sent.
• The citation check rejected real records. A quoted MD -0.37 or AUC 84% failed, because numbers were matched without sign and percentages were not read as proportions. This affected 7 of 101 records.
• Spreadsheet ids. PMIDs exported from Excel as =HYPERLINK(url, "id") were used whole as document ids.
• Derived events are now computed only when a spec selects a derived type, not for every table.
• Report: a kept baseline was misdescribed. When candidates passed the search bound but none was confirmed, the report said the kept baseline "beat the starting pipeline". It now says how many trials passed the search and that none kept its gain on the confirmation folds.
• Repeated feature proposals. LLM feature turns were wasted repeating features tried more than 12 trials earlier, which the prompt no longer showed. The prompt now lists every feature tried, the retry names the repeats, and trial notes are no longer labelled as space clipping.
• Plain any() definitions. '1 if either 1 if there was any X …, else 0 or … is above 0, else 0' now reads '1 if there was any X … or there was any Y …, else 0'. Selected features' definitions go to clinician review.

Tests

tests/core/test_autoimprove_knowledge.py (10 tests):

• Citation check: a real record is kept, and its wrong direction is corrected from the value. An invented source, a misquote, a wrong effect size and a wrong CI are each rejected with the reason.
• Extraction: it cannot choose its own source. A citation the LLM supplies is ignored, and a fabricated record is rejected.
• Curated tables: a risk-factor table is parsed and verified.
• PubMed: every outgoing query contains only task-card words and fixed operators.
• Mapping: have, build, a refused spec (moved to not in data with the reason) and not in data. Conflicting sources give an ambiguous direction. Save and load round-trip.
• Number matching and ids: negative effects and percentages are found; a near miss is still rejected; HYPERLINK cells and 33831622.0 give clean PMIDs; fetching by id sends only the ids.
• End to end: a search driven by QueueProposer(kb.ranked_specs()) keeps the planted feature, and the report shows the literature and citations.

All 65 autoimprove and metrics tests pass, as do the doctests, ruff and tools/check_pr_rules.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ntMLEmFsQEQJKTmPGGw4T

[github-actions]

Test Coverage

Total: 71.06% (20085 / 28266 lines)

Modified file	Coverage	Lines
pyhealth/autoimprove/__init__.py	100.00%	9 / 9
pyhealth/autoimprove/evaluate.py	92.57%	162 / 175
pyhealth/autoimprove/explain.py	89.16%	181 / 203
pyhealth/autoimprove/features.py	88.67%	532 / 600
pyhealth/autoimprove/improver.py	91.79%	816 / 889
pyhealth/autoimprove/knowledge.py	86.68%	319 / 368
pyhealth/autoimprove/proposers.py	90.96%	171 / 188
pyhealth/autoimprove/report.py	94.66%	585 / 618
pyhealth/autoimprove/space.py	97.40%	187 / 192
pyhealth/metrics/__init__.py	100.00%	12 / 12
pyhealth/metrics/clinical.py	100.00%	94 / 94
pyhealth/tasks/__init__.py	88.57%	31 / 35
pyhealth/tasks/base_task.py	83.33%	30 / 36
pyhealth/tasks/task_card.py	98.04%	50 / 51

Workflow run · ff6447d

[jhnwu3]

/review

[github-actions]

AI Review

suggestion: human review
decision: not available

This PR is too large for AI review (over 300 files or 200000 diff characters), so it is labeled status: manual and needs a human review.