Prepare the Python-only CopraRNA 3.0.0 release

.github/workflows/python.yml

@@ -1,4 +1,4 @@
-name: Python migration
+name: CopraRNA 3
 on:
   push:
   pull_request:
@@ -35,7 +35,7 @@ jobs:
           /tmp/coprarna-wheel/bin/python -m pip install --force-reinstall --no-deps dist/*.whl
           cd /tmp
           /tmp/coprarna-wheel/bin/coprarna --version
-          /tmp/coprarna-wheel/bin/coprarna status
+          /tmp/coprarna-wheel/bin/coprarna helpers
           /tmp/coprarna-wheel/bin/coprarna helper archive-readme
 
   scientific:
@@ -57,4 +57,5 @@ jobs:
         env:
           COPRARNA_PIPELINE_TESTS: '1'
           COPRARNA_BROWSER_TESTS: '1'
+          COPRARNA_REFERENCE_TESTS: '1'
         run: python -m pytest -q

.gitignore

@@ -1 +1,9 @@
 /.project
+
+__pycache__/
+*.py[cod]
+.pytest_cache/
+build/
+dist/
+*.egg-info/
+core_genome_archive/metadata/*.lock

AGENTS.md

@@ -1,91 +1,67 @@
-# CopraRNA2 architecture and review ledger
+# CopraRNA 3 architecture and release policy
 
-Issue: https://github.com/BackofenLab/CopraRNA/issues/1.
-Rewrite PR: https://github.com/BackofenLab/CopraRNA/pull/2.
-Follow-up PR: https://github.com/BackofenLab/CopraRNA/pull/3.
-
-## Current scope
-
-Martin's review (https://github.com/BackofenLab/CopraRNA/pull/3#issuecomment-6324702688)
-changes the follow-up to a CopraRNA2-only runtime. Its requested behavior changes
-supersede historical whole-workflow parity. Remove retired-mode code, tests and
-user documentation from the new implementation. Keep every original tracked
-legacy source byte unchanged for historical comparison; never invoke it from
-production. Keep substantial review changes in separate commits.
+[Issue #4](https://github.com/BackofenLab/CopraRNA/issues/4) prepares version 3.0.0
+from `issue-1-python3-bugfixes`. Its explicit removal of the 2.1 code base supersedes
+the earlier instruction to retain original source files. Historical sources remain
+in Git history; production and tests must never invoke them. Keep substantial
+review changes in separate commits.
 
 ## Architecture
 
-- `pipeline.py`, `cli.py`: prediction configuration, isolated batches and cleanup.
-- `homology.py`, `genome.py`, `ncbi.py`, `annotation.py`: genome retrieval, shared
-  compressed genome database, streaming GenBank parsing and homology clustering.
-- `intarna.py`: IntaRNA >=3 / IntaRNAsTar, file or inline parameter configuration,
+- `pipeline.py`, `cli.py`: configuration, isolated batches, cleanup and genome CLI.
+- `organisms.py`: on-demand NCBI assembly resolution, versioned metadata, cache
+  validation and duplicate-assembly rejection; no static organism allow-list.
+- `homology.py`, `genome.py`, `ncbi.py`, `annotation.py`: shared compressed genomes,
+  streaming GenBank parsing, short job-local native species codes and clustering.
+- `intarna.py`, `parameters.py`: IntaRNA >=3 / IntaRNAsTar, native parameter files,
   gzipped interactions and removal of near-full-length perfect helices.
-- `energy.py`, `combination.py`, `numerics.py`: validated GEV and comparative
-  statistical algorithms. Pin the validated NumPy/SciPy and BLAS reduction kernel.
+- `energy.py`, `combination.py`, `numerics.py`, `statistics.py`: validated GEV and
+  comparative statistics; preserve the validated NumPy/SciPy and BLAS kernel pins.
 - `conservation.py`, `alignments.py`: consensus matrices and Jalview exports.
 - `plots.py`, `phylogeny.py`: PDF/SVG scientific figures and tree rendering.
 - `david.py`, `enrichment.py`, `webserver.py`: optional service and web exports.
-- `io.py`: streaming compressed/plain file access shared by producers/consumers.
-- Adjacent `.md` mapping documents and `migration.json` distinguish active and
-  intentionally retired scripts; no retired helper is advertised as implemented.
-
-## Code documentation
-
-Document each new or changed public function with its purpose, input/output
-contract, file side effects and failure behavior where relevant. For scientific
-algorithms, explain coordinate systems, matrix dimensions, statistical conventions
-and differences deliberately retained for oracle comparisons. Add inline comments
-for non-obvious numerical choices, native-tool interface constraints and cache
-publication/lifetime rules. Keep documentation accurate when changing behavior;
-do not add comments that merely restate individual statements. Record release-facing
-changes and their rationale in `ChangeLog.md`.
-
-## Validation policy
-
-Keep exact captured CopraRNA2 helper/statistical oracles where behavior is retained.
-Test requested changes against explicit contracts and real native integrations.
-Do not regenerate old numerical expectations from the translated implementation.
-R-data objects are compared semantically; original sources have checksum tests.
-SVG/PDF are primary output formats; no PostScript/Ghostscript dependency remains.
-Preserve the user's original input FASTA, including when its name is input_sRNA.fa.
-Keep downloaded genomes compressed and reuse shared database paths directly.
-
-## Progress
-
-- Previous rewrite and fixes were delivered with CI green on Python 3.11–3.13,
-  exact statistical/report fixtures, 51 region comparisons on 17 reference genomes,
-  native workflows, browser rendering and installed-wheel checks.
-- Current review: current-only runtime, native parameter files, gzip/cache support,
-  helix filtering, vector reports, documentation and current dependencies implemented.
-  Fresh environment, retained numerical fixtures and native workflows pass; final
-  release checks and CI are recorded in the follow-up PR.
-- Live external service availability is not certified. Offline adapters retain
-  request tests. New dependencies/UI versions must be checked before pin updates.
-
-
-## Completeness audit — 2026-10-22
-
-The original issue and latest review were checked together; the current review
-supersedes historical defaults only where explicitly requested. Original sources
-remain subject to the checksum test. No additional inline review requests exist.
-
-| Requested behavior | Implementation and verification |
-| --- | --- |
-| Current comparative runtime only | `Config`, helper registry and `migration.json`; retired mode/fixtures removed, 39 active and 2 retired mappings verified by `test_inventory.py` and `test_review.py`. |
-| IntaRNA >=3 and sTar default | `parameters.check_version`, native homology/prediction/web entry points; old releases rejected before side effects, plus real native workflow tests. |
-| File/inline native parameters | `parameters.parameter_file`/`arguments`; mutual exclusion, exact snapshots, temporary-file lifetime and native file/inline equivalence tests. |
-| Native CSV and no duplicate CLI settings | `intarna.py`, `webserver.py`, `cli.py`; gzip CSV parsing and retired CLI flag checks. |
-| Current dependency interfaces | Fresh `environment.yml`, native BLAST+/DomClust integrations, exact retained numerical fixtures, Python 3.11–3.13 CI and separate optional oracle environments. |
-| Compressed intermediates/genomes | `io.py`, `ncbi.py`, native `.csv.gz` producers; streamed HTTP gzip/plain handling, atomic truncation rejection, direct cache reuse and concurrent-access tests. |
-| Python tree SVG | `phylogeny.render_tree`; SVG vector-path contract and native output checks. |
-| PDF/SVG diagrams, no PostScript | Python plotting and enrichment; matching vector files, all extended pages, no embedded raster images and browser adapter tests. |
-| Original FASTA preservation | Working copy, output-name/symlink collision preflight and guarded cleanup sources/destinations; failure-path and complete native tests. |
-| Shared compressed genome database | `--genome-db`, manifest and per-accession locking; two real jobs share one cache without copying genomes. |
-| Remove near-full-length perfect helices | `filter_perfect_helices` before GEV and web selection; boundary/loop cases, native helix and discarded-best-hit report regressions. |
-| Function/inline documentation | Public module API docstrings, numerical/interface comments and directives above. |
-| Rewrite and fix ChangeLog | `ChangeLog.md` identifies source version, architecture and each fix/change; included in sdist. |
-| Original issue CLI/package/mappings | Separate rewrite and fix PRs, isolated batches, per-script sidecars, wheel/sdist and installed native prediction checks. |
-
-When adding outputs, update `io._JOB_OUTPUTS` and preserve both original inputs
-and any cleanup destination that resolves to them. Keep every independent native
-entry point's version check and detected-version cache signature in place.
+- `io.py`: shared compressed/plain I/O and protected input/output namespaces.
+- `helpers.py`: active Python helper contracts; no original-script aliases or
+  defect-emulation mode. `resources.py` provides current report resources only.
+
+## Documentation
+
+Document each new/changed public function with its purpose, input/output contract,
+file side effects and failure behavior where relevant. For scientific algorithms,
+explain coordinate systems, matrix dimensions, statistical conventions and retained
+oracle differences. Comment non-obvious numerical choices, native interfaces and
+cache publication/lifetime rules. Record release-facing changes and rationale in
+`ChangeLog.md`; keep `README.md` consistent with CLI and packaging.
+
+## Validation
+
+Retain exact independently captured scientific fixtures for unchanged behavior.
+Their provenance and checksums live in `tests/fixtures`; do not regenerate expected
+numbers from the Python implementation. Compare R-data objects semantically.
+Test changed behavior against explicit contracts and real native integrations.
+No legacy source/interpreter environments or generated availability catalogs belong
+in the release. The current source version, package metadata and option-file
+version must agree.
+
+Native integration uses `COPRARNA_PIPELINE_TESTS=1`; reference-genome extraction
+uses `COPRARNA_REFERENCE_TESTS=1`; optional browser integration uses
+`COPRARNA_BROWSER_TESTS=1`. CI checks Python 3.11–3.13, builds and installed wheels.
+Refresh the 17 NCBI reference records with `python -m tools.refresh_core_genomes`;
+verify exact identities, checksums and annotation dates in the archive manifest.
+Live external-service availability is not certified by offline request tests.
+Check new dependency/UI releases before changing pins.
+
+## Invariants
+
+Preserve the user's original FASTA and native parameter file, including inputs
+named `input_sRNA.fa`. When adding outputs, update `io._JOB_OUTPUTS`; preserve
+cleanup destinations that resolve to original inputs. Every independent native
+entry point retains its version check and detected-version cache signature.
+
+Keep genomes compressed and reuse shared paths directly. Cache metadata pins exact
+assembly/sequence versions until explicitly refreshed. Publish only completed
+validated downloads/resolutions, under accession locks; failed refreshes preserve
+prior entries. A chromosome and plasmid from the same assembly must never count
+as independent comparative organisms. Scientific report labels are unversioned;
+job metadata and cache paths preserve exact versions. SVG/PDF remain primary
+figure formats; no PostScript/Ghostscript dependency is permitted.

ChangeLog.md

@@ -1,8 +1,82 @@
 # ChangeLog
 
-## Unreleased — Python 3 CopraRNA2
-
-Development release `3.0.0.dev0`, addressing
+## 3.0.0 — release preparation, 2026-10-22
+
+Addresses [issue #4](https://github.com/BackofenLab/CopraRNA/issues/4), based on and
+targeting `issue-1-python3-bugfixes`. This release completes the transition from
+the original 2.1.3 scripts to the Python-only CopraRNA2 runtime.
+
+- **Correct genomic accessibility boundaries ([issue #6](https://github.com/BackofenLab/CopraRNA/issues/6)).**
+  Extend native target inputs on both sides by the target accessibility base-pair
+  span (`tAccL`, 100 nt by default), honoring shared `accL` and the native zero/window
+  semantics. Wrap circular genomes and stop at physical linear ends; preserve
+  strand, spliced CDS and short-truncated-UTR selection conventions. Restrict
+  interactions with `--tRegion` and native indexing so all reported positions,
+  seeds and structures refer to the requested sequence. Trim target CSV sequences
+  and normalize `E_norm` before filtering/statistics; reuse context for web exports.
+  Keep context sidecars in prediction signatures and retained job artifacts, and
+  invalidate old predictions. This intentionally changes accessibility energies and
+  potentially rankings; independently captured fixtures for unchanged extraction,
+  statistics and reports remain untouched. Test explicit boundary contracts and
+  real IntaRNAsTar interactions, including the effect of genomic folding context.
+- **Simplify missing report fields and update DAVID access.** Following review of
+  [PR #7](https://github.com/BackofenLab/CopraRNA/pull/7) and the convention in
+  [upstream issue #22](https://github.com/PatrickRWright/CopraRNA/issues/22), write
+  unavailable values as empty cells in published result and web tables instead of
+  the internal `NA` marker. This makes CSV imports unambiguous without changing
+  statistical missing-value handling. Authenticate DAVID requests with the shared
+  email account instead of the former personal address.
+- **Remove static organism catalogs.** `CopraRNA_available_organisms.txt` supplied
+  display names and an input allow-list; `kegg2refseqnew.csv` supplied assembly
+  membership and arbitrary organism codes. Neither is necessary: resolve current
+  complete RefSeq assemblies and every chromosome/plasmid through NCBI
+  E-utilities and Datasets v2, and obtain names from metadata/GenBank. No global
+  table generation, KEGG identifiers or release-bound availability updates remain.
+- **Versioned, reusable resolution.** Accept `NC_`, `NZ_` and `GCF_` IDs with optional
+  versions. Cache validated metadata under the shared genome database using locks
+  and atomic publication. Reject ambiguous/missing/draft assemblies, incomplete
+  sequence reports and multiple inputs representing the same assembly. Follow
+  all report pages. Use short, deterministic job-local species codes to respect
+  DomClust's native name limits; snapshot those codes and exact genome identities
+  in `genome_metadata.json`. Resolve assembly/plasmid/versioned FASTA headers to
+  primary-chromosome IDs in a separate working copy for alignment and reporting.
+  Scientific table labels stay unversioned.
+- **Inspect, prefetch and refresh genomes.** Add `coprarna genomes`, `--download`
+  and `--refresh`. Cache exact-version gzip GenBank files, fetch accessions directly
+  without arbitrary first-hit searches, and retain old files on failed refresh.
+  Pace NCBI requests with timeouts and bounded transient retries. Reuse shared
+  compressed paths directly; prefetch metadata plus genomes for offline jobs.
+- **Remove the original code base.** Delete the Perl/R/Python 2/shell/PhantomJS
+  programs, old HTML wrappers, per-script mapping documents, migration inventory,
+  catalog maintenance code/resources, optional interpreter environments and
+  legacy source-capture tooling. Remove defect-emulation branches, script-name
+  aliases and `coprarna status`. Keep independently captured scientific data and
+  provenance with checksum protection; tests no longer execute original sources.
+- **Refresh all 17 reference genomes.** Download current NCBI GenBank annotations
+  and sequence versions, replacing the old bzip2 snapshots with versioned gzip
+  files. The archive manifest records source URLs, retrieval time, record dates,
+  lengths, CDS counts and SHA-256 hashes. A Python refresh tool stages and checks
+  all records before publishing them. Reference tests check identities and all
+  three target-region modes without invoking legacy scripts.
+- **Publish one complete guide.** Replace the old README with the expanded Python
+  guide covering installation, input choice, examples, supported assemblies,
+  database updates, native settings, every prediction option, outputs, batches,
+  migration, testing and biological citations. Remove README.2.md. Update the
+  architecture ledger, wheel/sdist contents, package/CLI version and job-option
+  version consistently to 3.0.0; include the current HTML report resource.
+
+The validated numerical dependency/kernel pins are unchanged. Captured GEV,
+correlation, aggregation, consensus and Jalview expectations are not regenerated.
+Native integration verifies the current scientific interfaces, complete predictions,
+file/inline equivalence, shared caches, preserved inputs and vector reports.
+Live NCBI access was exercised for the reference refresh; offline request tests do
+not certify future NCBI or DAVID availability. Cached metadata remains fixed until
+explicit refresh; NCBI annotation updates can change predictions even when the
+sequence accession version is unchanged. Use fresh job directories for new data.
+
+## 3.0 development history — issues #1 and #3
+
+The `3.0.0.dev0` development series addressed
 [issue #1](https://github.com/BackofenLab/CopraRNA/issues/1).
 The rewrite and subsequent corrections are reviewed separately in
 [PR #2](https://github.com/BackofenLab/CopraRNA/pull/2) and
@@ -12,11 +86,11 @@ The rewrite and subsequent corrections are reviewed separately in
 
 The source workflow identifies itself as **CopraRNA 2.1.3**. Its controller and
 auxiliary Perl, R, Python 2, shell and JavaScript programs were translated into a
-Python 3 package. Original tracked files and bundled biological data remain
-byte unchanged; the Python runtime never invokes those scripts. The initial
+Python 3 package. During that rewrite, original tracked files and bundled biological data were kept
+byte unchanged; the Python runtime never invoked those scripts. The initial
 rewrite provided counterparts for all 41 scripts, with adjacent mapping documents
-and a machine-readable migration inventory. The reviewed runtime now retains 39
-active counterparts and marks two obsolete scripts retired.
+and a machine-readable migration inventory. At the end of PR #3, the runtime retained 39
+active counterparts and marked two obsolete scripts retired.
 
 The architecture separates the command-line interface and isolated batch runner
 (`cli.py`, `pipeline.py`) from the scientific stages:
@@ -75,8 +149,8 @@ workflows additionally exercise real external-tool integration.
   supported Linux x86-64/AVX2 systems before NumPy import, respecting explicit
   caller overrides. CPU/BLAS differences outside that setup remain a parity limit.
 
-Historical helper comparisons can explicitly reproduce retained defects in a
-thread-local context. Complete production predictions always use corrected code.
+PR #3 supported explicit thread-local reproduction of retained defects for
+historical helper comparisons. That comparison mode was removed for 3.0.0.
 
 ### Review-driven behavior and integration changes — PR #3
 
@@ -143,11 +217,9 @@ whole-workflow equality is no longer the acceptance criterion.
 
 Live NCBI/DAVID service availability is not certified by offline request tests.
 The conda environment targets Linux; it is not a published Bioconda recipe or a
-portable lockfile. Original sources remain for auditing, with optional interpreter
-requirements separated in `environment-legacy.yml` and `environment-legacy-r.yml`.
-Keep these optional interpreters in separate environments to avoid incompatible
-XML/ICU libraries. The retained RobustRankAggreg 1.1 oracle installs from its
-archived pure-R source, avoiding obsolete conda ABI pins.
+portable lockfile. PR #3 retained original sources and separate optional Perl/R oracle environments
+for auditing. Version 3.0.0 removes them from the distribution; captured results
+and provenance remain in the test suite, and the sources remain in Git history.
 
 
 ### Follow-up completeness audit — 2026-10-22