From 990c3c8638f45933bad26f666d1e2e1e85d6f5e6 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Fri, 2 Oct 2026 19:41:19 -0400 Subject: [PATCH 1/8] sp run --in-allocation: run a campaign inside one SLURM allocation Runs both invocations under snakemake's local executor inside the allocation sp is called from, so a small campaign pays one queue wait instead of one per job. sp derives --cores from the cpuset and --resources mem_mb from the tightest cgroup memory limit (less a reserve), refuses to run outside an allocation, and defaults the node-local tile store to $SLURM_TMPDIR/tile-store through the existing tile_store_root rebind. README documents sizing and the resume caveat. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz --- workflow/README.md | 37 +++++++++++++++++++++ workflow/bin/sp | 81 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 114 insertions(+), 4 deletions(-) diff --git a/workflow/README.md b/workflow/README.md index 4f0b135c3..6cbad9490 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -230,6 +230,43 @@ neighbours (queue-latency amortization, per the PRD) are **not yet wired**: they require labels in `workflow/rules/*.smk`, out of scope for this profile-only pass. +## Execution mode: one allocation + +One SLURM job per rule pays one queue wait per job, and at a low fair share a +small campaign is mostly queue wait. `sp run --in-allocation` runs both +invocations under snakemake's local executor inside the allocation it is +called from, so the campaign waits once: + +```bash +#!/usr/bin/env bash +#SBATCH --account= +#SBATCH --nodes=1 --ntasks=1 +#SBATCH --cpus-per-task=96 --mem=383000M +#SBATCH --time=4:00:00 +SP_PROFILE=nibi /path/to/checkout/workflow/bin/sp run --in-allocation -c run.yaml +``` + +`sp` reads the budget from the allocation itself (cores from the cpuset, +memory from the tightest cgroup limit, less a reserve for snakemake) and +passes `--executor local --cores N --resources mem_mb=M`; later explicit +flags override them. Everything else is the profile's. `group:` labels are +inactive under local execution, so `--groups` does nothing; the tile_shape +members still share their node-local store because every job runs on this +node. That store defaults to `$SLURM_TMPDIR/tile-store`, which Slurm reclaims +when the job ends (an explicit `tile_store_root:` wins). + +A rule's `mem_mb` becomes a packing budget rather than a per-job cgroup: a job +over its share is killed only if the allocation as a whole runs out. Size the +allocation so the widest rule's last retry fits (a retry asking more than the +allocation has stops the run), at no more than 4 GB per core so memory does +not raise the billing. Keep it to one node: the tile store is node-local, so +a multi-node allocation would split a tile's vignets from its ngmix chunks. +For the same reason a time limit that ends the job mid-tile leaves that tile's +`tile_vignets.json` without its store, and a resume trips tile.smk's +"node-local vignette store is missing" guard. Delete the manifest of each tile +that has no `final_cat` yet before resuming, as the guard's message says, and +give the time limit room. + ## Layout ``` diff --git a/workflow/bin/sp b/workflow/bin/sp index cb6754108..241c920b1 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -15,6 +15,11 @@ # FILE is a user-specified config file that can be used # to add and/or overwrite entries in the default config # file workflow/config.yaml (read automatically) +# sp run --in-allocation [-c FILE] [ARGS...] +# the same, but every job runs HERE, inside the SLURM +# allocation sp is called from, under snakemake's local +# executor (see "one allocation" below). Refuses to run +# outside an allocation. # sp report [ARGS...] emit run_report.json now (mid-run is fine). # sp container VERB manage the image every job runs inside: pull, status, # sandbox, exec, resolve. `sp container --help` documents @@ -68,13 +73,17 @@ PROFILE="${SP_PROFILE:-nibi}" # the delegated command (`sp container exec python -c ...`) or for snakemake's # own escape hatch (`sp -- ...`) is left alone. `--config` is NOT sp's: keeping # it free lets snakemake's own `--config k=v` (what the Snakefile's flag() -# reads) pass through untouched. +# reads) pass through untouched. --in-allocation is sp's too (see "one +# allocation" below); snakemake never sees it. RUN_CONFIG="${SP_RUN_CONFIG:-}" +IN_ALLOCATION=0 _sp_args=() while [ $# -gt 0 ]; do case "$1" in --) shift; _sp_args+=("$@"); break ;; + --in-allocation) + IN_ALLOCATION=1; shift ;; -c|--config-file) [ $# -ge 2 ] || { echo "sp: $1 needs a file" >&2; exit 2; } RUN_CONFIG="$2"; shift 2 ;; @@ -153,6 +162,70 @@ fi STATE_DIR="${SP_STATE_DIR:-${RUN_DIR}-state}" [ "$NEEDS_CAMPAIGN" = 0 ] || mkdir -p "$STATE_DIR" +# --- one allocation (`sp run --in-allocation`) ----------------------------- +# The profile's SLURM executor submits one job per rule (or group) and so pays +# one queue wait per job; at a low fair share a small campaign is mostly queue +# wait. --in-allocation runs the same two invocations under snakemake's LOCAL +# executor instead, inside the allocation sp is called from (an sbatch script +# or salloc), so the campaign pays one wait. Everything else is the profile's: +# the apptainer wrapping, the rerun triggers, keep-going. +# +# What changes in local execution, and why each is safe: +# * `group:` labels are inactive ("Group jobs: inactive (local execution)"), +# so --groups and the rule files' groups do nothing. tile_shape's members +# still share the node-local store because they all run on THIS node. +# * The allocation's cores and memory are the scheduler's global budget, +# read here: cores from the cpuset (`nproc`), memory from the tightest +# cgroup limit above this process (an srun step's own --mem, not the +# job's), less a reserve for snakemake and the localrules. A rule's mem_mb +# is then a packing budget, not a cgroup: a job over its share is killed +# only if the whole allocation runs out. Attempt-scaled retries still +# raise the budget, and a retry asking more than the allocation has stops +# the run (snakemake's "Job needs mem_mb=..."), so size the allocation for +# the widest rule's third attempt. +# * The tile store defaults to $SLURM_TMPDIR/tile-store. The profile cannot +# name $SLURM_TMPDIR (its `--bind /local` comment says why), but sp runs +# inside the job here and knows the value, so it rebinds /local/scratch +# there exactly as `tile_store_root:` does. Slurm reclaims it at job end, +# and no other campaign on the node can share it. An explicit +# `tile_store_root:` still wins. +# Explicit --cores/--resources after it override the derived values. +ALLOC_ARGS=() +TILE_STORE_ROOT="" +[ "$NEEDS_CAMPAIGN" = 0 ] || TILE_STORE_ROOT="$(cfg tile_store_root)" +if [ "$IN_ALLOCATION" = 1 ]; then + [ "${1:-}" = run ] || { echo "sp: --in-allocation applies to \`sp run\` only" >&2; exit 2; } + [ -n "${SLURM_JOB_ID:-}" ] && [ -n "${SLURM_TMPDIR:-}" ] || { + echo "sp: --in-allocation runs every job on this machine; call it inside a SLURM" >&2 + echo " allocation (an sbatch script, or salloc), never on a login node" >&2; exit 2; } + ALLOC_CORES="$(nproc)" + ALLOC_MEM_MB="$(python - <<'PY' +import os, pathlib +limits = [] +try: + rel = next(l.split(":", 2)[2].strip() for l in open("/proc/self/cgroup") + if l.startswith("0::")) + p = pathlib.Path("/sys/fs/cgroup" + rel) + while p != p.parent: + f = p / "memory.max" + if f.exists() and f.read_text().strip() not in ("", "max"): + limits.append(int(f.read_text()) // 2**20) + p = p.parent +except (OSError, StopIteration, ValueError): + pass +if not limits and os.environ.get("SLURM_MEM_PER_NODE"): + limits.append(int(os.environ["SLURM_MEM_PER_NODE"])) +total = min(limits) if limits else 0 +print(total - max(4000, total // 20) if total else 0) +PY +)" + [ "$ALLOC_MEM_MB" -gt 0 ] || { + echo "sp: --in-allocation could not read this allocation's memory limit" >&2; exit 2; } + ALLOC_ARGS=(--executor local --cores "$ALLOC_CORES" --resources "mem_mb=$ALLOC_MEM_MB") + [ -n "$TILE_STORE_ROOT" ] || TILE_STORE_ROOT="$SLURM_TMPDIR/tile-store" + echo "sp: in allocation $SLURM_JOB_ID: ${ALLOC_ARGS[*]}" >&2 +fi + # --- the launch code snapshot ---------------------------------------------- # THE ONE HOME for this concept; everything else points here. # @@ -227,7 +300,7 @@ PY # directory (NFS) trades ngmix read latency for room; the campaign-unique store # names (LOCAL_TAG) keep concurrent campaigns apart in one root. Profiles with # no `X:/local/scratch` bind (nibi's `--bind /local`) get one appended. - python - "$SNAPSHOT/profiles/$PROFILE/config.yaml" "$(cfg tile_store_root)" <<'PY' + python - "$SNAPSHOT/profiles/$PROFILE/config.yaml" "$TILE_STORE_ROOT" <<'PY' import os, pathlib, re, sys f, root = pathlib.Path(sys.argv[1]), sys.argv[2] if root: @@ -324,7 +397,7 @@ case "$cmd" in # the compute parse's index build enforces over the WHOLE tile list — so we # record the failure and go on rather than letting `set -e` abort here. prep_rc=0 - SP_PHASE=prepare sm prepare_all_tiles "$@" || prep_rc=$? + SP_PHASE=prepare sm prepare_all_tiles ${ALLOC_ARGS[@]+"${ALLOC_ARGS[@]}"} "$@" || prep_rc=$? if [ "$prep_rc" -ne 0 ]; then echo "" >&2 echo "############################################################" >&2 @@ -338,7 +411,7 @@ case "$cmd" in fi comp_rc=0 - SP_PHASE=compute sm all "$@" || comp_rc=$? + SP_PHASE=compute sm all ${ALLOC_ARGS[@]+"${ALLOC_ARGS[@]}"} "$@" || comp_rc=$? if [ "$prep_rc" -ne 0 ] || [ "$comp_rc" -ne 0 ]; then exit "$([ "$comp_rc" -ne 0 ] && echo "$comp_rc" || echo "$prep_rc")" fi From 914510be98981b05971314162c4408e66b422e60 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Fri, 2 Oct 2026 19:42:53 -0400 Subject: [PATCH 2/8] sp --in-allocation: campaign-unique default tile store; two campaigns per allocation Data campaigns name node-local stores by bare tile id, so two campaigns over the same tiles sharing one allocation would share stores. Default to $SLURM_TMPDIR/tile-store/, and document one srun step per campaign. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01Dj4HnkNqqPkVCeDeUGEGuz --- workflow/README.md | 14 ++++++++++++-- workflow/bin/sp | 15 ++++++++------- 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/workflow/README.md b/workflow/README.md index 6cbad9490..1fdd9e556 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -252,8 +252,18 @@ passes `--executor local --cores N --resources mem_mb=M`; later explicit flags override them. Everything else is the profile's. `group:` labels are inactive under local execution, so `--groups` does nothing; the tile_shape members still share their node-local store because every job runs on this -node. That store defaults to `$SLURM_TMPDIR/tile-store`, which Slurm reclaims -when the job ends (an explicit `tile_store_root:` wins). +node. That store defaults to `$SLURM_TMPDIR/tile-store/`, which Slurm +reclaims when the job ends (an explicit `tile_store_root:` wins). + +Two campaigns can share one allocation, one `srun` step each. Each step's +cpuset and cgroup become that campaign's budget, and `` keeps their +stores apart: + +```bash +srun --exact -n1 -c96 --mem=380000M sp run --in-allocation -c arm_a.yaml & +srun --exact -n1 -c96 --mem=380000M sp run --in-allocation -c arm_b.yaml & +wait +``` A rule's `mem_mb` becomes a packing budget rather than a per-job cgroup: a job over its share is killed only if the allocation as a whole runs out. Size the diff --git a/workflow/bin/sp b/workflow/bin/sp index 241c920b1..8f06cf8ca 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -183,12 +183,13 @@ STATE_DIR="${SP_STATE_DIR:-${RUN_DIR}-state}" # raise the budget, and a retry asking more than the allocation has stops # the run (snakemake's "Job needs mem_mb=..."), so size the allocation for # the widest rule's third attempt. -# * The tile store defaults to $SLURM_TMPDIR/tile-store. The profile cannot -# name $SLURM_TMPDIR (its `--bind /local` comment says why), but sp runs -# inside the job here and knows the value, so it rebinds /local/scratch -# there exactly as `tile_store_root:` does. Slurm reclaims it at job end, -# and no other campaign on the node can share it. An explicit -# `tile_store_root:` still wins. +# * The tile store defaults to $SLURM_TMPDIR/tile-store/. The profile +# cannot name $SLURM_TMPDIR (its `--bind /local` comment says why), but sp +# runs inside the job here and knows the value, so it rebinds +# /local/scratch there exactly as `tile_store_root:` does. Slurm reclaims +# it at job end. keeps two campaigns over the same tiles apart when +# they share one allocation (one srun step each): data campaigns name +# their stores by bare tile id. An explicit `tile_store_root:` still wins. # Explicit --cores/--resources after it override the derived values. ALLOC_ARGS=() TILE_STORE_ROOT="" @@ -222,7 +223,7 @@ PY [ "$ALLOC_MEM_MB" -gt 0 ] || { echo "sp: --in-allocation could not read this allocation's memory limit" >&2; exit 2; } ALLOC_ARGS=(--executor local --cores "$ALLOC_CORES" --resources "mem_mb=$ALLOC_MEM_MB") - [ -n "$TILE_STORE_ROOT" ] || TILE_STORE_ROOT="$SLURM_TMPDIR/tile-store" + [ -n "$TILE_STORE_ROOT" ] || TILE_STORE_ROOT="$SLURM_TMPDIR/tile-store/$(cfg run)" echo "sp: in allocation $SLURM_JOB_ID: ${ALLOC_ARGS[*]}" >&2 fi From 5ee28fdd299e5e607ed166724dfeba3c4c4fec36 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Mon, 5 Oct 2026 09:10:19 -0400 Subject: [PATCH 3/8] sp run --in-allocation: resubmit as-is after a timeout A job cut short by its time limit leaves two things a plain resubmission trips over. Its node-local tile store is gone, so a tile whose tile_vignets.json survived without a final_cat stops at tile.smk's "vignette store is missing" guard; and a killed snakemake leaves its lock. sp now deletes those manifests before the run, and records the job that holds the state dir so that a lock whose job has left the queue is released. Any other lock stops the run before anything is swept. With -n both only report. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01D8wdTpGpBtyK6uHwU7qWQW --- workflow/README.md | 14 ++++++---- workflow/bin/sp | 64 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 70 insertions(+), 8 deletions(-) diff --git a/workflow/README.md b/workflow/README.md index 1fdd9e556..70c688db5 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -271,11 +271,15 @@ allocation so the widest rule's last retry fits (a retry asking more than the allocation has stops the run), at no more than 4 GB per core so memory does not raise the billing. Keep it to one node: the tile store is node-local, so a multi-node allocation would split a tile's vignets from its ngmix chunks. -For the same reason a time limit that ends the job mid-tile leaves that tile's -`tile_vignets.json` without its store, and a resume trips tile.smk's -"node-local vignette store is missing" guard. Delete the manifest of each tile -that has no `final_cat` yet before resuming, as the guard's message says, and -give the time limit room. + +A job cut short by its time limit is resumed by submitting the same script +again. The store died with the job, so before it runs `sp` deletes +`tile_vignets.json` for each listed tile that has no `final_cat` yet (tile.smk's +"node-local vignette store is missing" guard would stop its chunks otherwise); +tile_vignets rebuilds the store and the chunks rerun after it. `sp` also +records the job holding the campaign's state dir, and releases that job's lock +once the job has left the queue; any other lock stops the run. With `-n` both +only report. ## Layout diff --git a/workflow/bin/sp b/workflow/bin/sp index 8f06cf8ca..afc277a4e 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -187,9 +187,20 @@ STATE_DIR="${SP_STATE_DIR:-${RUN_DIR}-state}" # cannot name $SLURM_TMPDIR (its `--bind /local` comment says why), but sp # runs inside the job here and knows the value, so it rebinds # /local/scratch there exactly as `tile_store_root:` does. Slurm reclaims -# it at job end. keeps two campaigns over the same tiles apart when -# they share one allocation (one srun step each): data campaigns name -# their stores by bare tile id. An explicit `tile_store_root:` still wins. +# it at job end. gives each campaign sharing one allocation (one +# srun step each) its own directory. An explicit `tile_store_root:` still +# wins. +# * A resume finds the previous job's store gone. A tile whose +# tile_vignets.json survived it but whose final_cat did not would trip +# tile.smk's "node-local vignette store is missing" guard, so that +# manifest is deleted first: tile_vignets reruns and rebuilds the store, +# and the tile's chunks rerun after it. Tiles with a final_cat are never +# touched. +# * A lock left by a killed in-allocation run is released. sp records the +# job that holds the state dir; when that job is no longer in the queue, +# its lock is stale. Any other lock (a run sp did not start in an +# allocation, or one still running) stops the run instead. +# With -n both only report. # Explicit --cores/--resources after it override the derived values. ALLOC_ARGS=() TILE_STORE_ROOT="" @@ -227,6 +238,52 @@ PY echo "sp: in allocation $SLURM_JOB_ID: ${ALLOC_ARGS[*]}" >&2 fi +# The resume housekeeping (a stale lock, then stale vignette manifests; the +# lock goes first so a campaign still running is never swept). `sp run` calls +# it before the snapshot, with its own arguments, so -n only reports. +alloc_resume() { + local a alloc_dry=0 products_dir n_swept=0 tile m holder holder_file + local holder_state="" holder_rc=1 + for a in "$@"; do + case "$a" in -n|--dry-run|--dryrun|-[a-zA-Z]*n*) alloc_dry=1 ;; esac + done + holder_file="$STATE_DIR/in-allocation.job" + if [ -n "$(ls -A "$STATE_DIR/.snakemake/locks" 2>/dev/null)" ]; then + holder="$(cat "$holder_file" 2>/dev/null || true)" + if [ -n "$holder" ]; then + holder_state="$(squeue -h -j "$holder" -o %T 2>&1)" && holder_rc=0 || holder_rc=$? + fi + # Ended: gone from the controller, or listed in a final state. + if [ -n "$holder" ] && [ "$holder" != "$SLURM_JOB_ID" ] && { + [[ "$holder_state" == *"Invalid job id"* ]] || { [ "$holder_rc" = 0 ] && + [[ "$holder_state" =~ ^(|COMPLETED|FAILED|CANCELLED|TIMEOUT|OUT_OF_MEMORY|NODE_FAIL|PREEMPTED|BOOT_FAIL|DEADLINE)$ ]]; }; }; then + if [ "$alloc_dry" = 1 ]; then + echo "sp: stale lock of job $holder in $STATE_DIR (dry run, kept)" >&2 + else + echo "sp: releasing the stale lock of job $holder in $STATE_DIR" >&2 + sm --unlock >/dev/null + fi + else + echo "sp: $STATE_DIR is locked${holder:+ (job $holder: ${holder_state:-running})}." >&2 + echo " If nothing runs on this campaign, \`sp -c ... --unlock\` and resubmit." >&2 + exit 2 + fi + fi + + products_dir="$(cfg outputs.products_dir)" + while read -r tile; do + [ -n "$tile" ] || continue + m="$RUN_DIR/tiles/${tile:0:2}/$tile/manifests/tile_vignets.json" + [ -e "$m" ] || continue + [ -e "$products_dir/tiles/${tile:0:2}/$tile/final_cat-$tile.hdf5" ] && continue + if [ "$alloc_dry" = 1 ]; then echo "sp: stale (dry run, kept): $m" >&2 + else rm -f "$m"; echo "sp: deleted stale $m" >&2; fi + n_swept=$((n_swept + 1)) + done < <(sed 's/[[:space:]]//g' "$(cfg tile_list)") + echo "sp: $n_swept stale tile_vignets.json (no final_cat, store gone with its job)" >&2 + [ "$alloc_dry" = 1 ] || printf '%s\n' "$SLURM_JOB_ID" > "$holder_file" +} + # --- the launch code snapshot ---------------------------------------------- # THE ONE HOME for this concept; everything else points here. # @@ -391,6 +448,7 @@ cmd="${1:-}" case "$cmd" in run) shift + [ "$IN_ALLOCATION" = 0 ] || alloc_resume "$@" snapshot_code # PREPARE failing is NOT fatal to the run: keep-going means a failed tile # poisons only its own cone, and the tiles that lost their exposure list are From 313ccd729bf0635b0179080cb563fd42fbd11ce8 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Fri, 16 Oct 2026 22:38:16 -0400 Subject: [PATCH 4/8] sp: scope sm()'s SP_RUN_CONFIG to its snakemake call sm() exported SP_RUN_CONFIG as the snapshot's copy into the caller's shell, so on a resumed 'sp run' the snapshot refresh copied run_config.yaml onto itself and cp's non-zero exit failed the run in seconds (smk-b122, smk-b124 after timeouts). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01XekXwUoqNTMvZiEEsCtwZx (cherry picked from commit ab4c29345f77c752782286f5caafb8acec809ccf) --- workflow/bin/sp | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/workflow/bin/sp b/workflow/bin/sp index afc277a4e..39d6a50bf 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -435,13 +435,17 @@ sm() { local root; root="$(code_root)" # The run config the Snakefile reads: the snapshot's copy when this run has # one, so every parse of the campaign -- head and jobs -- sees the same file. - if [ -n "${SP_RUN_CONFIG:-}" ] && [ -f "$root/run_config.yaml" ]; then - export SP_RUN_CONFIG="$root/run_config.yaml" - fi - SP_PHASE="${SP_PHASE:-passthrough}" \ - snakemake --snakefile "$root/workflow/Snakefile" \ - --profile "$root/profiles/$PROFILE" \ - --directory "$STATE_DIR" "$@" + # Scoped to this call (subshell): SP_RUN_CONFIG in the caller stays the + # external config, so a later snapshot refresh never copies the file onto itself. + ( + if [ -n "${SP_RUN_CONFIG:-}" ] && [ -f "$root/run_config.yaml" ]; then + export SP_RUN_CONFIG="$root/run_config.yaml" + fi + SP_PHASE="${SP_PHASE:-passthrough}" \ + snakemake --snakefile "$root/workflow/Snakefile" \ + --profile "$root/profiles/$PROFILE" \ + --directory "$STATE_DIR" "$@" + ) } cmd="${1:-}" From ca224f7e2e73427ece71c8328243a7e6f6d3f167 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Sat, 17 Oct 2026 06:00:02 -0400 Subject: [PATCH 5/8] fix(workflow): rebuild swept vignettes before make_cat on resume The in-allocation sweep (5ee28fdd, #964) removes tile_vignets.json when a tile has no final catalogue. If ngmix and merge_cats already finished, make_cat had no dependency on that manifest and ran alone against a lost node-local store (smk-b81, tile 291.293, job 24195960). Declare the vignets manifest as a make_cat input. A plain edge also queues all completed ngmix chunks: Snakemake 9.23.1 propagates a missing producer output even with ancient(). Instead, cut that edge for completed chunks while the vignets manifest is missing. Require either the chunk directory or a completed merge, so a lost pre-merge chunk still waits for the writer. Finished tiles also cut the edge: otherwise the restored manifest's newer mtime would wake preserved chunks on the next invocation, when the sweep skips the now-finished tile. Unfinished tiles with an existing vignets manifest retain normal timestamp and structural invalidation. Trade-off: storage-only recovery and finished tiles trust existing chunk completion manifests instead of rerunning expensive science to restore a transient store. This uses the same disabled input rerun trigger and finished-product assumption as the exposure-edge cut. To explicitly recompute completed chunks after a sweep, delete their manifests too; to rebuild a finished tile, delete its final catalogue first. No shell body or params.pre changes, so the campaign-boundary pins stay unchanged. Add planning regressions for merged outputs (including reclaimed temp chunks), unmerged completed chunks, one unfinished chunk, and a lost chunk directory, in data and image-sims modes. Assert writer-before-reader edges, preserved completed chunks, profile trigger policy, stability on the invocation after recovery, and ordinary upstream invalidation. --- tests/workflow/test_resume.py | 105 ++++++++++++++++++++++++++++++++++ workflow/README.md | 17 +++--- workflow/bin/sp | 8 ++- workflow/rules/tile.smk | 38 +++++++++--- 4 files changed, 149 insertions(+), 19 deletions(-) create mode 100644 tests/workflow/test_resume.py diff --git a/tests/workflow/test_resume.py b/tests/workflow/test_resume.py new file mode 100644 index 000000000..ffa80d3ca --- /dev/null +++ b/tests/workflow/test_resume.py @@ -0,0 +1,105 @@ +"""Resume a tile after its node-local vignette manifest has been swept.""" + +import os +import shutil +from collections import Counter +from pathlib import Path + +import pytest +from snakemake.io import is_flagged + + +def test_resume_edge_cut_requires_input_trigger_off(profile): + """Changing a satisfied chunk's input set must not invalidate it.""" + assert "input" not in profile["rerun-triggers"] + + +def seed_outputs(campaign, resolve_dag): + """Materialise satisfied outputs without executing the science pipeline.""" + with resolve_dag(campaign) as dag: + for job in dag.jobs: + for output in job.output: + path = Path(output) + path.parent.mkdir(parents=True, exist_ok=True) + if is_flagged(output, "directory"): + path.mkdir(exist_ok=True) + stamp = path / ".snakemake_timestamp" + stamp.touch() + os.utime(stamp, (1, 1)) + else: + path.touch() + os.utime(path, (1, 1)) + + +@pytest.mark.parametrize("state", [ + "merged", "chunks_done", "partial", "missing_chunkdir", +]) +def test_resume_after_vignets_swept(campaign, resolve_dag, state): + """Only unfinished chunks read the rebuilt store; make_cat always does.""" + seed_outputs(campaign, resolve_dag) + tile = next(iter(campaign.ready)) + root = campaign.tile_manifest(tile, "tile_vignets").parents[1] + campaign.tile_manifest(tile, "tile_vignets").unlink() + campaign.tile_manifest(tile, "tile_make_cat").unlink() + campaign.final_cat(tile).unlink() + if state == "merged": + # merge_cats reclaims its temp inputs in production. + for path in (root / "output").glob("run_sp_tile_ngmix_Ng*u"): + shutil.rmtree(path) + else: + campaign.tile_manifest(tile, "tile_merge_cats").unlink() + if state == "partial": + campaign.tile_manifest(tile, "tile_ngmix_8").unlink() + if state == "missing_chunkdir": + shutil.rmtree(root / "output/run_sp_tile_ngmix_Ng8u") + with resolve_dag(campaign) as dag: + pending = list(dag.workflow.dag.needrun_jobs()) + counts = Counter(job.rule.name for job in pending) + print(f"Resume dry-run jobs ({state}):", dict(sorted(counts.items()))) + assert counts["tile_vignets"] == 1 + assert counts["tile_make_cat"] == 1 + assert counts["tile_ngmix"] == (state in {"partial", "missing_chunkdir"}) + assert counts["tile_merge_cats"] == (state != "merged") + vignets = next(j for j in pending if j.rule.name == "tile_vignets") + for job in pending: + if job.rule.name in {"tile_make_cat", "tile_ngmix"}: + assert vignets in dag.workflow.dag.dependencies[job] + + +def test_finished_tile_after_store_only_rebuild(campaign, resolve_dag): + """A resumed tile must stay finished on the next invocation.""" + seed_outputs(campaign, resolve_dag) + tile = next(iter(campaign.ready)) + # The store-only rebuild is newer than the preserved chunk products; + # make_cat then succeeds. alloc_resume skips tiles with a final catalogue. + campaign.tile_manifest(tile, "tile_vignets").touch() + with resolve_dag(campaign) as dag: + outputs = [ + Path(output) + for job in dag.jobs_for("tile_make_cat") + if job.wildcards.tile == tile + for output in job.output + ] + # Include any additional durable outputs, such as published tile stats. + for path in outputs: + path.touch() + with resolve_dag(campaign) as dag: + counts = Counter(j.rule.name for j in dag.workflow.dag.needrun_jobs()) + assert counts["tile_ngmix"] == 0 + assert counts["tile_merge_cats"] == 0 + assert counts["tile_vignets"] == 0 + assert counts["tile_make_cat"] == 0 + + +def test_real_vignets_rebuild_still_invalidates_chunks(campaign, resolve_dag): + """The storage-only exception must not hide upstream science changes.""" + seed_outputs(campaign, resolve_dag) + tile = next(iter(campaign.ready)) + campaign.tile_manifest(tile, "tile_detect").touch() + campaign.tile_manifest(tile, "tile_make_cat").unlink() + campaign.final_cat(tile).unlink() + with resolve_dag(campaign) as dag: + pending = list(dag.workflow.dag.needrun_jobs()) + counts = Counter(j.rule.name for j in pending) + assert counts["tile_vignets"] == 1 + assert counts["tile_ngmix"] == dag.namespace["NGMIX_CHUNKS"] diff --git a/workflow/README.md b/workflow/README.md index 70c688db5..a124fb4f1 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -272,14 +272,15 @@ allocation has stops the run), at no more than 4 GB per core so memory does not raise the billing. Keep it to one node: the tile store is node-local, so a multi-node allocation would split a tile's vignets from its ngmix chunks. -A job cut short by its time limit is resumed by submitting the same script -again. The store died with the job, so before it runs `sp` deletes -`tile_vignets.json` for each listed tile that has no `final_cat` yet (tile.smk's -"node-local vignette store is missing" guard would stop its chunks otherwise); -tile_vignets rebuilds the store and the chunks rerun after it. `sp` also -records the job holding the campaign's state dir, and releases that job's lock -once the job has left the queue; any other lock stops the run. With `-n` both -only report. +A job cut short by its time limit is resumed by submitting the same script again. +The store died with the job, so before it runs `sp` deletes `tile_vignets.json` for each listed tile that has no `final_cat` yet. +`tile_make_cat` declares that manifest as an input, so `tile_vignets` rebuilds the store even if all chunks and their merge are complete. +Completed chunks keep their products; only unfinished chunks wait for the rebuilt store and run again. +A chunk whose temporary directory is missing must also rerun if its merge has not succeeded. +This storage-only recovery trusts the chunk completion manifests: to explicitly recompute a completed chunk, remove its manifest too. +To rebuild a finished tile, remove its final catalogue first, as for the exposure-edge cut in `tile.smk`. +`sp` also records the job holding the campaign's state dir, and releases that job's lock once the job has left the queue; any other lock stops the run. +With `-n` both only report. ## Layout diff --git a/workflow/bin/sp b/workflow/bin/sp index 39d6a50bf..77cd60ebf 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -193,9 +193,11 @@ STATE_DIR="${SP_STATE_DIR:-${RUN_DIR}-state}" # * A resume finds the previous job's store gone. A tile whose # tile_vignets.json survived it but whose final_cat did not would trip # tile.smk's "node-local vignette store is missing" guard, so that -# manifest is deleted first: tile_vignets reruns and rebuilds the store, -# and the tile's chunks rerun after it. Tiles with a final_cat are never -# touched. +# manifest is deleted first. tile_make_cat declares it as an input, so +# tile_vignets rebuilds the store even if merge_cats is already complete. +# Completed chunks skip this storage-only rebuild (ngmix_vignets in +# tile.smk); unfinished chunks wait for it. Tiles with a final_cat are +# never touched. # * A lock left by a killed in-allocation run is released. sp records the # job that holds the state dir; when that job is no longer in the queue, # its lock is stale. Any other lock (a run sp did not start in an diff --git a/workflow/rules/tile.smk b/workflow/rules/tile.smk index 107bce4f9..80771e60e 100644 --- a/workflow/rules/tile.smk +++ b/workflow/rules/tile.smk @@ -614,6 +614,30 @@ rule tile_vignets: sp_shell("tile_vignets", f"config_tile_PiViVi_{PSF_MODEL}.ini", check_args=' --run-dir "$SP_LOCAL" --unit {wildcards.tile}') +def ngmix_vignets(wc): + """Keep completed chunks when only the node-local store was swept. + + A missing temp chunk directory is harmless after merge_cats succeeds, but + before that merge it must be rebuilt and therefore needs the store edge. + Unfinished tiles with existing vignets manifests retain normal rerun + propagation. Finished tiles also cut the edge, so a restored manifest's + newer timestamp cannot wake old chunks on the next invocation. As with + the finished-tile exposure cut above, the profile disables the input + trigger. To force a chunk to recompute after a sweep, remove its manifest + too; to rebuild a finished tile, remove its final catalogue first. + """ + root = Path(tile_dir(wc.tile)) + vignets = root / "manifests/tile_vignets.json" + chunk = root / f"manifests/tile_ngmix_{wc.chunk}.json" + chunkdir = root / f"output/run_sp_tile_ngmix_Ng{wc.chunk}u" + merged = root / "manifests/tile_merge_cats.json" + if (not vignets.exists() or tile_finished(wc.tile)) and chunk.exists() and ( + chunkdir.exists() or merged.exists() + ): + return [] + return [str(vignets)] + + # ngmix shape measurement — N chunks per tile (D4). Each chunk LOOKS UP its own # CLOSED catalogue-row range in the file tile_vignets materialised at the top of this # group job (TILE_NGMIX_RANGES); the ranges are knowable only at EXECUTION time, @@ -626,11 +650,9 @@ rule tile_vignets: rule tile_ngmix: group: TILE_GROUP input: - # The manifest, and only the manifest. The vignette store is no longer a - # declared input because it is no longer a declared output: it is - # node-local, produced by tile_vignets earlier in THIS SAME group job - # (see TILE_LOCAL). The manifest was always the real edge. - vignets = rules.tile_vignets.output.manifest, + # A completed chunk doesn't read a rebuilt store on resume. Missing + # chunks keep the manifest edge and wait for this group's writer. + vignets = ngmix_vignets, sx = rules.tile_detect.output.manifest, output: manifest = f"{TILE_DIR}/manifests/tile_ngmix_{{chunk}}.json", @@ -871,9 +893,9 @@ rule tile_merge_cats: rule tile_make_cat: group: TILE_GROUP input: - # No store input: it is node-local, written by tile_vignets in this same - # group job. make_cat reads its configured PSF-interpolation output - # through $NGMIX_VIGNET_DIR (config_tile_Mc.ini). + # The store is node-local, but its manifest is the ordering edge: + # after a resume sweep, rebuild it even when merge_cats is complete. + vignets = rules.tile_vignets.output.manifest, ms = rules.tile_merge_cats.output.manifest, output: manifest = f"{TILE_DIR}/manifests/tile_make_cat.json", From 7c3fcdd73e205933eb6d404ea2f94de825c4ac62 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Sat, 17 Oct 2026 06:17:30 -0400 Subject: [PATCH 6/8] fix(workflow): keep vignette dependencies unconditional on resume Fix forward the unsafe conditional edge in ca224f7e. A completion manifest does not mean a chunk will not run: after the vignets manifest is swept, a newer tile_detect input schedules all ngmix chunks, but the conditional removed their dependency on the writer. They could start before the store exists or race its rebuild. Remove ngmix_vignets and restore every chunk unconditional vignets input. Keep the unconditional make_cat input. The accepted trade-off is that sweeping a tile now rebuilds its store and reruns all ngmix chunks and their merge, even if earlier outputs survived. Correct ordering is worth the extra ngmix time on these resumed tiles; no completed-output inference may remove a reader edge. Replace the reuse-oriented tests and documentation. Test the original swept-store scenario and the review counterexample with newer tile_detect, in data and image-sims modes, with completed, reclaimed, partial and missing chunk products. Assert every scheduled chunk and make_cat depends directly on tile_vignets. The new merged-state regressions fail on ca224f7e. Shell bodies and params.pre are unchanged. --- tests/workflow/test_resume.py | 90 +++++++++++++---------------------- workflow/README.md | 7 ++- workflow/bin/sp | 7 +-- workflow/rules/tile.smk | 30 ++---------- 4 files changed, 44 insertions(+), 90 deletions(-) diff --git a/tests/workflow/test_resume.py b/tests/workflow/test_resume.py index ffa80d3ca..ef36cc5d5 100644 --- a/tests/workflow/test_resume.py +++ b/tests/workflow/test_resume.py @@ -1,4 +1,4 @@ -"""Resume a tile after its node-local vignette manifest has been swept.""" +"""Every resumed reader must wait for the rebuilt node-local vignette store.""" import os import shutil @@ -9,11 +9,6 @@ from snakemake.io import is_flagged -def test_resume_edge_cut_requires_input_trigger_off(profile): - """Changing a satisfied chunk's input set must not invalidate it.""" - assert "input" not in profile["rerun-triggers"] - - def seed_outputs(campaign, resolve_dag): """Materialise satisfied outputs without executing the science pipeline.""" with resolve_dag(campaign) as dag: @@ -32,74 +27,57 @@ def seed_outputs(campaign, resolve_dag): @pytest.mark.parametrize("state", [ - "merged", "chunks_done", "partial", "missing_chunkdir", + "merged", "merged_reclaimed", "chunks_done", "partial", "missing_chunkdir", +]) +@pytest.mark.parametrize("invalidate_detect", [False, True], ids=[ + "swept_only", "swept_and_upstream_invalidated", ]) -def test_resume_after_vignets_swept(campaign, resolve_dag, state): - """Only unfinished chunks read the rebuilt store; make_cat always does.""" +def test_resume_after_vignets_swept( + campaign, resolve_dag, state, invalidate_detect, +): + """Sweep alone and sweep plus upstream invalidation both preserve ordering. + + The newer tile_detect input reproduces the review's counterexample: + completed chunks can still be scheduled and must retain their writer edge. + """ seed_outputs(campaign, resolve_dag) tile = next(iter(campaign.ready)) root = campaign.tile_manifest(tile, "tile_vignets").parents[1] campaign.tile_manifest(tile, "tile_vignets").unlink() campaign.tile_manifest(tile, "tile_make_cat").unlink() campaign.final_cat(tile).unlink() - if state == "merged": + if state == "merged_reclaimed": # merge_cats reclaims its temp inputs in production. for path in (root / "output").glob("run_sp_tile_ngmix_Ng*u"): shutil.rmtree(path) - else: + elif state != "merged": campaign.tile_manifest(tile, "tile_merge_cats").unlink() if state == "partial": campaign.tile_manifest(tile, "tile_ngmix_8").unlink() if state == "missing_chunkdir": shutil.rmtree(root / "output/run_sp_tile_ngmix_Ng8u") + if invalidate_detect: + campaign.tile_manifest(tile, "tile_detect").touch() with resolve_dag(campaign) as dag: pending = list(dag.workflow.dag.needrun_jobs()) counts = Counter(job.rule.name for job in pending) - print(f"Resume dry-run jobs ({state}):", dict(sorted(counts.items()))) + print( + f"Resume dry-run jobs ({state}, invalidate_detect={invalidate_detect}):", + dict(sorted(counts.items())), + ) assert counts["tile_vignets"] == 1 + assert counts["tile_ngmix"] == dag.namespace["NGMIX_CHUNKS"] + assert counts["tile_merge_cats"] == 1 assert counts["tile_make_cat"] == 1 - assert counts["tile_ngmix"] == (state in {"partial", "missing_chunkdir"}) - assert counts["tile_merge_cats"] == (state != "merged") - vignets = next(j for j in pending if j.rule.name == "tile_vignets") - for job in pending: - if job.rule.name in {"tile_make_cat", "tile_ngmix"}: - assert vignets in dag.workflow.dag.dependencies[job] - - -def test_finished_tile_after_store_only_rebuild(campaign, resolve_dag): - """A resumed tile must stay finished on the next invocation.""" - seed_outputs(campaign, resolve_dag) - tile = next(iter(campaign.ready)) - # The store-only rebuild is newer than the preserved chunk products; - # make_cat then succeeds. alloc_resume skips tiles with a final catalogue. - campaign.tile_manifest(tile, "tile_vignets").touch() - with resolve_dag(campaign) as dag: - outputs = [ - Path(output) - for job in dag.jobs_for("tile_make_cat") - if job.wildcards.tile == tile - for output in job.output + writer = next(j for j in pending if j.rule.name == "tile_vignets") + readers = [ + job for job in pending + if job.rule.name in {"tile_make_cat", "tile_ngmix"} ] - # Include any additional durable outputs, such as published tile stats. - for path in outputs: - path.touch() - with resolve_dag(campaign) as dag: - counts = Counter(j.rule.name for j in dag.workflow.dag.needrun_jobs()) - assert counts["tile_ngmix"] == 0 - assert counts["tile_merge_cats"] == 0 - assert counts["tile_vignets"] == 0 - assert counts["tile_make_cat"] == 0 - - -def test_real_vignets_rebuild_still_invalidates_chunks(campaign, resolve_dag): - """The storage-only exception must not hide upstream science changes.""" - seed_outputs(campaign, resolve_dag) - tile = next(iter(campaign.ready)) - campaign.tile_manifest(tile, "tile_detect").touch() - campaign.tile_manifest(tile, "tile_make_cat").unlink() - campaign.final_cat(tile).unlink() - with resolve_dag(campaign) as dag: - pending = list(dag.workflow.dag.needrun_jobs()) - counts = Counter(j.rule.name for j in pending) - assert counts["tile_vignets"] == 1 - assert counts["tile_ngmix"] == dag.namespace["NGMIX_CHUNKS"] + assert len(readers) == dag.namespace["NGMIX_CHUNKS"] + 1 + for job in readers: + assert job.wildcards.tile == tile + assert writer in dag.workflow.dag.dependencies[job], ( + f"pending reader {job.rule.name} {job.wildcards} has no writer " + f"dependency; inputs={list(job.input)}" + ) diff --git a/workflow/README.md b/workflow/README.md index a124fb4f1..571728bc4 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -275,10 +275,9 @@ a multi-node allocation would split a tile's vignets from its ngmix chunks. A job cut short by its time limit is resumed by submitting the same script again. The store died with the job, so before it runs `sp` deletes `tile_vignets.json` for each listed tile that has no `final_cat` yet. `tile_make_cat` declares that manifest as an input, so `tile_vignets` rebuilds the store even if all chunks and their merge are complete. -Completed chunks keep their products; only unfinished chunks wait for the rebuilt store and run again. -A chunk whose temporary directory is missing must also rerun if its merge has not succeeded. -This storage-only recovery trusts the chunk completion manifests: to explicitly recompute a completed chunk, remove its manifest too. -To rebuild a finished tile, remove its final catalogue first, as for the exposure-edge cut in `tile.smk`. +Every ngmix chunk also declares the vignets manifest unconditionally, so all chunks rerun after the store is rebuilt, followed by their merge and `make_cat`. +This spends ngmix time on swept tiles, but guarantees that every scheduled reader waits for the writer, even if another input or rerun trigger invalidates a previously completed chunk. +A completion manifest alone cannot prove that a chunk will not run in this invocation. `sp` also records the job holding the campaign's state dir, and releases that job's lock once the job has left the queue; any other lock stops the run. With `-n` both only report. diff --git a/workflow/bin/sp b/workflow/bin/sp index 77cd60ebf..09185cd0b 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -195,9 +195,10 @@ STATE_DIR="${SP_STATE_DIR:-${RUN_DIR}-state}" # tile.smk's "node-local vignette store is missing" guard, so that # manifest is deleted first. tile_make_cat declares it as an input, so # tile_vignets rebuilds the store even if merge_cats is already complete. -# Completed chunks skip this storage-only rebuild (ngmix_vignets in -# tile.smk); unfinished chunks wait for it. Tiles with a final_cat are -# never touched. +# Every ngmix chunk retains its dependency on the writer and reruns after +# it, even if its previous output survived. This costs ngmix time on resume +# but ensures no reader can race the rebuild. Tiles with a final_cat are +# never swept. # * A lock left by a killed in-allocation run is released. sp records the # job that holds the state dir; when that job is no longer in the queue, # its lock is stale. Any other lock (a run sp did not start in an diff --git a/workflow/rules/tile.smk b/workflow/rules/tile.smk index 80771e60e..d2e58bd35 100644 --- a/workflow/rules/tile.smk +++ b/workflow/rules/tile.smk @@ -614,30 +614,6 @@ rule tile_vignets: sp_shell("tile_vignets", f"config_tile_PiViVi_{PSF_MODEL}.ini", check_args=' --run-dir "$SP_LOCAL" --unit {wildcards.tile}') -def ngmix_vignets(wc): - """Keep completed chunks when only the node-local store was swept. - - A missing temp chunk directory is harmless after merge_cats succeeds, but - before that merge it must be rebuilt and therefore needs the store edge. - Unfinished tiles with existing vignets manifests retain normal rerun - propagation. Finished tiles also cut the edge, so a restored manifest's - newer timestamp cannot wake old chunks on the next invocation. As with - the finished-tile exposure cut above, the profile disables the input - trigger. To force a chunk to recompute after a sweep, remove its manifest - too; to rebuild a finished tile, remove its final catalogue first. - """ - root = Path(tile_dir(wc.tile)) - vignets = root / "manifests/tile_vignets.json" - chunk = root / f"manifests/tile_ngmix_{wc.chunk}.json" - chunkdir = root / f"output/run_sp_tile_ngmix_Ng{wc.chunk}u" - merged = root / "manifests/tile_merge_cats.json" - if (not vignets.exists() or tile_finished(wc.tile)) and chunk.exists() and ( - chunkdir.exists() or merged.exists() - ): - return [] - return [str(vignets)] - - # ngmix shape measurement — N chunks per tile (D4). Each chunk LOOKS UP its own # CLOSED catalogue-row range in the file tile_vignets materialised at the top of this # group job (TILE_NGMIX_RANGES); the ranges are knowable only at EXECUTION time, @@ -650,9 +626,9 @@ def ngmix_vignets(wc): rule tile_ngmix: group: TILE_GROUP input: - # A completed chunk doesn't read a rebuilt store on resume. Missing - # chunks keep the manifest edge and wait for this group's writer. - vignets = ngmix_vignets, + # Unconditional: even a completed chunk can be invalidated by another + # input or rerun trigger. Every scheduled reader must wait for the store. + vignets = rules.tile_vignets.output.manifest, sx = rules.tile_detect.output.manifest, output: manifest = f"{TILE_DIR}/manifests/tile_ngmix_{{chunk}}.json", From 3b225146cc8f00f0bf8e16f7a2ed7edc78906122 Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Sat, 17 Oct 2026 10:58:28 -0400 Subject: [PATCH 7/8] docs(sp): keep persistent map recovery in the DAG Resume housekeeping invalidates scratch manifests, not durable map products. A map-specific sweep would protect only --in-allocation and could reopen exp_persist and a costly PSF fit. Document the boundary here; the functional change on feat/pointing-maps declares fragments and recovers them from staged raw MEFs after store reclamation. --- workflow/bin/sp | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/workflow/bin/sp b/workflow/bin/sp index 09185cd0b..8784b7517 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -244,6 +244,11 @@ fi # The resume housekeeping (a stale lock, then stale vignette manifests; the # lock goes first so a campaign still running is never swept). `sp run` calls # it before the snapshot, with its own arguments, so -n only reports. +# This repairs ephemeral stores, not persistent products. In particular, do not +# add an exp_maps sweep here: the map rule must declare its fragment as an +# output and the campaign merge must require it, even after reclamation. +# Missing fragments can then recover from staged raw MEFs without invalidating +# exp_persist or refitting PSFs. The same DAG also protects non-allocation runs. alloc_resume() { local a alloc_dry=0 products_dir n_swept=0 tile m holder holder_file local holder_state="" holder_rc=1 From 26be430a1a143fc7710a92bd4abcc51f7b60f2bd Mon Sep 17 00:00:00 2001 From: Cail Daley Date: Sat, 17 Oct 2026 11:21:43 -0400 Subject: [PATCH 8/8] docs(sp): require deliberate map backfills after reclamation Correct the recovery boundary: reclaimed fragments must have no producer, not an automatic raw-MEF fallback. Code and params edits must never cause Snakemake to remove products whose sources have already gone. --- workflow/bin/sp | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/workflow/bin/sp b/workflow/bin/sp index 8784b7517..e314d33ef 100755 --- a/workflow/bin/sp +++ b/workflow/bin/sp @@ -246,9 +246,11 @@ fi # it before the snapshot, with its own arguments, so -n only reports. # This repairs ephemeral stores, not persistent products. In particular, do not # add an exp_maps sweep here: the map rule must declare its fragment as an -# output and the campaign merge must require it, even after reclamation. -# Missing fragments can then recover from staged raw MEFs without invalidating -# exp_persist or refitting PSFs. The same DAG also protects non-allocation runs. +# output while the store is live, and reclamation must require it first. +# After reclamation, existing fragments must be producer-free immutable leaves; +# missing ones must fail planning with an explicit out-of-DAG backfill command. +# This prevents code/params edits from deleting products whose sources are gone. +# The same lifecycle boundary also protects non-allocation runs. alloc_resume() { local a alloc_dry=0 products_dir n_swept=0 tile m holder holder_file local holder_state="" holder_rc=1