Skip to content

mctest/mcrun: --scan_split fixes (statfactor nesting, NeXus merge) - #2891

Merged
willend merged 4 commits into
mccode-dev:mainfrom
willend:2026-10-mctest-statfactor-scan-split
Oct 23, 2026
Merged

willend merged 4 commits into
mccode-dev:mainfrom
willend:2026-10-mctest-statfactor-scan-split

Conversation

@willend

@willend willend commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Two fixes around --scan_split, so that split scans match the normal (serial) scan.

1. mctest: --scan_split=auto for %Scan tests only

6253f3f (--statfactor) inserted its block between the %Scan branch and the --scan_split=auto that belongs to it, so:

  • with --statfactor (and no MPI), every test got --scan_split=auto: a plain %Example then fails in mcrun with CRITICAL: No interval range specified
  • without --statfactor, no %Scan test was run split

Fix: move the two lines back under if test.scan: (as introduced in e5e7b58).

mctest --no-mpi, TAS1_Diff_Slit runtime errors %Example split %Scan split
--statfactor 0.05 0 (before: 1) no yes
no --statfactor 0 no yes (before: no)

2. mcrun: NeXus --scan_split merge as a serial NeXus scan writes it

Each parallel step writes dir/i/mccode.h5 with its data as entry1; these were merged with h5copy (/entry1 → /entryN), keeping references to entry1 and the step dir:

  • the target attribute of every linked monitor in entry2.. pointed to /entry1/instrument/.... nexusformat / NeXpy follow it and showed entry1's data for every scan step (h5py-based readers were not affected)
  • Directory, name, title, file_name named the step dir dir/i
  • the step dirs with their mccode.h5 copies were left behind

merge_nexus_steps() now copies the entries with h5py, rewrites those references (keeping fixed-length strings) and removes the step dirs.

Failed steps: a failed step can leave a partial, unreadable dir/i/mccode.h5, on which the merge (by file existence, with h5copy as well) crashed mcrun before mccode.dat was written. Now only the steps with detector values (the rows of mccode.dat) are merged, numbered entry1..N as a serial NeXus scan does, and all step dirs are kept when a step failed.

middle of 3 steps fails files mccode.dat mccode.h5
NeXus normal (blueprint) mccode.dat, mccode.h5 2 rows entry1 (w=0.02), entry2 (w=0.06)
NeXus split, before 0/ 1/ 2/, broken mccode.h5 missing (mcrun crashed) -
NeXus split, now mccode.dat, mccode.h5 + kept 0/ 1/ 2/ 2 rows as normal, item by item apart from times

Worker logging: split-scan steps run in multiprocessing workers. With the spawn start method (macOS, Windows) these lacked mcrun's log setup: warnings were printed bare (Scan step 1 failed ..., no WARNING:) and their INFO lines (e.g. call to mpirun failed) were lost. A Pool initializer now sets up mcrun's logging where the worker has none (with fork, Linux, it is inherited, so no duplicate handler), and steps are numbered i+1/N as in the serial scan: WARNING: Scan step 2/3 failed (...). Checked on macOS with spawn and a forced fork: one formatted line per message.

Tested (macOS, mcstas, same seed): 3-point scans of a 0D/1D/2D test instrument and of Test_NeXus_output (+ a dummy scan parameter: 0D, 1D, 2D, Monitor_nD 2D/3D BINS maps, event list), normal vs --scan_split=auto:

  • ASCII: identical file layout and data (already before this PR)
  • NeXus: same layout (mccode.dat, mccode.h5 only), and mccode.h5 identical item by item (all groups, datasets, attributes incl. types) apart from times; every target resolves to its own entry; nexusformat reads the same data as h5py
  • per-step seeds identical in all four cases (1234, 2258, 3282)

Dependency: NeXus --scan_split scans now need h5py in mcrun (instead of the h5copy tool). In mcstas-suite-feedstock h5py currently only comes indirectly (mcstas → nexpy, mcstas-vis → mcstasscript); it should be added to mcstas-core (and likewise for McXtrace).

Not tested on Linux/Windows or with McXtrace (shared code).

🤖 Generated with Claude Code

willend and others added 2 commits October 23, 2026 13:01
6253f3f (--statfactor) put its block between the %Scan branch and the
--scan_split=auto that belongs to it, so the option was added to every
test when --statfactor was given (a %Example then failed in mcrun with
"No interval range specified"), and to no %Scan test without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each parallel step writes its own dir/i/mccode.h5 with its data as entry1.
These were merged into dir/mccode.h5 with h5copy (/entry1 -> /entryN),
which keeps references to entry1 and the step dir:

- the 'target' attribute of each linked monitor in entry2.. pointed to
  /entry1/instrument/...: nexusformat/NeXpy follow it and showed entry1's
  data for every scan step (h5py-based readers were not affected)
- Directory/name/title/file_name named the step dir dir/i
- the step dirs with their mccode.h5 copies were left behind

merge_nexus_steps() now copies the entries with h5py, rewrites those
references (keeping fixed-length strings) and removes the step dirs, so
the result matches a serial NeXus scan item by item apart from times.
h5py is needed for NeXus --scan_split scans instead of the h5copy tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@willend willend changed the title mctest: --scan_split=auto for %Scan tests only, not with --statfactor mctest/mcrun: --scan_split fixes (statfactor nesting, NeXus merge) Oct 23, 2026
willend and others added 2 commits October 23, 2026 13:23
…n failure

A failed step can leave a partial, unreadable dir/i/mccode.h5: the merge
(by file existence, also with h5copy before) then crashed mcrun before
mccode.dat was written. Merge only the steps with detector values (the
rows of mccode.dat), numbered entry1..N as a serial NeXus scan does, and
keep all step dirs when a step failed, for inspection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The step workers of a split scan run in their own processes. With the
spawn start method (macOS, Windows) they lacked mcrun's log setup, so
their warnings were printed bare (no "WARNING:") and their INFO messages
were lost; they also numbered steps from 0 without the total. A Pool
initializer now sets up mcrun's logging where the worker has none (fork,
Linux, already inherits it), and the step messages read
"Scan step i+1/N", as in the serial scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@willend
willend merged commit b328817 into mccode-dev:main Oct 23, 2026
14 of 15 checks passed
@willend
willend deleted the 2026-10-mctest-statfactor-scan-split branch October 23, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant