[Copilot]

Fixes #1102

Problem

When a Slurm job left squeue and its _o.h5 file wasn't yet visible on the shared filesystem, the task was failed with "terminated without producing output", even though the job had completed successfully.

Change

• When a Slurm job is no longer in the queue, query sacct for its state.
• COMPLETED: don't fail; allow a 60 s grace period for the output file to appear (it is loaded as soon as it does). If it never appears, the task fails after the grace period.
• Still queued/running states: don't fail.
• Any other state (TIMEOUT, FAILED, OOM, ...) or unknown: fail promptly as before.
• Flux behavior is unchanged.

Testing

Added unit tests (mocked queue/accounting) for delayed output, failed jobs and grace-period expiry; the targeted file-backend tests pass. A live Slurm reproduction wasn't possible as no Slurm is available in this environment.

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: ba87328b-6ff3-4358-a66d-f8cf6dfebfbe

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[codecov]

Codecov Report

❌ Patch coverage is 69.69697% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.87%. Comparing base (aaff878) to head (35167ab).

Files with missing lines	Patch %	Lines
src/executorlib/standalone/command_pysqa.py	69.69%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1104      +/-   ##
==========================================
- Coverage   94.24%   93.87%   -0.37%     
==========================================
  Files          39       39              
  Lines        2190     2221      +31     
==========================================
+ Hits         2064     2085      +21     
- Misses        126      136      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jan-janssen]

@copilot Can the same analysis be extended to support the flux job scheduler and not only the slurm scheduler? In particular it would be great to use the pysqa package which should already provide an abstraction layer for checking the status of the job.

[Copilot]

@copilot Can the same analysis be extended to support the flux job scheduler and not only the slurm scheduler? In particular it would be great to use the pysqa package which should already provide an ...

Extended the shared output grace period to Flux using pysqa’s finished status; error still fails immediately. Added coverage for delayed output and bounded expiry. Targeted tests and mypy pass. Commit: 35167ab.