Repository navigation
feat(cli): add --max-failures to test and trigger [RED-1227] [ship] - #1561
Merged
Merged
Conversation
…[RED-1227] AbstractCheckRunner takes an optional maxFailures limit. Once that many checks have a failed final outcome (failed result, run error or timeout; retried attempts, degraded and cancelled results do not count) while other checks are still unfinished, the run stops right away: later results are dropped, timeouts are cleared, and MAX_FAILURES_REACHED is emitted before RUN_FINISHED. When the failure that reaches the limit is the last check, the run finishes normally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reporters get onMaxFailuresReached. Checks without a result are then shown as "not finished" in the list, dot and ci summaries, which also say the run stopped after reaching the max failures limit. The json and github reports no longer assume every check has a result: they emit a "Not finished" row, and the json report adds a top-level stoppedEarly field when the run stopped early. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`checkly test` and `checkly trigger` take --max-failures=<n>, and the Checkly config takes cli.maxFailures (the flag wins). Once n checks have failed, the CLI stops waiting for the rest, writes its reports, asks Checkly to cancel the test session and exits with an error. Checks that pass after a retry do not count. The limit cannot be combined with --detach, which never waits for results, or with --update-snapshots on `checkly test`, where checks the run stopped waiting for would keep their old snapshots. A configured limit is ignored in those cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorccu
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: RED-1227
Summary
checkly testandcheckly triggerget--max-failures=<n>, and the Checkly config getscli.maxFailures, where the flag wins. Oncenchecks have failed, the CLI stops waiting for the rest. It writes its reports, asks Checkly to cancel the test session, and exits 1. In CI, a broken change then fails early instead of after the whole suite.--detach, which never waits for results. Oncheckly testit also can't be combined with--update-snapshots, because checks the run stopped waiting for would keep their old snapshots. A configured limit is ignored in those cases, with a notice for--update-snapshots.checkly pw-testis unchanged: Playwright's own--max-failuresalready covers fail-fast inside a suite.Review guide (one commit per step)
AbstractCheckRunnercounts final failures. The stop is decided synchronously in theCHECK_FINISHEDlistener, which every terminal path emits right after the outcome event. Once the run has stopped, no later or in-flight result is reported, including a final result that was still fetching its logs.NOT_FINISHEDin every summary, and theciheader says the run stopped. Thejsonandgithubreporters used to throw on a check without a result; they now emit a "Not finished" row. JSON output gains a"result": "Not finished"value and a top-levelstoppedEarly: { reason: 'maxFailures', maxFailures }field, present only when the run stopped early.prepareMaxFailures), reporter wiring, and an awaited best-effort cancel through the newcancelTestSessionBestEffort, which the Ctrl+C handler now reuses too. Config codegen keepsmaxFailureswhen it regenerates the config.Testing
triggerwiring, and the cancel helper.--max-failures=1, the run must exit 1 well before 90s, and the JSON report must showstoppedEarlyand a "Not finished" row.Docs: checkly/docs#590 (merge after this PR is released).
🤖 Generated with Claude Code