[Svitlana-Dykun]

Summary

Fixes the test healer missing hard failures from earlier attempts when an evidence run was re-run. When a GitHub Actions workflow run is re-run, the run ID stays the same but the attempt number increments. Previously, the healer only considered the latest attempt, missing tests that failed in earlier attempts.

Problem

The Playwright test healer in auto mode was only considering the latest attempt of the evidence workflow run. When the evidence run was re-run:

• A test that failed hard in an earlier attempt but not in the latest one was missed
• Artifacts from earlier attempts were not considered
• Job logs from earlier attempts were not scanned

This meant healing candidates were being overlooked.

Solution

Created a new run-attempts.js module that:

• Queries all attempts of a workflow run using the GitHub API
• Aggregates failures and artifacts across all attempts
• Downloads artifacts by unique ID instead of name to handle duplicates

Updated the healer scripts to:

• collect-log-hard-failures.js: Scan job logs from all attempts and track which attempts each test failed in
• find-test-evidence.js: Query job logs to determine which attempts a test failed in, then prioritize artifacts from those specific attempts
• restore-nightly-playwright-builds.js: Handle duplicate artifact names by downloading by ID instead of name

How Attempt/Artifact Selection Works

Evidence Selection (Key Refinement)

When searching for evidence for a specific test:

1. Query job logs to find which attempt(s) the test actually failed in
2. Prioritize artifacts from those failed attempts first
3. Fall back to other attempts only if no evidence found in failed attempts
4. Record which attempt the evidence came from

This correctly handles scenarios like:

• Test fails in attempt 1, passes in attempt 2 → uses evidence from attempt 1
• Test fails in both attempts → uses evidence from the latest failed attempt (attempt 2)
• Test not found in logs → falls back to checking all attempts

Artifact Download

When multiple attempts exist:

1. Job logs: All failed jobs from all attempts are scanned with attempt numbers tracked
2. Artifacts: All artifacts are listed with their attempt numbers via workflow_run.run_attempt
3. Downloads: Artifacts are downloaded by unique artifact ID rather than name to handle duplicates correctly

The healer prefers:

1. Evidence with traces over log-only evidence
2. Evidence from attempts where the test actually failed
3. Evidence from later attempts when equally valid
4. Earlier shard indices as a tiebreaker

Why restore-nightly-playwright-builds.js Was Changed

The restore-nightly-playwright-builds.js change is necessary for this fix. When downloading build artifacts by name from a re-run using gh run download -n <artifact-name>, the command fails or gets confused when multiple artifacts with the same name exist across attempts.

By switching to downloadArtifactById(), we:

• Handle duplicate artifact names correctly
• Always get the specific artifact we want (latest attempt for builds)
• Prevent download failures that would block verification

Without this change, the healer would fail to restore builds from re-run evidence runs, preventing verification even when the fix is correct.

Testing

• Updated existing tests in find-test-evidence.test.js to include attempt field
• Added new test: "handles the case where test failed in attempt 1 but not attempt 2"
• Added new test: "prefers later failed attempt when test failed in multiple attempts"
• Added collect-log-hard-failures.test.js with tests for parseHardFailureTitles
• Added basic module structure tests for run-attempts.test.js
• All 359 tests pass (6 new tests added)

Jira

https://elementor.atlassian.net/browse/ED-25748

  

[wiz-9a149474ff]

Wiz Scan Summary

Scanner	Findings
Vulnerabilities	-
Sensitive Data	-
Secrets	-
IaC Misconfigurations	-
SAST Findings	1
Software Management Findings	-
Total	1

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.