Skip to content

ci: time-limit and retry Playwright installs, cache release-compat apt packages - #1038

Open
owjs3901 wants to merge 6 commits into
mainfrom
ci/playwright-install-retry
Open

owjs3901 wants to merge 6 commits into
mainfrom
ci/playwright-install-retry

Conversation

@owjs3901

@owjs3901 owjs3901 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

release-compat kept getting cancelled at its 30-minute limit inside bunx playwright install chromium firefox webkit --with-deps (run 44485606333 jobs 135428180958 / 135443285660 on #1037, also #1021, #1023, #1024, #1030).

Root cause

The job logs show the browser cache hit (490 MB restored in 3 s), so the browser download was not the problem. The time went to --with-deps: Firefox and WebKit need 185 Ubuntu packages (140 MB) that the ubuntu26 image does not ship (GStreamer, GTK 4, codecs...). apt downloads them one at a time from the runner's mirror (us-east-2.ec2.archive.ubuntu.com or azure.archive.ubuntu.com, depending on the runner), and at times that mirror served a few kB/s, often 5-30 s per small package. After 25 minutes only about 134 of 185 packages had arrived and the job was cancelled. Over the last 40 runs the step normally took 21-62 s but took 578 s, 746 s, 792 s and 1038 s in others. Retrying alone does not help: in this PR's first run three 4-minute attempts got about 12 packages each.

Change

  • .github/scripts/install-playwright.sh is used by every Playwright install in publish.yml (the shared Chromium step, release-compat, webpack-extends-theme):
    • install-deps runs as root with .github/scripts/apt-get-prefetch.sh first on PATH as apt-get. For apt-get install it lists the URIs apt would fetch (--print-uris, resolving the image's mirror+file: list), downloads them 8 at a time, drops any connection slower than 50 kB/s for 15 s and moves on to archive.ubuntu.com / azure.archive.ubuntu.com, checks each file's hash, then runs the real apt-get, which finds everything already downloaded. Any package the prefetch misses is still downloaded by apt.
    • Each attempt has a time limit (timeout runs inside sudo, so it reaches apt-get), with up to 3 attempts. Downloaded packages are kept, so a retry resumes. Between attempts leftover processes are killed and dpkg --configure -a runs. apt is configured with Acquire::Retries 3, a 30 s stall timeout and a dpkg lock wait.
    • Browsers then install as the runner user, also time-limited.
  • Each install step has timeout-minutes: 14.
  • release-compat caches the downloaded .deb files (~/.cache/playwright-apt, keyed on runner image + Playwright browser revisions + ISO week, with a prefix restore key). On a hit nothing is downloaded. The existing browser cache (~/.cache/ms-playwright) is unchanged.
  • Same browsers as before: test:release (e2e/release/drive-browser.mjs) drives chromium, firefox and webkit. No tests, benchmark samples or job behaviour changed. Actions stay on version tags.

Timings (release-compat "Install Playwright browsers")

before this PR
no apt cache, slow mirror 25+ min, cancelled 128 s (prefetch 186/186 in 111 s, apt then fetches 0 B)
apt cache hit n/a 19 s (Need to get 0 B/140 MB)

요약

release-compat이 bunx playwright install chromium firefox webkit --with-deps에서 30분 제한에 걸려 계속 취소되었습니다 (#1037의 run 44485606333 job 135428180958 / 135443285660, 그리고 #1021, #1023, #1024, #1030).

원인

잡 로그를 보면 브라우저 캐시는 적중했고(490 MB를 3초에 복원) 브라우저 다운로드는 문제가 아니었습니다. 시간은 --with-deps에서 쓰였습니다. Firefox와 WebKit에는 ubuntu26 이미지에 없는 Ubuntu 패키지 185개(140 MB)(GStreamer, GTK 4, 코덱 등)가 필요합니다. apt는 이를 러너의 미러(러너에 따라 us-east-2.ec2.archive.ubuntu.com 또는 azure.archive.ubuntu.com)에서 하나씩 받는데, 이 미러가 때때로 수 kB/s로, 작은 패키지 하나에 530초씩 걸려 내려주었습니다. 25분 동안 185개 중 약 134개만 받은 채 잡이 취소되었습니다. 최근 40회 실행에서 이 단계는 보통 2162초였지만 578초, 746초, 792초, 1038초가 걸린 경우도 있었습니다. 재시도만으로는 해결되지 않았습니다. 이 PR의 첫 실행에서 4분짜리 시도 3번이 각각 약 12개 패키지만 받았습니다.

변경

  • .github/scripts/install-playwright.sh를 publish.yml의 모든 Playwright 설치(공용 Chromium 단계, release-compat, webpack-extends-theme)에서 사용합니다.
    • install-deps를 root로 실행하고, PATH 맨 앞에 .github/scripts/apt-get-prefetch.sh를 apt-get으로 둡니다. apt-get install이면 apt가 받을 URI를 뽑아(--print-uris, 이미지의 mirror+file: 목록 해석) 8개씩 병렬로 받습니다. 15초 동안 50 kB/s보다 느린 연결은 끊고 archive.ubuntu.com / azure.archive.ubuntu.com으로 넘어가며, 받은 파일마다 해시를 확인한 뒤 진짜 apt-get을 실행합니다. 진짜 apt-get은 모두 받아진 상태에서 시작하고, 미리 받지 못한 패키지는 apt가 직접 받습니다.
    • 시도마다 시간 제한을 걸고(timeout을 sudo 안에서 실행해야 apt-get까지 닿습니다) 최대 3번 시도합니다. 받은 패키지는 남으므로 재시도는 이어서 받습니다. 시도 사이에는 남은 프로세스를 정리하고 dpkg --configure -a를 실행합니다. apt 설정은 Acquire::Retries 3, 멈춘 연결 30초 타임아웃, dpkg 잠금 대기입니다.
    • 브라우저는 그다음 러너 사용자로 설치하며 여기에도 시간 제한이 있습니다.
  • 각 설치 단계에 timeout-minutes: 14를 둡니다.
  • release-compat은 받은 .deb 파일을 캐시합니다(~/.cache/playwright-apt, 러너 이미지 + Playwright 브라우저 리비전 + ISO 주차 키, 접두사 restore key). 적중하면 아무것도 받지 않습니다. 기존 브라우저 캐시(~/.cache/ms-playwright)는 그대로입니다.
  • 브라우저는 이전과 같습니다. test:release(e2e/release/drive-browser.mjs)가 chromium, firefox, webkit을 모두 씁니다. 테스트, 벤치마크 샘플, 잡 동작은 바꾸지 않았고 액션은 버전 태그를 유지합니다.

소요 시간 (release-compat "Install Playwright browsers")

이전 이 PR
apt 캐시 없음, 느린 미러 25분 이상, 취소 128초 (prefetch 186/186개 111초, 이후 apt가 받을 양 0 B)
apt 캐시 적중 없음 19초 (Need to get 0 B/140 MB)

🤖 Generated with Claude Code

owjs3901 and others added 3 commits November 19, 2026 18:21
…t packages

`release-compat` restored its browser cache but still hung in
`playwright install --with-deps`: Firefox and WebKit need 185 Ubuntu
packages (140 MB) the runner image lacks, and the runner's Ubuntu mirror
served them at a few kB/s until the job was cancelled at 30 minutes.

Every Playwright install now goes through .github/scripts/install-playwright.sh,
which limits each attempt, retries up to three times (downloaded packages
are kept, so a retry resumes), and makes apt drop stalled connections.
release-compat also keeps the downloaded .deb files in actions/cache, so a
normal run installs them without the mirror.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Time-limited retries alone did not get through: the runner's Ubuntu mirror
delivered about 12 of 185 packages per 4-minute attempt. install-deps now
runs as root with apt-get-prefetch.sh first on PATH, which downloads the
packages apt would fetch eight at a time, drops connections slower than
50 kB/s and falls back to archive.ubuntu.com and azure.archive.ubuntu.com,
checks each file's hash, and then runs the real apt-get.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

owjs3901 and others added 2 commits November 19, 2026 18:59
The hosted images give apt `mirror+file:/etc/apt/apt-mirrors.txt/...`
URIs, which curl cannot fetch, so the prefetch downloaded nothing. Read the
mirrors from that list first, then the fallback mirrors, and report how
many packages the prefetch got.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prefetch built the checker name from apt's hash label and ran
`md5sumsum` for MD5Sum hashes, so it discarded every download. And the
`timeout` around `sudo` never reached apt-get, which sudo runs in its own
session: a timed-out attempt kept downloading and held the dpkg lock
through the next attempt. Map each hash label to its checker, run
`timeout` inside sudo, and time-limit the browser download separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@owjs3901
owjs3901 force-pushed the ci/playwright-install-retry branch from 9551e67 to 90d67f7 Compare November 19, 2026 10:19
mktemp creates the prefetched .deb files with mode 0600, so copying them
into the actions/cache directory as the runner user failed. Make them 0644
like apt's own downloads, and copy the cache as root and hand it back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@owjs3901

Copy link
Copy Markdown
Contributor Author

Install step timings from this PR's CI:

  • Before (perf(next-plugin): share what the root layout imports across every route #1037, run 44485606333): browser cache hit, then --with-deps crawled through 185 apt packages until the job was cancelled at ~25 min in the step.
  • This PR, first version (run 44495743685): time-limited retries alone, 3 × 4 min attempts got ~12 packages each, failed at 725 s. That is why the parallel prefetch was added.
  • This PR, no apt cache (run 44507871041, job 135504094099): 128 s, prefetch got 186/186 packages in 111 s.
  • This PR, apt cache hit (rerun job 135515187674): 19 s, 186 cached packages seeded, Need to get 0 B/140 MB.
  • Chromium-only jobs in the same run: 9-112 s; webpack-extends-theme: 105 s.

이 PR CI의 설치 단계 소요 시간:

  • 이전 (perf(next-plugin): share what the root layout imports across every route #1037, run 44485606333): 브라우저 캐시는 적중했지만 --with-deps가 apt 패키지 185개를 아주 느리게 받다가 이 단계에서 약 25분 만에 잡이 취소됨.
  • 이 PR 첫 버전 (run 44495743685): 시간 제한 재시도만으로는 4분 시도 3번이 각각 약 12개씩만 받고 725초에 실패. 그래서 병렬 prefetch를 추가함.
  • 이 PR, apt 캐시 없음 (run 44507871041, job 135504094099): 128초, prefetch가 186/186개를 111초에 받음.
  • 이 PR, apt 캐시 적중 (재실행 job 135515187674): 19초, 캐시된 패키지 186개 사용, Need to get 0 B/140 MB.
  • 같은 실행의 Chromium 전용 잡: 9~112초, webpack-extends-theme: 105초.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant