Repository navigation
Conversation
…t packages `release-compat` restored its browser cache but still hung in `playwright install --with-deps`: Firefox and WebKit need 185 Ubuntu packages (140 MB) the runner image lacks, and the runner's Ubuntu mirror served them at a few kB/s until the job was cancelled at 30 minutes. Every Playwright install now goes through .github/scripts/install-playwright.sh, which limits each attempt, retries up to three times (downloaded packages are kept, so a retry resumes), and makes apt drop stalled connections. release-compat also keeps the downloaded .deb files in actions/cache, so a normal run installs them without the mirror. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Time-limited retries alone did not get through: the runner's Ubuntu mirror delivered about 12 of 185 packages per 4-minute attempt. install-deps now runs as root with apt-get-prefetch.sh first on PATH, which downloads the packages apt would fetch eight at a time, drops connections slower than 50 kB/s and falls back to archive.ubuntu.com and azure.archive.ubuntu.com, checks each file's hash, and then runs the real apt-get. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
The hosted images give apt `mirror+file:/etc/apt/apt-mirrors.txt/...` URIs, which curl cannot fetch, so the prefetch downloaded nothing. Read the mirrors from that list first, then the fallback mirrors, and report how many packages the prefetch got. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prefetch built the checker name from apt's hash label and ran `md5sumsum` for MD5Sum hashes, so it discarded every download. And the `timeout` around `sudo` never reached apt-get, which sudo runs in its own session: a timed-out attempt kept downloading and held the dpkg lock through the next attempt. Map each hash label to its checker, run `timeout` inside sudo, and time-limit the browser download separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
owjs3901
force-pushed
the
ci/playwright-install-retry
branch
from
November 19, 2026 10:19
9551e67 to
90d67f7
Compare
mktemp creates the prefetched .deb files with mode 0600, so copying them into the actions/cache directory as the runner user failed. Make them 0644 like apt's own downloads, and copy the cache as root and hand it back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Install step timings from this PR's CI:
이 PR CI의 설치 단계 소요 시간:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
release-compatkept getting cancelled at its 30-minute limit insidebunx playwright install chromium firefox webkit --with-deps(run 44485606333 jobs 135428180958 / 135443285660 on #1037, also #1021, #1023, #1024, #1030).Root cause
The job logs show the browser cache hit (490 MB restored in 3 s), so the browser download was not the problem. The time went to
--with-deps: Firefox and WebKit need 185 Ubuntu packages (140 MB) that theubuntu26image does not ship (GStreamer, GTK 4, codecs...). apt downloads them one at a time from the runner's mirror (us-east-2.ec2.archive.ubuntu.comorazure.archive.ubuntu.com, depending on the runner), and at times that mirror served a few kB/s, often 5-30 s per small package. After 25 minutes only about 134 of 185 packages had arrived and the job was cancelled. Over the last 40 runs the step normally took 21-62 s but took 578 s, 746 s, 792 s and 1038 s in others. Retrying alone does not help: in this PR's first run three 4-minute attempts got about 12 packages each.Change
.github/scripts/install-playwright.shis used by every Playwright install inpublish.yml(the shared Chromium step,release-compat,webpack-extends-theme):install-depsruns as root with.github/scripts/apt-get-prefetch.shfirst on PATH asapt-get. Forapt-get installit lists the URIs apt would fetch (--print-uris, resolving the image'smirror+file:list), downloads them 8 at a time, drops any connection slower than 50 kB/s for 15 s and moves on toarchive.ubuntu.com/azure.archive.ubuntu.com, checks each file's hash, then runs the realapt-get, which finds everything already downloaded. Any package the prefetch misses is still downloaded by apt.timeoutruns insidesudo, so it reaches apt-get), with up to 3 attempts. Downloaded packages are kept, so a retry resumes. Between attempts leftover processes are killed anddpkg --configure -aruns. apt is configured withAcquire::Retries 3, a 30 s stall timeout and a dpkg lock wait.timeout-minutes: 14.release-compatcaches the downloaded.debfiles (~/.cache/playwright-apt, keyed on runner image + Playwright browser revisions + ISO week, with a prefix restore key). On a hit nothing is downloaded. The existing browser cache (~/.cache/ms-playwright) is unchanged.test:release(e2e/release/drive-browser.mjs) drives chromium, firefox and webkit. No tests, benchmark samples or job behaviour changed. Actions stay on version tags.Timings (
release-compat"Install Playwright browsers")Need to get 0 B/140 MB)요약
release-compat이bunx playwright install chromium firefox webkit --with-deps에서 30분 제한에 걸려 계속 취소되었습니다 (#1037의 run 44485606333 job 135428180958 / 135443285660, 그리고 #1021, #1023, #1024, #1030).원인
잡 로그를 보면 브라우저 캐시는 적중했고(490 MB를 3초에 복원) 브라우저 다운로드는 문제가 아니었습니다. 시간은
--with-deps에서 쓰였습니다. Firefox와 WebKit에는ubuntu26이미지에 없는 Ubuntu 패키지 185개(140 MB)(GStreamer, GTK 4, 코덱 등)가 필요합니다. apt는 이를 러너의 미러(러너에 따라us-east-2.ec2.archive.ubuntu.com또는azure.archive.ubuntu.com)에서 하나씩 받는데, 이 미러가 때때로 수 kB/s로, 작은 패키지 하나에 530초씩 걸려 내려주었습니다. 25분 동안 185개 중 약 134개만 받은 채 잡이 취소되었습니다. 최근 40회 실행에서 이 단계는 보통 2162초였지만 578초, 746초, 792초, 1038초가 걸린 경우도 있었습니다. 재시도만으로는 해결되지 않았습니다. 이 PR의 첫 실행에서 4분짜리 시도 3번이 각각 약 12개 패키지만 받았습니다.변경
.github/scripts/install-playwright.sh를publish.yml의 모든 Playwright 설치(공용 Chromium 단계,release-compat,webpack-extends-theme)에서 사용합니다.install-deps를 root로 실행하고, PATH 맨 앞에.github/scripts/apt-get-prefetch.sh를apt-get으로 둡니다.apt-get install이면 apt가 받을 URI를 뽑아(--print-uris, 이미지의mirror+file:목록 해석) 8개씩 병렬로 받습니다. 15초 동안 50 kB/s보다 느린 연결은 끊고archive.ubuntu.com/azure.archive.ubuntu.com으로 넘어가며, 받은 파일마다 해시를 확인한 뒤 진짜apt-get을 실행합니다. 진짜apt-get은 모두 받아진 상태에서 시작하고, 미리 받지 못한 패키지는 apt가 직접 받습니다.timeout을sudo안에서 실행해야 apt-get까지 닿습니다) 최대 3번 시도합니다. 받은 패키지는 남으므로 재시도는 이어서 받습니다. 시도 사이에는 남은 프로세스를 정리하고dpkg --configure -a를 실행합니다. apt 설정은Acquire::Retries 3, 멈춘 연결 30초 타임아웃, dpkg 잠금 대기입니다.timeout-minutes: 14를 둡니다.release-compat은 받은.deb파일을 캐시합니다(~/.cache/playwright-apt, 러너 이미지 + Playwright 브라우저 리비전 + ISO 주차 키, 접두사 restore key). 적중하면 아무것도 받지 않습니다. 기존 브라우저 캐시(~/.cache/ms-playwright)는 그대로입니다.test:release(e2e/release/drive-browser.mjs)가 chromium, firefox, webkit을 모두 씁니다. 테스트, 벤치마크 샘플, 잡 동작은 바꾸지 않았고 액션은 버전 태그를 유지합니다.소요 시간 (
release-compat"Install Playwright browsers")Need to get 0 B/140 MB)🤖 Generated with Claude Code