Repository navigation
perf(plugins): read each source once across the prepasses and compilers - #1033
Merged
Merged
Conversation
The import graph, the class-slot reservation and the prewarm each read the project's sources from disk, and a Next production build runs all of them again for its server, edge and client compilers. They now read through a stamp-checked cache that one build shares: plugin-utils' `readSource` serves a file again from the cache while its modification stamp is unchanged, inside `withSourceCache`. Next hands one cache to the webpack plugins of all its production compilers and uses one for the Turbopack setup; a plain webpack build shares one per preparation. Watch and dev builds still start from a fresh cache for each compilation. The module resolver keeps its own cache and reads misses through the shared one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Before the bundler starts, the import graph, the class-slot reservation and the prewarm each read the project's sources from disk, and the prewarm re-reads the files the reservation just read. A Next production build then runs all of them again for each of its three compilers (server, edge-server, client), because the engine is reset between compilers. So every source was read about 6–9 times per
next build.They now read through one stamp-checked cache per build:
plugin-utils: newreadSource(path)/withSourceCache(run, cache). InsidewithSourceCache, a file is served from the cache while itsmtime/ctime/size stamp is unchanged. Checking the stamp costs about a fifth of reading the file (300 files: stat 2–3 ms vs read 10–16 ms on Windows). Outside it,readSourcereads the disk as before. The import graph, the reservation discovery, Next's prewarm collection and the module resolver's cache misses read through it.next-plugin: one cache for a production build's webpack compilers, handed to eachDevupUIWebpackPlugin(sourceCache, adapter-owned likeproductionPrewarmFiles); one for the Turbopack config-phase graph and prewarm.webpack-plugin:initializeand each compilation's preparation (reservation + prewarm) run inside the build's cache, or a fresh one per call.sourceCacheis only passed when!dev), so each compilation still starts from a fresh cache and nothing can go stale between edits.Engine work (scratch reservation and prewarm extraction) still runs per compiler: the engine is reset between compilers and the server/client resolve conditions differ. Sharing it would change the engine-state contract, so it is out of scope here.
Measurements (local, Windows, 4 alternating A/B runs, native engine)
CPU sampled in Devup frames (
--cpu-prof, all processes of the build):next-devup-ui(webpack)Flex/Box/Textcomponents)next.setup(DEVUP_UI_PROFILE, single-turbo)Per compiler on the 300-component app (instrumented, main): the reservation discovery took 36–70 ms, of which reads were most, and the prewarm read 300 files again (23–31 ms) every compiler. The local machine is shared with other builds, so absolute numbers are noisy; the direction held in every pair. On Linux CI reads are cheaper, so expect a smaller gain there. The CI benchmark job of this PR shows it.
Output
Byte-identical CSS and page JS (md5) on
next-devup-ui,next-devup-ui-single,next-devup-ui-single-turbo,next-devup-ui-turbo,next-devup-ui-collapseand the 300-component app.Notes
Test plan
source-cache.test.ts: no cache outside a run, a cache hit under any spelling of the path, a changed file read again, a deleted file reported and evicted, nested runs keep the outer cache, the active cache is restored when the run throwsMapshared by the server and client compilers of a production build, none for devbun test packages/plugin-utils packages/next-plugin packages/webpack-plugin: 5213 pass (with rsbuild-plugin), 0 fail; 100% for every changed source file요약
번들러가 시작되기 전에 import graph, class-slot 예약(reservation), prewarm이 각각 프로젝트 소스를 디스크에서 읽고, prewarm은 예약이 방금 읽은 파일을 다시 읽습니다. Next 프로덕션 빌드는 컴파일러 사이에 엔진을 리셋하므로 이 과정을 컴파일러 3개(server, edge-server, client)마다 다시 돕니다. 그래서
next build한 번에 소스 하나를 약 6~9번 읽었습니다.이제 빌드당 하나인, stamp로 검증하는 캐시를 거쳐 읽습니다.
plugin-utils:readSource(path)/withSourceCache(run, cache)를 추가했습니다.withSourceCache안에서는mtime/ctime/크기 stamp가 그대로인 동안 캐시에서 내줍니다. stamp 확인 비용은 읽기의 약 1/5입니다(Windows에서 300개 파일: stat 23 ms, read 1016 ms). 밖에서는 지금처럼 디스크를 읽습니다. import graph, 예약 탐색, Next prewarm 수집, 모듈 resolver의 캐시 미스가 이걸 거쳐 읽습니다.next-plugin: 프로덕션 빌드의 webpack 컴파일러들이 캐시 하나를 함께 쓰고, 각DevupUIWebpackPlugin에 넘깁니다(sourceCache,productionPrewarmFiles처럼 어댑터 전용). Turbopack config 단계의 graph와 prewarm도 캐시 하나를 씁니다.webpack-plugin:initialize와 컴파일마다의 준비(예약 + prewarm)가 빌드의 캐시 안에서, 없으면 호출마다 새 캐시 안에서 돕니다.!dev일 때만 넘김). 그래서 컴파일마다 새 캐시로 시작하고, 편집 사이에 오래된 내용이 남을 수 없습니다.엔진 작업(예약의 scratch 추출, prewarm 추출)은 여전히 컴파일러마다 돕니다. 컴파일러 사이에 엔진이 리셋되고 server/client의 resolve 조건이 다르기 때문입니다. 이걸 공유하려면 엔진 상태 계약이 바뀌므로 이 PR 범위 밖입니다.
측정 (로컬, Windows, A/B 교대 4회, native 엔진)
Devup 프레임에서 샘플링된 CPU(
--cpu-prof, 빌드의 모든 프로세스):next-devup-ui(webpack)Flex/Box/Text컴포넌트 300개 생성)next.setup(DEVUP_UI_PROFILE, single-turbo)컴포넌트 300개 앱에서 컴파일러마다(main 계측): 예약 탐색이 36
70 ms였고 대부분이 읽기였습니다. prewarm은 컴파일러마다 파일 300개를 다시 읽었습니다(2331 ms). 로컬 머신을 다른 빌드와 함께 쓰고 있어 절대값은 흔들리지만, 모든 짝에서 방향은 같았습니다. Linux CI에서는 읽기가 더 싸서 효과가 더 작을 것입니다. 이 PR의 CI benchmark job에 나옵니다.출력
next-devup-ui,next-devup-ui-single,next-devup-ui-single-turbo,next-devup-ui-turbo,next-devup-ui-collapse, 컴포넌트 300개 앱 모두 CSS와 페이지 JS가 바이트 단위로 같습니다(md5).참고
테스트
source-cache.test.ts: 실행 밖에서는 캐시 없음, 경로 표기가 달라도 캐시 적중, 바뀐 파일은 다시 읽음, 삭제된 파일은 오류를 내고 캐시에서 빠짐, 중첩 실행은 바깥 캐시 유지, 실행이 throw해도 활성 캐시 복원Map하나를 공유하고 dev는 없음bun test packages/plugin-utils packages/next-plugin packages/webpack-plugin: 5213 pass (with rsbuild-plugin), 0 fail; 바뀐 소스 파일 모두 100%🤖 Generated with Claude Code