Skip to content

Decompress each chunk once in the response leak scan - #1306

Merged
theoephraim merged 1 commit into
mainfrom
incremental-leak-scan-decompress
Nov 16, 2026
Merged

theoephraim merged 1 commit into
mainfrom
incremental-leak-scan-decompress

Conversation

@theoephraim

Copy link
Copy Markdown
Member

Fixes #1305. Approach and diagnosis by @andreastande, thanks!

The patched ServerResponse pushed each compressed chunk onto zlibChunks and re-decompressed the whole buffer on every write, so the scan was quadratic in response size. Streaming SSR (e.g. Next.js with compression, which flushes after every rendered chunk) hit this hard.

Now each response gets one decompressor that is fed a chunk at a time. Node's public sync zlib API closes the stream after each call, so this drives the stream's native handle directly (_handle.writeSync / _writeState / _chunkSize, the same loop as node's processChunkSync minus the close). If those internals are missing, it falls back to the previous whole-buffer path. A stream that fails to decode stops being scanned (fails open, as before).

Verified:

  • internals present and output byte-exact for gzip, deflate, br, zstd on Node 22.3.0, 22.15, 24, 25 and Bun 1.3.14
  • 2 MB response in 1024 flushes over real HTTP: secret split across flushes still caught on all encodings in node and bun; server time on node dropped from 403 to 128 ms (gzip), 1315 to 268 ms (br), 360 to 83 ms (zstd)
  • existing patch-server-response tests pass on both the incremental and the forced-fallback path

Known gap: concatenated gzip members in one response only get the first member scanned (raw handle doesn't reset between members).

The patched ServerResponse re-decompressed every compressed chunk seen so
far on each write, so scan cost grew with the square of the response size.
Keep one decompressor per response and feed it one chunk at a time by
driving the stream's native handle (node and bun both expose it). Falls
back to the old path when those internals are missing.

Approach proposed by @andreastande in #1305.

Fixes #1305

Co-authored-by: andreastande <114910149+andreastande@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

bumpy-frog

The changes in this PR will be included in the next version bump.

patch Patch releases

  • varlock 1.25.0 → 1.25.1

Bump files in this PR

Click here if you want to add another bump file to this PR


This comment is maintained by bumpy.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Bundle size

⚠️ grows the bundle by 19.5 KB (+0.4%)

Metric main This PR Δ
Total dist 5239.6 KB 5259.1 KB +19.5 KB (+0.4%)
JS 1998.6 KB 2007.4 KB +8.8 KB (+0.4%)
Sourcemaps 3097.1 KB 3107.8 KB +10.7 KB (+0.3%)
Type defs 143.9 KB 143.9 KB —
Other 0.0 KB 0.0 KB —

dist/ only; native binaries are versioned separately and not counted here.

@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/varlock@1306

commit: 5b66f8d

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes: Reviewed the incremental compressed-response leak scanner and its release entry.

  • Incremental decompression: Reuses a native decompressor per response instead of replaying all prior compressed chunks.
  • Scanning and compatibility: Preserves streaming UTF-8 decoding and cross-chunk detection, with the previous decompression path available when internals are missing.
  • Release entry: Marks the performance improvement for a varlock patch release.

All 70 focused tests passed on Node and Bun. The concatenated-gzip limitation mentioned in the description did not reproduce on Node 24.22.1 or Bun 1.4.2, including member-aligned and byte-at-a-time chunks.

Pullfrog  | View workflow run | Using gpt-6.1-sol | 𝕏

@theoephraim
theoephraim merged commit fa41a23 into main Nov 16, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Leak scan re-decompresses the whole response on every chunk (quadratic on large compressed responses)

1 participant