Skip to content

zlib: re-enable CRC32 SIMD optimization - #67148

Open
ptweezy wants to merge 1 commit into
nodejs:mainfrom
ptweezy:zlib-reenable-crc32-simd
Open

ptweezy wants to merge 1 commit into
nodejs:mainfrom
ptweezy:zlib-reenable-crc32-simd

Conversation

@ptweezy

@ptweezy ptweezy commented Oct 30, 2026 •

Copy link
Copy Markdown

Since #49511, x86 builds compile the bundled zlib without CRC32_SIMD_SSE42_PCLMUL. That worked around the heap corruption in #45268, but left gzip, gunzip, and zlib.crc32() on the portable CRC path.

The corruption came from crc_fold_copy() in deps/zlib/crc_folding.c, which handled the trailing len % 16 bytes of a copy with a 16-byte unaligned store—writing up to 15 bytes past the destination. With gzip at level 0 (as in the #45268 repro using createGzip({ level: 0 })), deflate_stored() copies input directly into strm->next_out via read_buf() → copy_with_crc(), landing the overrun past the end of the caller's output buffer. Chromium replaced that store with zmemcpy(), and Node picked up the fix in #52123 (v22.0.0, backported to v20.13.0 and v18.20.3). The CRC values themselves were always correct; the out-of-bounds store was the only bug.

This PR:

  • Re-enables zlib_crc32_simd in deps/zlib/zlib.gyp. Because the commented-out block predates clang-cl support on Windows (build: add option to enable clang-cl on Windows #52870), this also passes -msse4.2 -mpclmul to clang-cl via msvs_settings, matching zlib_adler32_simd.
  • Runs zlib's CPU feature detection on binding initialization. Bundled zlib only runs detection in deflateInit2() or when crc32()/adler32() is called with a NULL buffer; without this, standalone zlib.crc32() calls use the portable path until a stream is created.
  • Adds test-zlib-gzip-stored-output-boundary.js, which runs level-0 gzip into output buffers ending at every offset modulo 16.

Verification (AMD Ryzen 7 9800X3D, Ubuntu 24.04, GCC 13.3, Valgrind 3.22):

  • Standalone harness (x86_cpu_enable_simd=1): Built bundled zlib with the exact defines and per-file SIMD flags from zlib.gyp. Round-tripped 29,340 gzip streams (all levels, whole and piecewise input, into exact-size malloc() buffers at every mod-16 end offset) and verified gzip trailers plus 335,766 crc32() calls against a reference implementation (lengths 0–4200, alignments 0–63, incremental updates). Clean under both ASan and Valgrind (0 errors).
  • Negative control: Restoring the old 16-byte store triggers a heap-buffer-overflow under ASan and invalid writes under Valgrind at crc_folding.c:419 (via read_buf()).
  • Regression test: test-zlib-gzip-stored-output-boundary.js aborts the official v19.0.0 and v20.6.0 linux-x64 binaries on every run with glibc heap-corruption errors (free(): invalid pointer on this machine; Memory corruption and crash when streaming to zlib #45268 reported an _int_malloc assertion). It passes on v20.8.0 (SIMD disabled) and on this branch.
  • Test suite: Full Linux build of this branch (bundled zlib 1.3.2.1-motley-9f9f52c): the zlib_crc32_simd target builds and links, and all 100 test-zlib* tests pass under tools/test.py.
  • Lint: make lint-js on the new test and tools/cpplint.py src/node_zlib.cc pass.

Benchmark, same toolchain (AMD Ryzen 7 9800X3D, Ubuntu 24.04 on WSL2, GCC 13.3). "Before" is the parent commit (8d514c7), built from the same tree with the same configure flags, so the only difference is this PR's zlib.gyp and node_zlib.cc changes.

node benchmark/compare.js --old <before> --new <after> --runs 30 --filter crc32 --exclude crc32c --analyze zlib:

                                                 confidence   improvement   accuracy (*)    (**)   (***)
zlib/crc32.js n=4000000 len=32 type='buffer'                     -0.13 %   ±0.90%  ±1.20%  ±1.56%
zlib/crc32.js n=4000000 len=256 type='buffer'    ***           +289.33 %   ±2.24%  ±2.99%  ±3.90%
zlib/crc32.js n=4000000 len=4096 type='buffer'   ***           +156.06 %   ±2.15%  ±2.87%  ±3.74%
zlib/crc32.js n=4000000 len=65536 type='buffer'  ***           +136.22 %   ±1.32%  ±1.76%  ±2.31%
zlib/crc32.js n=4000000 len=32 type='string'                     -0.04 %   ±0.45%  ±0.60%  ±0.78%
zlib/crc32.js n=4000000 len=256 type='string'    ***           +272.05 %   ±1.87%  ±2.51%  ±3.31%
zlib/crc32.js n=4000000 len=4096 type='string'   ***           +119.52 %   ±1.89%  ±2.53%  ±3.32%
zlib/crc32.js n=4000000 len=65536 type='string'  ***           +107.58 %   ±1.32%  ±1.75%  ±2.28%

len=32 is unchanged because the SIMD path only handles inputs of 64 bytes or more (Z_CRC32_SSE42_MINIMUM_LENGTH in deps/zlib/crc32_simd.h).

The same comparison with --runs 20 --filter pipe --set algorithm=gzip shows no significant change (+1.06% string, −0.62% buffer). That benchmark pipes 1 KB writes through createGzip() at the default level into createGunzip(), so compression dominates.

Whole-buffer throughput on the first 14 MB of a tar of lib/, doc/api/, deps/zlib/ and deps/v8/src/. Each figure is the median of 3 rounds that alternate the two binaries, and each round takes the best of 5–30 calls:

Operation Before After Change
zlib.crc32 8.6 GB/s 20.3 GB/s 2.35x
zlib.crc32, before any zlib stream exists 8.6 GB/s 20.2 GB/s 2.34x
gzipSync (level 0) 4.6 GB/s 7.3 GB/s 1.58x
gunzipSync (level-0 input) 5.3 GB/s 8.4 GB/s 1.59x
gunzipSync (level-6 input) 814 MB/s 871 MB/s 1.07x
gzipSync (level 6) 79 MB/s 81 MB/s within run-to-run noise

A build with only the zlib.gyp change (and the parent's node_zlib.cc) runs zlib.crc32() at 8.5 GB/s until the process creates its first zlib stream, then at 20.4 GB/s. With this PR it runs at 20.4 GB/s from the start (median of 3 runs on the same input).

Benchmark (Xeon, 14 MB tar, best-of-N):
Only CRC32_SIMD_SSE42_PCLMUL is enabled; CRC32_SIMD_AVX512_PCLMUL remains off. Note that "before" uses the official v22.22.0 binary, so the baseline also differs in bundled zlib version (1.3.1 vs. 1.3.2.1-motley on main) and toolchain:

Operation Before (v22.22.0) After (this PR)
zlib.crc32 4.5 GB/s 18.8 GB/s
gzipSync (level 0) 2.15 GB/s 3.8 GB/s
gunzipSync ~485 MB/s ~540 MB/s
gzipSync (level 6) unchanged unchanged

AI assistance: I used Claude Code for parts of this PR. I personally reviewed the diff, traced the root cause in the zlib sources, reproduced the crash on v20.6.0, and ran the ASan/Valgrind harness and zlib test suite.

Refs: #45268
Refs: #49511
Refs: #52123

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/gyp
  • @nodejs/security-wg

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. dependencies PRs that add, update, or configure Node.js dependencies. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib module and its compression dependencies. labels Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Welcome to Node.js, and thank you for your first contribution!

Before review, please take a moment to read:

Please make sure every commit is signed off. For a first pull request, GitHub Actions require collaborator approval and Jenkins CI must be started by a collaborator or triager, so an initial wait is normal.

Caution

AgentScan found account activity patterns that may be consistent with automation. This is a heuristic, not proof that this pull request was opened by an agent or violates policy. AI-assisted contributions are permitted, but automated tooling must not open pull requests without advance approval, and contributors must personally understand, test, verify, and take responsibility for every submitted change. See the AgentScan analysis, AI use policy, and automation policy for additional context.

@ptweezy ptweezy changed the title Zlib reenable crc32 simd zlib: re-enable CRC32 SIMD optimization Oct 30, 2026
@ptweezy
ptweezy force-pushed the zlib-reenable-crc32-simd branch from feb58df to 233f483 Compare October 30, 2026 20:14
The memory corruption that led to disabling CRC32_SIMD_SSE42_PCLMUL
came from crc_fold_copy() in deps/zlib/crc_folding.c. For the final
(len % 16) bytes of a copy it did a 16-byte unaligned store, writing
up to 15 bytes past the destination. With gzip at level 0,
deflate_stored() copies input straight into strm->next_out through
read_buf() and copy_with_crc(), so the overrun landed past the end of
the caller's output buffer and corrupted the heap.

Chromium replaced that store with zmemcpy(), and the fix arrived here
with the zlib update in 70a0510 (v22.0.0). With the old code, the
added test aborts v19.0.0 and v20.6.0 builds every time. With the
current code it passes, and the CRC32 SIMD path is clean under ASan
and valgrind.

The restored zlib_crc32_simd target predates clang-cl support on
Windows, so also pass -msse4.2 and -mpclmul through msvs_settings, as
zlib_adler32_simd does.

The bundled zlib only detects CPU features in deflateInit2() and when
crc32() or adler32() is called with a NULL buffer. zlib.crc32() always
passes a buffer, so it used the portable code until a stream had
triggered the detection. Trigger it when the binding is initialized so
zlib.crc32() uses the SIMD path from the start.

Refs: nodejs#45268
Refs: nodejs#49511
Refs: nodejs#52123
Assisted-by: Claude Code
Signed-off-by: Parker Loflin <parker@cronstable.dev>
@ptweezy
ptweezy force-pushed the zlib-reenable-crc32-simd branch from 233f483 to e180d3c Compare October 30, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentscan:mixed-signals c++ Issues and PRs that require attention from people who are familiar with C++. dependencies PRs that add, update, or configure Node.js dependencies. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib module and its compression dependencies.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants