Skip to content

PYTHON-XXXX Reuse a zstd compressor per thread - #3141

Open
lohmag wants to merge 1 commit into
mongodb:mainfrom
lohmag:reuse-zstd-compressor
Open

lohmag wants to merge 1 commit into
mongodb:mainfrom
lohmag:reuse-zstd-compressor

Conversation

@lohmag

@lohmag lohmag commented Oct 25, 2026

Copy link
Copy Markdown

PYTHON-XXXX

Changes in this PR

ZstdContext.compress called zstd.compress() for every message, which creates and frees a zstd compression context each time. For a small message, setting up the context costs more than compressing it.

It now keeps one ZstdCompressor per thread in a threading.local, and compresses each message as a complete frame with compress(data, ZstdCompressor.FLUSH_FRAME).

  • Thread safety. A compressor serializes its calls, so a shared one would make threads wait on each other.
  • Errors. If a call raises, the thread's compressor is dropped and the next call creates a new one.

The output is a standard zstd frame, the same as before, and decompression is unchanged.

ZstdContext.compress, document-like payloads, master → this PR. Python 3.12, backports.zstd (zstd 1.5.7), Intel Core Ultra 7 265, one pinned core:

Size Time
50 B 1.56 → 0.54 µs
200 B 3.24 → 2.26 µs
1 KiB 4.62 → 3.49 µs
4 KiB 9.42 → 7.88 µs
16 KiB 31.2 → 27.8 µs
64 KiB 126.3 → 119.5 µs

Compressed sizes are unchanged.

Test Plan

  • test_zstd_compressor_reused_across_messages: 64 messages of different sizes through the same thread's compressor. Each one decompresses with zstd.decompress and with pymongo.compression_support.decompress.
  • test_zstd_compress_from_many_threads: 8 threads compress and decompress 200 messages each.
  • test/test_compression_support.py: 36 passed, 4 skipped (python-snappy not installed).
  • Against a MongoDB 8.0.20 standalone with COMPRESSORS=zstd: test/test_client.py -k compress (2 passed), and test/test_collection.py, test/test_cursor.py and test/test_bulk.py (187 passed, 15 skipped because they need a replica set).
  • ruff check and ruff format --check pass on the changed files.

Checklist

Checklist for Author

  • Did you update the changelog (if necessary)?
  • Is there test coverage?
  • Is any followup work tracked in a JIRA ticket? If so, add link(s).

Checklist for Reviewer

  • Does the title of the PR reference a JIRA Ticket?
  • Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
  • Is all relevant documentation (README or docstring) updated?

With Claude's help.

ZstdContext.compress called zstd.compress(), which creates and frees a compression context for every message.
@lohmag
lohmag requested a review from a team as a code owner October 25, 2026 20:42
@lohmag
lohmag requested a review from aclark4life October 25, 2026 20:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant