Skip to content

Release metadata references in partition_unique - #502

Open
harbinresearcher wants to merge 1 commit into
python-streamz:masterfrom
harbinresearcher:codex/partition-unique-metadata
Open

harbinresearcher wants to merge 1 commit into
python-streamz:masterfrom
harbinresearcher:codex/partition-unique-metadata

Conversation

@harbinresearcher

Copy link
Copy Markdown

partition_unique retains metadata references for all inputs, including records discarded by keep="first" and records replaced by keep="last". Completed partitions also emit nested metadata lists, so the shared reference helpers cannot release the retained counters. A synchronous sink can finish while those counters remain nonzero.

Release the metadata belonging to discarded records, and flatten the retained records metadata before emission. Downstream nodes still acquire their own references; a buffering collect retains them until its flush completes. Value ordering, partition size and the key API are unchanged. The similar timed-window node is outside this patch.

Fixes #501.

Validation:

  • Four new regression cases fail on unchanged source; the focused partition_unique suite passes after the fix: 9 passed.
  • Tests cover both retention modes, multiple metadata entries and ownership through a downstream collector.
  • Repository flake8 and git diff --check pass on Windows Python 3.12.14.
  • A separate read-only Codex agent reran the 9 focused cases and found no blocking issue. This is not manual human review.
  • Full, unfiltered Linux Python 3.12.15 suites run serially in the same environment: baseline 1041 passed; patched 1045 passed. Both have exactly the same 3 failures, 437 skips, 8 xfails and 96 xpasses. The failures are test_create_file (missing Graphviz dot), test_tcp_word_count_example (connection refused), and test_tcp_async (60-second timeout). No test filtering, timeout extension or coverage exclusion was added. This is a baseline comparison, not a claim that the complete suite is green.
  • Full-suite package statement coverage remains 86%; core.py remains 95%, with the same number of uncovered statements (43).
  • The 9 focused tests also pass on Windows Python 3.10.21; Windows Python 3.12.14 complete-suite verification was interrupted after a pristine-baseline Dask communication stall. Linux provided the completed full-suite comparison.
  • Repository flake8 also passes on Linux.

AI disclosure: Codex prepared this autonomous contribution under the account owner authorization, including investigation, implementation, tests and a separate agent review. No production incident or active deployment is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

partition_unique leaks metadata references for completed and duplicate records

1 participant