Skip to content

perf(bigtable): resolve O(n^2) performance bottleneck in ReadRowsQuery.add_range - #18705

Closed
sahananshylaja-glitch wants to merge 1 commit into
googleapis:mainfrom
sahananshylaja-glitch:main
Closed

sahananshylaja-glitch wants to merge 1 commit into
googleapis:mainfrom
sahananshylaja-glitch:main

Conversation

@sahananshylaja-glitch

@sahananshylaja-glitch sahananshylaja-glitch commented Oct 15, 2026 •

Copy link
Copy Markdown

Fixes O(n^2) performance bottleneck in ReadRowsQuery.add_range.
Linking b/573522284

Previously, add_range checked for duplicate ranges by calling self.row_ranges, which dynamically reconstructed Python RowRange wrappers for the entire query on every insert. This caused severe quadratic CPU scaling (e.g., 8,000 ranges took ~400s).

This CL optimizes the deduplication by maintaining a Python set of string signatures for O(1) lookups. This restores performant insertion (~0.14s for 8k ranges) while perfectly preserving the deduplication behavior expected by the shard() method.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #18196 🦕

Fixes O(n^2) performance bottleneck in ReadRowsQuery.add_range.

Previously, add_range checked for duplicate ranges by calling self.row_ranges, which dynamically reconstructed Python RowRange wrappers for the entire query on every insert. This caused severe quadratic CPU scaling (e.g., 8,000 ranges took ~400s).

This CL optimizes the deduplication by maintaining a Python set of string signatures for O(1) lookups. This restores performant insertion (~0.14s for 8k ranges) while perfectly preserving the deduplication behavior expected by the shard() method.
@sahananshylaja-glitch
sahananshylaja-glitch requested a review from a team as a code owner October 15, 2026 16:09

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the add_range method in read_rows_query.py by introducing a set of signatures (_range_signatures) to avoid O(n^2) wrapper reconstruction during deduplication. The reviewer suggested a more efficient implementation that uses the serialized byte representation (SerializeToString()) of the underlying protobuf messages as signatures, which avoids string formatting and accessing private attributes like _pb.

@sahananshylaja-glitch sahananshylaja-glitch changed the title Update read_rows_query.py perf(bigtable): resolve O(n^2) performance bottleneck in ReadRowsQuery.add_range Oct 15, 2026
@parthea parthea self-assigned this Oct 15, 2026
@sahananshylaja-glitch

Copy link
Copy Markdown
Author

Closing in favor of PR #18706 authored by the core maintainers based on offline feedback. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants