Repository navigation
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enables fine-grained control over BigQuery API billing and quota management by allowing users to specify a quota project distinct from the project where the data resides. By implementing the X-Goog-User-Project header across the primary Apache Beam SDKs, this change provides a consistent way to manage costs and API limits in multi-project environments. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for attributing BigQuery API quota and billing to a specific GCP project (quota project) across the Go, Java, and Python SDKs. The changes introduce the quota_project_id parameter and pipeline option, propagating it to the underlying BigQuery and BigQuery Storage API clients, along with comprehensive unit tests. The review feedback suggests improving robustness in the Python SDK by checking if Google Auth is available before referencing its adapter to prevent potential NameErrors, and broadening the exception handling when initializing BigQuery clients with a quota project to ensure a safe fallback to default credentials in case of any loading failures.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
Live verification against real GCP projects (all three SDKs) Each SDK ran the same three scenarios on a local runner, reading
The third scenario is a positive attribution proof with zero cost: the API rejected the request as a consumer of the quota project, demonstrating the request was attributed to it rather than to project A. Two findings from the live tests:
|
|
/gemini review |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #39203 +/- ##
============================================
+ Coverage 56.16% 56.31% +0.14%
============================================
Files 1124 868 -256
Lines 178336 153607 -24729
Branches 1489 0 -1489
============================================
- Hits 100170 86501 -13669
+ Misses 75645 65261 -10384
+ Partials 2521 1845 -676
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
|
/gemini review |
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Reminder, please take a look at this pr: @shunping @chamikaramj @jrmccluskey |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @claudevdm for label python. Available commands:
|
|
Reminder, please take a look at this pr: @claudevdm @ahmedabu98 @lostluck |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @jrmccluskey for label python. Available commands:
|
|
Reminder, please take a look at this pr: @jrmccluskey @kennknowles @shunping |
|
LGTM on Go implementation. |
claudevdm
left a comment
claudevdm
left a comment
There was a problem hiding this comment.
Should we plumb the transform-level quota_project_id through the following BQ API calls?
beam/sdks/python/apache_beam/io/gcp/bigquery.py
Line 1117 in e4779cf
beam/sdks/python/apache_beam/io/gcp/bigquery.py
Line 3020 in e4779cf
Two unresolved review comments from claudevdm on apache#39203: 1. Don't fall back when the user explicitly asked for a quota project. `auth.with_quota_project` now raises instead of warning-and-ignoring, and the BigQuery Storage / google-cloud-bigquery client factories no longer swallow credential errors. Silently falling back bills a different project than the one requested, which is the exact thing the option exists to control. 2. Passing `client=` to override both `self.client` (apitools BigqueryV2) and `self.gcp_bq_client` (google-cloud-bigquery Client) was error prone. `BigQueryWrapper` now takes `pipeline_options`/`quota_project_id` (both keyword-only, so the existing positional signature is unchanged) and builds both clients itself, with `client`/`gcp_client` as separate test-only overrides. `gcp_bq_client` is created on first use, since only the streaming insert path needs it. Production call sites no longer pass `client=`: both read sources build every wrapper through a `_create_bq_wrapper` helper, so quota attribution cannot be forgotten at an individual call site. That also fixes two call sites that silently ignored the transform-level `quota_project_id`: `estimate_size` in both sources, and the table schema lookup that `output_type='BEAM_ROW'` performs. Generated-by: Claude Code (claude-opus-5) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R4ARQfZ9k9G9YG2WESgV7u
Two unresolved review comments from claudevdm on apache#39203: 1. Don't fall back when the user explicitly asked for a quota project. `auth.with_quota_project` now raises instead of warning-and-ignoring, and the BigQuery Storage / google-cloud-bigquery client factories no longer swallow credential errors. Silently falling back bills a different project than the one requested, which is the exact thing the option exists to control. 2. Passing `client=` to override both `self.client` (apitools BigqueryV2) and `self.gcp_bq_client` (google-cloud-bigquery Client) was error prone. `BigQueryWrapper` now takes `pipeline_options`/`quota_project_id` (both keyword-only, so the existing positional signature is unchanged) and builds both clients itself, with `client`/`gcp_client` as separate test-only overrides. `gcp_bq_client` is created on first use, since only the streaming insert path needs it. Production call sites no longer pass `client=`: both read sources build every wrapper through a `_create_bq_wrapper` helper, so quota attribution cannot be forgotten at an individual call site. That also fixes two call sites that silently ignored the transform-level `quota_project_id`: `estimate_size` in both sources, and the table schema lookup that `output_type='BEAM_ROW'` performs. Generated-by: Claude Code (claude-opus-5) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R4ARQfZ9k9G9YG2WESgV7u
8c34256 to
70be9f9
Compare
|
Reminder, please take a look at this pr: @jrmccluskey @kennknowles @shunping @claudevdm |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @damccorm for label python. Available commands:
|
|
R: @claudevdm (keeping this sticky) |
|
waiting on author |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
@claudevdm would you mind taking another look? Thank you :) |
claudevdm
left a comment
claudevdm
left a comment
There was a problem hiding this comment.
Can you please fix merge conflicts? Otherwise I think this looks good.
Thank you, fixed and made some last changes. Edit: CI fails, I'll investigate and fix CC: @iemejia |
Two unresolved review comments from claudevdm on apache#39203: 1. Don't fall back when the user explicitly asked for a quota project. `auth.with_quota_project` now raises instead of warning-and-ignoring, and the BigQuery Storage / google-cloud-bigquery client factories no longer swallow credential errors. Silently falling back bills a different project than the one requested, which is the exact thing the option exists to control. 2. Passing `client=` to override both `self.client` (apitools BigqueryV2) and `self.gcp_bq_client` (google-cloud-bigquery Client) was error prone. `BigQueryWrapper` now takes `pipeline_options`/`quota_project_id` (both keyword-only, so the existing positional signature is unchanged) and builds both clients itself, with `client`/`gcp_client` as separate test-only overrides. `gcp_bq_client` is created on first use, since only the streaming insert path needs it. Production call sites no longer pass `client=`: both read sources build every wrapper through a `_create_bq_wrapper` helper, so quota attribution cannot be forgotten at an individual call site. That also fixes two call sites that silently ignored the transform-level `quota_project_id`: `estimate_size` in both sources, and the table schema lookup that `output_type='BEAM_ROW'` performs. Generated-by: Claude Code (claude-opus-5) Claude-Session: https://claude.ai/code/session_01R4ARQfZ9k9G9YG2WESgV7u
92d071c to
3fea05b
Compare
Add a quota_project_id parameter to ReadFromBigQuery and a --quota_project_id pipeline option (GoogleCloudOptions) that attribute BigQuery API quota and billing to a project other than the one the data resides in. The quota project is sent as the x-goog-user-project header on the apitools BigQuery client, and through client_options on the google-cloud-bigquery Client (streaming inserts) and the BigQuery Storage Read client. The shared pipeline credentials are left untouched instead of being copied with with_quota_project(), which for some credential types drops the cached token. The header is only added to requests to the BigQuery API hosts: the credentials refresh their token through the same Http, and the token endpoint, metadata server and IAM Credentials API must not receive it. It is honored by ReadFromBigQuery (EXPORT and DIRECT_READ, including temporary dataset cleanup), ReadAllFromBigQuery, WriteToBigQuery with FILE_LOADS or STREAMING_INSERTS, and ReadBigQueryChangeHistory. The value is resolved once at expand() time and passed to each DoFn so it reaches separately deserialized SDF copies. STORAGE_WRITE_API is Java-backed and not covered yet. Generated-By: Claude Opus 5.5 <noreply@anthropic.com>
Add BigQueryOptions.bigQueryQuotaProjectId to attribute BigQuery API quota and billing to a project other than the one the data resides in. For the REST client, the x-goog-user-project header is set by an execute interceptor, so it is re-applied on every attempt, including after HttpCredentialsAdapter re-initializes headers on a 401 refresh. The Storage Read and Write clients set it through setQuotaProjectId. Generated-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a WithQuotaProject query option that attributes BigQuery API quota and billing to a project other than the one the data resides in, via option.WithQuotaProject on the BigQuery client. Read now accepts query options too. Only WithQuotaProject applies to it; UseStandardSQL and WithQueryParameters panic, because Read generates a legacy SQL statement without parameters. Generated-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated-By: Claude Opus 5.5 <noreply@anthropic.com>
3fea05b to
d5703b7
Compare
5 participants
Closes #37431
Adds support for attributing quota and billing of BigQuery API requests to a specific GCP project (the "quota project", i.e. the
x-goog-user-projectheader), separate from the project the data resides in. The credentials used must haveserviceusage.services.useon the quota project.Python
quota_project_idparameter onbeam.io.ReadFromBigQuery(bothEXPORTandDIRECT_READ), falling back to a new--quota_project_idpipeline option onGoogleCloudOptions.ReadFromBigQuery,ReadAllFromBigQuery,WriteToBigQuerywithFILE_LOADS/STREAMING_INSERTS, andReadBigQueryChangeHistory. It is not yet honored byWriteToBigQuery(method=STORAGE_WRITE_API)(Java-backed; the Python option name does not translate to Java'sbigQueryQuotaProjectId) nor by other GCP IOs — tracked in [Feature Request]: Honor --quota_project_id in Python GCS/Pub/Sub IO and the Java-backed BigQuery STORAGE_WRITE_API path #40555. The option's help text states this scope.with_quota_project(), so the shared cached credentials are left untouched:BigqueryV2client: anhttplib2wrapper below the credentials layer setsx-goog-user-projecton every attempt (so it survives 401-refresh retries and wins over a quota project carried by ADC).google-cloud-bigqueryclient (streaming inserts) andBigQueryReadClient(Storage Read API):client_options={'quota_project_id': ...}.quota_project_idwhile no credentials can be resolved raises instead of silently billing another project.BigQueryWrapper/ storage client used by these transforms is built through a single path that receives the quota project, so a call site cannot forget it. For sources that re-split on workers, the resolved value is carried as a plain attribute so it survives pickling._CustomBigQueryStorageStreamSource.split()re-created the stream source without itstimeout, so the per-stream read timeout was lost after a split. Now preserved.Java
--bigQueryQuotaProjectIdoption onBigQueryOptions(named after the existing--bigQueryProject, which controls the job parent project — a related but distinct concept).BigQueryServicesImpl: the HTTPBigqueryclient setsx-goog-user-projectvia an execute interceptor (re-applied on every attempt, sinceHttpCredentialsAdapterre-initializes headers after a 401 refresh); the Storage Read/Write gRPC clients setquotaProjectIdon their gax settings. The HTTP job/dataset services are shared by the read and write paths, so the option covers all BigQuery API calls made by the SDK.TypedRead.withQuotaProjectId(...)) would require threading the value through theBigQueryServicesinterface and its fakes; left as a follow-up.Go
bigqueryio.WithQuotaProject(...)option, accepted bybigqueryio.Read(signature extended with variadic options — backward compatible) andbigqueryio.Query, applied viaoption.WithQuotaProject.ReadrejectsUseStandardSQL/WithQueryParameterswith a panic, since its generated statement is legacy SQL and takes no parameters.TypeScript
Not covered here: the TypeScript BigQuery IO delegates to the Java
schemaio_bigquery_read:v1expansion transform, so per-transform support there depends on the Java per-transform follow-up plus aBigQuerySchemaIOProviderconfiguration-schema change.Testing
bigquery-public-datawas run on a local runner (a) without a quota project — succeeds, (b) with a quota project the caller has noserviceusage.services.usepermission on — fails withUSER_PROJECT_DENIED, and (c) with a quota project whose BigQuery API is disabled — fails withSERVICE_DISABLEDnaming the quota project, positively confirming attribution. The live tests caught (and this PR fixes) a bug where the transform-levelquota_project_iddid not reach the BigQuery jobs client.bigquery_test.py,bigquery_tools_test.py,bigquery_read_internal_test.py,bigquery_file_loads_test.py,bigquery_change_history_test.py,pipeline_options_test.pypass locally (454 tests, ~45 quota-specific), including a test that the configured quota project reaches every request through a 401 refresh even when ADC carries its own quota project.BigQueryServicesImplTestpasses locally, including tests asserting the quota project on Storage client settings and that the header interceptor wins over a credential-set value on retry; checkstyle/spotless clean.go vetpass.Generative AI disclosure
This PR was generated end-to-end by Claude Code (Anthropic, models
claude-fable-5/claude-fable-5-1/claude-opus-5-5) under the author's supervision: design decisions, code review, and live verification against real GCP projects were performed by the author. Commits carryGenerated-by:trailers per the ASF Generative Tooling Guidance.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
🤖 Generated with Claude Code