Skip to content

WIP [Rust SDK] Add the Cargo workspace, Gradle integration and model crate - #40508

Draft
prodriguezdefino wants to merge 1 commit into
apache:masterfrom
prodriguezdefino:rust/01-workspace-model
Draft

prodriguezdefino wants to merge 1 commit into
apache:masterfrom
prodriguezdefino:rust/01-workspace-model

Conversation

@prodriguezdefino

Copy link
Copy Markdown
Contributor

First change in the series that adds the Apache Beam Rust SDK.

This change adds:

  • The Cargo workspace at sdks/rust (rustfmt and clippy configuration, workspace lints, dev profiles) with a single member.
  • apache-beam-model: protobuf and gRPC bindings generated at build time from the model protos in this repository (model/pipeline, model/fn-execution, model/job-management).
  • Gradle integration (:sdks:rust), so the SDK builds like the rest of the repository: build, check (rustfmt + clippy -D warnings), test, fmt and clean. build.gradle applies one script per build area from sdks/rust/gradle/; the tasks call cargo, which handles incremental builds. Gradle passes the Beam version from gradle.properties to cargo; no Beam version is hard-coded in the Rust sources.
  • Release tooling: release/src/main/scripts/set_version.sh bumps the Cargo manifests through sdks/rust/scripts/set_version.sh.
  • local-env-setup.sh installs Rust and protoc.
  • README.md and docs/development.md.

@github-actions github-actions Bot added the build label Oct 11, 2026
@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.91%. Comparing base (76f6e6d) to head (0443786).
⚠️ Report is 216 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40508       +/-   ##
=============================================
+ Coverage     56.16%   79.91%   +23.74%     
=============================================
  Files          1124      488      -636     
  Lines        178336    82562    -95774     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65981    -34189     
+ Misses        75645    16581    -59064     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 79.91% <ø> (+0.36%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@CH-Abhinav

Copy link
Copy Markdown

Hey @prodriguezdefino, I would like to help this PR as much as possible since I didn't know Beam is getting a Rust SDK. I am new to beam and I am excited to see Rust based work here.

I tested this branch locally on Windows with rustc 1.98 and protoc 36.0 — cargo check built cleanly and all proto/gRPC definitions generated without issues.

One small note: building on Windows requires protoc in PATH (or setting the PROTOC environment variable). Would it be helpful to add a quick mention in docs/development.md regarding installing protoc on Windows (e.g., via winget install Google.Protobuf or manual binary download)?

Happy to help test upcoming PRs on Windows or contribute to the downstream crates as the SDK progresses.

@prodriguezdefino
prodriguezdefino force-pushed the rust/01-workspace-model branch 2 times, most recently from 50d94ea to fbfacb2 Compare October 17, 2026 00:33
@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

Thanks for testing this out on Windows @CH-Abhinav, great to hear cargo check and proto generation worked cleanly!

In Beam, host dependencies are normally set up via local-env-setup.sh or by using the Dockerized development environment via start-build-env.sh at the root of the repository, rather than listing per-OS installation steps in each SDK's docs.

Once the RFC document (it's on the dev@ distro if you're curious about it) gets reviewed and hopefully, eventually approved, I planned to start working on having the initial PRs land. This draft is more about making sure checks are in place for once/if things get in motion.

It would be great to have your help making sure things work smoothly on Windows!

@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly scaffolds the new Apache Beam Rust SDK workspace (sdks/rust), Gradle integration (:sdks:rust), release version tooling, and the apache-beam-model crate for compiling portability Protobuf and gRPC definitions. The structure is well-organized and follows Beam's repository conventions.

🔍 General Feedback

  • Environment Setup (local-env-setup.sh): While protoc was added to the macOS setup path, it is currently missing from the Linux setup path (dev-support/docker/pkglist does not include protobuf-compiler).
  • Release Script (release/src/main/scripts/set_version.sh): The git add 'sdks/rust/*Cargo.toml' pathspec only matches single-depth subdirectories in Git, which misses crate manifests nested deeper such as sdks/rust/beam/model/Cargo.toml.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Comment thread local-env-setup.sh
Comment thread release/src/main/scripts/set_version.sh
Comment thread sdks/rust/README.md Outdated
First change in the series that adds the Apache Beam Rust SDK.

This change adds:
- The Cargo workspace at sdks/rust (rustfmt and clippy configuration,
  workspace lints, dev profiles) with a single member.
- apache-beam-model: protobuf and gRPC bindings generated at build time
  from the model protos in this repository (model/pipeline,
  model/fn-execution, model/job-management).
- Gradle integration (:sdks:rust), so the SDK builds like the rest of
  the repository: build, check (rustfmt + clippy -D warnings), test, fmt
  and clean. build.gradle applies one script per build area from
  sdks/rust/gradle/; the tasks call cargo, which handles incremental
  builds. Gradle passes the Beam version from gradle.properties to cargo;
  no Beam version is hard-coded in the Rust sources.
- Release tooling: release/src/main/scripts/set_version.sh bumps the
  Cargo manifests through sdks/rust/scripts/set_version.sh.
- local-env-setup.sh installs Rust and protoc (protobuf-compiler in
  dev-support/docker/pkglist on Linux and in the Docker build env).
- README.md and docs/development.md.
@prodriguezdefino

Copy link
Copy Markdown
Contributor Author

/gemini review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly sets up the initial Cargo workspace, Gradle build integration (:sdks:rust), version-bumping scripts, environment setup, and the apache-beam-model crate with generated Protobuf/gRPC bindings. All feedback from the previous review (adding protobuf-compiler to dev-support/docker/pkglist for Linux setups and resolving the broken link in README.md) has been addressed.

🔍 General Feedback

  • The Gradle-to-Cargo task delegation (build, check, test, fmt, clean) and version propagation from gradle.properties integrate smoothly with existing Beam build and release conventions.
  • Protobuf compilation in beam/model/build.rs properly registers cargo:rerun-if-changed directives for all model proto inputs.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants