Skip to content

fix(deps): bump rustls-webpki to 0.103.16 for the emSign CA compromise - #1463

Merged
kvinwang merged 3 commits into
nextfrom
security/emsign-rust-workspace
Oct 23, 2026
Merged

kvinwang merged 3 commits into
nextfrom
security/emsign-rust-workspace

Conversation

@Leechael

Copy link
Copy Markdown
Collaborator

Eight eMudhra/emSign TLS intermediate CAs had private keys that can be computed from their certificates, so anyone can issue a certificate for any domain that chains to emSign ECC Root CA - G3. Clients that do not enforce Certificate Transparency (all of our Rust services) are exposed.

This PR moves the main Rust workspace to rustls-webpki 0.103.16 (RUSTSEC-2026-0383, CVE-2026-111943). That release rejects any chain containing one of the compromised intermediate keys (CertDistrusted), whatever the trust store holds. On Linux our reqwest builds read the system trust store through rustls-platform-verifier, so this is the fix that covers every Rust binary in the workspace even before the image trust stores are updated.

Changes

  • dstack/Cargo.toml: workspace rustls-webpki = "0.103.10" -> "0.103.16".
  • dstack/Cargo.lock (cargo update -p rustls-webpki@0.103.13 --precise 0.103.16, cargo update -p webpki-root-certs --precise 1.0.10):
crate before after
rustls-webpki 0.103.13 0.103.16
webpki-root-certs 1.0.8 1.0.10
aws-lc-rs 1.17.0 1.18.2 (rustls-webpki 0.103.16 requires aws-lc-rs = "1.18")
aws-lc-sys 0.41.0 0.46.0 (adds pkg-config 0.3.34)

Nothing else in the lockfile moves.

Lockfile proof after the change (cargo tree --locked -e normal):

$ grep -c 'name = "rustls-webpki"' dstack/Cargo.lock
1
$ cargo tree --locked -i rustls-webpki -e normal --depth 1
rustls-webpki v0.103.16
├── rustls v0.23.40
├── rustls-platform-verifier v0.7.0
└── sev-snp-qvl v0.6.1
$ cargo tree --locked -i webpki-root-certs --target all -e normal
webpki-root-certs v1.0.10
└── rustls-platform-verifier v0.7.0 ...

The main workspace does not use webpki-roots. webpki-root-certs is only pulled in for non-Linux targets, so its bump does not change the Linux binaries.

Self-test against the emSign test site

I built a small reqwest client with the workspace's reqwest features (rustls, hickory-dns, ...) and resolved it from a copy of the workspace Cargo.lock, so it uses the same rustls, rustls-webpki, rustls-platform-verifier and reqwest versions. The box trust store is Debian trixie ca-certificates 20250419, which still trusts emSign ECC Root CA - G3.

== next (rustls-webpki v0.103.13)
TRUSTED (VULNERABLE): https://dv-g3b-emudhra.we-have-a-problem.com/ -> HTTP 200 OK
== this branch (rustls-webpki v0.103.16)
REJECTED: ... invalid peer certificate: Other(OtherError(CertDistrusted))

A control request to a normal site still returns HTTP 200 with the new lock.

Tests run

  • cargo check --workspace --locked --all-targets: passes.
  • cargo test --locked -p certbot -p ra-rpc -p sev-snp-qvl -p dstack-gateway: everything passes except proxy::tls_passthough::tests::test_resolve_app_address. That test does a live DNS TXT lookup of _dstack-app-address-wildcard.app.dstack.org, and the lookup returned "no records found" from this box. It does not touch TLS. 推断: the failure comes from the environment, not from this change. I did not run that test on next to compare.

Not run here

  • The release musl builds (cargo build --release --locked --target x86_64-unknown-linux-musl) and the Docker image builds. aws-lc-sys moves to 0.46.0, so the Docker Build Check jobs on this PR are the real check for the musl build.
  • The full workspace cargo test.

Rollout impact

Rebuilt gateway, KMS and verifier images get new digests, so the compose hashes of the CVMs that run them change. The guest OS embeds binaries built from this lockfile, so a rebuilt guest image has a new OS image hash and new measurements. The on-chain allowlists (compose hashes, OS image hashes) and the auth-config allowlists must be updated before the new builds can boot or get keys.

Related branches

The rest of the fix is split by component so each part can be reviewed and rebuilt on its own:

  • security/emsign-container-images: ca-certificates in the gateway/KMS/verifier images.
  • security/emsign-guest-os: mkosi Debian snapshot and Go 1.26.10.
  • security/emsign-kms-auth-images: auth-api Node 22.23.4, auth-mock Bun digest pin.
  • security/emsign-sdk: Rust and Python SDK locks.

rustls-webpki 0.103.16 (RUSTSEC-2026-0383, CVE-2026-111943) rejects any
chain that contains one of the compromised eMudhra/emSign intermediate
CA keys, whatever the trust store holds. Before this change the
workspace locked 0.103.13, and reqwest built like the workspace
(rustls-platform-verifier, system trust store) accepted a certificate
issued by one of those intermediates.

Raise the workspace minimum to 0.103.16 and update Cargo.lock. Also move
webpki-root-certs from 1.0.8 to 1.0.10; on Linux it is only pulled in for
other targets, so this is for consistency.

cargo also moves aws-lc-rs 1.17.0 -> 1.18.2 and aws-lc-sys
0.41.0 -> 0.46.0 (adds pkg-config), which rustls-webpki 0.103.16
requires.

Rebuilt binaries change the gateway, KMS and verifier image digests and
the guest OS image, so compose hashes and OS image hashes must be
registered again in the on-chain and auth-config allowlists.

Signed-off-by: Leechael Yim <yanleech@gmail.com>
rustls-webpki 0.103.16 pulls aws-lc-rs 1.18, which sometimes closes the
TCP connection instead of sending HandshakeFailure. reqwest then reports
Canceled / connection was not ready. The cert is still refused.

Signed-off-by: Leechael Yim <yanleech@gmail.com>
@Leechael
Leechael marked this pull request as ready for review October 22, 2026 02:03
@kvinwang
kvinwang merged commit 9cadf40 into next Oct 23, 2026
12 checks passed
@kvinwang
kvinwang deleted the security/emsign-rust-workspace branch October 23, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants