Skip to content

[SLES-3105] fix(traces): trust DD_TLS_CERT_FILE roots for the post-CONNECT proxy TLS handshake - #1525

Merged
shreyamalpani merged 6 commits into
mainfrom
sles-3105-proxy-tunnel-tls-roots
Nov 20, 2026
Merged

shreyamalpani merged 6 commits into
mainfrom
sles-3105-proxy-tunnel-tls-roots

Conversation

@shreyamalpani

Copy link
Copy Markdown
Contributor

Ported from #1520

Problem

With DD_PROXY_HTTPS set to a TLS-intercepting proxy and DD_TLS_CERT_FILE pointing at the proxy's CA, traces are dropped:

TRACES | Request failed after 4 attempts (0 timeouts, 1 network errors, 0 status code errors): No requests sent
FLUSHING_SERVICE | Dropping undelivered payloads after blocking flush: traces

Logs and metrics work through the same proxy with the same cert file. The trace flusher and the stats flusher share one HTTP client (startup.rs), so stats go through the same path.

Cause

traces/http_client.rs::create_client puts the custom CA in the inner connector, but hyper_http_proxy::ProxyConnector::from_proxy() only uses that connector to reach the proxy. The handshake inside the CONNECT tunnel uses its own TlsConnector, built from public webpki roots, and set_tls() is never called. The intake certificate re-signed by the customer's CA is rejected. Logs and metrics use the reqwest client in http.rs, which adds the custom CA to its root store, so they are unaffected. Present since #961 (v91).

This PR

Installs the tunnel's TLS config with ProxyConnector::set_tls(...). Certificate verification stays on unless DD_SKIP_SSL_VALIDATION is set.

  • DD_TLS_CERT_FILE: the tunnel trusts the public webpki roots (what from_proxy() trusts by default) plus the configured CAs, as the reqwest client in http.rs does. This keeps a non-intercepting proxy working, where Datadog's publicly signed certificate passes through.
  • DD_SKIP_SSL_VALIDATION: now also applies to the TLS handshake inside the proxy tunnel, matching the reqwest client used for logs and metrics (danger_accept_invalid_certs). Before this PR the tunnel handshake always verified against the public roots, so the setting had no effect there. When it is set, it still overrides DD_TLS_CERT_FILE, as in http.rs. Covered by proxy_tunnel_honours_skip_ssl_validation.
  • tokio-rustls moves from [dev-dependencies] to [dependencies], and webpki-roots is added as a direct dependency. Both are already in the build; Cargo.lock gains one dependency edge.

Three commits: the fix with unit tests, the process-level harness test, and the public-roots change.

Testing

Test 1: Process-level harness test (tests/proxy_tls_test.rs)

Steps

  1. Spawn the real bottlecap binary against the fake Extensions API with DD_PROXY_HTTPS set to a CONNECT proxy, DD_TLS_CERT_FILE set to a private CA, and DD_APM_DD_URL set to an https://localhost intake whose certificate that CA signed. The intake is the harness FaultIntake behind a small TLS front. The proxy tunnels only to that front and answers 403 to any other target.
  2. Post a trace to the trace agent and complete the invocation.
  3. Run cargo nextest run --test proxy_tls_test on main and on this branch.

Result

  • main: fails. The proxy logs the CONNECT, the extension logs Request failed after 4 attempts ... No requests sent, and the intake receives metrics and logs but no trace.
  • This branch: passes. The trace reaches the intake with the API key and the expected trace ID, the proxy logs the CONNECT, and /next is called before the invoke deadline.

Test 2: Unit tests (traces/http_client.rs)

Steps

  1. Run cargo test --lib traces::http_client. Six new tests use an in-test CONNECT proxy and a private-CA TLS server.
  2. proxy_tunnel_keeps_public_roots_when_tls_cert_file_is_set covers a non-intercepting proxy: the server's certificate is signed by a CA standing in for the public roots, while DD_TLS_CERT_FILE names an unrelated CA.

Result

  • All six pass, including the two negative controls (an unrelated CA, and the private CA without DD_TLS_CERT_FILE, are still rejected).
  • Without the set_tls() fix, proxy_tunnel_trusts_tls_cert_file_ca and proxy_tunnel_honours_skip_ssl_validation fail.
  • With the public roots removed from the tunnel config (the first revision of this PR), proxy_tunnel_keeps_public_roots_when_tls_cert_file_is_set fails and the other five pass.

Test 3: Real AWS, official layer v103 vs this change

Steps

  1. Two arm64 Lambdas in a VPC with no NAT; egress only through a mitmproxy TLS-intercepting proxy. DD_PROXY_HTTPS and DD_TLS_CERT_FILE (the proxy CA) set on both, sending to real Datadog. One uses the official v103 layer, the other a layer built from this branch's logic with build_bottlecap_layer.sh.
  2. Invoke each 3 times and check the extension log, the proxy log and the traces in Datadog.

Result

Extension log Proxy log (trace.agent.datadoghq.com) Traces delivered
official v103 Request failed after 4 attempts ... No requests sent Client TLS handshake failed ... does not trust the proxy's certificate 0 / 3
this change Successfully sent trace POST /api/v0.2/traces -> 202 Accepted 3 / 3

In the v103 run, logs and metrics still got 202 through the same proxy.

Stats use the same client. With DD_LAMBDA_EXTENSION_COMPUTE_STATS=true and invocations that send no trace, the v103 function logged STATS | Failed to send stats ... (attempt 1/3) through 3/3 and then Dropping undelivered payloads ... stats, with three TLS handshake failures at the proxy. The patched function logged STATS | Successfully flushed stats and the proxy returned 202 for POST /api/v0.2/stats.

Note: the AWS layer was built from a v102 base with the first revision of the fix (before the public roots were added to the tunnel config), and http_client.rs on main is byte-identical to v102. The harness test, the unit tests, and Clippy (CI's pinned 1.93.1 toolchain) were run against a build of current main plus the final change.

Jira: SLES-3105

…TLS handshake

When DD_PROXY_HTTPS is set, the trace/stats flusher tunnels through the proxy with
HTTP CONNECT and then performs a second TLS handshake to the intake. create_client()
builds a rustls config containing the customer's DD_TLS_CERT_FILE roots and hands it
to the inner connector, but hyper_http_proxy::ProxyConnector::from_proxy() only uses
that connector to reach the proxy. For the handshake inside the tunnel it builds its
own TlsConnector from public webpki roots, and nothing ever replaces it (set_tls is
never called). Behind a TLS-intercepting proxy the intake certificate is signed by the
customer's private CA, so verification fails and every trace/stats payload is dropped
("Request failed after 4 attempts ... No requests sent").

Install the same custom-root config on the ProxyConnector with set_tls(), for both the
tls_cert_file and skip_ssl_validation branches. Certificate verification stays on.

Regression introduced when the hyper-based trace client was added in #961 (v91).

Moves tokio-rustls from [dev-dependencies] to [dependencies]; it is already in the
shipped binary transitively and Cargo.lock is unchanged.

Adds 5 unit tests using an in-test CONNECT proxy and TLS server:
  proxy_tunnel_trusts_tls_cert_file_ca
  proxy_tunnel_rejects_unrelated_ca
  proxy_tunnel_rejects_private_ca_without_tls_cert_file
  proxy_tunnel_honours_skip_ssl_validation
  direct_connection_trusts_tls_cert_file_ca
…roxy with a custom CA

Runs the real bottlecap binary with DD_PROXY_HTTPS pointing at a CONNECT proxy,
DD_TLS_CERT_FILE holding a private CA, and an https DD_APM_DD_URL whose certificate that
CA signed. Asserts the trace reaches the intake with the API key, that the proxy saw the
CONNECT, and that /next is still called before the invoke deadline.

The intake is the harness FaultIntake behind a small TLS front; the proxy only tunnels to
that front and answers 403 to any other target, so the stats flusher's CONNECT to the real
intake cannot leave the machine.

Fails on main (the trace is dropped: "Request failed after 4 attempts ... No requests
sent") and passes with the set_tls() fix. Registered in the fixed-ports nextest group and
in tests/HARNESS.md.
…RT_FILE is set

The tunnel's TLS config held only the DD_TLS_CERT_FILE certificates, so installing it with
set_tls() replaced the public roots ProxyConnector::from_proxy() trusts by default. A proxy
that tunnels without intercepting passes Datadog's publicly signed certificate through, and
that certificate would now be rejected, dropping traces and stats for a configuration that
worked before.

Start the tunnel's root store from the public webpki roots and add the DD_TLS_CERT_FILE
certificates to it, matching how the reqwest client in http.rs layers a custom CA. The
skip_ssl_validation branch is unchanged. The public roots are a parameter of an internal
create_client_with_public_roots(), so a test can stand in for them.

Adds webpki-roots as a direct dependency; it is already in the build and Cargo.lock gains a
single dependency edge. Adds proxy_tunnel_keeps_public_roots_when_tls_cert_file_is_set.
@shreyamalpani
shreyamalpani requested review from a team as code owners November 17, 2026 20:46
@shreyamalpani
shreyamalpani requested review from jcaparelli15 and a balanced review from Copilot November 17, 2026 20:46
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T18:44:56.929176Z 22c321b New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The proxy-tunnel TLS trust changes have no active review issues.

No open issues 🎉

What changed in this PR

Fixes TLS trust handling in the Lambda extension’s shared trace and stats HTTP client so custom CA certificates apply to the handshake inside a CONNECT proxy tunnel.

Changes:

  • Configures tunnel TLS to trust public roots plus DD_TLS_CERT_FILE, or skip verification when DD_SKIP_SSL_VALIDATION is enabled.
  • Adds six unit tests and a process-level regression test covering proxy TLS behavior.
  • Updates dependencies, test serialization, and harness documentation.
File Description
bottlecap/​tests/​proxy_tls_test.rs Adds end-to-end trace delivery coverage through a CONNECT proxy to a private-CA intake.
bottlecap/​tests/​HARNESS.md Documents the new proxy TLS test and its run command.
bottlecap/​src/​traces/​http_client.rs Applies TLS settings inside proxy tunnels and adds trust-policy tests.
bottlecap/​Cargo.toml Promotes tokio-rustls to a runtime dependency and adds webpki-roots.
bottlecap/​Cargo.lock Records the direct public-root dependency.
bottlecap/​.config/​nextest.toml Serializes the new integration test with other fixed-port tests.

Review effort: Balanced

@datadog-prod-us1-4

This comment has been minimized.

@shreyamalpani

Copy link
Copy Markdown
Contributor Author

/remove

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-18 14:39:15 UTC ℹ️ Start processing command /remove


2026-11-18 14:39:20 UTC

Done

@shreyamalpani
shreyamalpani merged commit 02fd45b into main Nov 20, 2026
59 of 63 checks passed
@shreyamalpani
shreyamalpani deleted the sles-3105-proxy-tunnel-tls-roots branch November 20, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants