Repository navigation
[SLES-3105] fix(traces): trust DD_TLS_CERT_FILE roots for the post-CONNECT proxy TLS handshake - #1525
Conversation
…TLS handshake
When DD_PROXY_HTTPS is set, the trace/stats flusher tunnels through the proxy with
HTTP CONNECT and then performs a second TLS handshake to the intake. create_client()
builds a rustls config containing the customer's DD_TLS_CERT_FILE roots and hands it
to the inner connector, but hyper_http_proxy::ProxyConnector::from_proxy() only uses
that connector to reach the proxy. For the handshake inside the tunnel it builds its
own TlsConnector from public webpki roots, and nothing ever replaces it (set_tls is
never called). Behind a TLS-intercepting proxy the intake certificate is signed by the
customer's private CA, so verification fails and every trace/stats payload is dropped
("Request failed after 4 attempts ... No requests sent").
Install the same custom-root config on the ProxyConnector with set_tls(), for both the
tls_cert_file and skip_ssl_validation branches. Certificate verification stays on.
Regression introduced when the hyper-based trace client was added in #961 (v91).
Moves tokio-rustls from [dev-dependencies] to [dependencies]; it is already in the
shipped binary transitively and Cargo.lock is unchanged.
Adds 5 unit tests using an in-test CONNECT proxy and TLS server:
proxy_tunnel_trusts_tls_cert_file_ca
proxy_tunnel_rejects_unrelated_ca
proxy_tunnel_rejects_private_ca_without_tls_cert_file
proxy_tunnel_honours_skip_ssl_validation
direct_connection_trusts_tls_cert_file_ca
…roxy with a custom CA Runs the real bottlecap binary with DD_PROXY_HTTPS pointing at a CONNECT proxy, DD_TLS_CERT_FILE holding a private CA, and an https DD_APM_DD_URL whose certificate that CA signed. Asserts the trace reaches the intake with the API key, that the proxy saw the CONNECT, and that /next is still called before the invoke deadline. The intake is the harness FaultIntake behind a small TLS front; the proxy only tunnels to that front and answers 403 to any other target, so the stats flusher's CONNECT to the real intake cannot leave the machine. Fails on main (the trace is dropped: "Request failed after 4 attempts ... No requests sent") and passes with the set_tls() fix. Registered in the fixed-ports nextest group and in tests/HARNESS.md.
…RT_FILE is set The tunnel's TLS config held only the DD_TLS_CERT_FILE certificates, so installing it with set_tls() replaced the public roots ProxyConnector::from_proxy() trusts by default. A proxy that tunnels without intercepting passes Datadog's publicly signed certificate through, and that certificate would now be rejected, dropping traces and stats for a configuration that worked before. Start the tunnel's root store from the public webpki roots and add the DD_TLS_CERT_FILE certificates to it, matching how the reqwest client in http.rs layers a custom CA. The skip_ssl_validation branch is unchanged. The public roots are a parameter of an internal create_client_with_public_roots(), so a test can stand in for them. Adds webpki-roots as a direct dependency; it is already in the build and Cargo.lock gains a single dependency edge. Adds proxy_tunnel_keeps_public_roots_when_tls_cert_file_is_set.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟢 Approval recommended
The proxy-tunnel TLS trust changes have no active review issues.
No open issues 🎉
What changed in this PR
Fixes TLS trust handling in the Lambda extension’s shared trace and stats HTTP client so custom CA certificates apply to the handshake inside a CONNECT proxy tunnel.
Changes:
- Configures tunnel TLS to trust public roots plus
DD_TLS_CERT_FILE, or skip verification whenDD_SKIP_SSL_VALIDATIONis enabled. - Adds six unit tests and a process-level regression test covering proxy TLS behavior.
- Updates dependencies, test serialization, and harness documentation.
| File | Description |
|---|---|
bottlecap/tests/proxy_tls_test.rs |
Adds end-to-end trace delivery coverage through a CONNECT proxy to a private-CA intake. |
bottlecap/tests/HARNESS.md |
Documents the new proxy TLS test and its run command. |
bottlecap/src/traces/http_client.rs |
Applies TLS settings inside proxy tunnels and adds trust-policy tests. |
bottlecap/Cargo.toml |
Promotes tokio-rustls to a runtime dependency and adds webpki-roots. |
bottlecap/Cargo.lock |
Records the direct public-root dependency. |
bottlecap/.config/nextest.toml |
Serializes the new integration test with other fixed-port tests. |
Review effort: Balanced
This comment has been minimized.
This comment has been minimized.
|
/remove |
|
View all feedbacks in Devflow UI.
Done |
Ported from #1520
Problem
With
DD_PROXY_HTTPSset to a TLS-intercepting proxy andDD_TLS_CERT_FILEpointing at the proxy's CA, traces are dropped:Logs and metrics work through the same proxy with the same cert file. The trace flusher and the stats flusher share one HTTP client (
startup.rs), so stats go through the same path.Cause
traces/http_client.rs::create_clientputs the custom CA in the inner connector, buthyper_http_proxy::ProxyConnector::from_proxy()only uses that connector to reach the proxy. The handshake inside the CONNECT tunnel uses its ownTlsConnector, built from public webpki roots, andset_tls()is never called. The intake certificate re-signed by the customer's CA is rejected. Logs and metrics use thereqwestclient inhttp.rs, which adds the custom CA to its root store, so they are unaffected. Present since #961 (v91).This PR
Installs the tunnel's TLS config with
ProxyConnector::set_tls(...). Certificate verification stays on unlessDD_SKIP_SSL_VALIDATIONis set.DD_TLS_CERT_FILE: the tunnel trusts the public webpki roots (whatfrom_proxy()trusts by default) plus the configured CAs, as thereqwestclient inhttp.rsdoes. This keeps a non-intercepting proxy working, where Datadog's publicly signed certificate passes through.DD_SKIP_SSL_VALIDATION: now also applies to the TLS handshake inside the proxy tunnel, matching thereqwestclient used for logs and metrics (danger_accept_invalid_certs). Before this PR the tunnel handshake always verified against the public roots, so the setting had no effect there. When it is set, it still overridesDD_TLS_CERT_FILE, as inhttp.rs. Covered byproxy_tunnel_honours_skip_ssl_validation.tokio-rustlsmoves from[dev-dependencies]to[dependencies], andwebpki-rootsis added as a direct dependency. Both are already in the build;Cargo.lockgains one dependency edge.Three commits: the fix with unit tests, the process-level harness test, and the public-roots change.
Testing
Test 1: Process-level harness test (
tests/proxy_tls_test.rs)Steps
bottlecapbinary against the fake Extensions API withDD_PROXY_HTTPSset to a CONNECT proxy,DD_TLS_CERT_FILEset to a private CA, andDD_APM_DD_URLset to anhttps://localhostintake whose certificate that CA signed. The intake is the harnessFaultIntakebehind a small TLS front. The proxy tunnels only to that front and answers403to any other target.cargo nextest run --test proxy_tls_testonmainand on this branch.Result
main: fails. The proxy logs theCONNECT, the extension logsRequest failed after 4 attempts ... No requests sent, and the intake receives metrics and logs but no trace.CONNECT, and/nextis called before the invoke deadline.Test 2: Unit tests (
traces/http_client.rs)Steps
cargo test --lib traces::http_client. Six new tests use an in-test CONNECT proxy and a private-CA TLS server.proxy_tunnel_keeps_public_roots_when_tls_cert_file_is_setcovers a non-intercepting proxy: the server's certificate is signed by a CA standing in for the public roots, whileDD_TLS_CERT_FILEnames an unrelated CA.Result
DD_TLS_CERT_FILE, are still rejected).proxy_tunnel_trusts_tls_cert_file_caandproxy_tunnel_honours_skip_ssl_validationfail.proxy_tunnel_keeps_public_roots_when_tls_cert_file_is_setfails and the other five pass.Test 3: Real AWS, official layer v103 vs this change
Steps
DD_PROXY_HTTPSandDD_TLS_CERT_FILE(the proxy CA) set on both, sending to real Datadog. One uses the official v103 layer, the other a layer built from this branch's logic withbuild_bottlecap_layer.sh.Result
trace.agent.datadoghq.com)Request failed after 4 attempts ... No requests sentClient TLS handshake failed ... does not trust the proxy's certificateSuccessfully sent tracePOST /api/v0.2/traces->202 AcceptedIn the v103 run, logs and metrics still got
202through the same proxy.Stats use the same client. With
DD_LAMBDA_EXTENSION_COMPUTE_STATS=trueand invocations that send no trace, the v103 function loggedSTATS | Failed to send stats ... (attempt 1/3)through 3/3 and thenDropping undelivered payloads ... stats, with three TLS handshake failures at the proxy. The patched function loggedSTATS | Successfully flushed statsand the proxy returned202forPOST /api/v0.2/stats.Note: the AWS layer was built from a v102 base with the first revision of the fix (before the public roots were added to the tunnel config), and
http_client.rsonmainis byte-identical to v102. The harness test, the unit tests, and Clippy (CI's pinned 1.93.1 toolchain) were run against a build of currentmainplus the final change.Jira: SLES-3105