Skip to content

Fix intermittent "connection was not ready" failures in proxy_trusted_server tests #1485

Description

@prk-Jr

Description

Tests in crates/trusted-server-cli/tests/proxy_trusted_server.rs (added in #1251) fail intermittently in the cargo test (axum native) job of Run Tests. They fail at the shared send() helper (line 347):

should receive the actual Trusted Server response: hyper::Error(Canceled, "connection was not ready")

The failure is not in a test assertion: the request is never sent. It has hit two different tests in this file:

Run Commit Failing test
41109074229 5ee01b365 (main) actual_trace_authorization_rejects_foreign_origin_queries_and_missing_trust
41414595942 attempt 1 029f20467 (PR #1435) real_publisher_rewrites_public_port_and_strips_forwarding_credentials

The PR #1435 job passed when re-run with no code change, and later main commits pass this job; the main run was not re-run. PR #1435 does not touch this file or the dev-proxy code.

Likely cause (not confirmed): both failing tests send a second request over the same hyper::client::conn::http1::SendRequest (lines 428/443 and 506/558). send() calls send_request without first awaiting SendRequest::ready(). If the HTTP/1 connection hasn't finished the previous exchange, or has been closed, hyper rejects the new request with Canceled: connection was not ready. authenticated_proxy_reaches_real_trace_actions_for_every_upstream_mode also sends two requests per connection (line 385) and has not been seen to fail, so reuse alone is not sufficient; the failure looks like a race between the previous exchange finishing and the next send_request.

Reproduction steps

  1. Run cargo test --package trusted-server-cli --target <host-triple> --test proxy_trusted_server repeatedly, or let CI run it (it's part of ./scripts/test-cli.sh).
  2. Occasionally a test that reuses one connection panics at line 347 with the error above.

Expected behavior

The proxy integration tests pass deterministically. A second request on a reused connection waits until the connection is ready, or the test opens a new connection per request.

Affected area

CI / Tooling

Version

main at d1bb3fa19; the test file was last changed in #1251 (3d54bbd59).

Done when

  • send() awaits SendRequest::ready() before send_request (or each request uses its own connection), and a connection that closes early fails with a clear message rather than Canceled.
  • proxy_trusted_server passes in a repeated local loop (for example 50 iterations) on Linux and macOS.

Activity

  1. self-assigned this
    on Nov 2, 2026
  2. added theissue type on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions