Skip to content

[SVLS-10070] fix(proxy): remove finished tasks from the Runtime API proxy's JoinSet - #1509

Open
lym953 wants to merge 5 commits into
yiming.luo/listener-joinset-leakfrom
yiming.luo/proxy-joinset-leak
Open

lym953 wants to merge 5 commits into
yiming.luo/listener-joinset-leakfrom
yiming.luo/proxy-joinset-leak

Conversation

@lym953

@lym953 lym953 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The extension can run a proxy in front of the Lambda Runtime API. It starts the proxy when the function uses Lambda Web Adapter or the Datadog wrapper, for example with App and API Protection. For each invocation, the proxy starts two tasks in a Tokio JoinSet: one for the event and one for the response. A JoinSet keeps each finished task until something joins it, and the proxy joins only at shutdown. So the set grows by two tasks per invocation for the life of the execution environment.

This is the same pattern that #1508 fixes in the listener for the tracer's start and end calls. That leak made the extension's memory grow about 14 KB per invocation in #1496. I have not measured the memory growth from the proxy.

This PR

Before each new task, the proxy removes the tasks that have finished, with JoinSet::try_join_next(). This is the same change as #1508. Each module keeps its own copy of the helper, as each already keeps its own shutdown function for its set. The proxy still joins the tasks that are running at shutdown.

This PR is based on #1508. Merge #1508 first.

Testing

Unit test

Steps

  1. Run cargo test --lib proxy::interceptor.
  2. Remove the two calls before each spawn, and run the new test again.
  3. Run the new test 20 times with the fix.

Result

The unit test starts a fake Lambda Runtime API and sends 100 next and response cycles through the proxy's routes. It reads each body to the end, because the proxy starts its work for a body only when the whole body passes through. Then it counts the tasks left in the set and runs the proxy's shutdown.

Check Before After
Tasks in the set after 100 cycles 200 (test fails) 2 or fewer (test passes)
Shutdown finishes in 5 seconds and empties the set not reached passes

With the fix, the unit test passed in 20 of 20 runs.

Process-level test

The new test in the lifecycle contract tests starts the real binary with the Datadog wrapper and Data Streams Monitoring, which turns the proxy on. It runs 3,500 warm invocations. In each one, it sends five next and response cycles through the proxy on port 9000. On the first invocation, it checks that the fake Lambda API received both forwarded calls. On Linux, it checks that RSS grows at most 3 MiB from invocation 500 to the end, and then that the extension shuts down cleanly.

The test sends five cycles per invocation because one leaked proxy task keeps less memory than one leaked listener task. With one cycle per invocation, RSS grew only 1.7 to 2.6 MiB without the fix, under the limit. The proxy does not link its calls to the extension's invocations.

Steps

  1. In a Linux container (arm64, Rust 1.93.1), run cargo test --test lifecycle_contract_test runtime_api_proxy_memory_stays_bounded with this PR.
  2. Remove the two calls before each spawn, and run it again.

Result

Check Before After
RSS growth over the 3,000 measured invocations 6,428 and 7,032 KiB in 2 runs (test fails) 0 and 676 KiB in 2 runs (test passes)

On macOS, the test runs 10 invocations and checks only the calls and the shutdown, because the harness measures memory only on Linux. I did not run it on the x86_64 CI runners yet. I did not run the proxy on Lambda.

🤖 Generated with Claude Code

…roxy's JoinSet

The proxy in front of the Lambda Runtime API spawns two tasks per invocation
into a JoinSet that is joined only at shutdown, so the set grows by two finished
tasks per invocation. Remove finished tasks before each spawn, as the lifecycle
listener does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The helper-only test would pass even without production task reaping, so a process-level regression is needed to check bounded memory growth after warmup on Linux and clean shutdown.

1 open issue
What changed in this PR

This PR prevents completed tasks from accumulating in the Lambda Runtime API proxy, applying the lifecycle listener’s cleanup pattern from #1508.

Changes:

  • Drain completed tasks before spawning invocation-event and response tasks.
  • Preserve shutdown handling for tasks still running.
  • Add a unit test verifying that the cleanup helper removes completed tasks.
File Description
bottlecap/​src/​proxy/​interceptor.rs Adds completed-task cleanup at both spawn sites and a helper unit test.

Review effort: Balanced

Comment thread bottlecap/src/proxy/interceptor.rs Resolved
@datadog-official

datadog-official Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0876a08 | Docs | View more details | Give us feedback!

lym953 and others added 2 commits November 19, 2026 16:21
…on cycles

Replace the helper-only test with one that sends 100 next/response cycles
through the proxy's router to a fake Runtime API, then checks that the set
holds at most two tasks and that shutdown finishes. Without the calls before
each spawn, the set holds 200 tasks and the test fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the proxy task-cleanup change.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@lym953
lym953 marked this pull request as ready for review November 19, 2026 21:37
@lym953
lym953 requested a review from a team as a code owner November 19, 2026 21:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T23:43:19.949437Z 0876a08 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 263b912a95

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bottlecap/src/proxy/interceptor.rs Resolved
lym953 and others added 2 commits November 19, 2026 18:12
…the Runtime API proxy

Add a process-level test that starts the real binary with the proxy on (the
Datadog wrapper with Data Streams Monitoring) and runs 3,500 warm invocations,
each with five next/response cycles through the proxy. On Linux, RSS must grow
at most 3 MiB after a warm-up of 500 invocations. Without the fix, RSS grew
6.4 to 7.0 MiB over the 3,000 measured invocations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants