Skip to content

[SVLS-10070] fix(lifecycle): remove finished tasks from the listener's JoinSet - #1508

Open
lym953 wants to merge 3 commits into
mainfrom
yiming.luo/listener-joinset-leak
Open

lym953 wants to merge 3 commits into
mainfrom
yiming.luo/listener-joinset-leak

Conversation

@lym953

@lym953 lym953 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

With universal instrumentation on, the extension's memory grows on every warm invocation and is never freed. Universal instrumentation is the default for Go. A customer reported in #1496 that several 1024 MB Go functions reach their memory limit after an execution environment serves about 10,000 to 20,000 invocations. They saw the growth on v94 and v101. With DD_UNIVERSAL_INSTRUMENTATION=false, the memory stays flat.

Cause

With universal instrumentation, the tracer calls the extension's /lambda/start-invocation and /lambda/end-invocation endpoints once each per invocation. The extension's listener for these calls puts the work for each call into a Tokio JoinSet. A JoinSet keeps each finished task until something joins it, and the listener joins only at shutdown. So the set grows by two tasks per invocation for the life of the execution environment. #796 added this in August 2025.

This PR

Before each new task, the listener removes the tasks that have finished, with JoinSet::try_join_next(). The listener still joins the tasks that are running at shutdown.

The proxy in front of the Lambda Runtime API has the same pattern. A separate PR fixes it.

Testing

Unit test

Steps

  1. Run cargo test --lib lifecycle::listener.
  2. Remove the two new calls that remove finished tasks, and run the new test again.

Result

The unit test sends 100 start and end pairs through the listener and counts the tasks left in the set.

Check Before After
Tasks in the set after 200 requests 200 (test fails) 2 or fewer (test passes)

Process-level test

The new test in the lifecycle contract tests starts the real binary against the harness's fake Lambda API and fake intake. It runs 3,500 warm invocations, and each one makes a start-invocation call and an end-invocation call. On Linux, it checks that RSS grows at most 3 MiB from invocation 500 to the end, and then that the extension shuts down cleanly.

Steps

  1. In a Linux container (arm64, Rust 1.93.1), run cargo test --test lifecycle_contract_test universal_instrumentation_memory_stays_bounded with this PR.
  2. Remove the two calls that remove finished tasks, and run it again.

Result

Check Before After
RSS growth over the 3,000 measured invocations 7,132 to 7,236 KiB in 3 runs (test fails) 0 to 568 KiB in 2 runs (test passes)

With this PR, RSS stops growing by invocation 500. Without it, RSS grows in a straight line, about 2.2 KiB per invocation. On macOS, the test runs 10 invocations and checks only the calls and the shutdown, because the harness measures memory only on Linux. I did not run it on the x86_64 CI runners yet.

Lambda

Steps

  1. Deploy the customer's Go handler from Extension memory grows across warm invocations with universal instrumentation (v94, v101) #1496: datadog-lambda-go v1.32.0, provided.al2023, arm64, 1024 MB, reserved concurrency 1. Use their settings: payload capture on, X-Ray merge on, X-Ray active tracing.
  2. Send 6,000 direct invocations with an API Gateway HTTP API (v2) event, one at a time, so one execution environment serves all of them.
  3. Every 250 invocations, read the VmRSS of the datadog-agent process from /proc/<pid>/status in the handler.
  4. Repeat for four extension builds.

Result

Extension build Universal instrumentation Extension RSS at invocation 250 Extension RSS at invocation 6,000
Public layer v101 on 69 MB 150 MB
Public layer v101 off 69 MB 81 MB
main without this PR on 59 MB 117 MB
main with this PR on 64 MB 81 MB

The two main builds are the same commit, so the difference between them comes from this PR. With this PR, the extension's RSS stays between 76 and 89 MB from invocation 2,000 to 6,000. With universal instrumentation off, it stays between 73 and 83 MB over the same range. Only the Go tracer was tested.

Open questions

  • In a local test with a counting allocator, the set held about 2 KB of live heap per invocation. On Lambda, RSS grew 10 to 14 KB per invocation. Allocator fragmentation could explain the difference, but I did not check it. The Lambda runs show only that this PR removes the RSS growth.

🤖 Generated with Claude Code

…s JoinSet

The listener that serves the tracer's start-invocation and end-invocation calls
spawns each call's work into a JoinSet, which keeps every finished task until it
is joined. The set was joined only at shutdown, so it grew by two tasks per
invocation for the life of the execution environment. Remove finished tasks
before each spawn.

Fixes #1496

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the lifecycle-listener task cleanup change.

No open issues 🎉

What changed in this PR

Fixes lifecycle-listener memory growth during warm Lambda invocations with universal instrumentation, addressing the retained tasks described in #1496.

Changes:

  • Removes completed tasks from the listener’s JoinSet before spawning start- and end-invocation work.
  • Logs task errors during cleanup while preserving shutdown handling for running tasks.
  • Adds a regression test covering task retention across 100 start/end request pairs.
File Description
bottlecap/​src/​lifecycle/​listener.rs Adds completed-task cleanup to both invocation handlers and tests that retained task counts remain bounded.

Review effort: Balanced

@lym953
lym953 marked this pull request as ready for review November 19, 2026 21:25
@lym953
lym953 requested a review from a team as a code owner November 19, 2026 21:25
@lym953
lym953 requested a review from jcaparelli15 November 19, 2026 21:25
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T23:12:58.238623Z e548612 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c4fe592a3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread bottlecap/src/lifecycle/listener.rs Resolved
…mentation invocations

Add a process-level test that runs the real binary through 3,500 warm
invocations, each with a start-invocation and an end-invocation call, and
checks on Linux that RSS grows at most 3 MiB after a warm-up of 500. Without
the fix, RSS grew about 7 MiB over the 3,000 measured invocations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants