Repository navigation
Conversation
…s JoinSet The listener that serves the tracer's start-invocation and end-invocation calls spawns each call's work into a JoinSet, which keeps every finished task until it is joined. The set was joined only at shutdown, so it grew by two tasks per invocation for the life of the execution environment. Remove finished tasks before each spawn. Fixes #1496 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the lifecycle-listener task cleanup change.
No open issues 🎉
What changed in this PR
Fixes lifecycle-listener memory growth during warm Lambda invocations with universal instrumentation, addressing the retained tasks described in #1496.
Changes:
- Removes completed tasks from the listener’s
JoinSetbefore spawning start- and end-invocation work. - Logs task errors during cleanup while preserving shutdown handling for running tasks.
- Adds a regression test covering task retention across 100 start/end request pairs.
| File | Description |
|---|---|
bottlecap/src/lifecycle/listener.rs |
Adds completed-task cleanup to both invocation handlers and tests that retained task counts remain bounded. |
Review effort: Balanced
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c4fe592a3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…mentation invocations Add a process-level test that runs the real binary through 3,500 warm invocations, each with a start-invocation and an end-invocation call, and checks on Linux that RSS grows at most 3 MiB after a warm-up of 500. Without the fix, RSS grew about 7 MiB over the 3,000 measured invocations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
With universal instrumentation on, the extension's memory grows on every warm invocation and is never freed. Universal instrumentation is the default for Go. A customer reported in #1496 that several 1024 MB Go functions reach their memory limit after an execution environment serves about 10,000 to 20,000 invocations. They saw the growth on v94 and v101. With
DD_UNIVERSAL_INSTRUMENTATION=false, the memory stays flat.Cause
With universal instrumentation, the tracer calls the extension's
/lambda/start-invocationand/lambda/end-invocationendpoints once each per invocation. The extension's listener for these calls puts the work for each call into a TokioJoinSet. AJoinSetkeeps each finished task until something joins it, and the listener joins only at shutdown. So the set grows by two tasks per invocation for the life of the execution environment. #796 added this in August 2025.This PR
Before each new task, the listener removes the tasks that have finished, with
JoinSet::try_join_next(). The listener still joins the tasks that are running at shutdown.The proxy in front of the Lambda Runtime API has the same pattern. A separate PR fixes it.
Testing
Unit test
Steps
cargo test --lib lifecycle::listener.Result
The unit test sends 100 start and end pairs through the listener and counts the tasks left in the set.
Process-level test
The new test in the lifecycle contract tests starts the real binary against the harness's fake Lambda API and fake intake. It runs 3,500 warm invocations, and each one makes a start-invocation call and an end-invocation call. On Linux, it checks that RSS grows at most 3 MiB from invocation 500 to the end, and then that the extension shuts down cleanly.
Steps
cargo test --test lifecycle_contract_test universal_instrumentation_memory_stays_boundedwith this PR.Result
With this PR, RSS stops growing by invocation 500. Without it, RSS grows in a straight line, about 2.2 KiB per invocation. On macOS, the test runs 10 invocations and checks only the calls and the shutdown, because the harness measures memory only on Linux. I did not run it on the x86_64 CI runners yet.
Lambda
Steps
datadog-lambda-gov1.32.0,provided.al2023, arm64, 1024 MB, reserved concurrency 1. Use their settings: payload capture on, X-Ray merge on, X-Ray active tracing.VmRSSof thedatadog-agentprocess from/proc/<pid>/statusin the handler.Result
mainwithout this PRmainwith this PRThe two
mainbuilds are the same commit, so the difference between them comes from this PR. With this PR, the extension's RSS stays between 76 and 89 MB from invocation 2,000 to 6,000. With universal instrumentation off, it stays between 73 and 83 MB over the same range. Only the Go tracer was tested.Open questions
🤖 Generated with Claude Code