Skip to content

[SVLS-10210] fix: skip OOM metric for successful invocations at the memory limit - #1539

Draft
lym953 wants to merge 1 commit into
mainfrom
yiming.luo/oom-skip-successful-invocations
Draft

lym953 wants to merge 1 commit into
mainfrom
yiming.luo/oom-skip-successful-invocations

Conversation

@lym953

@lym953 lym953 commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Problem

Java functions report the out-of-memory (OOM) metric, aws.lambda.enhanced.out_of_memory, for invocations that succeed (#1536). The reporter sees about 1000 times more OOM metrics than errors after an upgrade from v97 to v99.

From v98 (#1241), the extension counts an OOM when a platform report has Max Memory Used equal to Memory Size. This applies to every runtime. A JVM heap can grow to fill the memory size and keep working. So a warm Java function can meet this condition on every invocation without running out of memory.

This PR

The extension counts the memory-equality case only when the invocation status is not Success. This PR does not change the two other OOM checks: the runtime-specific OOM log line, and the Runtime.OutOfMemory error type.

The OOM case that #1241 added the check for still counts. In the Node repro for #1237, the OOMs that only the equality check caught had status timeout. The other OOM cases had status error.

Testing

Unit tests

Steps

  1. cargo test --lib
  2. cargo fmt --check
  3. cargo clippy --workspace --all-targets --features default

Result

All 765 tests pass, including a new test: a successful invocation at the memory limit sends no OOM metric. The existing equality test now uses status Error. Format and Clippy are clean.

Java function in Lambda

Steps

  1. Deploy a Java 21 function with 512 MB. The handler keeps a static heap cache at 90% of the maximum heap, holds a 20 MB direct buffer, and returns normally.
  2. Invoke it several times, and read Max Memory Used and Status from each platform report.
  3. Query aws.lambda.enhanced.out_of_memory for the function in Datadog.

Result

Check Before (v99, x86_64) After (this branch, arm64)
Invocations 10 5
Status of each invocation Success Success
Max Memory Used / Memory Size 512 / 512 MB 512 / 512 MB
OOM metrics sent 9 0

The same function with v97 sent 0 OOM metrics. The before and after runs used different architectures. "After" ran inside the existing OOM integration suite, and the other OOM cases (Node, Python, Java, .NET, Go) each sent exactly 1 OOM metric. The Ruby case sent 0 in that run because the kernel killed the extension (Extension.Crash). It passed in the next run.

This PR adds no integration test. A successful invocation reports exactly 512 of 512 MB only in a narrow range of memory use. The same setup gave 512 MB in one run and 511 MB in the next. A test for this would be flaky, or it would pass without reaching the condition.

Impact

On arm64 functions with 512 MB, invocations that the kernel killed for memory reported Max Memory Used of 510 to 511 MB, not 512. So the memory-equality check does not catch those OOMs. Their reports had the error type Runtime.OutOfMemory, which the extension also checks. I measured this only at 512 MB in us-east-1.

🤖 Generated with Claude Code

…emory limit

The extension counted `aws.lambda.enhanced.out_of_memory` for every invocation
whose PlatformReport had max_memory_used_mb == memory_size_mb, including
successful ones. A JVM heap can fill the memory size and keep working, so
Java functions reported OOMs on invocations that succeeded (#1536).

Count the memory-equality case only when the invocation status is not
Success. The log-line and Runtime.OutOfMemory paths are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant