Repository navigation
feat(box): Send the spans of the agent in a box to X-Ray on AgentCore Runtime - #2885
Conversation
… Runtime
## Problem
A box runtime on AgentCore Runtime sends no span of its agent to X-Ray.
Thus the console shows no trace for the runtime. A live test on
2026-11-20 found four causes. That test was correct only after manual
edits.
1. `agentcore create` writes `instrumentation.enableOtel: false` for a
box template. The agent does not start with
`opentelemetry-instrument`.
2. Box makes the environment of the agent from the profile only. The
agent gets no ADOT variable. Box gives it only
`OTEL_EXPORTER_OTLP_ENDPOINT`, and keeps each `OTEL_EXPORTER_OTLP_*`
name for box.
3. The Runtime profile has no `cloudwatch_traces` target. Thus box sends
no span to X-Ray.
4. The `OTEL_RESOURCE_ATTRIBUTES` that the Runtime sets does not get to
the agent. Thus the console cannot connect a span to the runtime.
Also, after this change, a warning of build is not correct: "the agent
does not run under opentelemetry-instrument, so it sends no traces. Set
instrumentation.enableOtel to false."
## Solution
The change is only in the Runtime profile that build derives, and in the
`@box/agentcore` pack that build writes. The `instrumentation.enableOtel`
flag of the runtime controls each part. The local `profile.toml`, the
commands of the templates, `agentcore dev`, and box do not change. When
`enableOtel` is `false`, build adds nothing, and the profile and the pack
are the same as before.
- The additions rule starts the agent with `opentelemetry-instrument`
when the command starts the Python of the venv. On a container, the
command starts with
`${{ workspace }}/.venv/bin/opentelemetry-instrument`. On CodeZip, the
zip has `bin/opentelemetry-instrument` with a `#!/usr/bin/env python3`
line, and the box cannot use that line. Thus the Python of the Runtime
starts the script:
`${{ inputs.python.path }} ${{ inputs.code.dir }}/bin/opentelemetry-instrument`.
`RuntimePaths.otelInstrument` holds the two start commands. A command that
starts with the script does not get it two times. A command of a
different program does not change.
- The `@box/agentcore` pack gives `[agent.env]` these ADOT variables:
`OTEL_PYTHON_DISTRO`, `OTEL_PYTHON_CONFIGURATOR`,
`AGENT_OBSERVABILITY_ENABLED`, `OTEL_TRACES_EXPORTER`,
`OTEL_LOGS_EXPORTER`, `OTEL_METRICS_EXPORTER = "none"`, and
`OTEL_PYTHON_EXCLUDED_URLS = "/ping"`. The last variable stops a span
for each health check of the Runtime. The pack sets no
`OTEL_EXPORTER_OTLP_*` variable. The new input
`runtime.resource_attributes` reads `OTEL_RESOURCE_ATTRIBUTES`, with
the default `service.name=<runtime>`, and the pack gives it to the
agent. Box merges the profile after the packs. Thus box uses a value
that the user sets in `[agent.env]`. Build gives no warning for that
value. This is the same for each other value of the user.
`renderAgentCorePack` gets the runtime.
- The additions rule adds `[telemetry.agentcore_traces]`
(`DEFAULT_TRACES_TARGET`) when no target is `cloudwatch_traces`. It
sends to X-Ray in the region of the Runtime, `${{ inputs.aws.region }}`,
with `include = ["deny", "permit", "trace"]` and a bare `aws://`. Its
name is `agentcore_traces_default` when a different target has the
name `agentcore_traces`. The comment lines at the top of the Runtime
profile name the target.
- The Box network model adds this target to `BoxNetwork.telemetry`
(`defaultTracesTelemetry`). Thus `boxTelemetryPolicy` gives the
execution role `xray:PutSpans` and `xray:PutTraceSegments`.
- `boxOtelWarnings` reads the derived profile. It gives a warning only
when `enableOtel` is `true` and the agent does not start with
`opentelemetry-instrument`.
- `agentcore create` writes no `instrumentation` for a box template.
Thus `enableOtel` is `true`, the same as for each other template.
- `docs/box-local-testing.md` tells about the OTEL start and the default
X-Ray target.
`DISABLE_ADOT_OBSERVABILITY` stops only the default log target, as
before. The default X-Ray target stays until `enableOtel` is `false`.
## Tests
`bun test` gives `6000 pass`, `7 skip`, and `0 fail`. `bun run typecheck`,
`bun run lint:check`, `bun run format:check`, `bun run build`,
`bun run audit`, and `bun run secrets:check` have no errors.
- `additions.test.ts`, "with OTEL on": the template command and a
command of the user get the start command of each build. A command that
starts with the script, as the program or as the script of the Python,
does not change. A command of a different program does not change.
The default X-Ray target comes with OTEL on and not with OTEL off. It
does not come when the profile or a pack has a `cloudwatch_traces`
target. A user value of an OTEL variable stays in `[agent.env]`.
- `additions.test.ts`, the two tests that use `old-renderer-tables.json`: with OTEL off,
each template gives the recorded profile. With OTEL on, it gives the
recorded profile with only the OTEL start and the default X-Ray target.
- `boxProfile.test.ts`, `renderAgentCorePack`: the ADOT variables, no
`OTEL_EXPORTER_OTLP_*` variable, the resource attributes input and its
default, and none of them with OTEL off.
- `boxNetworkSource.test.ts`, "the default X-Ray target of a runtime
with OTEL on": the grant has X-Ray with OTEL on, and build removes the
grant with OTEL off.
- `manager.boxNetwork.test.ts`, "build with OTEL on, as create writes a
box runtime": a new project gets the start, the pack variables, the
target, and the grant. With `enableOtel` set to `false`, it gets none of them.
A second build changes nothing.
- `cdk.test.ts`: the warning comes only for a box runtime with OTEL on,
when its agent does not start with `opentelemetry-instrument`.
- `runtime.test.ts`: each box template writes no `instrumentation`.
For each part of the change, a mutation caused an error in one or more
of these tests. Then the test script removed the mutation. There were
20 mutations.
A live test used a test account in us-west-2, box
0.1.11-preview.10, and `@aws/agentcore-cdk` from the `box-integration`
branch of the L3 at 08f8b23. `agentcore create` made a container project
from this branch, with no manual edit. Build, deploy, and three invokes
had no error, and build gave no OTEL warning. In `aws/spans`, the
runtime had 32 agent spans and 46 box spans in three traces. 20 box
decision spans had an agent span as their parent. There was no
`GET /ping` span. Each agent span had the `service.name` and the
`cloud.resource_id` of the runtime. The log stream `otel-rt-logs` of the
runtime had 97 records, and 46 of them were box records. The Box gateway
had 20 calls with no error, and each call was in one of the three
traces. The execution role had `BoxTelemetryToXRay`. `box profile show`
of the macOS preview.10 box gave the same values. It used the value of
the user for `OTEL_TRACES_EXPORTER` and `OTEL_RESOURCE_ATTRIBUTES`, and
the default `service.name=agent` when the Runtime sets no attributes.
The live test of CodeZip did not start. The runtime stopped with
"`[agent]` filesystem entry "/usr" is refused ... a system root is never
a grantable tree". The Python of the Runtime has the prefix `/usr`. A
second deploy with `enableOtel` set to `false` stopped with the same error. Thus
this change does not cause it, and CodeZip box on the Runtime does not
start today. Only unit tests examine the CodeZip start command. The test removed the
two stacks, and then CloudFormation had no stack of the test.
`box-integration` has one new change, [aws#2870](aws#2870). It writes `docs/box-local-testing.md` again. Git merged the file with no conflict, and the OTEL paragraphs of this branch stay in "What build and deploy do with the box profile". The merge adds one paragraph to step 10, which tells how to read box telemetry. That paragraph names the default X-Ray target and the log group `aws/spans`.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## box-integration #2885 +/- ##
=================================================
Coverage 96.79% 96.80%
=================================================
Files 766 766
Lines 66803 66928 +125
=================================================
+ Hits 64665 64791 +126
+ Misses 2138 2137 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
leo237
left a comment
There was a problem hiding this comment.
Thanks for this one, and for the live-test evidence. I read through the whole diff. The design holds together: the wrap goes only into the derived Runtime profile, the ADOT env goes only into the .box-dist pack, the local profile and agentcore dev don't change, and everything is gated on otelOn. I'm leaving this as a comment because of one item that has to be fixed before merge. Everything else is non-blocking.
Blocking
- An AWS account ID is in the commit message and the PR body. Commit 53a2962 ("A live test used account 005365278906 in us-west-2 ...") and the PR description's "Tests that CI does not do" section both name the test account. This is a public repo, and the squash-merge message usually comes from the PR body. Please drop the account ID from the PR body, and make sure it isn't in the final squash message. "a test account in us-west-2" is enough.
What I checked (no action needed)
- The reason 380e748 turned OTEL off. That commit was waiting on the Box Core fix for SigV4 signing of
X-Amzn-Trace-Id. strands-agents/box#537 is still open onmain, but the pinned release ships the fix. I downloadedbox-v0.1.11-preview.10-aarch64-unknown-linux-gnu.tar.gz, and its sha256 matchesboxRelease.json. Itsboxbinary contains the 14-name unsigned-header list right beside the SigV4 timestamp error string:baggage connection expect keep-alive proxy-authenticate proxy-authorization te traceparent tracestate trailer transfer-encoding upgrade user-agent x-amzn-trace-id. That is the botocore list #537 describes. Therelease.jsonin the archive names core commit8e4fa6b8. With that and your live result (20 gateway calls, no signature error), I'm satisfied that turning OTEL back on doesn't re-break gateway calls on the pinned box. One caveat: a box built fromstrands-agents/boxsource would not have the fix until #537 merges. - The distro is in every box template. Each box-capable template, including
agent-python-strands-box, hasaws-opentelemetry-distro ~= 0.21.0, so the wrapped command has a script to run. - IAM. The default
agentcore_tracestarget goes intoBoxNetwork.telemetrythroughdefaultTracesTelemetry.boxTelemetryPolicythen emitsBoxTelemetryToXRay(xray:PutSpans,xray:PutTraceSegments). That action has no resource type, so the Runtime-region vs build-region difference in the target doesn't matter. The xray grants are intact. - Must-survive fixes.
runtimeRunsInBox, the #2866 derivation rules, theenableApplicationLogsnotice text in the docs, and the preview.10 pin are all intact. Nothing in the diff reverts them. No internal URLs or names in the code or the docs. - Tests. Coverage is good. The template command and user commands are covered for both builds. Already-wrapped commands (as program or as script), non-venv programs plus the warning,
enableOtel:false(nothing added),DISABLE_ADOT_OBSERVABILITY, and user or pack-resolvedcloudwatch_tracestargets are all covered. So are the pack env with and without OTEL, the grant appearing and going away, and the end-to-end CodeZip command after the paths rule inmanager.boxNetwork.test.ts. I found no unsafe test patterns: no host paths, spawning, or network.
Non-blocking
- The default change. You've flagged this for the owners you named, and I'll leave the decision to them. One thing to note for them: projects created between 380e748 and this PR have
instrumentation.enableOtel: falsewritten inagentcore.json, so they stay off. That's probably fine, but it may be worth a line in release notes on how to turn traces on (delete the field). - CodeZip has only unit-test coverage. As you say, CodeZip box on the Runtime doesn't start today (the
/usrprefix grant). So${{ inputs.python.path }} ${{ inputs.code.dir }}/bin/opentelemetry-instrumentis unit-tested only, and it relies on the L3's zip havingbin/opentelemetry-instrument, which this repo can't check. Please re-run the CodeZip live test after the/usrfix (already listed in your follow-ups). DISABLE_ADOT_OBSERVABILITY=truewith OTEL on still wraps the command and sets the ADOT env, but there's no default traces or logs target, so the agent produces spans and logs that box drops. See the inline comment.- Merge order with #2882. A trial merge of #2882's head (490c197) onto this branch conflicts in
runtimeProfile/context.ts,runtimeProfile/types.ts, andruntimeProfile/derive.test.ts(RuntimePathsfields on both sides). Whichever PR lands second will need a small rebase. - Back-merge. This branch isn't on
box-integration-developmentyet, so it will need a back-merge after it lands.
CI
At head 34f9b0e, Linux, Windows, CodeQL/Analyze, and the CodeBuild e2e pass. verify / macOS was still pending when I posted.
…rue (comment of Leo on additions.ts)
|
The PR body no longer has the AWS account ID. The message of commit 53a29623 has the ID. Thus, for the squash merge, make the commit message from the PR title and the PR body only, and not from the commit messages. |
leo237
left a comment
There was a problem hiding this comment.
Re-review of the delta 34f9b0e..3e6d401 (one new commit).
Addressed
- The
additions.tsnit:otelOnnow also returns false when the runtime'senvVarssetDISABLE_ADOT_OBSERVABILITYtotrue. Because of that, theopentelemetry-instrumentwrap, the ADOT variables andresource_attributesinput of the@box/agentcorepack, the defaultagentcore_tracestarget, and the CDKopentelemetry-instrumentwarning all turn off together. The defaultagentcorelog target is still governed byobservabilityOffalone. I checked every caller (additions.ts,cdk.ts,boxNetworkWriters.ts->renderAgentCorePackwith the full spec runtime,manager.tsxnotice filter). The enabled path is unchanged: with no opt-out,otelOnis stillenableOtel ?? true. Tests cover the start command for each build, the pack (byte-equal toenableOtel: false), the CDK warning, and the manager build. The docs update is accurate. - Removed from the PR body: the test account ID is gone.
Still open (blocking)
- The test account ID is still in the message of commit 53a2962, which is still in the branch history. This repo allows merge commits and rebase merges. Its squash default is
squash_merge_commit_message = COMMIT_MESSAGES, so a default squash would also copy the ID into publicbox-integrationhistory. Thank you for asking for a squash from the title and body only. But that depends on whoever merges remembering to replace the message. Please either rewrite 53a2962's message (for example, reword it and force-push) or have the person merging confirm on this thread that it will be a squash with the cleaned message. - The branch is 2 commits behind
box-integration(#2884). It merges cleanly. - CI:
verify / Linuxandverify / Windowswere still pending when I posted. CodeQL, Analyze, macOS, and CodeBuild e2e pass.
Non-blocking
- Only
truecounts as the opt-out (case-insensitive, trimmed).1andyesdo not, which is consistent with the ADOT-style== "true"check. - Still deferred from before: the decision to change the default stays with the owners you named. The back-merge to
box-integration-developmentis still needed. #2882 was closed unmerged, so the CodeZip box still cannot start on the Runtime. CodeZip OTEL is therefore still validated only by unit tests, and the CodeZip start command has no live validation.
leo237
left a comment
There was a problem hiding this comment.
Re-review at 2ff18fe (merge of box-integration, bringing in #2883 and #2884).
Merge resolution: clean. The net diff against base (22 files, +950/-140) matches the net diff at 3e6d401 against its old base hunk for hunk, so no PR change was lost. boxMemory.ts (boxBedrockPolicy/boxMemoryPolicy partition handling), shortcuts.ts (boxGatewayBedrockHost/boxGatewayAgentCoreHost partition DNS suffix) and boxTelemetry.ts (BoxTelemetryToXRay) are byte-identical to base, so nothing from #2883 was reverted. In network.ts the PR only touches the telemetry merge and an import. The new agentcore_traces target uses the Runtime's region, and the X-Ray grant is Resource: "*", so no ARN or host is hard-coded and partition handling from #2883 stays consistent. No conflict markers. Branch is up to date with base. CI is green (Linux, macOS, Windows verify, CodeQL, CodeBuild e2e).
Still blocking: the message of commit 53a2962 still contains the test account ID (the PR body is clean). Please reword that commit and force-push, or confirm this will be squash-merged with a cleaned commit message.
Still open, non-blocking:
- Owners still need to sign off on changing the default (OTEL/X-Ray on by default for boxed agents).
- Back-merge to
box-integration-development. - CodeZip OTEL is only unit-tested, since #2882 was closed.
2ff18fe to
a1e611c
Compare
|
@leo237 I reworded the commit with the test account ID and force-pushed. The branch is now a1e611cd. The first commit is now 7136fba0, and its message says "a test account in us-west-2". Only the commit messages changed: the tree of the new head is the same as the tree of 2ff18fe. No commit message in this PR has an account ID now. |
leo237
left a comment
There was a problem hiding this comment.
Re-review at a1e611c: approving.
Blocker resolved: the account ID that was in a commit message is gone. The commits were reworded and history rewritten. The resulting tree is byte-identical to 2ff18fe, which we already reviewed. No commit message on the PR contains an account-ID-shaped number, and the PR body is clean too.
CI: CodeQL, Analyze, and the CodeBuild e2e run pass. verify (Linux/Windows/macOS) is still pending on the rewritten head. The same tree passed all checks at 2ff18fe. Merge only after checks pass.
Still open (non-blocking):
- The default-change decision still needs sign-off from the owners you named.
- Back-merge to
box-integration-development. - CodeZip OTEL is only unit-tested, since #2882 was closed.
Description
A box runtime on AgentCore Runtime sends no span of its agent to X-Ray. Thus the console shows
no trace for the runtime. A live test on 2026-11-20 found four causes:
agentcore createwritesinstrumentation.enableOtel: falsefor each box template, and theagent does not start with
opentelemetry-instrument.variable.
cloudwatch_tracestarget, so box sends no span to X-Ray.OTEL_RESOURCE_ATTRIBUTESof the Runtime does not get to the agent, so the console cannotconnect a span to the runtime.
This change adds only what is necessary on the Runtime. It changes the Runtime profile that build derives
(the rules of aws/agentcore-cli#2866) and the
@box/agentcorepack that build writes. Theinstrumentation.enableOtelflag of the runtimecontrols each part. The local
profile.toml, the commands of the templates,agentcore dev, andbox do not change. When
enableOtelisfalse, build adds nothing.This change sets a different default for box templates.
agentcore createwrites noinstrumentationfor a box template, soenableOtelistrue, the same as for each othertemplate. Varun and Quang own this area. Please examine this decision.
Decisions
venv with
opentelemetry-instrument. On a container, the command starts with${{ workspace }}/.venv/bin/opentelemetry-instrument. On CodeZip, the zip hasbin/opentelemetry-instrumentwith a#!/usr/bin/env python3line, and the box cannot use thatline. Thus the Python of the Runtime starts the script. A command that starts with the script
does not get it two times. A command of a different program does not change, and build gives a
warning for it.
OTEL_PYTHON_DISTRO,OTEL_PYTHON_CONFIGURATOR,AGENT_OBSERVABILITY_ENABLED,OTEL_TRACES_EXPORTER,OTEL_LOGS_EXPORTER,OTEL_METRICS_EXPORTER = "none",OTEL_PYTHON_EXCLUDED_URLS = "/ping",and
OTEL_RESOURCE_ATTRIBUTES. Box merges the profile after the packs, so box uses a value thatthe user sets in
[agent.env]. Build gives no warning for that value, the same as for each othervalue of the user. The pack sets no
OTEL_EXPORTER_OTLP_*variable, because box sets each one.runtime.resource_attributesreadsOTEL_RESOURCE_ATTRIBUTESfrom the Runtime, with the defaultservice.name=<runtime>.cloudwatch_tracestarget, the Runtime profile gets[telemetry.agentcore_traces], in the region of the Runtime, the same as the default log target.The Box network model gives this target to the telemetry policy, so the execution role gets
xray:PutSpansandxray:PutTraceSegments.envVarsof the runtime setDISABLE_ADOT_OBSERVABILITYto"true",build adds no default X-Ray target. This is the same as for the default log target. Then the
execution role gets no X-Ray grant for it. The opt-out also stops the
opentelemetry-instrumentstart, the ADOT
[agent.env]of the pack, and theruntime.resource_attributesinput. One check,otelOn, controls all of these parts. Acloudwatch_tracestarget of the user stays.enableOtelistrueand the agent does not start withopentelemetry-instrument.Tests that CI does not do
(aws/agentcore-cli#2871), and the L3 at
aws/agentcore-l3-cdk-constructs@08f8b230.
agentcore createmade the project from this branch, with no manual edit. Build, deploy, andthree invokes had no error, and build gave no OTEL warning.
aws/spanshad 32 agent spans and 46 box spans in three traces. 20 box decision spans had anagent span as their parent. There was no
GET /pingspan.service.nameand thecloud.resource_idof the runtime.otel-rt-logsof the runtime had 97 records. 46 of them were box records.three traces. strands-agents/box#537 is in
preview.10.
BoxTelemetryToXRay.box profile showof the macOS preview.10 box on the Runtime profile in.box-dist/gavethe ADOT variables, the resource attributes of the Runtime, and the default X-Ray target. It used
the value of the user for
OTEL_TRACES_EXPORTERandOTEL_RESOURCE_ATTRIBUTES. It usedservice.name=agentwhen the Runtime set no attributes.with "
[agent]filesystem entry "/usr" is refused ... a system root is never a grantable tree".The Python of the Runtime has the prefix
/usr. A second deploy withenableOtelset tofalsestopped with the same error. With
enableOtelon, the L3 startsstart.pywithopentelemetry-instrument, andstart.pystarted box correctly. Only unit tests examine theCodeZip start command of the agent.
The test also removed the five log groups that stay after a stack: the three log groups of the
runtimes, the CodeBuild log group, and the Lambda log group. The spans stay in
aws/spans.Follow-ups
opentelemetry-instrument. Each box template hasaws-opentelemetry-distro. If a user removes it, the agent does not start on the Runtime.agentcore createmade while box templates had OTEL off keepinstrumentation.enableOtel: falseinagentcore.json. The release note must tell users to remove that field to turn traces on./usrgrant. That problem is not part of this PR, and the CodeZip in a box work owns it. Thus the CodeZip
OTEL path has unit tests only.
Related Issue
None.
Documentation PR
This change updates
docs/box-local-testing.md.Type of Change
Testing
How have you tested the change?
bun test:6009 pass,7 skip,0 fail.bun run test:e2e, or explained why they are not applicable(CI does not run them for you): I did not run
bun run test:e2e. The live test above deployed andinvoked a container project and a CodeZip project.
bun run typecheckbun run lint:checkbun run format:checkbun run auditbun run secrets:checkbun run buildsrc/assets/, I updated affected snapshots withbun test <test-file> --update-snapshotsand committed themby this PR
command.md(the release workflow regenerates it)For each part of the change, a mutation caused an error in one or more tests. Then the test script
removed the mutation. There were 19 mutations in the first commit. The message of the first commit
says 20 mutations, but the correct number is 19. For the opt-out commit, one more mutation removed
the
DISABLE_ADOT_OBSERVABILITYcondition, and 3 tests showed an error.Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.