Repository navigation
test(e2e): Deploy and invoke every Strands Box template - #2865
Conversation
Adds a box tag and a test for each box template: create, deploy, invoke, then check the Runtime's logs show box running the agent, and tear the stack down. They skip agentcore dev, which runs box on a fixed ingress port.
Add agent-python-strands-box and agent-python-strands-box-container to the runtime template cases. CI deploys with the released @aws/agentcore-cdk, which has no Box support yet, and has no box on PATH for CodeZip dev, so the cases skip in CI. A CodeZip box agent's dev serves on the profile's ingress port, so that case pins dev to 8080.
The box cases test create, deploy and invoke on AgentCore Runtime, not agentcore dev, so they no longer start dev or invoke locally.
Add a py_strands_box case for PYTHON_3_10 through PYTHON_3_13, beside the template's default PYTHON_3_14, by setting runtimeVersion in agentcore.json after create. The remote invoke now also needs a 2xx status and a body, as the local invoke does, so an agent that failed to start in its box fails the case.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## box-integration #2865 +/- ##
===================================================
- Coverage 96.75% 96.75% -0.01%
===================================================
Files 757 757
Lines 65642 65659 +17
===================================================
+ Hits 63512 63528 +16
- Misses 2130 2131 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
leo237
left a comment
There was a problem hiding this comment.
Review: MQ-7 (e2e cases for the Strands Box templates)
This is MQ-7 of the squash plan. The plan doc returned 401 for me, so I checked the commits against the branches directly.
Commit list check
- The four commits have the same patches as
a6d52f82,e48bf2db,91fa62baand8d25cd6donbox-integration-development.e2eTest/at this head is byte-identical toe2eTest/on dev, so the dev-only1b6ccdb5revert andc17ecfc8reapply cancel out and nothing is lost. - Leaving out
868325c0is correct. It is onbox-integrationasb69851ca. - There are no extra commits. The diff touches only
e2eTest/README.md,e2eTest/constants.tsande2eTest/project/templates.test.ts. None of the protectedbox-integration-only fixes (#2845, #2848/#2849, #2850, #2851, #2852, #2853, #2857) are touched, and nothing is applied twice. - The open PRs #2859, #2860, #2862, #2863 and #2864 change no
e2eTest/files, so there is no conflict with them. - The base is
box-integration, and the PR is mergeable.
CI: all checks pass (verify Linux, Windows and macOS, CodeQL, Analyze). The e2e job is skipped on PRs, so these cases have not run in CI. As the description says, the e2e suite itself was not run.
Blocking
- The
BOX_TEMPLATESdescribe is not skipped in CI, so it will fail whenever the full e2e suite runs there.e48bf2dbsays the Box cases must skip in CI because CI deploys with the released@aws/agentcore-cdk, which has no Box support, and has noboxon PATH.createfails withoutbox. Theskip: process.env.CI === "true"guard went onto the newRUNTIME_TEMPLATESentries but not onto the sevenBOX_TEMPLATEScases; only the container case has a skip, and that one is the Linux-only skip. The block is also taggedruntime, so plainbun run test:e2e(the push CI step inci.yml) and--tagsFilter=runtimeboth pick it up.e2e-test.ymlinstalls nobox. Today the e2e job onbox-integrationpushes is skipped by the authorization gate, so CI is green, but the first authorized push or dispatch will go red on all three OSes. Dev has the same code, so no later plan commit fixes this. A fix: addskip: process.env.CI === "true"to eachBOX_TEMPLATEScase, or skip the whole describe in CI (see the inline comment).
Non-blocking
- The FALLBACK/FAILED check can only see lines that contain
running profile.--querybecomes a CloudWatch filter pattern (handlers/observability/logs.ts→logEventFilterPattern), so the logs returned are only the events that match"running profile".not.toMatch(/FALLBACK|FAILED/)catches a failure only if box prints it on that same line. Either run a second query for those markers, or note in a comment that box prints them on therunning profileline. - The Box describe's remote invoke checks less than the runtime describe now does.
8d25cd6dadded the 2xx status and non-empty body checks to the runtime block only. The Box block checks justcomplete, so an HTTP agent that box started but that returns 500 passes, becauserunning profileis still logged. - The four per-Python cases (
py_strands_box_3_10–3_13) do not check that box ran. They are inRUNTIME_TEMPLATES, so they get the new 2xx/body check but notassertRunsInBox. Becausestart.pyexecsboxwith no fallback, a 2xx answer does mean the agent started. Still, callingassertRunsInBoxforruntime.box/Box cases would match what the PR says it does. - The same templates deploy twice.
py_strands_box(runtime block) andagent_python_strands_box(Box block) cover the same template, andpy_strands_box_containeris in both lists under the same name. A full run outside CI deploys 12 Box stacks. Consider keeping the per-Python matrix and dropping the duplicate default and container entries fromRUNTIME_TEMPLATES, or giving the duplicates different names. - New type field
box?: booleanis never read..map(... box: true)sets it, but nothing reads it. Its doc comment describesagentcore devbehaviour that no code branches on. Either use it (for example to callassertRunsInBox, as in item 4) or drop it. - The
ALL_TEMPLATEScomment is wrong. It says the list gives each project "its own name and local dev port". The dev port still comes fromRUNTIME_TEMPLATES.indexOf, and Box cases never start dev;ALL_TEMPLATESsets only the project name. - The
runtimeVersionoverride is a raw edit ofagentcore.json. That is fine for a test. The{ runtimes: { runtimeVersion?: string }[] }cast passes the other keys through unchanged, andpath.joinkeeps it Windows-safe.
Windows safety: the paths use path.join, and there are no POSIX modes, sh spawns, empty-env spawns or : splits.
Planned open items: this PR is test-only and fixes none of them. The "no local-profile test for the A2A, AG-UI and MCP box templates" item is still open: this PR adds deploy and invoke coverage for those templates on the Runtime, not local-profile tests.
I am using COMMENT rather than APPROVE only because of item 1. With the CI skip added, this is ready to approve.
- Skip the Box template describe in CI, which deploys with the released CDK and has no box on PATH. - Look for the README's error lines with their own logs query, as --query returns only the events it matches. - Require a 2xx status and a body from a Box agent's remote invoke. - Check that box ran for the per-Python cases too, through the box field. - Drop py_strands_box and py_strands_box_container, which deployed the same templates as the Box describe. - Say that ALL_TEMPLATES sets only the project name.
leo237
left a comment
There was a problem hiding this comment.
Re-review: MQ-7 (e2e cases for the Strands Box templates), head 8d5f619
This is MQ-7 of the squash plan. The plan doc still returns 401 for me, so I checked against the branches directly.
What changed since 7d45b881: one new commit, 8d5f6199 ("Address the review of #2865"), on top of the earlier four. It is not a rebase, and the base has not moved: the merge-base is still f5503f02 (#2857). The PR still touches only e2eTest/README.md, e2eTest/constants.ts and e2eTest/project/templates.test.ts, so none of the box-integration-only fixes (#2845, #2848/#2849, #2850, #2851, #2853, #2857, 68d2bbb4, c1bc4576) are affected.
Dev parity: 8d5f6199 exists only on box-integration. It has no match on box-integration-development, where the newest e2eTest/ commit is still 8d25cd6d, and dev's templates.test.ts differs from this head by exactly this commit. See item 1 below.
Earlier comments
All of them are addressed:
- Blocking, CI skip: fixed. The Box describe is now
describe.skipIf(process.env.CI === "true"), and the per-Python cases keepskip: process.env.CI === "true". GitHub Actions setsCI=true, so neither the push job's plainbun run test:e2enor--tagsFilter=runtimepicks them up. - FALLBACK/FAILED filter: fixed. A second
logsquery,?ERROR ?"blocked by egress control" ?AccessDenied, looks for the error lines. That is valid OR syntax in a CloudWatch filter pattern, andbuildFilterPatternpasses it through unchanged.logswrites only the events to stdout; the announcement goes to stderr. That means thenot.toMatchcan't hit header text, andexitCodeis checked. The search terms match what the rootREADME.mdsays a working box agent never logs. - Box invoke 2xx and body checks: fixed.
- Per-Python cases now check box ran: fixed, through
box: trueandif (runtime.box) await assertRunsInBox(...). - Duplicate deploys: fixed.
py_strands_boxand the runtime-blockpy_strands_box_containerare gone, and the Box container case is renamedagent_python_strands_box_container. A correction to my earlier review: before this change a full non-CI run deployed 13 Box stacks, not 12 (7 in the Box block, pluspy_strands_box, the four per-Python cases and the runtime-block container case). It now deploys 11, which matches the author'svitest listcount. The names are unique, so theALL_TEMPLATES.indexOfproject names stay distinct. box?: booleanunused: fixed. It is now read, and its doc comment says what it does.ALL_TEMPLATEScomment: fixed.
CI: all checks pass at 8d5f619: verify on Linux, Windows and macOS, CodeQL, Analyze, and the agentcore-e2e CodeBuild check. The e2e job is skipped on PRs, so the e2e cases still have not run in CI. Known flaky checks on the base are non-blocking: the Windows transactionSearch EFAULT test and the 5 s timeouts in the macOS screen tests (the macOS queue is slow).
New findings (all non-blocking)
- Port
8d5f6199tobox-integration-development. It exists only onbox-integration. If a later phased PR squashese2eTest/project/templates.test.tsfrom dev, it brings back the missing Box-describe CI skip (the earlier blocking item) and the duplicate deploys. A cherry-pick to dev keeps the two branches in step. - The error query runs once, with no wait. It runs as soon as the
running profileline shows up (see the inline comment). Box logsrunning profilewhen it starts, before it handles the request, so an error logged during the invoke can reach CloudWatch after the check has run. The 2xx and body checks catch most of these failures. Moving the error query into the retry, or doing one short re-check, would close the gap. ERRORis a broad term. Any ERROR-level line in the runtime's log group fails the case, whether it comes from the agent or from a library it uses (for example an OTEL exporter warning logged at ERROR). That matches the README's promise, so I'm only flagging it as a likely source of flakes. If it does flake, narrow it to box's own error prefix.--tagsFilter='box'does not run the four per-Python box cases.e2eTest/README.mdgives this command for the box tests. The four cases are in theruntime/canarydescribe and lack theboxtag, so they run only with--tagsFilter=runtimeor a full run. Either say this in the README or move or tag them.- Nit:
BOX_TEMPLATESno longer has.filter((r) => !r.skip). No entry setsskiptoday, but the shared type still allows it, and askipadded later to a Box entry would do nothing. Either keep the filter or leave a comment. - Nit: the new comment says "the README". The list it refers to is in the root
README.md, note2eTest/README.md. Naming the file would help.
Windows safety: the new code adds no hard-coded / paths, : splits, POSIX modes, sh spawns or empty-env spawns. The logs arguments go to cli.run as an argv array, so the quotes in the filter pattern don't depend on a shell.
Planned open items: this PR is test-only and fixes none of them. The "no local-profile test for the A2A, AG-UI and MCP box templates" item is still open.
Decision: APPROVE. The one blocking item is fixed, and the items above are non-blocking.
- Wait 30 s after 'running profile' before looking for error lines, so lines from the invoke have reached CloudWatch. - Say in e2eTest/README.md that --tagsFilter='box' leaves out the per-Python box cases, and that the Box describe's skipIf is the only skip for BOX_TEMPLATES. - Name the root README.md as the source of the error lines.
tejaskash
left a comment
There was a problem hiding this comment.
Requesting changes. Thanks for working through leo237's points so quickly: the CI skip, the second log query for ERROR, blocked by egress control and AccessDenied, the 2xx and body check, and the dropped duplicates all look good. What's left is that, as far as I can tell, these cases can't pass yet on any machine, and nobody has run the suite at this head (the PR's e2e check is skipped), so the opt-in and the per-Python cases need another look (inline).
One coverage gap worth noting: the only Box-specific assertions are the running profile line and no error lines. A profile whose egress rules block Bedrock would still pass for the minimal and FastMCP templates, which make no outbound call. One case that calls out through the Box gateway and checks the response would cover the allowed path, and a denied host would cover the other side.
Concurrent build and deploy commands each download box into their own work folder, then moved it into ~/.agentcore/box/<version>-<target>/ by removing what was there first. A command that finished second removed the folder the first had just installed, while that command was copying out of it (ENOTEMPTY, EEXIST, or ENOENT on strands-box-contain-trampoline). An install now keeps a complete folder another process renamed in, and drops its own copy.
The Box tests deployed each Strands Box template but skipped agentcore dev, because box listens on the protocol's fixed ingress port and the tests run concurrently. A sequential group now creates each template, runs dev on that port, invokes it locally and checks that box started the agent. A box-egress test runs the minimal box template with an agent that fetches two hosts. Its local profile allows checkip.amazonaws.com with no credential, so that call returns 200 and box blocks example.com with a 403. It reaches public hosts, so it retries twice and has its own tag, which keeps it out of --tagsFilter='box'.
Create, build and deploy of a Box template run box, and box ships only arm64 builds, so on the x86_64 Linux CI runner every Box test failed at create. The Box groups now skip there and run wherever a box does.
leo237
left a comment
There was a problem hiding this comment.
Re-review: MQ-7 (e2e cases for the Strands Box templates), head 61458ca
This is MQ-7 of the squash plan. The plan doc still returns 401 for me, so I checked against the branches directly.
What changed since 8d5f619: four new commits on top. It is not a rebase: 8d5f619 is an ancestor of the head, and the merge-base is still f5503f02 (#2857).
dacbf9c5(wait before the Box error check, comment fixes): exists only onbox-integration, with no match onbox-integration-development.a3326d85(keep a box another process installed into the shared cache): dev port ofd4ef79a4. On this base it also coversinstallFfi, which dev'sboxRelease.tsdoesn't have.83ab7874(runagentcore devfor every Box template): dev port of44a151c4, with the same content.61458ca1(skip when noboxruns from PATH): dev port ofd5372686, with the same content, plus the CI skip this PR already had.
Scope: the PR no longer touches only e2eTest/. It now also changes src/core/project/backends/boxRelease.ts and boxRelease.test.ts. Against the merge-base, those are the only source files in the diff. None of the box-integration-only fixes are touched: boxProfileCredentials.ts (#2845), boxProfile.ts (#2848/#2849), the Windows-safe tests and Dockerfile changes (#2850), the xray grants and boxMemoryPolicy region (#2851), the #2853 items, and #2857's signing region, 68d2bbb4 and c1bc4576 are all intact. The PR still merges cleanly with the current box-integration.
Earlier comments (ours)
- Cherry-pick
8d5f6199to dev: still open, anddacbf9c5now needs the same. Dev'stemplates.test.tsstill has no CI skip on the Box describe (only!BOX_ON_PATH), still has thepy_strands_boxduplicates and the FALLBACK/FAILED check, and has no 30 s wait. - Single error-line query: addressed.
dacbf9c5waits 30 s before the query. That is a fixed sleep rather than a retry, but it is enough for the purpose. - Broad
ERRORterm: unchanged. Still non-blocking; I'm not repeating it. - README
--tagsFilter='box'skips the per-Python cases: addressed. The new paragraph says how they run. - Nit,
.filter((r) => !r.skip): addressed. The doc comment now says the describe'sskipIfis the only skip. - Nit, "the README": addressed. It now says "the root README.md".
tejaskash's review (CHANGES_REQUESTED on 8d5f619): none of its three inline threads has a reply, and the new commits don't change what they describe. The Box cases are still gated on box being on PATH rather than on a Box-capable CDK (src/assets/cdk/package.json still pins @aws/agentcore-cdk 1.0.2), and boxRelease.json still pins an ffi shim only for PYTHON_3_14. The new box-egress test partly covers the allowed/denied gap tejaskash raised, but only under dev, not on Runtime. I'm leaving those threads to tejaskash.
Blocking
verify / Linuxfails at this head, at "Smoke test npm installation with host TypeScript versions":error: Script not found "test:package". CI runs the base'sverify.yml, which now hasbun run test:package(from the #2858 v1x sync), but it checks outpull_request.head.sha. This branch is behindbox-integration, so itspackage.jsonhas notest:packagescript. That step stops the job before the Linux binary smoke test andbun test. The fix is to merge or rebasebox-integrationintobox/mq-7. #2858 changes none of this PR's files, and the merge is clean. No code change is needed. Windows and macOS verify passed, so the newboxReleaseunit test passes there.
Non-blocking
- The
box-egresstest can't pass on this base until #2859 lands (inline). It edits.box/agent/local/profile.toml, which is dev's per-agent layout. Onbox-integration,agentcore createwrites<project>/.box/profile.toml(boxProfilePath,manager.tsx), soreadFilethrows ENOENT. #2859 (open) moves the layout to.box/<agent>/{local,aws}, so this is fixed by a planned PR. Either land #2859 first, or say in the README that--tagsFilter='box-egress'needs it. CI skips the test either way, since there is noboxon the runners. - The per-Python box cases ignore
BOX_ON_PATH(inline). The Box describes skip withoutbox, and the README says the tests "run only when aboxruns from your PATH, and skip otherwise". Butpy_strands_box_3_10–3_13still skip only in CI, so a full run or--tagsFilter='runtime'on a machine withoutboxfails at create. UsingBOX_ON_PATHthere means moving its definition aboveRUNTIME_TEMPLATES. publishInstall's comment promises more than the code ensures (inline). There is a check-then-remove window: process A seesinstalled()false, B renames its complete payload intoroot, then A'srmSync(root)deletes B's install while B's command may be copying out of it. The end state is still a complete install, so this is rare and self-healing, but "onceinstalled()holds it is never removed" isn't strictly true. Either reword the comment, or rename first and only removeroot(then retry) when the rename fails andinstalled()is false. Also, on Windows a rename onto an existing folder fails withEPERM, notEEXIST/ENOTEMPTY. The code handles that, because it checksinstalled()rather than the code, but the comment doesn't mention it.- Test coverage of
publishInstall: the new unit test covers only the early return, where the other process finished before this one publishes. It doesn't cover thecatchpath (the rename fails and the winner's copy is kept), or theinstallFficaller, which is thebox-integration-only part of this port. - The box
devcases check less than the Runtime cases.invokeLocallycheckscomplete, and the new describe checksbox: running profile. Neither checks the 2xx status or the non-empty body that the Runtime path now requires, so an HTTP box agent that starts and then answers with an error passes underdev. - Nit, possible silent skip:
test.skipIf(!runtime)in the egress describe skips silently ifagent-python-minimal-boxis renamed or removed fromBOX_TEMPLATES. Anexpect(runtime).toBeDefined()(or a throw) would make that loud. - Nit, README: "box ships only arm64 builds, so they skip on the x86_64 Linux CI runner". The skip comes from there being no
boxon PATH, and the CI-only skip on the Runtime describe. Neither depends on the runner's architecture.
Windows safety: no new unsafe patterns. spawnSync("box", ["--version"]) uses no shell and inherits the environment, paths use join, and the new unit test uses no POSIX modes.
CI: verify / Linux fails (blocking item 1). verify / Windows, verify / macOS, CodeQL, Analyze and the agentcore-e2e CodeBuild check pass. The e2e job is skipped on PRs, so none of the new e2e cases have run in CI. Nothing is pending. The known flaky checks on the base are non-blocking: the Windows transactionSearch EFAULT test and the 5 s timeouts in the macOS screen tests. Neither showed up here.
Decision: COMMENT. Only the Linux verify failure blocks, and a merge of box-integration into the branch should clear it. Everything else is non-blocking.
Address the review of #2865. - The Box cases that deploy run only with AGENTCORE_E2E_BOX_CDK, an @aws/agentcore-cdk version or file: tarball that deploys Box projects; createTemplateProject installs it in each Box project's agentcore/cdk. The pinned release still rejects boundRuntime, so the CI-only skip hid that a laptop run fails at synth the same way. CI sets no opt-in. - The per-Python Box cases take the same gate, so they also skip without box on PATH. - The README says which steps need box: create for a CodeZip box template, build and deploy (box profile show) for every Box template. - The egress test throws at collection when its template case is gone, instead of skipping silently. - publishInstall renames first and removes the cache folder only when the rename fails and the folder is not a complete install, and its comment names the window that remains and EPERM on Windows. A test covers replacing an interrupted install.
|
Replies to the review bodies, at @leo237 (review on
@tejaskash (coverage gap): the Strands templates ( |
Addressed in 2e704fc: the Box deploy cases are gated on the AGENTCORE_E2E_BOX_CDK opt-in (a Box-capable @aws/agentcore-cdk build), the README says which steps need box, and the ffi shim point is fixed on box-integration-development by 264da50 and 7de308e. Replies are on each thread. Dismissing because Tejas is out today; happy to revisit when he's back.
leo237
left a comment
There was a problem hiding this comment.
Re-review: MQ-7 (e2e cases for the Strands Box templates), head 2e704fc
This is MQ-7 of the squash plan (phase 14). The plan doc loaded this time. It lists MQ-7 as a6d52f82, e48bf2db, 91fa62ba and 8d25cd6d on box-integration-development, plus review fixes. It has no MQ-8 row yet, so I checked #2867 against the branches directly.
What changed since 61458ca: two new commits. This is not a rebase.
216710cbmergesbox-integrationat1405932a, which brings #2858 and #2859. The merge was clean:git show --remerge-diffshows no conflict hunks. The PR's diff against the base is still onlye2eTest/README.md,e2eTest/constants.ts,e2eTest/project/templates.test.ts,boxRelease.tsandboxRelease.test.ts. So none of the fixes that exist only onbox-integration(#2845, #2848/#2849, #2850, #2851, #2853, #2857, #2858, #2859) are touched.2e704fcecontains the review fixes: theAGENTCORE_E2E_BOX_CDKopt-in, the per-Python gate, the README, the egress throw, and thepublishInstallrework with its test. It exists only onbox-integration. Dev'spublishInstall(d4ef79a4) still has the old remove-then-rename code and the old comment, and dev has noAGENTCORE_E2E_BOX_CDK. You said the plan's back-merge ofbox-integrationinto dev will bring this commit,8d5f6199anddacbf9c5. That works for me as long as the back-merge happens. Non-blocking.
CI: verify / Linux and verify / macOS pass, and so do CodeQL and CodeBuild e2e. e2e is skipped. verify / Windows is still pending (in progress since 03:58 UTC) after a 10-minute wait, so I am posting without it. The Windows transactionSearch EFAULT test and the 5 s macOS screen-test timeouts are known flaky on the base and would not block. The two new boxRelease cache tests are the Windows-relevant ones: a rename onto a non-empty folder gives EPERM there, which the new catch handles.
Earlier comments: status
- Blocking,
verify / Linuxtest:package: fixed by the merge. Linux passes. - The per-Python cases skip only in CI: fixed. They use
skip: !BOX_DEPLOYS, andBOX_ON_PATHnow sits aboveRUNTIME_TEMPLATES. box-egressneeds #2859's layout: fixed by the merge..box/<agent>/local/profile.tomlmatchesboxProfile.tson the base.publishInstallcomment, the race andEPERM: mostly fixed. It now tries the rename first, the comment namesEPERM, and a test covers thecatchand remove path for an interrupted install. One sentence is still off; see below.- The unit test covers only the early return: fixed for
installBox. You skipped aninstallFfitest because264da50aremoves the ffi installer. That is fine. - The box
devcases lack the 2xx and non-empty-body checks: I withdraw this one.invokeLocallyalready parsesLocalRuntimeInvokeResponseSchema, which requires both. My mistake. - Nit,
test.skipIf(!runtime): fixed. The describe now throws at collection. - Nit, the README's reason for the CI skip: fixed. It now says which steps need
boxand names the opt-in. - Cherry-pick
8d5f6199/dacbf9c5to dev: deferred to the back-merge, as above. - The broad
ERRORterm: unchanged. It is still non-blocking, and I won't repeat it. - tejaskash, the CDK pin: addressed by the
AGENTCORE_E2E_BOX_CDKopt-in.createTemplateProjectinstalls that build inagentcore/cdkand fails loudly whennpmfails. You report a run ofagent_python_minimal_boxwith a tarball packed from L3box-integration. - tejaskash, the ffi shim pinned only for
PYTHON_3_14: not fixed here. It is planned. The fix is264da50a(CodeZip without the ffi shim) and7de308e8(box preview.9), and I checked that both are onbox-integration-development. #2867 (MQ-8) ports them ase8069ddaand3934bbed. Until #2867 lands, the fourpy_strands_box_3_1xcases fail even with the opt-in, as you said. They are gated off in CI, so the gate does not hide a CI failure. Non-blocking because a planned PR fixes it. - tejaskash,
boxon PATH and--skip-install: your reply is right.buildanddeployrunbox profile showfor every Box runtime, and--skip-installwould skip the CDK app'snpm install, whichdeploynever runs. The README now says so. - tejaskash, a Runtime egress case: deferred to a follow-up. That is reasonable, since the Bedrock-calling templates' 2xx and body checks cover the allowed path.
New findings (all non-blocking)
- #2867 conflicts with this PR in
boxRelease.tsandboxRelease.test.ts. I tried merging #2867's head into this head, and both files conflict. #2867 removesinstallFfiand still uses the oldrmSyncthenrenameSyncininstallBox. Whichever PR lands second should keeppublishInstall(payload, root, () => installedBox(...))ininstallBox. It should dropwriteFileSync(onlyinstallFfiused it) but keepexistsSync, and drop theffioption from the two new cache tests. Otherwise #2867 silently brings back the race this PR fixes (dev has the same fix ind4ef79a4). - README (inline): it says the per-Python cases run with
--tagsFilter='runtime'"givenboxand the same opt-in". On this base they fail with the opt-in until #2867 lands. One clause saying so would save a confused laptop run. This is planned, so non-blocking. - The
publishInstalldoc comment (inline): it says the last sentence's removal "takes a complete install away, and the next command downloads again". After thermSync, this process renames its own complete payload intoroot, so the cache ends complete and the next command does not download. The real cost is that a command of the other process that is still reading fromrootcan lose files under it. - Nit: the reflowed doc comment on
RuntimeTemplateTestCase.boxbreaks after "runs box on the", leaving a short line.
The opt-in gate is solid. BOX_CDK is trimmed and treated as unset when empty, runCommand uses cross-spawn with the inherited env, and a non-zero npm exit throws with stdout and stderr. I found no Windows-unsafe patterns in the new code: no hard-coded / paths, no : splits, no POSIX modes, no sh, and no empty env.
Decision: APPROVE. Nothing left blocks. The open items are either planned (#2867, the dev back-merge) or non-blocking.
Address leo237's last comments on #2862: - boxStartFolder read the folder box starts in from the local profile's header comment, so a user who edited or removed it moved box back to the project root, with its project-wide grant, and the agent's ${{ workspace }}/.venv went missing. It now takes a CodeZip runtime's codeLocation from agentcore.json; the project root stays for a container and for a profile written before, whose paths name the code folder under the project root. - A comment in serve() still said box starts the agent in the project folder. - EnvLocalFile.upsert replaced only the first line of a key, so a later one, which the parser keeps, left the old key in force while build said it updated it. It replaces every line of the key, and keeps an export line an export line.
vvargu
left a comment
There was a problem hiding this comment.
I reviewed the diff and the code around it. I ran bun test src/core/project/backends/boxRelease.test.ts on 2e704fc: 12 pass, 0 fail. I did not run the e2e suite. I found no bug that blocks the merge, but please fix the two items below first.
1. The PR description does not agree with the branch.
- The body lists 4 commits. The branch has 10 commits, which include a merge and
a3326d8 fix(box): keep a box another process installed into the shared cache. - The body says "MQ-7 changes only
e2eTest/". But the PR also changessrc/core/project/backends/boxRelease.tsand its test. - The body says the log check looks for
FALLBACKandFAILED. The code looks forERROR,blocked by egress controlandAccessDenied. - The body says the tests do not depend on the
.box/<agent>/{local,aws}layout. But the egress test writes.box/agent/local/profile.toml. That layout is now onbox-integration, so the test is correct. Only the claim is old.
Please update the body. Also, add the cache fix to it, or move the fix to a separate PR.
2. Nobody has run the new e2e tests.
The body says "Not run: the e2e suite itself". So these tests have never passed. Please run --tagsFilter='box' and --tagsFilter='box-egress' one time with a Box-capable AGENTCORE_E2E_BOX_CDK. Then add the result to the PR before the merge.
The inline comments have the other items.
leo237
left a comment
There was a problem hiding this comment.
Plan PR: MQ-7 (phase 14) of the Box integration squash plan.
What changed since 2e704fce. There was no rebase. Two commits were added:
d6a007e7mergesbox-integrationat7252504c, which brings in #2862 (MQ-6). The merge-base is now thebox-integrationtip, so the branch is up to date. I compared the merge withgit merge-tree 2e704fce box-integration, and its tree matches the automatic merge exactly: there were no conflicts and no hand edits.git diff 2e704fce..HEADshows no change toe2eTest/,boxRelease.tsorboxRelease.test.ts.2e455e36fixes items from our last #2862 review. It exists only onbox-integration:box-integration-developmenthas noboxStartFolderat all, and the plan does not list this commit under MQ-7. The SYNC back-merge has to carry it, along with8d5f6199,dacbf9c5and2e704fce, as already agreed.
Do the e2e expectations still hold on top of #2862? Yes.
- The
box-egresstest writesapp/<runtime>/main.pyand appends[egress.checkip]to.box/<runtime>/local/profile.toml. It does not depend on the folder box starts in. - The dev cases only look for
box: running profile. - #2862 runs
start.pyon Python 3.10 withouttomli, which suits thepy_strands_box_3_10case. - The 3.10 to 3.13 cases still need #2867. See the earlier item below, which is still open.
2e455e36, checked against our #2862 findings.
- Finding 2 (
boxStartFolderdepended on the header comment): addressed. The start folder now comes fromruntime.codeLocationfor a CodeZip runtime and is the project root for a Container runtime. Both callers pass the runtime (box.ts, andreadResolvedPolicyinboxNetworkSource.ts).boxProfile.test.tscovers three cases: a profile with its comments removed, a Container runtime, and a profile written before.box.test.tsstill covers both cwd/profile-argument cases. One edge case remains in the profile-written-before check (inline). - Finding 4 (stale comment in
serve()): addressed. But the comment above the call inbox.tsnow has the same problem (inline). - Finding 5 (
upsertwith duplicate keys andexport): addressed, with tests for both. Every line for the key is replaced, soread()returns the new value, and anexportprefix is kept. - Findings 1 (no test with two boxed runtimes) and 3 (the
boxProfileCredentials.ts:55docstring) were not touched. They stay non-blocking.
Must-survive box-integration fixes. This PR touches source files beyond e2eTest/ and boxRelease.*, but only these: box.ts (one line), boxNetworkSource.ts (it passes the runtime through), boxProfile.ts (boxStartFolder and a comment in serve()), envLocal.ts (upsert) and their tests. No removed line touches boxProfileCredentials.ts, the #2848/#2849 profile code, the #2850 Windows-safe tests or Dockerfile flags, the #2851 grants or region, the #2853 items, #2857's signing, the c1bc4576 .venv, or anything else #2858, #2859 or #2862 added. Of #2862's changes, all of these are intact: the per-runtime dev secrets, the .venv/bin/python check, start.py reading policies as JSON on 3.10, OTEL off, the A2A runtime.url, and upsert (made stricter).
Review of the new code
- Tests: good. Each behaviour change has a test, and the new tests are Windows-safe: they use
tempRoot()paths and literal POSIX profile strings that are never used as file paths, they spawn nosh, they rely on no file modes, and they pass no empty env. - Silent failures: nothing blocking. The check for a profile written before is broad enough to send box back to the project root without any warning (inline).
encrypted[key] ?? ""would write an empty value ifencryptever dropped a key. That can't happen with the current cipher, so it's a nit. - Comments: one is stale (inline,
box.ts). - New types:
boxStartFoldernow takes{ readonly build: string; readonly codeLocation: string }. That's small and structural, andProjectRuntimesatisfies it. Typing it asPick<ProjectRuntime, "build" | "codeLocation">would make a new build type a type error here instead of a silent""(nit). - Nit:
codeLocationis any non-empty string. A hand-writtenapp\agentwould giveposix.relativea single segment, sobox runwould get../.box/.... The old header comment had the same limit, because the generator always writesapp/<name>/.
Earlier comments on this PR. None of them changed, because this push did not touch e2eTest/ or boxRelease.*. From our review on 2e704fce, all non-blocking and not repeated inline:
- The #2867 conflict in
boxRelease.tsandboxRelease.test.tsis still open. Whichever lands second must keeppublishInstallininstallBox. - The README's 3.10 to 3.13 wording is still open. Those cases are fixed by #2867 (MQ-8, planned:
264da50aand7de308e8, both verified on dev). - The
publishInstallcomment wording is still open. - The short-line nit on
RuntimeTemplateTestCase.boxis still open.
vvargu's four comments on 2e704fce (the fixed 30 s wait, the publishInstall rename-failure test, the shared deploy helper, and the projects left behind on retry) have no reply yet.
CI (at 2e455e36): verify / Linux, verify / Windows and verify / macOS pass, as do Analyze, CodeQL and CodeBuild e2e. The GitHub e2e job is skipped. Neither known flake (the Windows transactionSearch EFAULT test or the macOS 5 s screen-test timeouts) showed up; they would be non-blocking anyway.
Decision: APPROVE. The merge of #2862 is clean, and the e2e expectations still match it. 2e455e36 fixes three of our #2862 findings, with tests. No must-survive fix is lost. The new findings below and the items above are non-blocking.
New findings (all non-blocking)
boxStartFolder's check for a profile written before matches any${{ workspace }}/app/(inline).- A stale comment in
box.ts(inline). - Nits:
Pick<ProjectRuntime, ...>for the parameter;encrypted[key] ?? "";codeLocationwith backslashes. 2e455e36exists only onbox-integration. Please list it for the SYNC back-merge, and in the PR description as a #2862 follow-up, because it is outside MQ-7's planned scope.
boxStartFolder kept the project root for a profile whose text named
${{ workspace }}/app/, as a profile from before box started in the code
folder does. A current profile that grants ${{ workspace }}/app/<x>
matched it too, and box quietly started at the project root again, with
its project-wide grant. Box projects are new, so there is no older
profile to run: a CodeZip runtime's box starts in its codeLocation, and
a container's in the project root. The dev runner's comment says so.
On Python 3.10, with no tomllib, start.py reads the staged profile's policies with a regex, and a line in any other form gave no policy files and no message, so box failed later with an unclear error. start.py now stops with an error that names the line. A test runs staged_policies with no tomllib on the profiles deploy renders.
…ild reads EnvLocalFile.upsert snapshotted the file before its in-place write, then insertIfNew snapshotted the half-updated file, and the first one came back only if insertIfNew succeeded. It now keeps the earliest snapshot, as removeKeys does, so rollback restores the original file. The local testing guide said a build with GITHUB_TOKEN unset keeps the stored key. Build reads GITHUB_TOKEN from .env.local too, so a line there replaces the stored key, even with an older one.
The Box deploy cases waited a fixed 30 s after 'running profile' and then checked that no error line was logged, so a line CloudWatch delivered later passed unseen. They now wait for the agent server's access line for the invoke's POST first. Also: a README note that the Python 3.10-3.13 box cases pass only with #2867, a test of publishInstall's lost rename race, and two comment fixes.
Box is only for new projects, so the guide gives no cleanup hints for .env.local lines that earlier box-integration builds wrote.
|
@vvargu The PR body is updated, and the 7 Box deploy e2e cases passed (us-west-2, 7/7, results in the body). The dev groups and |
leo237
left a comment
There was a problem hiding this comment.
Re-review of 2e455e36..f7391f24. The five new commits (21e2c91, acb9b4c, ad4617d, 5a6a441, f7391f2) sit on top of the head I approved. There is no rebase and no new box-integration merge, and the head already contains box-integration (7252504, #2862).
Blocking: verify / Windows fails because of the new boxStartPy.test.ts. This is not the known transactionSearch EFAULT flake. Bun.spawnSync throws Executable not found in $PATH: "python3" (ENOENT) when the command does not exist. It does not return a non-zero exit code. The runner has no python3, so the .find throws while the file loads ("Unhandled error between tests"). It never tries python, and the skipIf never runs. Details are inline. Linux and macOS pass, CodeQL and Analyze pass, and e2e is skipped.
Earlier comments now addressed
boxProfile.tssilent fallback (mine): fixed in 21e2c91.boxStartFolderno longer takes the profile text, so no profile content can move box back to the project root. The boxProfile/box tests cover CodeZip,./, a trailing/, Container, and a profile with its comments removed.box.tsstale comment (mine): fixed.- README: the per-Python cases now say they need #2867. Fixed.
publishInstalldoc comment accuracy: fixed. It now says the removal takes the other install away while its command may be reading it, and that this process's payload replaces it.- The short line in the
boxdoc comment: fixed. - vvargu's fixed 30 s wait before the error check: fixed. The check now retries until the invoke's own
"POST /"access line shows up, then queries for errors. - vvargu's comment that the race test does not reach the rename-failure branch: fixed. The new
publishInstalltests run the lost-rename branch (checksis 2), the branch that replaces an incomplete folder, and the plain rename. - vvargu's helper-extraction nit and the nit about cleaning up the
retry: 2projects: the author declined both with reasons. Both stay non-blocking. - The PR description now names 2e455e3 and 21e2c91 as follow-ups to #2862.
Other new changes look right
start.py(acb9b4c) now stops with a clear error on apoliciesline the 3.10 regex can't read, instead of starting box with no policy files. The tests runstaged_policieswithouttomllibon profiles fromrenderRuntimeBoxProfile. Only the Windows load problem above is wrong.EnvLocalFile.upsert(ad4617d) keeps the earliest snapshot, and thefinallyputs it back wheninsertIfNewfails. The new test covers that case.- The docs are accurate:
GITHUB_TOKENcan also come from.env.local, and the troubleshooting item for earlier branch builds is gone.
Must-survive fixes: this PR's diff against box-integration touches only the box e2e, boxRelease, boxProfile/boxNetworkSource/box.ts, envLocal, start.py and the docs above. The #2845-#2862 markers I grepped for are all still there, and none were lost.
Non-blocking
upsert's comment says it keeps the earliest snapshot "as removeKeys does". Whenupdatedis empty, though,insertIfNewstill overwrites any earlier snapshot (this.snapshot = existing). That behavior is older than this PR. You could makeinsertIfNewkeep the earliest snapshot too, so all three methods agree.- My earlier nits are still open: the
Pick<ProjectRuntime, "build" | "codeLocation">parameter type,encrypted[key] ?? "", and backslashes incodeLocation. - SYNC back-merge: 21e2c91, acb9b4c, ad4617d, 5a6a441 and f7391f2 exist only on
box-integration.box-integration-developmenthas noboxStartFolderand no newstart.pyerror. They join 8d5f619, dacbf9c, 2e704fc and 2e455e3 on the back-merge list. - Older items still apply: the #2867 conflict in
boxRelease.ts/boxRelease.test.ts(whichever PR merges second keepspublishInstallininstallBoxand drops the ffi half; the newpublishInstalltests are part of the half to keep). The broadERRORterm also still applies. - The author did not run the
agentcore devgroups orbox-egress(they say these need a normal terminal). The 7 Box deploy cases passed.
Bun.spawnSync throws on Windows when the executable is not on PATH, so the python3 probe failed the file before any test ran.
leo237
left a comment
There was a problem hiding this comment.
Re-review of f7391f24..96c63bd9. There is one new commit, 96c63bd ("test(box): find Python on Windows without throwing"), on top of the head I last reviewed. There is no rebase and no new box-integration merge. The head still contains box-integration (7252504, #2862), so test:package is present.
Earlier comments now addressed
- Blocking,
boxStartPy.test.tscrashingverify / Windows(mine): fixed in 96c63bd. The probe now wrapsBun.spawnSyncintry/catchand returnsfalsewhen it throws. A missingpython3no longer throws while the module loads, so.findgoes on topython, and thedescribe.skipIf(PYTHON === undefined)works as the doc comment says. I suggestedBun.which, andtry/catchis just as good. It also handles a name that resolves to something that can't be spawned. The Windows Storepython3alias stub exits non-zero, so it is also rejected andpythongets tried. I checked the rest of the file for Windows problems: it usesjoin/tmpdirpaths, no POSIX modes, nosh, and no empty env.start.pyis pure ASCII, andstaged_policiesopens the profile in"rb", so the harness'sopen(...).read()doesn't depend on the Windows locale encoding. - Everything else I listed as addressed in my last review is still addressed. Nothing new came in from the author or vvargu after that review. The author's last replies (05:24-05:25) came before it.
New finding (nit, non-blocking)
boxStartPy.test.ts:22: the comment says "On Windows, spawnSync throws, rather than failing, for a name not on PATH." Bun throwsExecutable not found in $PATHfor a missing command on Linux and macOS as well. Windows is just the runner wherepython3is missing. Something like "spawnSync throws, rather than failing, for a name not on PATH (the Windows runner has nopython3)" would be accurate.
CI: everything passes on 96c63bd9: verify / Windows, verify / Linux, verify / macOS, Analyze, CodeQL and the CodeBuild e2e project. The GitHub e2e job is skipped. On the Windows runner, all three start.py with no tomllib tests ran and passed (they were not skipped), so the probe found python. This run hit none of the known base flakes (the Windows transactionSearch EFAULT test, the macOS 5 s screen-test timeouts). Those would be non-blocking anyway.
Must-survive fixes: 96c63bd touches only boxStartPy.test.ts. The PR's diff against box-integration is still the same 16 files (box e2e, boxRelease, boxProfile/boxNetworkSource/box.ts, envLocal, start.py, docs). It loses none of the #2845-#2862 fixes.
Coordination with #2867 (still open, at f40ebce): I trial-merged #2867 into this head. boxNetworkSource.ts merges cleanly. This PR changes readResolvedPolicy's parameter to runtime: ProjectRuntime and calls boxStartFolder(runtime). #2867 only rewords the @box/bedrock comment inside that function, moves BOX_BEDROCK_PACK out, and drops "ffi" from BOX_DIST_NAMES. Neither PR has refuseOldShippedNames left in this file. The only conflicts are the known ones in boxRelease.ts and boxRelease.test.ts. Whichever PR merges second keeps publishInstall (and its tests) in installBox and drops the ffi half.
Still open, non-blocking (unchanged from my last review)
upsert's comment says it keeps the earliest snapshot "as removeKeys does", butinsertIfNewstill overwrites an earlier snapshot whenupdatedis empty. That behavior is older than this PR.- Earlier nits: the
Pick<ProjectRuntime, "build" | "codeLocation">parameter type,encrypted[key] ?? "", and backslashes incodeLocation. - The broad
ERRORterm in the e2e error check. - The
agentcore devgroups andbox-egresshave not been run yet (the author says they need a normal terminal). The 7 Box deploy cases passed. - SYNC back-merge: 96c63bd exists only on
box-integration(box-integration-developmenthas noboxStartPy.test.tsat all). Add it to 8d5f619, dacbf9c, 2e704fc, 2e455e3, 21e2c91, acb9b4c, ad4617d, 5a6a441 and f7391f2 on the back-merge list (the SYNC row in the squash plan). - FYI: #2854 (containment, base
box-integration) is also open.
Approving. The one blocking item is fixed and CI is green. Everything left is a nit, already-known non-blocking, or SYNC back-merge bookkeeping.
…gent-v5 [aws#2865](aws#2865) adds e2e tests for each Strands Box template, reads the start folder of box from agentcore.json, and puts a box download in the cache with publishInstall. The resolution of each conflict: - templates/boxProfile.ts: the `boxStartFolder(runtime)` of box-integration, and `CODEZIP_START_FOLDER` and `RUNTIME_URL` of this branch. `START_FOLDER_LINE` goes. - boxNetworkSource.ts: the profile source gets the start folder that readResolvedPolicy reads from the runtime. - backends/boxRelease.ts and its test: publishInstall and its tests of box-integration, with no `installFfi`, because this branch removes the ffi shim. The two race tests give no ffi release. - dev/box.test.ts: the tests of box-integration, and the test of this branch "runs each runtime with the box profile of that runtime" with the start folder of each runtime. - backends/boxStartPy.test.ts, new in box-integration, wrote its Runtime profiles with `renderRuntimeBoxProfile`, which this branch removes. It writes them with `serializeRuntimeProfile`, as the runtime profile writer does. The e2e tests examine what an invoke and the logs give, not the text of the Runtime profile, so no change to them is necessary here.
… starts box in [aws#2865](aws#2865) makes `boxStartFolder` read `build` and `codeLocation` from agentcore.json, not the header line of the profile. Thus dev starts box in the code folder of each CodeZip runtime, also for a profile that names the code folder `${{ workspace }}/<codeLocation>`. The path model of this branch had a condition for such a profile, of the earlier layout. A Box project is only for new projects, so Varun on [aws#2866](aws#2866) told us to remove that condition. - `runtimePaths(runtime)` names the local code folder `${{ workspace }}` for each CodeZip runtime, and has no start folder parameter. - The profile source of the Box network model and the context of the derivation have no `startFolder`. - The rule tests use the profiles of the code-folder layout. The test of a path of the project that is not in the code folder goes, because `${{ workspace }}` is the code folder. Tests: derive.test.ts "moves a CodeZip runtime's code folder, ${{ workspace }}, and Python to the inputs of @box/agentcore-codezip", and the tests of the four rules, on the code-folder layout.
tejaskash
left a comment
There was a problem hiding this comment.
Thanks, the explicit CDK opt-in and the recorded 7 of 7 Box deploy cases look good, and #2867 has since pinned the shims for the per-Python cases. Two follow-ups, since they aren't covered yet:
- A deployed Box runtime case with one allowed host and one denied host. The egress test added here runs only under
agentcore dev. - One recorded run of the 12 Box cases that haven't passed anywhere yet (the 4 per-Python cases, the 7 dev cases and the egress test), on
box-integrationafter #2867, withboxon PATH.
|
|
||
| That runs each Box template on its default Python. The CodeZip box template on Python 3.10 to 3.13 | ||
| (`py_strands_box_3_10` to `py_strands_box_3_13`) is in the runtime templates describe, so it runs | ||
| with `--tagsFilter='runtime'` or a full run, given `box` and the same opt-in. Those four cases |
There was a problem hiding this comment.
Now that #2867 has merged, the sentence saying the four per-Python cases pass only once it lands can go.
| for (const entry of spec.runtimes) entry.runtimeVersion = runtime.runtimeVersion; | ||
| await writeFile(specPath, `${JSON.stringify(spec, null, 2)}\n`); | ||
| } | ||
| if (runtime.box && BOX_CDK) { |
There was a problem hiding this comment.
Small one: the dev cases and the egress test go through this too, so they install the opt-in CDK build they never use. That's 8 extra npm install runs inside the create time budget. Could this only install for cases that deploy?
Description
MQ-7 of the Box integration squash plan: e2e cases for the Strands Box templates. It also carries one cache fix and the follow-ups to #2862's last review.
box-integrationis merged in (d6a007e), with #2859 and #2862.e2e cases (
e2eTest/), ported frombox-integration-development:running profile.agentcore devfor every Box template, one at a time, and abox-egresstest: box allows the profile's host and blocks any other.boxruns from PATH, and the deploy cases need a Box-capable CDK (AGENTCORE_E2E_BOX_CDK).running profile, wait for the invoke's own access line, then query for the error lines the root README.md names:ERROR,blocked by egress controlandAccessDenied. The Box invoke needs a 2xx and a body. The per-Python cases also check that box ran. Duplicate deploys are dropped.The cases write the agent's profile under
.box/<agent>/local/, #2859's layout, now onbox-integration.Cache fix (
boxRelease.ts), a3326d8 fromd4ef79a4: concurrent build and deploy commands share~/.agentcore/box. One that finished second removed the folder the first was still copying out of. An install now keeps a complete folder another process renamed in. This suite runs deploys concurrently, which is why it is here. 5a6a441 adds a test of the lost rename race.Follow-ups from #2862's review (merged before they landed):
codeLocationfrom agentcore.json, not in a folder the profile's header comment names, and with no fallback to the project root. So an edited comment, or a grant under${{ workspace }}/app/, can't bring back the project-wide grant.start.pystops with a clear error on apoliciesline it cannot read, instead of starting box with no policy files. A test runs it with notomllibon the profiles deploy renders.EnvLocalFile.upsertreplaces every line of a key and keepsexport. It keeps its first snapshot, so a rollback restores the original file. The local testing guide says build also readsGITHUB_TOKENfrom.env.local.Left out:
868325c0, already onbox-integrationasb69851ca.Related Issue
Box integration squash plan, MQ-7.
Testing
bun testwith an emptyHOME, as CI runs it: 5528 pass, 1 fail (the knownthe box on PATHtest, which CI skips).bun run typecheck,bun run lint:check, Prettier,bun run secrets:check: pass.-t "create, deploy, invoke for Strands Box templates"), at f7391f2 in us-west-2 with box 0.1.11-linux-preview.6 and the L3box-integrationbuild asAGENTCORE_E2E_BOX_CDK: 7 of 7 passed (agent_python_minimal_box, agent_python_strands_box, agent_python_strands_box_container, agent_python_langchain_box, mcp_python_fastmcp_box, a2a_python_strands_box, agui_python_strands_box), 899 s. Each created, deployed, invoked, foundrunning profileand then the invoke's access line in the Runtime's logs, with no error line, and tore its stack down.agentcore devgroups andbox-egress. CodeZip dev runs box's macOS sandbox, which can't run inside the agent's sandbox. They need a run in a normal terminal:--tagsFilter='box' -t "create and dev for Strands Box templates"and--tagsFilter='box-egress'. The Python 3.10 to 3.13 cases pass only with fix(box): Ship only the pinned box, and drop the old box/<name> pack-name stop #2867.