Skip to content

test(e2e): Deploy and invoke every Strands Box template - #2865

Merged
vvargu merged 19 commits into
box-integrationfrom
box/mq-7
Nov 20, 2026
Merged

vvargu merged 19 commits into
box-integrationfrom
box/mq-7

Conversation

@mquang-amazon

@mquang-amazon mquang-amazon commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

MQ-7 of the Box integration squash plan: e2e cases for the Strands Box templates. It also carries one cache fix and the follow-ups to #2862's last review. box-integration is merged in (d6a007e), with #2859 and #2862.

e2e cases (e2eTest/), ported from box-integration-development:

  • ad7dda3, 6ca1d00, 204da9e, 7d45b88: create, deploy and invoke every Box template, and the CodeZip box template on each Runtime Python. Each case checks the Runtime's logs for running profile.
  • 83ab787: agentcore dev for every Box template, one at a time, and a box-egress test: box allows the profile's host and blocks any other.
  • 61458ca, 2e704fc: the Box cases skip where no box runs from PATH, and the deploy cases need a Box-capable CDK (AGENTCORE_E2E_BOX_CDK).
  • 8d5f619, dacbf9c, 5a6a441 (review): skip in CI. After running profile, wait for the invoke's own access line, then query for the error lines the root README.md names: ERROR, blocked by egress control and AccessDenied. The Box invoke needs a 2xx and a body. The per-Python cases also check that box ran. Duplicate deploys are dropped.

The cases write the agent's profile under .box/<agent>/local/, #2859's layout, now on box-integration.

Cache fix (boxRelease.ts), a3326d8 from d4ef79a4: concurrent build and deploy commands share ~/.agentcore/box. One that finished second removed the folder the first was still copying out of. An install now keeps a complete folder another process renamed in. This suite runs deploys concurrently, which is why it is here. 5a6a441 adds a test of the lost rename race.

Follow-ups from #2862's review (merged before they landed):

  • 2e455e3, 21e2c91: box starts in a CodeZip runtime's codeLocation from agentcore.json, not in a folder the profile's header comment names, and with no fallback to the project root. So an edited comment, or a grant under ${{ workspace }}/app/, can't bring back the project-wide grant.
  • acb9b4c: on Python 3.10, start.py stops with a clear error on a policies line it cannot read, instead of starting box with no policy files. A test runs it with no tomllib on the profiles deploy renders.
  • 2e455e3, ad4617d: EnvLocalFile.upsert replaces every line of a key and keeps export. It keeps its first snapshot, so a rollback restores the original file. The local testing guide says build also reads GITHUB_TOKEN from .env.local.

Left out: 868325c0, already on box-integration as b69851ca.

Related Issue

Box integration squash plan, MQ-7.

Testing

  • bun test with an empty HOME, as CI runs it: 5528 pass, 1 fail (the known the box on PATH test, which CI skips).
  • bun run typecheck, bun run lint:check, Prettier, bun run secrets:check: pass.
  • e2e, Box deploy cases (-t "create, deploy, invoke for Strands Box templates"), at f7391f2 in us-west-2 with box 0.1.11-linux-preview.6 and the L3 box-integration build as AGENTCORE_E2E_BOX_CDK: 7 of 7 passed (agent_python_minimal_box, agent_python_strands_box, agent_python_strands_box_container, agent_python_langchain_box, mcp_python_fastmcp_box, a2a_python_strands_box, agui_python_strands_box), 899 s. Each created, deployed, invoked, found running profile and then the invoke's access line in the Runtime's logs, with no error line, and tore its stack down.
  • Not run here: the agentcore dev groups and box-egress. CodeZip dev runs box's macOS sandbox, which can't run inside the agent's sandbox. They need a run in a normal terminal: --tagsFilter='box' -t "create and dev for Strands Box templates" and --tagsFilter='box-egress'. The Python 3.10 to 3.13 cases pass only with fix(box): Ship only the pinned box, and drop the old box/<name> pack-name stop #2867.

Adds a box tag and a test for each box template: create, deploy, invoke, then check the
Runtime's logs show box running the agent, and tear the stack down. They skip agentcore dev,
which runs box on a fixed ingress port.
Add agent-python-strands-box and agent-python-strands-box-container to the
runtime template cases. CI deploys with the released @aws/agentcore-cdk,
which has no Box support yet, and has no box on PATH for CodeZip dev, so
the cases skip in CI. A CodeZip box agent's dev serves on the profile's
ingress port, so that case pins dev to 8080.
The box cases test create, deploy and invoke on AgentCore Runtime, not
agentcore dev, so they no longer start dev or invoke locally.
Add a py_strands_box case for PYTHON_3_10 through PYTHON_3_13, beside
the template's default PYTHON_3_14, by setting runtimeVersion in
agentcore.json after create. The remote invoke now also needs a 2xx
status and a body, as the local invoke does, so an agent that failed to
start in its box fails the case.
@mquang-amazon
mquang-amazon requested a review from a team November 19, 2026 23:44
@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.75%. Comparing base (7252504) to head (96c63bd).

Files with missing lines Patch % Lines
src/core/project/backends/boxRelease.ts 94.11% 1 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                 @@
##           box-integration    #2865      +/-   ##
===================================================
- Coverage            96.75%   96.75%   -0.01%     
===================================================
  Files                  757      757              
  Lines                65642    65659      +17     
===================================================
+ Hits                 63512    63528      +16     
- Misses                2130     2131       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: MQ-7 (e2e cases for the Strands Box templates)

This is MQ-7 of the squash plan. The plan doc returned 401 for me, so I checked the commits against the branches directly.

Commit list check

  • The four commits have the same patches as a6d52f82, e48bf2db, 91fa62ba and 8d25cd6d on box-integration-development. e2eTest/ at this head is byte-identical to e2eTest/ on dev, so the dev-only 1b6ccdb5 revert and c17ecfc8 reapply cancel out and nothing is lost.
  • Leaving out 868325c0 is correct. It is on box-integration as b69851ca.
  • There are no extra commits. The diff touches only e2eTest/README.md, e2eTest/constants.ts and e2eTest/project/templates.test.ts. None of the protected box-integration-only fixes (#2845, #2848/#2849, #2850, #2851, #2852, #2853, #2857) are touched, and nothing is applied twice.
  • The open PRs #2859, #2860, #2862, #2863 and #2864 change no e2eTest/ files, so there is no conflict with them.
  • The base is box-integration, and the PR is mergeable.

CI: all checks pass (verify Linux, Windows and macOS, CodeQL, Analyze). The e2e job is skipped on PRs, so these cases have not run in CI. As the description says, the e2e suite itself was not run.

Blocking

  1. The BOX_TEMPLATES describe is not skipped in CI, so it will fail whenever the full e2e suite runs there. e48bf2db says the Box cases must skip in CI because CI deploys with the released @aws/agentcore-cdk, which has no Box support, and has no box on PATH. create fails without box. The skip: process.env.CI === "true" guard went onto the new RUNTIME_TEMPLATES entries but not onto the seven BOX_TEMPLATES cases; only the container case has a skip, and that one is the Linux-only skip. The block is also tagged runtime, so plain bun run test:e2e (the push CI step in ci.yml) and --tagsFilter=runtime both pick it up. e2e-test.yml installs no box. Today the e2e job on box-integration pushes is skipped by the authorization gate, so CI is green, but the first authorized push or dispatch will go red on all three OSes. Dev has the same code, so no later plan commit fixes this. A fix: add skip: process.env.CI === "true" to each BOX_TEMPLATES case, or skip the whole describe in CI (see the inline comment).

Non-blocking

  1. The FALLBACK/FAILED check can only see lines that contain running profile. --query becomes a CloudWatch filter pattern (handlers/observability/logs.ts → logEventFilterPattern), so the logs returned are only the events that match "running profile". not.toMatch(/FALLBACK|FAILED/) catches a failure only if box prints it on that same line. Either run a second query for those markers, or note in a comment that box prints them on the running profile line.
  2. The Box describe's remote invoke checks less than the runtime describe now does. 8d25cd6d added the 2xx status and non-empty body checks to the runtime block only. The Box block checks just complete, so an HTTP agent that box started but that returns 500 passes, because running profile is still logged.
  3. The four per-Python cases (py_strands_box_3_10–3_13) do not check that box ran. They are in RUNTIME_TEMPLATES, so they get the new 2xx/body check but not assertRunsInBox. Because start.py execs box with no fallback, a 2xx answer does mean the agent started. Still, calling assertRunsInBox for runtime.box/Box cases would match what the PR says it does.
  4. The same templates deploy twice. py_strands_box (runtime block) and agent_python_strands_box (Box block) cover the same template, and py_strands_box_container is in both lists under the same name. A full run outside CI deploys 12 Box stacks. Consider keeping the per-Python matrix and dropping the duplicate default and container entries from RUNTIME_TEMPLATES, or giving the duplicates different names.
  5. New type field box?: boolean is never read. .map(... box: true) sets it, but nothing reads it. Its doc comment describes agentcore dev behaviour that no code branches on. Either use it (for example to call assertRunsInBox, as in item 4) or drop it.
  6. The ALL_TEMPLATES comment is wrong. It says the list gives each project "its own name and local dev port". The dev port still comes from RUNTIME_TEMPLATES.indexOf, and Box cases never start dev; ALL_TEMPLATES sets only the project name.
  7. The runtimeVersion override is a raw edit of agentcore.json. That is fine for a test. The { runtimes: { runtimeVersion?: string }[] } cast passes the other keys through unchanged, and path.join keeps it Windows-safe.

Windows safety: the paths use path.join, and there are no POSIX modes, sh spawns, empty-env spawns or : splits.

Planned open items: this PR is test-only and fixes none of them. The "no local-profile test for the A2A, AG-UI and MCP box templates" item is still open: this PR adds deploy and invoke coverage for those templates on the Runtime, not local-profile tests.

I am using COMMENT rather than APPROVE only because of item 1. With the CI skip added, this is ready to approve.

Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
- Skip the Box template describe in CI, which deploys with the released
  CDK and has no box on PATH.
- Look for the README's error lines with their own logs query, as
  --query returns only the events it matches.
- Require a 2xx status and a body from a Box agent's remote invoke.
- Check that box ran for the per-Python cases too, through the box field.
- Drop py_strands_box and py_strands_box_container, which deployed the
  same templates as the Box describe.
- Say that ALL_TEMPLATES sets only the project name.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review: MQ-7 (e2e cases for the Strands Box templates), head 8d5f619

This is MQ-7 of the squash plan. The plan doc still returns 401 for me, so I checked against the branches directly.

What changed since 7d45b881: one new commit, 8d5f6199 ("Address the review of #2865"), on top of the earlier four. It is not a rebase, and the base has not moved: the merge-base is still f5503f02 (#2857). The PR still touches only e2eTest/README.md, e2eTest/constants.ts and e2eTest/project/templates.test.ts, so none of the box-integration-only fixes (#2845, #2848/#2849, #2850, #2851, #2853, #2857, 68d2bbb4, c1bc4576) are affected.

Dev parity: 8d5f6199 exists only on box-integration. It has no match on box-integration-development, where the newest e2eTest/ commit is still 8d25cd6d, and dev's templates.test.ts differs from this head by exactly this commit. See item 1 below.

Earlier comments

All of them are addressed:

  1. Blocking, CI skip: fixed. The Box describe is now describe.skipIf(process.env.CI === "true"), and the per-Python cases keep skip: process.env.CI === "true". GitHub Actions sets CI=true, so neither the push job's plain bun run test:e2e nor --tagsFilter=runtime picks them up.
  2. FALLBACK/FAILED filter: fixed. A second logs query, ?ERROR ?"blocked by egress control" ?AccessDenied, looks for the error lines. That is valid OR syntax in a CloudWatch filter pattern, and buildFilterPattern passes it through unchanged. logs writes only the events to stdout; the announcement goes to stderr. That means the not.toMatch can't hit header text, and exitCode is checked. The search terms match what the root README.md says a working box agent never logs.
  3. Box invoke 2xx and body checks: fixed.
  4. Per-Python cases now check box ran: fixed, through box: true and if (runtime.box) await assertRunsInBox(...).
  5. Duplicate deploys: fixed. py_strands_box and the runtime-block py_strands_box_container are gone, and the Box container case is renamed agent_python_strands_box_container. A correction to my earlier review: before this change a full non-CI run deployed 13 Box stacks, not 12 (7 in the Box block, plus py_strands_box, the four per-Python cases and the runtime-block container case). It now deploys 11, which matches the author's vitest list count. The names are unique, so the ALL_TEMPLATES.indexOf project names stay distinct.
  6. box?: boolean unused: fixed. It is now read, and its doc comment says what it does.
  7. ALL_TEMPLATES comment: fixed.

CI: all checks pass at 8d5f619: verify on Linux, Windows and macOS, CodeQL, Analyze, and the agentcore-e2e CodeBuild check. The e2e job is skipped on PRs, so the e2e cases still have not run in CI. Known flaky checks on the base are non-blocking: the Windows transactionSearch EFAULT test and the 5 s timeouts in the macOS screen tests (the macOS queue is slow).

New findings (all non-blocking)

  1. Port 8d5f6199 to box-integration-development. It exists only on box-integration. If a later phased PR squashes e2eTest/project/templates.test.ts from dev, it brings back the missing Box-describe CI skip (the earlier blocking item) and the duplicate deploys. A cherry-pick to dev keeps the two branches in step.
  2. The error query runs once, with no wait. It runs as soon as the running profile line shows up (see the inline comment). Box logs running profile when it starts, before it handles the request, so an error logged during the invoke can reach CloudWatch after the check has run. The 2xx and body checks catch most of these failures. Moving the error query into the retry, or doing one short re-check, would close the gap.
  3. ERROR is a broad term. Any ERROR-level line in the runtime's log group fails the case, whether it comes from the agent or from a library it uses (for example an OTEL exporter warning logged at ERROR). That matches the README's promise, so I'm only flagging it as a likely source of flakes. If it does flake, narrow it to box's own error prefix.
  4. --tagsFilter='box' does not run the four per-Python box cases. e2eTest/README.md gives this command for the box tests. The four cases are in the runtime/canary describe and lack the box tag, so they run only with --tagsFilter=runtime or a full run. Either say this in the README or move or tag them.
  5. Nit: BOX_TEMPLATES no longer has .filter((r) => !r.skip). No entry sets skip today, but the shared type still allows it, and a skip added later to a Box entry would do nothing. Either keep the filter or leave a comment.
  6. Nit: the new comment says "the README". The list it refers to is in the root README.md, not e2eTest/README.md. Naming the file would help.

Windows safety: the new code adds no hard-coded / paths, : splits, POSIX modes, sh spawns or empty-env spawns. The logs arguments go to cli.run as an argv array, so the quotes in the filter pattern don't depend on a shell.

Planned open items: this PR is test-only and fixes none of them. The "no local-profile test for the A2A, AG-UI and MCP box templates" item is still open.

Decision: APPROVE. The one blocking item is fixed, and the items above are non-blocking.

Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/README.md Resolved
- Wait 30 s after 'running profile' before looking for error lines, so
  lines from the invoke have reached CloudWatch.
- Say in e2eTest/README.md that --tagsFilter='box' leaves out the
  per-Python box cases, and that the Box describe's skipIf is the only
  skip for BOX_TEMPLATES.
- Name the root README.md as the source of the error lines.

@tejaskash tejaskash left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. Thanks for working through leo237's points so quickly: the CI skip, the second log query for ERROR, blocked by egress control and AccessDenied, the 2xx and body check, and the dropped duplicates all look good. What's left is that, as far as I can tell, these cases can't pass yet on any machine, and nobody has run the suite at this head (the PR's e2e check is skipped), so the opt-in and the per-Python cases need another look (inline).

One coverage gap worth noting: the only Box-specific assertions are the running profile line and no error lines. A profile whose egress rules block Bedrock would still pass for the minimal and FastMCP templates, which make no outbound call. One case that calls out through the Box gateway and checks the response would cover the allowed path, and a denied host would cover the other side.

Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/README.md Resolved
Concurrent build and deploy commands each download box into their own work folder, then
moved it into ~/.agentcore/box/<version>-<target>/ by removing what was there first. A command
that finished second removed the folder the first had just installed, while that command was
copying out of it (ENOTEMPTY, EEXIST, or ENOENT on strands-box-contain-trampoline). An install
now keeps a complete folder another process renamed in, and drops its own copy.
The Box tests deployed each Strands Box template but skipped agentcore dev,
because box listens on the protocol's fixed ingress port and the tests run
concurrently. A sequential group now creates each template, runs dev on
that port, invokes it locally and checks that box started the agent.

A box-egress test runs the minimal box template with an agent that fetches
two hosts. Its local profile allows checkip.amazonaws.com with no
credential, so that call returns 200 and box blocks example.com with a 403.
It reaches public hosts, so it retries twice and has its own tag, which
keeps it out of --tagsFilter='box'.
Create, build and deploy of a Box template run box, and box ships only
arm64 builds, so on the x86_64 Linux CI runner every Box test failed at
create. The Box groups now skip there and run wherever a box does.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review: MQ-7 (e2e cases for the Strands Box templates), head 61458ca

This is MQ-7 of the squash plan. The plan doc still returns 401 for me, so I checked against the branches directly.

What changed since 8d5f619: four new commits on top. It is not a rebase: 8d5f619 is an ancestor of the head, and the merge-base is still f5503f02 (#2857).

  • dacbf9c5 (wait before the Box error check, comment fixes): exists only on box-integration, with no match on box-integration-development.
  • a3326d85 (keep a box another process installed into the shared cache): dev port of d4ef79a4. On this base it also covers installFfi, which dev's boxRelease.ts doesn't have.
  • 83ab7874 (run agentcore dev for every Box template): dev port of 44a151c4, with the same content.
  • 61458ca1 (skip when no box runs from PATH): dev port of d5372686, with the same content, plus the CI skip this PR already had.

Scope: the PR no longer touches only e2eTest/. It now also changes src/core/project/backends/boxRelease.ts and boxRelease.test.ts. Against the merge-base, those are the only source files in the diff. None of the box-integration-only fixes are touched: boxProfileCredentials.ts (#2845), boxProfile.ts (#2848/#2849), the Windows-safe tests and Dockerfile changes (#2850), the xray grants and boxMemoryPolicy region (#2851), the #2853 items, and #2857's signing region, 68d2bbb4 and c1bc4576 are all intact. The PR still merges cleanly with the current box-integration.

Earlier comments (ours)

  1. Cherry-pick 8d5f6199 to dev: still open, and dacbf9c5 now needs the same. Dev's templates.test.ts still has no CI skip on the Box describe (only !BOX_ON_PATH), still has the py_strands_box duplicates and the FALLBACK/FAILED check, and has no 30 s wait.
  2. Single error-line query: addressed. dacbf9c5 waits 30 s before the query. That is a fixed sleep rather than a retry, but it is enough for the purpose.
  3. Broad ERROR term: unchanged. Still non-blocking; I'm not repeating it.
  4. README --tagsFilter='box' skips the per-Python cases: addressed. The new paragraph says how they run.
  5. Nit, .filter((r) => !r.skip): addressed. The doc comment now says the describe's skipIf is the only skip.
  6. Nit, "the README": addressed. It now says "the root README.md".

tejaskash's review (CHANGES_REQUESTED on 8d5f619): none of its three inline threads has a reply, and the new commits don't change what they describe. The Box cases are still gated on box being on PATH rather than on a Box-capable CDK (src/assets/cdk/package.json still pins @aws/agentcore-cdk 1.0.2), and boxRelease.json still pins an ffi shim only for PYTHON_3_14. The new box-egress test partly covers the allowed/denied gap tejaskash raised, but only under dev, not on Runtime. I'm leaving those threads to tejaskash.

Blocking

  1. verify / Linux fails at this head, at "Smoke test npm installation with host TypeScript versions": error: Script not found "test:package". CI runs the base's verify.yml, which now has bun run test:package (from the #2858 v1x sync), but it checks out pull_request.head.sha. This branch is behind box-integration, so its package.json has no test:package script. That step stops the job before the Linux binary smoke test and bun test. The fix is to merge or rebase box-integration into box/mq-7. #2858 changes none of this PR's files, and the merge is clean. No code change is needed. Windows and macOS verify passed, so the new boxRelease unit test passes there.

Non-blocking

  1. The box-egress test can't pass on this base until #2859 lands (inline). It edits .box/agent/local/profile.toml, which is dev's per-agent layout. On box-integration, agentcore create writes <project>/.box/profile.toml (boxProfilePath, manager.tsx), so readFile throws ENOENT. #2859 (open) moves the layout to .box/<agent>/{local,aws}, so this is fixed by a planned PR. Either land #2859 first, or say in the README that --tagsFilter='box-egress' needs it. CI skips the test either way, since there is no box on the runners.
  2. The per-Python box cases ignore BOX_ON_PATH (inline). The Box describes skip without box, and the README says the tests "run only when a box runs from your PATH, and skip otherwise". But py_strands_box_3_10–3_13 still skip only in CI, so a full run or --tagsFilter='runtime' on a machine without box fails at create. Using BOX_ON_PATH there means moving its definition above RUNTIME_TEMPLATES.
  3. publishInstall's comment promises more than the code ensures (inline). There is a check-then-remove window: process A sees installed() false, B renames its complete payload into root, then A's rmSync(root) deletes B's install while B's command may be copying out of it. The end state is still a complete install, so this is rare and self-healing, but "once installed() holds it is never removed" isn't strictly true. Either reword the comment, or rename first and only remove root (then retry) when the rename fails and installed() is false. Also, on Windows a rename onto an existing folder fails with EPERM, not EEXIST/ENOTEMPTY. The code handles that, because it checks installed() rather than the code, but the comment doesn't mention it.
  4. Test coverage of publishInstall: the new unit test covers only the early return, where the other process finished before this one publishes. It doesn't cover the catch path (the rename fails and the winner's copy is kept), or the installFfi caller, which is the box-integration-only part of this port.
  5. The box dev cases check less than the Runtime cases. invokeLocally checks complete, and the new describe checks box: running profile. Neither checks the 2xx status or the non-empty body that the Runtime path now requires, so an HTTP box agent that starts and then answers with an error passes under dev.
  6. Nit, possible silent skip: test.skipIf(!runtime) in the egress describe skips silently if agent-python-minimal-box is renamed or removed from BOX_TEMPLATES. An expect(runtime).toBeDefined() (or a throw) would make that loud.
  7. Nit, README: "box ships only arm64 builds, so they skip on the x86_64 Linux CI runner". The skip comes from there being no box on PATH, and the CI-only skip on the Runtime describe. Neither depends on the runner's architecture.

Windows safety: no new unsafe patterns. spawnSync("box", ["--version"]) uses no shell and inherits the environment, paths use join, and the new unit test uses no POSIX modes.

CI: verify / Linux fails (blocking item 1). verify / Windows, verify / macOS, CodeQL, Analyze and the agentcore-e2e CodeBuild check pass. The e2e job is skipped on PRs, so none of the new e2e cases have run in CI. Nothing is pending. The known flaky checks on the base are non-blocking: the Windows transactionSearch EFAULT test and the 5 s timeouts in the macOS screen tests. Neither showed up here.

Decision: COMMENT. Only the Linux verify failure blocks, and a merge of box-integration into the branch should clear it. Everything else is non-blocking.

Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread src/core/project/backends/boxRelease.ts Resolved
Address the review of #2865.

- The Box cases that deploy run only with AGENTCORE_E2E_BOX_CDK, an
  @aws/agentcore-cdk version or file: tarball that deploys Box projects;
  createTemplateProject installs it in each Box project's agentcore/cdk.
  The pinned release still rejects boundRuntime, so the CI-only skip hid
  that a laptop run fails at synth the same way. CI sets no opt-in.
- The per-Python Box cases take the same gate, so they also skip without
  box on PATH.
- The README says which steps need box: create for a CodeZip box
  template, build and deploy (box profile show) for every Box template.
- The egress test throws at collection when its template case is gone,
  instead of skipping silently.
- publishInstall renames first and removes the cache folder only when
  the rename fails and the folder is not a complete install, and its
  comment names the window that remains and EPERM on Windows. A test
  covers replacing an interrupted install.
@mquang-amazon

Copy link
Copy Markdown
Contributor Author

Replies to the review bodies, at 2e704fce.

@leo237 (review on 61458ca):

  • Blocking, verify / Linux: merged box-integration in 216710cb, so package.json has test:package.
  • 1, port 8d5f6199/dacbf9c5 to dev: not from this PR. box-integration-development stays as is, and the plan's back-merge of box-integration into it brings them.
  • 6, dev cases: invokeLocally parses LocalRuntimeInvokeResponseSchema, which already requires a 2xx status and a non-empty body.
  • 8, README arm64 line: rewritten in 2e704fce; it now says which steps need box and that the deploy cases need the AGENTCORE_E2E_BOX_CDK opt-in.

@tejaskash (coverage gap): the Strands templates (agent_python_strands_box, the container, LangChain, A2A, AG-UI) call Bedrock through the Box gateway, so a profile that blocks Bedrock fails their 2xx and non-empty-body checks. The minimal and FastMCP templates make no outbound call, as you say. The denied side is covered under dev by box-egress, which runs on this base now that #2859 is merged. A Runtime case with an allowed and a denied host is not in this PR; it needs its own deployed stack, so I'd keep it for a follow-up.

@mquang-amazon
mquang-amazon dismissed tejaskash’s stale review November 20, 2026 04:16

Addressed in 2e704fc: the Box deploy cases are gated on the AGENTCORE_E2E_BOX_CDK opt-in (a Box-capable @aws/agentcore-cdk build), the README says which steps need box, and the ffi shim point is fixed on box-integration-development by 264da50 and 7de308e. Replies are on each thread. Dismissing because Tejas is out today; happy to revisit when he's back.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review: MQ-7 (e2e cases for the Strands Box templates), head 2e704fc

This is MQ-7 of the squash plan (phase 14). The plan doc loaded this time. It lists MQ-7 as a6d52f82, e48bf2db, 91fa62ba and 8d25cd6d on box-integration-development, plus review fixes. It has no MQ-8 row yet, so I checked #2867 against the branches directly.

What changed since 61458ca: two new commits. This is not a rebase.

  • 216710cb merges box-integration at 1405932a, which brings #2858 and #2859. The merge was clean: git show --remerge-diff shows no conflict hunks. The PR's diff against the base is still only e2eTest/README.md, e2eTest/constants.ts, e2eTest/project/templates.test.ts, boxRelease.ts and boxRelease.test.ts. So none of the fixes that exist only on box-integration (#2845, #2848/#2849, #2850, #2851, #2853, #2857, #2858, #2859) are touched.
  • 2e704fce contains the review fixes: the AGENTCORE_E2E_BOX_CDK opt-in, the per-Python gate, the README, the egress throw, and the publishInstall rework with its test. It exists only on box-integration. Dev's publishInstall (d4ef79a4) still has the old remove-then-rename code and the old comment, and dev has no AGENTCORE_E2E_BOX_CDK. You said the plan's back-merge of box-integration into dev will bring this commit, 8d5f6199 and dacbf9c5. That works for me as long as the back-merge happens. Non-blocking.

CI: verify / Linux and verify / macOS pass, and so do CodeQL and CodeBuild e2e. e2e is skipped. verify / Windows is still pending (in progress since 03:58 UTC) after a 10-minute wait, so I am posting without it. The Windows transactionSearch EFAULT test and the 5 s macOS screen-test timeouts are known flaky on the base and would not block. The two new boxRelease cache tests are the Windows-relevant ones: a rename onto a non-empty folder gives EPERM there, which the new catch handles.

Earlier comments: status

  • Blocking, verify / Linux test:package: fixed by the merge. Linux passes.
  • The per-Python cases skip only in CI: fixed. They use skip: !BOX_DEPLOYS, and BOX_ON_PATH now sits above RUNTIME_TEMPLATES.
  • box-egress needs #2859's layout: fixed by the merge. .box/<agent>/local/profile.toml matches boxProfile.ts on the base.
  • publishInstall comment, the race and EPERM: mostly fixed. It now tries the rename first, the comment names EPERM, and a test covers the catch and remove path for an interrupted install. One sentence is still off; see below.
  • The unit test covers only the early return: fixed for installBox. You skipped an installFfi test because 264da50a removes the ffi installer. That is fine.
  • The box dev cases lack the 2xx and non-empty-body checks: I withdraw this one. invokeLocally already parses LocalRuntimeInvokeResponseSchema, which requires both. My mistake.
  • Nit, test.skipIf(!runtime): fixed. The describe now throws at collection.
  • Nit, the README's reason for the CI skip: fixed. It now says which steps need box and names the opt-in.
  • Cherry-pick 8d5f6199/dacbf9c5 to dev: deferred to the back-merge, as above.
  • The broad ERROR term: unchanged. It is still non-blocking, and I won't repeat it.
  • tejaskash, the CDK pin: addressed by the AGENTCORE_E2E_BOX_CDK opt-in. createTemplateProject installs that build in agentcore/cdk and fails loudly when npm fails. You report a run of agent_python_minimal_box with a tarball packed from L3 box-integration.
  • tejaskash, the ffi shim pinned only for PYTHON_3_14: not fixed here. It is planned. The fix is 264da50a (CodeZip without the ffi shim) and 7de308e8 (box preview.9), and I checked that both are on box-integration-development. #2867 (MQ-8) ports them as e8069dda and 3934bbed. Until #2867 lands, the four py_strands_box_3_1x cases fail even with the opt-in, as you said. They are gated off in CI, so the gate does not hide a CI failure. Non-blocking because a planned PR fixes it.
  • tejaskash, box on PATH and --skip-install: your reply is right. build and deploy run box profile show for every Box runtime, and --skip-install would skip the CDK app's npm install, which deploy never runs. The README now says so.
  • tejaskash, a Runtime egress case: deferred to a follow-up. That is reasonable, since the Bedrock-calling templates' 2xx and body checks cover the allowed path.

New findings (all non-blocking)

  1. #2867 conflicts with this PR in boxRelease.ts and boxRelease.test.ts. I tried merging #2867's head into this head, and both files conflict. #2867 removes installFfi and still uses the old rmSync then renameSync in installBox. Whichever PR lands second should keep publishInstall(payload, root, () => installedBox(...)) in installBox. It should drop writeFileSync (only installFfi used it) but keep existsSync, and drop the ffi option from the two new cache tests. Otherwise #2867 silently brings back the race this PR fixes (dev has the same fix in d4ef79a4).
  2. README (inline): it says the per-Python cases run with --tagsFilter='runtime' "given box and the same opt-in". On this base they fail with the opt-in until #2867 lands. One clause saying so would save a confused laptop run. This is planned, so non-blocking.
  3. The publishInstall doc comment (inline): it says the last sentence's removal "takes a complete install away, and the next command downloads again". After the rmSync, this process renames its own complete payload into root, so the cache ends complete and the next command does not download. The real cost is that a command of the other process that is still reading from root can lose files under it.
  4. Nit: the reflowed doc comment on RuntimeTemplateTestCase.box breaks after "runs box on the", leaving a short line.

The opt-in gate is solid. BOX_CDK is trimmed and treated as unset when empty, runCommand uses cross-spawn with the inherited env, and a non-zero npm exit throws with stdout and stderr. I found no Windows-unsafe patterns in the new code: no hard-coded / paths, no : splits, no POSIX modes, no sh, and no empty env.

Decision: APPROVE. Nothing left blocks. The open items are either planned (#2867, the dev back-merge) or non-blocking.

Comment thread e2eTest/README.md Resolved
Comment thread src/core/project/backends/boxRelease.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Address leo237's last comments on #2862:

- boxStartFolder read the folder box starts in from the local profile's
  header comment, so a user who edited or removed it moved box back to
  the project root, with its project-wide grant, and the agent's
  ${{ workspace }}/.venv went missing. It now takes a CodeZip runtime's
  codeLocation from agentcore.json; the project root stays for a
  container and for a profile written before, whose paths name the
  code folder under the project root.
- A comment in serve() still said box starts the agent in the project
  folder.
- EnvLocalFile.upsert replaced only the first line of a key, so a later
  one, which the parser keeps, left the old key in force while build
  said it updated it. It replaces every line of the key, and keeps an
  export line an export line.

@vvargu vvargu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the diff and the code around it. I ran bun test src/core/project/backends/boxRelease.test.ts on 2e704fc: 12 pass, 0 fail. I did not run the e2e suite. I found no bug that blocks the merge, but please fix the two items below first.

1. The PR description does not agree with the branch.

  • The body lists 4 commits. The branch has 10 commits, which include a merge and a3326d8 fix(box): keep a box another process installed into the shared cache.
  • The body says "MQ-7 changes only e2eTest/". But the PR also changes src/core/project/backends/boxRelease.ts and its test.
  • The body says the log check looks for FALLBACK and FAILED. The code looks for ERROR, blocked by egress control and AccessDenied.
  • The body says the tests do not depend on the .box/<agent>/{local,aws} layout. But the egress test writes .box/agent/local/profile.toml. That layout is now on box-integration, so the test is correct. Only the claim is old.

Please update the body. Also, add the cache fix to it, or move the fix to a separate PR.

2. Nobody has run the new e2e tests.
The body says "Not run: the e2e suite itself". So these tests have never passed. Please run --tagsFilter='box' and --tagsFilter='box-egress' one time with a Box-capable AGENTCORE_E2E_BOX_CDK. Then add the result to the PR before the merge.

The inline comments have the other items.

Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved
Comment thread e2eTest/project/templates.test.ts Resolved

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plan PR: MQ-7 (phase 14) of the Box integration squash plan.

What changed since 2e704fce. There was no rebase. Two commits were added:

  • d6a007e7 merges box-integration at 7252504c, which brings in #2862 (MQ-6). The merge-base is now the box-integration tip, so the branch is up to date. I compared the merge with git merge-tree 2e704fce box-integration, and its tree matches the automatic merge exactly: there were no conflicts and no hand edits. git diff 2e704fce..HEAD shows no change to e2eTest/, boxRelease.ts or boxRelease.test.ts.
  • 2e455e36 fixes items from our last #2862 review. It exists only on box-integration: box-integration-development has no boxStartFolder at all, and the plan does not list this commit under MQ-7. The SYNC back-merge has to carry it, along with 8d5f6199, dacbf9c5 and 2e704fce, as already agreed.

Do the e2e expectations still hold on top of #2862? Yes.

  • The box-egress test writes app/<runtime>/main.py and appends [egress.checkip] to .box/<runtime>/local/profile.toml. It does not depend on the folder box starts in.
  • The dev cases only look for box: running profile.
  • #2862 runs start.py on Python 3.10 without tomli, which suits the py_strands_box_3_10 case.
  • The 3.10 to 3.13 cases still need #2867. See the earlier item below, which is still open.

2e455e36, checked against our #2862 findings.

  • Finding 2 (boxStartFolder depended on the header comment): addressed. The start folder now comes from runtime.codeLocation for a CodeZip runtime and is the project root for a Container runtime. Both callers pass the runtime (box.ts, and readResolvedPolicy in boxNetworkSource.ts). boxProfile.test.ts covers three cases: a profile with its comments removed, a Container runtime, and a profile written before. box.test.ts still covers both cwd/profile-argument cases. One edge case remains in the profile-written-before check (inline).
  • Finding 4 (stale comment in serve()): addressed. But the comment above the call in box.ts now has the same problem (inline).
  • Finding 5 (upsert with duplicate keys and export): addressed, with tests for both. Every line for the key is replaced, so read() returns the new value, and an export prefix is kept.
  • Findings 1 (no test with two boxed runtimes) and 3 (the boxProfileCredentials.ts:55 docstring) were not touched. They stay non-blocking.

Must-survive box-integration fixes. This PR touches source files beyond e2eTest/ and boxRelease.*, but only these: box.ts (one line), boxNetworkSource.ts (it passes the runtime through), boxProfile.ts (boxStartFolder and a comment in serve()), envLocal.ts (upsert) and their tests. No removed line touches boxProfileCredentials.ts, the #2848/#2849 profile code, the #2850 Windows-safe tests or Dockerfile flags, the #2851 grants or region, the #2853 items, #2857's signing, the c1bc4576 .venv, or anything else #2858, #2859 or #2862 added. Of #2862's changes, all of these are intact: the per-runtime dev secrets, the .venv/bin/python check, start.py reading policies as JSON on 3.10, OTEL off, the A2A runtime.url, and upsert (made stricter).

Review of the new code

  • Tests: good. Each behaviour change has a test, and the new tests are Windows-safe: they use tempRoot() paths and literal POSIX profile strings that are never used as file paths, they spawn no sh, they rely on no file modes, and they pass no empty env.
  • Silent failures: nothing blocking. The check for a profile written before is broad enough to send box back to the project root without any warning (inline). encrypted[key] ?? "" would write an empty value if encrypt ever dropped a key. That can't happen with the current cipher, so it's a nit.
  • Comments: one is stale (inline, box.ts).
  • New types: boxStartFolder now takes { readonly build: string; readonly codeLocation: string }. That's small and structural, and ProjectRuntime satisfies it. Typing it as Pick<ProjectRuntime, "build" | "codeLocation"> would make a new build type a type error here instead of a silent "" (nit).
  • Nit: codeLocation is any non-empty string. A hand-written app\agent would give posix.relative a single segment, so box run would get ../.box/.... The old header comment had the same limit, because the generator always writes app/<name>/.

Earlier comments on this PR. None of them changed, because this push did not touch e2eTest/ or boxRelease.*. From our review on 2e704fce, all non-blocking and not repeated inline:

  1. The #2867 conflict in boxRelease.ts and boxRelease.test.ts is still open. Whichever lands second must keep publishInstall in installBox.
  2. The README's 3.10 to 3.13 wording is still open. Those cases are fixed by #2867 (MQ-8, planned: 264da50a and 7de308e8, both verified on dev).
  3. The publishInstall comment wording is still open.
  4. The short-line nit on RuntimeTemplateTestCase.box is still open.

vvargu's four comments on 2e704fce (the fixed 30 s wait, the publishInstall rename-failure test, the shared deploy helper, and the projects left behind on retry) have no reply yet.

CI (at 2e455e36): verify / Linux, verify / Windows and verify / macOS pass, as do Analyze, CodeQL and CodeBuild e2e. The GitHub e2e job is skipped. Neither known flake (the Windows transactionSearch EFAULT test or the macOS 5 s screen-test timeouts) showed up; they would be non-blocking anyway.

Decision: APPROVE. The merge of #2862 is clean, and the e2e expectations still match it. 2e455e36 fixes three of our #2862 findings, with tests. No must-survive fix is lost. The new findings below and the items above are non-blocking.

New findings (all non-blocking)

  1. boxStartFolder's check for a profile written before matches any ${{ workspace }}/app/ (inline).
  2. A stale comment in box.ts (inline).
  3. Nits: Pick<ProjectRuntime, ...> for the parameter; encrypted[key] ?? ""; codeLocation with backslashes.
  4. 2e455e36 exists only on box-integration. Please list it for the SYNC back-merge, and in the PR description as a #2862 follow-up, because it is outside MQ-7's planned scope.

Comment thread src/core/dev/box.ts Resolved
boxStartFolder kept the project root for a profile whose text named
${{ workspace }}/app/, as a profile from before box started in the code
folder does. A current profile that grants ${{ workspace }}/app/<x>
matched it too, and box quietly started at the project root again, with
its project-wide grant. Box projects are new, so there is no older
profile to run: a CodeZip runtime's box starts in its codeLocation, and
a container's in the project root. The dev runner's comment says so.
On Python 3.10, with no tomllib, start.py reads the staged profile's
policies with a regex, and a line in any other form gave no policy
files and no message, so box failed later with an unclear error.
start.py now stops with an error that names the line. A test runs
staged_policies with no tomllib on the profiles deploy renders.
…ild reads

EnvLocalFile.upsert snapshotted the file before its in-place write,
then insertIfNew snapshotted the half-updated file, and the first one
came back only if insertIfNew succeeded. It now keeps the earliest
snapshot, as removeKeys does, so rollback restores the original file.

The local testing guide said a build with GITHUB_TOKEN unset keeps the
stored key. Build reads GITHUB_TOKEN from .env.local too, so a line
there replaces the stored key, even with an older one.
The Box deploy cases waited a fixed 30 s after 'running profile' and
then checked that no error line was logged, so a line CloudWatch
delivered later passed unseen. They now wait for the agent server's
access line for the invoke's POST first. Also: a README note that the
Python 3.10-3.13 box cases pass only with #2867, a test of
publishInstall's lost rename race, and two comment fixes.
Box is only for new projects, so the guide gives no cleanup hints for
.env.local lines that earlier box-integration builds wrote.
@mquang-amazon

Copy link
Copy Markdown
Contributor Author

@vvargu The PR body is updated, and the 7 Box deploy e2e cases passed (us-west-2, 7/7, results in the body). The dev groups and box-egress still need a run outside my sandbox.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of 2e455e36..f7391f24. The five new commits (21e2c91, acb9b4c, ad4617d, 5a6a441, f7391f2) sit on top of the head I approved. There is no rebase and no new box-integration merge, and the head already contains box-integration (7252504, #2862).

Blocking: verify / Windows fails because of the new boxStartPy.test.ts. This is not the known transactionSearch EFAULT flake. Bun.spawnSync throws Executable not found in $PATH: "python3" (ENOENT) when the command does not exist. It does not return a non-zero exit code. The runner has no python3, so the .find throws while the file loads ("Unhandled error between tests"). It never tries python, and the skipIf never runs. Details are inline. Linux and macOS pass, CodeQL and Analyze pass, and e2e is skipped.

Earlier comments now addressed

  • boxProfile.ts silent fallback (mine): fixed in 21e2c91. boxStartFolder no longer takes the profile text, so no profile content can move box back to the project root. The boxProfile/box tests cover CodeZip, ./, a trailing /, Container, and a profile with its comments removed.
  • box.ts stale comment (mine): fixed.
  • README: the per-Python cases now say they need #2867. Fixed.
  • publishInstall doc comment accuracy: fixed. It now says the removal takes the other install away while its command may be reading it, and that this process's payload replaces it.
  • The short line in the box doc comment: fixed.
  • vvargu's fixed 30 s wait before the error check: fixed. The check now retries until the invoke's own "POST /" access line shows up, then queries for errors.
  • vvargu's comment that the race test does not reach the rename-failure branch: fixed. The new publishInstall tests run the lost-rename branch (checks is 2), the branch that replaces an incomplete folder, and the plain rename.
  • vvargu's helper-extraction nit and the nit about cleaning up the retry: 2 projects: the author declined both with reasons. Both stay non-blocking.
  • The PR description now names 2e455e3 and 21e2c91 as follow-ups to #2862.

Other new changes look right

  • start.py (acb9b4c) now stops with a clear error on a policies line the 3.10 regex can't read, instead of starting box with no policy files. The tests run staged_policies without tomllib on profiles from renderRuntimeBoxProfile. Only the Windows load problem above is wrong.
  • EnvLocalFile.upsert (ad4617d) keeps the earliest snapshot, and the finally puts it back when insertIfNew fails. The new test covers that case.
  • The docs are accurate: GITHUB_TOKEN can also come from .env.local, and the troubleshooting item for earlier branch builds is gone.

Must-survive fixes: this PR's diff against box-integration touches only the box e2e, boxRelease, boxProfile/boxNetworkSource/box.ts, envLocal, start.py and the docs above. The #2845-#2862 markers I grepped for are all still there, and none were lost.

Non-blocking

  • upsert's comment says it keeps the earliest snapshot "as removeKeys does". When updated is empty, though, insertIfNew still overwrites any earlier snapshot (this.snapshot = existing). That behavior is older than this PR. You could make insertIfNew keep the earliest snapshot too, so all three methods agree.
  • My earlier nits are still open: the Pick<ProjectRuntime, "build" | "codeLocation"> parameter type, encrypted[key] ?? "", and backslashes in codeLocation.
  • SYNC back-merge: 21e2c91, acb9b4c, ad4617d, 5a6a441 and f7391f2 exist only on box-integration. box-integration-development has no boxStartFolder and no new start.py error. They join 8d5f619, dacbf9c, 2e704fc and 2e455e3 on the back-merge list.
  • Older items still apply: the #2867 conflict in boxRelease.ts/boxRelease.test.ts (whichever PR merges second keeps publishInstall in installBox and drops the ffi half; the new publishInstall tests are part of the half to keep). The broad ERROR term also still applies.
  • The author did not run the agentcore dev groups or box-egress (they say these need a normal terminal). The 7 Box deploy cases passed.

Bun.spawnSync throws on Windows when the executable is not on PATH,
so the python3 probe failed the file before any test ran.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of f7391f24..96c63bd9. There is one new commit, 96c63bd ("test(box): find Python on Windows without throwing"), on top of the head I last reviewed. There is no rebase and no new box-integration merge. The head still contains box-integration (7252504, #2862), so test:package is present.

Earlier comments now addressed

  • Blocking, boxStartPy.test.ts crashing verify / Windows (mine): fixed in 96c63bd. The probe now wraps Bun.spawnSync in try/catch and returns false when it throws. A missing python3 no longer throws while the module loads, so .find goes on to python, and the describe.skipIf(PYTHON === undefined) works as the doc comment says. I suggested Bun.which, and try/catch is just as good. It also handles a name that resolves to something that can't be spawned. The Windows Store python3 alias stub exits non-zero, so it is also rejected and python gets tried. I checked the rest of the file for Windows problems: it uses join/tmpdir paths, no POSIX modes, no sh, and no empty env. start.py is pure ASCII, and staged_policies opens the profile in "rb", so the harness's open(...).read() doesn't depend on the Windows locale encoding.
  • Everything else I listed as addressed in my last review is still addressed. Nothing new came in from the author or vvargu after that review. The author's last replies (05:24-05:25) came before it.

New finding (nit, non-blocking)

  • boxStartPy.test.ts:22: the comment says "On Windows, spawnSync throws, rather than failing, for a name not on PATH." Bun throws Executable not found in $PATH for a missing command on Linux and macOS as well. Windows is just the runner where python3 is missing. Something like "spawnSync throws, rather than failing, for a name not on PATH (the Windows runner has no python3)" would be accurate.

CI: everything passes on 96c63bd9: verify / Windows, verify / Linux, verify / macOS, Analyze, CodeQL and the CodeBuild e2e project. The GitHub e2e job is skipped. On the Windows runner, all three start.py with no tomllib tests ran and passed (they were not skipped), so the probe found python. This run hit none of the known base flakes (the Windows transactionSearch EFAULT test, the macOS 5 s screen-test timeouts). Those would be non-blocking anyway.

Must-survive fixes: 96c63bd touches only boxStartPy.test.ts. The PR's diff against box-integration is still the same 16 files (box e2e, boxRelease, boxProfile/boxNetworkSource/box.ts, envLocal, start.py, docs). It loses none of the #2845-#2862 fixes.

Coordination with #2867 (still open, at f40ebce): I trial-merged #2867 into this head. boxNetworkSource.ts merges cleanly. This PR changes readResolvedPolicy's parameter to runtime: ProjectRuntime and calls boxStartFolder(runtime). #2867 only rewords the @box/bedrock comment inside that function, moves BOX_BEDROCK_PACK out, and drops "ffi" from BOX_DIST_NAMES. Neither PR has refuseOldShippedNames left in this file. The only conflicts are the known ones in boxRelease.ts and boxRelease.test.ts. Whichever PR merges second keeps publishInstall (and its tests) in installBox and drops the ffi half.

Still open, non-blocking (unchanged from my last review)

  • upsert's comment says it keeps the earliest snapshot "as removeKeys does", but insertIfNew still overwrites an earlier snapshot when updated is empty. That behavior is older than this PR.
  • Earlier nits: the Pick<ProjectRuntime, "build" | "codeLocation"> parameter type, encrypted[key] ?? "", and backslashes in codeLocation.
  • The broad ERROR term in the e2e error check.
  • The agentcore dev groups and box-egress have not been run yet (the author says they need a normal terminal). The 7 Box deploy cases passed.
  • SYNC back-merge: 96c63bd exists only on box-integration (box-integration-development has no boxStartPy.test.ts at all). Add it to 8d5f619, dacbf9c, 2e704fc, 2e455e3, 21e2c91, acb9b4c, ad4617d, 5a6a441 and f7391f2 on the back-merge list (the SYNC row in the squash plan).
  • FYI: #2854 (containment, base box-integration) is also open.

Approving. The one blocking item is fixed and CI is green. Everything left is a nit, already-known non-blocking, or SYNC back-merge bookkeeping.

@vvargu
vvargu merged commit b805512 into box-integration Nov 20, 2026
7 checks passed
@vvargu
vvargu deleted the box/mq-7 branch November 20, 2026 06:17
SaiAbhishek98 pushed a commit to SaiAbhishek98/agentcore-cli that referenced this pull request Nov 20, 2026
…gent-v5

[aws#2865](aws#2865)
adds e2e tests for each Strands Box template, reads the start folder of
box from agentcore.json, and puts a box download in the cache with
publishInstall. The resolution of each conflict:

- templates/boxProfile.ts: the `boxStartFolder(runtime)` of
  box-integration, and `CODEZIP_START_FOLDER` and `RUNTIME_URL` of this
  branch. `START_FOLDER_LINE` goes.
- boxNetworkSource.ts: the profile source gets the start folder that
  readResolvedPolicy reads from the runtime.
- backends/boxRelease.ts and its test: publishInstall and its tests of
  box-integration, with no `installFfi`, because this branch removes the
  ffi shim. The two race tests give no ffi release.
- dev/box.test.ts: the tests of box-integration, and the test of this
  branch "runs each runtime with the box profile of that runtime" with
  the start folder of each runtime.
- backends/boxStartPy.test.ts, new in box-integration, wrote its Runtime
  profiles with `renderRuntimeBoxProfile`, which this branch removes. It
  writes them with `serializeRuntimeProfile`, as the runtime profile
  writer does.

The e2e tests examine what an invoke and the logs give, not the text of
the Runtime profile, so no change to them is necessary here.
SaiAbhishek98 pushed a commit to SaiAbhishek98/agentcore-cli that referenced this pull request Nov 20, 2026
… starts box in

[aws#2865](aws#2865)
makes `boxStartFolder` read `build` and `codeLocation` from agentcore.json,
not the header line of the profile. Thus dev starts box in the code
folder of each CodeZip runtime, also for a profile that names the code
folder `${{ workspace }}/<codeLocation>`. The path model of this branch
had a condition for such a profile, of the earlier layout. A Box project
is only for new projects, so Varun on
[aws#2866](aws#2866)
told us to remove that condition.

- `runtimePaths(runtime)` names the local code folder `${{ workspace }}`
  for each CodeZip runtime, and has no start folder parameter.
- The profile source of the Box network model and the context of the
  derivation have no `startFolder`.
- The rule tests use the profiles of the code-folder layout. The test of
  a path of the project that is not in the code folder goes, because
  `${{ workspace }}` is the code folder.

Tests: derive.test.ts "moves a CodeZip runtime's code folder,
${{ workspace }}, and Python to the inputs of @box/agentcore-codezip",
and the tests of the four rules, on the code-folder layout.

@tejaskash tejaskash left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the explicit CDK opt-in and the recorded 7 of 7 Box deploy cases look good, and #2867 has since pinned the shims for the per-Python cases. Two follow-ups, since they aren't covered yet:

  1. A deployed Box runtime case with one allowed host and one denied host. The egress test added here runs only under agentcore dev.
  2. One recorded run of the 12 Box cases that haven't passed anywhere yet (the 4 per-Python cases, the 7 dev cases and the egress test), on box-integration after #2867, with box on PATH.

Comment thread e2eTest/README.md

That runs each Box template on its default Python. The CodeZip box template on Python 3.10 to 3.13
(`py_strands_box_3_10` to `py_strands_box_3_13`) is in the runtime templates describe, so it runs
with `--tagsFilter='runtime'` or a full run, given `box` and the same opt-in. Those four cases

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that #2867 has merged, the sentence saying the four per-Python cases pass only once it lands can go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #2886 (89b4acf): the sentence is removed.

for (const entry of spec.runtimes) entry.runtimeVersion = runtime.runtimeVersion;
await writeFile(specPath, `${JSON.stringify(spec, null, 2)}\n`);
}
if (runtime.box && BOX_CDK) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small one: the dev cases and the egress test go through this too, so they install the opt-in CDK build they never use. That's 8 extra npm install runs inside the create time budget. Could this only install for cases that deploy?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in #2886 (89b4acf): createTemplateProject takes { deploys }, and only the deploy cases install the AGENTCORE_E2E_BOX_CDK build. The dev and dev-egress cases no longer run npm install for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants