Repository navigation
ci(e2e): Run the Box e2e tests on macOS and Linux arm64 runners with the pinned box - #2872
mquang-amazon wants to merge 8 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## box-integration #2872 +/- ##
================================================
Coverage 96.80% 96.80%
================================================
Files 766 766
Lines 66803 66804 +1
================================================
+ Hits 64666 64667 +1
Misses 2137 2137 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
leo237
left a comment
There was a problem hiding this comment.
First review, at head 2c2bb679 (macOS install plus the new Linux arm64 job). Thanks, the step is small and readable, and it works mechanically against the current pin.
What I verified (read-only, nothing executed)
- #2871 has merged, so the step now reads
0.1.11-preview.10. That release publishesbox-v0.1.11-preview.10-aarch64-apple-darwin.tar.gz,...-aarch64-unknown-linux-gnu.tar.gz, andSHA256SUMS.txt. The SUMS lines use the<hash> <name>format, sogrep " ${ASSET}$"matches for bothTARGETs, and the hashes equal the GitHub asset digests. preview.9 has the same layout. - Failure handling holds up. The default
runshell isbash -e.curl -ffails on a 404, including whenjqprintsnull. Ifgrepmatches nothing,shasum -c -gets empty input and exits 1: Perlshasumsets$erron "no properly formatted SHA checksum lines".jqandshasumare on bothmacos-latestandubuntu-24.04-arm,/usr/local/binis on PATH, andsudois passwordless on hosted runners. - Both tarballs keep
./boxand./resources/at the root, so-C /opt/boxplus the symlink keeps the layout box needs.BOX_ON_PATHturns true, socreate and dev for Strands Box templatesruns.BOX_DEPLOYSstays false withoutAGENTCORE_E2E_BOX_CDK. - Values from
boxRelease.jsongo only into quoted shell arguments.TAGSgoes throughenvand"$TAGS", not${{ }}. The workflow runs only on an authorizedworkflow_call/workflow_dispatch, so I see no new injection path. - Branch: 1 behind
box-integration(#2871 only, files disjoint, mergeable). Thee2ejob skips on the PR, so none of these steps has run in CI yet.verify/Analyze on2c2bb679were still pending when I posted.
Blocking
- The box binary isn't checked against the in-repo pin, and it runs with the E2E AWS role in the environment. The step checks the tarball against
SHA256SUMS.txtfrom the same release. Both releases reportimmutable: false, so anyone who can edit the release can replace the tarball and SUMS together, and the check still passes. The binary then runs right afterConfigure AWS credentials. For Linux arm64 the pin already exists:.box.linux.sha256/.box.linux.assetinboxRelease.json. #2867 deliberately moved the CLI offSHA256SUMS.txtonto that pin. Please verify againstboxRelease.json: use.box.linuxfor Linux arm64, and add adarwinentry (target/asset/sha256, written byscripts/bump-box-release.ts) for macOS. Inline suggestion below.
Non-blocking
2. The Linux arm64 test selection via contains(inputs.tags, 'box') is a substring check. tags: '!box' or 'runtime || box' makes the arm64 job run that whole expression, so it duplicates the x86 Linux job's deploys (inline).
3. The dev path on hosted runners is untested. That covers both macOS and arm64 with the AppArmor sysctl. A failure turns the job red rather than skipping, so please do one authorized workflow_dispatch of this branch (for example tags: box) before merge. On macOS the default "Run all E2E tests" also runs the network-dependent box-egress test (public hosts, retry: 2). Neither the description nor the README mentions it.
4. Nit: sudo tar keeps the archive owners: mquang/staff uid 501 on darwin, uid 18484610 on Linux. Files stay world-readable, so box works, but --no-same-owner would match the Dockerfile fix.
5. Nit: the PR title and description still describe only the macOS runner. They also mention a Box build test, but only the create + dev Box describe runs. The new README sentence runs well past the ~100-column wrap.
6. This is off-plan in the squash plan v2, and it isn't on box-integration-development (e2e-test.yml there has no Install box). Please back-merge so a later phased squash doesn't drop it.
…with sha256sum on Linux
Pin the macOS arm64 build in boxRelease.json beside the Linux one, written by scripts/bump-box-release.ts. The e2e install step checks each archive against that pin, not against SHA256SUMS.txt of the same release, which anyone who can edit the release could replace with the archive. It also extracts with --no-same-owner.
leo237
left a comment
There was a problem hiding this comment.
Re-review at 6a487349 (last review was at 2c2bb679). Scope: the PR's own commits c1adceff and 6a487349, plus the conflict resolution in merge c4d53f4b. I did not re-review the base commits it brought in (#2860, #2871). Everything below is read-only. I did not run or download anything.
The earlier blocker is fixed
- "Install box" now reads
repo,version,assetandsha256fromboxRelease.json, using.box.darwinon macOS and.box.linuxon Linux arm64. It checks the tarball withecho "${SHA} ${WORK}/${ASSET}" | $SUM -c -, which has the right two-space format, usingsha256sumorshasum -a 256.SHA256SUMS.txtis gone from CI, and the CLI still doesn't read it (downloadCheckedcompares against the pin). - Failures stop the step.
jq -eexits non-zero on a missing key ornull, and under the defaultbash -ea failed$(...)assignment ends the step. An empty or malformed hash leaves-cwith no properly formatted line, so it exits 1. Thesha256regex inBoxReleaseSchemaalso stops a malformeddarwinhash at import, sobun testcatches it. - I checked the pins against the GitHub asset digests on
mquang-amazon/box-releasesv0.1.11-preview.10.darwinis76a3909c...ef57e18andlinuxis45740d3e...c2dc583, and both match. The release is stillimmutable: false, which is why the in-repo pin matters. bump-box-release.tswritesdarwinthrough the samepin()aslinux, and it throws if the release doesn't list the asset.BoxReleaseSchemaaddsdarwinas optional. Zod's defaultobjectstrips unknown keys anyway, and nothing iterates over the entries, so the CLI can't pickdarwinby mistake:resolveBoxLinuxBuildandbox-release-test-supportonly use.linux.- Values from the JSON go only into quoted arguments.
BUILDcomes from the matrix throughenv. Triggers are unchanged (workflow_call/workflow_dispatch, still behindauthorize), and so are the permissions (id-token: write,contents: read).
Merge c4d53f4b: The branch is 0 behind box-integration. Compared with the merge base, the PR touches only its 6 files, all additions to or edits of its own hunks. The pin is still 0.1.11-preview.10 from #2871. No base fix is reverted.
Earlier comments
- Blocking, verify against the pin: addressed (see above).
contains(inputs.tags, 'box')substring match: moot.c1adceffdropped the arm64-specific selection, and arm64 now runs the same command as the other jobs.- Untested on hosted runners: still open (non-blocking). The description lists
box runonmacos-latest/ubuntu-24.04-armas not tested, and I see noe2e-test.ymldispatch ofbox/e2e-macos-box. One authorized dispatch would also show whethermacos-latesttakes thesha256sumor theshasumbranch. Thebox-egress/ public-hosts behaviour is now documented in the description and ine2eTest/README.md, so that part is addressed. On OIDC: the macOS job already assumes the E2E role from a GitHub-hosted runner, so the trust policy probably acceptsubuntu-24.04-armtoo, unless it has a runner-specific condition. --no-same-owner: addressed.- Title, description and README wrap: addressed. The title and description cover both runners, the PR no longer claims a Box
buildtest, and the new lines are within the wrap width. - Squash plan / back-merge: acknowledged. The description's follow-up is to merge
box-integrationback intobox-integration-development, which doesn't have it yet. That's fine after merge.
New findings (all non-blocking)
linuxOnlynow also runsLinux-arm64(inline).canary.ymlcalls this workflow withlinuxOnly: true,tags: canary, so after this reaches the default branch, every canary run deploys twice and its heartbeat also depends on a hosted arm64 runner.- Running the whole suite on arm64 doubles the x86 Linux job's deploys on every push to the branch, with 4 jobs in parallel in one account instead of 3. Names are
E2E_PREFIX+Date.now()base36 (plus a template index), the same scheme the 3 existing parallel jobs already share, and the stale-stack sweep only removes stacks older than 2h. So I don't expect collisions, but watch the account's Runtime/Gateway quotas. The description says this is deliberate. - The AppArmor sysctl is scoped to
matrix.name == 'Linux-arm64'on a throwaway hosted VM. That's fine. - The L3
enableApplicationLogsgap (aws/agentcore-l3-cdk-constructs#470) doesn't turn this red. The Box deploy tests still skip withoutAGENTCORE_E2E_BOX_CDK, and only Box gateways write that field. - Test nit (inline): nothing asserts the
darwinpin's shape against the version the way thelinuxone is checked. bump-box-release.tsnow refuses a release with no macOS build. That's probably what you want, since CI needs it. It's just a new coupling to be aware of for any Linux-only preview.
CI: verify / Linux, verify / Windows, Analyze, CodeQL and the CodeBuild check pass. verify / macOS was still pending when I posted, and e2e skips on PRs as expected. The branch is mergeable and 0 behind base.
Thanks, the blocker is fixed cleanly.
…ox pin canary.yml runs this workflow with linuxOnly, so a Linux arm64 job there doubled each canary's deploys and made its heartbeat wait on a hosted arm64 runner. The pinned releases test now checks the darwin entry that the e2e workflow installs from, so a stale pin fails in bun test.
leo237
left a comment
There was a problem hiding this comment.
Re-review of the delta 6a48734...bc71015 (one commit: the matrix expression in e2e-test.yml and 5 lines in boxRelease.test.ts).
Now addressed from my last review:
- Canary runs twice: fixed.
linuxOnlyis declaredtype: booleanunder bothworkflow_callandworkflow_dispatch, and those are the only two triggers. Soinputs.linuxOnly && '["Linux"]' || '[...all four]'gives onlyLinux(the x86_64 CodeBuild runner) when it is true.canary.yml(linuxOnly: true,tags: canary) now deploys once. When it is false or unset, the matrix is unchanged: Linux, Linux-arm64, Windows and macOS. Theruns-onmapping and the per-OS stepif:s did not change, so macOS and Windows behave as before. - No unit test for the darwin pin: fixed. The test now checks
BOX_RELEASE.darwin?.assetagainstbox-v${version}-aarch64-apple-darwin.tar.gzand checks thatdarwin?.sha256is 64 hex characters. If the darwin entry is missing,undefinedwill not match, so the test fails. A wrong version or target also fails it. The repo is shared across platforms and is already covered by theBOX_RELEASE_PAGEassertion. The test only checks constants, with no network, file paths, file modes or subprocesses, so it is safe on all CI runners.
No new findings. CI on the new head: CodeQL, Analyze and verify / Windows pass. verify / Linux and verify / macOS were still running when I posted this.
The hosted macOS arm64 runner has no nested virtualization, so it cannot run a container tool, and agentcore dev of the box container template fails there. The Linux arm64 runner still runs it.
leo237
left a comment
There was a problem hiding this comment.
Re-review of the delta since bc71015 (887297f, skip the Box container dev test on CI macOS). The change looks right. I'm leaving this as a comment rather than an approval for two reasons. verify / Linux, macOS and Windows are still pending at 887297f. The branch is also 1 commit behind box-integration (e2ac9d7, docs-only, touching docs/box-local-testing.md; no conflict). Please update the branch once verify goes green.
The skip is narrow and matches the README:
- It filters only
agent-python-strands-box-container, and only out of the sequentialcreate and devdescribe. The deploy describe still runs the container case. - It applies only when
CI === "true"and the platform is not Linux. Local macOS runs and the Linux arm64 job still run the test. - The reason is the real one. Dispatch run 44820634395 attempt 2 (at bc71015) failed on macOS only, on this test, with
'container runtime' was not found on your PATH. The other 6 Box dev tests passed. Linux-arm64 ranagent_python_strands_box_containerand passed (7 passed). Linux and Windows also passed.
That dispatch's ref is the sha bc71015. The queued attempt-3 rerun of macOS will therefore check out bc71015 again and fail the same way, so it won't validate this fix. A new dispatch at 887297f would.
Non-blocking: see the inline note about using a capability check. The earlier open items still stand: the back-merge to box-integration-development, the full suite on arm64 doubling deploys and quotas, and darwin being optional in the schema.
| const cli = new CliRunner(); | ||
| // The container template's dev needs a container tool, which a CI macOS runner cannot run | ||
| // (no nested virtualization), as py_strands_container notes. Linux arm64 runs it. | ||
| const noLocalContainers = process.env.CI === "true" && process.platform !== "linux"; |
There was a problem hiding this comment.
Non-blocking: this mirrors py_strands_container's CI && !linux check, which is consistent. A capability check would be more precise, for example skipping when none of docker, podman or finch is on PATH (the same tools agentcore dev looks for). That would also cover a local Mac with no container tool, and it would start running the test automatically if a runner gains one. Note too that !== "linux" also matches Windows, which the comment and README don't mention. It's harmless, since box isn't installed on Windows and the describe skips there anyway.
There was a problem hiding this comment.
Done in 5049104: the container case skips when none of docker, podman or finch runs from PATH, the tools agentcore dev looks for, instead of CI && !linux. Checked with vitest list: 6 dev cases with no tool on PATH, 7 with one. The README says the same.
…s from PATH Addresses the review of #2872: a capability check in place of CI && !linux. It also covers a local machine with no container tool, says nothing about Windows by accident, and runs the test once a runner gains one.
leo237
left a comment
There was a problem hiding this comment.
Re-reviewed 5049104. The new capability check looks good. spawnSync from node:child_process returns status: null and does not throw when a binary is missing, so the test file can't crash. It checks the same docker/podman/finch --version order that resolveContainerTool in src/core/dev/container.ts uses. The skip only covers agent-python-strands-box-container, so Linux x86/arm64 runners and a local Mac with Docker still run it. This also fixes the macOS failure in the e2e dispatch at bc71015 (run 44820634395, 'container runtime' was not found on your PATH).
There's no e2e dispatch at this head yet, so it's worth running one before merge. verify / Linux and verify / Windows were still pending when I posted, with no failures so far. Please merge only after they pass.
Non-blocking, still open: back-merge to box-integration-development, the full suite on arm64 doubling deploys and quota use, and darwin being optional in the schema.
| template a container tool. Box listens on the protocol's fixed port (8080, 8000 or 9000), so the | ||
| `dev` tests run one at a time and need those ports free: | ||
| template a container tool. The container template's `dev` test skips when no `docker`, `podman` | ||
| or `finch` runs from PATH, as on the CI macOS runner; the Linux arm64 runner runs it. Box listens on the protocol's fixed port (8080, |
There was a problem hiding this comment.
Nit: this line runs to 133 columns while the rest of the paragraph wraps at 100. Could you rewrap it? One caveat: docker --version passes even when the engine is down, so on a Mac where Docker Desktop is installed but stopped, this test runs and fails instead of skipping. That's probably fine for a local run.
Description
The Box e2e tests skip on every CI runner. They need a
boxon PATH (BOX_ON_PATH), and box ships only arm64 builds. The Linux and Windows runners are x86_64, and the macOS runner has no box installed.This PR runs them on two arm64 runners:
macos-latest, Apple silicon): installs box, then runs the suite as before.ubuntu-24.04-arm, new): installs box, then runs the full suite, like the other jobs. It is the same OS and CPU as AgentCore Runtime. Asysctlstep lets an unprivileged process make a user namespace, which Ubuntu 24.04's AppArmor blocks andbox runneeds for its sandbox. The runner is a throwaway VM.The "Install box" step:
src/core/project/backends/boxRelease.json(.box.darwinon macOS,.box.linuxon Linux).jq -efails on a missing key.curl. The release is public, so no token is needed.SHA256SUMS.txtof the same release./opt/boxwith--no-same-owner, because box reads its packs fromresources/beside it. Then links/usr/local/bin/boxto it.boxRelease.jsonnow pins the macOS arm64 build beside the Linux one.scripts/bump-box-release.tswrites both. The schema takesdarwinas optional, and the CLI does not read it.What runs in CI after this PR: on macOS and Linux arm64, the Box
createanddevtests (create and dev for Strands Box templates) and thebox-egresstest, which reaches public hosts and retries twice. The Box deploy tests still skip, because they also needAGENTCORE_E2E_BOX_CDK(a Box-enabled@aws/agentcore-cdk), which CI does not set.Follow-up: after this merges, merge
box-integrationback intobox-integration-development, so that a later squash keeps it.Related Issue
Closes #
Documentation PR
Not applicable.
e2eTest/README.mdandREADME.mdare updated in this PR.Type of Change
Testing
How have you tested the change?
bun test(src/core/project/backends/: 234 pass)bun run test:e2e, or explained why they are not applicable(CI does not run them for you): the workflow has not run yet. It needs an authorized
workflow_dispatchof this branch.bun run typecheckbun run lint:checkbun run format:checkbun run audit: not applicable, no dependency changebun run secrets:check: not applicablebun run build: not applicablesrc/assets/, I updated affected snapshots: not applicableby this PR: not applicable
I ran the "Install box" script by hand:
box 0.1.11-preview.10installs, the checksum passes, and the files are owned by root. With a changedsha256in the pin, the step fails.box 0.1.11-preview.10installs and the checksum passes.box profile show --format json --fullresolves@box/base, so box findsresources/. In a privileged container as a non-root user,box runstarts its sandbox and runs a command. In an unprivileged container,box runfails with "may not create a user namespace", which is what thesysctlstep addresses.Not tested:
box runon the hostedmacos-latestandubuntu-24.04-armrunners.ubuntu-24.04-arm. The E2E role's trust policy must accept a GitHub-hosted runner.