Skip to content

ci(e2e): Run the Box e2e tests on macOS and Linux arm64 runners with the pinned box - #2872

Open
mquang-amazon wants to merge 8 commits into
box-integrationfrom
box/e2e-macos-box
Open

mquang-amazon wants to merge 8 commits into
box-integrationfrom
box/e2e-macos-box

Conversation

@mquang-amazon

@mquang-amazon mquang-amazon commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

The Box e2e tests skip on every CI runner. They need a box on PATH (BOX_ON_PATH), and box ships only arm64 builds. The Linux and Windows runners are x86_64, and the macOS runner has no box installed.

This PR runs them on two arm64 runners:

  • macOS (macos-latest, Apple silicon): installs box, then runs the suite as before.
  • Linux arm64 (ubuntu-24.04-arm, new): installs box, then runs the full suite, like the other jobs. It is the same OS and CPU as AgentCore Runtime. A sysctl step lets an unprivileged process make a user namespace, which Ubuntu 24.04's AppArmor blocks and box run needs for its sandbox. The runner is a throwaway VM.

The "Install box" step:

  1. Reads the repository, version, archive, and SHA-256 of the runner's build from src/core/project/backends/boxRelease.json (.box.darwin on macOS, .box.linux on Linux). jq -e fails on a missing key.
  2. Downloads the archive with curl. The release is public, so no token is needed.
  3. Checks the archive against the SHA-256 pinned in the repository, not against SHA256SUMS.txt of the same release.
  4. Extracts the whole release to /opt/box with --no-same-owner, because box reads its packs from resources/ beside it. Then links /usr/local/bin/box to it.

boxRelease.json now pins the macOS arm64 build beside the Linux one. scripts/bump-box-release.ts writes both. The schema takes darwin as optional, and the CLI does not read it.

What runs in CI after this PR: on macOS and Linux arm64, the Box create and dev tests (create and dev for Strands Box templates) and the box-egress test, which reaches public hosts and retries twice. The Box deploy tests still skip, because they also need AGENTCORE_E2E_BOX_CDK (a Box-enabled @aws/agentcore-cdk), which CI does not set.

Follow-up: after this merges, merge box-integration back into box-integration-development, so that a later squash keeps it.

Related Issue

Closes #

Documentation PR

Not applicable. e2eTest/README.md and README.md are updated in this PR.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe): CI, the e2e workflow installs the pinned box on a macOS runner and a new Linux arm64 runner

Testing

How have you tested the change?

  • I ran bun test (src/core/project/backends/: 234 pass)
  • I ran the relevant end-to-end tests with bun run test:e2e, or explained why they are not applicable
    (CI does not run them for you): the workflow has not run yet. It needs an authorized workflow_dispatch of this branch.
  • I ran bun run typecheck
  • I ran bun run lint:check
  • I ran bun run format:check
  • I ran bun run audit: not applicable, no dependency change
  • I ran bun run secrets:check: not applicable
  • I ran bun run build: not applicable
  • If I modified src/assets/, I updated affected snapshots: not applicable
  • If I re-recorded fixtures or golden files, every change beyond account identifiers is explained
    by this PR: not applicable

I ran the "Install box" script by hand:

  • Ubuntu 24.04 arm64 container (finch): box 0.1.11-preview.10 installs, the checksum passes, and the files are owned by root. With a changed sha256 in the pin, the step fails.
  • Apple silicon Mac: box 0.1.11-preview.10 installs and the checksum passes.
  • Through the symlink: box profile show --format json --full resolves @box/base, so box finds resources/. In a privileged container as a non-root user, box run starts its sandbox and runs a command. In an unprivileged container, box run fails with "may not create a user namespace", which is what the sysctl step addresses.

Not tested:

  • box run on the hosted macos-latest and ubuntu-24.04-arm runners.
  • The AWS login through OIDC from ubuntu-24.04-arm. The E2E role's trust policy must accept a GitHub-hosted runner.

@mquang-amazon
mquang-amazon requested a review from a team November 20, 2026 18:16
@mquang-amazon mquang-amazon changed the title box/e2e macos box ci(e2e): Install the pinned box on the macOS runner so the Box tests run Nov 20, 2026
@codecov-commenter

codecov-commenter commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.80%. Comparing base (f98cfd6) to head (5049104).
⚠️ Report is 7 commits behind head on box-integration.

Additional details and impacted files
@@               Coverage Diff                @@
##           box-integration    #2872   +/-   ##
================================================
  Coverage            96.80%   96.80%           
================================================
  Files                  766      766           
  Lines                66803    66804    +1     
================================================
+ Hits                 64666    64667    +1     
  Misses                2137     2137           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review, at head 2c2bb679 (macOS install plus the new Linux arm64 job). Thanks, the step is small and readable, and it works mechanically against the current pin.

What I verified (read-only, nothing executed)

  • #2871 has merged, so the step now reads 0.1.11-preview.10. That release publishes box-v0.1.11-preview.10-aarch64-apple-darwin.tar.gz, ...-aarch64-unknown-linux-gnu.tar.gz, and SHA256SUMS.txt. The SUMS lines use the <hash> <name> format, so grep " ${ASSET}$" matches for both TARGETs, and the hashes equal the GitHub asset digests. preview.9 has the same layout.
  • Failure handling holds up. The default run shell is bash -e. curl -f fails on a 404, including when jq prints null. If grep matches nothing, shasum -c - gets empty input and exits 1: Perl shasum sets $err on "no properly formatted SHA checksum lines". jq and shasum are on both macos-latest and ubuntu-24.04-arm, /usr/local/bin is on PATH, and sudo is passwordless on hosted runners.
  • Both tarballs keep ./box and ./resources/ at the root, so -C /opt/box plus the symlink keeps the layout box needs. BOX_ON_PATH turns true, so create and dev for Strands Box templates runs. BOX_DEPLOYS stays false without AGENTCORE_E2E_BOX_CDK.
  • Values from boxRelease.json go only into quoted shell arguments. TAGS goes through env and "$TAGS", not ${{ }}. The workflow runs only on an authorized workflow_call/workflow_dispatch, so I see no new injection path.
  • Branch: 1 behind box-integration (#2871 only, files disjoint, mergeable). The e2e job skips on the PR, so none of these steps has run in CI yet. verify/Analyze on 2c2bb679 were still pending when I posted.

Blocking

  1. The box binary isn't checked against the in-repo pin, and it runs with the E2E AWS role in the environment. The step checks the tarball against SHA256SUMS.txt from the same release. Both releases report immutable: false, so anyone who can edit the release can replace the tarball and SUMS together, and the check still passes. The binary then runs right after Configure AWS credentials. For Linux arm64 the pin already exists: .box.linux.sha256 / .box.linux.asset in boxRelease.json. #2867 deliberately moved the CLI off SHA256SUMS.txt onto that pin. Please verify against boxRelease.json: use .box.linux for Linux arm64, and add a darwin entry (target/asset/sha256, written by scripts/bump-box-release.ts) for macOS. Inline suggestion below.

Non-blocking
2. The Linux arm64 test selection via contains(inputs.tags, 'box') is a substring check. tags: '!box' or 'runtime || box' makes the arm64 job run that whole expression, so it duplicates the x86 Linux job's deploys (inline).
3. The dev path on hosted runners is untested. That covers both macOS and arm64 with the AppArmor sysctl. A failure turns the job red rather than skipping, so please do one authorized workflow_dispatch of this branch (for example tags: box) before merge. On macOS the default "Run all E2E tests" also runs the network-dependent box-egress test (public hosts, retry: 2). Neither the description nor the README mentions it.
4. Nit: sudo tar keeps the archive owners: mquang/staff uid 501 on darwin, uid 18484610 on Linux. Files stay world-readable, so box works, but --no-same-owner would match the Dockerfile fix.
5. Nit: the PR title and description still describe only the macOS runner. They also mention a Box build test, but only the create + dev Box describe runs. The new README sentence runs well past the ~100-column wrap.
6. This is off-plan in the squash plan v2, and it isn't on box-integration-development (e2e-test.yml there has no Install box). Please back-merge so a later phased squash doesn't drop it.

Comment thread .github/workflows/e2e-test.yml Resolved
Comment thread .github/workflows/e2e-test.yml Resolved
Comment thread .github/workflows/e2e-test.yml Resolved
Comment thread .github/workflows/e2e-test.yml Resolved
Comment thread e2eTest/README.md Resolved
Pin the macOS arm64 build in boxRelease.json beside the Linux one, written
by scripts/bump-box-release.ts. The e2e install step checks each archive
against that pin, not against SHA256SUMS.txt of the same release, which
anyone who can edit the release could replace with the archive. It also
extracts with --no-same-owner.
@mquang-amazon mquang-amazon changed the title ci(e2e): Install the pinned box on the macOS runner so the Box tests run ci(e2e): Run the Box e2e tests on macOS and Linux arm64 runners with the pinned box Nov 20, 2026

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 6a487349 (last review was at 2c2bb679). Scope: the PR's own commits c1adceff and 6a487349, plus the conflict resolution in merge c4d53f4b. I did not re-review the base commits it brought in (#2860, #2871). Everything below is read-only. I did not run or download anything.

The earlier blocker is fixed

  • "Install box" now reads repo, version, asset and sha256 from boxRelease.json, using .box.darwin on macOS and .box.linux on Linux arm64. It checks the tarball with echo "${SHA} ${WORK}/${ASSET}" | $SUM -c -, which has the right two-space format, using sha256sum or shasum -a 256. SHA256SUMS.txt is gone from CI, and the CLI still doesn't read it (downloadChecked compares against the pin).
  • Failures stop the step. jq -e exits non-zero on a missing key or null, and under the default bash -e a failed $(...) assignment ends the step. An empty or malformed hash leaves -c with no properly formatted line, so it exits 1. The sha256 regex in BoxReleaseSchema also stops a malformed darwin hash at import, so bun test catches it.
  • I checked the pins against the GitHub asset digests on mquang-amazon/box-releases v0.1.11-preview.10. darwin is 76a3909c...ef57e18 and linux is 45740d3e...c2dc583, and both match. The release is still immutable: false, which is why the in-repo pin matters.
  • bump-box-release.ts writes darwin through the same pin() as linux, and it throws if the release doesn't list the asset. BoxReleaseSchema adds darwin as optional. Zod's default object strips unknown keys anyway, and nothing iterates over the entries, so the CLI can't pick darwin by mistake: resolveBoxLinuxBuild and box-release-test-support only use .linux.
  • Values from the JSON go only into quoted arguments. BUILD comes from the matrix through env. Triggers are unchanged (workflow_call / workflow_dispatch, still behind authorize), and so are the permissions (id-token: write, contents: read).

Merge c4d53f4b: The branch is 0 behind box-integration. Compared with the merge base, the PR touches only its 6 files, all additions to or edits of its own hunks. The pin is still 0.1.11-preview.10 from #2871. No base fix is reverted.

Earlier comments

  1. Blocking, verify against the pin: addressed (see above).
  2. contains(inputs.tags, 'box') substring match: moot. c1adceff dropped the arm64-specific selection, and arm64 now runs the same command as the other jobs.
  3. Untested on hosted runners: still open (non-blocking). The description lists box run on macos-latest / ubuntu-24.04-arm as not tested, and I see no e2e-test.yml dispatch of box/e2e-macos-box. One authorized dispatch would also show whether macos-latest takes the sha256sum or the shasum branch. The box-egress / public-hosts behaviour is now documented in the description and in e2eTest/README.md, so that part is addressed. On OIDC: the macOS job already assumes the E2E role from a GitHub-hosted runner, so the trust policy probably accepts ubuntu-24.04-arm too, unless it has a runner-specific condition.
  4. --no-same-owner: addressed.
  5. Title, description and README wrap: addressed. The title and description cover both runners, the PR no longer claims a Box build test, and the new lines are within the wrap width.
  6. Squash plan / back-merge: acknowledged. The description's follow-up is to merge box-integration back into box-integration-development, which doesn't have it yet. That's fine after merge.

New findings (all non-blocking)

  • linuxOnly now also runs Linux-arm64 (inline). canary.yml calls this workflow with linuxOnly: true, tags: canary, so after this reaches the default branch, every canary run deploys twice and its heartbeat also depends on a hosted arm64 runner.
  • Running the whole suite on arm64 doubles the x86 Linux job's deploys on every push to the branch, with 4 jobs in parallel in one account instead of 3. Names are E2E_PREFIX + Date.now() base36 (plus a template index), the same scheme the 3 existing parallel jobs already share, and the stale-stack sweep only removes stacks older than 2h. So I don't expect collisions, but watch the account's Runtime/Gateway quotas. The description says this is deliberate.
  • The AppArmor sysctl is scoped to matrix.name == 'Linux-arm64' on a throwaway hosted VM. That's fine.
  • The L3 enableApplicationLogs gap (aws/agentcore-l3-cdk-constructs#470) doesn't turn this red. The Box deploy tests still skip without AGENTCORE_E2E_BOX_CDK, and only Box gateways write that field.
  • Test nit (inline): nothing asserts the darwin pin's shape against the version the way the linux one is checked.
  • bump-box-release.ts now refuses a release with no macOS build. That's probably what you want, since CI needs it. It's just a new coupling to be aware of for any Linux-only preview.

CI: verify / Linux, verify / Windows, Analyze, CodeQL and the CodeBuild check pass. verify / macOS was still pending when I posted, and e2e skips on PRs as expected. The branch is mergeable and 0 behind base.

Thanks, the blocker is fixed cleanly.

Comment thread .github/workflows/e2e-test.yml Resolved
Comment thread src/core/project/backends/boxRelease.ts Resolved
…ox pin

canary.yml runs this workflow with linuxOnly, so a Linux arm64 job there
doubled each canary's deploys and made its heartbeat wait on a hosted arm64
runner. The pinned releases test now checks the darwin entry that the e2e
workflow installs from, so a stale pin fails in bun test.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of the delta 6a48734...bc71015 (one commit: the matrix expression in e2e-test.yml and 5 lines in boxRelease.test.ts).

Now addressed from my last review:

  • Canary runs twice: fixed. linuxOnly is declared type: boolean under both workflow_call and workflow_dispatch, and those are the only two triggers. So inputs.linuxOnly && '["Linux"]' || '[...all four]' gives only Linux (the x86_64 CodeBuild runner) when it is true. canary.yml (linuxOnly: true, tags: canary) now deploys once. When it is false or unset, the matrix is unchanged: Linux, Linux-arm64, Windows and macOS. The runs-on mapping and the per-OS step if:s did not change, so macOS and Windows behave as before.
  • No unit test for the darwin pin: fixed. The test now checks BOX_RELEASE.darwin?.asset against box-v${version}-aarch64-apple-darwin.tar.gz and checks that darwin?.sha256 is 64 hex characters. If the darwin entry is missing, undefined will not match, so the test fails. A wrong version or target also fails it. The repo is shared across platforms and is already covered by the BOX_RELEASE_PAGE assertion. The test only checks constants, with no network, file paths, file modes or subprocesses, so it is safe on all CI runners.

No new findings. CI on the new head: CodeQL, Analyze and verify / Windows pass. verify / Linux and verify / macOS were still running when I posted this.

The hosted macOS arm64 runner has no nested virtualization, so it
cannot run a container tool, and agentcore dev of the box container
template fails there. The Linux arm64 runner still runs it.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of the delta since bc71015 (887297f, skip the Box container dev test on CI macOS). The change looks right. I'm leaving this as a comment rather than an approval for two reasons. verify / Linux, macOS and Windows are still pending at 887297f. The branch is also 1 commit behind box-integration (e2ac9d7, docs-only, touching docs/box-local-testing.md; no conflict). Please update the branch once verify goes green.

The skip is narrow and matches the README:

  • It filters only agent-python-strands-box-container, and only out of the sequential create and dev describe. The deploy describe still runs the container case.
  • It applies only when CI === "true" and the platform is not Linux. Local macOS runs and the Linux arm64 job still run the test.
  • The reason is the real one. Dispatch run 44820634395 attempt 2 (at bc71015) failed on macOS only, on this test, with 'container runtime' was not found on your PATH. The other 6 Box dev tests passed. Linux-arm64 ran agent_python_strands_box_container and passed (7 passed). Linux and Windows also passed.

That dispatch's ref is the sha bc71015. The queued attempt-3 rerun of macOS will therefore check out bc71015 again and fail the same way, so it won't validate this fix. A new dispatch at 887297f would.

Non-blocking: see the inline note about using a capability check. The earlier open items still stand: the back-merge to box-integration-development, the full suite on arm64 doubling deploys and quotas, and darwin being optional in the schema.

Comment thread e2eTest/project/templates.test.ts Outdated
const cli = new CliRunner();
// The container template's dev needs a container tool, which a CI macOS runner cannot run
// (no nested virtualization), as py_strands_container notes. Linux arm64 runs it.
const noLocalContainers = process.env.CI === "true" && process.platform !== "linux";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this mirrors py_strands_container's CI && !linux check, which is consistent. A capability check would be more precise, for example skipping when none of docker, podman or finch is on PATH (the same tools agentcore dev looks for). That would also cover a local Mac with no container tool, and it would start running the test automatically if a runner gains one. Note too that !== "linux" also matches Windows, which the comment and README don't mention. It's harmless, since box isn't installed on Windows and the describe skips there anyway.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5049104: the container case skips when none of docker, podman or finch runs from PATH, the tools agentcore dev looks for, instead of CI && !linux. Checked with vitest list: 6 dev cases with no tool on PATH, 7 with one. The README says the same.

…s from PATH

Addresses the review of #2872: a capability check in place of CI && !linux.
It also covers a local machine with no container tool, says nothing about
Windows by accident, and runs the test once a runner gains one.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 5049104. The new capability check looks good. spawnSync from node:child_process returns status: null and does not throw when a binary is missing, so the test file can't crash. It checks the same docker/podman/finch --version order that resolveContainerTool in src/core/dev/container.ts uses. The skip only covers agent-python-strands-box-container, so Linux x86/arm64 runners and a local Mac with Docker still run it. This also fixes the macOS failure in the e2e dispatch at bc71015 (run 44820634395, 'container runtime' was not found on your PATH).

There's no e2e dispatch at this head yet, so it's worth running one before merge. verify / Linux and verify / Windows were still pending when I posted, with no failures so far. Please merge only after they pass.

Non-blocking, still open: back-merge to box-integration-development, the full suite on arm64 doubling deploys and quota use, and darwin being optional in the schema.

Comment thread e2eTest/README.md
template a container tool. Box listens on the protocol's fixed port (8080, 8000 or 9000), so the
`dev` tests run one at a time and need those ports free:
template a container tool. The container template's `dev` test skips when no `docker`, `podman`
or `finch` runs from PATH, as on the CI macOS runner; the Linux arm64 runner runs it. Box listens on the protocol's fixed port (8080,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this line runs to 133 columns while the rest of the paragraph wraps at 100. Could you rewrap it? One caveat: docker --version passes even when the engine is down, so on a Mac where Docker Desktop is installed but stopped, this test runs and fails instead of skipping. That's probably fine for a local run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants