Skip to content

fix(box): Grant the standard library of the Runtime Python on CodeZip, not its prefix - #2882

Closed
SaiAbhishek98 wants to merge 1 commit into
aws:box-integrationfrom
SaiAbhishek98:sa/codezip-python-grants
Closed

SaiAbhishek98 wants to merge 1 commit into
aws:box-integrationfrom
SaiAbhishek98:sa/codezip-python-grants

Conversation

@SaiAbhishek98

Copy link
Copy Markdown

Description

Problem

A CodeZip box runtime does not start on AgentCore Runtime. The deploy stops with CREATE_FAILED
(The runtime process exited unexpectedly), and box gives this error:

[agent] filesystem entry "/usr" is refused: ... a system root is never a grantable tree

A control run on 2026-11-20 gave the same result with each value of instrumentation.enableOtel.
A container runtime of the same project starts.

The Runtime profile moves the read grant and the exec grant of the uv Pythons of the local profile
to ${{ inputs.python.prefix }}. start.py sets this input to sys.base_prefix. On the Runtime,
the value is /usr. Box Core refuses a grant of /usr as a system root, and each Box Core release
refuses it.

Thus box did not start on a CodeZip Runtime before this change:

  • aws/agentcore-cli#2789 added the CodeZip
    Runtime profile with the prefix grant. When box did not start, start.sh started the agent with no
    box.
  • 4700b621
    (in aws/agentcore-cli#2850) and
    aws/agentcore-cli#2853 removed this fallback.
    Since then, the Runtime stops the runtime.
  • aws/agentcore-cli#2866 moved the grant into
    the derivation, with the same value. Its fixture of the renderer that it replaced also has the prefix.
  • The removal of the ffi shim did not change the Python or its grants.

The branch box-integration-development has the same problem
(435b6ba1, with box 0.1.11-preview.10).
This PR does not change that branch.

The necessary grants for the Runtime Python

A probe deploy to a PYTHON_3_14 CodeZip Runtime gave these facts:

  • The Python is /usr/bin/python3.14, and its sys.base_prefix is /usr. Its libpython is
    static.
  • The standard library is /usr/lib/python3.14, with the native modules in lib-dynload.
  • The native modules use libraries in /usr/lib64, for example libssl.so.3 and libffi.so.8.

box mounts a read grant that is not a loader folder with no exec. Thus a read grant of the
standard library is not sufficient: on the Runtime, import socket stops with
failed to map segment from shared object.

The change

With this change, the uv Pythons of the local profile go to the standard library of the Runtime
Python, ${{ inputs.python.stdlib }}, and not to its prefix.

  • start.py exports BOX_PYTHON_STDLIB (sysconfig.get_path("stdlib")). It does not export
    BOX_PYTHON_PREFIX.
  • The @box/agentcore-codezip pack has the input python.stdlib. It does not have the input
    python.prefix.
  • The paths rule moves ${{ home }}/.local/share/uv/python to ${{ inputs.python.stdlib }}.

After this change, the Runtime profile of each CodeZip template has these grants:

read = [ "${{ inputs.code.dir }}", "${{ inputs.python.stdlib }}", "/usr/lib64", "/lib64" ]
exec = [ "${{ inputs.code.dir }}", "${{ inputs.python.stdlib }}" ]

The Python itself is the command of the agent, and box gives it exec.

Decisions

  • The standard library, and not the loader folder /usr/lib. A read grant of /usr/lib also
    lets the Python start, and an exec grant is not necessary. But box makes a loader folder
    executable, thus that grant gives read and exec on all of /usr/lib. The grant of the standard
    library gives less access.
  • start.py gives the path. agentcore build can also make the path from runtimeVersion, for example
    ${{ inputs.python.prefix }}/lib/python3.14. That path is correct only for the lib layout of
    this Runtime. The value of sysconfig is the path of the Python that operates, also for a
    lib64 layout.
  • The input python.prefix goes. The prefix grant was its only use.
  • /usr/lib64 and /lib64 stay. On the Runtime, /lib64 is a link to /usr/lib64, and box
    accepts the two grants. This change does not change them.

Tests that CI cannot do

  • A local test in an Amazon Linux 2023 arm64 container, with box 0.1.11-preview.10 and a Python
    3.14.3 that has the paths of the Runtime Python. The zip of box-integration stops with the
    /usr refusal. The zip of this change starts the agent, and /ping gives 200. The zip of
    box-integration-development stops with the /usr refusal.
  • A live test in a dev account. We deploy a new CodeZip box project of this branch, with no
    change. box shows the grants read /usr/lib/python3.14 and exec /usr/lib/python3.14. Three
    invocations get status 200. The log group -DEFAULT of the runtime has 54 box records: 48
    permit decisions and the record box_started. The test removes all its resources.

Known problems and subsequent work

  • A CodeZip box runtime that agentcore create made before this change has a start.py that exports
    the prefix. With this CLI, box stops on the Runtime, because the input python.stdlib has no
    value. Put a copy of the start.py of this change into the code folder. agentcore build does not examine
    start.py. A subsequent change can make agentcore build stop for a start.py that exports the
    prefix.
  • This PR does not change box-integration-development. A different PR must apply this change to
    that branch.
  • boxProfileResolver.test.ts shows an error when box 0.1.11-preview.10 is on PATH. This error is
    also on box-integration without this change.

Related Issue

No issue is open for this problem.

Documentation PR

Not applicable. No document names the input python.prefix.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe):

Testing

How have you tested the change?

  • I ran bun test: 5835 pass, 6 skip, 2 fail. This change does not cause the two
    errors. boxProfileResolver.test.ts also shows its error on box-integration.
    remove/index.test.ts stops at its timeout only when all tests operate together.
  • I ran the relevant end-to-end tests with bun run test:e2e, or explained why they are not
    applicable (CI does not run them for you). Not done. The live test above is on AgentCore
    Runtime.
  • I ran bun run typecheck
  • I ran bun run lint:check
  • I ran bun run format:check
  • I ran bun run audit
  • I ran bun run secrets:check
  • I ran bun run build
  • If I modified src/assets/, I updated affected snapshots with bun test <test-file> --update-snapshots and committed them. No snapshot has start.py.
  • If I re-recorded fixtures or golden files, every change beyond account identifiers is explained
    by this PR. In old-renderer-tables.json, only the 204 values of the prefix change.
  • This PR does not change command.md (the release workflow regenerates it)

Each new or changed test shows an error on the code before this change (115 tests). Five
mutations of the code each cause an error in these tests.

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

…, not its prefix

## Problem

A CodeZip box runtime does not start on AgentCore Runtime. box stops with this error:

`[agent] filesystem entry "/usr" is refused: ... a system root is never a grantable tree`

The Runtime profile moves the read grant and the exec grant of the uv Pythons of the local profile
to `${{ inputs.python.prefix }}`. start.py sets this input to `sys.base_prefix`. The Python of a
PYTHON_3_14 Runtime is `/usr/bin/python3.14`. Thus the value is `/usr`. Box Core refuses a grant of
`/usr`, and each Box Core release refuses it. Thus box did not start on a CodeZip Runtime before
this change. Before
[4700b62](aws@4700b62),
start.sh started the agent with no box when box did not start. That commit and
[aws#2853](aws#2853) removed this fallback. At
this time, the Runtime stops the runtime, and the deploy stops with an error. The branch
box-integration-development has the same problem.

## Solution

With this change, the uv Pythons of the local profile go to the standard library of the Runtime
Python, `${{ inputs.python.stdlib }}`, and not to its prefix.

- start.py exports `BOX_PYTHON_STDLIB`, the value of `sysconfig.get_path("stdlib")`, in place of
  `BOX_PYTHON_PREFIX`. On a PYTHON_3_14 Runtime, the value is `/usr/lib/python3.14`.
- The `@box/agentcore-codezip` pack has the input `python.stdlib` in place of `python.prefix`, with
  the same pattern.
- The paths rule moves `${{ home }}/.local/share/uv/python` to `${{ inputs.python.stdlib }}`.

After this change, the Runtime profile of each CodeZip template has these grants:

- `read = [code folder, standard library, "/usr/lib64", "/lib64"]`. The Python reads its modules.
  `/usr/lib64` contains the libraries that its native modules use, for example `libssl.so.3`.
- `exec = [code folder, standard library]`. box mounts a folder that is not a loader folder with
  no exec. Thus the Python can use the native modules in `lib-dynload` only with this grant.
- The Python itself is the command of the agent, and box gives it exec.

A read grant of the loader folder `/usr/lib` also lets the Python start. This change does not use
it, because it gives read and exec on all of `/usr/lib`, and not only on the standard library.

A CodeZip box runtime that `agentcore create` made before this change has a start.py that exports
the prefix. Copy the start.py of this change into its code folder.

## Tests

- `start.py > gives box the standard library of the Runtime's Python, and not its base prefix`
  (`boxStartPy.test.ts`) starts `main()` of start.py, and replaces `os.execve` with a function that
  records the environment. It makes sure that `BOX_PYTHON_STDLIB` is the standard library, and that
  no value is the prefix.
- `the Runtime profile of each CodeZip box template of create` (`additions.test.ts`) derives the
  Runtime profile of the six CodeZip box templates. It makes sure that the read grant and the exec
  grant of the standard library are in the profile. It also makes sure that no grant has a value
  on the Runtime that Box Core refuses: a system root in a list, or a loader folder in a list that
  is not `read`.
- The pack test makes sure that the pack has the input `python.stdlib`. The paths test and the
  test of the recorded profiles in `old-renderer-tables.json` use the standard library. In that
  file, only these 204 values change.
- On the code before this change, 115 of the new and changed tests show an error.
- Five mutations of the code, and the errors that each mutation causes:
  - start.py exports the prefix again: the start.py test shows an error.
  - The paths rule moves the uv Pythons to the prefix again: 111 tests show an error.
  - The pack has `python.prefix` again: 2 pack tests show an error.
  - `runtimeCodeZipRunner` gives no exec grant of the uv Pythons: 110 tests show an error.
  - `runtimeCodeZipRunner` also gives a grant of the prefix: the six template tests show an error
    at the check of the refused grants.
- `bun test`: `5835 pass`, `6 skip`, `2 fail`. This change does not cause the two errors.
  `boxProfileResolver.test.ts` also shows the error on box-integration with the same box
  (0.1.11-preview.10) on PATH. `remove/index.test.ts` stops at its 10 s timeout when all tests
  operate together. When it operates with no other test, it shows no error.
- `bun run typecheck`, `bun run lint:check`, `bun run format:check`, `bun run build`,
  `bun run audit`, and `bun run secrets:check`: no errors.
- A local test in an Amazon Linux 2023 arm64 container, with box 0.1.11-preview.10 and a Python
  3.14.3 that has the prefix `/usr` and the standard library `/usr/lib/python3.14`. The Python of
  the Runtime has the same paths. The results:
  - The zip of box-integration stops with the `/usr` refusal.
  - The zip of this change starts the agent, and `/ping` gives 200.
  - The zip of box-integration-development stops with the `/usr` refusal.
- A live test on AgentCore Runtime in a dev account. In the test, we deploy a new CodeZip box
  project of this branch, with no change. Three invocations get status 200, and the box records
  are in the log group `-DEFAULT` of the runtime. The test removes all its resources.

We did not do `bun run test:e2e`. This change has no test on the CodeZip Runtimes PYTHON_3_10 to
PYTHON_3_13, and no test of an `[mcp]` server or a `[tool]` on the Runtime Python.
@SaiAbhishek98
SaiAbhishek98 requested review from a team, leo237 and mquang-amazon November 20, 2026 22:50
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.79%. Comparing base (e2ac9d7) to head (490c197).

Additional details and impacted files
@@               Coverage Diff                @@
##           box-integration    #2882   +/-   ##
================================================
  Coverage            96.79%   96.79%           
================================================
  Files                  766      766           
  Lines                66803    66803           
================================================
  Hits                 64665    64665           
  Misses                2138     2138           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@leo237 leo237 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I found no blocking issues. The change is small, it fixes the actual cause, and it has a regression test for the exact failure.

What I checked (head 490c1972, merge-base = current box-integration e2ac9d78, so not behind base)

  • The /usr refusal. start.py now exports BOX_PYTHON_STDLIB = sysconfig.get_path("stdlib") and no longer exports BOX_PYTHON_PREFIX. The pack declares inputs.python.stdlib with ABSOLUTE_PATH_PATTERN, and the paths rule moves the uv Pythons folder to it. The derived CodeZip grants are read = [code, stdlib, /usr/lib64, /lib64] and exec = [code, stdlib].
  • Regression test. The new additions.test.ts case resolves every grant of every CodeZip template against the measured Runtime values (/usr, /usr/lib/python3.14, /var/task). It asserts that no grant is a Box Core system root and that no loader folder appears outside read. It also checks that stdlib is in both read and exec, which covers the failed to map segment problem in lib-dynload. With the old prefix grant this test fails, which is what we want.
  • /usr/lib64 (libssl, libffi). The grant was already there as a read-only loader folder and is unchanged. The PR body reports a live CodeZip deploy where box started, gave 3x 200 and logged box_started, plus an AL2023 arm64 container repro.
  • Fixture. All 204 changed lines in old-renderer-tables.json are a pure python.prefix to python.stdlib substitution. Every removed line pairs with an added line once that string is replaced.
  • Leftover references. None. Outside tests and the fixture, nothing in the repo mentions python.prefix, BOX_PYTHON_PREFIX, pythonPrefix or base_prefix.
  • Test hygiene. The start.py harness test skips without Python (describe.skipIf(PYTHON === undefined), and the spawnSync probe is guarded with try/catch). It stubs os.execve and points all folders at a mkdtemp folder. It builds no host paths by hand.
  • Must-survive fixes. The diff touches only these 12 files, and the #2866 derivation, .venv/bin/python, boxStartFolder, the ffi shim removal, BOX_OTHER_MACHINE_HINT and the preview.10 pin are all untouched.
  • CI. Linux, Windows, CodeQL/Analyze and CodeBuild e2e pass. verify / macOS was still pending when I posted this.

Non-blocking

  1. Back-merge to box-integration-development. That branch still has "BOX_PYTHON_PREFIX": sys.base_prefix in start.py and inputs.python.prefix in its fixture. A later squash from dev would bring the /usr refusal back, so it is worth tracking the follow-up PR explicitly.
  2. Old start.py in existing projects. As the body says, a CodeZip project created before this change still exports only BOX_PYTHON_PREFIX, so box fails with an unresolved python.stdlib input. That failure is loud, and those projects never started on the Runtime anyway, so nothing that works today breaks. The suggested follow-up (build rejects a start.py that exports the prefix, or prints how to fix it) would still be good.
  3. Runtime versions other than PYTHON_3_14 were not probed. start.py still supports 3.10 (the tomllib fallback). If any Runtime Python uses a lib64 layout, the stdlib grant would be /usr/lib64/python3.X. That puts an exec grant under a loader folder and nested inside the /usr/lib64 read grant. I do not know whether Box Core accepts that. The Decisions section says sysconfig makes the lib64 layout work, but that case has not been run, so the claim is untested.

"BOX_CODE": CODE,
"BOX_PYTHON": box_python(sys.executable, sys.version_info),
"BOX_PYTHON_PREFIX": sys.base_prefix,
"BOX_PYTHON_STDLIB": sysconfig.get_path("stdlib"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this is right for the measured PYTHON_3_14 layout (/usr/lib/python3.14). On a lib64-layout Python (Fedora/RHEL style), stdlib becomes /usr/lib64/python3.X, which sits inside the /usr/lib64 loader-folder read grant and also lands in exec. Has a Runtime version other than 3.14 been probed, or do we know Box Core accepts exec on a subfolder of a loader folder? If not, a sentence in the PR body or a comment would set expectations.

runtimeCode: WORKSPACE,
runtimePython: `${WORKSPACE}/.venv/bin/python`,
runtimePythonPrefix: "/usr/local",
runtimePythonStdlib: "/usr/local",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: for a container, this field is the identity mapping of /usr/local (the local Pythons folder). It is not a standard library. With the rename to runtimePythonStdlib, a reader might think the container gets a stdlib grant. A short comment here, or keeping a neutral name such as runtimePythons, would avoid that.

* {@link AGENTCORE_CODEZIP_PACK_PATH}.
*/
export function renderAgentCoreCodeZipPack(): string {
return `# The CodeZip Runtime's Python and code directory, read from the environment start.py exports.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the pack's header comment still says "The CodeZip Runtime's Python and code directory". It could mention the standard library too, to match the new inputs.python.stdlib and the updated JSDoc above.

@SaiAbhishek98

Copy link
Copy Markdown
Author

I close this PR, because CodeZip box runtimes are not in the scope of our work. Our work is OTEL for box runtimes.

The problem that this PR found stays open: on box-integration, a CodeZip box runtime does not start on AgentCore Runtime. Box refuses the grant of ${{ inputs.python.prefix }}, which start.py sets to /usr on the Runtime, with "a system root is never a grantable tree". The branch of this PR has a fix that grants only the standard library of the Runtime Python, and a live test of that fix. The owners of CodeZip in a box can use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants