Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions docs/box-local-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,12 @@ A profile that names a pack `box/<name>` fails: box refuses it and names the `@b
`[tool.<name>]`;
- the `@box/agentcore` packs, the project name, and the build region come in, and each CloudWatch
target signs with the execution role.
- with `instrumentation.enableOtel` on, which is the default for a box template too, a command
of the Python of the venv starts with `opentelemetry-instrument`. On CodeZip, the
Python of the Runtime runs the script in the zip, `<code>/bin/opentelemetry-instrument`. The
`@box/agentcore` pack gives the agent the ADOT variables and the `OTEL_RESOURCE_ATTRIBUTES`
that the Runtime sets. When your `[agent.env]` sets one of these variables, box uses your value.
The pack sets no `OTEL_EXPORTER_OTLP_*` variable, because box sets each one.

**The default log target.** With no `cloudwatch_logs` target, the Runtime profile gets
`[telemetry.agentcore]`: box sends its deny, permit, and trace records to the log group of the
Expand All @@ -250,9 +256,19 @@ A profile that names a pack `box/<name>` fails: box refuses it and names the `@b
Runtime does not set it, box does not start. The Runtime makes that log group, so its retention
applies, and it stays after you remove the runtime. CloudWatch Logs charges for the records it
gets. To send none, set `DISABLE_ADOT_OBSERVABILITY` to `"true"` in the `envVars` of the runtime
in `agentcore.json`. To send them to another log group, add a `cloudwatch_logs` target to the
local profile. `agentcore create` and `agentcore add runtime` give this notice for each box
runtime.
in `agentcore.json`. Then the Runtime profile also gets no default X-Ray target. The agent does
not start with `opentelemetry-instrument`, and the `@box/agentcore` pack gives it no ADOT
variable. To send the records to another log group, add a `cloudwatch_logs` target to the local
profile. `agentcore create` and `agentcore add runtime` give this notice for each box runtime.

**The default X-Ray target.** With `enableOtel` on and no `cloudwatch_traces` target, the
Runtime profile gets `[telemetry.agentcore_traces]`: box sends the spans of the agent, and its
own deny, permit, and trace spans, to X-Ray in the region of the Runtime. Build gives the
execution role `xray:PutSpans` and `xray:PutTraceSegments` in
`agentcore/box-telemetry-policy-<runtime>.json`. To send the spans to a different target, add a
`cloudwatch_traces` target to the local profile. To send no spans, set
`instrumentation.enableOtel` to `false` in `agentcore.json`. When `enableOtel` is on and the
agent does not start with `opentelemetry-instrument`, build gives a warning.

Build stops, and names the key and its line, for a value that cannot work on the Runtime: a
required grant or a command path in the project outside the code folder, in the venv, or in a
Expand Down Expand Up @@ -466,6 +482,11 @@ to the Runtime profile. If the profile has a target with the name `agentcore`, t
`AWS_REGION`, and its log group and stream from `OTEL_EXPORTER_OTLP_LOGS_HEADERS`. The target sends
the deny, permit, and trace records, and box signs it with the execution role.

With `instrumentation.enableOtel` on, box also sends the spans of the agent and its own spans to
X-Ray, through the target `[telemetry.agentcore_traces]`. Build gives the execution role the X-Ray
grant for it. "The default X-Ray target" in step 4 gives the rules. CloudWatch Transaction Search
keeps the spans in the log group `aws/spans`.

The CLI and the L3 constructs make no log group for box. The role that the CDK makes for the
runtime can write to the log group of the Runtime, thus build adds no grant for it. A runtime that
names an `executionRoleArn` must have `logs:CreateLogStream` and `logs:PutLogEvents` on
Expand Down
2 changes: 1 addition & 1 deletion src/core/project/backends/boxCodeZip.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ describe("stageBoxCodeZip", () => {
// The profile imports these generated packs, which box finds beside its binary. They are
// written after box's own resources are copied, so the copy does not remove them.
expect(await readFile(join(dist, AGENTCORE_PACK_PATH), "utf8")).toBe(
renderAgentCorePack(demo.spec),
renderAgentCorePack(demo.spec, demo.spec.runtimes[0]!),
);
expect(await readFile(join(dist, AGENTCORE_CODEZIP_PACK_PATH), "utf8")).toBe(
renderAgentCoreCodeZipPack(),
Expand Down
4 changes: 3 additions & 1 deletion src/core/project/backends/boxContainer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,9 @@ describe("stageBoxContainers", () => {
`BOX_SHA256=${BOX_RELEASE.linux.sha256}\n`,
);
// The container also stages the @box/agentcore pack, beside box's own packs in the image.
expect(await readFile(join(dist, PACK_PATH), "utf8")).toBe(renderAgentCorePack(demo.spec));
expect(await readFile(join(dist, PACK_PATH), "utf8")).toBe(
renderAgentCorePack(demo.spec, demo.spec.runtimes[0]!),
);
expect((await readdir(dist)).sort()).toEqual([...STAGED, "resources"].sort());
// The runtime's .box/agent/aws/ holds a copy of what the profile writer stages, and no box.
const aws = join(demo.rootPath, ".box", "agent", "aws");
Expand Down
145 changes: 115 additions & 30 deletions src/core/project/backends/cdk.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,10 @@ function unusedIdentity(): CredentialProviderCalls {
const NO_BOX_NETWORKS = { boxNetworks: [] };

const OTEL_WARNING =
"Runtime 'agent' runs its agent in a box, where the agent does not run under " +
"opentelemetry-instrument, so it sends no traces. Set instrumentation.enableOtel to false.";
"Runtime 'agent' runs its agent in a box, and [agent] command in " +
".box/agent/local/profile.toml does not run the venv's Python, so on AgentCore Runtime the " +
"agent does not start under opentelemetry-instrument and sends no traces. Run the agent with " +
"the venv's Python, or set instrumentation.enableOtel to false.";

function deployInput(overrides: Partial<DeployBackendInput> = {}): DeployBackendInput {
return { target: TARGET, confirmTeardown: async () => false, boxNetworks: [], ...overrides };
Expand Down Expand Up @@ -332,32 +334,121 @@ describe("CdkBackend.build", () => {
expect(subject.commands).toEqual([{ command: synthCommand(input), cwd: cdkDirectory(input) }]);
});

/** The Box network model of the runtime `agent`, with its derived Runtime profile. */
const boxNetwork = (runtimeProfile: Record<string, unknown>) => ({
runtime: "agent",
gateway: "agent-box-egress",
region: "us-west-2",
hosts: [],
envRoutes: [],
policyFiles: [],
runtimeProfile,
grantedActions: [],
warnings: [],
failures: [],
});
const PLAIN = ["${{ workspace }}/.venv/bin/python", "-m", "main"];
const UNDER_OTEL: Record<string, string[]> = {
Container: ["${{ workspace }}/.venv/bin/opentelemetry-instrument", ...PLAIN],
CodeZip: [
"${{ inputs.python.path }}",
"${{ inputs.code.dir }}/bin/opentelemetry-instrument",
"${{ inputs.python.path }}",
"-m",
"main",
],
};

// A CodeZip box build downloads box before it warns, so the cases check the warning itself.
test.each([
["a box container", "box", "Container", true, true],
["a box CodeZip runtime", "box", "CodeZip", true, true],
["a box container with enableOtel false", "box", "Container", false, false],
["a container outside a Box project", undefined, "Container", true, false],
] as const)("warns that %s sends no traces: %p", (_label, projectType, build, otel, warns) => {
[
"a box container that starts under opentelemetry-instrument",
"box",
"Container",
true,
true,
false,
],
[
"a box CodeZip runtime that starts under opentelemetry-instrument",
"box",
"CodeZip",
true,
true,
false,
],
["a box container that does not", "box", "Container", true, false, true],
["a box CodeZip runtime that does not", "box", "CodeZip", true, false, true],
["a box container with enableOtel false", "box", "Container", false, false, false],
["a container outside a Box project", undefined, "Container", true, false, false],
] as const)(
"warns that %s sends no traces: %p",
(_label, projectType, build, otel, started, warns) => {
const spec = ProjectSpecSchema.parse({
name: "example",
version: 2,
...(projectType && { projectType }),
runtimes: [
{
name: "agent",
build,
...(build === "CodeZip" && { runtimeVersion: "PYTHON_3_14" }),
entrypoint: "main.py",
codeLocation: "app/agent",
...(projectType && { box: { version: BOX_RELEASE.version } }),
...(!otel && { instrumentation: { enableOtel: false } }),
},
],
agentCoreGateways: projectType ? [boxGatewaySpec("agent")] : [],
});

const command = started ? UNDER_OTEL[build]! : PLAIN;

expect(boxOtelWarnings(spec, [boxNetwork({ agent: { command } })])).toEqual(
warns ? [OTEL_WARNING] : [],
);
},
);

test("gives no OTEL warning for a box runtime whose envVars set DISABLE_ADOT_OBSERVABILITY to true", () => {
const spec = ProjectSpecSchema.parse({
name: "example",
version: 2,
projectType: "box",
runtimes: [
{
name: "agent",
build: "Container",
entrypoint: "main.py",
codeLocation: "app/agent",
envVars: [{ name: "DISABLE_ADOT_OBSERVABILITY", value: "true" }],
box: { version: BOX_RELEASE.version },
},
],
agentCoreGateways: [boxGatewaySpec("agent")],
});

expect(boxOtelWarnings(spec, [boxNetwork({ agent: { command: PLAIN } })])).toEqual([]);
});

test("gives no OTEL warning for a box runtime that has no Box network model", () => {
const spec = ProjectSpecSchema.parse({
name: "example",
version: 2,
...(projectType && { projectType }),
projectType: "box",
runtimes: [
{
name: "agent",
build,
...(build === "CodeZip" && { runtimeVersion: "PYTHON_3_14" }),
build: "Container",
entrypoint: "main.py",
codeLocation: "app/agent",
...(projectType && { box: { version: BOX_RELEASE.version } }),
...(!otel && { instrumentation: { enableOtel: false } }),
box: { version: BOX_RELEASE.version },
},
],
agentCoreGateways: projectType ? [boxGatewaySpec("agent")] : [],
agentCoreGateways: [boxGatewaySpec("agent")],
});

expect(boxOtelWarnings(spec)).toEqual(warns ? [OTEL_WARNING] : []);
expect(boxOtelWarnings(spec, [])).toEqual([]);
});

test("build yields the OTEL warning of a box runtime", async () => {
Expand All @@ -381,23 +472,17 @@ describe("CdkBackend.build", () => {
await writeFile(join(input.rootPath, "app", "agent", "Dockerfile"), "FROM scratch\n");

// Build stages box from the runtime's Box network model, with its derived Runtime profile.
const boxNetworks = [
{
runtime: "agent",
gateway: "agent-box-egress",
region: "us-west-2",
hosts: [],
envRoutes: [],
policyFiles: [],
runtimeProfile: { packs: ["@box/base", "@box/agentcore"] },
grantedActions: [],
warnings: [],
failures: [],
},
];
const events = await collect(harness().backend.build(input, { boxNetworks }));
const plain = { packs: ["@box/base", "@box/agentcore"], agent: { command: PLAIN } };
const started = { ...plain, agent: { command: UNDER_OTEL.Container } };
const warned = await collect(
harness().backend.build(input, { boxNetworks: [boxNetwork(plain)] }),
);
const quiet = await collect(
harness().backend.build(input, { boxNetworks: [boxNetwork(started)] }),
);

expect(events).toContainEqual({ type: "warning", message: OTEL_WARNING } as ProjectEvent);
expect(warned).toContainEqual({ type: "warning", message: OTEL_WARNING } as ProjectEvent);
expect(quiet).not.toContainEqual({ type: "warning", message: OTEL_WARNING } as ProjectEvent);
});

test("warns but continues when the installed CDK is legacy", async () => {
Expand Down
33 changes: 23 additions & 10 deletions src/core/project/backends/cdk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ import type {
} from "../../../handlers/project/types";
import { stageBoxCodeZip } from "./boxCodeZip";
import { stageBoxContainers } from "./boxContainer";
import { runtimeRunsInBox } from "../templates/boxProfile";
import { boxLocalProfileFile, otelOn, runtimeRunsInBox } from "../templates/boxProfile";
import { agentStartsUnderOtel } from "../runtimeProfile/rules/additions";
import { boxDownloadMessage, boxReleaseFor, resolveBoxLinuxBuild } from "./boxRelease";
import { cdkCompatibilityWarning } from "./cdk/compatibility";
import {
Expand Down Expand Up @@ -160,18 +161,28 @@ export type CdkBackendConfig = {
};

/**
* A warning for each runtime of `spec` that runs its agent in a box with OTEL on: the agent in the
* box does not run under opentelemetry-instrument, CodeZip or container, so it sends no traces.
* A warning for each runtime of `spec` that runs its agent in a box with OTEL on, and whose
* AgentCore Runtime profile in `boxNetworks` does not start the agent under opentelemetry-instrument:
* build starts only a command of the venv's Python under it, and the agent then sends no traces.
*/
export function boxOtelWarnings(spec: Project["spec"]): string[] {
export function boxOtelWarnings(
spec: Project["spec"],
boxNetworks: BuildBackendInput["boxNetworks"],
): string[] {
return spec.runtimes
.filter(
(runtime) => runtimeRunsInBox(spec, runtime) && (runtime.instrumentation?.enableOtel ?? true),
)
.filter((runtime) => runtimeRunsInBox(spec, runtime) && otelOn(runtime))
.filter((runtime) => {
const profile = boxNetworks.find(
(network) => network.runtime === runtime.name,
)?.runtimeProfile;
return profile !== undefined && !agentStartsUnderOtel(profile);
})
.map(
(runtime) =>
`Runtime '${runtime.name}' runs its agent in a box, where the agent does not run under ` +
"opentelemetry-instrument, so it sends no traces. Set instrumentation.enableOtel to false.",
`Runtime '${runtime.name}' runs its agent in a box, and [agent] command in ` +
`${boxLocalProfileFile(runtime.name)} does not run the venv's Python, so on AgentCore ` +
"Runtime the agent does not start under opentelemetry-instrument and sends no traces. Run " +
"the agent with the venv's Python, or set instrumentation.enableOtel to false.",
);
}

Expand Down Expand Up @@ -267,7 +278,9 @@ export class CdkBackend implements ProjectBackend {
): AsyncGenerator<ProjectEvent, void> {
yield* await this.ensureCdkDependencies(project, boxNetworks);

for (const message of boxOtelWarnings(project.spec)) yield { type: "warning", message };
for (const message of boxOtelWarnings(project.spec, boxNetworks)) {
yield { type: "warning", message };
}

const compatibilityWarning = await cdkCompatibilityWarning(this.cdkDirectory(project));
if (compatibilityWarning) {
Expand Down
14 changes: 11 additions & 3 deletions src/core/project/boxNetwork/network.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import { describeEntry, describeRule, ruleOrigin } from "./places";
import { placeOf, type ParsedPolicy } from "./policyParser";
import { runtimeProfileContext } from "../runtimeProfile/context";
import { deriveRuntimeProfile } from "../runtimeProfile/derive";
import { runtimeProfileHeader } from "../runtimeProfile/rules/additions";
import { defaultTracesTelemetry, runtimeProfileHeader } from "../runtimeProfile/rules/additions";
import type { TomlTable } from "../runtimeProfile/types";
import { withDeployRegion, type CloudWatchTelemetry } from "./telemetry";
import { outboundAuthWords, sameOutboundAuth, targetOutboundAuth } from "./projectSpec";
Expand Down Expand Up @@ -122,7 +122,10 @@ export interface BoxNetwork {
* Runtime profile.
*/
readonly policyFiles: readonly string[];
/** The CloudWatch targets of the profile's `[telemetry]`; absent means none. */
/**
* The CloudWatch targets of the profile's `[telemetry]`, then the default X-Ray target of
* {@link runtimeProfile} (`DEFAULT_TRACES_TARGET`); absent means none.
*/
readonly telemetry?: readonly CloudWatchTelemetry[];
/**
* The AgentCore Runtime profile that build derives from the local profile (`deriveRuntimeProfile`
Expand Down Expand Up @@ -217,14 +220,19 @@ export function buildBoxNetwork(
failures.push(
...(derived?.problems ?? []).map(({ code, reason, place }) => ({ code, reason, place })),
);
const defaultTraces = derived ? defaultTracesTelemetry(derived.profile, project.region) : [];
const sent =
telemetry || defaultTraces.length > 0
? [...(telemetry?.telemetry ?? []), ...defaultTraces]
: undefined;
return {
runtime: project.runtime.name,
gateway: project.gateway,
region: project.region,
hosts: targets.hosts,
envRoutes: envRoutesOf(targets.hosts),
policyFiles: parsed.profile.policyFiles,
...(telemetry && { telemetry: telemetry.telemetry }),
...(sent && { telemetry: sent }),
...(derived &&
context && {
runtimeProfile: derived.profile,
Expand Down
Loading
Loading