diff --git a/AGENTS.md b/AGENTS.md index f07305138..bf0f08071 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -106,9 +106,9 @@ and deserialized through a registry. | Package | Contents | |---|---| -| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset` and `load_file_artifact_universe`. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]`: files, a `python` requirements file for a venv, and the services the base's init runs; its deploy is refused until agent-env creates computers), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch with this machine's Docker, the other arch through its emulation, tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile and build context so any env with the same inputs reuses it; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; builds end `unverified`, and its deploy is refused until a computer can be booted). `env/gateway/gateway.py` is the env-side gateway server, container-only code. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | +| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]`: files, a `python` requirements file for a venv, and the services the base's init runs; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch with this machine's Docker, the other arch through its emulation, tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile and build context so any env with the same inputs reuses it; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 then boots each native build on a configured sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver). An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name). `env/gateway/gateway.py` is the env-side gateway server, container-only code. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | | `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Every artifact but an environment universe, whose `metadata` key holds its legacy metadata file ids, carries a `metadata` dict outside its versioned fields, changed by compare-and-swap (`merge_metadata`). Built-ins: `CliArtifact` (`cli`), `ComputerArtifact` (`computer`, a base computer: its architectures, OS, session user, settings and container form, whose image `put` reads from its registry into `metadata.validated_images`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or a published image's tag with its `image_digest` and its build context, which sandboxes pull by the digest and build from the context when they can't (the infra images, and a bundle agent built from a shipped agent's sources, `Plan.published`), or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | -| `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, `start_computer_recording` and `end_computer_recording`, whose open recordings `teardown_run` and `teardown_sandboxes` end first, and more). `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | +| `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, `start_computer_recording` and `end_computer_recording`, and more). `teardown_run` and the `teardown_sandboxes` step share one core (`task/teardown.py` `tear_down`): the recordings the run left open end with fresh grants, each env still up closes (`Env.close`; a computer env ends any recording still open), what stayed open is registered from its namespace, then every sandbox is terminated by id. `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the `@local` namespace: while the CLI or a `build_app` server runs, an `@local/…` id and what an `@local` task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. | | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. | | `providers/` | `providers/sandbox_providers/` holds the sandbox providers `local`, `modal`, `modal_vm`, `e2b`, `sail_vm` (the `sail` extra), `smol_vm` (the `smol` extra), `tensorlake` (the `tensorlake` extra), `tenki` (the `tenki` extra), `vercel` (the `vercel` extra), `boxd` (the `boxd` extra), `daytona` (the `daytona` extra), `boat` (the `boat` extra); `[sandbox] default` and `agent_default` accept a comma-separated fallback chain, whose deploys skip a provider that can't run them. A sandbox provider declares what it runs (`ON_THIS_MACHINE`, and for VMs on this machine the address their requests to it arrive at, `REACHES_HOST_AT`; `CONTAINER_ACCEPTS`, `PRIVATE_NETWORK`; `SUPPORTED_SANDBOX_MODES`, derived from whether it implements `create_sandbox` and `create_vm`; and `SANDBOX_ACCEPTS`, the images an agent's sandbox runs, which defaults to those a VM loads when it implements `create_vm` and to `CONTAINER_ACCEPTS` otherwise, so only a provider that implements `create_vm` yet gives an agent a container, like the local one, declares it), each sandbox carries the `platform` of the images it runs (linux/amd64 unless its provider sets another when it creates it; None, any, unchecked, for a local sandbox), and code outside `providers/sandbox_providers/` reads those, never which provider it has. `providers/env_providers/` holds the environment providers: `EnvironmentProvider` (an env's containers and state store), `EnvironmentGatewayProvider` (`gateway`, an env's default), which renders a docker-compose for the gateway and its MCP servers inside the sandbox, and `EnvironmentServerProvider` (`server`), one MCP server in its own container with no gateway in front and no state store; `providers/env_state/` holds env-state providers (`local_postgres` built in). | @@ -123,8 +123,8 @@ and deserialized through a registry. | `cli/` | Click CLI with the groups `a2a-agent`, `artifact`, `bundle`, `config`, `env`, `eval`, `plugin`, `run`, `task`, `up`. `task run`, `task run-batch` and `eval run` hand their runs to the configured `[runner]` and follow them. | | `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. | | `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, supervisord as the session user, settings as `AGENTENV_SETTING_`), aliased through `[artifacts]`. It has no tasks either. | -| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`. It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. `tst/integration/computer/` serves it on the `ubuntu-24.04` base in a scaffold image and runs the conformance probe and an agent against it. | -| `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, ` = ""` shipping `src///`. | +| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`. It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | +| `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, ` = ""` shipping `src///`. | ## Extension points diff --git a/packages/agentenv-protocol/README.md b/packages/agentenv-protocol/README.md index 6a9d66065..8d8dc1182 100644 --- a/packages/agentenv-protocol/README.md +++ b/packages/agentenv-protocol/README.md @@ -240,6 +240,32 @@ at the gateway. Where the binding lives depends on the app. On mcp's own FastMCP it wraps the tool manager's dispatch; on the standalone `fastmcp` package it joins the app's middleware chain as the outermost layer, so a handler's own middleware runs under it too. An app with neither seam mounts unbound, with a warning, unless a tool declares a `ToolContext` parameter or the handler defines `on_tool_call`: then the mount is refused rather than serving an empty caller. +### Getting part of the data + +A `@get_data` handler that takes a `paths` parameter returns only the data under the paths a caller +names (`client.get_data(base_url, paths=[...])`), and is called with None when the caller names +none. A handler without one answers `invalid_params` to a call that names paths, rather than +returning everything. + +### A skill for agents + +An env can hand agents a skill: a SKILL.md an agent reads before it uses the env. Mark the method +that writes it with `@agent_skill`; it returns the skill's description and its markdown body, +sync or async: + +```python +@agent_skill +def guide(self): + return "How to place and track orders.", "# The shop\n\nPlace orders with `place_order`.\n" +``` + +The card then declares `urn:agentenv:skill/v1` (`SKILL_EXTENSION_URI`), whose `get {name}` +answers `{name, description, skill_md}`, the shape `skill-config/v1` takes, with the frontmatter +written under the name the caller gives; a name that isn't lowercase letters, digits and single +hyphens, up to 64, is refused `invalid_argument`. agent-env's `add_skills` step installs it on +an agent through `env_ids`, named after the env's environment name; for an env behind a gateway, +whose card nests its servers' cards, it installs each server's skill under that server's name. + ### Uploading an export through a grant An export too large to return from `data/get` can go straight to the caller's object store. A @@ -396,6 +422,11 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope (`stale_frame`) when the display geometry it was captured at changes. - **The input queue.** Input calls run one at a time in arrival order; one that waits past `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue. +- **The computer skill.** `computer/v1` declares the seven tools, its params the driver's + features. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base + class): a SKILL.md written from the computer as it booted (its platform, session user and home, + the live screen size, what its input takes, its OS's shortcuts) and ending with the driver + class's `notes`, markdown for what its card can't say. - **Framework extensions.** `computer-ready/v1` (`status`: real checks of the display and a capture, plus the driver's own `ready_checks()`; tools refuse `device_unavailable` until they pass once), `computer-display/v1`, `computer-spaces/v1` (the default space, written into the tool @@ -405,8 +436,10 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope - **The data plane.** `data/add` takes file parts that name their path, relative to the session user's home, under the `data-objects/v1` marker (`{"path": ...}`), as inline bytes, an https URL, or a `transfers.ReadObject` read grant under the marker's `read`, and writes them atomically. - `data/get` returns the files under the settings' `data_paths`, inline or uploaded under a - `write_namespace` grant. + `data/get` returns the files under the `paths` it's given, relative to the home, or under the + card's `default_data_paths` (`Desktop`, `Documents`, `Downloads`) when it's given none, inline or + uploaded under a `write_namespace` grant. An environment whose `@get_data` hook takes no `paths` + refuses them. - **The card** (the name from `ENVIRONMENT_NAME`, then `@computer_driver`'s), and per-role tool access (`tool_access` is on). @@ -417,7 +450,7 @@ was sent (`applied: "no"`); `Timeout`, `DeliveryFailed` and `Internal` mean it m other exception from an input operation is reported `delivery_failed` with `applied: "unknown"`. **Settings** reach the driver as one JSON value in `AGENTENV_COMPUTER_SETTINGS` -(`ComputerSettings`): `data_paths`, `default_space`, `default_norm_scale`, `queue_wait_s`, +(`ComputerSettings`): `default_space`, `default_norm_scale`, `queue_wait_s`, `tap_hold_ms`, `type_delay_ms`, `max_hold_ms`, `layout`, `drag_duration_ms`, `exec_output_bytes` and `exec_timeout_s`. A driver reads them as `self.settings`, for example its tap hold in `@press`. @@ -456,13 +489,16 @@ class MyDriver(ComputerDriverEnvironment): `@pointer_position`: the slow path, at most 15 fps. - **The methods.** `start {request_id, scope?, fps?, max_edge?, cursor?, upload?}` returns `{recording_id, started_at, format}`; `end {recording_id, upload?, reason?}` closes the last segment - and returns the manifest. A repeated `request_id`, or a repeated `end`, answers the same. A start - past `max_concurrent` is refused `limit_exceeded`; `max_duration_s`, free disk under 512 MB or a stream + and returns the manifest; `end_all {reason?}` ends every recording not yet ended, each under the + grant it started with, and returns `{manifests}`, as the framework does before it resets or tears a + computer down. A repeated `request_id`, or a repeated `end`, answers the same. A start past + `max_concurrent` is refused `limit_exceeded`; `max_duration_s`, free disk under 512 MB or a stream failing for 60 s end a recording by itself. - **Segments.** A segment closes at `max_segment_s`, when the frame size changes or after a gap in the stream. Each uploads as it closes under `start`'s `write_namespace` grant, at `/.mp4`, and is then deleted; `end`'s own grant uploads any that missed. With no - grant they stay under `~/.agentenv-recordings/`. + grant they stay under `~/.agentenv-recordings/`. An ended recording's manifest uploads beside its + segments, at `/manifest.json`, so a reader that didn't end it finds it there. - **The manifest** (`RecordingManifest`). Its `segments` (`ManifestSegment`) carry each one's path, `started_at` and `ended_at` (its first and last frames' capture times), size, frame count, byte size, SHA-256 and whether it uploaded, and `ended_by` says why the recording ended: `end`, `reset`, diff --git a/packages/agentenv-protocol/src/agentenv_protocol/__init__.py b/packages/agentenv-protocol/src/agentenv_protocol/__init__.py index fd209caf4..a22d66607 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/__init__.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/__init__.py @@ -12,6 +12,7 @@ AgentEnvFastMCPApplication, AgentEnvStarletteApplication, add_data, + agent_skill, create_fastmcp_app, environment_card, extension, @@ -58,6 +59,7 @@ METHOD_RESET, PROTOCOL_VERSION, RPC_PATH, + SKILL_EXTENSION_URI, WELL_KNOWN_PATH, AddDataRequest, AddDataResponse, @@ -91,6 +93,7 @@ "reset_data", "add_data", "get_data", + "agent_skill", "extension", "tool", "injecting", @@ -117,6 +120,7 @@ "intake_fit_check", "INTAKE_EXTENSION_URI", "DATA_OBJECTS_EXTENSION_URI", + "SKILL_EXTENSION_URI", "uploaded_file_part", "uploaded_object_path", "MCP_PATH", diff --git a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py index 02b1c8896..b8b522a5b 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py @@ -9,6 +9,7 @@ import json import logging import os +import re import sys import types import uuid @@ -26,7 +27,7 @@ from .tool_access import extensions as tool_access_extensions from .tool_context import EMPTY, ToolContext, bound, context_from_http, context_from_mcp from .transfers import WriteNamespaceGrant -from .types import DATA_OBJECTS_EXTENSION_URI, MCP_TRANSPORT, METHOD_ADD, METHOD_GET, METHOD_RESET, RPC_PATH, WELL_KNOWN_PATH, AddDataRequest, AddDataResponse, EnvironmentCapabilities, EnvironmentCard, EnvironmentExtension, EnvironmentInterface, EnvironmentTool, GetDataResponse, ResetDataResponse, error_body +from .types import DATA_OBJECTS_EXTENSION_URI, MCP_TRANSPORT, SKILL_EXTENSION_URI, METHOD_ADD, METHOD_GET, METHOD_RESET, RPC_PATH, WELL_KNOWN_PATH, AddDataRequest, AddDataResponse, EnvironmentCapabilities, EnvironmentCard, EnvironmentExtension, EnvironmentInterface, EnvironmentTool, GetDataResponse, ResetDataResponse, error_body logger = logging.getLogger(__name__) @@ -53,6 +54,7 @@ RPC_ERROR_CODE_METHOD_NOT_FOUND = -32601 RPC_ERROR_CODE_INVALID_PARAMS = -32602 RPC_ERROR_CODE_SERVER_ERROR = -32000 +_SKILL_NAME = re.compile(r"[a-z0-9]+(-[a-z0-9]+)*") def reset_data(fn: Callable) -> Callable: @@ -126,6 +128,23 @@ def deco(fn: Callable) -> Callable: return deco +def agent_skill(fn: Callable) -> Callable: + """Mark the method that writes this env's skill for agents, returning its description and its markdown body; it + is served under ``SKILL_EXTENSION_URI`` as ``get {name}``, which writes the SKILL.md under the name asked for.""" + + async def get(self, name: str) -> dict: + if len(name) > 64 or not _SKILL_NAME.fullmatch(name): + raise AgentEnvError("invalid_argument", + f"a skill's name is lowercase letters, digits and single hyphens, up to 64, not {name!r}") + description, body = await fn(self) if inspect.iscoroutinefunction(fn) else fn(self) + frontmatter = f"---\nname: {name}\ndescription: {json.dumps(description)}\n---\n" + return {"name": name, "description": description, "skill_md": f"{frontmatter}\n{body}"} + + get.__name__ = fn.__name__ + return extension(SKILL_EXTENSION_URI, op="get", method="GET", + description="get returns a SKILL.md for agents under the name it's given.")(get) + + def _extension_slug(uri: str) -> str: """``urn:agentenv:computer-display/v1`` -> ``computer-display``; another namespace keeps its name as a path.""" name = uri.removeprefix("urn:").removeprefix("agentenv:") @@ -1162,6 +1181,14 @@ async def _invoke_add(fn: Callable, params: dict) -> dict: async def _invoke_get(fn: Callable, params: dict) -> dict: kwargs = {} + takes_paths = "paths" in inspect.signature(fn).parameters + if isinstance(params, dict) and "paths" in params: + if not takes_paths: + raise _InvalidParams("data/get on this environment takes no paths: it returns all of its data") + if not isinstance(params["paths"], list) or not all(isinstance(path, str) for path in params["paths"]): + raise _InvalidParams("paths must be a list of strings") + if takes_paths: + kwargs["paths"] = params.get("paths") if isinstance(params, dict) else None if _takes_write_namespace(fn): if not isinstance(params, dict): raise _InvalidParams("data/get takes its params by name") diff --git a/packages/agentenv-protocol/src/agentenv_protocol/client.py b/packages/agentenv-protocol/src/agentenv_protocol/client.py index e934b7720..0d0bfe648 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/client.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/client.py @@ -57,11 +57,15 @@ async def add_data(base_url: str, parts: list[Part], timeout: int = 120, verify: async def get_data( - base_url: str, timeout: int = 30, verify: bool = True, *, write_namespace: WriteNamespaceGrant | None = None + base_url: str, timeout: int = 30, verify: bool = True, *, write_namespace: WriteNamespaceGrant | None = None, + paths: list[str] | None = None, ) -> GetDataResponse: """`data/get`. A server advertising `DATA_OBJECTS_EXTENSION_URI` may be handed `write_namespace` to upload - its export under; `uploaded_object_path` reads from the answer where it did.""" + its export under; `uploaded_object_path` reads from the answer where it did. `paths` names what a computer + returns, relative to its user's home; an env that returns all of its data refuses them.""" params = {} if write_namespace is None else {"write_namespace": write_namespace.model_dump(mode="json", exclude_none=True)} + if paths is not None: + params["paths"] = paths return GetDataResponse.model_validate(await _rpc(base_url, METHOD_GET, params, timeout, verify)) diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py index bf6e1fffb..812cd0e44 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/_declare.py @@ -77,7 +77,8 @@ def declare(folder: Path, file: str, name: str) -> dict: return {"name": cards[0][1].name, "supports": list(cls.supports), "platforms": [ {"platform": platform.as_dict(), "provides": [e.uri for e in card.capabilities.extensions], "tools": [t.name for t in card.capabilities.tools], - "features": next(e.params for e in card.capabilities.extensions if e.uri == COMPUTER)} + "features": next({k: v for k, v in e.params.items() if k not in ("endpoint", "methods")} + for e in card.capabilities.extensions if e.uri == COMPUTER)} for platform, card in cards]} diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/card.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/card.py index f07319d4a..a8453b7e2 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/card.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/card.py @@ -1,10 +1,11 @@ """What a driver's card says: the extensions, the tools, extensions and data-plane methods its operations support, -the extensions' params, the extensions it declares without handlers, and the descriptions that name the default -coordinate space.""" +the extensions' params (computer/v1's are the driver's features), the extensions it declares without handlers, and the +descriptions that name the default coordinate space.""" from __future__ import annotations from typing import TYPE_CHECKING +from .dataplane import DEFAULT_DATA_PATHS from .recording import recording_params from ..types import METHOD_ADD, EnvironmentExtension, EnvironmentTool, IntakeDeclaration, IntakeFormat, intake_extension @@ -79,18 +80,23 @@ def extension_params(driver: ComputerDriverEnvironment, uri: str) -> dict: return {} -def declared_extensions(driver: ComputerDriverEnvironment) -> list[EnvironmentExtension]: - features = { +def computer_features(driver: ComputerDriverEnvironment) -> dict: + """computer/v1's params: what the seven tools take, how input is delivered, the platform and the default data + paths.""" + return { "buttons": list(driver.buttons), "max_click_count": driver.max_click_count, "keys": driver.keys if driver.keys == "all" else list(driver.keys), "max_text_length": driver.max_text_length, "max_wait_ms": driver.max_wait_ms, "max_drag_points": driver.max_drag_points, "max_duration_ms": driver.max_duration_ms, "max_key_repeat": driver.max_key_repeat, - "max_scroll_amount": driver.max_scroll_amount, + "max_scroll_amount": driver.max_scroll_amount, "delivery": "foreground", "platform": driver.platform.as_dict(), + "supports": list(driver.supports), "default_data_paths": list(DEFAULT_DATA_PATHS), } + + +def declared_extensions(driver: ComputerDriverEnvironment) -> list[EnvironmentExtension]: declared = [EnvironmentExtension( uri=COMPUTER, description="The seven computer_* tools: screenshot, click, drag, scroll, type, key, wait.", - params={**features, "delivery": "foreground", "platform": driver.platform.as_dict(), - "supports": list(driver.supports)})] + params=computer_features(driver))] if driver.has("pointer"): declared.append(EnvironmentExtension( uri=COMPUTER_POINTER, description="The computer_pointer_* tools, and click and scroll at the pointer.", @@ -99,7 +105,7 @@ def declared_extensions(driver: ComputerDriverEnvironment) -> list[EnvironmentEx if driver.has("write_files"): intake.add = [IntakeFormat(part="file", format="file-tree", load="additive")] declared.append(intake_extension(intake, "data/add writes files under the session user's home; data/get " - "returns the env's data paths.")) + "returns those under the paths it's given, or the default data paths.")) return declared diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/conformance/checks.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/conformance/checks.py index bb5da2b9e..a13bff5e9 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/conformance/checks.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/conformance/checks.py @@ -21,7 +21,7 @@ from ... import client as protocol from ...client import ExtensionCallError from ...tool_access import DISABLE_TOOL_EXTENSION_URI, ENABLE_TOOL_EXTENSION_URI -from ...types import DATA_OBJECTS_EXTENSION_URI, INTAKE_EXTENSION_URI +from ...types import DATA_OBJECTS_EXTENSION_URI, INTAKE_EXTENSION_URI, SKILL_EXTENSION_URI from ..card import ( COMPUTER, COMPUTER_DISPLAY, @@ -88,7 +88,7 @@ async def basic(self) -> None: for check, section in (("P0", self._ready), ("P1", self._card), ("P2", self._frame), ("P5", self._refusals), ("X", self._display), ("X", self._spaces), ("X", self._keyboard_settings), ("X", self._exec), ("X", self._data_get), ("X", self._tool_access), - ("X", self._pointer)): + ("X", self._pointer), ("X", self._skill)): await self._guarded(check, section) self._grants() @@ -388,6 +388,18 @@ async def missing() -> str | None: await self.check("X", "output past max_output_bytes is cut there", capped, COMPUTER_EXEC) await self.check("X", "a program that isn't there is refused with a code", missing, COMPUTER_EXEC) + async def _skill(self) -> None: + if not self.driver.declares(SKILL_EXTENSION_URI): + return + + async def written() -> str | None: + got = await self.driver.invoke(SKILL_EXTENSION_URI, "get", {"name": "probe-skill"}) + if got.get("name") != "probe-skill" or not str(got.get("skill_md")).startswith("---\nname: probe-skill\n"): + return f"it answered {got}" + return None if got.get("description") else "its description is empty" + + await self.check("X", "the skill is written under the name it's asked for", written, SKILL_EXTENSION_URI) + async def _data_get(self) -> None: if not self.driver.declares(INTAKE_EXTENSION_URI): return diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/dataplane.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/dataplane.py index e1621dd1a..ba9f8dcb8 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/dataplane.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/dataplane.py @@ -1,5 +1,6 @@ """The data plane on a computer: ``data/add`` writes files under the session user's home, ``data/get`` returns the -env's ``data_paths``. Each file part names its path, relative to the home, under the ``data-objects/v1`` marker.""" +files under the paths its caller names, or under ``DEFAULT_DATA_PATHS`` when it names none. Each file part names its +path, relative to the home, under the ``data-objects/v1`` marker.""" from __future__ import annotations import base64 @@ -10,10 +11,10 @@ from typing import TYPE_CHECKING, Optional import httpx -from pydantic import ValidationError +from pydantic import TypeAdapter, ValidationError from ..agent_env_environment import add_data, get_data -from ..transfers import NamespaceUploader, ReadObject, WriteNamespaceGrant, download +from ..transfers import NamespaceUploader, ReadObject, RelativePath, WriteNamespaceGrant, download from ..types import DATA_OBJECTS_EXTENSION_URI, FilePart, FileWithBytes, FileWithUri, uploaded_file_part, uploaded_object_path if TYPE_CHECKING: @@ -22,6 +23,9 @@ #: The most an https URL part may fetch; a read grant carries its own limit. MAX_URL_BYTES = 1 << 30 +#: What ``data/get`` returns when its caller names no paths: the folders a desktop keeps its user's files in. +DEFAULT_DATA_PATHS = ("Desktop", "Documents", "Downloads") +_PATHS = TypeAdapter(list[RelativePath]) class DataPlane: @@ -47,8 +51,13 @@ async def _computer_data_add(self, parts: list) -> None: await self._computer.call("write_files", files) @get_data - async def _computer_data_get(self, write_namespace: Optional[WriteNamespaceGrant] = None) -> list: - files = _data_files(Path.home(), self.settings.data_paths) + async def _computer_data_get(self, write_namespace: Optional[WriteNamespaceGrant] = None, + paths: Optional[list[str]] = None) -> list: + try: + named = _PATHS.validate_python(paths) if paths is not None else list(DEFAULT_DATA_PATHS) + except ValidationError: + raise ValueError(f"data/get on a computer takes paths relative to the home, not {paths!r}") from None + files = _data_files(Path.home(), named) if write_namespace is not None: uploader = NamespaceUploader(write_namespace) parts = [] @@ -98,11 +107,11 @@ async def _get_url(file: FileWithUri) -> bytes: return bytes(data) -def _data_files(home: Path, data_paths: list[str]) -> list[tuple[str, Path]]: - """Every file under the data paths, by its path relative to the home; a path not there yet is skipped.""" +def _data_files(home: Path, paths: list[str]) -> list[tuple[str, Path]]: + """Every file under ``paths``, by its path relative to the home; a path not there yet is skipped.""" root = home.resolve() found: dict[str, Path] = {} - for relative in data_paths: + for relative in paths: target = home / relative if not target.exists() or not target.resolve().is_relative_to(root): continue diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py index f8f28aa6c..b70320e9d 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/driver.py @@ -46,6 +46,8 @@ class ComputerDriverEnvironment(Tools, Extensions, DataPlane, AgentEnvEnvironmen #: computer-recording/v1, from ``@record`` if the driver has one, else by polling ``@capture``; ``None`` turns #: it off. recording: ClassVar[Recording | None] = Recording() + #: What an agent should know about this driver's computer beyond what its card says, as markdown: the end of its skill. + notes: ClassVar[str] = "" tool_access = True #: What the base class keeps for the driver, built by ``__init__``. _computer: ComputerRuntime | None = None diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/extensions.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/extensions.py index b69aefd28..52a522112 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/extensions.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/extensions.py @@ -1,10 +1,10 @@ -"""The framework extensions agentenv calls, never agents: readiness, display, coordinate spaces, keyboard settings, -commands in the session and recordings.""" +"""The framework extensions agentenv calls, never agents: the computer skill, readiness, display, coordinate spaces, +keyboard settings, commands in the session and recordings.""" from __future__ import annotations from typing import TYPE_CHECKING, Callable, Optional -from ..agent_env_environment import extension +from ..agent_env_environment import agent_skill, extension from . import card from .card import ( COMPUTER_DISPLAY, @@ -18,6 +18,7 @@ ) from .errors import Internal, InvalidArgument, Unsupported from .operations import Display, ExecResult +from .skill import skill from .spaces import check_space if TYPE_CHECKING: @@ -36,6 +37,10 @@ class Extensions: layouts: tuple[str, ...] mounted_card: Callable[[], EnvironmentCard] + @agent_skill + async def _computer_skill(self) -> tuple[str, str]: + return await skill(self) + @extension(COMPUTER_READY, op="status", method="GET", description="Whether the computer is usable now, by real checks; never sends input.") async def _computer_ready_status(self) -> dict: @@ -114,7 +119,8 @@ async def _computer_exec_run(self, argv: list, request_id: str, cwd: Optional[st @extension(COMPUTER_RECORDING, op="start", description="Record the display: start returns a recording whose segments upload through its grant as " - "they close, and end seals it and returns its manifest; repeats answer the same.") + "they close; end seals it and returns its manifest, uploaded beside them; end_all ends " + "every one not yet ended; repeats answer the same.") async def _computer_recording_start(self, request_id: str, scope: str = "display", window: Optional[str] = None, fps: Optional[int] = None, max_edge: Optional[int] = None, cursor: bool = True, upload: Optional[dict] = None) -> dict: @@ -125,6 +131,10 @@ async def _computer_recording_end(self, recording_id: str, upload: Optional[dict reason: str = "end") -> dict: return await self._computer.recordings.end(recording_id, upload, reason) + @extension(COMPUTER_RECORDING, op="end_all") + async def _computer_recording_end_all(self, reason: str = "end") -> dict: + return {"manifests": await self._computer.recordings.end_all(reason)} + async def _run(computer: ComputerRuntime, cap: int, argv: list, cwd: Optional[str], env: Optional[dict], timeout: float) -> dict: diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/recording.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/recording.py index fbd248d3b..8feca498e 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/recording.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/recording.py @@ -7,6 +7,7 @@ import contextlib import dataclasses import hashlib +import json import shutil import threading import time @@ -38,6 +39,8 @@ RECORDINGS_DIR = ".agentenv-recordings" SCOPES = ("display", "window") REASONS = ("end", "reset", "teardown") +#: What an ended recording's manifest is uploaded as, under its id beside its segments. +MANIFEST = "manifest.json" DEFAULT_FPS = 10 #: The fallback polls ``@capture``, a full read of the screen each pass, so it is capped lower than streams. MAX_POLLED_FPS = 15 @@ -179,6 +182,10 @@ async def end(self, recording_id: str, upload: Optional[dict], reason: str) -> d self.ended.popitem(last=False) return manifest + async def end_all(self, reason: str) -> list[dict]: + """End every recording not yet ended, each uploading under the grant it was started with; their manifests.""" + return [await self.end(recording_id, None, reason) for recording_id in list(self.running)] + def _hints(self, features: Recording, scope: str, window: Optional[str], fps: Optional[int], max_edge: Optional[int], cursor: bool, upload: Optional[dict]) -> tuple[RecordHints, Any]: if scope not in SCOPES: @@ -287,11 +294,15 @@ async def end(self, reason: str, grant: Optional[WriteNamespaceGrant]) -> dict: with contextlib.suppress(OSError): self.hints.directory.rmdir() entries = [s.entry for s in self.segments] - return RecordingManifest( + manifest = RecordingManifest( recording_id=self.id, scope=self.hints.scope, format=MP4, fps=self.hints.fps, cursor=self.cursor, source="stream" if self.streams else "capture", started_at=entries[0].started_at if entries else None, ended_at=entries[-1].ended_at if entries else None, ended_by=self.ended_by, segments=entries, error=self.error if self.ended_by == "error" else None).wire() + if uploader is not None: # beside the segments, so a reader who didn't end the recording finds it + with contextlib.suppress(Exception): + await uploader.upload(f"{self.id}/{MANIFEST}", json.dumps(manifest).encode()) + return manifest async def _add_frame(self, item: RecordedFrame) -> None: frame = await self._in_thread(_video_frame, item.image, self.hints.max_edge) diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/settings.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/settings.py index 320ed3335..60e8d6d23 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/settings.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/settings.py @@ -7,7 +7,6 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationError -from ..transfers import RelativePath SETTINGS_ENV = "AGENTENV_COMPUTER_SETTINGS" @@ -15,12 +14,10 @@ class ComputerSettings(BaseModel): - """``data_paths`` are relative to the session user's home and are what ``data/get`` returns; the rest are the - defaults the card reports and the framework extensions change.""" + """The defaults the card reports and the framework extensions change.""" model_config = ConfigDict(extra="forbid", validate_assignment=True) - data_paths: list[RelativePath] = Field(default_factory=list) default_space: Space = "frame" default_norm_scale: int = Field(default=1000, ge=1, le=1_000_000) queue_wait_s: float = Field(default=30.0, gt=0, le=3600) diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/skill.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/skill.py new file mode 100644 index 000000000..5c31f55cb --- /dev/null +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/skill.py @@ -0,0 +1,68 @@ +"""The computer skill: a SKILL.md an agent reads before it uses the computer, written from the computer as it booted: +its platform, its session user, its screen, its OS's shortcuts, what its card declares, and the driver's notes. It +says nothing about any task.""" +from __future__ import annotations + +import getpass +import os +from pathlib import Path +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from .driver import ComputerDriverEnvironment + +#: The chords each OS's desktops keep for the everyday actions, as ``computer_key`` takes them. +SHORTCUTS = { + "linux": """\ +- Copy, cut and paste: `['Control', 'c']`, `['Control', 'x']`, `['Control', 'v']`; in a terminal, with `'Shift'` too. +- Undo `['Control', 'z']`, save `['Control', 's']`, select all `['Control', 'a']`, find `['Control', 'f']`. +- Switch windows `['Alt', 'Tab']`; close the focused one `['Alt', 'F4']`. +- In a file manager, `['Control', 'l']` takes a typed path, `['F2']` renames and `['Control', 'Shift', 'n']` makes a + folder. +""", +} +_SESSIONS = {"x11": "an X11 session", "wayland": "a Wayland session"} + + +async def skill(driver: ComputerDriverEnvironment) -> tuple[str, str]: + """The skill's description and its markdown body, as ``agent_skill`` takes them.""" + display = await driver._computer.display_now() + platform = driver.platform + system = " ".join(part for part in ((platform.distribution or platform.os or "").title(), platform.version, + f"({platform.arch})" if platform.arch else None) if part) + session = _SESSIONS.get(platform.display or "", "a desktop session") + if desktop := os.environ.get("XDG_CURRENT_DESKTOP"): + session += f" ({desktop})" + description = (f"How to use this computer through its computer_* tools: {system}, {display.width}x{display.height}, " + "what its input takes, and its shortcuts.") + sections = [ + "# Using this computer", + "You see this computer through `computer_screenshot` and drive it through the other `computer_*` tools, as a " + "person at its screen would: look, then click, type and press keys on what the screen shows.", + "## The computer", + "\n".join([f"- {system}, {session}.", f"- You act as `{getpass.getuser()}`, whose home is `{Path.home()}`.", + f"- The screen is {display.width}x{display.height}."]), + "## Input", + _input(driver), + ] + if shortcuts := SHORTCUTS.get(platform.os or ""): + sections += ["## Shortcuts", shortcuts.rstrip()] + if notes := driver.notes.strip(): + sections += ["## Notes", notes] + return description, "\n\n".join(sections) + "\n" + + +def _input(driver: ComputerDriverEnvironment) -> str: + keys = ("any W3C key name, such as `'Enter'`, `'Control'`, `'ArrowLeft'` or `'a'`" if driver.keys == "all" else + "only " + ", ".join(f"`{key!r}`" for key in driver.keys)) + space, scale = driver.settings.default_space, driver.settings.default_norm_scale + points = ("pixels of the latest screenshot" if space == "frame" else + f"0 to {scale} across and down the latest screenshot") + return "\n".join([ + f"- Buttons: {', '.join(driver.buttons)}; up to {driver.max_click_count} clicks in one `computer_click`.", + f"- Keys: {keys}. A chord is pressed in order and released in reverse.", + f"- Up to {driver.max_text_length} characters in one `computer_type`; `computer_wait` waits up to " + f"{driver.max_wait_ms} ms.", + f"- Points are {points}, unless a call names another space.", + "- Input is foreground: it goes to the focused window and moves the pointer, as a person's would.", + ]) diff --git a/packages/agentenv-protocol/src/agentenv_protocol/types.py b/packages/agentenv-protocol/src/agentenv_protocol/types.py index 56a860ea6..448b430b6 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/types.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/types.py @@ -37,6 +37,11 @@ def error_body(code: str, message: str, **fields: Any) -> dict: # it, answering with the part `uploaded_file_part` builds. A server that does not advertise it # is never sent one. DATA_OBJECTS_EXTENSION_URI = "urn:agentenv:data-objects/v1" + +# `get {name}` answers with the env's skill for agents, `{name, description, skill_md}` as +# skill-config/v1 takes one: a SKILL.md an agent reads before it uses the env, written under the +# name the caller gives. A server serves it through an `@agent_skill` handler. +SKILL_EXTENSION_URI = "urn:agentenv:skill/v1" _RELATIVE_PATH = TypeAdapter(RelativePath) diff --git a/packages/agentenv-protocol/tests/test_agent_env_environment.py b/packages/agentenv-protocol/tests/test_agent_env_environment.py index 5d145b312..8c26a270b 100644 --- a/packages/agentenv-protocol/tests/test_agent_env_environment.py +++ b/packages/agentenv-protocol/tests/test_agent_env_environment.py @@ -20,6 +20,7 @@ from agentenv_protocol import ( HIDDEN_BY_DEFAULT_META_KEY, + SKILL_EXTENSION_URI, AgentEnvEnvironment, AgentEnvFastMCPApplication, AgentEnvStarletteApplication, @@ -30,6 +31,7 @@ EnvironmentInterface, EnvironmentTool, add_data, + agent_skill, create_fastmcp_app, environment_card, extension, @@ -143,6 +145,44 @@ async def test_add_invalid_params_returns_error(): r = await client.post("/agentenv", json=_rpc("data/add", {"parts": [{"kind": "bogus"}]})) assert r.json()["error"]["code"] == -32602 + r = await client.post("/agentenv", json=_rpc("data/get", {"paths": ["items"]})) + assert r.json()["error"]["code"] == -32602 and "takes no paths" in r.json()["error"]["data"]["detail"] + + +class _PathsHandler: + @get_data + async def _export(self, paths): + return [{"kind": "data", "data": {"paths": paths}}] + + +@pytest.mark.asyncio +async def test_a_get_hook_that_takes_paths_gets_none_when_the_call_names_none(): + async with _client(_PathsHandler) as client: + named = await client.post("/agentenv", json=_rpc("data/get", {"paths": ["a"]})) + unnamed = await client.post("/agentenv", json=_rpc("data/get")) + assert named.json()["result"]["parts"][0]["data"] == {"paths": ["a"]} + assert unnamed.json()["result"]["parts"][0]["data"] == {"paths": None} + + +class _SkillHandler: + @agent_skill + def _guide(self): + return "How to use the shop's API.", "# The shop\n\nOrders are placed with `place_order`.\n" + + +@pytest.mark.asyncio +async def test_an_env_serves_its_skill_under_the_name_its_asked_for_and_refuses_one_that_isnt_a_skill_name(): + async with _client(_SkillHandler) as client: + card = (await client.get("/.well-known/agent-env.json")).json() + answer = (await client.get("/agentenv/ext/skill/get", params={"name": "shop"})).json() + refused = await client.get("/agentenv/ext/skill/get", params={"name": "The Shop"}) + extensions = {e["uri"]: e for e in card["capabilities"]["extensions"]} + assert extensions[SKILL_EXTENSION_URI]["params"]["methods"]["get"]["method"] == "GET" + assert answer == {"name": "shop", "description": "How to use the shop's API.", "skill_md": + '---\nname: shop\ndescription: "How to use the shop\'s API."\n---\n\n# The shop\n\n' + "Orders are placed with `place_order`.\n"} + assert refused.status_code == 400 and "lowercase letters, digits and single hyphens" in refused.text + @pytest.mark.asyncio async def test_unknown_method_returns_error(): diff --git a/packages/agentenv-protocol/tests/test_computer_conformance.py b/packages/agentenv-protocol/tests/test_computer_conformance.py index 6b57262e1..5f14c3cdd 100644 --- a/packages/agentenv-protocol/tests/test_computer_conformance.py +++ b/packages/agentenv-protocol/tests/test_computer_conformance.py @@ -62,7 +62,7 @@ async def test_the_fake_driver_passes_every_check_at_full_and_cleans_up(home): report = await _probe(driver) assert report.passed, _failed(report) assert report.driver == "FixtureDriver" and report.level == "full" - assert set(report.extensions.values()) == {"verified"} and len(report.extensions) == 12 + assert set(report.extensions.values()) == {"verified"} and len(report.extensions) == 13 counts = report.counts() assert counts["failed"] == counts["unverified"] == 0 and counts["not_applicable"] == 1 assert {c.check for c in report.checks} == {f"P{n}" for n in range(10)} | {"X"} diff --git a/packages/agentenv-protocol/tests/test_computer_dataplane.py b/packages/agentenv-protocol/tests/test_computer_dataplane.py index 212d99123..76108af3b 100644 --- a/packages/agentenv-protocol/tests/test_computer_dataplane.py +++ b/packages/agentenv-protocol/tests/test_computer_dataplane.py @@ -1,5 +1,6 @@ """The data plane on a computer: data/add writes file parts atomically under the session user's home, from inline -bytes, an https URL or a read grant, with paths checked; data/get returns the data paths, inline or uploaded.""" +bytes, an https URL or a read grant, with paths checked; data/get returns the files under the paths it's given, or under +the default data paths, inline or uploaded.""" from __future__ import annotations import base64 @@ -15,7 +16,7 @@ from computer_fake import FakeDriver, http_client, rpc # noqa: E402 from agentenv_protocol import DATA_OBJECTS_EXTENSION_URI # noqa: E402 -from agentenv_protocol.computer import ComputerSettings # noqa: E402 +from agentenv_protocol.computer.dataplane import DEFAULT_DATA_PATHS # noqa: E402 @pytest.fixture @@ -110,7 +111,7 @@ async def fake_download(source, destination): @pytest.mark.asyncio -async def test_data_get_returns_the_data_paths_inline(home, tmp_path_factory): +async def test_data_get_returns_the_files_under_the_paths_it_names_inline(home, tmp_path_factory): (home / "app" / "db").mkdir(parents=True) (home / "app" / "db" / "state.json").write_text('{"n": 1}') (home / "app" / "db" / "blob.bin").write_bytes(b"\x01") @@ -118,9 +119,9 @@ async def test_data_get_returns_the_data_paths_inline(home, tmp_path_factory): outside = tmp_path_factory.mktemp("outside") (outside / "secret").write_text("no") (home / "app" / "db" / "escape").symlink_to(outside / "secret") - driver = FakeDriver(settings=ComputerSettings(data_paths=["app", "notes.md", "missing", "app/db"])) - async with http_client(driver) as http: - parts = (await http.post("/agentenv", json=rpc("data/get"))).json()["result"]["parts"] + async with http_client(FakeDriver()) as http: + parts = (await http.post("/agentenv", json=rpc("data/get", {"paths": ["app", "notes.md", "missing", "app/db"]})) + ).json()["result"]["parts"] by_path = {p["metadata"][DATA_OBJECTS_EXTENSION_URI]["path"]: p["file"] for p in parts} assert sorted(by_path) == ["app/db/blob.bin", "app/db/state.json", "notes.md"] assert base64.b64decode(by_path["app/db/state.json"]["bytes"]) == b'{"n": 1}' @@ -128,6 +129,27 @@ async def test_data_get_returns_the_data_paths_inline(home, tmp_path_factory): assert by_path["app/db/blob.bin"]["mimeType"] == "application/octet-stream" +@pytest.mark.asyncio +async def test_data_get_naming_no_paths_returns_the_default_data_paths(home): + for folder in DEFAULT_DATA_PATHS: + (home / folder).mkdir() + (home / folder / "f.txt").write_text(folder) + (home / ".cache").mkdir() + (home / ".cache" / "noise").write_text("no") + async with http_client(FakeDriver()) as http: + parts = (await http.post("/agentenv", json=rpc("data/get"))).json()["result"]["parts"] + assert sorted(p["metadata"][DATA_OBJECTS_EXTENSION_URI]["path"] for p in parts) == [ + "Desktop/f.txt", "Documents/f.txt", "Downloads/f.txt"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("paths, code", [(["../out"], -32000), (["/etc"], -32000), ("notes", -32602), ([1], -32602)]) +async def test_data_get_refuses_paths_outside_the_home(home, paths, code): + async with http_client(FakeDriver()) as http: + error = (await http.post("/agentenv", json=rpc("data/get", {"paths": paths}))).json()["error"] + assert error["code"] == code + + @pytest.mark.asyncio async def test_data_get_uploads_under_a_write_namespace_grant(home): (home / "notes.md").write_text("# notes") @@ -138,9 +160,9 @@ async def test_data_get_uploads_under_a_write_namespace_grant(home): upload = AsyncMock() with patch("agentenv_protocol.computer.dataplane.NamespaceUploader") as uploader: uploader.return_value.upload = upload - driver = FakeDriver(settings=ComputerSettings(data_paths=["notes.md"])) - async with http_client(driver) as http: - parts = (await http.post("/agentenv", json=rpc("data/get", {"write_namespace": grant}))).json()["result"]["parts"] + async with http_client(FakeDriver()) as http: + parts = (await http.post("/agentenv", json=rpc("data/get", {"write_namespace": grant, "paths": ["notes.md"]})) + ).json()["result"]["parts"] assert upload.await_args.args[0] == "notes.md" and upload.await_args.args[1] == home / "notes.md" assert parts == [{"kind": "file", "file": {"uri": "notes.md", "mimeType": "text/markdown", "name": "notes.md"}, "metadata": {DATA_OBJECTS_EXTENSION_URI: {"path": "notes.md"}}}] diff --git a/packages/agentenv-protocol/tests/test_computer_driver.py b/packages/agentenv-protocol/tests/test_computer_driver.py index a0e3331ce..c6ceec511 100644 --- a/packages/agentenv-protocol/tests/test_computer_driver.py +++ b/packages/agentenv-protocol/tests/test_computer_driver.py @@ -147,9 +147,9 @@ def test_the_platform_is_detected(tmp_path, monkeypatch): def test_settings_come_from_one_json_value(): assert load_settings({}) == ComputerSettings() - settings = load_settings({SETTINGS_ENV: '{"data_paths": ["notes"], "default_space": "norm", "tap_hold_ms": 20}'}) - assert settings.data_paths == ["notes"] and settings.default_space == "norm" and settings.tap_hold_ms == 20 - for raw in ('{"data_paths": ["../out"]}', '{"tap_hold_ms": -1}', '{"unknown": 1}', "not json"): + settings = load_settings({SETTINGS_ENV: '{"default_space": "norm", "tap_hold_ms": 20}'}) + assert settings.default_space == "norm" and settings.tap_hold_ms == 20 + for raw in ('{"default_space": "pixels"}', '{"tap_hold_ms": -1}', '{"data_paths": ["notes"]}', "not json"): with pytest.raises(ValueError, match=SETTINGS_ENV): load_settings({SETTINGS_ENV: raw}) @@ -169,6 +169,8 @@ async def test_the_card_declares_the_contract_the_driver_implements(): computer = extensions["urn:agentenv:computer/v1"]["params"] assert computer["buttons"] == ["left", "right", "middle"] and computer["max_click_count"] == 3 assert computer["keys"] == "all" and computer["platform"] == LINUX.as_dict() and computer["delivery"] == "foreground" + assert computer["default_data_paths"] == ["Desktop", "Documents", "Downloads"] + assert "methods" not in computer and extensions["urn:agentenv:skill/v1"]["params"]["methods"]["get"]["method"] == "GET" assert extensions["urn:agentenv:computer-pointer/v1"]["params"]["buttons"] == ["left", "right", "middle"] assert set(extensions["urn:agentenv:computer-keyboard/v1"]["params"]["methods"]) == {"get", "set"} assert set(extensions["urn:agentenv:computer-display/v1"]["params"]["methods"]) == {"get", "set"} @@ -181,6 +183,24 @@ async def test_the_card_declares_the_contract_the_driver_implements(): assert card["additionalInterfaces"] == [{"url": "/mcp", "transport": "mcp"}] +@pytest.mark.asyncio +async def test_the_skill_is_written_from_the_computer_as_it_booted_under_the_name_its_given(monkeypatch): + monkeypatch.setenv("XDG_CURRENT_DESKTOP", "XFCE") + monkeypatch.setattr("agentenv_protocol.computer.skill.getpass.getuser", lambda: "user") + monkeypatch.setattr(FakeDriver, "notes", "The dock is at the bottom.") + async with http_client(FakeDriver()) as http: + answer = (await http.get("/agentenv/ext/skill/get", params={"name": "computer-desktop"})).json() + refused = await http.get("/agentenv/ext/skill/get", params={"name": "Computer Desktop"}) + skill_md = answer["skill_md"] + assert answer["name"] == "computer-desktop" and skill_md.startswith("---\nname: computer-desktop\ndescription: ") + assert answer["description"] in skill_md and "Ubuntu 24.04 (amd64), 1280x800" in answer["description"] + for line in ("- Ubuntu 24.04 (amd64), an X11 session (XFCE).", "- You act as `user`", "- The screen is 1280x800.", + "- Buttons: left, right, middle; up to 3 clicks", "pixels of the latest screenshot", + "`['Alt', 'Tab']`", "## Notes\n\nThe dock is at the bottom."): + assert line in skill_md, line + assert refused.status_code == 400 and "lowercase letters, digits and single hyphens" in refused.text + + @pytest.mark.asyncio async def test_a_minimal_driver_declares_only_what_it_has(): async with http_client(MinimalDriver(platform=MACOS)) as http: diff --git a/packages/agentenv-protocol/tests/test_computer_recording.py b/packages/agentenv-protocol/tests/test_computer_recording.py index c238dd9bf..24224ec08 100644 --- a/packages/agentenv-protocol/tests/test_computer_recording.py +++ b/packages/agentenv-protocol/tests/test_computer_recording.py @@ -4,6 +4,7 @@ import asyncio import hashlib +import json import threading import time from datetime import datetime, timedelta, timezone @@ -32,6 +33,7 @@ from agentenv_protocol.transfers import TransferError # noqa: E402 START, END = "/agentenv/ext/computer-recording/start", "/agentenv/ext/computer-recording/end" +END_ALL = "/agentenv/ext/computer-recording/end_all" @pytest.fixture(autouse=True) @@ -129,7 +131,7 @@ class Off(FakeDriver): async with http_client(Off()) as http: off = (await http.get("/.well-known/agent-env.json")).json() extension = next(e for e in card["capabilities"]["extensions"] if e["uri"] == COMPUTER_RECORDING) - assert set(extension["params"]["methods"]) == {"start", "end"} + assert set(extension["params"]["methods"]) == {"start", "end", "end_all"} assert {k: v for k, v in extension["params"].items() if k not in ("methods", "endpoint")} == { "formats": ["video/mp4; codecs=avc1"], "scopes": ["display"], "fps": {"default": 10, "max": 15}, "cursor": True, "source": "capture", "max_segment_s": 15, "max_concurrent": 2, "max_duration_s": 86_400} @@ -270,7 +272,7 @@ def uploader(grant): (segment,) = manifest["segments"] assert segment["uploaded"] is True and "upload_error" not in segment assert [root for root, _ in calls] == ["rec/run-1", "rec/run-2"] - assert calls[1][1].await_args.args[0] == segment["path"] + assert [call.args[0] for call in calls[1][1].await_args_list] == [segment["path"], f"{manifest['recording_id']}/manifest.json"] assert not (home / ".agentenv-recordings" / segment["path"]).exists() @@ -340,12 +342,33 @@ async def test_bad_calls_are_refused_with_a_code(path, body, status, code): @pytest.mark.asyncio -async def test_uploaded_segments_are_not_sent_again_at_end(home): +async def test_uploaded_segments_are_not_sent_again_at_end_and_the_manifest_goes_beside_them(home): upload = AsyncMock() with patch.object(recording_module, "NamespaceUploader") as uploader: uploader.return_value.upload = upload _, manifest = await _record(Recorded(), 1.4, start={"upload": _grant()}) - assert all(s["uploaded"] for s in manifest["segments"]) and upload.await_count == len(manifest["segments"]) >= 2 + paths = [call.args[0] for call in upload.await_args_list] + assert all(s["uploaded"] for s in manifest["segments"]) and len(manifest["segments"]) >= 2 + assert paths == [s["path"] for s in manifest["segments"]] + [f"{manifest['recording_id']}/manifest.json"] + assert json.loads(upload.await_args_list[-1].args[1]) == manifest + + +@pytest.mark.asyncio +async def test_end_all_ends_every_recording_not_yet_ended_with_the_grant_it_started_with(home): + upload = AsyncMock() + driver = Recorded() + with patch.object(recording_module, "NamespaceUploader") as uploader: + uploader.return_value.upload = upload + async with http_client(driver) as http: + started = [(await http.post(START, json={"request_id": rid, "fps": 10, "upload": _grant()})).json() + for rid in ("a", "b")] + await _first_frame(driver, started[0]) + ended = (await http.post(END, json={"recording_id": started[0]["recording_id"]})).json() + rest = (await http.post(END_ALL, json={"reason": "teardown"})).json()["manifests"] + again = (await http.post(END_ALL, json={"reason": "teardown"})).json()["manifests"] + assert ended["ended_by"] == "end" and [m["recording_id"] for m in rest] == [started[1]["recording_id"]] + assert rest[0]["ended_by"] == "teardown" and again == [] + assert f"{started[1]['recording_id']}/manifest.json" in [call.args[0] for call in upload.await_args_list] @pytest.mark.asyncio diff --git a/packages/agentenv-protocol/tests/test_data_objects.py b/packages/agentenv-protocol/tests/test_data_objects.py index dd235e690..d19337bcb 100644 --- a/packages/agentenv-protocol/tests/test_data_objects.py +++ b/packages/agentenv-protocol/tests/test_data_objects.py @@ -186,7 +186,7 @@ async def test_params_given_by_position_are_refused(): @pytest.mark.asyncio -async def test_get_data_sends_no_params_without_a_grant_and_the_grant_with_one(monkeypatch): +async def test_get_data_sends_no_params_without_a_grant_or_paths_and_each_it_is_given(monkeypatch): sent: list[dict] = [] real = httpx.AsyncClient @@ -199,9 +199,11 @@ def handle(request: httpx.Request) -> httpx.Response: grant = _grant() await protocol_v1.get_data("http://env") await protocol_v1.get_data("http://env", write_namespace=grant) + await protocol_v1.get_data("http://env", paths=["Documents"]) assert sent[0]["params"] == {} assert WriteNamespaceGrant.model_validate(sent[1]["params"]["write_namespace"]) == grant + assert sent[2]["params"] == {"paths": ["Documents"]} @pytest.mark.asyncio diff --git a/pyproject.toml b/pyproject.toml index 2f6aa591d..f01712612 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -131,6 +131,7 @@ agent-env = "agent_env.cli:cli" agents = "agent_env" computers = "agent_env" drivers = "agent_env" +desktop = "agent_env.examples" hello = "agent_env.examples" echo-mcp = "agent_env.examples" mail-mcp = "agent_env.examples" diff --git a/src/agent_env/a2a_agent/a2a_agent.py b/src/agent_env/a2a_agent/a2a_agent.py index f43f205f8..05947fdba 100644 --- a/src/agent_env/a2a_agent/a2a_agent.py +++ b/src/agent_env/a2a_agent/a2a_agent.py @@ -32,7 +32,7 @@ ) from agent_env.a2a_agent.staging import transfer_store from agent_env.providers.sandbox_providers.chained_sandbox_provider import skip_refusing -from agent_env.providers.sandbox_providers.local_sandbox import LOCAL_TRUST_ENV, local_grant_trust, start_trusting +from agent_env.providers.sandbox_providers.local_docker.provider import LOCAL_TRUST_ENV, local_grant_trust, start_trusting from agent_env.providers.sandbox_providers.sandbox_provider import ( SANDBOX_MODE_VM, Accepts, diff --git a/src/agent_env/cli/run.py b/src/agent_env/cli/run.py index 9a0dbd0d6..24b5fb549 100644 --- a/src/agent_env/cli/run.py +++ b/src/agent_env/cli/run.py @@ -16,11 +16,10 @@ path_like, run_name) from agent_env.plugins._bundles import ALIAS_TABLES, installed_aliases from agent_env.plugins._discovery import discovery_error -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.task.interrupts import Interrupts -from agent_env.task.teardown import RecordedSandbox, TeardownReport, kind, recorded_sandboxes, sandbox_ids +from agent_env.task.teardown import TORN_DOWN_KEY, RecordedSandbox, TeardownReport, kind, recorded_sandboxes, sandbox_ids from agent_env.task_step.context import TaskStepContext -from agent_env.task_step.task_steps.teardown_sandboxes import TORN_DOWN_KEY _COLORS = {Outcome.PASSED: "green", Outcome.BELOW_ONE: "yellow", Outcome.UNSCORED: "yellow", Outcome.FAILED: "red", Outcome.CANCELLED: "yellow"} diff --git a/src/agent_env/cli/task/run.py b/src/agent_env/cli/task/run.py index dce593ff8..70310ad17 100644 --- a/src/agent_env/cli/task/run.py +++ b/src/agent_env/cli/task/run.py @@ -15,7 +15,7 @@ from agent_env.cli import submitted_runs from agent_env.cli.identity import get_agent_env_client_id from agent_env.cli.teardown_output import echo_teardown -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.runner.runner import KEEP_KEY, RunStatus from agent_env.store.ids import derive_id, fs_safe, is_local_id, validate_local_id from agent_env.store.routing import run_scope diff --git a/src/agent_env/computers/README.md b/src/agent_env/computers/README.md index fd03b9515..aaa54ccc3 100644 --- a/src/agent_env/computers/README.md +++ b/src/agent_env/computers/README.md @@ -1,7 +1,8 @@ Base computers that come with agent-env, each a `computer` artifact a computer env composes a driver and apps onto. `artifacts/ubuntu-24.04/` is Ubuntu 24.04 (`amd64` and `arm64`) with XFCE on Xvfb, D-Bus and AT-SPI. tini starts -supervisord, which runs every program as the session user, `user` (uid 1000); nothing in it runs as root. Its +supervisord, which runs every program as the session user, `user` (uid 1000); nothing in it runs as root. Its home +has a desktop's standard folders (`Desktop`, `Documents`, `Downloads`, `Pictures` and the rest, from `xdg-user-dirs`). Its container form's image is built from the folder's `Dockerfile`, and a release publishes it as `ghcr.io/scaleapi/agentenv-ubuntu-24.04:src-`, tagged by its sources, so a run pulls it by digest instead of building it. Name it from another bundle as `ubuntu-24.04`, the alias `aliases.toml` gives it, or by its id, @@ -21,7 +22,8 @@ with `startsecs=0`, so X dying however often never leaves it `FATAL`, as the XFC A computer env composes a driver onto it at put: an `envs//env.toml` of `type = "computer"` names `computer = "ubuntu-24.04"` and `driver = "reference"`, and its put builds an image of the two for each arch this -machine's Docker can build. +machine's Docker can build, then boots the one this machine runs natively and probes it. The shipped `desktop` example +is that env, and `agent-env run desktop` deploys it. This bundle has no tasks of its own: other bundles pull its computers by alias, and `agent-env run computers` has nothing to run. diff --git a/src/agent_env/computers/artifacts/ubuntu-24.04/Dockerfile b/src/agent_env/computers/artifacts/ubuntu-24.04/Dockerfile index 43e83a842..d46579c51 100644 --- a/src/agent_env/computers/artifacts/ubuntu-24.04/Dockerfile +++ b/src/agent_env/computers/artifacts/ubuntu-24.04/Dockerfile @@ -1,11 +1,12 @@ # The ubuntu-24.04 base computer: Ubuntu 24.04 with XFCE on Xvfb, D-Bus and AT-SPI, every program run by supervisord -# as the session user. Composing a driver or an app onto it adds its services as programs in /etc/supervisor/conf.d. +# as the session user, whose home has a desktop's standard folders (Documents, Downloads, ...). Composing a driver or an +# app onto it adds its services as programs in /etc/supervisor/conf.d. FROM public.ecr.aws/docker/library/ubuntu:24.04@sha256:24737cef17f4e22b1977df7d9a05e77c4e28b7da5b8911f8d8ccda136bf3573c ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ xvfb x11-utils x11-xserver-utils libxcb1 libxcb-xinerama0 \ xfce4-session xfwm4 xfce4-panel xfdesktop4 xfce4-settings xfconf thunar \ - dbus dbus-x11 at-spi2-core \ + dbus dbus-x11 at-spi2-core xdg-user-dirs \ supervisor tini \ python3 python3-venv python3-tk \ fonts-dejavu-core locales tzdata ca-certificates procps \ @@ -13,6 +14,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && sed -i 's/^# *en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen \ && userdel --remove ubuntu \ && useradd --create-home --uid 1000 --shell /bin/bash user \ + && runuser -u user -- xdg-user-dirs-update \ && mkdir -p /etc/supervisor/conf.d /var/log/agentenv /run/agentenv \ && chown user:user /var/log/agentenv /run/agentenv COPY supervisord.conf /etc/supervisor/supervisord.conf diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index 6193e30f9..349b45d22 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -10,8 +10,9 @@ alias `aliases.toml` gives it, or by its id, `@local/agentenv-framework/drivers/ A run that names it writes it: its files, what its class declares for each platform it supports, read without `mss`, `python-xlib` or PyAV installed, and how a computer installs it. Its `env.toml`'s `[install.linux]` has a computer env's composer put the folder in `/opt/agentenv/drivers/reference/`, install `requirements.txt` into a venv -there, and run `agentenv_protocol.computer.serve driver.py:ReferenceDriver` as a session service on `port`, 18765. It deploys inside a computer, which agent-env can't create yet, so a deploy of it is -refused until then. `python -m agentenv_protocol.computer.conformance ` checks it, or any driver, where it runs. +there, and run `agentenv_protocol.computer.serve driver.py:ReferenceDriver` as a session service on `port`, 18765. It deploys only inside a computer env, as +the shipped `desktop` example's driver. `python -m agentenv_protocol.computer.conformance ` checks it, or any +driver, where it runs. This bundle has no tasks of its own: other bundles pull its drivers by alias, and `agent-env run drivers` has nothing to run. diff --git a/src/agent_env/env/env.py b/src/agent_env/env/env.py index 56dedcf68..c92199f7f 100644 --- a/src/agent_env/env/env.py +++ b/src/agent_env/env/env.py @@ -356,6 +356,12 @@ def provider_toml(self, out: ExportContext, env_provider_type: str) -> dict: async def deploy(self, **kwargs) -> DeployedEnv: raise NotImplementedError(f"{type(self).__name__} must implement deploy()") + def deploy_refusal(self, *, sandbox_type: Optional[str] = None, env_params: Optional[dict] = None, + **options) -> Optional[str]: + """Why deploy() would refuse these options before building anything, or None. A type that checks its options + overrides this; one that doesn't takes no ``env_params``.""" + return f"{self.type} envs take no env_params" if env_params else None + @classmethod async def from_deployed_env(cls, deployed: DeployedEnv) -> Self: """The env a running deployment serves, for its caller to drive. A type that can be @@ -364,6 +370,11 @@ async def from_deployed_env(cls, deployed: DeployedEnv) -> Self: """ raise NotImplementedError(f"{cls.__name__} cannot be rebuilt from a deployment") + async def close(self) -> None: + """Release what deploy() made. Teardown calls it on each env a run deployed before it terminates what's left by + sandbox id; a type that can't close raises this.""" + raise NotImplementedError(f"{type(self).__name__} does not implement close()") + async def reset(self, deployed: "DeployedEnv") -> None: """Return an already-deployed env to a clean between-tasks state. diff --git a/src/agent_env/env/envs/_deployment.py b/src/agent_env/env/envs/_deployment.py index 2d4db4d75..88b169943 100644 --- a/src/agent_env/env/envs/_deployment.py +++ b/src/agent_env/env/envs/_deployment.py @@ -134,19 +134,20 @@ async def load_by_signed_url(env: MCPServerEnv | WebsiteEnv, file_artifact: File async def close_deployed(deployed: DeployedEnv, env_class: type[Env]) -> None: - """Close what a task deployed of an env_class env: a built-in's through its env, a plugin's by terminating the sandboxes its record names.""" - from agent_env.task_step.task_steps.teardown_sandboxes import _env_sandbox_ids, _terminate + """Close what a task deployed of an env_class env: through the env, unless a plugin's provider deployed a built-in, whose + record's sandboxes are terminated instead. A plugin's own env class, which has no env provider of agent-env's, closes itself.""" + from agent_env.task.teardown import RecordedSandbox, env_sandbox_ids, terminate env = await env_class.from_deployed_env(deployed) - if as_builtin(env._env_provider): + if not hasattr(env, "_env_provider") or as_builtin(env._env_provider): await env.close() return - sandboxes = _env_sandbox_ids(deployed) + sandboxes = env_sandbox_ids(deployed) if not sandboxes: logger.warning(f"env '{deployed.env_id}' was deployed by env_provider_type '{deployed.env_provider_type}' outside " "agent-env's sandboxes, so it isn't closed here; the deployment must end by itself") - results = await asyncio.gather(*(_terminate(sandbox_id, sandbox_type, "env") for sandbox_id, sandbox_type in sandboxes), - return_exceptions=True) + results = await asyncio.gather(*(terminate(RecordedSandbox(sandbox_id, sandbox_type, "env")) + for sandbox_id, sandbox_type in sandboxes), return_exceptions=True) for (sandbox_id, _), result in zip(sandboxes, results): if isinstance(result, BaseException): logger.warning(f"env '{deployed.env_id}': terminate {sandbox_id} failed: {result!r}") @@ -280,13 +281,15 @@ def provider_refusal(provider_class: type[EnvironmentProvider], env_class: type[ def option_refusal(env: Env, provider: EnvironmentProvider | type[EnvironmentProvider], options: dict) -> str | None: - """Why provider, or a provider of that class, wouldn't take options: one its deploy() doesn't name, set away from the env's deploy() default.""" + """Why provider, or a provider of that class, wouldn't take options: one its deploy() doesn't name, set away from the env's deploy() default, + or one the env's deploy() doesn't name either.""" from agent_env.providers.env_providers.env_provider import _SandboxEnvironmentProvider taken = inspect.signature(provider.deploy).parameters if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in taken.values()): return None defaults = inspect.signature(type(env).deploy).parameters - refused = [name for name, value in options.items() if name not in taken and value != defaults[name].default] + refused = [name for name, value in options.items() + if name not in taken and (name not in defaults or value != defaults[name].default)] if not refused: return None builtin = issubclass(provider if isinstance(provider, type) else type(provider), _SandboxEnvironmentProvider) diff --git a/src/agent_env/env/envs/computer/boot.py b/src/agent_env/env/envs/computer/boot.py new file mode 100644 index 000000000..37bee0cfd --- /dev/null +++ b/src/agent_env/env/envs/computer/boot.py @@ -0,0 +1,118 @@ +"""Booting a composed computer: one build of it in a container, on a sandbox provider that runs its arch natively, with +its settings in the environment the base and the driver read them from, until its driver says it's ready. A deploy and +gate 3 boot computers this way.""" + +from __future__ import annotations + +import asyncio +import contextlib +import json +import secrets +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, Optional + +from agentenv_protocol import client as protocol_v1 +from agentenv_protocol.computer import COMPUTER_READY, SETTINGS_ENV + +from agent_env.config import get_config +from agent_env.providers.env_providers.env_provider import _size_kwargs +from agent_env.providers.sandbox_providers.local_docker.provider import remove_local_work_dir + +if TYPE_CHECKING: + from agent_env.artifact.artifacts.computer import ComputerArtifact + from agent_env.attribution import Attribution + from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy, Sandbox + from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider + +#: How long a computer may take from its container starting to its driver saying it's ready. +READY_TIMEOUT_S = 180.0 +_POLL_S = 0.5 +#: How a setting a record keeps shows a secret's or a generated one's value, which it never holds. +WITHHELD = "" + + +@dataclass +class Booted: + sandbox: Sandbox + url: str # the driver's address, which its card's paths join onto + card: dict + + +def native_links(sandbox_provider: SandboxProvider, arch: str) -> list[SandboxProvider]: + """The providers a boot of ``arch`` may run on, in the order a chain tries them: those that run it natively.""" + return [link for link in sandbox_provider.links if f"linux/{arch}" in link.native_platforms()] + + +def settings_environment(computer: ComputerArtifact, values: dict[str, Any], *, environment_name: str, + driver_settings: Optional[dict] = None) -> dict[str, str]: + """The environment a boot of ``computer`` gives it: each deploy-phase setting as ``AGENTENV_SETTING_``, from + ``values`` over its default, minted afresh when it's generated and resolved through the secret store when it's a + secret; ``driver_settings``, when there are any, in ``AGENTENV_COMPUTER_SETTINGS``; and the name its card is served + under.""" + environment = {"ENVIRONMENT_NAME": environment_name} + for name, setting in computer.config.items(): + if setting.phase != "deploy": + continue + value = secrets.token_urlsafe(24) if setting.generate else values.get(name, setting.default) + if setting.secret and not setting.generate: + value = get_config().resolve_reference(value) + environment[f"AGENTENV_SETTING_{name.upper()}"] = str(value) + if driver_settings: + environment[SETTINGS_ENV] = json.dumps(driver_settings) + return environment + + +def recorded_settings(computer: ComputerArtifact, values: dict[str, Any]) -> dict[str, Any]: + """The deploy-phase settings a boot applies as a record keeps them, a secret's and a generated one's withheld.""" + return {name: WITHHELD if setting.secret or setting.generate else values.get(name, setting.default) + for name, setting in computer.config.items() if setting.phase == "deploy"} + + +async def boot(sandbox_provider: SandboxProvider, *, image: str, port: int, environment: dict[str, str], + network_policy: Optional[NetworkPolicy], timeout: int, attribution: Optional[Attribution], + disk_size_gb: float = 10, cpu: Optional[float] = None, memory_mb: Optional[int] = None) -> Booted: + """``image`` booted with ``environment`` on ``sandbox_provider``, which the caller chose to run its arch natively, + its driver's ``port`` reached on the sandbox, once the driver says the computer is ready; a boot that fails is torn + down.""" + sandbox = await sandbox_provider.create_container( + image_name=image, port=port, env=environment, network_policy=network_policy, + disk_size_gb=disk_size_gb, timeout=timeout, attribution=dict(attribution or {}), **_size_kwargs(cpu, memory_mb)) + try: + url = sandbox.tunnel_urls[port] + return Booted(sandbox, url, await ready(url, READY_TIMEOUT_S)) + except BaseException: + await discard(sandbox) + raise + + +async def discard(sandbox: Sandbox) -> None: + """Tear down a sandbox no run records, as a run's teardown would: terminate it, then remove its work folder when it + ran on this machine.""" + try: + await sandbox.terminate() + except Exception: + return + if sandbox.ON_THIS_MACHINE: + with contextlib.suppress(OSError): + await asyncio.to_thread(remove_local_work_dir, sandbox.sandbox_id) + + +async def ready(url: str, timeout: float) -> dict: + """The card of the driver at ``url`` once its ``computer-ready`` status says it's ready; raises TimeoutError, naming + what it last saw, when it isn't within ``timeout`` seconds.""" + last = "nothing answered" + try: + async with asyncio.timeout(timeout): + while True: + try: + card = await protocol_v1.get_card(url, timeout=5) + status = await protocol_v1.invoke_extension(url, card, COMPUTER_READY, None, 10, method="status") + if status.get("ready"): + return card + last = "not ready: " + "; ".join(f"{check['name']}: {check.get('detail')}" + for check in status.get("checks", []) if not check.get("ok")) + except Exception as e: + last = f"{type(e).__name__}: {e}" + await asyncio.sleep(_POLL_S) + except TimeoutError: + raise TimeoutError(f"the computer at {url} wasn't ready within {timeout:.0f} s ({last})") from None diff --git a/src/agent_env/env/envs/computer/composer.py b/src/agent_env/env/envs/computer/composer.py index 52e0ba33e..053f93412 100644 --- a/src/agent_env/env/envs/computer/composer.py +++ b/src/agent_env/env/envs/computer/composer.py @@ -6,6 +6,7 @@ from __future__ import annotations +import asyncio import copy import hashlib import importlib.metadata @@ -25,13 +26,15 @@ from collections.abc import Callable from typing import TYPE_CHECKING, Any +from agentenv_protocol.computer.conformance import run_probe + from agent_env.artifact.artifacts.computer import ComputerArtifact from agent_env.artifact.artifacts.docker_image import DockerImageArtifact, docker_login, push_local_image from agent_env.artifact.artifacts.file import FileArtifact from agent_env.config import get_config from agent_env.entity.export import Unwritable, checked_path from agent_env.env.env import Env -from agent_env.env.envs.computer import gate +from agent_env.env.envs.computer import boot, gate from agent_env.env.envs.computer.install import Install from agent_env.providers.sandbox_providers.host_images import build_for, host_platform, platform_tag from agent_env.store.base import ConcurrentModificationError @@ -42,6 +45,9 @@ from agent_env.utils.docker.build import DockerBuildError, build_image if TYPE_CHECKING: + from agentenv_protocol.computer.conformance import ProbeReport + + from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider from agent_env.env.envs.computer.env import ComputerEnv from agent_env.env.envs.computer_driver import ComputerDriverEnv @@ -61,6 +67,10 @@ # A pair's statuses once its image is in the store. _BUILT = ("unverified", "ready") _WRITE_TRIES = 4 +#: How long a probe's computer may run before its sandbox provider stops it. +_PROBE_TTL_S = 1800 +#: How long the probe at full may take on a computer that booted: about ten seconds on a healthy one. +_PROBE_TIMEOUT_S = 120 _PROTOCOL = "agentenv-framework-protocol" # What building the protocol from its source reads, beside src/. _PROTOCOL_FILES = ("pyproject.toml", "README.md", "LICENSE", "NOTICE", "THIRD_PARTY_NOTICES.md") @@ -68,23 +78,25 @@ def compose(env: ComputerEnv, pairs: list[tuple[str, str]] | None = None, on_build: Callable[[], None] | None = None) -> None: - """Build each of ``pairs``, by default each form the base has in each arch the env composes, in parallel, - recording each as it starts and finishes, then write the env's computer over every pair built so far. ``on_build`` - is called once, just before the first image is built here; reusing them all calls it never.""" + """Build each of ``pairs``, by default each form the base has in each arch the env composes, in parallel, and probe + each one a configured sandbox provider runs natively (gate 3), recording each as it starts and finishes, then write + the env's computer over every pair built so far. ``on_build`` is called once, just before the first image is built + here; reusing them all calls it never.""" base, driver = env.computer, env.driver building = _once(on_build) pairs = pairs if pairs is not None else [(form, arch) for form in gate.forms(base) for arch in gate.arches(base, driver)] record = _Record(env) record.started(pairs) with ThreadPoolExecutor(max_workers=max(len(pairs), 1)) as pool: - futures = {pool.submit(_built, env, base, driver, form, arch, building): (form, arch) for form, arch in pairs} + futures = {pool.submit(_composed, env, base, driver, form, arch, building): (form, arch) for form, arch in pairs} for future in as_completed(futures): form, arch = futures[future] result = future.result() if result["status"] == "failed": logger.warning(f"{env.stored_id}: {form}/{arch} failed at {result['stage']}: {result['reason']}") else: - logger.info(f"{env.stored_id}: {form}/{arch} {'reused' if result['reused'] else 'built'}: {result['image']}") + logger.info(f"{env.stored_id}: {form}/{arch} {'reused' if result['reused'] else 'built'}, " + f"{result['status']}: {result['image']}") record.finished(form, arch, result) record.computer(_computer(env, base, record.built())) @@ -96,10 +108,70 @@ def current(env: ComputerEnv) -> bool: def pending(env: ComputerEnv) -> list[tuple[str, str]]: - """The forms and arches of ``env`` not yet built: never started, failed, or left building by a put that stopped.""" - composed = env.metadata.get("composed", {}) - return [(form, arch) for form in gate.forms(env.computer) for arch in gate.arches(env.computer, env.driver) - if composed.get(form, {}).get(arch, {}).get("status") not in _BUILT] + """The forms and arches of ``env`` left to compose: never started, failed, left building by a put that stopped, or + built but unprobed though a configured sandbox provider runs the arch natively.""" + from agent_env.providers import get_env_sandbox_provider + + composed, provider = env.metadata.get("composed", {}), get_env_sandbox_provider() + statuses = {(form, arch): composed.get(form, {}).get(arch, {}).get("status") + for form in gate.forms(env.computer) for arch in gate.arches(env.computer, env.driver)} + return [(form, arch) for (form, arch), status in statuses.items() + if status not in _BUILT or (status == "unverified" and boot.native_links(provider, arch))] + + +def _composed(env: ComputerEnv, base: ComputerArtifact, driver: ComputerDriverEnv, form: str, arch: str, + building: Callable[[], None]) -> dict: + """``form`` and ``arch`` built or reused, then probed when it built (gate 3).""" + result = _built(env, base, driver, form, arch, building) + return _probed(env, driver, form, arch, result) if result["status"] == "unverified" else result + + +def _probed(env: ComputerEnv, driver: ComputerDriverEnv, form: str, arch: str, result: dict) -> dict: + """Gate 3: the build of ``result`` booted on the first configured sandbox provider that runs ``arch`` natively, and + probed at ``full``: ready, with the driver's card, or failed at the probe; each with a summary of the probe and its + report stored. Still unverified, saying why, when no configured provider runs ``arch`` natively.""" + from agent_env.providers import get_env_sandbox_provider + + if not (links := boot.native_links(get_env_sandbox_provider(), arch)): + return {**result, "reason": f"no configured sandbox provider runs linux/{arch} natively to probe it"} + try: + report, card, provider = asyncio.run(_probe(links, env, driver, result["image"])) + except TimeoutError: + return _failed(env, form, arch, "probe", result["key"], f"the probe didn't finish within {_PROBE_TIMEOUT_S} s") + except Exception as e: + return _failed(env, form, arch, "probe", result["key"], f"it didn't boot: {e}") + stored = FileArtifact.put_bytes(derive_id(env.stored_id, f"probe_report_{form}_{arch}"), description=( + f"the {form}/{arch} probe of {env.stored_id} v{env.version}"), filename="probe.json", + content=report.model_dump_json(indent=1).encode(), content_type="application/json") + probe = {"level": report.level, "provider": provider, "passed": report.passed, "checks": report.counts(), + "extensions": report.extensions, "report": {"id": stored.id, "version": stored.version}, + "probed_at": _now()} + if not report.passed: + failing = "; ".join(f"{c.check} {c.name}: {c.detail}" for c in report.checks if c.status == "failed") + return {**result, "status": "failed", "stage": "probe", "reason": f"the probe failed: {failing}", + "failed_at": _now(), "probe": probe} + return {**result, "status": "ready", "card": card, "probe": probe} + + +async def _probe(links: list[SandboxProvider], env: ComputerEnv, driver: ComputerDriverEnv, + image: str) -> tuple[ProbeReport, dict, str]: + """``image`` booted at the env's settings, as a deploy boots it, on the first of ``links`` that boots it, probed at + ``full`` within ``_PROBE_TIMEOUT_S`` and torn down; the report, the driver's card and the provider's name.""" + environment = boot.settings_environment(env.computer, _values(env), environment_name=env.environment_name) + problems = [] + for link in links: + try: + booted = await boot.boot(link, image=image, port=driver.port, environment=environment, + network_policy=None, timeout=_PROBE_TTL_S, attribution=None) + except Exception as e: + problems.append(f"{type(link).__name__}: {e}") + continue + try: + async with asyncio.timeout(_PROBE_TIMEOUT_S): + return await run_probe(booted.url, "full"), booted.card, type(link).__name__ + finally: + await boot.discard(booted.sandbox) + raise RuntimeError("; ".join(problems)) def _built(env: ComputerEnv, base: ComputerArtifact, driver: ComputerDriverEnv, form: str, arch: str, @@ -245,14 +317,15 @@ def _protocol_source() -> Path | None: def _build_settings(env: ComputerEnv, base: ComputerArtifact) -> dict[str, Any]: - """The env's values of the base's build-phase settings, which the composed image fixes: its display is the - resolution's, for a base that reads that at build.""" - values = {**env.computer_settings, **({"resolution": _resolution(env.display)} if env.display else {})} + """The env's values of the base's build-phase settings, which the composed image fixes.""" + values = _values(env) return {name: values[name] for name, setting in base.config.items() if setting.phase == "build" and name in values} -def _resolution(display: dict) -> str: - return f"{display['width']}x{display['height']}" +def _values(env: ComputerEnv) -> dict[str, Any]: + """The env's values of its base's settings: its ``computer_settings``, and its display as the resolution.""" + display = env.display + return {**env.computer_settings, **({"resolution": f"{display['width']}x{display['height']}"} if display else {})} def _dockerfile(source: str, layer: str, install: Install, base: ComputerArtifact, build_settings: dict, @@ -348,15 +421,16 @@ def _computer(env: ComputerEnv, base: ComputerArtifact, built: dict[str, str]) - image = DockerImageArtifact.put_ref(derive_id(env.stored_id, "computer_image"), description=( f"{env.stored_id}'s composed computer"), image_name=index) config = {name: setting.model_dump() for name, setting in base.config.items()} - for name, value in env.computer_settings.items(): + for name, value in _values(env).items(): config[name]["default"] = value - if env.display is not None: - config["resolution"]["default"] = _resolution(env.display) computer = ComputerArtifact.put( id=derive_id(env.stored_id, "computer"), description=f"{base.id} composed with {env.driver.id} for {env.stored_id}", arch=arches, os=base.os.model_dump(), session=base.session.model_dump(), config=config, forms={"container": {"image": {"id": image.id, "version": image.version}, "init": base.forms.container.init, "input_channel": base.forms.container.input_channel}}) + composed = env.metadata.get("composed", {}).get("container", {}) + if probes := {arch: composed[arch]["probe"] for arch in arches if "probe" in composed.get(arch, {})}: + computer.merge_metadata({"validated_boot": {"container": probes}}) return {"id": computer.id, "version": computer.version} @@ -386,8 +460,8 @@ def _now() -> str: class _Record: """The env's ``composed`` metadata, by form and arch. Each result is merged into the stored table pair by pair, the - table read again when another build wrote it first, so builds of one env at once keep each other's pairs, and a pair - built stays built whatever another build of it records.""" + table read again when another build wrote it first, so builds of one env at once keep each other's pairs, and a + result never undoes one that got further (``_rank``).""" def __init__(self, env: ComputerEnv): self.env, self.lock = env, threading.Lock() @@ -416,7 +490,7 @@ def _merge(self, results: dict[tuple[str, str], dict]) -> None: composed = copy.deepcopy(self.env.metadata.get("composed", {})) for (form, arch), result in results.items(): stored = composed.setdefault(form, {}).get(arch, {}) - if stored.get("status") not in _BUILT or result["status"] in _BUILT: + if _rank(result) >= _rank(stored): composed[form][arch] = {"started_at": stored.get("started_at"), **result} try: self.env.update_metadata({**self.env.metadata, "composed": composed}) @@ -425,3 +499,11 @@ def _merge(self, results: dict[tuple[str, str], dict]) -> None: if attempt == _WRITE_TRIES - 1: raise self.env.metadata = Env.get(self.env.stored_id, self.env.version).metadata + + +def _rank(result: dict) -> int: + """How far a pair got: ready, then built and either unprobed or failed at its probe, then anything short of an + image. A result takes a stored one's place only when it got at least as far.""" + if result.get("status") == "ready": + return 2 + return 1 if result.get("status") == "unverified" or result.get("stage") == "probe" else 0 diff --git a/src/agent_env/env/envs/computer/env.py b/src/agent_env/env/envs/computer/env.py index a38ae5a53..940edd22e 100644 --- a/src/agent_env/env/envs/computer/env.py +++ b/src/agent_env/env/envs/computer/env.py @@ -4,24 +4,41 @@ from __future__ import annotations +import logging from collections.abc import Callable from typing import TYPE_CHECKING, Any, ClassVar, Optional, Self from agent_env.artifact.artifacts.computer import ComputerArtifact, SettingValue +from agent_env.attribution import Attribution from agent_env.entity.refs import EntityRef -from agent_env.env.env import Env +from agent_env.env.env import Env, gateway_url_of +from agent_env.env.envs._deployment import ( + as_builtin, + builtin_provider_for, + close_replaced, + deploy_refusal, + deploy_through_provider, + provider_or_class, +) from agent_env.env.envs.computer import composer, gate +from agent_env.env.envs.computer import params as computer_params from agent_env.env.envs.computer_driver import ComputerDriverEnv +from agent_env.env.gateway import GatewayMode from agent_env.env.store import get_env_store +from agent_env.providers.env_providers.env_computer_provider import end_recordings if TYPE_CHECKING: from agent_env.bundle.authoring import AuthoringContext from agent_env.env.env import DeployedEnv + from agent_env.providers.env_providers.env_provider import EnvironmentProvider + +logger = logging.getLogger(__name__) class ComputerEnv(Env): type: ClassVar[str] = "computer" description = "A computer: a base computer and a driver, composed at put into an image for each form and arch" + env_provider_type: ClassVar[str] = "computer" toml_refs: ClassVar[tuple[EntityRef, ...]] = (EntityRef.artifact("computer", artifact_type="computer"), EntityRef.env("driver", env_type=ComputerDriverEnv.type)) @@ -33,6 +50,11 @@ def __init__(self, id: str, version: Optional[int], environment_name: str, compu self.computer, self.driver = computer, driver self.display, self.network = display, network self.computer_settings = computer_settings or {} + self._sandbox = None + self._env_provider: Optional[EnvironmentProvider] = builtin_provider_for(self.env_provider_type) + self._gateway_url = None + self._deployed: Optional[DeployedEnv] = None + self._replaced: list[tuple] = [] # the provider and sandbox of each deployment a later deploy replaced, for close() def to_dict(self) -> dict: return {**super().to_dict(), "service_name": self.environment_name, @@ -88,8 +110,83 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerEnv: except ValueError as e: ctx.refuse([str(e)]) - async def deploy(self, **kwargs) -> DeployedEnv: - statuses = {f"{form}/{arch}": result.get("status") for form, by_arch in self.metadata.get("composed", {}).items() - for arch, result in by_arch.items()} - raise NotImplementedError(f"{self.id} is a computer env, which agent-env can't boot yet; its builds are " - f"{statuses or 'none yet'}") + def computer_artifact(self) -> ComputerArtifact: + """The env's computer: its base composed with its driver, over the images built.""" + return ComputerArtifact.get(self.metadata["computer"]["id"], self.metadata["computer"]["version"]) + + def ready_builds(self, form: Optional[str] = None) -> list[tuple[str, str, dict]]: + """Each ready build, as (form, arch, record), in the base's order of forms and of arches; only + ``form``'s when it's given.""" + composed, arches = self.metadata.get("composed", {}), self.computer.arch + return [(name, arch, composed[name][arch]) for name in gate.forms(self.computer) if form in (None, name) + for arch in arches if composed.get(name, {}).get(arch, {}).get("status") == "ready"] + + def build_statuses(self) -> dict[str, str]: + return {f"{form}/{arch}": result.get("status") for form, by_arch in self.metadata.get("composed", {}).items() + for arch, result in by_arch.items()} + + async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, + gateway_mode: GatewayMode = GatewayMode.PERFORMANCE, cpu: float | None = None, + memory_mb: int | None = None, sandbox_type: str | None = None, env_state_type: str | None = None, + env_state_instance_id: str | None = None, *, attribution: Optional[Attribution] = None, + env_params: Optional[dict] = None) -> DeployedEnv: + return await deploy_through_provider( + self, environment_name=self.environment_name, ttl_seconds=ttl_seconds, sandbox_type=sandbox_type, + disk_size_gb=disk_size_gb, gateway_mode=gateway_mode, cpu=cpu, memory_mb=memory_mb, + env_state_type=env_state_type, env_state_instance_id=env_state_instance_id, attribution=attribution, + env_params=env_params) + + def deploy_refusal(self, *, sandbox_type: str | None = None, env_params: Optional[dict] = None, + **options) -> str | None: + """Why deploy() would refuse before booting anything, or None: an option the computer provider doesn't take, a + problem with ``env_params``, or a sandbox provider that runs none of the env's arches natively or can't enforce + its network.""" + from agent_env.providers import build_sandbox_provider, get_env_sandbox_provider + from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy + + if refusal := deploy_refusal(self, provider_or_class(self), {**options, "env_params": env_params}): + return refusal + _, _, problems = computer_params.checked(self, env_params) + sandbox_provider = build_sandbox_provider(sandbox_type) if sandbox_type else get_env_sandbox_provider() + links, arches = sandbox_provider.links, gate.arches(self.computer, self.driver) + if not any(f"linux/{arch}" in link.native_platforms() for link in links for arch in arches): + problems.append(f"{', '.join(type(link).__name__ for link in links)} runs none of its arches " + f"({', '.join(arches)}) natively") + policy = NetworkPolicy.from_dict(self.network) if self.network else None + if policy is not None and not any(link.supports_network_policy(policy) for link in links): + problems.append(f"{', '.join(type(link).__name__ for link in links)} can't enforce its network") + return f"computer env {self.id!r}: " + "; ".join(problems) if problems else None + + @classmethod + async def from_deployed_env(cls, deployed: DeployedEnv) -> ComputerEnv: + env = Env.get(deployed.env_id, deployed.env_version) + if not isinstance(env, ComputerEnv): + raise TypeError(f"Expected ComputerEnv, got {type(env).__name__}") + env._sandbox = await as_builtin(env._env_provider)._reattach(env, deployed) + env._gateway_url = gateway_url_of(deployed) + env._instance_id = deployed.instance_id + env._deployed = deployed + return env + + async def reset(self, deployed: DeployedEnv) -> None: + """Return the computer to boot state: its recordings ended, its container replaced with a fresh one of the same + image, on the same sandbox and port, so its address stays. An agent's MCP session ends with the old driver; one + held across the reset is answered 404, and MCP clients then start a new one.""" + await as_builtin(self._env_provider).reboot(self, deployed) + + async def close(self) -> None: + """End the computer's recordings, so what they recorded reaches the store, then tear it down.""" + if self._deployed is not None and self._sandbox is not None and self._deployed.sandbox_id == self._sandbox.sandbox_id: + await end_recordings(self._deployed, "teardown") + await close_replaced(self) + if self._env_provider is not None: + try: + await self._env_provider.close() + except Exception as e: + logger.warning(f"Failed to close env provider: {e}") + if self._sandbox is not None: + try: + await self._sandbox.terminate() + except Exception as e: # a cancellation propagates, and keeps the sandbox for the next close() + logger.warning(f"Failed to terminate sandbox {self._sandbox.sandbox_id}: {e}") + self._sandbox = None diff --git a/src/agent_env/env/envs/computer/gate.py b/src/agent_env/env/envs/computer/gate.py index a1334e837..187be2d7a 100644 --- a/src/agent_env/env/envs/computer/gate.py +++ b/src/agent_env/env/envs/computer/gate.py @@ -8,6 +8,7 @@ from agentenv_protocol.computer.platform import Platform, supported from agent_env.artifact.artifacts.computer import FORMS, ComputerArtifact, SettingValue +from agent_env.config.loader import ENV_REF_PREFIX, SECRET_REF_PREFIX from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy from agent_env.store.image_store.oci_registry_credentials import names_registry @@ -29,7 +30,7 @@ def check(base: ComputerArtifact, driver: ComputerDriverEnv, *, apps: list, disp problems += _image(base) problems += _display(base, display, computer_settings) problems += _network(network) - values, invalid = _computer_settings(base, computer_settings) + values, invalid = check_computer_settings(base, computer_settings) return values, problems + invalid @@ -45,15 +46,22 @@ def arches(base: ComputerArtifact, driver: ComputerDriverEnv) -> list[str]: return [arch for arch in base.arch if arch in matching] -def _computer_settings(base: ComputerArtifact, values: Any) -> tuple[dict[str, SettingValue], list[str]]: - """``values``, of the base's settings by name, and the problems with them: a setting the base doesn't declare, or a - value of another type than its default's.""" +def check_computer_settings(base: ComputerArtifact, values: Any) -> tuple[dict[str, SettingValue], list[str]]: + """``values``, of the base's settings by name, and the problems with them: a setting the base doesn't declare, a + value of another type than its default's, or a secret's given as anything but a bare reference, since the env, the + task and the deploy record all keep what's given; a secret's reference is a string whatever its default is.""" if not isinstance(values, dict): return {}, [f"computer_settings must be a table of the computer's setting values, not {values!r}"] checked, problems = {}, [] for name, value in values.items(): if (setting := base.config.get(name)) is None: problems.append(f"computer_settings.{name}: {base.id} declares no setting {name!r}") + elif setting.secret: + if _names_a_secret(value): + checked[name] = value + else: + problems.append(f"computer_settings.{name} is a secret, so it takes a {SECRET_REF_PREFIX}KEY or " + f"{ENV_REF_PREFIX}NAME reference with no ?default, not its value") elif type(value) is not type(setting.default): problems.append(f"computer_settings.{name} must be a {type(setting.default).__name__}, like its default " f"{setting.default!r}, not {value!r}") @@ -62,6 +70,12 @@ def _computer_settings(base: ComputerArtifact, values: Any) -> tuple[dict[str, S return checked, problems +def _names_a_secret(value: Any) -> bool: + """Whether ``value`` names a secret rather than holding it: a bare reference, as a stored context keeps one.""" + return (isinstance(value, str) and value.startswith((ENV_REF_PREFIX, SECRET_REF_PREFIX)) and "?" not in value + and value not in (ENV_REF_PREFIX, SECRET_REF_PREFIX)) + + def _platform(base: ComputerArtifact, driver: ComputerDriverEnv) -> list[str]: os = base.os if not _matching(base, driver): diff --git a/src/agent_env/env/envs/computer/params.py b/src/agent_env/env/envs/computer/params.py new file mode 100644 index 000000000..a49bd1f38 --- /dev/null +++ b/src/agent_env/env/envs/computer/params.py @@ -0,0 +1,54 @@ +"""What a deploy of a computer env takes as ``deploy_env`` ``env_params``: the form to boot, ``computer_settings``, +values of the base's deploy-phase settings, and ``driver_settings``, the driver's ``ComputerSettings``, such as the +default coordinate space its tools describe. The env's display and network are its own, which no deploy overrides.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Optional + +from agentenv_protocol.computer import ComputerSettings +from pydantic import BaseModel, ConfigDict, Field, ValidationError + +from agent_env.artifact.artifacts.computer import SettingValue +from agent_env.env.envs.computer import gate + +if TYPE_CHECKING: + from agent_env.env.envs.computer.env import ComputerEnv + + +class ComputerParams(BaseModel): + model_config = ConfigDict(extra="forbid") + + form: Optional[str] = None + computer_settings: dict[str, Any] = Field(default_factory=dict) + driver_settings: dict[str, Any] = Field(default_factory=dict) + + +def checked(env: ComputerEnv, env_params: Optional[dict]) -> tuple[ComputerParams, dict[str, SettingValue], list[str]]: + """``env_params`` read for ``env``, its values of the base's settings, and every problem with them.""" + try: + params = ComputerParams.model_validate(env_params or {}) + except ValidationError as e: + return ComputerParams(), {}, _problems("env_params", e) + problems = [] + if params.form is not None and params.form not in (forms := gate.forms(env.computer)): + problems.append(f"env_params.form: {params.form!r} isn't one of {env.computer.id}'s forms ({', '.join(forms)})") + base = env.computer + values, invalid = gate.check_computer_settings(base, params.computer_settings) + problems += [f"env_params.{problem}" for problem in invalid] + for name in values: + if base.config[name].phase == "build": + problems.append(f"env_params.computer_settings: {name} is a build-phase setting, which only another image " + "changes") + if name == "resolution" and env.display is not None: + problems.append("env_params.computer_settings: the env's display sets the screen's size, which no deploy " + "changes") + try: + ComputerSettings.model_validate(params.driver_settings) + except ValidationError as e: + problems += _problems("env_params.driver_settings", e) + return params, values, problems + + +def _problems(at: str, error: ValidationError) -> list[str]: + return [f"{at}.{'.'.join(map(str, problem['loc']))}: {problem['msg']}" for problem in error.errors()] diff --git a/src/agent_env/env/envs/computer_driver.py b/src/agent_env/env/envs/computer_driver.py index b8b39f36a..be2b2c05e 100644 --- a/src/agent_env/env/envs/computer_driver.py +++ b/src/agent_env/env/envs/computer_driver.py @@ -91,9 +91,8 @@ def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: return fields async def deploy(self, **kwargs) -> DeployedEnv: - raise NotImplementedError(f"{self.id} is a computer_driver env, which deploys inside a computer, and " - "agent-env can't create a computer yet; serve the driver from an image of its own " - "as an mcp_server env meanwhile") + raise NotImplementedError(f"{self.id} is a computer_driver env, which deploys inside a computer, never on its " + "own: name it as a computer env's driver and deploy that") def _missing(os_type: str, install: Install, files: dict) -> list[str]: diff --git a/src/agent_env/env/envs/mcp_server.py b/src/agent_env/env/envs/mcp_server.py index 46128d48e..3f33a286e 100644 --- a/src/agent_env/env/envs/mcp_server.py +++ b/src/agent_env/env/envs/mcp_server.py @@ -112,8 +112,9 @@ async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gatew artifact_id=artifact_id, artifact_version=artifact_version, ) - def deploy_refusal(self, **options) -> str | None: - """Why deploy() would refuse these options before building anything, or None; raises, as deploy() does, for a type this process can't find.""" + def deploy_refusal(self, *, sandbox_type: str | None = None, **options) -> str | None: + """Why deploy() would refuse these options before building anything, or None; raises, as deploy() does, for a type this process can't find. + ``sandbox_type`` is deploy()'s own, which no provider takes.""" return deploy_refusal(self, provider_or_class(self), options) @classmethod diff --git a/src/agent_env/env/envs/multi_env.py b/src/agent_env/env/envs/multi_env.py index b157834d4..b64be5552 100644 --- a/src/agent_env/env/envs/multi_env.py +++ b/src/agent_env/env/envs/multi_env.py @@ -146,8 +146,9 @@ async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gatew _hand_record_to_child_envs(self, deployed_env) return deployed_env - def deploy_refusal(self, **options) -> str | None: - """Why deploy() would refuse these options before building anything, or None; raises, as deploy() does, for a type this process can't find.""" + def deploy_refusal(self, *, sandbox_type: str | None = None, **options) -> str | None: + """Why deploy() would refuse these options before building anything, or None; raises, as deploy() does, for a type this process can't find. + ``sandbox_type`` is deploy()'s own, which no provider takes.""" return deploy_refusal(self, provider_or_class(self), options) @classmethod diff --git a/src/agent_env/env/envs/website.py b/src/agent_env/env/envs/website.py index a35dd30e7..8a706c7cd 100644 --- a/src/agent_env/env/envs/website.py +++ b/src/agent_env/env/envs/website.py @@ -132,8 +132,9 @@ async def deploy(self, ttl_seconds: int = 10800, disk_size_gb: float = 10, gatew artifact_id=artifact_id, artifact_version=artifact_version, ) - def deploy_refusal(self, **options) -> str | None: - """Why deploy() would refuse these options before building anything, or None; raises, as deploy() does, for a type this process can't find.""" + def deploy_refusal(self, *, sandbox_type: str | None = None, **options) -> str | None: + """Why deploy() would refuse these options before building anything, or None; raises, as deploy() does, for a type this process can't find. + ``sandbox_type`` is deploy()'s own, which no provider takes.""" return deploy_refusal(self, provider_or_class(self), options) @classmethod diff --git a/src/agent_env/examples/desktop/README.md b/src/agent_env/examples/desktop/README.md new file mode 100644 index 000000000..f7ca52e3a --- /dev/null +++ b/src/agent_env/examples/desktop/README.md @@ -0,0 +1,15 @@ +A Linux desktop an agent uses through the `computer_*` tools: M0's smoke example of a computer env. + +`envs/desktop/env.toml` is a computer env of the shipped `ubuntu-24.04` base computer and the `reference` driver, +named by their aliases. A run composes it first: it builds an image of the base with the driver installed for each +arch this machine's Docker builds, and probes the one this machine runs natively, which takes a few minutes the first +time; a later run reuses the image. Deploying it boots that image in a container on local Docker. + +`tasks/probe.json` deploys it and runs the conformance probe at `full` against it, with `artifacts/check_probe/`. It +needs Docker and the local registry on `localhost:5000`, and no model. + +`tasks/agent.json` has Claude Code, given the computer's skill (`add_skills` `env_ids`), open the file +manager and make a folder named `Reports` in `Documents`, while the display is recorded; `artifacts/check_folder/` +checks the folder through the driver. It needs a model endpoint as well. + +Run them with `agent-env run desktop`, or one with `agent-env run desktop --task probe`. diff --git a/src/agent_env/examples/desktop/artifacts/check_folder/verify.py b/src/agent_env/examples/desktop/artifacts/check_folder/verify.py new file mode 100644 index 000000000..31d7de83d --- /dev/null +++ b/src/agent_env/examples/desktop/artifacts/check_folder/verify.py @@ -0,0 +1,15 @@ +"""Whether Documents/Reports is a folder in the session user's home, asked through the driver's computer-exec.""" +import uuid + +from agentenv_protocol import client as protocol + +EXEC = "urn:agentenv:computer-exec/v1" + + +async def verify(mcp_url): + base = mcp_url.rsplit("/mcp", 1)[0] + card = await protocol.get_card(base) + out = await protocol.invoke_extension(base, card, EXEC, {"argv": ["test", "-d", "Documents/Reports"], + "request_id": uuid.uuid4().hex}, method="run") + return [{"id": "folder", "result": out["exit_code"] == 0, + "description": "Documents/Reports is a folder" if out["exit_code"] == 0 else "no folder Documents/Reports"}] diff --git a/src/agent_env/examples/desktop/artifacts/check_probe/verify.py b/src/agent_env/examples/desktop/artifacts/check_probe/verify.py new file mode 100644 index 000000000..bf1ac8a34 --- /dev/null +++ b/src/agent_env/examples/desktop/artifacts/check_probe/verify.py @@ -0,0 +1,9 @@ +"""The conformance probe at full against the deployed computer: one result per check, passing when it passed or +didn't apply.""" +from agentenv_protocol.computer.conformance import run_probe + + +async def verify(mcp_url): + report = await run_probe(mcp_url.rsplit("/mcp", 1)[0], "full") + return [{"id": f"{check.check} {check.name}", "result": check.status in ("passed", "not_applicable"), + "description": check.detail or check.status} for check in report.checks] diff --git a/src/agent_env/examples/desktop/envs/desktop/env.toml b/src/agent_env/examples/desktop/envs/desktop/env.toml new file mode 100644 index 000000000..e1dca97f8 --- /dev/null +++ b/src/agent_env/examples/desktop/envs/desktop/env.toml @@ -0,0 +1,3 @@ +type = "computer" +computer = "ubuntu-24.04" +driver = "reference" diff --git a/src/agent_env/examples/desktop/tasks/agent.json b/src/agent_env/examples/desktop/tasks/agent.json new file mode 100644 index 000000000..09b1e89f1 --- /dev/null +++ b/src/agent_env/examples/desktop/tasks/agent.json @@ -0,0 +1,11 @@ +[ + {"id": "env", "type": "deploy_env", "env_id": "desktop"}, + {"id": "agent", "type": "deploy_agent", "env_ids": ["desktop"]}, + {"id": "skills", "type": "add_skills", "env_ids": ["desktop"]}, + {"id": "record", "type": "start_computer_recording", "env_id": "desktop", "name": "agent", "depends_on": ["env"]}, + {"id": "task", "type": "prompt_agent", "depends_on": ["skills", "record"], + "prompt": "Open the file manager on this computer and make a new folder named Reports in the Documents folder. Use only the computer_* tools, and reply DONE once the folder is there."}, + {"id": "check", "type": "env_outcome_verifier", "env_id": "desktop", "file_artifact_id": "check_folder", + "verifier_id": "folder", "depends_on": ["task"]}, + {"id": "end", "type": "end_computer_recording", "name": "agent", "depends_on": ["task"]} +] diff --git a/src/agent_env/examples/desktop/tasks/probe.json b/src/agent_env/examples/desktop/tasks/probe.json new file mode 100644 index 000000000..2fd068aec --- /dev/null +++ b/src/agent_env/examples/desktop/tasks/probe.json @@ -0,0 +1,5 @@ +[ + {"id": "env", "type": "deploy_env", "env_id": "desktop"}, + {"id": "probe", "type": "env_outcome_verifier", "env_id": "desktop", "file_artifact_id": "check_probe", + "verifier_id": "probe"} +] diff --git a/src/agent_env/providers/__init__.py b/src/agent_env/providers/__init__.py index 3c403dcb0..f7228fa81 100644 --- a/src/agent_env/providers/__init__.py +++ b/src/agent_env/providers/__init__.py @@ -15,7 +15,7 @@ get_agent_sandbox_provider, set_agent_sandbox_provider, reset_agent_sandbox_provider, ) from agent_env.providers.sandbox_providers.chained_sandbox_provider import ChainedSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider diff --git a/src/agent_env/providers/env_providers/env_computer_provider.py b/src/agent_env/providers/env_providers/env_computer_provider.py new file mode 100644 index 000000000..ba5fbdad9 --- /dev/null +++ b/src/agent_env/providers/env_providers/env_computer_provider.py @@ -0,0 +1,169 @@ +"""The computer environment provider: a computer env's computer, one ready build of it, booted with no gateway in front +on a sandbox provider that runs its arch natively; the driver's card is the env card.""" + +from __future__ import annotations + +import functools +import logging +from datetime import datetime, timezone +from typing import TYPE_CHECKING, ClassVar, Optional + +from agentenv_protocol.computer import COMPUTER_DISPLAY, COMPUTER_RECORDING + +from agent_env.artifact.artifacts.computer import ComputerArtifact +from agent_env.attribution import Attribution +from agent_env.env.env import DeployedSandboxEnv, EnvCapabilityUnsupported +from agent_env.env.envs.computer import params as computer_params +from agent_env.env.envs.computer.boot import READY_TIMEOUT_S, boot, discard, ready, recorded_settings, settings_environment +from agent_env.env.gateway.constants import WELL_KNOWN_PATH +from agent_env.providers.env_providers.env_provider import DeployRefused, _SandboxEnvironmentProvider +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy +from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider + +if TYPE_CHECKING: + from agent_env.env.env import Env + from agent_env.env.envs.computer.env import ComputerEnv + from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider + +logger = logging.getLogger(__name__) + +#: How long a driver may take to end its recordings, uploading the segments still to go. +_END_RECORDINGS_TIMEOUT_S = 300 + + +class EnvironmentComputerProvider(_SandboxEnvironmentProvider): + """Deploys a computer env with no gateway in front; the driver's own card is the env card.""" + + type: ClassVar[str] = "computer" + record_class = DeployedSandboxEnv + + async def deploy( + self, + env: Env, + sandbox_provider: SandboxProvider, + *, + ttl_seconds: int = 10800, + disk_size_gb: float = 10, + cpu: float | None = None, + memory_mb: int | None = None, + attribution: Optional[Attribution] = None, + env_params: Optional[dict] = None, + ) -> DeployedSandboxEnv: + """The env's first ready build, in its base's order of forms and arches (or of ``env_params.form``), that the sandbox + provider runs natively, booted; returns the record once the driver is ready with the env's display.""" + from agent_env.env.envs.computer.env import ComputerEnv + + if not isinstance(env, ComputerEnv): + raise TypeError(f"env_provider_type '{self.type}' deploys a computer env, not a {env.type} env") + params, values, problems = computer_params.checked(env, env_params) + if problems: + raise DeployRefused(f"{env.id}: " + "; ".join(problems)) + if not (builds := env.ready_builds(params.form)): + raise DeployRefused(f"{env.id} has no ready build to deploy; its builds are {env.build_statuses() or 'none yet'}") + attempt = functools.partial(self._boot, env, builds, params, values, ttl_seconds=ttl_seconds, + disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, attribution=attribution) + return await self._run(sandbox_provider, env.id, attempt) + + async def install_changelog_triggers(self, environment_name: str) -> None: + """No state store, so no changelog to install.""" + + async def _boot(self, env: ComputerEnv, builds: list[tuple[str, str, dict]], params: computer_params.ComputerParams, + values: dict, sandbox_provider: SandboxProvider, *, ttl_seconds: int, + disk_size_gb: float, cpu: float | None, memory_mb: int | None, + attribution: Optional[Attribution]) -> DeployedSandboxEnv: + """One attempt on one sandbox provider: the first build it runs natively, booted and readied; its record.""" + from agent_env.env.envs.computer.env import ComputerEnv + + native = [(form, arch, build) for form, arch, build in builds + if f"linux/{arch}" in sandbox_provider.native_platforms()] + if not native: + raise ValueError(f"{type(sandbox_provider).__name__} runs none of {env.id}'s ready builds natively " + f"({', '.join(f'{form}/{arch}' for form, arch, _ in builds)})") + form, arch, build = native[0] + computer, name = env.computer_artifact(), env.environment_name + values = _values(computer, values) + policy = NetworkPolicy.from_dict(env.network) if env.network else None + booted = await boot( + sandbox_provider, image=build["image"], port=env.driver.port, network_policy=policy, + environment=_environment(env, computer, values, params), timeout=ttl_seconds, attribution=attribution, + disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb) + self._container_sandboxes.append(booted.sandbox) + self._environment_sandboxes[name] = booted.sandbox + try: + _require_name(env, booted.card) + record = DeployedSandboxEnv( + env_id=env.id, env_version=env.version, env_provider_type=self.type, + environment_card_url=f"{booted.url}{WELL_KNOWN_PATH}", environment_card=booted.card, + environment_card_read_at_utc=datetime.now(timezone.utc).isoformat(), + sandbox_id=booted.sandbox.sandbox_id, sandbox_type=booted.sandbox.type, + sandbox_ids={ComputerEnv.type: {name: booted.sandbox.sandbox_id}}, + metadata={"computer": { + "form": form, "arch": arch, "provider": type(sandbox_provider).__name__, "image": build["image"], + "computer": {"id": env.metadata["computer"]["id"], "version": env.metadata["computer"]["version"]}, + "display": values.get("resolution"), "network": env.network, + "computer_settings": recorded_settings(computer, values), + "env_params": params.model_dump(exclude_defaults=True)}}, + ) + await _require_display(record, values.get("resolution")) + except BaseException: + await discard(booted.sandbox) + raise + return record + + async def reboot(self, env: ComputerEnv, deployed: DeployedSandboxEnv) -> None: + """Replace the computer ``deployed`` records with a fresh container of the image it booted, once its recordings + ended, its settings applied as that deploy applied them (a generated one minted afresh), on the same sandbox and + port, readied as a deploy readies it.""" + record = deployed.metadata["computer"] + params, values, _ = computer_params.checked(env, record["env_params"]) + computer = ComputerArtifact.get(record["computer"]["id"], record["computer"]["version"]) + values = _values(computer, values) + await end_recordings(deployed, "reset") + sandbox_provider = build_sandbox_provider(deployed.sandbox_type) + sandbox = await sandbox_provider.get_sandbox(deployed.sandbox_id) + try: + await sandbox_provider.reset_container(sandbox, image_name=record["image"], port=env.driver.port, + env=_environment(env, computer, values, params)) + except NotImplementedError as e: # a reset the env supports, which this provider can't carry out + raise RuntimeError(f"'{env.id}' can't be reset on {type(sandbox_provider).__name__}: {e}") from None + _require_name(env, await ready(deployed.environment_url, READY_TIMEOUT_S)) + await _require_display(deployed, values.get("resolution")) + + +async def end_recordings(record: DeployedSandboxEnv, reason: str) -> None: + """Have the computer's driver end every recording not yet ended, each uploading what's left and its manifest under + the grant it was started with, before the computer goes; a driver that doesn't record has none.""" + try: + await record.invoke(COMPUTER_RECORDING, "end_all", {"reason": reason}, timeout=_END_RECORDINGS_TIMEOUT_S) + except EnvCapabilityUnsupported: + return + except Exception as e: + logger.warning(f"'{record.env_id}': its recordings weren't ended before its computer went: {e}") + + +def _values(computer: ComputerArtifact, values: dict) -> dict: + """The computer's settings a boot applies: each one's default, which carries the env's value, under what + ``env_params`` set.""" + return {**{key: setting.default for key, setting in computer.config.items()}, **values} + + +def _environment(env: ComputerEnv, computer: ComputerArtifact, values: dict, + params: computer_params.ComputerParams) -> dict[str, str]: + return settings_environment(computer, values, environment_name=env.environment_name, + driver_settings=params.driver_settings) + + +def _require_name(env: ComputerEnv, card: dict) -> None: + if card.get("name") != env.environment_name: + raise RuntimeError(f"'{env.environment_name}' serves a card named '{card.get('name')}'; a computer env's driver " + "must serve its card under the env's environment_name") + + +async def _require_display(record: DeployedSandboxEnv, resolution: Optional[str]) -> None: + """Fail unless the driver reports the screen size the boot set, as ``WIDTHxHEIGHT``.""" + if resolution is None: + return + display = await record.invoke(COMPUTER_DISPLAY, "get") + if f"{display['width']}x{display['height']}" != resolution: + raise RuntimeError(f"'{record.env_id}' booted at {display['width']}x{display['height']}, not the {resolution} " + "its display sets") diff --git a/src/agent_env/providers/env_providers/env_gateway_provider.py b/src/agent_env/providers/env_providers/env_gateway_provider.py index 761976edf..e6fb04717 100644 --- a/src/agent_env/providers/env_providers/env_gateway_provider.py +++ b/src/agent_env/providers/env_providers/env_gateway_provider.py @@ -49,7 +49,7 @@ _size_kwargs, _tool_names, ) -from agent_env.providers.sandbox_providers.local_sandbox import remove_local_work_dir +from agent_env.providers.sandbox_providers.local_docker.provider import remove_local_work_dir from agent_env.providers.sandbox_providers.sandbox import Sandbox, VmSandbox, port_bindings from agent_env.providers.sandbox_providers.sandbox_provider import ( SANDBOX_MODE_CONTAINER, diff --git a/src/agent_env/providers/env_providers/env_provider.py b/src/agent_env/providers/env_providers/env_provider.py index 16dd2982f..a6f4d1913 100644 --- a/src/agent_env/providers/env_providers/env_provider.py +++ b/src/agent_env/providers/env_providers/env_provider.py @@ -213,10 +213,11 @@ async def _tool_names(mcp_url: str) -> list[str]: def _builtin_env_providers() -> dict[str, type[_SandboxEnvironmentProvider]]: """The built-in providers, by type.""" + from agent_env.providers.env_providers.env_computer_provider import EnvironmentComputerProvider from agent_env.providers.env_providers.env_gateway_provider import EnvironmentGatewayProvider from agent_env.providers.env_providers.env_server_provider import EnvironmentServerProvider - return {p.type: p for p in (EnvironmentGatewayProvider, EnvironmentServerProvider)} + return {p.type: p for p in (EnvironmentGatewayProvider, EnvironmentServerProvider, EnvironmentComputerProvider)} def _build_registry(source: Config | None = None) -> dict[str, type[EnvironmentProvider]]: diff --git a/src/agent_env/providers/sandbox_providers/AGENTS.md b/src/agent_env/providers/sandbox_providers/AGENTS.md index 62e0ef81e..e25a2f38b 100644 --- a/src/agent_env/providers/sandbox_providers/AGENTS.md +++ b/src/agent_env/providers/sandbox_providers/AGENTS.md @@ -11,7 +11,7 @@ and runs verifiers over that machine's shell. A provider is two classes: `tunnel_urls`. Read `boxd/` (the smallest complete VM provider) and `e2b/` before writing anything. `tensorlake/`, `vercel/` and `tenki/` -are further VM backends; `modal_sandbox.py` is a container backend; `local_sandbox.py` runs on this machine. +are further VM backends; `modal_sandbox.py` is a container backend; `local_docker/` runs on this machine's Docker, with the relay a container closed off from every network is reached through. ## Workflow @@ -55,6 +55,8 @@ with `exposed_ports=[port]` and returns the VM, and the deploy then loads and st | `get_sandbox(sandbox_id) -> Sandbox` | yes | every step after the one that created the sandbox, and teardown | | `terminate_sandbox(sandbox_id)` | no | teardown. The default calls `get_sandbox` then `terminate()`, and treats `SandboxNotFoundError` as done; override when you can terminate without a live handle | | `create_container(...)` | inherited | containers that run an env or agent image. Override only when the platform boots images natively | +| `reset_container(sandbox, *, image_name, port, env)` | inherited | a computer env's reset, which replaces the container `create_container` ran with a fresh one on the same sandbox and port. The default works on a VM backend; one whose sandbox is the container itself raises `NotImplementedError` | +| `native_platforms()` | no | a computer env's deploy and gate 3, which boot a build only on a provider that runs its arch without emulation. The default is `linux/amd64`; local Docker returns this machine's | | `prepare_image(image, *, attribution)` | no | before a sandbox is asked for an image, outside any creation deadline; build a context-only image here, and pull one with an `image_digest` by it, building it from its context when it can't be pulled | | `from_config(**config)` | no | receives `[sandbox.providers..config]` with `secret:` references resolved. Override to fail early with a `ConfigError` that names the missing key (`boxd` also rejects unknown keys) | | `close()` | no | release SDK clients | diff --git a/src/agent_env/providers/sandbox_providers/__init__.py b/src/agent_env/providers/sandbox_providers/__init__.py index b0f13b2ab..378003475 100644 --- a/src/agent_env/providers/sandbox_providers/__init__.py +++ b/src/agent_env/providers/sandbox_providers/__init__.py @@ -10,7 +10,7 @@ get_agent_sandbox_provider, set_agent_sandbox_provider, reset_agent_sandbox_provider, ) from agent_env.providers.sandbox_providers.chained_sandbox_provider import ChainedSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox, E2BSandboxProvider diff --git a/src/agent_env/providers/sandbox_providers/local_docker/__init__.py b/src/agent_env/providers/sandbox_providers/local_docker/__init__.py new file mode 100644 index 000000000..fc1bf96d5 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/local_docker/__init__.py @@ -0,0 +1,6 @@ +"""The local Docker sandbox provider: this machine's Docker, and the relay a container closed off from every network is +reached through.""" + +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider + +__all__ = ["LocalSandbox", "LocalSandboxProvider"] diff --git a/src/agent_env/providers/sandbox_providers/local_sandbox.py b/src/agent_env/providers/sandbox_providers/local_docker/provider.py similarity index 87% rename from src/agent_env/providers/sandbox_providers/local_sandbox.py rename to src/agent_env/providers/sandbox_providers/local_docker/provider.py index a7995b098..13ca76d9b 100644 --- a/src/agent_env/providers/sandbox_providers/local_sandbox.py +++ b/src/agent_env/providers/sandbox_providers/local_docker/provider.py @@ -38,8 +38,16 @@ from agent_env import config from agent_env.attribution import Attribution from agent_env.config.paths import state_root -from agent_env.providers.sandbox_providers.host_images import build_here, built_here, present, pull_here -from agent_env.providers.sandbox_providers.sandbox import SANDBOX_LABEL, NetworkPolicy, VmSandbox +from agent_env.providers.sandbox_providers.host_images import build_here, built_here, host_platform, present, pull_here +from agent_env.providers.sandbox_providers.local_docker.relay import relay_image +from agent_env.providers.sandbox_providers.sandbox import ( + SANDBOX_LABEL, + NetworkPolicy, + NetworkPolicyUnsupportedError, + Sandbox, + VmSandbox, + port_bindings, +) from agent_env.providers.sandbox_providers.sandbox_provider import ( SANDBOX_MODE_CONTAINER, SANDBOX_MODE_VM, @@ -105,6 +113,8 @@ def trust_env(directory: str) -> dict[str, str]: _CONTAINER_MODE_MARKER = ".agent-container-mode" # The sandbox's container port -> host port map, so a sandbox rebuilt from its id publishes and finds the same ports. _PORT_MAP_FILE = ".port-map.json" +# The internal network a container with no network runs on, so a sandbox rebuilt from its id removes it and its relay. +_CLOSED_NETWORK_FILE = ".closed-network" # The host ports sandboxes publish on: below the default ranges outgoing connections take their local ports from (32768 # up on Linux, the VM Docker Desktop or Rancher Desktop runs Docker in included; 49152 up on macOS and Windows) and # below Kubernetes' node ports (30000-32767). @@ -280,6 +290,12 @@ def scoped_name(self, name: str) -> str: """``name`` made this sandbox's own: every local sandbox shares this machine's Docker and /tmp.""" return f"{name}-{self.sandbox_id}" + @property + def closed_network(self) -> str | None: + """The internal network create_container ran this sandbox's container on, with no route out, if it did.""" + marker = self._work_dir / _CLOSED_NETWORK_FILE + return marker.read_text() if marker.exists() else None + @property def owns_container(self) -> bool: """Whether create_container started ``container_name`` for this sandbox, not an agent placed on it.""" @@ -323,6 +339,8 @@ async def terminate(self) -> None: await self._remove_labeled() finally: # a container that wouldn't go must not keep the compose stack up await self.exec_script(f"docker rm -f {shlex.quote(self.container_name)} >/dev/null 2>&1 || true") + if self.closed_network is not None: + await self._remove_labeled() if self.mode == SANDBOX_MODE_VM and (self._work_dir / "docker-compose.yml").exists(): # No `|| true`: `docker compose down` is idempotent (an already-down stack exits 0), so a # non-zero exit is a real failure — surface it (exec_script raises, with the compose output) @@ -615,6 +633,15 @@ def _prepared(self) -> set[str]: def url_from_sandbox(self, url: str) -> str: return self.get_external_url(url) + @classmethod + def native_platforms(cls) -> frozenset[str]: + return frozenset({host_platform()}) + + @classmethod + def supports_network_policy(cls, policy: NetworkPolicy) -> bool: + """Unrestricted, or a container with no network at all (``create_container``).""" + return not policy.restricts_egress or policy.closed + async def create_vm( self, *, @@ -630,6 +657,8 @@ async def create_vm( network_policy: Optional[NetworkPolicy] = None, ) -> LocalSandbox: refuse_unenforceable_policy(self, network_policy) + if network_policy is not None and network_policy.restricts_egress: + raise NetworkPolicyUnsupportedError(f"{type(self).__name__} takes a container's network away, not a VM's") # Allocated here rather than in the sandbox constructor: probing for a free port # is I/O, and callers that only need a handle must not pay for it. Off the event # loop, since a draw waits for any other process's draw to finish. @@ -653,17 +682,22 @@ async def create_container( attribution: Optional[Attribution] = None, network_policy: Optional[NetworkPolicy] = None, ) -> LocalSandbox: - sandbox = await super().create_container( - image_name=image_name, - port=port, - env=env, - cpu=cpu, - memory=memory, - disk_size_gb=disk_size_gb, - timeout=timeout, - attribution=attribution, - network_policy=network_policy, - ) + refuse_unenforceable_policy(self, network_policy) + if network_policy is not None and network_policy.closed: + sandbox = await self._create_closed(image_name=image_name, port=port, env=env, timeout=timeout, + attribution=attribution) + else: + sandbox = await super().create_container( + image_name=image_name, + port=port, + env=env, + cpu=cpu, + memory=memory, + disk_size_gb=disk_size_gb, + timeout=timeout, + attribution=attribution, + network_policy=network_policy, + ) # The marker is how a later get_sandbox() (post-run teardown rebuilds the sandbox from disk) # learns this is a container and removes it. If we can't persist it, that reconstructed # teardown would take the VM path and leak the running container — so we can't just swallow @@ -689,9 +723,64 @@ async def create_container( "be running and holding its host port — remove it manually.", sandbox.container_name, exc_info=True, ) + if sandbox.closed_network is not None: + await sandbox._remove_labeled() raise return sandbox + async def _create_closed(self, *, image_name: str, port: int, env: dict[str, str], timeout: int, + attribution: Optional[Attribution]) -> LocalSandbox: + """The container on an internal network of its own, with no route out, and a relay on that network and the + default bridge publishing its ``port``: Docker publishes no port of a container on an internal network alone.""" + relay = await relay_image() + sandbox = await self.create_vm(exposed_ports=[port], timeout=timeout, attribution=attribution) + try: + await sandbox.pull_images([image_name]) + network = sandbox.scoped_name("closed") + label = shlex.quote(f"{SANDBOX_LABEL}={sandbox.sandbox_id}") + (sandbox.work_dir / _CLOSED_NETWORK_FILE).write_text(network) + await sandbox.exec_script(f"docker network create --internal --label {label} {network} >/dev/null") + await self._run_closed(sandbox, relay, image_name=image_name, port=port, env=env) + sandbox.mode = SANDBOX_MODE_CONTAINER + except BaseException: + try: + await sandbox.terminate() + await asyncio.to_thread(remove_local_work_dir, sandbox.sandbox_id) + except Exception: + pass + raise + sandbox.network_policy = NetworkPolicy(mode="allowlist") + return sandbox + + async def reset_container(self, sandbox: Sandbox, *, image_name: str, port: int, env: dict[str, str]) -> None: + """Reset the sandbox's container to a fresh one, and on a closed network its relay too, since the fresh + container's address on that network may differ.""" + if not (isinstance(sandbox, LocalSandbox) and sandbox.closed_network): + await super().reset_container(sandbox, image_name=image_name, port=port, env=env) + return + await sandbox.exec_script(f"docker rm -f {shlex.quote(sandbox.container_name)} " + f"{shlex.quote(sandbox.scoped_name('relay'))} > /dev/null") + await self._run_closed(sandbox, await relay_image(), image_name=image_name, port=port, env=env) + + async def _run_closed(self, sandbox: LocalSandbox, relay: str, *, image_name: str, port: int, + env: dict[str, str]) -> None: + """The container on the sandbox's closed network, then the relay publishing its ``port`` at its address there.""" + network, relay_name = sandbox.closed_network, sandbox.scoped_name("relay") + label = shlex.quote(f"{SANDBOX_LABEL}={sandbox.sandbox_id}") + await self._start_container(sandbox, image_name=image_name, port=port, env=env) + address = await sandbox.exec_script( + f"docker inspect -f '{{{{(index .NetworkSettings.Networks \"{network}\").IPAddress}}}}' " + f"{shlex.quote(sandbox.container_name)}") + publish = " ".join(f"-p {spec}" for spec in port_bindings(sandbox.host_ips, sandbox.host_port(port), port)) + await sandbox.exec_script( + f"docker run -d --name {relay_name} --label {label} {publish} {relay} {port}={address.strip()}:{port} " + f">/dev/null && docker network connect {network} {relay_name}") + + def _publish_args(self, sandbox: VmSandbox, port: int) -> str: + """A container with no network is reached through its relay; any other is published as everywhere.""" + network = sandbox.closed_network if isinstance(sandbox, LocalSandbox) else None + return f"--network {shlex.quote(network)}" if network is not None else super()._publish_args(sandbox, port) + async def _start_container(self, sandbox: VmSandbox, *, image_name: str, port: int, env: dict[str, str]) -> None: """Where the configured store hands out local grants, create the container, copy the local transfer CA's trust files in, then start it, so its TLS clients trust the grant server. Variables the caller sets win.""" diff --git a/src/agent_env/providers/sandbox_providers/local_docker/relay/Dockerfile b/src/agent_env/providers/sandbox_providers/local_docker/relay/Dockerfile new file mode 100644 index 000000000..931a0ec15 --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/local_docker/relay/Dockerfile @@ -0,0 +1,5 @@ +FROM public.ecr.aws/docker/library/python:3.11-slim@sha256:c20888b6acdd1e63e1c433a185bf3ad162c0288fe484616ce062e0d28add2900 + +COPY relay.py /relay.py + +ENTRYPOINT ["python", "/relay.py"] diff --git a/src/agent_env/providers/sandbox_providers/local_docker/relay/__init__.py b/src/agent_env/providers/sandbox_providers/local_docker/relay/__init__.py new file mode 100644 index 000000000..db9771a2c --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/local_docker/relay/__init__.py @@ -0,0 +1,27 @@ +"""The image local Docker runs beside a container it closes off from every network, to publish that container's port: +built here from this folder the first time it's needed, under a tag of its sources.""" + +import asyncio +import hashlib +from pathlib import Path + +from agent_env.providers.sandbox_providers.host_images import present +from agent_env.utils.docker.build import build_image + +_FOLDER = Path(__file__).parent +_SOURCES = ("Dockerfile", "relay.py") + + +async def relay_image() -> str: + """The relay image's tag, built here unless this machine's Docker already holds it.""" + tag = f"agentenv-relay:{_sources_digest()}" + if not await asyncio.to_thread(present, tag): + await asyncio.to_thread(build_image, _FOLDER / "Dockerfile", _FOLDER, tag, platform=None) + return tag + + +def _sources_digest() -> str: + hashed = hashlib.sha256() + for name in _SOURCES: + hashed.update(name.encode() + b"\0" + (_FOLDER / name).read_bytes()) + return hashed.hexdigest()[:16] diff --git a/src/agent_env/providers/sandbox_providers/local_docker/relay/relay.py b/src/agent_env/providers/sandbox_providers/local_docker/relay/relay.py new file mode 100644 index 000000000..9e24245ee --- /dev/null +++ b/src/agent_env/providers/sandbox_providers/local_docker/relay/relay.py @@ -0,0 +1,38 @@ +"""Relay TCP connections: each argument ``PORT=HOST:PORT`` accepts connections on the first port and joins each to the +second address, both ways, until either side closes.""" + +import asyncio +import sys + + +async def main(routes: list[str]) -> None: + servers = [] + for route in routes: + port, target = route.split("=") + host, _, target_port = target.rpartition(":") + servers.append(await asyncio.start_server( + lambda reader, writer, host=host, target_port=int(target_port): join(reader, writer, host, target_port), + "0.0.0.0", int(port))) + await asyncio.gather(*(server.serve_forever() for server in servers)) + + +async def join(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, host: str, port: int) -> None: + try: + upstream_reader, upstream_writer = await asyncio.open_connection(host, port) + except OSError: + writer.close() + return + await asyncio.gather(pipe(reader, upstream_writer), pipe(upstream_reader, writer), return_exceptions=True) + + +async def pipe(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + try: + while data := await reader.read(65536): + writer.write(data) + await writer.drain() + finally: + writer.close() + + +if __name__ == "__main__": + asyncio.run(main(sys.argv[1:])) diff --git a/src/agent_env/providers/sandbox_providers/sandbox.py b/src/agent_env/providers/sandbox_providers/sandbox.py index cb40c2386..86c740656 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/sandbox.py @@ -114,6 +114,11 @@ def __post_init__(self) -> None: def restricts_egress(self) -> bool: return self.mode is not NetworkMode.ALLOW_ALL + @property + def closed(self) -> bool: + """An allowlist of nothing: no egress at all.""" + return self.mode is NetworkMode.ALLOWLIST and not self.allow_hosts and not self.allow_cidrs + def with_hosts(self, hosts: Iterable[str]) -> "NetworkPolicy": """Copy with ``hosts`` unioned in; a no-op for ALLOW_ALL, which needs no list.""" if self.mode is not NetworkMode.ALLOWLIST: diff --git a/src/agent_env/providers/sandbox_providers/sandbox_provider.py b/src/agent_env/providers/sandbox_providers/sandbox_provider.py index bf0575f97..ac3483577 100644 --- a/src/agent_env/providers/sandbox_providers/sandbox_provider.py +++ b/src/agent_env/providers/sandbox_providers/sandbox_provider.py @@ -15,6 +15,7 @@ from agent_env.config import runtime from agent_env.config import ENV_REF_PREFIX from agent_env.config.errors import ConfigError +from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM from agent_env.providers.sandbox_providers.sandbox import ( NetworkPolicy, NetworkPolicyUnsupportedError, @@ -143,6 +144,16 @@ async def create_container( pass raise + async def reset_container(self, sandbox: Sandbox, *, image_name: str, port: int, env: dict[str, str]) -> None: + """Reset the container create_container ran on ``sandbox`` to a fresh one of ``image_name`` with ``env``, reached + on ``port`` as before: the sandbox, its id and its URLs stay. A backend whose sandbox is the container itself + can't reset it in place.""" + if not isinstance(sandbox, VmSandbox): + raise NotImplementedError(f"{type(self).__name__}'s sandbox is the container itself, so it can't be reset " + "in place") + await sandbox.exec_script(f"docker rm -f {shlex.quote(sandbox.container_name)} > /dev/null") + await self._start_container(sandbox, image_name=image_name, port=port, env=env) + async def prepare_image(self, image: DockerImageArtifact, *, attribution: Optional[Attribution] = None) -> None: """Get ready to run ``image`` by its name, before create_sandbox or create_container is asked for it and outside any deadline on creating one, for the deploy ``attribution`` describes, as it describes the sandbox. Does @@ -157,12 +168,15 @@ def _container_args(self, sandbox: VmSandbox, *, image_name: str, port: int, env """The ``docker run`` / ``docker create`` arguments for the sandbox's container.""" env_flags = " \\\n ".join(f"-e {shlex.quote(k)}={shlex.quote(v)}" for k, v in env.items()) extra_args = f"{self.EXTRA_CONTAINER_RUN_ARGS} " if self.EXTRA_CONTAINER_RUN_ARGS else "" - publish = " ".join(f"-p {spec}" for spec in port_bindings(sandbox.host_ips, sandbox.host_port(port), port)) return ( - f"--name {shlex.quote(sandbox.container_name)} {publish} {extra_args}\\\n " + f"--name {shlex.quote(sandbox.container_name)} {self._publish_args(sandbox, port)} {extra_args}\\\n " f"{env_flags} \\\n {shlex.quote(image_name)}" ) + def _publish_args(self, sandbox: VmSandbox, port: int) -> str: + """How the sandbox's container is reached on ``port``: published on the sandbox's host.""" + return " ".join(f"-p {spec}" for spec in port_bindings(sandbox.host_ips, sandbox.host_port(port), port)) + async def get_sandbox(self, sandbox_id: str) -> Sandbox: raise NotImplementedError(f"{type(self).__name__} does not support get_sandbox") @@ -297,6 +311,11 @@ def supports_network_policy(cls, policy: NetworkPolicy) -> bool: """Whether this backend can enforce ``policy``. Default: unrestricted only.""" return not policy.restricts_egress + @classmethod + def native_platforms(cls) -> frozenset[str]: + """The platforms its containers run without emulation.""" + return frozenset({DEFAULT_BUILD_PLATFORM}) + @classmethod def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPolicy: """``policy`` with the platform's own hosts unioned in; identity unless it allowlists.""" @@ -322,7 +341,7 @@ def effective_network_policy(cls, policy: Optional[NetworkPolicy]) -> NetworkPol "boxd": "agent_env.providers.sandbox_providers.boxd.provider:BoxdSandboxProvider", "daytona": "agent_env.providers.sandbox_providers.daytona.provider:DaytonaSandboxProvider", "boat": "agent_env.providers.sandbox_providers.boat.provider:BoatSandboxProvider", - "local": "agent_env.providers.sandbox_providers.local_sandbox:LocalSandboxProvider", + "local": "agent_env.providers.sandbox_providers.local_docker.provider:LocalSandboxProvider", } diff --git a/src/agent_env/providers/sandbox_providers/smol_vm/provider.py b/src/agent_env/providers/sandbox_providers/smol_vm/provider.py index 6c2c026d9..85fec7066 100644 --- a/src/agent_env/providers/sandbox_providers/smol_vm/provider.py +++ b/src/agent_env/providers/sandbox_providers/smol_vm/provider.py @@ -24,7 +24,7 @@ from agent_env.store.image_store.local_registry_image_store import LocalRegistryImageStore from agent_env.store.routing import LocalRunImageStore from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy -from agent_env.providers.sandbox_providers.local_sandbox import LOCAL_TRUST_ENV, LocalSandboxProvider, local_grant_trust, start_trusting +from agent_env.providers.sandbox_providers.local_docker.provider import LOCAL_TRUST_ENV, LocalSandboxProvider, local_grant_trust, start_trusting from agent_env.providers.sandbox_providers.sandbox_provider import ( Accepts, SandboxProvider, refuse_unenforceable_policy, ) diff --git a/src/agent_env/providers/sandbox_providers/smol_vm/sandbox.py b/src/agent_env/providers/sandbox_providers/smol_vm/sandbox.py index 31046017e..27e441c4e 100644 --- a/src/agent_env/providers/sandbox_providers/smol_vm/sandbox.py +++ b/src/agent_env/providers/sandbox_providers/smol_vm/sandbox.py @@ -13,7 +13,7 @@ from agent_env.config import get_config from agent_env.providers.sandbox_providers.sandbox import CURL_RETRY_FLAGS, VmSandbox from agent_env.providers.sandbox_providers.host_images import host_platform, saved_here -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.store.object_store.local.store import LocalFilesystemObjectStore from agent_env.store.object_store.local.tls import local_ca from agent_env.utils.docker.build import normalize_platform diff --git a/src/agent_env/task/teardown.py b/src/agent_env/task/teardown.py index f5fffb014..9f11627c4 100644 --- a/src/agent_env/task/teardown.py +++ b/src/agent_env/task/teardown.py @@ -1,28 +1,36 @@ """Tear down the sandboxes a finished run deployed. -``teardown_run`` terminates every sandbox a run's context records: its bare sandboxes', its envs' and its -agents'. It removes compute only; the instance, its outputs and the artifacts it collected stay. A local -sandbox's work folder goes too, once the sandbox is down. Before any goes, what has to reach the stores does -(``before_teardown``): the changelogs staged on its agents and the recordings open on its envs. ``agent-env run`` -and ``agent-env eval run`` call it after each run. ``Task.run`` never does, so a caller that keeps a run's -environment up, as the hub and the worker do, owns its teardown. +``teardown_run`` takes down everything a run's context records: its bare sandboxes, its envs and its agents. It +removes compute only; the instance, its outputs and the artifacts it collected stay. What has to reach the stores goes +first: the changelogs staged on its agents (``before_teardown``), the recordings the run left open, ended with fresh +grants whatever env serves them, and what each env saves as it closes (``Env.close``); a recording still open after +that is registered from what its namespace holds. Then every sandbox is terminated by id, and a local one's work +folder goes once it's down. ``agent-env run`` and ``agent-env eval run`` call it after each run, and the +``teardown_sandboxes`` step takes down part of a run the same way (``tear_down``). ``Task.run`` never does, so a +caller that keeps a run's environment up, as the hub and the worker do, owns its teardown. """ from __future__ import annotations import asyncio import logging +from collections.abc import Sequence from dataclasses import dataclass from typing import NamedTuple, Optional -from agent_env.env.env import DeployedEnv -from agent_env.providers.sandbox_providers.local_sandbox import remove_local_work_dir -from agent_env.providers.sandbox_providers.sandbox_provider import build_sandbox_provider +from agent_env.a2a_agent.staging import drain_all, staged_changelogs +from agent_env.env.env import DeployedEnv, DeployedSandboxEnv, Env +from agent_env.env.envs._deployment import close_deployed +from agent_env.providers.sandbox_providers.local_docker.provider import remove_local_work_dir +from agent_env.providers.sandbox_providers.sandbox_provider import ( + build_sandbox_provider, + get_agent_sandbox_provider, + get_env_sandbox_provider, + get_sandbox_provider, +) from agent_env.task.interrupts import tearing_down from agent_env.task_step.context import TaskStepContext -from agent_env.task_step.task_steps.teardown_sandboxes import ( - _DEFAULT_PROVIDERS, TORN_DOWN_KEY, _env_sandbox_ids, _sandbox_ids_of, before_teardown, -) +from agent_env.task_step.task_steps.computer_recording import end_open_recordings, register_open_recordings logger = logging.getLogger(__name__) @@ -30,6 +38,10 @@ TERMINATE_TIMEOUT_SECONDS = 120 +#: ``context.metadata[TORN_DOWN_KEY]``: the ids of the sandboxes a run has terminated, since its ``deployed_*`` +#: records stay (context list fields persist additions only). +TORN_DOWN_KEY = "torn_down_sandbox_ids" + class RecordedSandbox(NamedTuple): sandbox_id: str @@ -51,7 +63,7 @@ def still_up(self) -> tuple[RecordedSandbox, ...]: @classmethod def skipped(cls, context: TaskStepContext) -> TeardownReport: """A teardown that didn't start: everything ``context`` records that's still up is left.""" - return cls(left=_pending(context)) + return cls(left=_pending(context, recorded_sandboxes(context))) def recorded_sandboxes(context: TaskStepContext) -> tuple[RecordedSandbox, ...]: @@ -62,7 +74,7 @@ def recorded_sandboxes(context: TaskStepContext) -> tuple[RecordedSandbox, ...]: for sandbox in context.deployed_sandboxes: found.setdefault(sandbox.sandbox_id, RecordedSandbox(sandbox.sandbox_id, sandbox.sandbox_type, "sandbox")) for env in context.deployed_envs: - for sandbox_id, sandbox_type in _env_sandbox_ids(env): + for sandbox_id, sandbox_type in env_sandbox_ids(env): found.setdefault(sandbox_id, RecordedSandbox(sandbox_id, sandbox_type, "env")) for agent in context.deployed_agents: if agent.sandbox_id: @@ -82,25 +94,47 @@ def recorded_sandboxes(context: TaskStepContext) -> tuple[RecordedSandbox, ...]: def sandbox_ids(record) -> list[str]: """The ids of the sandboxes behind one deployed env, agent or bare sandbox record.""" if isinstance(record, DeployedEnv): - return [sandbox_id for sandbox_id, _ in _env_sandbox_ids(record)] + return [sandbox_id for sandbox_id, _ in env_sandbox_ids(record)] return [record.sandbox_id] if record.sandbox_id else [] async def teardown_run(context: TaskStepContext, *, timeout: float = TERMINATE_TIMEOUT_SECONDS) -> TeardownReport: - """Terminate every sandbox ``context`` records, all at once, each within ``timeout`` seconds, and remove a - local one's work folder once it's down. + """Take down everything ``context`` records, as ``tear_down`` does, once the changelogs staged on its agents have + reached the stores. Best-effort: a failure is logged and reported, never raised. A sandbox already torn down, by a ``teardown_sandboxes`` step or an earlier call, isn't terminated again, though a local one's folder still goes. Under ``Interrupts.gather`` the first signal spares it. Cancelling it stops it early: it returns what it reached, with the rest in ``left``, rather than raising.""" tearing_down() - pending = _pending(context) - terminated, failed = [], [] try: await before_teardown(context, timeout) + except asyncio.CancelledError: + return TeardownReport.skipped(context) + return await tear_down(context, recorded_sandboxes(context), context.deployed_envs, timeout=timeout) + + +async def tear_down(context: TaskStepContext, sandboxes: Sequence[RecordedSandbox], envs: Sequence[DeployedEnv], *, + timeout: float) -> TeardownReport: + """End the recordings the run left open on ``envs`` with fresh grants, close each env still up, register any + recording still open from its namespace, then terminate each of ``sandboxes`` by id, all at once and each within + ``timeout`` seconds, and remove a local one's work folder once it's down. A close that fails is logged: its env's sandboxes are terminated by id regardless, which leaves one the + close took down as it is. Cancelling it returns what it reached, with the rest in ``left``.""" + pending = _pending(context, sandboxes) + up = {sandbox.sandbox_id for sandbox in pending} + env_ids = [env.env_id for env in envs] + try: + try: + async with asyncio.timeout(timeout): + await end_open_recordings(context, env_ids) + except TimeoutError: + logger.warning("teardown: the run's open recordings weren't all ended within %gs", timeout) + await asyncio.gather(*(_close(env, timeout) for env in envs + if any(sandbox_id in up for sandbox_id, _ in env_sandbox_ids(env)))) + await asyncio.to_thread(register_open_recordings, context, env_ids) except asyncio.CancelledError: return TeardownReport(left=pending) + terminated, failed = [], [] async def take_down(sandbox: RecordedSandbox) -> None: if sandbox not in pending: # torn down already: only a local one's folder can be left @@ -109,7 +143,7 @@ async def take_down(sandbox: RecordedSandbox) -> None: return try: async with asyncio.timeout(timeout): - on_this_machine = await _terminate(sandbox) + on_this_machine = await terminate(sandbox) except Exception as e: reason = f"timed out after {timeout:g}s" if isinstance(e, TimeoutError) else _described(e) logger.warning("teardown: %s (%s) was not terminated: %s", sandbox.sandbox_id, kind(sandbox), reason) @@ -122,29 +156,31 @@ async def take_down(sandbox: RecordedSandbox) -> None: terminated.append(sandbox) try: - await asyncio.gather(*(take_down(sandbox) for sandbox in recorded_sandboxes(context))) + await asyncio.gather(*(take_down(sandbox) for sandbox in sandboxes)) except asyncio.CancelledError: pass reached = {sandbox.sandbox_id for sandbox in terminated} | {sandbox.sandbox_id for sandbox, _ in failed} return TeardownReport(tuple(terminated), tuple(failed), tuple(s for s in pending if s.sandbox_id not in reached)) -def _pending(context: TaskStepContext) -> tuple[RecordedSandbox, ...]: - torn_down = set(context.metadata.get(TORN_DOWN_KEY) or []) - return tuple(sandbox for sandbox in recorded_sandboxes(context) if sandbox.sandbox_id not in torn_down) +async def before_teardown(context: TaskStepContext, within: float, agent_names: Optional[list[str]] = None) -> None: + """Drain the changelogs staged on ``agent_names``, or on every agent when None, within ``within`` seconds each.""" + for name in [None] if agent_names is None else agent_names: + await drain_all(staged_changelogs(context.metadata, agent_name=name), within=within) -async def _remove_folder(sandbox: RecordedSandbox) -> str | None: - """Remove a local sandbox's work folder, and say why when it can't be.""" - try: - await asyncio.to_thread(remove_local_work_dir, sandbox.sandbox_id) - except OSError as e: - logger.warning("teardown: %s's work folder was not removed: %s", sandbox.sandbox_id, e) - return f"terminated, but its work folder wasn't removed: {e}" - return None +def env_sandbox_ids(env: DeployedEnv) -> list[tuple[str, Optional[str]]]: + """Every (sandbox_id, sandbox_type) of a deployed env: its primary sandbox plus ``sandbox_ids``. + ``sandbox_ids`` values are an id or a ``{service: id}`` dict and inherit the env's type, unless + the key itself names a registered backend (a sandbox on another backend than the gateway's). + """ + if not isinstance(env, DeployedSandboxEnv): # an env outside our sandboxes owns its own lifetime + return [] + return _sandbox_ids_of(env.sandbox_id, env.sandbox_type, env.sandbox_ids) -async def _terminate(sandbox: RecordedSandbox) -> bool: + +async def terminate(sandbox: RecordedSandbox) -> bool: """Terminate ``sandbox`` and say whether it ran on this machine: by id through the backend that made it, or, for a record that doesn't name one, through a live handle, since "not found" from a provider that didn't make the sandbox doesn't mean it's gone.""" @@ -164,10 +200,64 @@ async def _terminate(sandbox: RecordedSandbox) -> bool: return handle.ON_THIS_MACHINE +def kind(sandbox: RecordedSandbox) -> str: + """The backend a sandbox is on, as the CLI prints it.""" + return sandbox.sandbox_type or f"the configured {sandbox.slot} provider" + + +async def _close(deployed: DeployedEnv, timeout: float) -> None: + """Close one env a run deployed through its type, so it ends what it runs before its sandboxes go (a computer env, + its recordings). A type that can't close a deployment leaves them to be terminated by id.""" + try: + async with asyncio.timeout(timeout): + await close_deployed(deployed, type(Env.get(deployed.env_id, deployed.env_version))) + except NotImplementedError: + return + except Exception as e: + reason = f"timed out after {timeout:g}s" if isinstance(e, TimeoutError) else _described(e) + logger.warning("teardown: env %s wasn't closed (%s); its sandboxes are terminated by id", deployed.env_id, reason) + + +def _pending(context: TaskStepContext, sandboxes: Sequence[RecordedSandbox]) -> tuple[RecordedSandbox, ...]: + torn_down = set(context.metadata.get(TORN_DOWN_KEY) or []) + return tuple(sandbox for sandbox in sandboxes if sandbox.sandbox_id not in torn_down) + + +async def _remove_folder(sandbox: RecordedSandbox) -> str | None: + """Remove a local sandbox's work folder, and say why when it can't be.""" + try: + await asyncio.to_thread(remove_local_work_dir, sandbox.sandbox_id) + except OSError as e: + logger.warning("teardown: %s's work folder was not removed: %s", sandbox.sandbox_id, e) + return f"terminated, but its work folder wasn't removed: {e}" + return None + + +def _sandbox_ids_of( + sandbox_id: Optional[str], sandbox_type: Optional[str], sandbox_ids: Optional[dict], +) -> list[tuple[str, Optional[str]]]: + ids = [(sandbox_id, sandbox_type)] if sandbox_id else [] + for key, value in (sandbox_ids or {}).items(): + entry_type = key if _is_backend(key) else sandbox_type + values = [value] if isinstance(value, str) else [v for v in value.values() if isinstance(v, str)] + ids.extend((v, entry_type) for v in values) + return ids + + +def _is_backend(name: str) -> bool: + try: + build_sandbox_provider(name) + except Exception: + return False + return True + + def _described(error: Exception) -> str: return f"{type(error).__name__}: {error}" if str(error) else type(error).__name__ -def kind(sandbox: RecordedSandbox) -> str: - """The backend a sandbox is on, as the CLI prints it.""" - return sandbox.sandbox_type or f"the configured {sandbox.slot} provider" +_DEFAULT_PROVIDERS = { + "agent": get_agent_sandbox_provider, + "env": get_env_sandbox_provider, + "sandbox": get_sandbox_provider, +} diff --git a/src/agent_env/task_step/context.py b/src/agent_env/task_step/context.py index 413954de6..e6d541bae 100644 --- a/src/agent_env/task_step/context.py +++ b/src/agent_env/task_step/context.py @@ -9,7 +9,7 @@ from agent_env.config.loader import ENV_REF_PREFIX, SECRET_REF_PREFIX from agent_env.env.env import DeployedEnv -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, host_url_for +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, host_url_for from agent_env.providers.sandbox_providers.sandbox_provider import reachable_url HEARTBEAT_KEY = "_heartbeat_fn" @@ -81,6 +81,8 @@ class DeployedAgent: on_host: bool = False # The container install_agent put it in, as the sandbox's host names it; none for a host install or a deploy. installed_container: str | None = None + # The envs deploy_agent registered with it, by id. + env_ids: list[str] = field(default_factory=list) @classmethod def from_dict(cls, data: dict) -> DeployedAgent: @@ -96,6 +98,7 @@ def from_dict(cls, data: dict) -> DeployedAgent: network_policy=data.get("network_policy"), on_host=bool(data.get("on_host", False)), installed_container=data.get("installed_container"), + env_ids=list(data.get("env_ids") or []), ) def url_for(self, sandbox_type: str | None, *, on_host: bool = False) -> str: diff --git a/src/agent_env/task_step/task_steps/add_skills.py b/src/agent_env/task_step/task_steps/add_skills.py index 64b06887c..2a44c5b98 100644 --- a/src/agent_env/task_step/task_steps/add_skills.py +++ b/src/agent_env/task_step/task_steps/add_skills.py @@ -8,13 +8,20 @@ from dataclasses import dataclass from typing import ClassVar, Optional -from agent_env.task_step.context import TaskStepContext +from agentenv_protocol import SKILL_EXTENSION_URI +from agentenv_protocol.client import find_extension_method + +from agent_env.task_step.context import DeployedAgent, TaskStepContext from agent_env.entity.refs import EntityRef from agent_env.entity.scopes import FieldScope from agent_env.task_step.task_step import TaskStep, TaskStepDependency logger = logging.getLogger(__name__) +#: ``context.metadata[ENV_SKILLS_KEY][agent_name][env_id][skill_name]``: the SKILL.md add_skills installed from an env, as +#: the env, or one of its child envs, wrote it for that deploy. +ENV_SKILLS_KEY = "env_skills" + @dataclass class Skill: @@ -166,6 +173,16 @@ def _files_skill_name(universe_id: str) -> str: return f"{slug}-{digest}-files" if slug else f"{digest}-files" +def _env_skill_name(environment_name: str, env_id: str) -> str: + """The env's environment name, as the agent's MCP alias for it is; for a name that isn't a skill name as it stands, + a slug of it and a hash of the env's id, so two envs' names never come out the same.""" + if re.fullmatch(r"[a-z0-9]+(-[a-z0-9]+)*", environment_name) and len(environment_name) <= 64: + return environment_name + slug = re.sub(r"[^a-z0-9]+", "-", environment_name.lower()).strip("-")[:51].rstrip("-") + digest = hashlib.sha256(env_id.encode("utf-8")).hexdigest()[:12] + return f"{slug}-{digest}" if slug else digest + + def _build_skill_for_loaded_file_artifact_universe(universe_id: str, entry: dict) -> Skill: destination_path = entry["destination_path"] return Skill( @@ -181,6 +198,7 @@ class AddSkillsTaskStep(TaskStep): EntityRef.artifact("skills[].skill_artifact_id", version_field="skill_artifact_version", artifact_type="skill"), EntityRef.artifact("cli_artifact_ids[]", artifact_type="cli"), EntityRef.artifact("file_artifact_universe_ids[]"), + EntityRef.env("env_ids[]"), ) field_scopes = {**TaskStep.field_scopes, "skills[].object_url": FieldScope.USER, "skills[].s3_url": FieldScope.USER} @@ -194,11 +212,13 @@ def __init__( agent_name: Optional[str] = None, depends_on: Optional[list[TaskStepDependency]] = None, fail_task_on_error: bool = True, + env_ids: Optional[list[str]] = None, ): super().__init__(id, version, depends_on=depends_on, fail_task_on_error=fail_task_on_error) self.skills = [s if isinstance(s, Skill) else Skill.from_dict(s) for s in (skills or [])] self.cli_artifact_ids = list(cli_artifact_ids or []) self.file_artifact_universe_ids = list(file_artifact_universe_ids or []) + self.env_ids = list(env_ids or []) self.agent_name = agent_name or TaskStep.DEFAULT_AGENT_NAME def to_dict(self) -> dict: @@ -206,6 +226,8 @@ def to_dict(self) -> dict: base["skills"] = [s.to_dict() for s in self.skills] base["cli_artifact_ids"] = self.cli_artifact_ids base["file_artifact_universe_ids"] = self.file_artifact_universe_ids + if self.env_ids: # left out otherwise, so a step without one serializes as before + base["env_ids"] = self.env_ids base["agent_name"] = self.agent_name return base @@ -217,6 +239,7 @@ def from_dict(cls, data: dict) -> AddSkillsTaskStep: cli_artifact_ids=data.get("cli_artifact_ids"), file_artifact_universe_ids=data.get("file_artifact_universe_ids"), agent_name=data.get("agent_name"), + env_ids=data.get("env_ids"), ) def config_problems(self) -> list[str]: @@ -260,9 +283,42 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: ) skills_to_add.append(_build_skill_for_loaded_file_artifact_universe(fau_id, entry)) + env_skills = {env_id: await _env_skills(context, agent, env_id) for env_id in self.env_ids} + deployed = get_a2a_agent_instance_store().get(agent.instance_id) for skill in skills_to_add: result = await A2AAgent.add_skill(deployed, skill) logger.info(f"Registered skill: {result}") + for env_id, skills in env_skills.items(): + for skill in skills: + result = await A2AAgent.register_skill(deployed, name=skill["name"], description=skill["description"], + skill_md=skill["skill_md"]) + installed = context.metadata.setdefault(ENV_SKILLS_KEY, {}).setdefault(self.agent_name, {}) + installed.setdefault(env_id, {})[skill["name"]] = skill["skill_md"] + logger.info(f"Registered the skill {skill['name']} of env {env_id}: {result}") return context + + +async def _env_skills(context: TaskStepContext, agent: DeployedAgent, env_id: str) -> list[dict]: + """The skills the env ``env_id`` serves, each under its card's environment name: its own card's, or else each of its + child envs' (a gateway's servers); refused for an env this run didn't deploy or bind to ``agent``, or that serves + none.""" + deployed = next((d for d in context.deployed_envs if d.env_id == env_id), None) + if deployed is None: + raise RuntimeError(f"env '{env_id}' isn't deployed in this run") + if env_id not in agent.env_ids: + raise RuntimeError(f"env '{env_id}' isn't bound to agent '{agent.agent_name}': name it in the env_ids of its " + "deploy_agent") + card = deployed.environment_card or {} + if find_extension_method(card, SKILL_EXTENSION_URI, "get") is not None: + serving = [(None, _env_skill_name(card.get("name") or env_id, env_id))] + else: + serving = [(child["name"], _env_skill_name(child["name"], f"{env_id}/{child['name']}")) + for child in card.get("children_environments") or [] + if find_extension_method(child, SKILL_EXTENSION_URI, "get") is not None] + if not serving: + raise RuntimeError(f"env '{env_id}' serves no skill: neither its card nor a child env's declares " + f"{SKILL_EXTENSION_URI}") + return [await deployed.invoke(SKILL_EXTENSION_URI, "get", {"name": name}, environment_name=child) + for child, name in serving] diff --git a/src/agent_env/task_step/task_steps/computer_recording.py b/src/agent_env/task_step/task_steps/computer_recording.py index 628ac2552..4841cec61 100644 --- a/src/agent_env/task_step/task_steps/computer_recording.py +++ b/src/agent_env/task_step/task_steps/computer_recording.py @@ -1,8 +1,10 @@ """Record a computer env's display. ``start_computer_recording`` starts a recording on a deployed env whose card, or whose child env's card (``environment_name``), declares computer-recording/v1, with a grant to upload its segments under a namespace of its own; -``end_computer_recording`` ends it and registers its segments and manifest as one FileArtifactUniverse. Teardown -ends whatever a run left open (``end_open_recordings``).""" +``end_computer_recording`` ends it and registers its segments and manifest as one FileArtifactUniverse. Teardown ends +what a run left open, with fresh grants, before any env closes (``end_open_recordings``), and registers from its +namespace one it didn't end in time (``register_open_recordings``); a computer env also ends its own as it closes or +resets.""" from __future__ import annotations import json @@ -12,6 +14,7 @@ from agentenv_protocol.computer.card import COMPUTER_RECORDING from agentenv_protocol.computer.manifest import ManifestSegment, RecordingManifest +from agentenv_protocol.computer.recording import MANIFEST from agentenv_protocol.client import find_extension from agent_env.a2a_agent.object_transfer import ObjectLimits, namespace_grant @@ -20,6 +23,7 @@ from agent_env.artifact.store import get_artifact_store from agent_env.config import get_config from agent_env.entity.refs import EntityRef +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy from agent_env.store.base import GrantUnavailableError from agent_env.store.ids import derive_id, is_local_id, validate_local_id from agent_env.store.object_store.object_store import issues_grants_to @@ -83,6 +87,7 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: request_id = f"{context.instance_id}:{self.id}" if context.instance_id else uuid.uuid4().hex try: deployed = _deployed(context, self.env_id) + _refuse_no_network(deployed) namespace = _namespace(request_id) grant = _grant(deployed, namespace, self.environment_name) started = await deployed.invoke(COMPUTER_RECORDING, "start", { @@ -140,20 +145,33 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: async def end_open_recordings(context: TaskStepContext, env_ids: Optional[list[str]] = None) -> None: - """End and register every recording still open in ``context`` (of ``env_ids`` only, when given), before its - computer is torn down.""" + """End and register every recording still open in ``context`` (of ``env_ids`` only, when given), each with a fresh + grant, before its computer is torn down, whatever env serves it; ``end_recording`` says what happened to each.""" + for name, entry in list((context.metadata.get(RECORDINGS_KEY) or {}).items()): + if entry.get("state") == "open" and (env_ids is None or entry["env_id"] in env_ids): + await end_recording(context, name, "teardown") + + +def register_open_recordings(context: TaskStepContext, env_ids: Optional[list[str]] = None) -> None: + """Register every recording still open in ``context`` (of ``env_ids`` only, when given) from what its namespace + holds: the manifest its driver uploaded when its env ended it, or else the segments alone. Teardown calls it once + the envs have closed, for one ``end_open_recordings`` didn't end in time. Never raises: each entry says what + happened.""" for name, entry in list((context.metadata.get(RECORDINGS_KEY) or {}).items()): if entry.get("state") == "open" and (env_ids is None or entry["env_id"] in env_ids): try: - await end_recording(context, name, "teardown") - except Exception as e: # what tears the computer down goes on regardless + manifest = _from_namespace(entry) + except Exception as e: + logger.warning("recording %s: listing its namespace failed: %s", name, e) entry["state"], entry["error"] = "failed", f"{type(e).__name__}: {e}"[:500] - logger.warning("ending recording %s: %s", name, e) + continue + _registered(context, name, entry, manifest) async def end_recording(context: TaskStepContext, name: str, reason: str) -> None: - """End the open recording ``name`` with a fresh grant and register it. When the driver no longer knows it, as - after a restart, what its namespace holds is registered instead. Never raises: the entry says what happened.""" + """End the open recording ``name`` with a fresh grant and register it. When the driver no longer knows it, as after + a restart or once its env ended it, what its namespace holds is registered instead. Never raises: the entry says + what happened.""" entry = context.metadata[RECORDINGS_KEY][name] if entry.get("state") != "open": return @@ -171,16 +189,21 @@ async def end_recording(context: TaskStepContext, name: str, reason: str) -> Non logger.warning("end_computer_recording %s: listing its namespace failed: %s", name, listing) entry["state"], entry["error"] = "failed", f"{entry['error']}; then {type(listing).__name__}: {listing}"[:500] return + _registered(context, name, entry, manifest) + + +def _registered(context: TaskStepContext, name: str, entry: dict, manifest: RecordingManifest) -> None: + """Register the recording ``name`` as ``manifest`` says it went, and its entry as ended, or as failed.""" try: entry["artifact"] = _register(context, name, entry["namespace"], manifest) except Exception as e: - logger.warning("end_computer_recording %s: registering it failed: %s", name, e) + logger.warning("recording %s: registering it failed: %s", name, e) entry["state"], entry["error"] = "failed", f"{type(e).__name__}: {e}"[:500] return uploaded = [s for s in manifest.segments if s.uploaded] entry.update(state="ended", ended_by=manifest.ended_by, segments=len(uploaded), missing_segments=len(manifest.segments) - len(uploaded)) - logger.info("end_computer_recording %s: %d segments, ended by %s", name, len(uploaded), manifest.ended_by) + logger.info("recording %s: %d segments, ended by %s", name, len(uploaded), manifest.ended_by) def _deployed(context: TaskStepContext, env_id: str): @@ -190,6 +213,14 @@ def _deployed(context: TaskStepContext, env_id: str): return deployed +def _refuse_no_network(deployed) -> None: + """A computer deployed with no network can't reach the store its recording's segments upload to.""" + network = ((deployed.metadata or {}).get("computer") or {}).get("network") + if network and NetworkPolicy.from_dict(network).closed: + raise RuntimeError(f"env {deployed.env_id!r} was deployed with no network, so its computer can't upload a " + "recording's segments") + + def _namespace(request_id: str) -> str: """Where a recording's segments go: named by its start's request id, so a retried start uploads alongside.""" config = get_config() @@ -209,7 +240,7 @@ def _grant(deployed, namespace: str, environment_name: Optional[str] = None) -> f"({sandbox_type or 'unknown'}), so its recording couldn't upload") duration = params["max_duration_s"] cap = min(store.max_single_upload_bytes or _MAX_SEGMENT_BYTES, _MAX_SEGMENT_BYTES) - limits = ObjectLimits(max_objects=MAX_SEGMENTS, max_object_bytes=cap, max_total_bytes=MAX_RECORDING_BYTES) + limits = ObjectLimits(max_objects=MAX_SEGMENTS + 1, max_object_bytes=cap, max_total_bytes=MAX_RECORDING_BYTES) try: grant = namespace_grant(store, namespace, limits=limits, expires_in=duration + GRANT_MARGIN_S) except GrantUnavailableError: @@ -218,10 +249,14 @@ def _grant(deployed, namespace: str, environment_name: Optional[str] = None) -> def _from_namespace(entry: dict) -> RecordingManifest: - """A manifest of what the recording's namespace holds, for a recording its driver can't end.""" + """What the recording's namespace holds, for a recording its driver didn't end for the run: the manifest the driver + uploaded beside the segments when it ended it, or else one of the segments there.""" store = get_config().get_object_store() root = entry["namespace"].rstrip("/") + "/" - paths = sorted(url[len(root):] for url in store.list_at(root) if url.endswith(".mp4")) + urls = store.list_at(root) + if (uploaded := f"{root}{entry.get('recording_id')}/{MANIFEST}") in urls: + return RecordingManifest.model_validate(json.loads(store.get(uploaded))) + paths = sorted(url[len(root):] for url in urls if url.endswith(".mp4")) return RecordingManifest(recording_id=entry.get("recording_id"), ended_by="error", error=entry.get("error"), segments=[ManifestSegment(index=i, path=path, uploaded=True) for i, path in enumerate(paths)]) diff --git a/src/agent_env/task_step/task_steps/deploy_agent.py b/src/agent_env/task_step/task_steps/deploy_agent.py index f2b3728b6..53baf931e 100644 --- a/src/agent_env/task_step/task_steps/deploy_agent.py +++ b/src/agent_env/task_step/task_steps/deploy_agent.py @@ -28,7 +28,7 @@ from agent_env.config import get_config from agent_env.env.env import DeployedEnv, DeployedSandboxEnv, Env from agent_env.env.gateway import AGENT_ENV_ROLE_HEADER -from agent_env.providers.sandbox_providers.local_sandbox import host_url_for +from agent_env.providers.sandbox_providers.local_docker.provider import host_url_for from agent_env.a2a_agent.a2a_agent import DEFAULT_A2A_PORT, unwritten_local_agent from agent_env.providers.sandbox_providers.sandbox_provider import ( SANDBOX_MODE_VM, @@ -505,6 +505,7 @@ async def _execute(self, context: TaskStepContext, deployed_agents: list) -> Tas instance_id=deployed.instance_id, role=self.role, network_policy=deployed.network_policy, + env_ids=list(self.env_ids), )) return context diff --git a/src/agent_env/task_step/task_steps/deploy_env.py b/src/agent_env/task_step/task_steps/deploy_env.py index d31618bc6..4efc589cc 100644 --- a/src/agent_env/task_step/task_steps/deploy_env.py +++ b/src/agent_env/task_step/task_steps/deploy_env.py @@ -46,6 +46,7 @@ def __init__( metadata: Optional[dict] = None, artifact_id: Optional[str] = None, artifact_version: Optional[int] = None, + env_params: Optional[dict] = None, ): super().__init__(id, version, depends_on=depends_on, fail_task_on_error=fail_task_on_error) self.env_id = env_id @@ -62,6 +63,8 @@ def __init__( # Existing env state instance to attach to; the provider interprets it. self.env_state_instance_id = env_state_instance_id self.metadata = metadata or {} + # The env type's own deploy options, which this step passes to its deploy() without reading them. + self.env_params = env_params def to_dict(self) -> dict: base = super().to_dict() @@ -78,6 +81,7 @@ def to_dict(self) -> dict: base["env_state_type"] = self.env_state_type base["env_state_instance_id"] = self.env_state_instance_id base["metadata"] = self.metadata + base["env_params"] = self.env_params return base @classmethod @@ -97,14 +101,12 @@ def from_dict(cls, data: dict) -> DeployEnvTaskStep: env_state_type=data.get("env_state_type"), env_state_instance_id=data.get("env_state_instance_id"), metadata=dict(data.get("metadata") or {}), + env_params=data.get("env_params"), ) def preflight(self) -> list[str]: """An invalid gateway_mode, an env this step can't load, or an option it would refuse at deploy, reported at save instead.""" from agent_env.env.env import Env - from agent_env.env.envs.mcp_server import MCPServerEnv - from agent_env.env.envs.multi_env import MultiEnv - from agent_env.env.envs.website import WebsiteEnv from agent_env.store.base import NotFoundError try: @@ -113,17 +115,19 @@ def preflight(self) -> list[str]: return [f"deploy_env '{self.id}': {e}"] # the words execute() would raise try: env = Env.get(self.env_id, self.env_version) - if not isinstance(env, (MCPServerEnv, WebsiteEnv, MultiEnv)): - return [] - # As stored: run-time overrides aren't known at save, and deploy() consumes sandbox_type itself. Every run passes an - # attribution, so the check does too, whatever it will hold. + # As stored: run-time overrides aren't known at save. Every run passes an attribution, so the check does too, whatever it + # will hold. refusal = env.deploy_refusal(ttl_seconds=self.ttl_seconds, disk_size_gb=self.disk_size_gb, gateway_mode=self.gateway_mode, cpu=self.cpu, memory_mb=self.memory_mb, env_state_type=self.env_state_type, - env_state_instance_id=self.env_state_instance_id, attribution={}) + env_state_instance_id=self.env_state_instance_id, attribution={}, + sandbox_type=self.sandbox_type, **self._env_params_option()) except (NotFoundError, ValueError) as e: # missing, or a type or env_provider_type this process can't load return [f"deploy_env '{self.id}': env '{self.env_id}' can't be loaded: {e}"] return [f"deploy_env '{self.id}': {refusal}"] if refusal else [] + def _env_params_option(self) -> dict: + return {"env_params": self.env_params} if self.env_params else {} + async def execute(self, context: TaskStepContext) -> TaskStepContext: from agent_env.env.env import Env @@ -176,6 +180,10 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext: deploy_kwargs["artifact_id"] = self.artifact_id if self.artifact_version is not None and ("artifact_version" in params or accepts_kwargs): deploy_kwargs["artifact_version"] = self.artifact_version + if self.env_params: + if "env_params" not in params and not accepts_kwargs: + raise ValueError(f"deploy_env '{self.id}': {env.type} envs take no env_params") + deploy_kwargs["env_params"] = self.env_params deployed_env = await env.deploy(**deploy_kwargs) # create_instance runs inside env.deploy(), so an annotation set after it # returns has to be persisted explicitly to reach the instance record. diff --git a/src/agent_env/task_step/task_steps/install_agent.py b/src/agent_env/task_step/task_steps/install_agent.py index dfa2e80a9..7ddbbd8f9 100644 --- a/src/agent_env/task_step/task_steps/install_agent.py +++ b/src/agent_env/task_step/task_steps/install_agent.py @@ -24,7 +24,7 @@ from agent_env.a2a_agent import A2AAgent from agent_env.config import get_config from agent_env.entity.scopes import FieldScope -from agent_env.providers.sandbox_providers.local_sandbox import ( +from agent_env.providers.sandbox_providers.local_docker.provider import ( LOCAL_TRUST_DIR, LOCAL_TRUST_ENV, host_url_for, diff --git a/src/agent_env/task_step/task_steps/load_artifact.py b/src/agent_env/task_step/task_steps/load_artifact.py index 185ac73ae..cba850321 100644 --- a/src/agent_env/task_step/task_steps/load_artifact.py +++ b/src/agent_env/task_step/task_steps/load_artifact.py @@ -13,7 +13,7 @@ from typing import Callable, ClassVar, Optional from urllib.parse import unquote, urlparse -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.providers.sandbox_providers.sandbox import into_container from agent_env.providers.sandbox_providers.sandbox_provider import reachable_url from agent_env.task_step.context import DeployedAgent, TaskStepContext diff --git a/src/agent_env/task_step/task_steps/teardown_sandboxes.py b/src/agent_env/task_step/task_steps/teardown_sandboxes.py index d299a86c9..76af0ff98 100644 --- a/src/agent_env/task_step/task_steps/teardown_sandboxes.py +++ b/src/agent_env/task_step/task_steps/teardown_sandboxes.py @@ -1,43 +1,39 @@ -"""Terminate the sandboxes behind named agents, envs and bare sandboxes, mid-run. +"""Take down the sandboxes behind named agents, envs and bare sandboxes, mid-run. A long pipeline keeps every sandbox it deployed until the run ends or its TTL expires. Place this -step after the last step that uses a target (via ``depends_on``) to stop paying for it early. +step after the last step that uses a target (via ``depends_on``) to stop paying for it early. It takes +them down as the end of a run does (``agent_env.task.teardown.tear_down``): the named agents' staged +changelogs drain, each named env closes, then every sandbox behind them is terminated by id. Strictly best-effort: a sandbox that is already gone or fails to terminate is logged and the run continues. The step never raises on a terminate, so a retry never re-runs it over partly torn-down -state, and ``fail_task_on_error`` must stay false. The ids it terminated are appended to ``metadata["torn_down_sandbox_ids"]``; the -``deployed_*`` records stay, since context list fields persist additions only. +state, and ``fail_task_on_error`` must stay false. The ids it terminated are appended to +``metadata["torn_down_sandbox_ids"]``; the ``deployed_*`` records stay, since context list fields persist additions only. """ from __future__ import annotations -import asyncio import logging from typing import Optional -from agent_env.a2a_agent.staging import LAST_DRAIN_SECONDS, drain_all, staged_changelogs +from agent_env.a2a_agent.staging import LAST_DRAIN_SECONDS from agent_env.entity.refs import EntityRef -from agent_env.env.env import DeployedSandboxEnv -from agent_env.providers.sandbox_providers.sandbox_provider import ( - SandboxProvider, - build_sandbox_provider, - get_agent_sandbox_provider, - get_env_sandbox_provider, - get_sandbox_provider, +from agent_env.env.env import DeployedEnv +from agent_env.task.teardown import ( + TERMINATE_TIMEOUT_SECONDS, + TORN_DOWN_KEY, + RecordedSandbox, + before_teardown, + env_sandbox_ids, + tear_down, ) from agent_env.task_step.context import TaskStepContext -from agent_env.task_step.task_steps.computer_recording import end_open_recordings from agent_env.task_step.task_step import TaskStep, TaskStepDependency logger = logging.getLogger(__name__) -PROVIDER_CLOSE_TIMEOUT_SECONDS = 5 - -TORN_DOWN_KEY = "torn_down_sandbox_ids" - - class TeardownSandboxesTaskStep(TaskStep): - """Terminate every sandbox behind the named deployed agents, envs and bare sandboxes.""" + """Take down every sandbox behind the named deployed agents, envs and bare sandboxes.""" type = "teardown_sandboxes" entity_refs = (EntityRef.env("env_ids[]"),) @@ -84,36 +80,24 @@ def from_dict(cls, data: dict) -> TeardownSandboxesTaskStep: ) async def execute(self, context: TaskStepContext) -> TaskStepContext: - targets, missing = self._resolve(context) + sandboxes, envs, missing = self._resolve(context) if missing: logger.warning(f"teardown_sandboxes: not deployed in this run, skipping: {', '.join(missing)}") already = set(context.metadata.get(TORN_DOWN_KEY) or []) - pending = {sid: (stype, default) for sid, (stype, default) in targets.items() if sid not in already} - await before_teardown(context, LAST_DRAIN_SECONDS, env_ids=self.env_ids, agent_names=self.agent_names) - results = await asyncio.gather( - *(_terminate(sid, stype, default) for sid, (stype, default) in pending.items()), - return_exceptions=True, - ) - terminated, failed = [], [] - for sandbox_id, result in zip(pending, results): - if isinstance(result, BaseException): - failed.append(sandbox_id) - logger.warning(f"teardown_sandboxes: terminate {sandbox_id} failed: {result!r}"[:300]) - else: - terminated.append(sandbox_id) - if terminated: - context.metadata[TORN_DOWN_KEY] = [*context.metadata.get(TORN_DOWN_KEY, []), *terminated] + await before_teardown(context, LAST_DRAIN_SECONDS, agent_names=self.agent_names) + report = await tear_down(context, sandboxes, envs, timeout=TERMINATE_TIMEOUT_SECONDS) + for sandbox, why in report.failed: + logger.warning(f"teardown_sandboxes: {sandbox.sandbox_id}: {why}"[:300]) logger.info( - f"teardown_sandboxes: terminated={len(terminated)} failed={len(failed)} " - f"already_torn_down={len(targets) - len(pending)} missing={len(missing)}" + f"teardown_sandboxes: terminated={len(report.terminated)} failed={len(report.failed)} " + f"already_torn_down={sum(sandbox.sandbox_id in already for sandbox in sandboxes)} missing={len(missing)}" ) return context - def _resolve( - self, context: TaskStepContext - ) -> tuple[dict[str, tuple[Optional[str], str]], list[str]]: - """``{sandbox_id: (sandbox_type, default_slot)}`` for every target, plus unmatched names.""" - targets: dict[str, tuple[Optional[str], str]] = {} + def _resolve(self, context: TaskStepContext) -> tuple[list[RecordedSandbox], list[DeployedEnv], list[str]]: + """Every sandbox behind the targets, once each, the env records named, and the names that matched nothing.""" + sandboxes: dict[str, RecordedSandbox] = {} + envs: list[DeployedEnv] = [] missing: list[str] = [] for name in self.agent_names: agents = [a for a in context.deployed_agents if a.agent_name == name] @@ -121,86 +105,18 @@ def _resolve( missing.append(f"agent {name!r}") for agent in agents: if agent.sandbox_id: - targets[agent.sandbox_id] = (agent.sandbox_type, "agent") + sandboxes[agent.sandbox_id] = RecordedSandbox(agent.sandbox_id, agent.sandbox_type, "agent") for name in self.sandbox_names: - sandboxes = [s for s in context.deployed_sandboxes if s.sandbox_name == name] - if not sandboxes: + found = [s for s in context.deployed_sandboxes if s.sandbox_name == name] + if not found: missing.append(f"sandbox {name!r}") - for sandbox in sandboxes: - targets[sandbox.sandbox_id] = (sandbox.sandbox_type, "sandbox") + for sandbox in found: + sandboxes[sandbox.sandbox_id] = RecordedSandbox(sandbox.sandbox_id, sandbox.sandbox_type, "sandbox") for env_id in self.env_ids: - envs = [e for e in context.deployed_envs if e.env_id == env_id] - if not envs: + named = [e for e in context.deployed_envs if e.env_id == env_id] + if not named: missing.append(f"env {env_id!r}") - for env in envs: - targets.update({sid: (stype, "env") for sid, stype in _env_sandbox_ids(env)}) - return targets, missing - - -async def before_teardown(context: TaskStepContext, within: float, env_ids: Optional[list[str]] = None, - agent_names: Optional[list[str]] = None) -> None: - """Save what has to reach the stores before sandboxes go, each part within ``within`` seconds: the changelogs - staged on ``agent_names`` and the recordings open on ``env_ids``, or on every agent and env when None.""" - for name in [None] if agent_names is None else agent_names: - await drain_all(staged_changelogs(context.metadata, agent_name=name), within=within) - if env_ids is None or env_ids: - try: - async with asyncio.timeout(within): - await end_open_recordings(context, env_ids) - except TimeoutError: - logger.warning("open recordings were not all ended within %gs", within) - - -def _env_sandbox_ids(env) -> list[tuple[str, Optional[str]]]: - """Every (sandbox_id, sandbox_type) of a deployed env: its primary sandbox plus ``sandbox_ids``. - - ``sandbox_ids`` values are an id or a ``{service: id}`` dict and inherit the env's type, unless - the key itself names a registered backend (a sandbox on another backend than the gateway's). - """ - if not isinstance(env, DeployedSandboxEnv): # an env outside our sandboxes owns its own lifetime - return [] - return _sandbox_ids_of(env.sandbox_id, env.sandbox_type, env.sandbox_ids) - - -def _sandbox_ids_of( - sandbox_id: Optional[str], sandbox_type: Optional[str], sandbox_ids: Optional[dict], -) -> list[tuple[str, Optional[str]]]: - ids = [(sandbox_id, sandbox_type)] if sandbox_id else [] - for key, value in (sandbox_ids or {}).items(): - entry_type = key if _is_backend(key) else sandbox_type - values = [value] if isinstance(value, str) else [v for v in value.values() if isinstance(v, str)] - ids.extend((v, entry_type) for v in values) - return ids - - -def _is_backend(name: str) -> bool: - try: - build_sandbox_provider(name) - except Exception: - return False - return True - - -_DEFAULT_PROVIDERS = { - "agent": get_agent_sandbox_provider, - "env": get_env_sandbox_provider, - "sandbox": get_sandbox_provider, -} - - -async def _terminate(sandbox_id: str, sandbox_type: Optional[str], default_slot: str) -> None: - """By id through the backend that made it; a record that doesn't name one goes through a live handle, since - "not found" from a provider that didn't make the sandbox doesn't mean it's gone.""" - if sandbox_type: - provider = build_sandbox_provider(sandbox_type) - try: - await provider.terminate_sandbox(sandbox_id) - finally: - try: - async with asyncio.timeout(PROVIDER_CLOSE_TIMEOUT_SECONDS): - await provider.close() - except Exception as error: - logger.warning("Closing teardown provider %s failed: %s: %s", sandbox_type, type(error).__name__, error) - return - provider: SandboxProvider = _DEFAULT_PROVIDERS[default_slot]() - await (await provider.get_sandbox(sandbox_id)).terminate() + envs.extend(named) + for env in named: + sandboxes.update({sid: RecordedSandbox(sid, stype, "env") for sid, stype in env_sandbox_ids(env)}) + return list(sandboxes.values()), envs, missing diff --git a/src/agent_env/task_step/task_steps/verifiers/run_container_unit_tests_verifier.py b/src/agent_env/task_step/task_steps/verifiers/run_container_unit_tests_verifier.py index c0e7b7d02..f289bba51 100644 --- a/src/agent_env/task_step/task_steps/verifiers/run_container_unit_tests_verifier.py +++ b/src/agent_env/task_step/task_steps/verifiers/run_container_unit_tests_verifier.py @@ -34,7 +34,7 @@ from typing import Any, ClassVar, Optional from agent_env.config.model import MODEL_HEADERS_ENV -from agent_env.providers.sandbox_providers.local_sandbox import host_url_for +from agent_env.providers.sandbox_providers.local_docker.provider import host_url_for from agent_env.providers.sandbox_providers.sandbox_provider import ( all_sandbox_container_env, registered_sandbox_provider_classes, diff --git a/tst/installer/test_container_journey.py b/tst/installer/test_container_journey.py index af104de73..477460382 100644 --- a/tst/installer/test_container_journey.py +++ b/tst/installer/test_container_journey.py @@ -133,7 +133,7 @@ def test_a_new_user_installs_agent_env_adds_plugins_runs_a_task_and_removes_them container.ok("python", "-m", "pip", "install", "-q", "--no-index", "--find-links", "/wheels", "uv") container.ok("uv", "tool", "install", "agentenv-framework") rows = container.ok("agent-env", "plugin", "list").stdout.split("STATUS\n", 1)[1] - assert re.fullmatch(r"agentenv-framework +\S+ +7 bundles +ok\n", rows), rows + assert re.fullmatch(r"agentenv-framework +\S+ +8 bundles +ok\n", rows), rows added = container.ok("agent-env", "plugin", "add", browser, grader, "--yes") assert "toy_navigate active" in added.stdout and "toy_grade active" in added.stdout diff --git a/tst/integration/cli/bundle_init_scopes_test.py b/tst/integration/cli/bundle_init_scopes_test.py index 039d393c8..6701f0510 100644 --- a/tst/integration/cli/bundle_init_scopes_test.py +++ b/tst/integration/cli/bundle_init_scopes_test.py @@ -17,7 +17,7 @@ from agent_env.bundle.resolve import build_step from agent_env.cli import cli from agent_env.config import configure, reset_config -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.sandbox_provider import ( reset_agent_sandbox_provider, reset_env_sandbox_provider, diff --git a/tst/integration/cli/run_bundle_claude_code_install_test.py b/tst/integration/cli/run_bundle_claude_code_install_test.py index 7ef041bf9..f3b76d0af 100644 --- a/tst/integration/cli/run_bundle_claude_code_install_test.py +++ b/tst/integration/cli/run_bundle_claude_code_install_test.py @@ -17,7 +17,7 @@ from agent_env.artifact.store import reset_artifact_store from agent_env.cli import cli from agent_env.config import configure, reset_config -from agent_env.providers.sandbox_providers.local_sandbox import _host_ips +from agent_env.providers.sandbox_providers.local_docker.provider import _host_ips from agent_env.store.object_store.local.grant_server import grant_server from tst.util.fake_anthropic import FakeAnthropic, Fault, tool_then_answer diff --git a/tst/integration/cli/run_bundle_claude_code_test.py b/tst/integration/cli/run_bundle_claude_code_test.py index 7e8eeb2f3..019d0728b 100644 --- a/tst/integration/cli/run_bundle_claude_code_test.py +++ b/tst/integration/cli/run_bundle_claude_code_test.py @@ -17,7 +17,7 @@ from agent_env.bundle.installed import find_bundle from agent_env.cli import cli from agent_env.config import configure, get_config, reset_config -from agent_env.providers.sandbox_providers.local_sandbox import _host_ips +from agent_env.providers.sandbox_providers.local_docker.provider import _host_ips from agent_env.store.object_store.local.grant_server import grant_server from agent_env.store.routing import namespace_routing from agent_env.task import Task diff --git a/tst/integration/cli/run_bundle_openhands_test.py b/tst/integration/cli/run_bundle_openhands_test.py index 7ffa94ca6..1ecd21167 100644 --- a/tst/integration/cli/run_bundle_openhands_test.py +++ b/tst/integration/cli/run_bundle_openhands_test.py @@ -18,7 +18,7 @@ from agent_env.artifact.store import reset_artifact_store from agent_env.cli import cli from agent_env.config import configure, reset_config -from agent_env.providers.sandbox_providers.local_sandbox import _host_ips +from agent_env.providers.sandbox_providers.local_docker.provider import _host_ips from agent_env.store.object_store.local.grant_server import grant_server from agent_env.store.routing import namespace_routing from agent_env.task.teardown import teardown_run diff --git a/tst/integration/cli/run_bundle_pydantic_ai_test.py b/tst/integration/cli/run_bundle_pydantic_ai_test.py index 9a9ac397b..7953f9dda 100644 --- a/tst/integration/cli/run_bundle_pydantic_ai_test.py +++ b/tst/integration/cli/run_bundle_pydantic_ai_test.py @@ -17,7 +17,7 @@ from agent_env.artifact.store import reset_artifact_store from agent_env.cli import cli from agent_env.config import configure, reset_config -from agent_env.providers.sandbox_providers.local_sandbox import _host_ips +from agent_env.providers.sandbox_providers.local_docker.provider import _host_ips from agent_env.store.object_store.local.grant_server import grant_server from agent_env.store.routing import namespace_routing from agent_env.task.teardown import teardown_run diff --git a/tst/integration/computer/base_computer_test.py b/tst/integration/computer/base_computer_test.py index b208e24ef..3920ea10e 100644 --- a/tst/integration/computer/base_computer_test.py +++ b/tst/integration/computer/base_computer_test.py @@ -49,6 +49,7 @@ def test_the_desktop_runs_as_the_session_user_with_the_settings_a_deploy_passes( wm = _exec(name, "xprop", "-root", "_NET_SUPPORTING_WM_CHECK") environment = _exec(name, "sh", "-c", 'tr "\\0" "\\n" < /proc/$(pgrep -x xfwm4)/environ') accessibility = _exec(name, "pgrep", "-c", "at-spi2-registr") + folders = set(_exec(name, "ls", "/home/user").split()) finally: subprocess.run(["docker", "rm", "-f", name], capture_output=True) assert {line.split()[0] for line in running.splitlines()} == {"dbus", "desktop", "xvfb"} @@ -57,6 +58,7 @@ def test_the_desktop_runs_as_the_session_user_with_the_settings_a_deploy_passes( assert "dimensions: 1600x900 pixels" in screen and "window id #" in wm assert "TZ=Europe/Paris" in environment.splitlines() and "DISPLAY=:99" in environment.splitlines() assert int(accessibility) == 1 + assert {"Desktop", "Documents", "Downloads", "Pictures"} <= folders def test_the_desktop_comes_back_however_often_x_dies(base): diff --git a/tst/integration/computer/computer_env_test.py b/tst/integration/computer/computer_env_test.py index 473a80020..e953f28b7 100644 --- a/tst/integration/computer/computer_env_test.py +++ b/tst/integration/computer/computer_env_test.py @@ -1,11 +1,12 @@ -"""A computer env composed on this machine: a bundle's env naming the shipped ubuntu-24.04 and reference by alias builds -an image for each arch this machine's Docker can build, pushed to the local registry and written as the env's computer; -the image of this machine's arch boots with everything running as the session user and passes the conformance probe -at full three times in a row; and another env that changes only what doesn't change the image reuses it. Needs Docker -and the local registry on localhost:5000.""" +"""The shipped desktop example's computer env, composed on this machine: naming ubuntu-24.04 and reference by alias, it +builds an image for each arch this machine's Docker can build, pushed to the local registry and written as the env's +computer, and gate 3 makes the image of this machine's arch ready; that image boots with everything running as the +session user and passes the conformance probe at full three times in a row; another env that changes only what doesn't +change the image reuses it; one with no network, deployed, is reached but reaches nothing, comes back from a reset +fresh, and leaves nothing once torn down; and a recording open when the computer resets or is torn down is ended by +the computer and registered whole. Needs Docker and the local registry on localhost:5000.""" import asyncio -import json import platform import subprocess import time @@ -18,72 +19,51 @@ from agentenv_protocol.computer.conformance.client import Driver from agent_env.artifact.artifacts.computer import ComputerArtifact -from agent_env.artifact.store import reset_artifact_store -from agent_env.bundle import parse_bundle -from agent_env.bundle.installed import installed_bundles -from agent_env.bundle.materialize import materialize -from agent_env.bundle.plan import plan_bundle -from agent_env.bundle.resolve import BuiltImage, resolve_bundle -from agent_env.config import configure, reset_config from agent_env.env import Env from agent_env.env.envs.computer.env import ComputerEnv +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.store.routing import namespace_routing +from agent_env.store.object_store.local.grant_server import grant_server +from agent_env.task.teardown import teardown_run +from agent_env.task_step.context import TaskStepContext +from agent_env.task_step.task_steps.computer_recording import ( + RECORDINGS_KEY, + EndComputerRecordingTaskStep, + StartComputerRecordingTaskStep, +) +from agent_env.task_step.task_steps.reset_env import ResetEnvTaskStep from tst.integration.computer.conftest import docker pytestmark = [pytest.mark.integration, pytest.mark.int_test_slow] ARCH = {"x86_64": "amd64", "amd64": "amd64", "aarch64": "arm64", "arm64": "arm64"}[platform.machine().lower()] PORT = 18765 -DESKTOP = 'type = "computer"\ncomputer = "ubuntu-24.04"\ndriver = "reference"\n' - - -@pytest.fixture(scope="module") -def composed(tmp_path_factory): - """The bundle's computer env, composed once for the module's tests, in stores of the module's own.""" - root = tmp_path_factory.mktemp("composed") - patch = pytest.MonkeyPatch() - patch.chdir(root) - patch.setenv("XDG_STATE_HOME", str(root / "state")) - patch.setenv("AGENT_ENV_DOCUMENT_STORE", "local") - patch.setenv("AGENT_ENV_OBJECT_STORE", "local") - configure() - reset_artifact_store() - bundle = root / "desk" - for path, text in {"envs/desktop/env.toml": DESKTOP, "artifacts/notes/notes.txt": "a note\n", "tasks/t.json": json.dumps( - [{"id": "load", "type": "load_artifact", "env_id": "desktop", "artifact_id": "notes"}])}.items(): - (bundle / path).parent.mkdir(parents=True, exist_ok=True) - (bundle / path).write_text(text) - with namespace_routing(): - plan = plan_bundle(resolve_bundle(parse_bundle(bundle)), installed=installed_bundles) - done = materialize(plan, build_here=frozenset(w.id for w in plan.writes if isinstance(w.source, BuiltImage))) - env = next(Env.get(item.write.id) for item in done.writes if item.write.id.endswith("/desktop")) - yield env - reset_artifact_store() - reset_config() - patch.undo() -def test_each_arch_this_machine_builds_is_composed_pushed_and_written_as_the_envs_computer(composed): - results = composed.metadata["composed"]["container"] +def test_each_arch_this_machine_builds_is_composed_pushed_and_written_as_the_envs_computer(desktop): + results = desktop.metadata["composed"]["container"] with namespace_routing(): - computer = ComputerArtifact.get(composed.metadata["computer"]["id"], composed.metadata["computer"]["version"]) + computer = ComputerArtifact.get(desktop.metadata["computer"]["id"], desktop.metadata["computer"]["version"]) - assert results[ARCH]["status"] == "unverified" and results[ARCH]["emulated"] is False + assert results[ARCH]["status"] == "ready" and results[ARCH]["emulated"] is False + assert results[ARCH]["probe"]["passed"] and results[ARCH]["probe"]["level"] == "full" + assert results[ARCH]["card"]["name"] == desktop.environment_name assert {"mss", "python-xlib", "av", "agentenv-framework-protocol"} <= set(results[ARCH]["installed"]) for arch, result in results.items(): - assert result["status"] == "unverified" or "its Docker has no emulation for it" in result["reason"], result - built = sorted(arch for arch, result in results.items() if result["status"] == "unverified") + if arch != ARCH: + assert result["status"] == "unverified" or "its Docker has no emulation for it" in result["reason"], result + built = sorted(arch for arch, result in results.items() if result["status"] in ("ready", "unverified")) assert sorted(computer.arch) == built assert {arch: result["status"] for arch, result in computer.metadata["validated_images"]["container"].items()} == { arch: "passed" for arch in built} + assert computer.metadata["validated_boot"]["container"][ARCH] == results[ARCH]["probe"] @pytest.mark.asyncio -async def test_the_image_of_this_machines_arch_boots_as_the_session_user_and_passes_the_probe_at_full(composed): - image = composed.metadata["composed"]["container"][ARCH]["image"] +async def test_the_image_of_this_machines_arch_boots_as_the_session_user_and_passes_the_probe_at_full(desktop): + image = desktop.metadata["composed"]["container"][ARCH]["image"] name = f"agentenv-test-computer-{uuid.uuid4().hex[:8]}" - docker("run", "-d", "--name", name, "-p", f"127.0.0.1::{PORT}", - "-e", 'AGENTENV_COMPUTER_SETTINGS={"data_paths": ["fixture"]}', image) + docker("run", "-d", "--name", name, "-p", f"127.0.0.1::{PORT}", image) try: url = await _ready(name) reports = [await run_probe(url, "full") for _ in range(3)] @@ -96,20 +76,77 @@ async def test_the_image_of_this_machines_arch_boots_as_the_session_user_and_pas assert users == {"user"} -def test_an_env_that_changes_only_its_display_and_name_reuses_the_image(composed): +def test_an_env_that_changes_only_its_display_and_name_reuses_the_image(desktop): with namespace_routing(): started = time.monotonic() - other = ComputerEnv.put(id=f"{composed.id}-other", computer=composed.computer, driver=composed.driver, + other = ComputerEnv.put(id=f"{desktop.id}-other", computer=desktop.computer, driver=desktop.driver, environment_name="other", display={"width": 1024, "height": 768}) took = time.monotonic() - started results = Env.get(other.id).metadata["composed"]["container"] - first = composed.metadata["composed"]["container"] + first = desktop.metadata["composed"]["container"] assert results[ARCH]["reused"] is True and results[ARCH]["image"] == first[ARCH]["image"] + assert results[ARCH]["status"] == "ready" and results[ARCH]["card"]["name"] == "other" assert results[ARCH]["installed"] == first[ARCH]["installed"] and results[ARCH]["emulated"] is False assert took < 60, took +@pytest.mark.asyncio +async def test_with_no_network_it_is_reached_reaches_nothing_comes_back_fresh_and_leaves_nothing(desktop): + with namespace_routing(): + closed = ComputerEnv.put(id=f"{desktop.id}-closed", computer=desktop.computer, driver=desktop.driver, + environment_name="closed", network={"mode": "allowlist"}) + deployed = await Env.get(closed.id).deploy() + context = TaskStepContext(deployed_envs=[deployed]) + try: + driver = await Driver.connect(deployed.environment_url, 30) + egress = await driver.run("python3", "-c", "import urllib.request; urllib.request.urlopen('http://1.1.1.1', timeout=5)") + await driver.run("sh", "-c", "echo x > Desktop/left.txt") + await ResetEnvTaskStep(id="reset", version=None, env_id=closed.id, fail_task_on_error=True).execute(context) + left = await (await Driver.connect(deployed.environment_url, 30)).run("test", "-e", "Desktop/left.txt") + finally: + report = await teardown_run(context) + sandbox = deployed.sandbox_id + + assert egress["exit_code"] != 0 and "Network is unreachable" in egress["stderr"], egress + assert left["exit_code"] != 0 + assert [s.sandbox_id for s in report.terminated] == [sandbox] and not report.failed + for listing in (["ps", "-aq", "--filter", f"label=agentenv.sandbox={sandbox}"], + ["ps", "-aq", "--filter", f"name=agent-{sandbox}"], + ["network", "ls", "-q", "--filter", f"label=agentenv.sandbox={sandbox}"]): + assert not docker(*listing).stdout.strip(), listing + assert LocalSandbox.find_work_dir(sandbox) is None + + +@pytest.mark.asyncio +async def test_a_recording_open_at_a_reset_or_at_teardown_is_ended_by_its_computer_and_registered_whole(desktop): + with namespace_routing(): + deployed = await Env.get(desktop.id).deploy() + context = TaskStepContext(instance_id=f"run-{uuid.uuid4().hex[:8]}", deployed_envs=[deployed]) + try: + context = await _record("before-reset", desktop.id).execute(context) + await asyncio.sleep(3) + await ResetEnvTaskStep(id="reset", version=None, env_id=desktop.id, fail_task_on_error=True).execute(context) + context = await EndComputerRecordingTaskStep(id="end", version=None, name="before-reset").execute(context) + context = await _record("left-open", desktop.id).execute(context) + await asyncio.sleep(3) + finally: + report = await teardown_run(context) + grant_server(None, None).close() + recordings = context.metadata[RECORDINGS_KEY] + + assert not report.failed + for name, ended_by in (("before-reset", "reset"), ("left-open", "teardown")): + entry = recordings[name] + assert (entry["state"], entry["ended_by"], entry["missing_segments"]) == ("ended", ended_by, 0), entry + assert entry["segments"] >= 1 + + +def _record(name: str, env_id: str) -> StartComputerRecordingTaskStep: + return StartComputerRecordingTaskStep(id=f"start-{name}", version=None, env_id=env_id, name=name, + fail_task_on_error=True) + + async def _ready(name: str) -> str: for _ in range(240): port = subprocess.run(["docker", "port", name, str(PORT)], capture_output=True, text=True).stdout diff --git a/tst/integration/computer/conftest.py b/tst/integration/computer/conftest.py index c4543ba42..fa1bbe76e 100644 --- a/tst/integration/computer/conftest.py +++ b/tst/integration/computer/conftest.py @@ -1,5 +1,6 @@ -"""The ubuntu-24.04 base computer, built from its folder once for the tests here and tagged for the local registry, and -``docker``, which runs the docker CLI for them.""" +"""The ubuntu-24.04 base computer, built from its folder once for the tests here and tagged for the local registry; the +shipped desktop example's computer env, composed once for a module's tests; and ``docker``, which runs the docker CLI +for them.""" import subprocess import uuid @@ -8,6 +9,15 @@ import pytest import agent_env +from agent_env.artifact.store import reset_artifact_store +from agent_env.bundle import parse_bundle +from agent_env.bundle.installed import find_bundle, installed_bundles +from agent_env.bundle.materialize import materialize +from agent_env.bundle.plan import plan_bundle +from agent_env.bundle.resolve import BuiltImage, resolve_bundle +from agent_env.config import configure, reset_config +from agent_env.env import Env +from agent_env.store.routing import namespace_routing BASE = Path(agent_env.__file__).parent / "computers" / "artifacts" / "ubuntu-24.04" @@ -20,5 +30,27 @@ def base(): subprocess.run(["docker", "rmi", "-f", tag], capture_output=True) +@pytest.fixture(scope="module") +def desktop(tmp_path_factory): + """The shipped desktop example's computer env, composed for the module in stores of the module's own: an image for + each arch this machine's Docker builds, the native one probed by gate 3.""" + root = tmp_path_factory.mktemp("desktop") + patch = pytest.MonkeyPatch() + patch.chdir(root) + patch.setenv("XDG_STATE_HOME", str(root / "state")) + patch.setenv("AGENT_ENV_DOCUMENT_STORE", "local") + patch.setenv("AGENT_ENV_OBJECT_STORE", "local") + configure() + reset_artifact_store() + bundle = find_bundle("desktop") + with namespace_routing(): + plan = plan_bundle(resolve_bundle(parse_bundle(bundle.root, id_root=bundle.id_root)), installed=installed_bundles) + materialize(plan, build_here=frozenset(w.id for w in plan.writes if isinstance(w.source, BuiltImage))) + yield Env.get(f"{bundle.id_root}/desktop") + reset_artifact_store() + reset_config() + patch.undo() + + def docker(*args: str) -> subprocess.CompletedProcess: return subprocess.run(["docker", *args], check=True, capture_output=True) diff --git a/tst/integration/computer/scaffold_test.py b/tst/integration/computer/desktop_example_test.py similarity index 61% rename from tst/integration/computer/scaffold_test.py rename to tst/integration/computer/desktop_example_test.py index f269ce6f8..61afb2b7d 100644 --- a/tst/integration/computer/scaffold_test.py +++ b/tst/integration/computer/desktop_example_test.py @@ -1,13 +1,11 @@ -"""The reference driver on a desktop of its own before agent-env can compose a computer: the scaffold, an image under -this folder that lays it on the ubuntu-24.04 base, deployed by ``agent-env run`` as an MCP server on the server -provider, passes the conformance probe at full three runs in a row, records a scripted drive of its desk that lines up -with the drive's screenshots, and an agent drives its desktop through the computer_* tools under a recording. Needs a -Docker daemon and the local registry; the agent's task, a model endpoint too.""" +"""The shipped desktop example through ``agent-env run``: its computer env, deployed on local Docker, passes the +conformance probe at full; a scripted drive of the conformance fixture on it is recorded on the clock its screenshots +carry; and an agent given the computer's skill makes a folder through the computer_* tools under a recording. Needs +Docker and the local registry; the agent's task, a model endpoint too.""" import json import logging import re -import shutil from datetime import datetime from pathlib import Path @@ -16,8 +14,6 @@ from click.testing import CliRunner from PIL import Image -import agent_env -import agentenv_protocol from agent_env.artifact import Artifact from agent_env.artifact.store import reset_artifact_store from agent_env.cli import cli @@ -29,11 +25,7 @@ pytestmark = [pytest.mark.integration, pytest.mark.int_test_slow] -SCAFFOLD = Path(__file__).parent / "scaffold" -DRIVER = Path(agent_env.__file__).parent / "drivers" / "envs" / "reference" / "driver.py" -FIXTURE = Path(agentenv_protocol.__file__).parent / "computer" / "conformance" / "fixture.py" -PROTOCOL = Path(agentenv_protocol.__file__).parents[2] -PROTOCOL_FILES = ("pyproject.toml", "README.md", "LICENSE", "NOTICE", "THIRD_PARTY_NOTICES.md") +RECORDING = Path(__file__).parent / "recording" @pytest.fixture @@ -58,65 +50,15 @@ def state(monkeypatch, tmp_path): reset_config() -def _scaffold(root: Path, base: str) -> Path: - """The scaffold bundle on the base image ``base``, with the reference driver, the probe's fixture and this - checkout's protocol beside its Dockerfile.""" - shutil.copytree(SCAFFOLD, root, ignore=shutil.ignore_patterns("__pycache__")) - desktop = root / "envs" / "desktop" - dockerfile = desktop / "Dockerfile" - text = dockerfile.read_text() - assert "\nARG BASE\n" in text, "the scaffold's Dockerfile names its base by ARG BASE, which _scaffold sets" - dockerfile.write_text(text.replace("\nARG BASE\n", f"\nARG BASE={base}\n")) - shutil.copy(DRIVER, desktop / "driver.py") - shutil.copy(FIXTURE, desktop / "fixture.py") - shutil.copytree(PROTOCOL / "src", desktop / "protocol" / "src", ignore=shutil.ignore_patterns("__pycache__")) - for name in PROTOCOL_FILES: - shutil.copy(PROTOCOL / name, desktop / "protocol" / name) - return root - - -def _run(root: Path, task: str): - return CliRunner().invoke(cli, ["run", str(root), "--task", task]) - - -def _recorded(output: str, name: str, tmp: Path) -> tuple[dict, dict, list]: - """The run's context, the manifest of its recording ``name``, and every frame of the video with its time on the - computer's clock.""" - instance_id = re.search(r"instance (@local/\S+)", output).group(1) - with namespace_routing(): - context = get_task_instance_store().get(instance_id).context - entry = context["metadata"]["computer_recordings"][name] - universe = Artifact.get(entry["artifact"]["id"], version=entry["artifact"]["version"]) - objects = get_config().get_object_store() - files = {name: objects.get(Artifact.get(ref).object_url) for name, ref in universe.file_artifact_ids.items()} - manifest = json.loads(files["manifest.json"]) - frames = [] - for segment in manifest["segments"]: - path = tmp / segment["path"].replace("/", "-") - path.write_bytes(files[segment["path"].replace("/", "-")]) - with av.open(str(path)) as video: - start = _when(segment["started_at"]) - frames += [(start.timestamp() + frame.time, frame.to_image()) for frame in video.decode(video=0)] - return context, manifest, frames - - -def _when(iso: str) -> datetime: - return datetime.fromisoformat(iso.replace("Z", "+00:00")) - - -def _thumb(image: Image.Image) -> bytes: - return image.convert("L").resize((64, 40)).tobytes() - - -def test_the_reference_driver_passes_the_probe_at_full(state, base): - ran = _run(_scaffold(state / "scaffold", base), "probe") +def test_the_desktop_examples_computer_passes_the_probe_at_full(state): + ran = _run("desktop", "probe") assert ran.exit_code == 0, ran.output assert "passed (probe: 1)" in ran.output, ran.output -def test_a_scripted_drive_is_recorded_on_the_clock_its_screenshots_carry(state, base): - ran = _run(_scaffold(state / "scaffold", base), "recording") +def test_a_scripted_drive_is_recorded_on_the_clock_its_screenshots_carry(state): + ran = _run(str(RECORDING), "recording") assert ran.exit_code == 0, ran.output assert "passed (drive: 1)" in ran.output, ran.output @@ -137,16 +79,53 @@ def test_a_scripted_drive_is_recorded_on_the_clock_its_screenshots_carry(state, shots = drive["shots"] n = next(i for i, s in enumerate(shots) if s["args"].get("x") == x + w // 2) corner = lambda image: image.getpixel((x + 8, y + 8)) # noqa: E731 clear of the pointer drawn at the click - changed = next(t for t, image in frames if corner(image) != corner(frames[0][1])) - # The cell changes after the screenshot before its click and by the one after it, both on the computer's clock. - assert _when(shots[n - 1]["captured_at"]).timestamp() < changed <= _when(shots[n]["captured_at"]).timestamp() + interval + before = _when(shots[n - 1]["captured_at"]).timestamp() + _, unclicked = min(frames, key=lambda frame: abs(frame[0] - before)) + changed = next(t for t, image in frames if t > before and corner(image) != corner(unclicked)) + # The cell changes after the screenshot before its click, by the one after it, both on the computer's clock. + assert changed <= _when(shots[n]["captured_at"]).timestamp() + interval @skip_without_model_endpoint() -def test_an_agent_drives_the_reference_drivers_desktop_under_a_recording(state, base): - ran = _run(_scaffold(state / "scaffold", base), "agent") +def test_an_agent_given_the_computers_skill_makes_a_folder_under_a_recording(state): + ran = _run("desktop", "agent") assert ran.exit_code == 0, ran.output assert "passed (check: 1)" in ran.output, ran.output context, manifest, frames = _recorded(ran.output, "agent", state) - assert context["metadata"]["computer_recordings"]["agent"]["state"] == "ended" and frames + entry = context["metadata"]["computer_recordings"]["agent"] + assert (entry["state"], entry["missing_segments"]) == ("ended", 0) and frames + assert "## Shortcuts" in context["metadata"]["env_skills"]["default-agent"]["desktop"]["desktop"] + + +def _run(bundle: str, task: str): + return CliRunner().invoke(cli, ["run", bundle, "--task", task]) + + +def _recorded(output: str, name: str, tmp: Path) -> tuple[dict, dict, list]: + """The run's context, the manifest of its recording ``name``, and every frame of the video with its time on the + computer's clock.""" + instance_id = re.search(r"instance (@local/\S+)", output).group(1) + with namespace_routing(): + context = get_task_instance_store().get(instance_id).context + entry = context["metadata"]["computer_recordings"][name] + universe = Artifact.get(entry["artifact"]["id"], version=entry["artifact"]["version"]) + objects = get_config().get_object_store() + files = {name: objects.get(Artifact.get(ref).object_url) for name, ref in universe.file_artifact_ids.items()} + manifest = json.loads(files["manifest.json"]) + frames = [] + for segment in manifest["segments"]: + path = tmp / segment["path"].replace("/", "-") + path.write_bytes(files[segment["path"].replace("/", "-")]) + with av.open(str(path)) as video: + start = _when(segment["started_at"]) + frames += [(start.timestamp() + frame.time, frame.to_image()) for frame in video.decode(video=0)] + return context, manifest, frames + + +def _when(iso: str) -> datetime: + return datetime.fromisoformat(iso.replace("Z", "+00:00")) + + +def _thumb(image: Image.Image) -> bytes: + return image.convert("L").resize((64, 40)).tobytes() diff --git a/tst/integration/computer/scaffold/artifacts/drive_desk/verify.py b/tst/integration/computer/recording/artifacts/drive_desk/verify.py similarity index 76% rename from tst/integration/computer/scaffold/artifacts/drive_desk/verify.py rename to tst/integration/computer/recording/artifacts/drive_desk/verify.py index 898048e35..9eec904dd 100644 --- a/tst/integration/computer/scaffold/artifacts/drive_desk/verify.py +++ b/tst/integration/computer/recording/artifacts/drive_desk/verify.py @@ -1,17 +1,19 @@ -"""A scripted drive of the Fixture Desk while the desktop records: the clicks, typing and scroll a person would make, -each answered with a screenshot. A test finds each moment in the video by its send time, its screenshot's capture time +"""A scripted drive of the Fixture Desk, the conformance fixture started on the computer as the probe starts it, while +the desktop records: the clicks, typing and scroll a person would make, each answered with a screenshot. A test finds each moment in the video by its send time, its screenshot's capture time and thumbnail, and the clicked grid cell's box.""" import asyncio import base64 import io import json from datetime import datetime, timezone +from importlib import resources from PIL import Image from agentenv_protocol.computer.conformance.client import Driver THUMB = (64, 40) +FIXTURE = resources.files("agentenv_protocol.computer.conformance").joinpath("fixture.py").read_bytes() def _thumb(result): @@ -21,7 +23,9 @@ def _thumb(result): async def verify(mcp_url): driver = await Driver.connect(mcp_url.rsplit("/mcp", 1)[0], 60) - for _ in range(60): # the Fixture Desk starts beside the driver and writes its layout once it is up + await driver.add_file("desk/fixture.py", FIXTURE) + await driver.run("sh", "-c", "python3 desk/fixture.py fixture > desk/fixture.log 2>&1 &") + for _ in range(60): # the Fixture Desk writes its layout once it is up out = await driver.run("cat", "fixture/layout.json") if out["exit_code"] == 0 and out["stdout"].strip(): break diff --git a/tst/integration/computer/recording/tasks/recording.json b/tst/integration/computer/recording/tasks/recording.json new file mode 100644 index 000000000..ad000361f --- /dev/null +++ b/tst/integration/computer/recording/tasks/recording.json @@ -0,0 +1,8 @@ +[ + {"id": "env", "type": "deploy_env", "env_id": "@local/agentenv-framework/desktop/desktop"}, + {"id": "record", "type": "start_computer_recording", "env_id": "@local/agentenv-framework/desktop/desktop", + "name": "desk", "fps": 10, "depends_on": ["env"]}, + {"id": "drive", "type": "env_outcome_verifier", "env_id": "@local/agentenv-framework/desktop/desktop", + "file_artifact_id": "drive_desk", "verifier_id": "drive", "depends_on": ["record"]}, + {"id": "end", "type": "end_computer_recording", "name": "desk", "depends_on": ["drive"]} +] diff --git a/tst/integration/computer/recording_chaos_test.py b/tst/integration/computer/recording_chaos_test.py index 6cd1e2189..b619def33 100644 --- a/tst/integration/computer/recording_chaos_test.py +++ b/tst/integration/computer/recording_chaos_test.py @@ -1,13 +1,16 @@ -"""computer-recording/v1 on the scaffold's desktop when things go wrong: the X server killed mid-recording, segments -rolling over, the disk filling up, clicks while it records at 30 fps, and the driver restarted. Each test runs the -scaffold's image in a container of its own, so it can kill, fill and restart it, and decodes every segment it reads -back. Needs a Docker daemon.""" +"""computer-recording/v1 on the shipped desktop example's computer when things go wrong: the X server killed +mid-recording, segments rolling over, the disk filling up, clicks on the conformance fixture while it records at 30 fps, +and the driver restarted. Each test runs the computer's composed image in a container of its own, so it can kill, fill +and restart it, and decodes every segment it reads back. Needs Docker and the local registry on localhost:5000.""" import asyncio import json +import platform import subprocess import uuid from contextlib import asynccontextmanager +from datetime import datetime +from importlib import resources from pathlib import Path import av @@ -18,7 +21,6 @@ from agentenv_protocol.computer.conformance.client import Driver from agentenv_protocol.computer.recording import DISK_CHECK_S, RECORDINGS_DIR from tst.integration.computer.conftest import docker -from tst.integration.computer.scaffold_test import _scaffold, _when pytestmark = [pytest.mark.integration, pytest.mark.int_test_slow] @@ -28,13 +30,13 @@ FPS = 10 +ARCH = {"x86_64": "amd64", "amd64": "amd64", "aarch64": "arm64", "arm64": "arm64"}[platform.machine().lower()] +FIXTURE = resources.files("agentenv_protocol.computer.conformance").joinpath("fixture.py").read_bytes() + + @pytest.fixture(scope="module") -def image(tmp_path_factory, base): - tag = f"agentenv-test-recording-desk:{uuid.uuid4().hex[:8]}" - scaffold = _scaffold(tmp_path_factory.mktemp("desk") / "scaffold", base) - docker("build", "-q", "-t", tag, str(scaffold / "envs" / "desktop")) - yield tag - subprocess.run(["docker", "rmi", "-f", tag], capture_output=True) +def image(desktop): + return desktop.metadata["composed"]["container"][ARCH]["image"] @pytest.mark.asyncio @@ -113,8 +115,7 @@ async def test_a_restarted_driver_forgets_its_recording_and_keeps_the_segments_i async def _desk(image: str, *docker_args: str): """The desktop in a container of its own, and a client for its driver once it is ready.""" name = f"agentenv-test-desk-{uuid.uuid4().hex[:8]}" - docker("run", "-d", "--name", name, "-p", f"127.0.0.1::{PORT}", *docker_args, - "-e", 'AGENTENV_COMPUTER_SETTINGS={"data_paths": ["fixture"]}', image) + docker("run", "-d", "--name", name, "-p", f"127.0.0.1::{PORT}", *docker_args, image) try: yield name, await _ready(name) finally: @@ -137,6 +138,9 @@ async def _ready(name: str) -> Driver: async def _layout(driver: Driver) -> dict: + """The Fixture Desk, started through data/add and computer-exec as the probe starts it, once it wrote its layout.""" + await driver.add_file("desk/fixture.py", FIXTURE) + await driver.run("sh", "-c", "python3 desk/fixture.py fixture > desk/fixture.log 2>&1 &") for _ in range(120): out = await driver.run("cat", "fixture/layout.json") if out["exit_code"] == 0 and out["stdout"].strip(): @@ -165,6 +169,10 @@ def _decoded(name: str, manifest: RecordingManifest, tmp: Path) -> list[int]: return counts +def _when(iso: str) -> datetime: + return datetime.fromisoformat(iso.replace("Z", "+00:00")) + + def _gaps(manifest: RecordingManifest) -> list[float]: segments = manifest.segments return [(_when(b.started_at) - _when(a.ended_at)).total_seconds() for a, b in zip(segments, segments[1:])] diff --git a/tst/integration/computer/scaffold/artifacts/check_desk/verify.py b/tst/integration/computer/scaffold/artifacts/check_desk/verify.py deleted file mode 100644 index 414f26a89..000000000 --- a/tst/integration/computer/scaffold/artifacts/check_desk/verify.py +++ /dev/null @@ -1,25 +0,0 @@ -"""Whether the agent did the desk task through the computer: the Fixture Desk's own record of what reached it, read -through the driver's computer-exec.""" -import json -import uuid - -from agentenv_protocol import client as protocol - -EXEC = "urn:agentenv:computer-exec/v1" - - -async def verify(mcp_url): - base = mcp_url.rsplit("/mcp", 1)[0] - card = await protocol.get_card(base) - out = await protocol.invoke_extension(base, card, EXEC, {"argv": ["cat", "fixture/state.json"], - "request_id": uuid.uuid4().hex}, method="run") - state = json.loads(out["stdout"]) - return [ - {"id": "name", "result": state["name"] == "Ada Lovelace", "description": f"name {state['name']!r}"}, - {"id": "subscribed", "result": state["subscribed"] is True, "description": f"subscribed {state['subscribed']}"}, - {"id": "item", "result": state["selected"] == "Item 37", "description": f"selected {state['selected']!r}"}, - {"id": "saved", "result": state["saved_count"] >= 1, "description": f"saved {state['saved_count']} times"}, - {"id": "cell", "result": state["cells"].get("3,4", 0) >= 1, "description": f"cells {state['cells']}"}, - {"id": "typed-as-keys", "result": len(state["keys"]) >= len("Ada Lovelace"), - "description": f"{len(state['keys'])} key events reached the field"}, - ] diff --git a/tst/integration/computer/scaffold/artifacts/check_probe/verify.py b/tst/integration/computer/scaffold/artifacts/check_probe/verify.py deleted file mode 100644 index 3443b95cf..000000000 --- a/tst/integration/computer/scaffold/artifacts/check_probe/verify.py +++ /dev/null @@ -1,15 +0,0 @@ -"""The conformance probe at full against the desktop, RUNS times: a check passes only when it passed, or didn't -apply, in every run.""" -from agentenv_protocol.computer.conformance import run_probe - -RUNS = 3 - - -async def verify(mcp_url): - base, outcomes = mcp_url.rsplit("/mcp", 1)[0], {} - for _ in range(RUNS): - for check in (await run_probe(base, "full")).checks: - seen = outcomes.setdefault(f"{check.check} {check.name}", []) - seen.append(check.status if check.status in ("passed", "not_applicable") else f"{check.status}: {check.detail}") - return [{"id": name, "result": all(s in ("passed", "not_applicable") for s in seen), "description": "; ".join(seen)} - for name, seen in outcomes.items()] diff --git a/tst/integration/computer/scaffold/envs/desktop/Dockerfile b/tst/integration/computer/scaffold/envs/desktop/Dockerfile deleted file mode 100644 index c05df40f7..000000000 --- a/tst/integration/computer/scaffold/envs/desktop/Dockerfile +++ /dev/null @@ -1,12 +0,0 @@ -# The scaffold: the reference driver and the Fixture Desk on the ubuntu-24.04 base computer, until agent-env composes -# computers. The test builds the base and names it in BASE, and stages driver.py, fixture.py and the protocol package -# (protocol/) beside this file. -ARG BASE -FROM ${BASE} -USER root -COPY protocol /opt/protocol -RUN python3 -m venv /opt/driver \ - && /opt/driver/bin/pip install --no-cache-dir "/opt/protocol[computer]" "mss>=9,<11" "python-xlib>=0.33,<0.34" -COPY driver.py fixture.py /opt/desktop/ -COPY desktop.conf /etc/supervisor/conf.d/desktop.conf -USER user diff --git a/tst/integration/computer/scaffold/envs/desktop/desktop.conf b/tst/integration/computer/scaffold/envs/desktop/desktop.conf deleted file mode 100644 index 036b30b9e..000000000 --- a/tst/integration/computer/scaffold/envs/desktop/desktop.conf +++ /dev/null @@ -1,14 +0,0 @@ -[program:reference-driver] -command=/opt/driver/bin/python /opt/desktop/driver.py -priority=30 -autorestart=true -redirect_stderr=true -stdout_logfile=/var/log/agentenv/reference-driver.log - -[program:fixture-desk] -command=/usr/local/bin/start-desktop with-x python3 /opt/desktop/fixture.py /home/user/fixture -priority=30 -autorestart=true -startsecs=0 -redirect_stderr=true -stdout_logfile=/var/log/agentenv/fixture-desk.log diff --git a/tst/integration/computer/scaffold/envs/desktop/env.toml b/tst/integration/computer/scaffold/envs/desktop/env.toml deleted file mode 100644 index c7adc72e8..000000000 --- a/tst/integration/computer/scaffold/envs/desktop/env.toml +++ /dev/null @@ -1,2 +0,0 @@ -environment_name = "desktop" -env_provider_type = "server" diff --git a/tst/integration/computer/scaffold/tasks/agent.json b/tst/integration/computer/scaffold/tasks/agent.json deleted file mode 100644 index f2398aace..000000000 --- a/tst/integration/computer/scaffold/tasks/agent.json +++ /dev/null @@ -1,8 +0,0 @@ -[ - {"id": "env", "type": "deploy_env", "env_id": "desktop", "sandbox_type": "local"}, - {"id": "agent", "type": "deploy_agent", "a2a_agent_id": "claude-code", "env_ids": ["desktop"], "sandbox_type": "local"}, - {"id": "record", "type": "start_computer_recording", "env_id": "desktop", "name": "agent", "fps": 10, "depends_on": ["env"]}, - {"id": "task", "type": "prompt_agent", "prompt_id": "task", "depends_on": ["record", "agent"], "prompt": "You have a computer with a graphical desktop, which you use through the computer_* tools: take a screenshot to see it, then click, type, scroll and press keys. A window called Fixture Desk is open. In it: type the name Ada Lovelace into the Name field, tick the Subscribe checkbox, select Item 37 in the list on the left (scroll the list to find it), click the cell in the fourth row and fifth column of the white grid of squares, and finally click Save. Take a screenshot after each step to check it worked. Use only the computer_* tools, and reply DONE when everything is in place."}, - {"id": "check", "type": "env_outcome_verifier", "env_id": "desktop", "file_artifact_id": "check_desk", "verifier_id": "desk", "depends_on": ["task"]}, - {"id": "end", "type": "end_computer_recording", "name": "agent", "depends_on": ["task"]} -] diff --git a/tst/integration/computer/scaffold/tasks/probe.json b/tst/integration/computer/scaffold/tasks/probe.json deleted file mode 100644 index d71829cde..000000000 --- a/tst/integration/computer/scaffold/tasks/probe.json +++ /dev/null @@ -1,5 +0,0 @@ -[ - {"id": "env", "type": "deploy_env", "env_id": "desktop", "sandbox_type": "local"}, - {"id": "probe", "type": "env_outcome_verifier", "env_id": "desktop", "file_artifact_id": "check_probe", - "verifier_id": "probe", "depends_on": ["env"]} -] diff --git a/tst/integration/computer/scaffold/tasks/recording.json b/tst/integration/computer/scaffold/tasks/recording.json deleted file mode 100644 index a407607db..000000000 --- a/tst/integration/computer/scaffold/tasks/recording.json +++ /dev/null @@ -1,8 +0,0 @@ -[ - {"id": "env", "type": "deploy_env", "env_id": "desktop", "sandbox_type": "local"}, - {"id": "record", "type": "start_computer_recording", "env_id": "desktop", "name": "desk", "fps": 10, - "depends_on": ["env"]}, - {"id": "drive", "type": "env_outcome_verifier", "env_id": "desktop", "file_artifact_id": "drive_desk", - "verifier_id": "drive", "depends_on": ["record"]}, - {"id": "end", "type": "end_computer_recording", "name": "desk", "depends_on": ["drive"]} -] diff --git a/tst/integration/env/envs/build_context_local_test.py b/tst/integration/env/envs/build_context_local_test.py index dedc04dc0..26e803b0e 100644 --- a/tst/integration/env/envs/build_context_local_test.py +++ b/tst/integration/env/envs/build_context_local_test.py @@ -17,7 +17,7 @@ from agent_env.env import legacy_protocol from agent_env.env.envs.mcp_server import MCPServerEnv from agent_env.providers.sandbox_providers import sandbox -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.utils.docker.build import normalize_platform from tst.integration.env.envs.server_env_local_test import local_stack as local_stack from tst.integration.store.local_parallel_deploy_test import _docker, _seed_bootstrap_envs diff --git a/tst/integration/env/envs/server_env_local_test.py b/tst/integration/env/envs/server_env_local_test.py index b725729d0..52ed515c1 100644 --- a/tst/integration/env/envs/server_env_local_test.py +++ b/tst/integration/env/envs/server_env_local_test.py @@ -35,7 +35,8 @@ from agent_env.store.object_store.local.grant_server import grant_server from agent_env.task import EpisodeSpec, Task from agent_env.task_step.context import TaskStepContext -from agent_env.task_step.task_steps.teardown_sandboxes import TORN_DOWN_KEY, TeardownSandboxesTaskStep +from agent_env.task.teardown import TORN_DOWN_KEY +from agent_env.task_step.task_steps.teardown_sandboxes import TeardownSandboxesTaskStep from tst.util.a2a_test_agent import put_test_agent pytestmark = [pytest.mark.integration, pytest.mark.int_test_slow] diff --git a/tst/integration/providers/sandbox_providers/vm_sandbox_build_test.py b/tst/integration/providers/sandbox_providers/vm_sandbox_build_test.py index de0fc1870..f16532bb1 100644 --- a/tst/integration/providers/sandbox_providers/vm_sandbox_build_test.py +++ b/tst/integration/providers/sandbox_providers/vm_sandbox_build_test.py @@ -13,7 +13,7 @@ from agent_env.artifact.store import reset_artifact_store from agent_env.config import configure, reset_config from agent_env.providers.sandbox_providers.host_images import host_platform -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from tst.util.capabilities import missing_capability_reason pytestmark = [pytest.mark.integration, pytest.mark.int_test_slow] diff --git a/tst/integration/slow_durations.json b/tst/integration/slow_durations.json index 1dc850079..56ba067e8 100644 --- a/tst/integration/slow_durations.json +++ b/tst/integration/slow_durations.json @@ -35,9 +35,9 @@ "tst/integration/cli/run_bundle_skills_test.py": 23.2, "tst/integration/cli/run_role_models_test.py": 36.8, "tst/integration/computer/base_computer_test.py": 75.1, - "tst/integration/computer/computer_env_test.py": 90.0, + "tst/integration/computer/computer_env_test.py": 120.0, + "tst/integration/computer/desktop_example_test.py": 60.0, "tst/integration/computer/recording_chaos_test.py": 167.6, - "tst/integration/computer/scaffold_test.py": 109.6, "tst/integration/env/env_test.py": 11.3, "tst/integration/env/envs/build_context_local_test.py": 182.4, "tst/integration/env/envs/multi_env_local_test.py": 111.4, diff --git a/tst/integration/task_step/test_verify_sandbox_local.py b/tst/integration/task_step/test_verify_sandbox_local.py index b64d5d468..78ea37600 100644 --- a/tst/integration/task_step/test_verify_sandbox_local.py +++ b/tst/integration/task_step/test_verify_sandbox_local.py @@ -21,7 +21,7 @@ from agent_env.a2a_agent.store import get_a2a_agent_instance_store from agent_env.artifact.store import reset_artifact_store from agent_env.config import configure, get_config, reset_config -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.task import Task from agent_env.task_step.context import DeployedAgent, DeployedSandbox, TaskStepContext from agent_env.task_step.task_steps.collect_artifacts import CollectArtifactsTaskStep diff --git a/tst/unit/a2a_agent/context_only_deploy_test.py b/tst/unit/a2a_agent/context_only_deploy_test.py index 53490a27f..c52e557b1 100644 --- a/tst/unit/a2a_agent/context_only_deploy_test.py +++ b/tst/unit/a2a_agent/context_only_deploy_test.py @@ -7,9 +7,10 @@ from agent_env.a2a_agent.a2a_agent import A2AAgent from agent_env.artifact.artifacts.docker_image import DockerImageArtifact -from agent_env.providers.sandbox_providers import local_sandbox, sandbox_provider +from agent_env.providers.sandbox_providers import sandbox_provider +from agent_env.providers.sandbox_providers.local_docker import provider as local_sandbox from agent_env.providers.sandbox_providers.chained_sandbox_provider import ChainedSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.sandbox_provider import Accepts, SandboxProvider diff --git a/tst/unit/artifact/test_docker_image_github_push.py b/tst/unit/artifact/test_docker_image_github_push.py index d76b8079e..fc4d170a3 100644 --- a/tst/unit/artifact/test_docker_image_github_push.py +++ b/tst/unit/artifact/test_docker_image_github_push.py @@ -14,7 +14,7 @@ import agent_env.providers as providers from agent_env.artifact import DockerImageArtifact from agent_env.config import get_config -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.store import ImageStore, RegistryAuth, set_object_store from agent_env.store.object_store.local import LocalFilesystemObjectStore diff --git a/tst/unit/bundle/computer_driver_env_test.py b/tst/unit/bundle/computer_driver_env_test.py index ef200e90a..cb49ac76c 100644 --- a/tst/unit/bundle/computer_driver_env_test.py +++ b/tst/unit/bundle/computer_driver_env_test.py @@ -178,11 +178,11 @@ def test_a_driver_whose_class_takes_too_long_to_read_is_refused(bundle_dir, monk @pytest.mark.asyncio -async def test_a_driver_env_doesnt_deploy_until_there_is_a_computer_to_deploy_it_in(bundle_dir): +async def test_a_driver_env_deploys_only_inside_a_computer_env(bundle_dir): _bundle(bundle_dir) _run(bundle_dir) - with pytest.raises(NotImplementedError, match="deploys inside a computer, and agent-env can't create a computer"): + with pytest.raises(NotImplementedError, match="deploys inside a computer, never on its own"): await Env.get(f"{ROOT}/desk").deploy() diff --git a/tst/unit/bundle/installed_test.py b/tst/unit/bundle/installed_test.py index ae34b07d9..be77271c8 100644 --- a/tst/unit/bundle/installed_test.py +++ b/tst/unit/bundle/installed_test.py @@ -127,7 +127,8 @@ def test_the_plugin_report_counts_the_bundles_agent_env_ships(monkeypatch, quiet assert result.exit_code == 0, result.output (core,) = json.loads(result.stdout)["plugins"] assert [(c["group"], c["name"], c["status"]) for c in core["contributions"]] == [ - (BUNDLES, "agents", "active"), (BUNDLES, "computers", "active"), (BUNDLES, "drivers", "active"), + (BUNDLES, "agents", "active"), (BUNDLES, "computers", "active"), (BUNDLES, "desktop", "active"), + (BUNDLES, "drivers", "active"), (BUNDLES, "echo-mcp", "active"), (BUNDLES, "hello", "active"), (BUNDLES, "mail-mcp", "active"), (BUNDLES, "next-env-event", "active")] assert "\n agent-env itself\n" in shown and "requires:" not in shown diff --git a/tst/unit/bundle/preflight_test.py b/tst/unit/bundle/preflight_test.py index 274446b91..394c2ab75 100644 --- a/tst/unit/bundle/preflight_test.py +++ b/tst/unit/bundle/preflight_test.py @@ -30,7 +30,7 @@ from agent_env.providers.env_providers.env_gateway_provider import EnvironmentGatewayProvider from agent_env.providers.env_state.env_state_provider import EnvStateInstance from agent_env.providers.env_state.store import register_env_state_instance -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider from agent_env.store.routing import namespace_routing from agent_env.task_step.context import PromptResponse, TaskStepContext diff --git a/tst/unit/bundle/shipped_bundles_test.py b/tst/unit/bundle/shipped_bundles_test.py index c71c7d701..8e1d687af 100644 --- a/tst/unit/bundle/shipped_bundles_test.py +++ b/tst/unit/bundle/shipped_bundles_test.py @@ -26,7 +26,7 @@ def _shipped() -> dict: def test_agentenv_framework_ships_its_bundles(): - assert SHIPPED == ["agents", "computers", "drivers", "echo-mcp", "hello", "mail-mcp", "next-env-event"] + assert SHIPPED == ["agents", "computers", "desktop", "drivers", "echo-mcp", "hello", "mail-mcp", "next-env-event"] @pytest.mark.parametrize("name", SHIPPED) diff --git a/tst/unit/env/envs/test_computer_env.py b/tst/unit/env/envs/test_computer_env.py index f3716c3e9..8e54d46de 100644 --- a/tst/unit/env/envs/test_computer_env.py +++ b/tst/unit/env/envs/test_computer_env.py @@ -3,14 +3,19 @@ whichever env built it, records each result as it finishes, and writes the arches built as the env's computer. Docker and the registry are faked.""" +import asyncio import io import json import logging import subprocess import tarfile from types import SimpleNamespace +from unittest.mock import AsyncMock import pytest +from agentenv_protocol.computer import COMPUTER_DISPLAY, COMPUTER_RECORDING +from agentenv_protocol.computer.conformance import CheckResult, ProbeReport +from agentenv_protocol.tool_access import DISABLE_TOOL_EXTENSION_URI from click.testing import CliRunner from agent_env.artifact.artifacts import docker_image @@ -22,12 +27,20 @@ from agent_env.cli import cli from agent_env.env import Env from agent_env.env import env as env_module -from agent_env.env.envs.computer import composer +from agent_env.env.envs.computer import boot, composer, gate +from agent_env.env.envs.computer.composer import _probe from agent_env.env.envs.computer.env import ComputerEnv from agent_env.env.envs.computer.install import Install, Service from agent_env.env.envs.computer_driver import ComputerDriverEnv from agent_env.plugins._bundles import ALIASES_FILE, InstalledBundle, installed_aliases +from agent_env.providers.env_providers import env_computer_provider +from agent_env.providers.env_providers.env_provider import DeployRefused +from agent_env.task_step.task_steps.deploy_env import DeployEnvTaskStep +from agent_env.task_step.task_steps.reset_env import ResetEnvTaskStep +from agent_env.providers.sandbox_providers.local_docker import provider as local_sandbox from agent_env.store.base import NotFoundError +from agent_env.task import teardown +from agent_env.task_step.context import TaskStepContext from agent_env.store.image_store.image_store import OciRegistryImageStore from agent_env.store.image_store.local_registry_image_store import LocalRegistryImageStore from agent_env.store.image_store.oci_registry_credentials import RegistryAuth @@ -38,15 +51,28 @@ CONFIG = {"resolution": {"default": "1280x800", "phase": "deploy"}, "timezone": {"default": "UTC", "phase": "deploy"}, "theme": {"default": "light", "phase": "build"}} SERVE = "venv/bin/python -m agentenv_protocol.computer.serve driver.py:Desk" +CARD_EXTENSION = "urn:agentenv:computer/v1" +CARD = {"name": "desktop", "capabilities": {"extensions": [{"uri": CARD_EXTENSION}]}} TOOLS = ["computer_screenshot", "computer_click", "computer_type", "computer_pointer_list"] class FakeDocker: """This machine's Docker for the composer, on arm64 emulating amd64: each build recorded with its Dockerfile, its - context's files and its programs, each push landing in the fake registry as an image of its arch.""" + context's files and its programs, each push landing in the fake registry as an image of its arch, and each probe + recorded by the arch and image it booted.""" def __init__(self, registry: Registry): self.registry, self.builds, self.fail, self.emulates, self.labels = registry, [], {}, {"linux/amd64"}, {} + self.probed, self.probe_fails = [], {} + + async def probe(self, links, env, driver, image): + arch = self._arch(self.registry.manifests[image.rpartition("@")[2]]["config"]["digest"]) + self.probed.append((arch, image)) + if arch in self.probe_fails: + checks = [CheckResult(check="P4", name="clicks show their effect", status="failed", detail=self.probe_fails[arch])] + else: + checks = [CheckResult(check="P0", name="the card", status="passed", extensions=[CARD_EXTENSION])] + return ProbeReport.of("full", "desk", checks, [CARD_EXTENSION]), CARD, "LocalSandboxProvider" def build_image(self, dockerfile, context, tag, *, platform, labels=None, build_args=None): if platform in self.fail: @@ -84,6 +110,8 @@ def docker(local_stores, cli_routing, monkeypatch): monkeypatch.setattr(composer, "pin_digest", registry.pin_digest) monkeypatch.setattr(docker_image, "pin_digest", registry.pin_digest) monkeypatch.setattr(composer, "host_platform", lambda: "linux/arm64") + monkeypatch.setattr(local_sandbox, "host_platform", lambda: "linux/arm64") + monkeypatch.setattr(composer, "_probe", fake.probe) monkeypatch.setattr(composer, "_emulated_platforms", lambda: {"linux/arm64", *fake.emulates}) monkeypatch.setattr(composer, "_installed", lambda local, layer: {"mss": "10.1.0"}) monkeypatch.setattr(composer, "_protocol_source", lambda: None) @@ -144,7 +172,13 @@ def test_a_computer_env_is_composed_for_each_arch_and_written_with_the_computer_ stored = Env.get(env.id) composed = stored.metadata["composed"]["container"] assert {arch: (r["status"], r["reused"], r["emulated"]) for arch, r in composed.items()} == { - "amd64": ("unverified", False, True), "arm64": ("unverified", False, False)} + "amd64": ("unverified", False, True), "arm64": ("ready", False, False)} + assert composed["amd64"]["reason"] == "no configured sandbox provider runs linux/amd64 natively to probe it" + assert docker.probed == [("arm64", composed["arm64"]["image"])] and composed["arm64"]["card"] == CARD + probe = composed["arm64"]["probe"] + assert (probe["level"], probe["passed"], probe["provider"], probe["extensions"]) == ( + "full", True, "LocalSandboxProvider", {CARD_EXTENSION: "verified"}) + assert b'"passed": true' in FileArtifact.get(probe["report"]["id"], probe["report"]["version"]).load() assert composed["arm64"]["installed"] == {"mss": "10.1.0"} and composed["arm64"]["key"] != composed["amd64"]["key"] assert composed["arm64"]["image"].startswith("localhost:5000/agentenv-computers:") computer = ComputerArtifact.get(stored.metadata["computer"]["id"], stored.metadata["computer"]["version"]) @@ -152,6 +186,7 @@ def test_a_computer_env_is_composed_for_each_arch_and_written_with_the_computer_ assert computer.image().image_name.startswith("localhost:5000/agentenv-computers:index-") assert {arch: r["status"] for arch, r in computer.metadata["validated_images"]["container"].items()} == { "amd64": "passed", "arm64": "passed"} + assert computer.metadata["validated_boot"] == {"container": {"arm64": probe}} assert (computer.config["resolution"].default, computer.config["timezone"].default) == ("1600x900", "Europe/Paris") assert stored.computer_settings == {"timezone": "Europe/Paris", "theme": "dark"} arm64 = next(build for build in docker.builds if build["platform"] == "linux/arm64") @@ -224,7 +259,7 @@ def test_a_pair_that_fails_is_recorded_with_its_log_and_the_rest_are_built_and_w assert amd64["reason"] == 'the build failed: #7 ERROR: process "/bin/sh -c pip install" did not complete successfully' log = FileArtifact.get(amd64["log"]["id"], amd64["log"]["version"]) assert b"did not complete successfully" in log.load() - assert env.metadata["composed"]["container"]["arm64"]["status"] == "unverified" + assert env.metadata["composed"]["container"]["arm64"]["status"] == "ready" assert ComputerArtifact.get(env.metadata["computer"]["id"]).arch == ["arm64"] @@ -248,7 +283,7 @@ def test_a_machine_that_cant_emulate_the_other_arch_fails_that_arch_alone(base, amd64 = env.metadata["composed"]["container"]["amd64"] assert amd64["reason"] == "this machine can't build linux/amd64: its Docker has no emulation for it" - assert env.metadata["composed"]["container"]["arm64"]["status"] == "unverified" + assert env.metadata["composed"]["container"]["arm64"]["status"] == "ready" assert [build["platform"] for build in docker.builds] == ["linux/arm64"] @@ -277,7 +312,8 @@ def test_a_base_built_from_its_context_is_built_here_for_each_arch_and_keyed_by_ assert {r["reused"] for r in again.metadata["composed"]["container"].values()} == {True} assert {build["dockerfile"].splitlines()[0] for build in docker.builds} == { "FROM local/base-1234:abcd-amd64", "FROM local/base-1234:abcd-arm64"} - assert {r["status"] for r in env.metadata["composed"]["container"].values()} == {"unverified"} + assert {arch: r["status"] for arch, r in env.metadata["composed"]["container"].items()} == { + "amd64": "unverified", "arm64": "ready"} def test_a_checkout_composes_its_own_protocol_into_the_drivers_venv(base, driver, docker, tmp_path, monkeypatch): @@ -295,21 +331,87 @@ def test_a_checkout_composes_its_own_protocol_into_the_drivers_venv(base, driver assert "protocol/pyproject.toml" in docker.builds[0]["files"] -def test_the_pairs_left_to_build_are_those_failed_or_interrupted(base, driver, docker): +@pytest.mark.parametrize("left", [{"status": "building"}, {"status": "unverified", "image": "built", "key": "k"}], + ids=["interrupted", "unprobed"]) +def test_the_pairs_left_to_compose_are_those_failed_interrupted_or_unprobed_on_a_native_arch(base, driver, docker, + left): docker.fail["linux/amd64"] = "boom" env = Env.get(_put(base, driver).id) composed = env.metadata["composed"] - composed["container"]["arm64"] = {"status": "building"} + composed["container"]["arm64"] = left env.metadata = {**env.metadata, "composed": composed} assert composer.pending(env) == [("container", "amd64"), ("container", "arm64")] docker.fail.clear() composer.compose(env, composer.pending(env)) - assert {r["status"] for r in Env.get(env.id).metadata["composed"]["container"].values()} == {"unverified"} + assert {arch: r["status"] for arch, r in Env.get(env.id).metadata["composed"]["container"].items()} == { + "amd64": "unverified", "arm64": "ready"} assert composer.pending(Env.get(env.id)) == [] +def test_the_probe_boots_a_build_at_the_envs_settings_as_a_deploy_does(base, driver, docker, monkeypatch): + env = _put(base, driver, compose=False, computer_settings={"timezone": "Europe/Paris"}, + display={"width": 1600, "height": 900}) + booted = [] + + async def fake_boot(link, **kwargs): + booted.append(kwargs["environment"]) + return boot.Booted(sandbox=None, url="http://computer", card=CARD) + + monkeypatch.setattr(boot, "boot", fake_boot) + monkeypatch.setattr(boot, "discard", AsyncMock()) + monkeypatch.setattr(composer, "run_probe", AsyncMock(return_value="report")) + + assert asyncio.run(_probe([SimpleNamespace()], env, driver, "img")) == ( + "report", CARD, "SimpleNamespace") + assert {k: v for k, v in booted[0].items() if k.startswith("AGENTENV_SETTING_")} == { + "AGENTENV_SETTING_RESOLUTION": "1600x900", "AGENTENV_SETTING_TIMEZONE": "Europe/Paris"} + + +def test_a_build_whose_probe_fails_or_that_never_gets_ready_fails_at_the_probe_and_isnt_deployed(base, driver, docker, + monkeypatch): + docker.probe_fails["arm64"] = "the click never landed" + failed = Env.get(_put(base, driver).id).metadata["composed"]["container"]["arm64"] + + async def never_ready(*args): + raise RuntimeError("LocalSandboxProvider: the computer at http://127.0.0.1:1 wasn't ready within 180 s " + "(not ready: display: no X)") + + async def overran(*args): + raise TimeoutError + + monkeypatch.setattr(composer, "_probe", never_ready) + unbooted = Env.get(_put(base, driver, id="@local/~/kit/other", environment_name="other").id).metadata[ + "composed"]["container"]["arm64"] + monkeypatch.setattr(composer, "_probe", overran) + stuck = Env.get(_put(base, driver, id="@local/~/kit/stuck", environment_name="stuck").id).metadata[ + "composed"]["container"]["arm64"] + + assert (failed["status"], failed["stage"], failed["reason"]) == ( + "failed", "probe", "the probe failed: P4 clicks show their effect: the click never landed") + assert not failed["probe"]["passed"] and failed["image"] + assert (unbooted["status"], unbooted["stage"]) == ("failed", "probe") + assert unbooted["reason"].startswith("it didn't boot: LocalSandboxProvider: the computer at http://127.0.0.1:1 " + "wasn't ready within 180 s") + assert (stuck["status"], stuck["stage"], stuck["reason"]) == ( + "failed", "probe", f"the probe didn't finish within {composer._PROBE_TIMEOUT_S} s") + + +@pytest.mark.parametrize("result, stored, replaces", [ + ({"status": "unverified"}, {"status": "ready"}, False), + ({"status": "failed", "stage": "build"}, {"status": "unverified"}, False), + ({"status": "building"}, {"status": "unverified"}, False), + ({"status": "failed", "stage": "probe"}, {"status": "unverified"}, True), + ({"status": "failed", "stage": "probe"}, {"status": "ready"}, False), + ({"status": "ready"}, {"status": "failed", "stage": "probe"}, True), + ({"status": "building"}, {"status": "failed", "stage": "build"}, True), +], ids=["unprobed-over-ready", "build-failure-over-built", "building-over-built", "probe-failure-over-unprobed", + "probe-failure-over-ready", "ready-over-probe-failure", "building-over-build-failure"]) +def test_a_result_takes_a_pairs_place_only_when_it_got_at_least_as_far(result, stored, replaces): + assert (composer._rank(result) >= composer._rank(stored)) is replaces + + def test_env_build_builds_what_a_put_left_and_prints_each_pair(base, driver, docker, caplog): caplog.set_level(logging.ERROR, logger=composer.__name__) # live logging would put pytest's stdout back mid-command docker.fail["linux/amd64"] = "boom" @@ -322,8 +424,9 @@ def test_env_build_builds_what_a_put_left_and_prints_each_pair(base, driver, doc again = runner.invoke(cli, ["env", "build", "--id", env.id]) assert failed.exit_code == 1 and "container/amd64: failed (the build failed: boom)" in failed.output - assert built.exit_code == 0 and "container/amd64: unverified (localhost:5000/agentenv-computers:" in built.output - assert "container/arm64: unverified (localhost:5000" in built.output and len(docker.builds) == 2 + assert built.exit_code == 0 and ("container/amd64: unverified (no configured sandbox provider runs linux/amd64 " + "natively to probe it)") in built.output + assert "container/arm64: ready (localhost:5000" in built.output and len(docker.builds) == 2 assert again.exit_code == 0 and len(docker.builds) == 2 assert ComputerArtifact.get(Env.get(env.id).metadata["computer"]["id"]).arch == ["amd64", "arm64"] @@ -335,11 +438,12 @@ def test_env_build_refuses_an_env_that_isnt_a_computer_env(base, driver, docker) @pytest.mark.asyncio -async def test_a_computer_env_doesnt_deploy_yet_and_says_how_far_it_got(base, driver, docker): +async def test_a_computer_env_with_no_ready_build_is_refused_at_deploy_naming_its_builds(base, driver, docker): + docker.probe_fails["arm64"] = "the click never landed" env = _put(base, driver) - with pytest.raises(NotImplementedError, match=r"agent-env can't boot yet; its builds are \{'container/amd64': " - r"'unverified', 'container/arm64': 'unverified'\}"): + with pytest.raises(DeployRefused, match=r"has no ready build to deploy; its builds are \{'container/amd64': " + r"'unverified', 'container/arm64': 'failed'\}"): await Env.get(env.id).deploy() @@ -521,5 +625,221 @@ def test_builds_of_one_env_at_once_keep_each_others_pairs_and_a_failure_doesnt_r stored = Env.get(env.id) composed = stored.metadata["composed"]["container"] - assert {arch: r["status"] for arch, r in composed.items()} == {"amd64": "unverified", "arm64": "unverified"} + assert {arch: r["status"] for arch, r in composed.items()} == {"amd64": "unverified", "arm64": "ready"} assert stored.metadata["computer"]["images"] == {arch: r["image"] for arch, r in composed.items()} + + +DRIVER_CARD = {"name": "desktop", "capabilities": {"extensions": [ + {"uri": COMPUTER_DISPLAY, "params": {"endpoint": "/computer/display", "methods": {"get": {"method": "GET"}}}}, + {"uri": DISABLE_TOOL_EXTENSION_URI, "params": {"endpoint": "/tools/disable", "methods": {"disable": {}}}}, + {"uri": COMPUTER_RECORDING, "params": {"endpoint": "/computer/recording", "methods": {"end_all": {}}}}]}} + + +@pytest.fixture +def boots(monkeypatch): + """The boots a deploy asks for, each a sandbox serving the desktop driver's card at 127.0.0.1:20001; the driver + shows a 1600x900 screen, and each extension call it gets is recorded.""" + made = SimpleNamespace(boots=[], calls=[], sandboxes=[], display=(1600, 900)) + + async def fake_boot(sandbox_provider, **kwargs): + made.boots.append({**kwargs, "provider": type(sandbox_provider).__name__}) + sandbox = SimpleNamespace(sandbox_id=f"local-{len(made.boots)}", type="local", terminated=False, + ON_THIS_MACHINE=True) + + async def terminate(): + sandbox.terminated = True + + sandbox.terminate = terminate + made.sandboxes.append(sandbox) + return boot.Booted(sandbox, "http://127.0.0.1:20001", DRIVER_CARD) + + async def invoke(base_url, card, uri, params=None, timeout=30, verify=True, *, method=None): + made.calls.append((uri, method, params)) + width, height = made.display + return {"width": width, "height": height, "scale": 1, "modes": []} if uri == COMPUTER_DISPLAY else {} + + monkeypatch.setattr(env_computer_provider, "boot", fake_boot) + monkeypatch.setattr(env_module.protocol_v1, "invoke_extension", invoke) + return made + + +@pytest.mark.asyncio +async def test_a_computer_env_deploys_its_first_native_ready_build_with_its_settings_and_display( + base, driver, docker, boots): + env = _put(base, driver, display={"width": 1600, "height": 900}) + + deployed = await Env.get(env.id).deploy(attribution={"run": "r1"}, env_params={ + "computer_settings": {"timezone": "Asia/Tokyo"}, "driver_settings": {"default_space": "norm"}}) + + (booted,) = boots.boots + composed = Env.get(env.id).metadata["composed"]["container"] + assert (booted["image"], booted["port"], booted["provider"]) == (composed["arm64"]["image"], 18765, + "LocalSandboxProvider") + environment = booted["environment"] + assert (environment["AGENTENV_SETTING_RESOLUTION"], environment["AGENTENV_SETTING_TIMEZONE"]) == ("1600x900", "Asia/Tokyo") + assert json.loads(environment["AGENTENV_COMPUTER_SETTINGS"]) == {"default_space": "norm"} + assert environment["ENVIRONMENT_NAME"] == "desktop" and "AGENTENV_SETTING_THEME" not in environment + assert boots.calls == [(COMPUTER_DISPLAY, "get", None)] + assert (deployed.env_provider_type, deployed.sandbox_ids) == ("computer", {"computer": {"desktop": "local-1"}}) + assert deployed.environment_card == DRIVER_CARD and deployed.mcp_url == "http://127.0.0.1:20001/mcp" + record = deployed.metadata["computer"] + assert {key: record[key] for key in ("form", "arch", "provider", "display", "computer_settings")} == { + "form": "container", "arch": "arm64", "provider": "LocalSandboxProvider", "display": "1600x900", + "computer_settings": {"resolution": "1600x900", "timezone": "Asia/Tokyo"}} + assert record["computer"]["id"] == f"{env.id}__computer" and record["env_params"] == { + "computer_settings": {"timezone": "Asia/Tokyo"}, "driver_settings": {"default_space": "norm"}} + + +@pytest.mark.asyncio +async def test_a_reset_ends_the_envs_recordings_then_boots_a_fresh_container_where_the_last_one_ran( + base, driver, docker, boots, monkeypatch): + env = _put(base, driver, display={"width": 1600, "height": 900}) + deployed = await Env.get(env.id).deploy(env_params={"computer_settings": {"timezone": "Asia/Tokyo"}}) + (sandbox,) = boots.sandboxes + resets = [] + + async def reset_container(self, sandbox, **kwargs): + resets.append((sandbox, kwargs)) + boots.calls.append("reset_container") + + monkeypatch.setattr(local_sandbox.LocalSandboxProvider, "get_sandbox", AsyncMock(return_value=sandbox)) + monkeypatch.setattr(local_sandbox.LocalSandboxProvider, "reset_container", reset_container) + monkeypatch.setattr(env_computer_provider, "ready", AsyncMock(return_value=DRIVER_CARD)) + boots.calls.clear() + + await ResetEnvTaskStep(id="reset", version=None, env_id=env.id, fail_task_on_error=True).execute( + TaskStepContext(deployed_envs=[deployed])) + + ((replaced, kwargs),) = resets + assert replaced is sandbox and not sandbox.terminated + assert (kwargs["image_name"], kwargs["port"]) == (deployed.metadata["computer"]["image"], 18765) + assert (kwargs["env"]["AGENTENV_SETTING_TIMEZONE"], kwargs["env"]["ENVIRONMENT_NAME"]) == ("Asia/Tokyo", "desktop") + env_computer_provider.ready.assert_awaited_once_with(deployed.environment_url, boot.READY_TIMEOUT_S) + assert boots.calls == [(COMPUTER_RECORDING, "end_all", {"reason": "reset"}), "reset_container", (COMPUTER_DISPLAY, "get", None)] + + +@pytest.mark.asyncio +async def test_a_reset_a_provider_cant_carry_out_fails_the_step_rather_than_skipping_it(base, driver, docker, boots, + monkeypatch): + env = _put(base, driver, display={"width": 1600, "height": 900}) + deployed = await Env.get(env.id).deploy() + monkeypatch.setattr(local_sandbox.LocalSandboxProvider, "get_sandbox", AsyncMock(return_value=boots.sandboxes[0])) + monkeypatch.setattr(local_sandbox.LocalSandboxProvider, "reset_container", + AsyncMock(side_effect=NotImplementedError("its sandbox is the container itself"))) + + with pytest.raises(RuntimeError, match="can't be reset on LocalSandboxProvider: its sandbox is the container"): + await ResetEnvTaskStep(id="reset", version=None, env_id=env.id, fail_task_on_error=True).execute( + TaskStepContext(deployed_envs=[deployed])) + + +@pytest.mark.asyncio +async def test_teardown_closes_a_computer_env_ending_its_recordings_before_its_computer_goes( + base, driver, docker, boots, monkeypatch): + env = _put(base, driver, display={"width": 1600, "height": 900}) + deployed = await Env.get(env.id).deploy() + (sandbox,) = boots.sandboxes + terminate = sandbox.terminate + + async def terminating(): + boots.calls.append("terminate") + await terminate() + + sandbox.terminate = terminating + monkeypatch.setattr(local_sandbox.LocalSandboxProvider, "get_sandbox", AsyncMock(return_value=sandbox)) + monkeypatch.setattr(teardown, "terminate", AsyncMock(return_value=False)) + boots.calls.clear() + + report = await teardown.teardown_run(TaskStepContext(deployed_envs=[deployed])) + + assert boots.calls == [(COMPUTER_RECORDING, "end_all", {"reason": "teardown"}), "terminate"] + assert [s.sandbox_id for s in report.terminated] == ["local-1"] and not report.failed + + +@pytest.mark.asyncio +@pytest.mark.parametrize("env_params, problem", [ + ({"form": "disk"}, "env_params.form: 'disk' isn't one of"), + ({"computer_settings": {"theme": "dark"}}, "env_params.computer_settings: theme is a build-phase setting"), + ({"computer_settings": {"resolution": "800x600"}}, "the env's display sets the screen's size"), + ({"computer_settings": {"timezone": 3}}, "env_params.computer_settings.timezone must be a str"), + ({"driver_settings": {"default_space": "polar"}}, "env_params.driver_settings.default_space"), + ({"driver_settings": {"colour": "blue"}}, "env_params.driver_settings.colour: Extra inputs are not permitted"), + ({"colour": "blue"}, "env_params.colour"), +], ids=["form", "build-setting", "resolution-over-display", "setting-type", "driver-setting", "driver-unknown", + "unknown"]) +async def test_env_params_a_computer_env_cant_take_are_refused_at_save_and_at_deploy(base, driver, docker, boots, + env_params, problem): + env = _put(base, driver, display={"width": 1600, "height": 900}) + + saved = DeployEnvTaskStep(id="d", version=None, env_id=env.id, env_params=env_params).preflight() + with pytest.raises(DeployRefused, match=problem.replace(".", "[.]")): + await Env.get(env.id).deploy(env_params=env_params) + + assert len(saved) == 1 and problem in saved[0] and boots.boots == [] + + +@pytest.mark.asyncio +async def test_a_computer_that_boots_at_another_size_than_its_display_fails_the_deploy_and_goes_with_its_folder( + base, driver, docker, boots, monkeypatch): + boots.display = (1280, 800) + removed = [] + monkeypatch.setattr(boot, "remove_local_work_dir", removed.append) + env = _put(base, driver, display={"width": 1600, "height": 900}) + + with pytest.raises(RuntimeError, match="booted at 1280x800, not the 1600x900 its display sets"): + await Env.get(env.id).deploy() + + assert [sandbox.terminated for sandbox in boots.sandboxes] == [True] and removed == ["local-1"] + + +@pytest.mark.asyncio +async def test_a_boot_that_never_gets_ready_takes_its_sandbox_and_work_folder_down(monkeypatch, tmp_path): + monkeypatch.setenv("AGENT_ENV_LOCAL_SANDBOX_DIR", str(tmp_path)) + sandbox = local_sandbox.LocalSandbox(exposed_ports=[18765], sandbox_id="local-unready") + sandbox.terminate = AsyncMock() + monkeypatch.setattr(boot, "ready", AsyncMock(side_effect=TimeoutError("not ready"))) + + with pytest.raises(TimeoutError): + await boot.boot(SimpleNamespace(create_container=AsyncMock(return_value=sandbox)), image="img", port=18765, + environment={}, network_policy=None, timeout=60, attribution=None) + + sandbox.terminate.assert_awaited_once() + assert local_sandbox.LocalSandbox.find_work_dir("local-unready") is None + + +def test_a_boot_mints_a_generated_setting_resolves_a_secret_its_record_withholds_both_and_a_secret_is_a_reference( + base, monkeypatch): + monkeypatch.setenv("DESK_KEY", "s3cret") + computer = base.model_copy(update={"config": { + **base.config, "token": base.config["timezone"].model_copy(update={"default": "", "generate": True}), + "api_key": base.config["timezone"].model_copy(update={"default": "env:DESK_KEY", "secret": True})}}) + + first, second = (boot.settings_environment(computer, {}, environment_name="desktop") for _ in range(2)) + + assert first["AGENTENV_SETTING_API_KEY"] == "s3cret" and first["AGENTENV_SETTING_TOKEN"] + assert first["AGENTENV_SETTING_TOKEN"] != second["AGENTENV_SETTING_TOKEN"] + assert boot.recorded_settings(computer, {}) == {"resolution": "1280x800", "timezone": "UTC", "token": boot.WITHHELD, + "api_key": boot.WITHHELD} + pin = computer.model_copy(update={"config": {"pin": base.config["timezone"].model_copy( + update={"default": 1234, "secret": True})}}) + for reference in ("env:DESK_KEY", "secret:desk/key"): + assert gate.check_computer_settings(computer, {"api_key": reference}) == ({"api_key": reference}, []) + assert gate.check_computer_settings(pin, {"pin": reference}) == ({"pin": reference}, []) + for value in ("s3cret", "env:DESK_KEY?s3cret", "env:", 3): + assert gate.check_computer_settings(computer, {"api_key": value}) == ({}, [ + "computer_settings.api_key is a secret, so it takes a secret:KEY or env:NAME reference with no ?default, " + "not its value"]) + + +@pytest.mark.parametrize("change, problem", [ + ("riscv64", "LocalSandboxProvider runs none of its arches (amd64, arm64) natively"), + ("network", "LocalSandboxProvider can't enforce its network"), +]) +def test_a_deploy_its_sandbox_provider_cant_carry_out_is_refused_at_save(base, driver, docker, monkeypatch, change, + problem): + env = _put(base, driver, network={"mode": "allowlist", "allow_hosts": ["example.com"]} if change == "network" else None) + if change == "riscv64": + monkeypatch.setattr(local_sandbox, "host_platform", lambda: "linux/riscv64") + + (refusal,) = DeployEnvTaskStep(id="d", version=None, env_id=env.id).preflight() + + assert problem in refusal diff --git a/tst/unit/env/envs/test_mcp_server_container_mode.py b/tst/unit/env/envs/test_mcp_server_container_mode.py index 4adc4d96e..1a9e7d178 100644 --- a/tst/unit/env/envs/test_mcp_server_container_mode.py +++ b/tst/unit/env/envs/test_mcp_server_container_mode.py @@ -28,7 +28,7 @@ from agent_env.providers.env_providers.env_provider import _builtin_env_providers, build_env_provider from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_CONTAINER, SANDBOX_MODE_VM from agent_env.task_step.context import TaskStepContext -from agent_env.task_step.task_steps.teardown_sandboxes import _env_sandbox_ids +from agent_env.task.teardown import env_sandbox_ids from tst.util.sandbox_providers import mock_provider _GW = "https://gw.example" @@ -62,7 +62,7 @@ async def test_a_container_deploy_stages_into_the_server_and_records_every_conta assert env._sandbox is sandboxes["srv"] assert (deployed.sandbox_id, deployed.sandbox_type) == ("gw", "modal") assert deployed.sandbox_ids == _RECORDED - assert {i for i, _ in _env_sandbox_ids(deployed)} == {sb.sandbox_id for sb in env._env_provider._container_sandboxes} + assert {i for i, _ in env_sandbox_ids(deployed)} == {sb.sandbox_id for sb in env._env_provider._container_sandboxes} sandboxes["srv"].write_file_from_object.assert_awaited_once_with("s3://bucket/email.json", "/data/email.json") sandboxes["gw"].write_file_from_object.assert_not_awaited() @@ -226,7 +226,7 @@ async def test_a_deploy_without_a_gateway_records_the_servers_own_card(): "server", "srv", "modal", {"mcp_server": {"email": "srv"}}) assert (deployed.environment_card_url, deployed.environment_card, deployed.mcp_url, deployed.mcp_server_name) == ( f"{_SRV}{WELL_KNOWN_PATH}", _LEAF_CARD, f"{_SRV}/mcp", "email") - assert {i for i, _ in _env_sandbox_ids(deployed)} == {"srv"} + assert {i for i, _ in env_sandbox_ids(deployed)} == {"srv"} @pytest.mark.asyncio diff --git a/tst/unit/env/envs/test_mcp_server_plugin_provider.py b/tst/unit/env/envs/test_mcp_server_plugin_provider.py index 4a1574605..02037b78c 100644 --- a/tst/unit/env/envs/test_mcp_server_plugin_provider.py +++ b/tst/unit/env/envs/test_mcp_server_plugin_provider.py @@ -25,6 +25,7 @@ from agent_env.providers.env_providers.env_gateway_provider import EnvironmentGatewayProvider from agent_env.providers.env_providers.env_provider import EnvironmentProvider from agent_env.providers.env_providers.env_server_provider import EnvironmentServerProvider +from agent_env.task.teardown import RecordedSandbox _URL = "https://plug.example" _CARD = {"name": "email", "url": "/agentenv"} @@ -469,9 +470,9 @@ async def test_validate_cleanup_terminates_the_sandboxes_a_plugins_record_names( context = SimpleNamespace(deployed_envs=[record], deployed_agents=[], instance_id="task-inst") with patch("agent_env.task.Task.put", return_value=MagicMock(id="t", version=1, run=AsyncMock(return_value=context))), \ patch("agent_env.env.env.Env.get", return_value=_env()), \ - patch("agent_env.task_step.task_steps.teardown_sandboxes._terminate", AsyncMock()) as terminate: + patch("agent_env.task.teardown.terminate", AsyncMock()) as terminate: assert await _env().validate() == "task-inst" - terminate.assert_awaited_once_with("sb-1", "local", "env") + terminate.assert_awaited_once_with(RecordedSandbox("sb-1", "local", "env")) @pytest.mark.asyncio @@ -517,10 +518,10 @@ async def test_validate_cleanup_terminates_every_sandbox_even_when_one_is_alread record = DeployedSandboxEnv(env_id="mcp-email", env_version=1, env_provider_type="plugin_kw", mcp_url=f"{_URL}/mcp", sandbox_id="sb-gone", sandbox_type="local", sandbox_ids={"mcp_server": {"email": "sb-2"}}) context = SimpleNamespace(deployed_envs=[record], deployed_agents=[], instance_id="task-inst") - terminate = AsyncMock(side_effect=lambda sandbox_id, *_: (_ for _ in ()).throw(RuntimeError("gone")) if sandbox_id == "sb-gone" else None) + terminate = AsyncMock(side_effect=lambda sandbox: (_ for _ in ()).throw(RuntimeError("gone")) if sandbox.sandbox_id == "sb-gone" else None) with patch("agent_env.task.Task.put", return_value=MagicMock(id="t", version=1, run=AsyncMock(return_value=context))), \ patch("agent_env.env.env.Env.get", return_value=_env()), \ - patch("agent_env.task_step.task_steps.teardown_sandboxes._terminate", terminate), caplog.at_level("WARNING"): + patch("agent_env.task.teardown.terminate", terminate), caplog.at_level("WARNING"): await _env().validate() - assert sorted(c.args[0] for c in terminate.await_args_list) == ["sb-2", "sb-gone"] + assert sorted(c.args[0].sandbox_id for c in terminate.await_args_list) == ["sb-2", "sb-gone"] assert "terminate sb-gone failed" in caplog.text diff --git a/tst/unit/env/envs/test_multi_env_plugin_provider.py b/tst/unit/env/envs/test_multi_env_plugin_provider.py index 60ae43408..5d4020da6 100644 --- a/tst/unit/env/envs/test_multi_env_plugin_provider.py +++ b/tst/unit/env/envs/test_multi_env_plugin_provider.py @@ -32,6 +32,7 @@ from agent_env.providers.env_providers.env_server_provider import EnvironmentServerProvider from agent_env.task_step.context import TaskStepContext from agent_env.task_step.task_steps.deploy_env import DeployEnvTaskStep +from agent_env.task.teardown import RecordedSandbox from tst.util.sandbox_providers import mock_provider _URL = "https://pods.example" @@ -291,9 +292,9 @@ async def test_validate_terminates_the_sandboxes_a_plugin_multi_env_record_names context = SimpleNamespace(deployed_envs=[record], deployed_agents=[], instance_id="task-inst") with patch("agent_env.task.Task.put", return_value=MagicMock(id="t", version=1, run=AsyncMock(return_value=context))), \ patch("agent_env.env.env.Env.get", return_value=_multi()), \ - patch("agent_env.task_step.task_steps.teardown_sandboxes._terminate", AsyncMock()) as terminate: + patch("agent_env.task.teardown.terminate", AsyncMock()) as terminate: assert await _multi().validate() == "task-inst" - terminate.assert_awaited_once_with("sb-1", "modal", "env") + terminate.assert_awaited_once_with(RecordedSandbox("sb-1", "modal", "env")) @pytest.mark.parametrize("env_class, make", [(MultiEnv, _multi), (WebsiteEnv, lambda t: _site("shop", t))], ids=["multi", "website"]) diff --git a/tst/unit/plugins_test.py b/tst/unit/plugins_test.py index e03049cf8..b6260a4cf 100644 --- a/tst/unit/plugins_test.py +++ b/tst/unit/plugins_test.py @@ -666,7 +666,7 @@ def test_a_plugin_cannot_replace_a_built_in_env_provider(monkeypatch): def test_building_a_provider_whose_plugin_was_skipped_says_why(monkeypatch): _install(monkeypatch, env_providers=[_EP("bare", "_BareEnvProvider")]) - with pytest.raises(ValueError, match=r"Unknown env_provider_type: 'bare' \(expected one of \['gateway', 'server'\]\) " + with pytest.raises(ValueError, match=r"Unknown env_provider_type: 'bare' \(expected one of \['computer', 'gateway', 'server'\]\) " r"\(.*_BareEnvProvider must implement close and deploy"): build_env_provider("bare") diff --git a/tst/unit/providers/env_providers/env_server_provider_test.py b/tst/unit/providers/env_providers/env_server_provider_test.py index d798d0e5b..30e26330b 100644 --- a/tst/unit/providers/env_providers/env_server_provider_test.py +++ b/tst/unit/providers/env_providers/env_server_provider_test.py @@ -182,7 +182,7 @@ def test_build_env_provider_builds_a_new_provider_of_the_declared_type(name, cls def test_build_env_provider_rejects_an_unknown_type(): - with pytest.raises(ValueError, match=r"Unknown env_provider_type: 'vm' \(expected one of \['gateway', 'server'\]\)"): + with pytest.raises(ValueError, match=r"Unknown env_provider_type: 'vm' \(expected one of \['computer', 'gateway', 'server'\]\)"): build_env_provider("vm") diff --git a/tst/unit/providers/sandbox_providers/attribution_wire_test.py b/tst/unit/providers/sandbox_providers/attribution_wire_test.py index 0af60203c..ad26aceb9 100644 --- a/tst/unit/providers/sandbox_providers/attribution_wire_test.py +++ b/tst/unit/providers/sandbox_providers/attribution_wire_test.py @@ -149,7 +149,7 @@ async def create_sandbox(self, *, attribution=None, **kwargs): @pytest.mark.asyncio async def test_the_quartet_is_no_longer_a_keyword_on_any_backend(): """A caller still on the pre-dict signature fails loud rather than billing unattributed.""" - from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider + from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider with pytest.raises(TypeError, match="team"): await LocalSandboxProvider().create_vm(team="t") diff --git a/tst/unit/providers/sandbox_providers/host_images_test.py b/tst/unit/providers/sandbox_providers/host_images_test.py index eb82caed3..f1b4cb356 100644 --- a/tst/unit/providers/sandbox_providers/host_images_test.py +++ b/tst/unit/providers/sandbox_providers/host_images_test.py @@ -12,8 +12,8 @@ from agent_env.artifact.artifacts.docker_image import DockerImageArtifact from agent_env.providers.sandbox_providers import host_images -from agent_env.providers.sandbox_providers import local_sandbox -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker import provider as local_sandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.sandbox import VmSandbox from agent_env.utils.docker.build import DockerBuildError diff --git a/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py b/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py index e19a1e92c..240aa2941 100644 --- a/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py +++ b/tst/unit/providers/sandbox_providers/local_sandbox_host_ports_test.py @@ -22,11 +22,11 @@ from agent_env.config.paths import state_root from agent_env.env.gateway import AGENT_ENV_GATEWAY_MCP_PORT -from agent_env.providers.sandbox_providers import local_sandbox as local_sandbox_module +from agent_env.providers.sandbox_providers.local_docker import provider as local_sandbox_module from agent_env.env.envs.service_db import DB_MCP_PORT, DB_WEB_PORT from agent_env.providers.sandbox_providers.boxd.sandbox import BoxdSandbox from agent_env.providers.sandbox_providers.daytona.sandbox import DaytonaSandbox -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.e2b import E2BSandbox from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox diff --git a/tst/unit/providers/sandbox_providers/local_sandbox_test.py b/tst/unit/providers/sandbox_providers/local_sandbox_test.py index 488cfb867..9d5819240 100644 --- a/tst/unit/providers/sandbox_providers/local_sandbox_test.py +++ b/tst/unit/providers/sandbox_providers/local_sandbox_test.py @@ -11,10 +11,11 @@ import pytest -import agent_env.providers.sandbox_providers.local_sandbox as ls +import agent_env.providers.sandbox_providers.local_docker.provider as ls from agent_env.a2a_agent.a2a_agent import A2AAgent from agent_env.config import reset_config, set_object_store -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_CONTAINER, SANDBOX_MODE_VM from agent_env.store import LocalFilesystemObjectStore from agent_env.store.object_store.local.tls import local_ca @@ -587,6 +588,129 @@ async def _rec(script, *, max_retries=0): assert "-p 127.0.0.1:41337:8000 -p 172.17.0.1:41337:8000 " in run_cmd +@pytest.mark.asyncio +async def test_a_container_with_no_network_runs_on_an_internal_network_that_its_relay_publishes_and_is_replaced_with_it( + monkeypatch, tmp_path): + import agent_env.config as cfg + + monkeypatch.setattr(cfg, "get_config", lambda: _StubConfig()) + monkeypatch.setattr(ls, "_free_host_port", lambda: 41337) + monkeypatch.setattr(ls, "relay_image", _relay_image) + made: list[LocalSandbox] = [] + + class _RecordingProvider(LocalSandboxProvider): + async def create_vm(self, **kwargs): + sandbox = await LocalSandboxProvider.create_vm(self, **kwargs) + sandbox._work_dir = tmp_path + sandbox.scripts = [] + + async def _record(script, *, max_retries=0): + sandbox.scripts.append(script) + return "172.30.0.2\n" if script.startswith("docker inspect") else "" + + sandbox.exec_script = _record + made.append(sandbox) + return sandbox + + sandbox = await _RecordingProvider().create_container(image_name="img:v1", port=8000, env={}, + network_policy=NetworkPolicy(mode="allowlist")) + + network, label = f"closed-{sandbox.sandbox_id}", f"agentenv.sandbox={sandbox.sandbox_id}" + steps = [s.split()[1] for s in sandbox.scripts] + create = next(s for s in sandbox.scripts if s.startswith("docker create")) + assert steps == ["pull", "network", "create", "start", "inspect", "run"] + assert sandbox.scripts[1] == f"docker network create --internal --label {label} {network} >/dev/null" + assert f"--network {network} " in create and "-p " not in create + assert sandbox.scripts[-1] == (f"docker run -d --name relay-{sandbox.sandbox_id} --label {label} " + "-p 127.0.0.1:41337:8000 agentenv-relay:test 8000=172.30.0.2:8000 >/dev/null && " + f"docker network connect {network} relay-{sandbox.sandbox_id}") + assert sandbox.mode == SANDBOX_MODE_CONTAINER and sandbox.closed_network == network + assert sandbox.network_policy.closed and sandbox.tunnel_urls[8000] == "http://127.0.0.1:41337" + + sandbox.scripts.clear() + await _RecordingProvider().reset_container(sandbox, image_name="img:v1", port=8000, env={"A": "1"}) + + assert [s.split()[1] for s in sandbox.scripts] == ["rm", "create", "start", "inspect", "run"] + assert sandbox.scripts[0] == f"docker rm -f agent-{sandbox.sandbox_id} relay-{sandbox.sandbox_id} > /dev/null" + assert f"--network {network} " in sandbox.scripts[1] and "-e A=1 " in sandbox.scripts[1] + assert "-p 127.0.0.1:41337:8000 " in sandbox.scripts[-1] + + +@pytest.mark.asyncio +async def test_a_container_with_no_network_whose_marker_cant_be_written_takes_its_relay_and_network_down( + tmp_path: Path, monkeypatch): + import agent_env.config as cfg + + monkeypatch.setattr(cfg, "get_config", lambda: _StubConfig()) + monkeypatch.setattr(ls, "_free_host_port", lambda: 41337) + monkeypatch.setattr(ls, "relay_image", _relay_image) + (tmp_path / ls._CONTAINER_MODE_MARKER).mkdir() + made: list[LocalSandbox] = [] + + class _RecordingProvider(LocalSandboxProvider): + async def create_vm(self, **kwargs): + sandbox = await LocalSandboxProvider.create_vm(self, **kwargs) + sandbox._work_dir = tmp_path + sandbox.scripts = [] + + async def _record(script, *, max_retries=0): + sandbox.scripts.append(script) + return "172.30.0.2\n" if script.startswith("docker inspect") else "" + + async def _listed(*args): + sandbox.scripts.append(" ".join(args)) + return 0, "r1\n", "" + + sandbox.exec_script, sandbox.exec_with_output = _record, _listed + made.append(sandbox) + return sandbox + + with pytest.raises(IsADirectoryError): + await _RecordingProvider().create_container(image_name="img:v1", port=8000, env={}, + network_policy=NetworkPolicy(mode="allowlist")) + + sandbox_id, scripts = made[0].sandbox_id, made[0].scripts + assert f"docker rm -f agent-{sandbox_id}" in scripts and "docker rm -f r1 >/dev/null" in scripts + assert any(f"docker network ls -q --filter label=agentenv.sandbox={sandbox_id}" in s for s in scripts) + + +@pytest.mark.asyncio +async def test_a_container_with_no_network_whose_setup_fails_goes_with_its_work_folder(monkeypatch, tmp_path): + monkeypatch.setenv("AGENT_ENV_LOCAL_SANDBOX_DIR", str(tmp_path)) + monkeypatch.setattr(ls, "relay_image", _relay_image) + + class _Unpullable(_RecordingLocalSandbox): + async def pull_images(self, image_names): + raise RuntimeError("pull access denied") + + class _RefusedPull(LocalSandboxProvider): + async def create_vm(self, **kwargs): + return _Unpullable(None, sandbox_id="local-closed", exposed_ports=kwargs["exposed_ports"]) + + with pytest.raises(RuntimeError, match="pull access denied"): + await _RefusedPull().create_container(image_name="img:v1", port=8000, env={}, + network_policy=NetworkPolicy(mode="allowlist")) + + assert LocalSandbox.find_work_dir("local-closed") is None + + +@pytest.mark.asyncio +async def test_a_container_with_no_network_takes_its_relay_and_network_with_it(tmp_path: Path): + sandbox = _RecordingLocalSandbox(tmp_path, sandbox_id="local-abc", listed=(0, "r1\n", "")) + sandbox.mode = SANDBOX_MODE_CONTAINER + (tmp_path / ls._CLOSED_NETWORK_FILE).write_text("closed-local-abc") + + await sandbox.terminate() + + assert sandbox.scripts[:3] == [f"docker rm -f {sandbox.container_name} >/dev/null 2>&1 || true", + "docker ps -aq --filter label=agentenv.sandbox=local-abc", "docker rm -f r1 >/dev/null"] + assert "docker network ls -q --filter label=agentenv.sandbox=local-abc" in sandbox.scripts[3] + + +async def _relay_image() -> str: + return "agentenv-relay:test" + + @pytest.mark.asyncio async def test_an_agent_placed_on_a_local_vm_publishes_only_on_the_host_ips(monkeypatch, tmp_path): monkeypatch.setattr(ls, "_host_ips", lambda: ("127.0.0.1", "172.17.0.1")) diff --git a/tst/unit/providers/sandbox_providers/network_policy_providers_test.py b/tst/unit/providers/sandbox_providers/network_policy_providers_test.py index 86fa1b7eb..fdddc320c 100644 --- a/tst/unit/providers/sandbox_providers/network_policy_providers_test.py +++ b/tst/unit/providers/sandbox_providers/network_policy_providers_test.py @@ -4,7 +4,7 @@ import pytest from agent_env.providers.sandbox_providers.chained_sandbox_provider import ChainedSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import ( @@ -17,6 +17,7 @@ from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider RESTRICTIVE = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("only-this.example.com",)) +CLOSED = NetworkPolicy(mode=NetworkMode.ALLOWLIST) ALLOWLIST = NetworkPolicy(mode=NetworkMode.ALLOWLIST, allow_hosts=("llm-proxy.example.com",)) ALLOW_ALL = NetworkPolicy() @@ -71,6 +72,14 @@ async def test_a_backend_refuses_before_provisioning_rather_than_ignoring_the_po await getattr(provider, method)(network_policy=RESTRICTIVE, **kwargs) +@pytest.mark.asyncio +async def test_local_takes_a_containers_network_away_but_not_a_vms(): + assert CLOSED.closed and not RESTRICTIVE.closed and not ALLOW_ALL.closed + assert LocalSandboxProvider.supports_network_policy(CLOSED) is True + with pytest.raises(NetworkPolicyUnsupportedError, match="takes a container's network away, not a VM's"): + await LocalSandboxProvider().create_vm(network_policy=CLOSED) + + @pytest.mark.asyncio async def test_local_provider_still_accepts_allow_all(): sandbox = await LocalSandboxProvider().create_vm(network_policy=ALLOW_ALL) diff --git a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py index ff64b3b1d..f609552e2 100644 --- a/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py +++ b/tst/unit/providers/sandbox_providers/network_policy_stamping_test.py @@ -18,7 +18,7 @@ from agent_env.providers.sandbox_providers.daytona.config import Settings as DaytonaSettings from agent_env.providers.sandbox_providers.daytona.provider import DaytonaSandboxProvider from agent_env.providers.sandbox_providers.e2b.provider import E2BSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandboxProvider from agent_env.providers.sandbox_providers.sail_vm import _sdk as sail_sdk diff --git a/tst/unit/providers/sandbox_providers/provider_capabilities_test.py b/tst/unit/providers/sandbox_providers/provider_capabilities_test.py index 86b3742b4..0c81f3195 100644 --- a/tst/unit/providers/sandbox_providers/provider_capabilities_test.py +++ b/tst/unit/providers/sandbox_providers/provider_capabilities_test.py @@ -20,7 +20,7 @@ from agent_env.providers.sandbox_providers.daytona.sandbox import DaytonaSandbox from agent_env.providers.sandbox_providers.e2b.provider import E2BSandboxProvider from agent_env.providers.sandbox_providers.e2b.sandbox import E2BSandbox -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider diff --git a/tst/unit/providers/sandbox_providers/relay_test.py b/tst/unit/providers/sandbox_providers/relay_test.py new file mode 100644 index 000000000..4d01c183a --- /dev/null +++ b/tst/unit/providers/sandbox_providers/relay_test.py @@ -0,0 +1,63 @@ +"""The relay local Docker runs beside a container with no network joins each connection to the container's port, both +ways, and closes a connection whose target refuses it.""" + +import asyncio +import socket + +import pytest +from pytest_socket import enable_socket + +from agent_env.providers.sandbox_providers.local_docker.relay import relay + + +@pytest.fixture(autouse=True) +def _disable_network(): + """Override the suite-wide socket block (tst/unit/conftest.py) for this module: loopback is the point here.""" + enable_socket() + yield + + +@pytest.mark.asyncio +async def test_the_relay_joins_each_connection_to_its_target_both_ways(): + async def echo(reader, writer): + writer.write(b"echo:" + await reader.readline()) + await writer.drain() + writer.close() + + target = await asyncio.start_server(echo, "127.0.0.1", 0) + port = _free_port() + relaying = asyncio.create_task(relay.main([f"{port}=127.0.0.1:{target.sockets[0].getsockname()[1]}"])) + try: + reader, writer = await _connect(port) + writer.write(b"hello\n") + await writer.drain() + assert await reader.read() == b"echo:hello\n" + finally: + relaying.cancel() + target.close() + + +@pytest.mark.asyncio +async def test_a_connection_whose_target_refuses_it_is_closed(): + port = _free_port() + relaying = asyncio.create_task(relay.main([f"{port}=127.0.0.1:{_free_port()}"])) + try: + reader, _ = await _connect(port) + assert await reader.read() == b"" + finally: + relaying.cancel() + + +async def _connect(port: int) -> tuple[asyncio.StreamReader, asyncio.StreamWriter]: + for _ in range(100): + try: + return await asyncio.open_connection("127.0.0.1", port) + except OSError: + await asyncio.sleep(0.02) + raise TimeoutError(f"the relay never listened on {port}") + + +def _free_port() -> int: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] diff --git a/tst/unit/providers/sandbox_providers/sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/sandbox_provider_test.py index 2340f65f9..fe9b8489c 100644 --- a/tst/unit/providers/sandbox_providers/sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/sandbox_provider_test.py @@ -8,12 +8,13 @@ import pytest import agent_env.providers.sandbox_providers.chained_sandbox_provider as chained_module +import agent_env.providers.sandbox_providers.local_docker.provider as lsm from agent_env.artifact.artifacts.docker_image import DockerImageArtifact from agent_env.config import get_config from agent_env.config.loader import load_impl from agent_env.providers.sandbox_providers.chained_sandbox_provider import ChainedSandboxProvider, skip_refusing -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandbox, ModalSandboxProvider from agent_env.providers.sandbox_providers.modal_vm_sandbox import ModalVmSandbox, ModalVmSandboxProvider from agent_env.providers.sandbox_providers import sandbox as sandbox_module @@ -88,9 +89,10 @@ def test_modal_sandbox_instance_mode_is_container(): @pytest.mark.asyncio -async def test_default_create_container_mutates_mode_to_container(): +async def test_default_create_container_mutates_mode_to_container_and_reset_container_replaces_it_in_place(): """The base SandboxProvider.create_container default impl should mutate the - returned VmSandbox's mode to 'container' once docker is running on it.""" + returned VmSandbox's mode to 'container' once docker is running on it; reset_container + removes that container and runs a fresh one on the same VM, and a container-only sandbox is refused.""" class _FakeVm(VmSandbox): type = "fake-vm" @@ -125,6 +127,13 @@ async def create_sandbox(self, **kwargs): assert result is fake_vm assert result.mode == "container" + fake_vm.scripts.clear() + await provider.reset_container(fake_vm, image_name="nginx:latest", port=8080, env={"A": "1"}) + assert fake_vm.scripts[0] == "docker rm -f agent-api > /dev/null" + assert fake_vm.scripts[1].startswith("docker run -d --name agent-api ") and "-e A=1" in fake_vm.scripts[1] + with pytest.raises(NotImplementedError, match="sandbox is the container itself"): + await provider.reset_container(_FakeSandbox(), image_name="nginx:latest", port=8080, env={}) + @pytest.mark.asyncio async def test_a_create_container_cancelled_during_the_pull_terminates_the_vm(): @@ -654,3 +663,10 @@ async def test_a_pull_that_fails_again_is_raised_as_it_was(no_pause): with pytest.raises(RuntimeError, match="pull access denied"): await _pull(sandbox, "registry/agent:v1") assert sandbox.scripts == ["docker pull registry/agent:v1"] * 2 + + +def test_local_docker_runs_this_machines_platform_natively_and_other_providers_linux_amd64(monkeypatch): + monkeypatch.setattr(lsm, "host_platform", lambda: "linux/arm64") + assert LocalSandboxProvider.native_platforms() == {"linux/arm64"} + assert ModalSandboxProvider.native_platforms() == {"linux/amd64"} + assert SandboxProvider.native_platforms() == {"linux/amd64"} diff --git a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py index d02d98311..84848e8bb 100644 --- a/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py +++ b/tst/unit/providers/sandbox_providers/test_config_sandbox_providers.py @@ -13,7 +13,7 @@ from agent_env.providers.sandbox_providers.daytona.config import Settings as DaytonaSettings from agent_env.providers.sandbox_providers.daytona.provider import DaytonaSandboxProvider from agent_env.providers.sandbox_providers.e2b.provider import E2BSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.sail_vm.provider import SailVmSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import ( diff --git a/tst/unit/providers/sandbox_providers/test_reachability_contributions.py b/tst/unit/providers/sandbox_providers/test_reachability_contributions.py index 2b931d907..9e1c10ad0 100644 --- a/tst/unit/providers/sandbox_providers/test_reachability_contributions.py +++ b/tst/unit/providers/sandbox_providers/test_reachability_contributions.py @@ -15,7 +15,7 @@ from agent_env.config import reset_config from agent_env.providers.sandbox_providers import sandbox_provider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import ( SandboxProvider, diff --git a/tst/unit/providers/sandbox_providers/test_url_rewrites.py b/tst/unit/providers/sandbox_providers/test_url_rewrites.py index 73ce28121..bbd1af140 100644 --- a/tst/unit/providers/sandbox_providers/test_url_rewrites.py +++ b/tst/unit/providers/sandbox_providers/test_url_rewrites.py @@ -7,8 +7,8 @@ from agent_env.config import reset_config from agent_env.env.env import DeployedEnv, DeployedSandboxEnv -from agent_env.providers.sandbox_providers import local_sandbox as local_sandbox_module -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider, host_url_for +from agent_env.providers.sandbox_providers.local_docker import provider as local_sandbox_module +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider, host_url_for from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider from agent_env.providers.sandbox_providers.sandbox import Sandbox from agent_env.providers.sandbox_providers.sandbox_provider import ( diff --git a/tst/unit/providers/sandbox_providers/vercel_sandbox_provider_test.py b/tst/unit/providers/sandbox_providers/vercel_sandbox_provider_test.py index 74be91c78..fd3149084 100644 --- a/tst/unit/providers/sandbox_providers/vercel_sandbox_provider_test.py +++ b/tst/unit/providers/sandbox_providers/vercel_sandbox_provider_test.py @@ -327,7 +327,6 @@ async def test_cleanup_records_an_already_deleted_sandbox_as_torn_down(monkeypat client = provider._client provider._client.get_sandbox.side_effect = type("Missing", (Exception,), {"status_code": 404})("gone") monkeypatch.setattr(teardown, "build_sandbox_provider", lambda _: provider) - monkeypatch.setattr(teardown_sandboxes, "build_sandbox_provider", lambda _: provider) context = TaskStepContext(deployed_sandboxes=[DeployedSandbox( sandbox_name="box", sandbox_id="gone", sandbox_mode="vm", sandbox_type="vercel", )]) @@ -339,7 +338,7 @@ async def test_cleanup_records_an_already_deleted_sandbox_as_torn_down(monkeypat await teardown_sandboxes.TeardownSandboxesTaskStep( id="cleanup", version=1, sandbox_names=["box"], ).execute(context) - assert context.metadata[teardown_sandboxes.TORN_DOWN_KEY] == ["gone"] + assert context.metadata[teardown.TORN_DOWN_KEY] == ["gone"] client.get_sandbox.assert_awaited_once_with(name="gone") client.aclose.assert_awaited_once() @@ -375,7 +374,6 @@ async def test_cleanup_does_not_mark_permission_failure_as_torn_down(monkeypatch provider = _provider() provider._client.get_sandbox.side_effect = type("Forbidden", (Exception,), {"status_code": 403})("forbidden") monkeypatch.setattr(teardown, "build_sandbox_provider", lambda _: provider) - monkeypatch.setattr(teardown_sandboxes, "build_sandbox_provider", lambda _: provider) context = TaskStepContext(deployed_sandboxes=[DeployedSandbox( sandbox_name="box", sandbox_id="box", sandbox_mode="vm", sandbox_type="vercel", )]) @@ -384,7 +382,7 @@ async def test_cleanup_does_not_mark_permission_failure_as_torn_down(monkeypatch assert len(report.failed) == 1 and report.still_up else: await teardown_sandboxes.TeardownSandboxesTaskStep(id="cleanup", version=1, sandbox_names=["box"]).execute(context) - assert not context.metadata.get(teardown_sandboxes.TORN_DOWN_KEY) + assert not context.metadata.get(teardown.TORN_DOWN_KEY) @pytest.mark.parametrize("tags", [{}, {"a": "1"}, {str(i): str(i) for i in range(5)}]) diff --git a/tst/unit/providers/sandbox_providers/vm_sandbox_test.py b/tst/unit/providers/sandbox_providers/vm_sandbox_test.py index 721123012..2e77e3540 100644 --- a/tst/unit/providers/sandbox_providers/vm_sandbox_test.py +++ b/tst/unit/providers/sandbox_providers/vm_sandbox_test.py @@ -997,7 +997,7 @@ async def test_a_vm_installs_the_trust_dir_in_its_container_through_itself_and_c @pytest.mark.asyncio async def test_start_trusting_a_container_on_a_vm_never_runs_docker_on_this_machine(tmp_path, monkeypatch): - import agent_env.providers.sandbox_providers.local_sandbox as ls + import agent_env.providers.sandbox_providers.local_docker.provider as ls monkeypatch.setattr(ls, "_copy_into_container", lambda *args: pytest.fail("docker cp ran on this machine")) (tmp_path / "ca.pem").write_bytes(b"ca") diff --git a/tst/unit/task/local_run_write_fence_test.py b/tst/unit/task/local_run_write_fence_test.py index a2d292dc6..eb8896ff4 100644 --- a/tst/unit/task/local_run_write_fence_test.py +++ b/tst/unit/task/local_run_write_fence_test.py @@ -22,7 +22,7 @@ from agent_env.store.ids import derive_id from agent_env.store.image_store import OciRegistryImageStore from agent_env.store.object_store import LocalFilesystemObjectStore -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.store.routing import LocalRunWriteError, run_scope from agent_env.task.task import Task from agent_env.task_step.context import DeployedSandbox, TaskStepContext diff --git a/tst/unit/task/teardown_test.py b/tst/unit/task/teardown_test.py index 73846835e..0008d979e 100644 --- a/tst/unit/task/teardown_test.py +++ b/tst/unit/task/teardown_test.py @@ -15,13 +15,11 @@ from agent_env.env.env import DeployedEnv, DeployedGatewayEnv from agent_env.providers.sandbox_providers.modal_sandbox import ModalSandboxProvider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, remove_local_work_dir +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, remove_local_work_dir from agent_env.task import teardown from agent_env.task.task import _rolled_back_sandboxes -from agent_env.task.teardown import RecordedSandbox, TeardownReport, recorded_sandboxes, teardown_run +from agent_env.task.teardown import TORN_DOWN_KEY, RecordedSandbox, TeardownReport, recorded_sandboxes, teardown_run from agent_env.task_step.context import DeployedAgent, DeployedSandbox, TaskStepContext -from agent_env.task_step.task_steps import teardown_sandboxes -from agent_env.task_step.task_steps.teardown_sandboxes import TORN_DOWN_KEY class _FakeSandbox: @@ -66,10 +64,9 @@ def __init__(self): @pytest.mark.asyncio -@pytest.mark.parametrize("step_entry", [False, True]) @pytest.mark.parametrize("failure", [False, True]) @pytest.mark.parametrize("close_failure", [False, True, "stall"]) -async def test_owned_teardown_provider_is_closed(monkeypatch, step_entry, failure, close_failure): +async def test_owned_teardown_provider_is_closed(monkeypatch, failure, close_failure): provider = MagicMock(ON_THIS_MACHINE=False) provider.terminate_sandbox = AsyncMock(side_effect=RuntimeError("denied") if failure else None) async def stalled_close(): @@ -77,11 +74,9 @@ async def stalled_close(): provider.close = AsyncMock(side_effect=(stalled_close if close_failure == "stall" else RuntimeError("client close failed") if close_failure else None)) - module = teardown_sandboxes if step_entry else teardown - monkeypatch.setattr(module, "build_sandbox_provider", lambda _: provider) - monkeypatch.setattr(module, "PROVIDER_CLOSE_TIMEOUT_SECONDS", 0.01, raising=False) - operation = (module._terminate("box", "vercel", "env") if step_entry else - module._terminate(RecordedSandbox("box", "vercel", "env"))) + monkeypatch.setattr(teardown, "build_sandbox_provider", lambda _: provider) + monkeypatch.setattr(teardown, "PROVIDER_CLOSE_TIMEOUT_SECONDS", 0.01) + operation = teardown.terminate(RecordedSandbox("box", "vercel", "env")) if failure: with pytest.raises(RuntimeError, match="denied"): await asyncio.wait_for(operation, 0.5) @@ -100,7 +95,6 @@ def build(name): return _FakeProvider(name, log) monkeypatch.setattr(teardown, "build_sandbox_provider", build) - monkeypatch.setattr(teardown_sandboxes, "build_sandbox_provider", build) monkeypatch.setattr(teardown, "_DEFAULT_PROVIDERS", { slot: (lambda slot=slot: _FakeProvider(f"default-{slot}", log)) for slot in ("agent", "env", "sandbox") }) @@ -171,6 +165,78 @@ def test_a_sandbox_a_step_already_tore_down_is_not_terminated_again(terminated): assert len(report.terminated) == 2 +class _Shop: + pass + + +@pytest.mark.parametrize("close, logged", [ + ("closes", None), + ("raises", "env shop wasn't closed (RuntimeError: its driver is gone); its sandboxes are terminated by id"), + ("hangs", "env shop wasn't closed (timed out after 0.05s)"), + ("can't", None), +]) +def test_the_runs_open_recordings_end_then_each_env_still_up_closes_then_what_stayed_open_registers_then_its_sandboxes_go_by_id( + terminated, monkeypatch, caplog, close, logged): + steps = [] + + async def close_deployed(deployed, env_class): + steps.append(("close", deployed.env_id, env_class, list(terminated))) + if close == "raises": + raise RuntimeError("its driver is gone") + if close == "hangs": + await asyncio.Event().wait() + if close == "can't": + raise NotImplementedError + + async def end_open_recordings(context, env_ids): + steps.append(("end", env_ids, list(terminated))) + + monkeypatch.setattr(teardown, "end_open_recordings", end_open_recordings) + monkeypatch.setattr(teardown, "close_deployed", close_deployed) + monkeypatch.setattr(teardown.Env, "get", lambda env_id, version: _Shop()) + monkeypatch.setattr(teardown, "register_open_recordings", + lambda context, env_ids: steps.append(("register", env_ids, list(terminated)))) + context = _context() + + report = _run(context, timeout=0.05) + + assert steps == [ + ("end", ["shop", "hosted"], []), ("close", "shop", _Shop, []), ("register", ["shop", "hosted"], []), + ] + assert {sandbox.sandbox_id for sandbox in report.terminated} == {"sb-box", "sb-db", "sb-gw", "sb-solver", "sb-vm"} + assert (logged in caplog.text) if logged else "wasn't closed" not in caplog.text + + +def test_an_env_class_of_a_plugins_own_closes_itself_and_its_sandboxes_still_go_by_id(terminated, monkeypatch): + closed = [] + + class _Own: + @classmethod + async def from_deployed_env(cls, deployed): + return cls() + + async def close(self): + closed.append("closed") + + monkeypatch.setattr(teardown.Env, "get", lambda env_id, version: _Own()) + + report = _run(_context()) + + assert closed == ["closed"] + assert {sandbox.sandbox_id for sandbox in report.terminated} == {"sb-box", "sb-db", "sb-gw", "sb-solver", "sb-vm"} + + +def test_an_env_a_step_already_tore_down_is_not_closed_again(terminated, monkeypatch): + close_deployed = AsyncMock() + monkeypatch.setattr(teardown, "close_deployed", close_deployed) + context = _context() + context.metadata[TORN_DOWN_KEY] = ["sb-gw", "sb-db", "sb-vm"] + + _run(context) + + close_deployed.assert_not_awaited() + + @pytest.mark.parametrize("behavior, why", [ ("raise", "RuntimeError: boom"), ("gone", "RuntimeError: no sandbox sb-gw"), diff --git a/tst/unit/task_step/add_skills_env_test.py b/tst/unit/task_step/add_skills_env_test.py new file mode 100644 index 000000000..0ba651fe0 --- /dev/null +++ b/tst/unit/task_step/add_skills_env_test.py @@ -0,0 +1,104 @@ +"""``add_skills`` ``env_ids``: each env's skill, or each of its child envs' behind a gateway, written under that card's +environment name, installed on the agent and recorded in the run; refused for an env the run didn't deploy or bind to +the agent, or that serves no skill.""" + +import hashlib +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from agentenv_protocol import SKILL_EXTENSION_URI + +from agent_env.a2a_agent import A2AAgent +from agent_env.env import env as env_module +from agent_env.env.env import DeployedSandboxEnv +from agent_env.task_step.context import DeployedAgent, TaskStepContext +from agent_env.task_step.task_steps.add_skills import ENV_SKILLS_KEY, AddSkillsTaskStep + +_SKILL = {"uri": SKILL_EXTENSION_URI, "params": {"methods": { + "get": {"method": "GET", "endpoint": "/agentenv/ext/skill/get"}}}} + + +def _env(env_id: str = "shop", name: str = "my-shop", extensions: tuple = (_SKILL,), children: tuple = () + ) -> DeployedSandboxEnv: + card = {"name": name, "capabilities": {"extensions": list(extensions)}} + if children: + card["children_environments"] = [{"name": child, "capabilities": {"extensions": list(served)}} + for child, served in children] + return DeployedSandboxEnv(env_id=env_id, env_version=1, environment_card_url=f"http://{env_id}/.well-known/agent-env.json", + environment_card=card, sandbox_id=f"sb-{env_id}", sandbox_type="local") + + +@pytest.fixture +def agent(monkeypatch): + """The agent's skill registrations and the env calls the step made.""" + made = SimpleNamespace(calls=[], register=AsyncMock(return_value={"ok": True})) + + async def invoke(base_url, card, uri, params=None, timeout=30, verify=True, *, method=None): + made.calls.append((base_url, card["name"], uri, method, params)) + return {"name": params["name"], "description": "How to use this env", "skill_md": f"---\nname: {params['name']}\n"} + + monkeypatch.setattr(env_module.protocol_v1, "invoke_extension", invoke) + monkeypatch.setattr(A2AAgent, "register_skill", made.register) + monkeypatch.setattr("agent_env.a2a_agent.store.get_a2a_agent_instance_store", + lambda: SimpleNamespace(get=lambda instance_id: "deployed-agent")) + return made + + +def _context(*envs, bound=("shop",)) -> TaskStepContext: + return TaskStepContext(deployed_envs=list(envs), deployed_agents=[DeployedAgent( + agent_name="solver", api_url="http://a", instance_id="inst-1", env_ids=list(bound))]) + + +def _step(*env_ids) -> AddSkillsTaskStep: + return AddSkillsTaskStep(id="skills", version=None, agent_name="solver", env_ids=list(env_ids)) + + +@pytest.mark.asyncio +async def test_each_envs_skill_is_installed_under_its_environment_name_and_recorded(agent): + context = await _step("shop").execute(_context(_env())) + + assert agent.calls == [("http://shop", "my-shop", SKILL_EXTENSION_URI, "get", {"name": "my-shop"})] + agent.register.assert_awaited_once_with("deployed-agent", name="my-shop", description="How to use this env", + skill_md="---\nname: my-shop\n") + assert context.metadata[ENV_SKILLS_KEY] == {"solver": {"shop": {"my-shop": "---\nname: my-shop\n"}}} + + +@pytest.mark.asyncio +async def test_an_env_behind_a_gateway_installs_each_child_envs_skill_under_the_childs_name(agent): + gateway = _env("office", name="office", extensions=(), children=(("mail", (_SKILL,)), ("Chat Room", (_SKILL,)), + ("calendar", ()))) + + context = await _step("office").execute(_context(gateway, bound=("office",))) + + digest = hashlib.sha256(b"office/Chat Room").hexdigest()[:12] + assert [(card, params["name"]) for _, card, *_, params in agent.calls] == [ + ("mail", "mail"), ("Chat Room", f"chat-room-{digest}")] + assert set(context.metadata[ENV_SKILLS_KEY]["solver"]["office"]) == {"mail", f"chat-room-{digest}"} + + +@pytest.mark.asyncio +async def test_envs_whose_names_come_out_alike_get_skills_of_their_own(agent): + await _step("shop", "other").execute(_context(_env(name="my-shop"), _env("other", name="My_Shop"), + bound=("shop", "other"))) + + digest = hashlib.sha256(b"other").hexdigest()[:12] + assert [params["name"] for *_, params in agent.calls] == ["my-shop", f"my-shop-{digest}"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("envs, bound, problem", [ + ((), ("shop",), "env 'shop' isn't deployed in this run"), + ((_env(),), (), "env 'shop' isn't bound to agent 'solver': name it in the env_ids of its deploy_agent"), + ((_env(extensions=(), children=(("mail", ()),)),), ("shop",), + f"env 'shop' serves no skill: neither its card nor a child env's declares {SKILL_EXTENSION_URI}"), +], ids=["not-deployed", "not-bound", "no-skill"]) +async def test_an_env_the_skill_cant_come_from_fails_the_step_naming_it(agent, envs, bound, problem): + with pytest.raises(RuntimeError, match=problem.replace(".", "[.]")): + await _step("shop").execute(_context(*envs, bound=bound)) + agent.register.assert_not_awaited() + + +def test_the_step_round_trips_and_a_step_without_envs_serializes_as_before(): + assert AddSkillsTaskStep.from_dict(_step("shop").to_dict()).env_ids == ["shop"] + assert "env_ids" not in AddSkillsTaskStep(id="s", version=None).to_dict() diff --git a/tst/unit/task_step/computer_recording_test.py b/tst/unit/task_step/computer_recording_test.py index be2be1910..42e6199c6 100644 --- a/tst/unit/task_step/computer_recording_test.py +++ b/tst/unit/task_step/computer_recording_test.py @@ -1,10 +1,11 @@ """start_computer_recording and end_computer_recording against a deployed env standing in for a driver: the grant it's handed, the artifact its uploaded segments and manifest become, what's registered when the driver forgot the -recording, teardown ending what a run left open, and a reset leaving it going.""" +recording, and what a run left open registered at teardown from its namespace, with the manifest its env uploaded +ending it.""" from __future__ import annotations -import asyncio +import json import pytest from agentenv_protocol.computer.card import COMPUTER_RECORDING @@ -12,7 +13,6 @@ from agent_env.artifact import Artifact from agent_env.config import set_object_store from agent_env.store.base import GrantUnavailableError -from agent_env.task.teardown import teardown_run from agent_env.task_step.context import TaskStepContext from agent_env.task_step.task_steps import computer_recording as recording_steps from agent_env.task_step.task_steps.computer_recording import ( @@ -21,7 +21,6 @@ StartComputerRecordingTaskStep, ) from agent_env.task_step.task_steps.reset_env import ResetEnvTaskStep -from agent_env.task_step.task_steps.teardown_sandboxes import TeardownSandboxesTaskStep from tst.util.granting_object_store import GrantingObjectStore _CARD = {"name": "desktop", "capabilities": {"extensions": [{"uri": COMPUTER_RECORDING, "params": { @@ -30,14 +29,14 @@ class _Computer: - """A deployed env whose driver records: each end uploads one segment per started recording under its grant, - or leaves the last one out when ``drop_last``; it ``forgets`` the recording, ``garbles`` its manifest, or - ``hangs`` at end.""" + """A deployed env whose driver records: an end uploads one segment and the manifest under its grant, and end_all + under the grant the recording started with, or leaves the segment out when ``drop_last``; it ``forgets`` the + recording or ``garbles`` its manifest.""" def __init__(self, store, env_id: str = "desktop", forgets: bool = False, garbles: bool = False, - hangs: bool = False, drop_last: bool = False) -> None: - self.env_id, self.environment_card, self.sandbox_type = env_id, _CARD, "local" - self.store, self.forgets, self.garbles, self.hangs, self.drop_last = store, forgets, garbles, hangs, drop_last + drop_last: bool = False) -> None: + self.env_id, self.environment_card, self.sandbox_type, self.metadata = env_id, _CARD, "local", None + self.store, self.forgets, self.garbles, self.drop_last = store, forgets, garbles, drop_last self.calls: list[tuple[str, dict]] = [] self.started: set[str] = set() @@ -47,19 +46,23 @@ async def invoke(self, uri, method, params=None, **_): if method == "start": if params["request_id"] not in self.started: # a repeated request_id joins the first recording self.started.add(params["request_id"]) - self.store.put(f"{params['upload']['root_path']}/rec-1/0000.mp4", b"mp4-0", content_type="video/mp4") + self.root = params["upload"]["root_path"] + self.store.put(f"{self.root}/rec-1/0000.mp4", b"mp4-0", content_type="video/mp4") return {"recording_id": "rec-1", "started_at": "2026-10-27T10:00:00.000Z", "format": "video/mp4"} if self.forgets: raise RuntimeError("400 invalid_argument: there is no recording 'rec-1' on this computer") if self.garbles: return {"recording_id": "rec-1", "segments": "all of them"} - if self.hangs: - await asyncio.sleep(3600) + if method == "end_all" and not self.started: + return {"manifests": []} + root = (params.get("upload") or {}).get("root_path", self.root) segments = [{"index": 0, "path": "rec-1/0000.mp4", "uploaded": True}, {"index": 1, "path": "rec-1/0001.mp4", "uploaded": not self.drop_last}] if not self.drop_last: - self.store.put(f"{params['upload']['root_path']}/rec-1/0001.mp4", b"mp4-1", content_type="video/mp4") - return {"recording_id": "rec-1", "ended_by": params["reason"], "segments": segments} + self.store.put(f"{root}/rec-1/0001.mp4", b"mp4-1", content_type="video/mp4") + manifest = {"recording_id": "rec-1", "ended_by": params["reason"], "segments": segments} + self.store.put(f"{root}/rec-1/manifest.json", json.dumps(manifest).encode(), content_type="application/json") + return {"manifests": [manifest]} if method == "end_all" else manifest @pytest.fixture @@ -154,7 +157,7 @@ def broken(*args): @pytest.mark.asyncio -async def test_a_reset_leaves_a_recording_going_and_teardown_ends_what_a_run_left_open(store): +async def test_what_a_run_left_open_registers_from_its_namespace_with_the_manifest_its_env_uploaded_ending_it(store): desktop, other = _Computer(store), _Computer(store, env_id="other") context = _context(desktop, other) context = await _start("a").execute(context) @@ -162,27 +165,31 @@ async def test_a_reset_leaves_a_recording_going_and_teardown_ends_what_a_run_lef await ResetEnvTaskStep(id="reset", version=None, env_id="other").execute(context) recordings = context.metadata[RECORDINGS_KEY] assert recordings["b"]["state"] == "open" and [method for method, _ in other.calls] == ["start"] - await teardown_run(context) - assert (recordings["a"]["ended_by"], recordings["b"]["ended_by"]) == ("teardown", "teardown") - assert [method for method, _ in desktop.calls] == ["start", "end"] + await desktop.invoke(COMPUTER_RECORDING, "end_all", {"reason": "teardown"}) + recording_steps.register_open_recordings(context) -@pytest.mark.asyncio -async def test_teardown_goes_on_when_a_recording_wont_end_in_time(store): - context = await _start().execute(_context(_Computer(store, hangs=True))) - report = await teardown_run(context, timeout=0.2) - assert context.metadata[RECORDINGS_KEY]["desk"]["state"] == "open" and not report.left + assert [(recordings[n]["state"], recordings[n]["ended_by"], recordings[n]["segments"]) for n in "ab"] == [ + ("ended", "teardown", 2), ("ended", "error", 1)] + universe = Artifact.get(recordings["a"]["artifact"]["id"]) + assert sorted(universe.file_artifact_ids) == ["manifest.json", "rec-1-0000.mp4", "rec-1-0001.mp4"] @pytest.mark.asyncio -async def test_a_teardown_sandboxes_step_ends_the_recordings_of_its_envs_only(store): - desktop, other = _Computer(store), _Computer(store, env_id="other") - context = _context(desktop, other) - context = await _start("a").execute(context) - context = await _start("b", env_id="other").execute(context) - await TeardownSandboxesTaskStep(id="down", version=None, env_ids=["desktop"]).execute(context) +async def test_ending_or_registering_one_envs_open_recordings_leaves_the_others(store): + desktop, other, third = _Computer(store), _Computer(store, env_id="other"), _Computer(store, env_id="third") + context = _context(desktop, other, third) + for name, env_id in (("a", "desktop"), ("b", "other"), ("c", "third")): + context = await _start(name, env_id=env_id).execute(context) recordings = context.metadata[RECORDINGS_KEY] - assert (recordings["a"]["state"], recordings["b"]["state"]) == ("ended", "open") + + await recording_steps.end_open_recordings(context, ["desktop"]) + recording_steps.register_open_recordings(context, ["other"]) + + assert [(recordings[n]["state"], recordings[n].get("ended_by")) for n in "abc"] == [ + ("ended", "teardown"), ("ended", "error"), ("open", None)] + (_, start), (method, end) = desktop.calls + assert (method, end["reason"], end["upload"]["root_path"]) == ("end", "teardown", start["upload"]["root_path"]) def test_the_steps_round_trip_and_refuse_what_they_cant_run(): @@ -219,9 +226,14 @@ async def test_a_store_that_cant_sign_for_the_whole_recording_grants_what_it_can @pytest.mark.asyncio -async def test_recording_an_env_this_run_didnt_deploy_fails_the_start(store): +async def test_recording_an_env_this_run_didnt_deploy_or_that_has_no_network_fails_the_start(store): context = await _start(env_id="elsewhere").execute(_context(_Computer(store))) assert "env 'elsewhere' isn't deployed in this run" in context.metadata[RECORDINGS_KEY]["desk"]["error"] + closed = _Computer(store) + closed.metadata = {"computer": {"network": {"mode": "allowlist"}}} + context = await _start().execute(_context(closed)) + assert "deployed with no network, so its computer can't upload" in context.metadata[RECORDINGS_KEY]["desk"]["error"] + assert closed.calls == [] @pytest.mark.asyncio @@ -233,7 +245,7 @@ async def test_a_retried_start_joins_the_first_attempts_namespace_and_the_grant_ (_, one), (_, two) = computer.calls assert one["request_id"] == two["request_id"] and one["upload"]["root_path"] == two["upload"]["root_path"] assert context.metadata[RECORDINGS_KEY]["desk"]["namespace"] == first["namespace"] - assert one["upload"]["max_objects"] == recording_steps.MAX_SEGMENTS + assert one["upload"]["max_objects"] == recording_steps.MAX_SEGMENTS + 1 @pytest.mark.asyncio @@ -248,34 +260,35 @@ async def test_a_start_reusing_an_open_recordings_name_is_refused_and_leaves_it_ @pytest.mark.asyncio -async def test_teardown_goes_on_when_a_forgotten_recordings_namespace_cant_be_listed(store, monkeypatch): +async def test_a_forgotten_recording_whose_namespace_cant_be_listed_is_marked_failed(store, monkeypatch): context = await _start().execute(_context(_Computer(store, forgets=True))) def unlisted(prefix): raise ConnectionError("the object store is down") monkeypatch.setattr(store, "list_at", unlisted) - await teardown_run(context) + await _end().execute(context) entry = context.metadata[RECORDINGS_KEY]["desk"] assert entry["state"] == "failed" and "then ConnectionError: the object store is down" in entry["error"] @pytest.mark.asyncio -async def test_a_recording_that_cant_be_ended_at_all_is_marked_and_the_rest_still_end(store, monkeypatch): +async def test_a_recording_that_cant_be_registered_is_marked_and_the_rest_still_register(store, monkeypatch): context = _context(_Computer(store), _Computer(store, env_id="other")) context = await _start("a").execute(context) context = await _start("b", env_id="other").execute(context) - real = recording_steps.end_recording + real = recording_steps._register - async def broken(context, name, reason): + def broken(context, name, namespace, manifest): if name == "a": - raise RuntimeError("unexpected") - await real(context, name, reason) + raise RuntimeError("the document store is down") + return real(context, name, namespace, manifest) - monkeypatch.setattr(recording_steps, "end_recording", broken) - await recording_steps.end_open_recordings(context) + monkeypatch.setattr(recording_steps, "_register", broken) + recording_steps.register_open_recordings(context) recordings = context.metadata[RECORDINGS_KEY] assert (recordings["a"]["state"], recordings["b"]["state"]) == ("failed", "ended") + assert recordings["a"]["error"] == "RuntimeError: the document store is down" class _Gateway(_Computer): diff --git a/tst/unit/task_step/deploy_env_test.py b/tst/unit/task_step/deploy_env_test.py index 119e87987..874832e4a 100644 --- a/tst/unit/task_step/deploy_env_test.py +++ b/tst/unit/task_step/deploy_env_test.py @@ -6,7 +6,7 @@ import pytest -from agent_env.env.env import DeployedGatewayEnv +from agent_env.env.env import DeployedGatewayEnv, Env from agent_env.task_step.context import TaskStepContext from agent_env.task_step.task_steps.deploy_env import DeployEnvTaskStep @@ -263,14 +263,23 @@ def test_preflight_reports_an_option_a_server_env_would_refuse(options, refused) ("other", {"gateway_mode": "consistent", "env_state_type": "remote_postgres"}), ], ids=["server-sizing", "gateway-env", "other-env-type"]) def test_preflight_passes_what_the_env_takes(kind, options): - env = MagicMock() if kind == "other" else _mcp_env(kind) + env = _OtherEnv("other", 1) if kind == "other" else _mcp_env(kind) assert _preflight(env, **options) == [] +@pytest.mark.parametrize("kind, refusal", [ + ("server", "env 'mcp-email' has env_provider_type 'server', which doesn't take env_params"), + ("other", "other envs take no env_params"), +]) +def test_preflight_refuses_env_params_an_env_doesnt_take(kind, refusal): + env = _OtherEnv("other", 1) if kind == "other" else _mcp_env(kind) + assert _preflight(env, env_params={"form": "container"}) == [f"deploy_env 'd': {refusal}"] + + @pytest.mark.parametrize("kind", ["server", "gateway", "other"]) @pytest.mark.parametrize("mode", ["bogus", "Performance", None]) def test_preflight_reports_an_invalid_gateway_mode_for_every_env(kind, mode): - env = MagicMock() if kind == "other" else _mcp_env(kind) + env = _OtherEnv("other", 1) if kind == "other" else _mcp_env(kind) assert _preflight(env, gateway_mode=mode) == [f"deploy_env 'd': {mode!r} is not a valid GatewayMode"] @@ -312,6 +321,12 @@ def test_a_task_reports_only_its_refused_deploy_step(): assert task.preflight() == ["deploy_env 'bad': env 'mcp-email' has env_provider_type 'server', which doesn't take gateway_mode"] +class _OtherEnv(Env): + """An env type that deploys itself, through no provider, and checks none of its options.""" + + type = "other" + + def _mcp_env(env_provider_type: str): from agent_env.env.envs.mcp_server import MCPServerEnv diff --git a/tst/unit/task_step/local_container_urls_test.py b/tst/unit/task_step/local_container_urls_test.py index 9f06b04e6..8320dfcd3 100644 --- a/tst/unit/task_step/local_container_urls_test.py +++ b/tst/unit/task_step/local_container_urls_test.py @@ -21,7 +21,7 @@ from agent_env.config.model import ModelConfig from agent_env.env.env import DeployedGatewayEnv from agent_env.providers.sandbox_providers import sandbox_provider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.sandbox_provider import SandboxProvider from agent_env.task_step.context import DeployedAgent, DeployedSandbox, TaskStepContext from agent_env.task_step.task_steps.install_agent import _build_param_resolvers diff --git a/tst/unit/task_step/local_step_containers_test.py b/tst/unit/task_step/local_step_containers_test.py index 83f2d6ca0..565a08d33 100644 --- a/tst/unit/task_step/local_step_containers_test.py +++ b/tst/unit/task_step/local_step_containers_test.py @@ -13,9 +13,9 @@ from agent_env.a2a_agent import a2a_agent as a2a_agent_module from agent_env.a2a_agent.a2a_agent import A2AAgent from agent_env.env.env import DeployedEnv -from agent_env.providers.sandbox_providers import local_sandbox as local_sandbox_module +from agent_env.providers.sandbox_providers.local_docker import provider as local_sandbox_module from agent_env.providers.sandbox_providers import sandbox_provider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.providers.sandbox_providers.sandbox import VmSandbox from agent_env.task_step.context import DeployedSandbox, TaskStepContext from agent_env.task_step.task_steps.deploy_agent import _choose_mcp_url diff --git a/tst/unit/task_step/run_docker_container_test.py b/tst/unit/task_step/run_docker_container_test.py index c570d6fbd..24dc37d27 100644 --- a/tst/unit/task_step/run_docker_container_test.py +++ b/tst/unit/task_step/run_docker_container_test.py @@ -11,7 +11,7 @@ from agent_env.config import set_object_store from agent_env.providers.sandbox_providers import sandbox_provider -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.task_step.context import DeployedSandbox, TaskStepContext from agent_env.task_step.task_steps.run_docker_container import ( RunDockerContainerTaskStep as Step, diff --git a/tst/unit/task_step/test_collect_artifacts.py b/tst/unit/task_step/test_collect_artifacts.py index 64cb52b08..31bc1ad76 100644 --- a/tst/unit/task_step/test_collect_artifacts.py +++ b/tst/unit/task_step/test_collect_artifacts.py @@ -19,7 +19,7 @@ import pytest from agent_env.task_step.context import DeployedAgent, PromptResponse, TaskStepContext -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.task_step.task_steps.collect_artifacts import CollectArtifactsTaskStep, _exec_args, _is_url_entry from agent_env.task_step.task_steps.sandbox_utils.sandbox_utils import find_agent_container from agent_env.env.env import DeployedEnv, DeployedGatewayEnv, EnvCapabilityUnsupported diff --git a/tst/unit/task_step/test_deploy_agent_role_delivery.py b/tst/unit/task_step/test_deploy_agent_role_delivery.py index f560f6483..d9e0707ae 100644 --- a/tst/unit/task_step/test_deploy_agent_role_delivery.py +++ b/tst/unit/task_step/test_deploy_agent_role_delivery.py @@ -47,7 +47,7 @@ def _card(*extensions: dict) -> dict: @pytest.mark.asyncio -async def test_a_card_with_both_paths_gets_the_role_on_both(wiring, caplog): +async def test_a_card_with_both_paths_gets_the_role_on_both_and_the_agent_records_its_envs(wiring, caplog): with caplog.at_level(logging.WARNING, logger=_LOGGER): context = await wiring.deploy(_card(HEADERS_MCP, ROLE_CONFIG), role="executor") @@ -56,7 +56,7 @@ async def test_a_card_with_both_paths_gets_the_role_on_both(wiring, caplog): ] assert wiring.bodies("/ext/agent-config") == [{"name": "solver", "role": "executor"}] assert not [m for m in caplog.messages if "role" in m] - assert [a.role for a in context.deployed_agents] == ["executor"] + assert [(a.role, a.env_ids) for a in context.deployed_agents] == [("executor", ["crm"])] @pytest.mark.asyncio diff --git a/tst/unit/task_step/test_entity_refs.py b/tst/unit/task_step/test_entity_refs.py index f91c3ab14..77d4d7fcb 100644 --- a/tst/unit/task_step/test_entity_refs.py +++ b/tst/unit/task_step/test_entity_refs.py @@ -160,7 +160,7 @@ def test_declared_paths_match_what_to_dict_writes(): LoadArtifactTaskStep(id="l", version=None, env_id="crm", artifact_id="seed", artifact_version=1), AddSkillsTaskStep( id="s", version=None, skills=[{"skill_artifact_id": "sk", "skill_artifact_version": 5}], - cli_artifact_ids=["cli-crm"], file_artifact_universe_ids=["fau-crm"], + cli_artifact_ids=["cli-crm"], file_artifact_universe_ids=["fau-crm"], env_ids=["desk"], ), DeployAgentTaskStep(id="a", version=None, env_ids=["crm", "mail"], a2a_agent_id="triage", a2a_agent_version=3), PromptAgentTaskStep(id="p", version=None, prompt="hi", snapshot_config=SnapshotConfig(env_id="tickets")), @@ -175,7 +175,7 @@ def test_declared_paths_match_what_to_dict_writes(): "load_artifact": [("env_id", "crm", None), ("artifacts[0].id", "seed", 1)], "add_skills": [ ("skills[0].skill_artifact_id", "sk", 5), ("cli_artifact_ids[0]", "cli-crm", None), - ("file_artifact_universe_ids[0]", "fau-crm", None), + ("file_artifact_universe_ids[0]", "fau-crm", None), ("env_ids[0]", "desk", None), ], "deploy_agent": [("env_ids[0]", "crm", None), ("env_ids[1]", "mail", None), ("a2a_agent_id", "triage", 3)], "prompt_agent": [("snapshot_config.env_id", "tickets", None)], diff --git a/tst/unit/task_step/test_run_code.py b/tst/unit/task_step/test_run_code.py index 53c3c50a8..1c7d30fae 100644 --- a/tst/unit/task_step/test_run_code.py +++ b/tst/unit/task_step/test_run_code.py @@ -20,7 +20,7 @@ from agent_env.artifact import FileArtifact from agent_env.artifact.artifacts.file_artifact_universe import FileArtifactUniverse from agent_env.env.env import DeployedEnv, DeployedGatewayEnv, DeployedSandboxEnv, EnvNeedsSandbox -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox from agent_env.providers.sandbox_providers.sandbox import Sandbox, VmSandbox from agent_env.task_step.context import DeployedAgent, DeployedSandbox, TaskStepContext from agent_env.task_step.task_steps import run_code as _run_code_module diff --git a/tst/unit/task_step/test_teardown_sandboxes.py b/tst/unit/task_step/test_teardown_sandboxes.py index 3d1d9f979..a7120fcf9 100644 --- a/tst/unit/task_step/test_teardown_sandboxes.py +++ b/tst/unit/task_step/test_teardown_sandboxes.py @@ -11,13 +11,16 @@ from agent_env.task_step.context import DeployedAgent, DeployedSandbox, TaskStepContext from agent_env.task_step.context_ops import build_context_update_ops from agent_env.task_step.registry import get_task_step_registry -from agent_env.task_step.task_steps import teardown_sandboxes -from agent_env.task_step.task_steps.teardown_sandboxes import TORN_DOWN_KEY, TeardownSandboxesTaskStep +from agent_env.task import teardown +from agent_env.task.teardown import TORN_DOWN_KEY +from agent_env.task_step.task_steps.teardown_sandboxes import TeardownSandboxesTaskStep _BACKENDS = {"modal", "modal_vm"} class _FakeSandbox: + ON_THIS_MACHINE = False + def __init__(self, provider, sandbox_id): self.provider, self.sandbox_id = provider, sandbox_id @@ -28,6 +31,8 @@ async def terminate(self): class _FakeProvider: + ON_THIS_MACHINE = False + def __init__(self, name, terminated, failing): self.name, self.terminated, self.failing = name, terminated, failing @@ -60,8 +65,8 @@ def build(name): raise ValueError(f"unknown backend {name}") return _FakeProvider(name, log, failing) - monkeypatch.setattr(teardown_sandboxes, "build_sandbox_provider", build) - monkeypatch.setattr(teardown_sandboxes, "_DEFAULT_PROVIDERS", { + monkeypatch.setattr(teardown, "build_sandbox_provider", build) + monkeypatch.setattr(teardown, "_DEFAULT_PROVIDERS", { slot: (lambda slot=slot: _FakeProvider(f"default-{slot}", log, failing)) for slot in ("agent", "env", "sandbox") }) diff --git a/tst/unit/task_step/verifiers/test_verify_sandbox.py b/tst/unit/task_step/verifiers/test_verify_sandbox.py index 1f703d4e4..03dbb6a6b 100644 --- a/tst/unit/task_step/verifiers/test_verify_sandbox.py +++ b/tst/unit/task_step/verifiers/test_verify_sandbox.py @@ -4,7 +4,7 @@ import pytest -from agent_env.providers.sandbox_providers.local_sandbox import LocalSandbox, LocalSandboxProvider +from agent_env.providers.sandbox_providers.local_docker.provider import LocalSandbox, LocalSandboxProvider from agent_env.task_step.context import DeployedAgent, DeployedSandbox, TaskStepContext from agent_env.task_step.task_step import TaskStep from agent_env.task_step.task_steps.verifiers import verify_sandbox diff --git a/tst/unit/test_clean_install_wheel.py b/tst/unit/test_clean_install_wheel.py index beb389838..7390d6487 100644 --- a/tst/unit/test_clean_install_wheel.py +++ b/tst/unit/test_clean_install_wheel.py @@ -63,6 +63,6 @@ def test_the_gate_checks_the_folder_of_every_shipped_bundle(): roots = {name: root.relative_to(SCRIPT.parents[2]).as_posix() for name, root in clean_install.bundle_roots().items()} assert roots == {"agents": "src/agent_env/agents", "computers": "src/agent_env/computers", - "drivers": "src/agent_env/drivers", "echo-mcp": "src/agent_env/examples/echo-mcp", + "desktop": "src/agent_env/examples/desktop", "drivers": "src/agent_env/drivers", "echo-mcp": "src/agent_env/examples/echo-mcp", "hello": "src/agent_env/examples/hello", "mail-mcp": "src/agent_env/examples/mail-mcp", "next-env-event": "src/agent_env/examples/next-env-event"}