Repository navigation
feat(bundle): register a folder bundle init wrote back into the store it came from - #356
Closed
earakely-scale wants to merge 32 commits into
Closed
earakely-scale wants to merge 32 commits into
earakely-scale wants to merge 32 commits into
Conversation
…back can tell an edit provenance.json, schema 2, now also holds: - the task map, each task file with the store's task id and version, and the eval's; - a fingerprint of the configured document store's section, which tells one store from another without naming where either is; - each file the import wrote, as its content hash (+x when executable). The pinned store references move from "store" to "pinned". Nothing reads the file yet. The file hash the ledger computes moves to _fs, so both use one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…igured stores The bundle is planned as a run plans it, then each @Local id it writes takes a registry id: - an entry takes <prefix><name>, the prefix the folder's name and a dash by default; - a task's output takes <task id>-<name>; - a derived id keeps its suffix; - a shipped agent named by its alias takes the alias. Each task and eval names the version of what it references. Each image the bundle builds is built for linux/amd64, pushed under a tag of its sources and registered by digest with its build context. A ledger in the target store's bundle_registrations makes a rerun write only what changed. Problems are refused before anything is written: an id the store holds that the bundle didn't register (unless --adopt), an @Local id left in a document, or, for a shared store, a credential written out or a URL only this machine reaches. A step that leaves its agent unset stays unset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…her than write the next one Two registrations of one bundle on different machines aren't serialized by the local locks, so both wrote every id, the second at v2. The registration ledger now holds each write to the version its check found next (expecting_version): a put of that id then takes exactly that version, through insert_next, or raises ConcurrentModificationError, and a rerun reuses what the winner wrote. Only an agent an installed bundle comes with is registered under its alias: no store env may start on an installed env alias's id, so an installed env is refused before anything is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pending row marks a write a run started, so the next run keeps the version an interrupted one wrote. One whose write raised wrote no version the run knows of, so the row is removed: a registration that lost a race left one behind in the shared store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed registration, recheck ownership at each write - A step naming no version took another step's pin of the same id; each now takes the version the bundle wrote, or the store's latest the plan read. - A registration killed after a write but before its ledger row was done couldn't resume: the rerun refused the id as unowned. A pending row of this bundle at the stored version now counts as its own, so the ledger adopts it. - Ownership is checked again as each write is checked, so an id another bundle registers after the first check is refused before it's written, not taken at its next version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… it came from When the folder's provenance.json records an import from the configured store: - each entity, image, task and eval takes the store id it was imported from, and an imported task's outputs the names its file gives them; - a write the folder still holds as the import wrote it, with all it needs at the versions they were imported at, keeps the version it was imported at, so an unedited folder writes nothing, and the ledger records it; - an edit re-versions only what it reaches; - each imported task's steps get back the user-scoped fields the import left out (its agent, env vars, sandbox type...), from the stored version, by step id, where the folder doesn't set them. A shared store's credential and URL checks skip those fields, which it holds already. A folder imported from another store, or recorded by an older agent-env, registers as a new bundle, and says why. Registering over a version someone else wrote since, the bundle's own or the imported one, is refused unless --adopt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bundle's own ledger row let it write an id whatever the store's latest was, so a version another tool or person wrote after the last registration was written over unseen. Now an id whose latest version isn't the one this bundle last registered is refused unless --adopt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/agent_env/bundle/register.py
…hich nothing resolves The shared-store check let a secret-named env var through when its value was a secret: reference, as if the run would resolve it. Nothing resolves env: or secret: references in a step's env vars, so the run would get the reference itself. Such a value is now refused, whatever the variable's name, with a reason saying so. The other refusals now say what each one exposes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests that a bundle's identity drops a remote's credentials held URLs with user info written out, which the pre-commit secret scan flags as new secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/agent_env/bundle/register.py
… timeout A task and an artifact whose names make one id live in different stores, so they no longer refuse each other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/agent_env/bundle/register.py
…d id its own, when registering the folder - A pin is given back only beside the reference it was stored with, not beside an entity the import wrote into the folder. - An env var the import dropped and a credential it masked come back where the folder still holds what it left. - A list item, such as a skill, takes back its left-out source from the stored item it was written from. - Envs sharing one stored image register as imported; an edit to one copy is refused. - An image whose toml names another Dockerfile is rebuilt. - An installed bundle's entries take nothing from the folder's record, and an output under a folder keeps its name. - The import numbers a name another kind holds for another id, and registering refuses one name taking two ids. - The store fingerprint resolves env: references and ignores credentials, and registering reads only the files of what the import wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ead each pinned version one way - materialize takes the ledger instead of a Registering wrapper: Ledger.pins_references now also decides whether tasks and evals are written pinned, and Ledger.pushes_images whether built images are pushed. - Tasks and evals pin through one helper, reading the versions straight from what was written rather than a copy kept in step by hand. An eval ref whose store version the plan didn't read now floats instead of raising, as a task's does. - The kind-to-store table is gone: an EntityKind's value is its store's key. - The registry-id check no longer rewrites "__", which the id pattern already allows. - translated() builds its fields in _translations, beside _KEPT, the fields that hold no @Local id; a test fails when a Plan field is in neither. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/agent_env/bundle/register.py
…urce the folder gives - The import records each image it names by registry ref with its stored id and version, so registering an untouched folder reuses it rather than writing it, its env and its task anew. - A list item's left-out fields come back together, and only when the item sets none of them, so a new object_url or a switch to body or skill_artifact_id is never undone by the stored s3_url. Matching by name no longer applies when the item sets a field its stored item doesn't. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and through one that's down Builds and pushes the env's image to the local registry, runs the registered task by its id to a passing score, and checks registering again writes nothing. A registry that can't be reached fails the registration before the image is built or named, and a rerun once it's back finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ration tests on the local registry main folded Plan's aliases and env_aliases into alias_uses, by kind, so a registered plan renames each of them there. The integration tests now use the local registry whatever AGENT_ENV_CONFIG names, so they never push to a hosted one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n unchanged imported write once - Registration compares each imported write with the files its writer reads: links inside the bundle are followed, so a linked folder in an image's context is an edit to it, and what the OS leaves behind, such as .DS_Store, isn't. - An unchanged imported write is adopted only when the ledger has no row of this bundle for it at that version and digest, so registering again adds no rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… it was masked from While a list is as the import left it, each item takes back its own position's value. Once it's edited, an item takes back the value of the one stored item whose scrubbed form it equals, so reordered downloads keep their own credentials, and an item that matches none, or several, keeps what it holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and refuse one that can't be given back - An edited list item that equals no stored item matches the one whose masked values it still holds, so a renamed download takes back its URL. - A field still holding what the import masked after everything is given back refuses the registration, naming it, rather than writing the masked text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ites A task the selection doesn't reach is never written, so what it still holds masked refuses nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gister An id the store holds already now takes the bundle's next version, whoever wrote the store's latest. Two bundles taking one id is unlikely, and the check cost a refusal, its recheck at each write, and --adopt to get past it. The strict version each write lands on stays, so two registrations racing for one id still can't both write it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # src/agent_env/bundle/register.py
…imports # Conflicts: # src/agent_env/bundle/register.py
…ted skill nothing back by its name - The store fingerprint resolves secret: references as well as env:, and hashes only the store's impl and the strings saying where it is: a URL's scheme, host and path, nothing a secret-named key holds, and no tuning option. Two stages whose configs hold the same text but whose secrets reach different databases no longer look like one store, and a teammate's different timeout no longer makes the folder register as a new bundle. - A list item matches its stored item only by being it apart from the fields the import left out, so an edited skill takes back no source by sharing a name, and registration says it names none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…portedWrites - ImportedWrites, built once from the record, the plan and what restore gave back, holds each imported write's source, which writes the folder holds as imported, and what couldn't be given back, and answers the record's ids, the shared image copies check and the versions the ledger reads. register.py drops _given, _held_as_imported, _files_of and _sources, and passes it in place of five separate values. - Writes are told apart by store and @Local id there, since a task and an env of one name share an @Local id. - A record's id and version are a Source, and an image's are an ImportedImage with its Dockerfile, in place of positional tuples and two maps of one key. - registry_ids's ids the record gives are `recorded`, no longer `kept`, which elsewhere means the writes left as imported. - One string walk and one key-path lookup, in _values, in place of _strings, _texts, _at and _at_keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Shelving this for now. Registering an imported folder back in place isn't needed yet. It's mostly a hub workflow, and that work hasn't started; it may handle editing stored tasks differently. The branch stays at e71ec77, with CI green, Greptile 5/5 and the dev round trip passed, in case we pick it up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
agent-env bundle initwrites a stored task into a folder you can run and edit. Registering that folder back should keep the task what it was. The folder's entities should go back to the ids they came from, and an untouched folder should write nothing. An edit should re-version only what it reaches, without losing the fields the import left out.Stacked on #355 (
bundle register), and it includes #353's commit (the import record v2), which this is the reader of. If you'd rather review one PR for the import side, close #353 and merge this after #355.What it does
When the folder's
provenance.jsonrecords an import from the configured store (its store fingerprint matches):Ids: each entity, image, task and eval takes the store id it was imported from. An imported task's outputs keep the names its file gives them, as the stored task did. Only ids register makes up are held to the registry id rule, so a source id like
nightly/regressionpasses.Untouched folders write nothing: a write whose files are as the import wrote them keeps the version it was imported at, as long as everything it needs is kept at the versions they were imported at too. The ledger records that version, so later runs compare digests as usual. The files compared are:
This compares against the import's file hashes, not the stored entity's digests, because import prunes contexts and pins their installs.
Edits re-version only the edited write and what references it.
Re-grafting: each imported task's steps get back the user-scoped fields the import left out, such as the agent, env vars, sandbox type and model. They come from the stored version, matched by step id and type, wherever the folder doesn't set them. A shared store's credential and loopback-URL checks skip these fields, since the store holds them already.
A version someone else wrote since the folder was imported: an untouched entry still keeps the version it was imported at, and an edit writes the store's next version.
A folder imported from another store, or recorded by an older agent-env, registers as a new bundle with prefix ids, and the run says why.
Verification
Unit tests:
tst/unit/bundle/register_imports_test.py, 8 tests, plus #353's import-record tests. They cover:The full unit tier passes: 10,912 passed. One a2a staging timing test failed once under load (the run took 15 minutes instead of 2) and passes 3/3 alone; this PR doesn't touch it.
Open-source stores on an agent-environments-admin VM (Mongo, SeaweedFS S3, and a TLS registry with auth):
echo-mcpregistered, then imported from those stores withbundle init. Registering the folder back, as a dry run and for real, kept every entity at its source id and v1, writing nothing.--adopt. (That refusal came from an ownership check since dropped; an edit now writes v4 directly.)Dev stage, from a laptop, since the dev cluster isn't reachable from the admin VM:
fd3965-e2e-echo-smoketask feat(bundle): agent-env bundle register writes a bundle into the configured stores #355's e2e registered was imported and registered back untouched, writing nothing.sandbox_type: localthe import had dropped, and ran on a Modal VM, score 1.0.task-rocket-prospecting-christina-rivera-v1(v7) was imported with--include-scope userand dry-run registered. All 21 writes showed as unchanged at their source versions, and nothing was written. The writes are 9 MCP-server envs and their images, the multi-env, theclaude-code-cliagent (v74) and its image, the universe artifact, and the task.bundle inititself. Two of its stored build contexts hold several Dockerfiles and record none, which import already refuses without--include-scope user.🤖 Generated with Claude Code


Confidence Score: 5/5
The changes appear safe to merge; no new blocking issue was found.Summary
Lets imported folders register back under their source IDs and versions, while restoring task settings omitted during import.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Imported folder] --> B[Read matching import record] B --> C[Restore omitted task settings] C --> D[Plan selected writes] D --> E[Match source IDs and compare files] E --> F{Files and dependencies unchanged?} F -->|Yes| G[Reuse imported version] F -->|No| H[Write next version]Reviews (13) · Last reviewed commit: "refactor(bundle): gather what an import ..." · Reviewed by Greptile