Skip to content

feat(connectors): add file-identity signals and a 15-step checkpoint to nebius/swe_agent_trajectories - #3

Merged
etienneRV merged 3 commits into
feature/nebius-swe-agent-trajectoriesfrom
feature/swe-agent-file-identity
Nov 21, 2026
Merged

etienneRV merged 3 commits into
feature/nebius-swe-agent-trajectoriesfrom
feature/swe-agent-file-identity

Conversation

@ziyad-m97

@ziyad-m97 ziyad-m97 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Adds two per-step signals and one checkpoint to the SWE-agent connector of OpenTSLM#832. Merge this into feature/nebius-swe-agent-trajectories and OpenTSLM#832 picks it up.

What changes

  • same_file_edit (bool): the step is an edit of the same file as the run's previous edit.
  • edit_target (enum none / source / test / own_script): the kind of file an edit changes. own_script is a file the agent created earlier in the run, usually its reproduction script.
  • A checkpoint at 15 steps: checkpoints are now 5, 10, 15, 20 and 30 steps. A typical run is 17 steps long (median), so the monitor gets a point between 10 and 20.
  • The status report gains one fact sentence: "N edits changed the same file as the edit before, and M edits changed a file the agent had created." The prediction sentence is unchanged.
  • README and dataset.yaml are updated with figures from a full build, plus a section on three channels that were tried and left out.

Why

  • An edit command names no file. SWE-agent applies it to the open file, and every reply names that file in its footer, (Open file: <path>). Following the footer separates patching the repository from patching the agent's own script, and it shows edit loops on one file.
  • It is the only extra channel that helped. Under 5-fold cross-validation by repository, a gradient-boosted model on the signals gains −0.001 / +0.006 / +0.010 AUROC at 5 / 10 / 20 steps with these two signals. Repeated errors, tests vs own script, and thought phrases gave no gain, so they stay out. The README lists their numbers.
  • Nothing about the ending leaks in. Both signals use only the step's command and the reply before it. Checkpoints still exist only for runs that go on after step k.
  • The test-file rule skips the repository's folder. Otherwise every file of a repository called pytest would count as a test. One of the tests covers this case.

Measured on a full build (80,036 runs)

  • Records: 235,978 records (previously 192,428) and 707,934 tasks. The 15-step checkpoint holds 43,550 runs, 9.2 % of them resolved.

  • Edits: over the steps the checkpoints show, there are 406,189 edits:

    • 235,311 change source files;
    • 163,682 change files the agent created;
    • 6,870 change test files;
    • 326 have no file open.

    67 % of edits change the same file as the edit before.

  • Split: unchanged (by repository). Per-partition rates at 15 steps: 9.1 / 9.9 / 9.4 % resolved for train / validation / test.

Checks run

  • make check (ruff format, ruff check, ty, schemas): passed
  • make test: 2,979 passed, 2 skipped, 2 xfailed
  • make test-connectors: all connector checks passed (this connector: 45 tests, 7 of them new or updated for the new signals)
  • Full build: timenet-build build nebius/swe_agent_trajectories --no-isolation took 3 min 41 s and peaked at 4.7 GB. I read the release back: all nine signals and the 15-step checkpoint are present. The status text rebuilt by agent-ecg from the signals matches the targets on 100 % of a 2,000-checkpoint sample.

The training side, etienneRV/agent-ecg#1, is pinned to this branch. On the full build, its signals baseline reaches same-issue pairs of 0.61 at 10 steps and 0.65 at 30 on test (0.74 at 30 under repository cross-validation).

🤖 Generated with Claude Code

ziyad-m97 and others added 2 commits November 21, 2026 14:35
…to nebius/swe_agent_trajectories

An edit command names no file: it changes the file open before it, which SWE-agent's
"(Open file: ...)" footer names. The connector now follows that footer and the files the agent
creates, and adds two per-step signals:

- same_file_edit: the edit changes the same file as the run's edit before it;
- edit_target: none / source / test / own_script (a file the agent created in this run).

A test file is recognised by its folders and name after the repository's folder, so a repository
called pytest does not make all its files tests. The status report gains one fact sentence, and a
checkpoint at 15 steps (the middle of a typical run) joins 5, 10, 20 and 30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
…checkpoint of nebius/swe_agent_trajectories

Measured on a full build: 235,978 records and 707,934 tasks; the 15-step checkpoint holds 43,550
runs (9.2 % resolved). The README explains how an edit's file is followed through SWE-agent's
"(Open file: ...)" footer, gives the edit counts by kind of file, and lists three channels that
were tried and left out because they did not improve a signals model under a repository split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
… card

The signal-selection results belong with the models (etienneRV/agent-ecg), not
with the dataset.
@etienneRV
etienneRV merged commit 4e0f11c into feature/nebius-swe-agent-trajectories Nov 21, 2026
etienneRV pushed a commit to etienneRV/agent-ecg that referenced this pull request Nov 21, 2026
…, calibration and replay

- Follow etienneRV/TimeNet#3: nine per-step signals (same_file_edit, edit_target) and a checkpoint
  at 15 steps; the connector is pinned to that branch. `prepare` reads label-set names from their
  `name` annotation, where the rebased connector API stores them (it crashed on `label_set.name`).
- OpenTSLM samples: the 0/1 series labels no longer state their counts ("2 of 5 steps"), which let
  the model answer from the text and skip the series; 19 series; `opentslm-export --context
  full|thoughts|none`; one Prompts class builds the pre-prompt for both the export and the replay.
- Evaluation: same-issue pair score; an issue+signals+thoughts baseline; `baselines --cv 5`
  (repository cross-validation over all checkpoints); file-identity features; calibrated
  probabilities for every baseline.
- monitor.py: isotonic calibration per checkpoint, health 0-100 as the percentile among validation
  runs of the same age, and a critical alarm where at least 95 % of validation runs failed (two
  steps in a row, held until health is back to 30). `agent-ecg replay` and `opentslm/replay.py`
  write one JSON line per step for the live ECG.
- GPU scripts: `predict.py --calibrate-from`, and the samples the run trained on by default;
  `load_model` loads an OpenTSLM checkpoint onto `--llm`, so `--init-from` works on the ungated
  unsloth/Llama-3.2-1B.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants