Repository navigation
feat(connectors): add file-identity signals and a 15-step checkpoint to nebius/swe_agent_trajectories - #3
Merged
etienneRV merged 3 commits intoNov 21, 2026
Conversation
…to nebius/swe_agent_trajectories An edit command names no file: it changes the file open before it, which SWE-agent's "(Open file: ...)" footer names. The connector now follows that footer and the files the agent creates, and adds two per-step signals: - same_file_edit: the edit changes the same file as the run's edit before it; - edit_target: none / source / test / own_script (a file the agent created in this run). A test file is recognised by its folders and name after the repository's folder, so a repository called pytest does not make all its files tests. The status report gains one fact sentence, and a checkpoint at 15 steps (the middle of a typical run) joins 5, 10, 20 and 30. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
…checkpoint of nebius/swe_agent_trajectories Measured on a full build: 235,978 records and 707,934 tasks; the 15-step checkpoint holds 43,550 runs (9.2 % resolved). The README explains how an edit's file is followed through SWE-agent's "(Open file: ...)" footer, gives the edit counts by kind of file, and lists three channels that were tried and left out because they did not improve a signals model under a repository split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
… card The signal-selection results belong with the models (etienneRV/agent-ecg), not with the dataset.
etienneRV
pushed a commit
to etienneRV/agent-ecg
that referenced
this pull request
Nov 21, 2026
…, calibration and replay - Follow etienneRV/TimeNet#3: nine per-step signals (same_file_edit, edit_target) and a checkpoint at 15 steps; the connector is pinned to that branch. `prepare` reads label-set names from their `name` annotation, where the rebased connector API stores them (it crashed on `label_set.name`). - OpenTSLM samples: the 0/1 series labels no longer state their counts ("2 of 5 steps"), which let the model answer from the text and skip the series; 19 series; `opentslm-export --context full|thoughts|none`; one Prompts class builds the pre-prompt for both the export and the replay. - Evaluation: same-issue pair score; an issue+signals+thoughts baseline; `baselines --cv 5` (repository cross-validation over all checkpoints); file-identity features; calibrated probabilities for every baseline. - monitor.py: isotonic calibration per checkpoint, health 0-100 as the percentile among validation runs of the same age, and a critical alarm where at least 95 % of validation runs failed (two steps in a row, held until health is back to 30). `agent-ecg replay` and `opentslm/replay.py` write one JSON line per step for the live ECG. - GPU scripts: `predict.py --calibrate-from`, and the samples the run trained on by default; `load_model` loads an OpenTSLM checkpoint onto `--llm`, so `--init-from` works on the ungated unsloth/Llama-3.2-1B. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two per-step signals and one checkpoint to the SWE-agent connector of OpenTSLM#832. Merge this into
feature/nebius-swe-agent-trajectoriesand OpenTSLM#832 picks it up.What changes
same_file_edit(bool): the step is aneditof the same file as the run's previous edit.edit_target(enumnone/source/test/own_script): the kind of file aneditchanges.own_scriptis a file the agent created earlier in the run, usually its reproduction script.dataset.yamlare updated with figures from a full build, plus a section on three channels that were tried and left out.Why
editcommand names no file. SWE-agent applies it to the open file, and every reply names that file in its footer,(Open file: <path>). Following the footer separates patching the repository from patching the agent's own script, and it shows edit loops on one file.pytestwould count as a test. One of the tests covers this case.Measured on a full build (80,036 runs)
Records: 235,978 records (previously 192,428) and 707,934 tasks. The 15-step checkpoint holds 43,550 runs, 9.2 % of them resolved.
Edits: over the steps the checkpoints show, there are 406,189 edits:
67 % of edits change the same file as the edit before.
Split: unchanged (by repository). Per-partition rates at 15 steps: 9.1 / 9.9 / 9.4 % resolved for train / validation / test.
Checks run
make check(ruff format, ruff check, ty, schemas): passedmake test: 2,979 passed, 2 skipped, 2 xfailedmake test-connectors: all connector checks passed (this connector: 45 tests, 7 of them new or updated for the new signals)timenet-build build nebius/swe_agent_trajectories --no-isolationtook 3 min 41 s and peaked at 4.7 GB. I read the release back: all nine signals and the 15-step checkpoint are present. The status text rebuilt by agent-ecg from the signals matches the targets on 100 % of a 2,000-checkpoint sample.The training side, etienneRV/agent-ecg#1, is pinned to this branch. On the full build, its signals baseline reaches same-issue pairs of 0.61 at 10 steps and 0.65 at 30 on test (0.74 at 30 under repository cross-validation).
🤖 Generated with Claude Code