Skip to content

feat(connectors): add the SWE-agent trajectories connector - #832

Open
etienneRV wants to merge 13 commits into
OpenTSLM:mainfrom
etienneRV:feature/nebius-swe-agent-trajectories
Open

etienneRV wants to merge 13 commits into
OpenTSLM:mainfrom
etienneRV:feature/nebius-swe-agent-trajectories

Conversation

@etienneRV

@etienneRV etienneRV commented Nov 20, 2026 •

Copy link
Copy Markdown

Description

Adds nebius/swe_agent_trajectories, a connector for nebius/SWE-agent-trajectories (CC BY 4.0): 80,036 runs of the SWE-agent coding agent trying to resolve real GitHub issues, each labelled with whether its final patch passed the issue's tests.

Why it matters: coding agents run for many steps and most runs fail (83 % here). A monitor that reads an agent's steps as they happen, its "ECG", could warn early that a run is going wrong. This dataset lets a time-series language model learn that from per-step signals and the agent's own reasoning.

The connector downloads the 12 parquet shards (1.1 GB) at a pinned revision and checks each shard's row count.

Mapping

Checkpoints, never whole runs. The end of a run gives its outcome away: every submitted run ends with submit, and 36 % of failed runs stop at the context limit. So each run is cut after 5, 10, 15, 20 and 30 steps, but only if it goes on after them. No input holds the outcome, the exit status, the run's length, the patch or the test log.

Record = one checkpoint (235,978). It holds one source, agent_steps, with nine signals on an OrdinalAxis. The steps have an order but the release has no timestamps.

Signal dtype Per step
thought_chars, action_chars, observation_chars int32 lengths of the agent's thought, its command and the environment's reply
context_chars int32 conversation length so far
repeated_action bool command repeats an earlier one
action_kind enum edit / create / run / read / search / submit / other
observation_kind enum edit_rejected / timeout / traceback / command_error / silent_success / output
same_file_edit bool the step edits the same file as the run's previous edit
edit_target enum for an edit, the kind of file it changes: source / test / own_script (a file the agent created); none otherwise

An edit command names no file, so the connector follows the (Open file: <path>) footer that ends every SWE-agent reply: an edit changes the file the reply before it names. Both file signals use only that step's command and earlier replies.

Text inputs:

  • The agent's thoughts: the record annotation agent_thoughts, one string per step. It's a LazyTextList, not a str signal, because TimeFTorchDataset has no tensor form for string series.
  • The issue: a dataset annotation, stored once per instance (3,599), with the instance_id in its metadata.

Both are named in each task's input_annotations.

Three tasks per checkpoint (707,934):

Task Target
ClassificationTask run_outcome resolved / unresolved
ClassificationTask run_ending submitted / context_limit / early_exit / submitted_no_patch / format_errors / cost_limit
AnswerTask a rule-made status report: the steps so far, then how many more steps the run will take, how it will end and whether it will resolve the issue

The kinds, signals and status texts come from fixed rules, which the README lists verbatim. No model writes them.

Decisions

  • Split by repository: StratifiedSplitter (8/1/1, stratified on the outcome, seed 42). No repository or issue is in two partitions, which matters because issues have 22 runs on average and 807 issues have both outcomes.
  • Checkpoint rule: a run counts for checkpoint k only if it has more than k steps, so its final action is never visible.
  • Imbalance kept as published: resolved rates fall from 16.6 % at 5 steps to 4.2 % at 30. The README recommends per-checkpoint AUROC/AUPRC, calibration, and weighting long runs down instead of accuracy.
  • Fields not read: generated_patch, eval_logs, mask, cutoff_date.
  • Rows are validated, never repaired: rows are read with the shared ParquetDataReader into a pydantic SourceRow. A missing text, an unknown exit status, a malformed instance id or messages out of turn raise TimeNetConnectorError. Thoughts are kept exactly as written.
  • convert reads every run first, on purpose: every row is checked before anything is written, and the 1.1 GB of shards are read once. Each run keeps only the signals of its first 30 steps and its compressed thoughts, and runs of one issue share one copy of its text, so memory stays bounded (peak RSS measured below). The Records' values still load lazily.
  • A known limit of the traceback rule: a file view that merely shows a traceback (after open or edit) counts as one: 69 of 9,375 tracebacks in shard 0. The README states it.

Source notes documented in the README

  • Licence conditions: the dataset page adds the Llama 3.1 License for the model outputs and the licences of the underlying repositories. The README states both.
  • No citation entry: the release has none, so the card cites the dataset page and the revision.
  • 13 instances appear with two slightly different issue texts, and 16 texts are shared by two instances of the same repository. Each (instance, text) pair gets its own annotation.
  • early_exit is not explained by the release; the connector keeps the name.

Files

packages/timenet-connectors/src/timenet_connectors/datasets/nebius/__init__.py
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/__init__.py
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/connector.py   # records, annotations, task stream
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/source.py      # parquet reading, step parsing and signals
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/tasks.py       # checkpoints, split, labels, status text
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/dataset.yaml
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/README.md
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/requirements.txt
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/tests/__init__.py
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/tests/test_connector.py

The tests write a synthetic shard, so the repo ships no dataset bytes.

How to load

import timenet

timenet.build("nebius/swe_agent_trajectories")  # or: timenet-build build nebius/swe_agent_trajectories
with timenet.TimeNet().open("nebius/swe_agent_trajectories") as reader:
    dataset = reader.read()
train = dataset.get_train()

Validation

Check Result
make check (ruff format, ruff check, ty) passed
make check-ci (all pre-commit hooks + ty on 3.11) passed
make test 2,979 passed, 2 skipped, 2 xfailed
make test-connectors all connectors passed (pytest + ty), incl. nebius/swe_agent_trajectories
make license-check passed
Connector tests (synthetic shard: parsing rules, file-identity rules, checkpoint rule, no outcome on records, thoughts kept verbatim, one issue annotation per instance, status text, repository split, storage round trip, 9 malformed-row cases, missing target, row count) 45 passed
timenet-build build nebius/swe_agent_trajectories from the Hub first build (7 signals, 4 checkpoints): ~5 min incl. the 1.1 GB download, peak RSS 1.9 GB; current version (9 signals, 5 checkpoints, --no-isolation): 3 min 41 s, peak 4.7 GB

Read-back of the built release:

  • 235,978 records; every signal of every record has exactly k values; every visible step has a reply.
  • 707,934 tasks (235,978 per task type): train 570,120, validation 67,254, test 70,560. Every task names issue and agent_thoughts as inputs; every issue annotation names the task's own instance (3,599 annotations).
  • No repository (of 1,275 with a checkpoint) and no issue appears in two partitions.
Partition Repositories Runs Resolved at 5 / 10 / 15 / 20 / 30 steps
train 993 60,985 16.6 / 13.7 / 9.1 / 6.2 / 3.9 %
validation 139 7,557 16.8 / 14.6 / 9.9 / 6.8 / 4.4 %
test 143 7,640 16.4 / 14.1 / 9.4 / 7.1 / 5.9 %

Example checkpoint (run train-00000-of-00012:0, 5 steps):

  • action_kind → other, search, search, search, read; observation_kind → traceback, output, output, output, command_error
  • run_outcome → unresolved; run_ending → context_limit
  • answer → "In its first 5 steps the agent edited files 0 times, created 0 files, ran code 0 times, read or searched the repository 4 times and repeated an earlier command 0 times. The linter rejected 0 edits, 1 reply showed a Python traceback, 1 reply a command error and 0 commands timed out. 0 edits changed the same file as the edit before, and 0 edits changed a file the agent had created. The conversation holds 12,857 characters. The run will go on for 41 more steps, then it will reach the context limit and SWE-agent will submit its changes, and the agent will not resolve the issue."

Signal check (not part of the connector; reproducible with etienneRV/agent-ecg, agent-ecg baselines): simple CPU models trained on this release's train split and scored on its test split (143 held-out repositories) reach AUROC 0.68 / 0.70 / 0.71 / 0.70 / 0.70 for "will it resolve" at 5 / 10 / 15 / 20 / 30 steps from the agent's thoughts, 0.61-0.73 from the step signals, and 0.61-0.62 from the issue text alone (chance 0.50).

Licence and citation

The data is CC BY 4.0 (dataset page). Using the model outputs also requires complying with the Llama 3.1 License, and the repositories keep their own licences. Cite:

Nebius. SWE-agent trajectories (nebius/SWE-agent-trajectories). Hugging Face, 2024. Revision 68195a1450865274106246d0d0296a1d6807b88e.

Limitations

  • One agent framework and one model family (93 % of runs from Llama 70B).
  • Signals and kinds are rule-made from the logged text. Status reports are templates.
  • The outcome is the release's test result.

Contributed by Team RAMadan during the TimeNet hackathon, Round 1 (November 2026).

Checklist

There is no PR template in the repo; this mirrors CONTRIBUTING.md and AGENTS.md.

  • Conventional Commits, Conventional Branch (feature/nebius-swe-agent-trajectories)
  • make check, make check-ci, make test-unit, make license-check pass
  • make test-connectors passes
  • Google-style docstrings; ty green
  • Source pinned by Hub revision and shard row counts; no raw data committed
  • Card validates against dataset-card-v1; licence, citation and source URL set
  • Source-data problems raise TimeNetConnectorError; nothing is filled

Note: we can't add labels on this repo. Could a maintainer add area:connectors? Thanks!

Parse each run of nebius/SWE-agent-trajectories into its outcome and the
per-step signals of its first 30 steps: thought, command and reply
lengths, conversation length, repeated commands, command kind and reply
kind. The build fails on an unknown exit status, a malformed instance id,
a trajectory out of turn, or a shard with another row count.
… checkpoint tasks

Cut each run after 5, 10, 20 and 30 steps while it is still going, so no
input shows how the run ends. Each checkpoint is one Record of ordinal
step signals with the agent's thoughts as text, the issue is a dataset
annotation, and three streamed tasks ask whether the run will resolve
its issue, how it will end, and for a rule-made status report. Runs are
split by repository.
- Read the shards with the shared ParquetDataReader into a SourceRow model, and raise on a
  missing message text or a system message with text, instead of filling in empty strings.
- Keep thoughts exactly as written; thought_chars now marks where each one ends.
- Store one issue annotation per instance and text, only for instances with a checkpoint, so
  no task names another instance's issue.
- Keep the exit statuses in one table, rename the label sets, and reuse the Record metadata for
  the tasks.
- Ask the status report for the remaining steps, the ending and the outcome, and write its
  target in the same future tense.
- Give repeated_action a dimensionless unit like the other signals.
…onnector API

Main changed the connector API. A label set carries its name as an
annotation, a Hugging Face connector passes its Hub settings as class
keywords and reads a mapping from part to path, BaseConnector names the
whole download result, and Annotation takes its value by keyword.
etienneRV added a commit to etienneRV/agent-ecg that referenced this pull request Nov 20, 2026
…update

OpenTSLM/TimeNet#832 now carries main's new connector API (b8385e7d).
Tests and lint pass against it.
ziyad-m97 and others added 3 commits November 21, 2026 16:02
…to nebius/swe_agent_trajectories

An edit command names no file: it changes the file open before it, which SWE-agent's
"(Open file: ...)" footer names. The connector now follows that footer and the files the agent
creates, and adds two per-step signals:

- same_file_edit: the edit changes the same file as the run's edit before it;
- edit_target: none / source / test / own_script (a file the agent created in this run).

A test file is recognised by its folders and name after the repository's folder, so a repository
called pytest does not make all its files tests. The status report gains one fact sentence, and a
checkpoint at 15 steps (the middle of a typical run) joins 5, 10, 20 and 30.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
…checkpoint of nebius/swe_agent_trajectories

Measured on a full build: 235,978 records and 707,934 tasks; the 15-step checkpoint holds 43,550
runs (9.2 % resolved). The README explains how an edit's file is followed through SWE-agent's
"(Open file: ...)" footer, gives the edit counts by kind of file, and lists three channels that
were tried and left out because they did not improve a signals model under a repository split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
… card

The signal-selection results belong with the models (etienneRV/agent-ecg), not
with the dataset.
@etienneRV

etienneRV commented Nov 21, 2026 •

Copy link
Copy Markdown
Author

Thanks for porting this to the new API, @masquare! FYI: we've since added two small things on top, a couple of signals that track which file each edit touches and a checkpoint at 15 steps.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants