Skip to content

feat(connectors): add the SWE-agent trajectories connector - #2

Closed
etienneRV wants to merge 8 commits into
mainfrom
feature/nebius-swe-agent-trajectories
Closed

etienneRV wants to merge 8 commits into
mainfrom
feature/nebius-swe-agent-trajectories

Conversation

@etienneRV

@etienneRV etienneRV commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Description

Adds nebius/swe_agent_trajectories, a connector for nebius/SWE-agent-trajectories (CC BY 4.0): 80,036 runs of the SWE-agent coding agent trying to resolve real GitHub issues, each labelled with whether its final patch passed the issue's tests.

Why it matters: coding agents run for many steps and most runs fail (83 % here). A monitor that reads an agent's steps as they happen, its "ECG", could warn early that a run is going wrong. This dataset lets a time-series language model learn that from per-step signals and the agent's own reasoning.

The connector downloads the 12 parquet shards (1.1 GB) at a pinned revision and checks each shard's row count.

Mapping

Checkpoints, never whole runs. The end of a run gives its outcome away: every submitted run ends with submit, and 36 % of failed runs stop at the context limit. So each run is cut after 5, 10, 20 and 30 steps, but only if it goes on after them. No input holds the outcome, the exit status, the run's length, the patch or the test log.

Record = one checkpoint (192,428). It holds one source, agent_steps, with seven signals on an OrdinalAxis. The steps have an order but the release has no timestamps.

Signal dtype Per step
thought_chars, action_chars, observation_chars int32 lengths of the agent's thought, its command and the environment's reply
context_chars int32 conversation length so far
repeated_action bool command repeats an earlier one
action_kind enum edit / create / run / read / search / submit / other
observation_kind enum edit_rejected / timeout / traceback / command_error / silent_success / output

Text inputs:

  • The agent's thoughts: the record annotation agent_thoughts, one string per step. It's a LazyTextList, not a str signal, because TimeFTorchDataset has no tensor form for string series.
  • The issue: a dataset annotation, stored once per instance (3,599), with the instance_id in its metadata.

Both are named in each task's input_annotations.

Three tasks per checkpoint (577,284):

Task Target
ClassificationTask run_outcome resolved / unresolved
ClassificationTask run_ending submitted / context_limit / early_exit / submitted_no_patch / format_errors / cost_limit
AnswerTask a rule-made status report: the steps so far, then how many more steps the run will take, how it will end and whether it will resolve the issue

The kinds, signals and status texts come from fixed rules, which the README lists verbatim. No model writes them.

Decisions

  • Split by repository: StratifiedSplitter (8/1/1, stratified on the outcome, seed 42). No repository or issue is in two partitions, which matters because issues have 22 runs on average and 807 issues have both outcomes.
  • Checkpoint rule: a run counts for checkpoint k only if it has more than k steps, so its final action is never visible.
  • Imbalance kept as published: resolved rates fall from 16.6 % at 5 steps to 4.2 % at 30. The README recommends per-checkpoint AUROC/AUPRC, calibration, and weighting long runs down instead of accuracy.
  • Fields not read: generated_patch, eval_logs, mask, cutoff_date.
  • Rows are validated, never repaired: rows are read with the shared ParquetDataReader into a pydantic SourceRow. A missing text, an unknown exit status, a malformed instance id or messages out of turn raise TimeNetConnectorError. Thoughts are kept exactly as written.
  • convert reads every run first, on purpose: every row is checked before anything is written, and the 1.1 GB of shards are read once. Each run keeps only the signals of its first 30 steps and its compressed thoughts, and runs of one issue share one copy of its text, so peak memory stays at 1.9 GB. The Records' values still load lazily.
  • A known limit of the traceback rule: a file view that merely shows a traceback (after open or edit) counts as one: 69 of 9,375 tracebacks in shard 0. The README states it.

Source notes documented in the README

  • Licence conditions: the dataset page adds the Llama 3.1 License for the model outputs and the licences of the underlying repositories. The README states both.
  • No citation entry: the release has none, so the card cites the dataset page and the revision.
  • 13 instances appear with two slightly different issue texts, and 16 texts are shared by two instances of the same repository. Each (instance, text) pair gets its own annotation.
  • early_exit is not explained by the release; the connector keeps the name.

Files

packages/timenet-connectors/src/timenet_connectors/datasets/nebius/__init__.py
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/__init__.py
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/connector.py   # records, annotations, task stream
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/source.py      # parquet reading, step parsing and signals
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/tasks.py       # checkpoints, split, labels, status text
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/dataset.yaml
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/README.md
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/requirements.txt
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/tests/__init__.py
packages/timenet-connectors/src/timenet_connectors/datasets/nebius/swe_agent_trajectories/tests/test_connector.py

The tests write a synthetic shard, so the repo ships no dataset bytes.

How to load

import timenet

timenet.build("nebius/swe_agent_trajectories")  # or: timenet-build build nebius/swe_agent_trajectories
with timenet.TimeNet().open("nebius/swe_agent_trajectories") as reader:
    dataset = reader.read()
train = dataset.get_train()

Validation

Check Result
make check (ruff format, ruff check, ty) passed
make check-ci (all pre-commit hooks + ty on 3.11) passed
make test-unit 1655 passed
make test-connectors all connectors passed (pytest + ty), incl. nebius/swe_agent_trajectories
make license-check passed
Connector tests (synthetic shard: parsing rules, checkpoint rule, no outcome on records, thoughts kept verbatim, one issue annotation per instance, status text, repository split, storage round trip, 9 malformed-row cases, missing target, row count) 35 passed
timenet-build build nebius/swe_agent_trajectories from the Hub built 1.0.0 in ~5 min incl. the 1.1 GB download, 171 MB, peak RSS 1.9 GB

Read-back of the built release:

  • 192,428 records; every signal of every record has exactly k values; every visible step has a reply.
  • 577,284 tasks (192,428 per task type): train 464,748, validation 55,029, test 57,507. Every task names issue and agent_thoughts as inputs; every issue annotation names the task's own instance (3,599 annotations).
  • No repository (of 1,275 with a checkpoint) and no issue appears in two partitions.
Partition Repositories Runs Resolved at 5 / 10 / 20 / 30 steps
train 993 60,985 16.6 / 13.7 / 6.2 / 3.9 %
validation 139 7,557 16.8 / 14.6 / 6.8 / 4.4 %
test 143 7,640 16.4 / 14.1 / 7.1 / 5.9 %

Example checkpoint (run train-00000-of-00012:0, 5 steps):

  • action_kind → other, search, search, search, read; observation_kind → traceback, output, output, output, command_error
  • run_outcome → unresolved; run_ending → context_limit
  • answer → "In its first 5 steps the agent edited files 0 times, created 0 files, ran code 0 times, read or searched the repository 4 times and repeated an earlier command 0 times. The linter rejected 0 edits, 1 reply showed a Python traceback, 1 reply a command error and 0 commands timed out. The conversation holds 12,857 characters. The run will go on for 41 more steps, then it will reach the context limit and SWE-agent will submit its changes, and the agent will not resolve the issue."

Preliminary signal check (one shard, 6,670 runs, folds grouped by repository, simple models; not part of the connector): AUROC for "will it resolve" at 10 steps was 0.70 from the numeric signals and 0.74 from a bag-of-words of the thoughts, against 0.50 for chance.

Licence and citation

The data is CC BY 4.0 (dataset page). Using the model outputs also requires complying with the Llama 3.1 License, and the repositories keep their own licences. Cite:

Nebius. SWE-agent trajectories (nebius/SWE-agent-trajectories). Hugging Face, 2024. Revision 68195a1450865274106246d0d0296a1d6807b88e.

Limitations

  • One agent framework and one model family (93 % of runs from Llama 70B).
  • Signals and kinds are rule-made from the logged text. Status reports are templates.
  • The outcome is the release's test result.

Contributed by Team RAMadan during the TimeNet hackathon, Round 1 (November 2026).

Checklist

There is no PR template in the repo; this mirrors CONTRIBUTING.md and AGENTS.md.

  • Conventional Commits, Conventional Branch (feature/nebius-swe-agent-trajectories)
  • make check, make check-ci, make test-unit, make license-check pass
  • make test-connectors passes
  • Google-style docstrings; ty green
  • Source pinned by Hub revision and shard row counts; no raw data committed
  • Card validates against dataset-card-v1; licence, citation and source URL set
  • Source-data problems raise TimeNetConnectorError; nothing is filled

Note: we can't add labels on this repo. Could a maintainer add area:connectors? Thanks!

Parse each run of nebius/SWE-agent-trajectories into its outcome and the
per-step signals of its first 30 steps: thought, command and reply
lengths, conversation length, repeated commands, command kind and reply
kind. The build fails on an unknown exit status, a malformed instance id,
a trajectory out of turn, or a shard with another row count.
… checkpoint tasks

Cut each run after 5, 10, 20 and 30 steps while it is still going, so no
input shows how the run ends. Each checkpoint is one Record of ordinal
step signals with the agent's thoughts as text, the issue is a dataset
annotation, and three streamed tasks ask whether the run will resolve
its issue, how it will end, and for a rule-made status report. Runs are
split by repository.
@etienneRV etienneRV mentioned this pull request Nov 20, 2026
7 of 8 tasks
- Read the shards with the shared ParquetDataReader into a SourceRow model, and raise on a
  missing message text or a system message with text, instead of filling in empty strings.
- Keep thoughts exactly as written; thought_chars now marks where each one ends.
- Store one issue annotation per instance and text, only for instances with a checkpoint, so
  no task names another instance's issue.
- Keep the exit statuses in one table, rename the label sets, and reuse the Record metadata for
  the tasks.
- Ask the status report for the remaining steps, the ending and the outcome, and write its
  target in the same future tense.
- Give repeated_action a dimensionless unit like the other signals.
@etienneRV etienneRV changed the title [Internal review] Add the nebius/swe_agent_trajectories connector (agent early warning) feat(connectors): add the SWE-agent trajectories connector Nov 20, 2026
@etienneRV

Copy link
Copy Markdown
Owner Author

Closing: this is now open upstream as OpenTSLM#832. Please review and comment there.

@etienneRV etienneRV closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant