Repository navigation
Conversation
Parse each run of nebius/SWE-agent-trajectories into its outcome and the per-step signals of its first 30 steps: thought, command and reply lengths, conversation length, repeated commands, command kind and reply kind. The build fails on an unknown exit status, a malformed instance id, a trajectory out of turn, or a shard with another row count.
… checkpoint tasks Cut each run after 5, 10, 20 and 30 steps while it is still going, so no input shows how the run ends. Each checkpoint is one Record of ordinal step signals with the agent's thoughts as text, the issue is a dataset annotation, and three streamed tasks ask whether the run will resolve its issue, how it will end, and for a rule-made status report. Runs are split by repository.
- Read the shards with the shared ParquetDataReader into a SourceRow model, and raise on a missing message text or a system message with text, instead of filling in empty strings. - Keep thoughts exactly as written; thought_chars now marks where each one ends. - Store one issue annotation per instance and text, only for instances with a checkpoint, so no task names another instance's issue. - Keep the exit statuses in one table, rename the label sets, and reuse the Record metadata for the tasks. - Ask the status report for the remaining steps, the ending and the outcome, and write its target in the same future tense. - Give repeated_action a dimensionless unit like the other signals.
…nd the status report
7 tasks done
…onnector API Main changed the connector API. A label set carries its name as an annotation, a Hugging Face connector passes its Hub settings as class keywords and reads a mapping from part to path, BaseConnector names the whole download result, and Annotation takes its value by keyword.
etienneRV
added a commit
to etienneRV/agent-ecg
that referenced
this pull request
Nov 20, 2026
…update OpenTSLM/TimeNet#832 now carries main's new connector API (b8385e7d). Tests and lint pass against it.
…to nebius/swe_agent_trajectories An edit command names no file: it changes the file open before it, which SWE-agent's "(Open file: ...)" footer names. The connector now follows that footer and the files the agent creates, and adds two per-step signals: - same_file_edit: the edit changes the same file as the run's edit before it; - edit_target: none / source / test / own_script (a file the agent created in this run). A test file is recognised by its folders and name after the repository's folder, so a repository called pytest does not make all its files tests. The status report gains one fact sentence, and a checkpoint at 15 steps (the middle of a typical run) joins 5, 10, 20 and 30. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
…checkpoint of nebius/swe_agent_trajectories Measured on a full build: 235,978 records and 707,934 tasks; the 15-step checkpoint holds 43,550 runs (9.2 % resolved). The README explains how an edit's file is followed through SWE-agent's "(Open file: ...)" footer, gives the edit counts by kind of file, and lists three channels that were tried and left out because they did not improve a signals model under a repository split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WLm9hY7SVui7tLGa6Fsxr
… card The signal-selection results belong with the models (etienneRV/agent-ecg), not with the dataset.
Author
|
Thanks for porting this to the new API, @masquare! FYI: we've since added two small things on top, a couple of signals that track which file each edit touches and a checkpoint at 15 steps. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
nebius/swe_agent_trajectories, a connector for nebius/SWE-agent-trajectories (CC BY 4.0): 80,036 runs of the SWE-agent coding agent trying to resolve real GitHub issues, each labelled with whether its final patch passed the issue's tests.Why it matters: coding agents run for many steps and most runs fail (83 % here). A monitor that reads an agent's steps as they happen, its "ECG", could warn early that a run is going wrong. This dataset lets a time-series language model learn that from per-step signals and the agent's own reasoning.
The connector downloads the 12 parquet shards (1.1 GB) at a pinned revision and checks each shard's row count.
Mapping
Checkpoints, never whole runs. The end of a run gives its outcome away: every
submittedrun ends withsubmit, and 36 % of failed runs stop at the context limit. So each run is cut after 5, 10, 15, 20 and 30 steps, but only if it goes on after them. No input holds the outcome, the exit status, the run's length, the patch or the test log.Record = one checkpoint (235,978). It holds one source,
agent_steps, with nine signals on anOrdinalAxis. The steps have an order but the release has no timestamps.thought_chars,action_chars,observation_charscontext_charsrepeated_actionaction_kindobservation_kindsame_file_editedit_targetAn
editcommand names no file, so the connector follows the(Open file: <path>)footer that ends every SWE-agent reply: an edit changes the file the reply before it names. Both file signals use only that step's command and earlier replies.Text inputs:
agent_thoughts, one string per step. It's aLazyTextList, not astrsignal, becauseTimeFTorchDatasethas no tensor form for string series.instance_idin its metadata.Both are named in each task's
input_annotations.Three tasks per checkpoint (707,934):
ClassificationTaskrun_outcomeresolved/unresolvedClassificationTaskrun_endingAnswerTaskThe kinds, signals and status texts come from fixed rules, which the README lists verbatim. No model writes them.
Decisions
StratifiedSplitter(8/1/1, stratified on the outcome, seed 42). No repository or issue is in two partitions, which matters because issues have 22 runs on average and 807 issues have both outcomes.generated_patch,eval_logs,mask,cutoff_date.ParquetDataReaderinto a pydanticSourceRow. A missing text, an unknown exit status, a malformed instance id or messages out of turn raiseTimeNetConnectorError. Thoughts are kept exactly as written.convertreads every run first, on purpose: every row is checked before anything is written, and the 1.1 GB of shards are read once. Each run keeps only the signals of its first 30 steps and its compressed thoughts, and runs of one issue share one copy of its text, so memory stays bounded (peak RSS measured below). The Records' values still load lazily.tracebackrule: a file view that merely shows a traceback (afteropenoredit) counts as one: 69 of 9,375 tracebacks in shard 0. The README states it.Source notes documented in the README
early_exitis not explained by the release; the connector keeps the name.Files
The tests write a synthetic shard, so the repo ships no dataset bytes.
How to load
Validation
make check(ruff format, ruff check, ty)make check-ci(all pre-commit hooks + ty on 3.11)make testmake test-connectorsnebius/swe_agent_trajectoriesmake license-checktimenet-build build nebius/swe_agent_trajectoriesfrom the Hub--no-isolation): 3 min 41 s, peak 4.7 GBRead-back of the built release:
issueandagent_thoughtsas inputs; every issue annotation names the task's own instance (3,599 annotations).Example checkpoint (run
train-00000-of-00012:0, 5 steps):action_kind→other, search, search, search, read;observation_kind→traceback, output, output, output, command_errorrun_outcome→unresolved;run_ending→context_limitSignal check (not part of the connector; reproducible with etienneRV/agent-ecg,
agent-ecg baselines): simple CPU models trained on this release's train split and scored on its test split (143 held-out repositories) reach AUROC 0.68 / 0.70 / 0.71 / 0.70 / 0.70 for "will it resolve" at 5 / 10 / 15 / 20 / 30 steps from the agent's thoughts, 0.61-0.73 from the step signals, and 0.61-0.62 from the issue text alone (chance 0.50).Licence and citation
The data is CC BY 4.0 (dataset page). Using the model outputs also requires complying with the Llama 3.1 License, and the repositories keep their own licences. Cite:
Limitations
Contributed by Team RAMadan during the TimeNet hackathon, Round 1 (November 2026).
Checklist
There is no PR template in the repo; this mirrors
CONTRIBUTING.mdandAGENTS.md.feature/nebius-swe-agent-trajectories)make check,make check-ci,make test-unit,make license-checkpassmake test-connectorspassestygreendataset-card-v1; licence, citation and source URL setTimeNetConnectorError; nothing is filledNote: we can't add labels on this repo. Could a maintainer add
area:connectors? Thanks!