Repository navigation
Conversation
Parse each run of nebius/SWE-agent-trajectories into its outcome and the per-step signals of its first 30 steps: thought, command and reply lengths, conversation length, repeated commands, command kind and reply kind. The build fails on an unknown exit status, a malformed instance id, a trajectory out of turn, or a shard with another row count.
… checkpoint tasks Cut each run after 5, 10, 20 and 30 steps while it is still going, so no input shows how the run ends. Each checkpoint is one Record of ordinal step signals with the agent's thoughts as text, the issue is a dataset annotation, and three streamed tasks ask whether the run will resolve its issue, how it will end, and for a rule-made status report. Runs are split by repository.
7 of 8 tasks
- Read the shards with the shared ParquetDataReader into a SourceRow model, and raise on a missing message text or a system message with text, instead of filling in empty strings. - Keep thoughts exactly as written; thought_chars now marks where each one ends. - Store one issue annotation per instance and text, only for instances with a checkpoint, so no task names another instance's issue. - Keep the exit statuses in one table, rename the label sets, and reuse the Record metadata for the tasks. - Ask the status report for the remaining steps, the ending and the outcome, and write its target in the same future tense. - Give repeated_action a dimensionless unit like the other signals.
…nd the status report
Owner
Author
|
Closing: this is now open upstream as OpenTSLM#832. Please review and comment there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
nebius/swe_agent_trajectories, a connector for nebius/SWE-agent-trajectories (CC BY 4.0): 80,036 runs of the SWE-agent coding agent trying to resolve real GitHub issues, each labelled with whether its final patch passed the issue's tests.Why it matters: coding agents run for many steps and most runs fail (83 % here). A monitor that reads an agent's steps as they happen, its "ECG", could warn early that a run is going wrong. This dataset lets a time-series language model learn that from per-step signals and the agent's own reasoning.
The connector downloads the 12 parquet shards (1.1 GB) at a pinned revision and checks each shard's row count.
Mapping
Checkpoints, never whole runs. The end of a run gives its outcome away: every
submittedrun ends withsubmit, and 36 % of failed runs stop at the context limit. So each run is cut after 5, 10, 20 and 30 steps, but only if it goes on after them. No input holds the outcome, the exit status, the run's length, the patch or the test log.Record = one checkpoint (192,428). It holds one source,
agent_steps, with seven signals on anOrdinalAxis. The steps have an order but the release has no timestamps.thought_chars,action_chars,observation_charscontext_charsrepeated_actionaction_kindobservation_kindText inputs:
agent_thoughts, one string per step. It's aLazyTextList, not astrsignal, becauseTimeFTorchDatasethas no tensor form for string series.instance_idin its metadata.Both are named in each task's
input_annotations.Three tasks per checkpoint (577,284):
ClassificationTaskrun_outcomeresolved/unresolvedClassificationTaskrun_endingAnswerTaskThe kinds, signals and status texts come from fixed rules, which the README lists verbatim. No model writes them.
Decisions
StratifiedSplitter(8/1/1, stratified on the outcome, seed 42). No repository or issue is in two partitions, which matters because issues have 22 runs on average and 807 issues have both outcomes.generated_patch,eval_logs,mask,cutoff_date.ParquetDataReaderinto a pydanticSourceRow. A missing text, an unknown exit status, a malformed instance id or messages out of turn raiseTimeNetConnectorError. Thoughts are kept exactly as written.convertreads every run first, on purpose: every row is checked before anything is written, and the 1.1 GB of shards are read once. Each run keeps only the signals of its first 30 steps and its compressed thoughts, and runs of one issue share one copy of its text, so peak memory stays at 1.9 GB. The Records' values still load lazily.tracebackrule: a file view that merely shows a traceback (afteropenoredit) counts as one: 69 of 9,375 tracebacks in shard 0. The README states it.Source notes documented in the README
early_exitis not explained by the release; the connector keeps the name.Files
The tests write a synthetic shard, so the repo ships no dataset bytes.
How to load
Validation
make check(ruff format, ruff check, ty)make check-ci(all pre-commit hooks + ty on 3.11)make test-unitmake test-connectorsnebius/swe_agent_trajectoriesmake license-checktimenet-build build nebius/swe_agent_trajectoriesfrom the HubRead-back of the built release:
issueandagent_thoughtsas inputs; every issue annotation names the task's own instance (3,599 annotations).Example checkpoint (run
train-00000-of-00012:0, 5 steps):action_kind→other, search, search, search, read;observation_kind→traceback, output, output, output, command_errorrun_outcome→unresolved;run_ending→context_limitPreliminary signal check (one shard, 6,670 runs, folds grouped by repository, simple models; not part of the connector): AUROC for "will it resolve" at 10 steps was 0.70 from the numeric signals and 0.74 from a bag-of-words of the thoughts, against 0.50 for chance.
Licence and citation
The data is CC BY 4.0 (dataset page). Using the model outputs also requires complying with the Llama 3.1 License, and the repositories keep their own licences. Cite:
Limitations
Contributed by Team RAMadan during the TimeNet hackathon, Round 1 (November 2026).
Checklist
There is no PR template in the repo; this mirrors
CONTRIBUTING.mdandAGENTS.md.feature/nebius-swe-agent-trajectories)make check,make check-ci,make test-unit,make license-checkpassmake test-connectorspassestygreendataset-card-v1; licence, citation and source URL setTimeNetConnectorError; nothing is filledNote: we can't add labels on this repo. Could a maintainer add
area:connectors? Thanks!