Repository navigation
Conversation
Connectors that read several files or define several kinds of task have each grown their own loader: TRACE has TraceDatasetLoader, and RATs40K repeats the same shape. The [TRACE review](https://review.aioniclabs.dev/OpenTSLM/TimeNet/pull/577#diff-comment-PRRC_kwDOTR9sUs8AAAABBJn8LA) asked for a generic class those loaders inherit, without duplicating what BaseConnector or TimeFDataset already do. This adds two small bases for that, in timenet_connectors.bases.loader. - DatasetTask is one task as the source defines it, such as TRACE's retrieval task. It declares its TIMENET_TYPE and its split, and builds the TimeNet task on demand with to_timenet_task(). - DatasetLoader[TTask] parses the downloaded files once. Each dataset fills in parse_from(), records() and tasks(), plus dataset_annotations() if it needs any. The base provides tasks_of(kind, split=), timenet_tasks() and populate(dataset). Where the boundaries are: - The connector keeps the outside world: its card, its download, its pinned source, and the engine's download/convert interface. Its convert() becomes parse_from(...) followed by populate(...). The download bases (Hugging Face, PhysioNet, EDF, Excel) are unchanged. - The loader keeps the downloaded files: parsing, Records and source tasks. It holds state, so it is its own object rather than more methods on a stateless connector. - TimeNet core is untouched. TimeFDataset still holds, selects and stores the TimeNet tasks. The loader's tasks_of() selects source task kinds, which a TimeNet type can't tell apart when two kinds become the same type. The base adds no runtime checks of its own. The type checker rejects a TASK_KINDS entry that isn't a DatasetTask, and populate() declares the kinds' TimeNet types, so the writer rejects a task of any other type. populate() streams the tasks, declaring each TimeNet type once. Shortcuts such as get_train_tasks(), pinned counts, key lookups and a default convert() are left out. Each joins the base once two connectors need it. TRACE and RATs40K move onto these bases in follow-up PRs. docs/connectors.md gains a "Loaders and source tasks" section. Two tests cover a toy loader with three task kinds, two of which become the same TimeNet type. Validation: make check, make lint-fix, make check-ci and make test (1606 passed, 1 skipped). make test-connectors passes for every connector this commit tracks.
The review found the DatasetTask class an extra step. Each task of a dataset needed its own class that held the Record and the parsed row, gave a split, and then built the TimeNet task. Simple connectors got more code, not less. This removes DatasetTask. tasks() now yields TimeNet tasks, and TASK_TYPES lists their TimeNet types, as TASK_KINDS listed the kinds before. timenet_tasks() goes, because tasks() already gives TimeNet tasks. tasks_of() goes too. When the dataset authors define two tasks of the same TimeNet type, the loader gives each one its own method, named as the authors name it, and tasks() yields from each method. Calling that method selects the tasks of the authors. After populate(), TimeFDataset.tasks_of and get_split select the tasks, so the base needs no selection method and no TTask type variable. A stored name for the task of the authors is left out until a connector needs it. The module docstring and the docs section now use simple English. The docs section is now "Converting with DatasetLoader" and matches the style of the page. The tests use a toy loader with two tasks of the same TimeNet type and a task without a split. Validation: make lint-fix, make check, make check-ci and make test (2142 passed, 1 skipped).
TRACE and RATs40K each carried the same RecordMetadata prototype: a small shared core (subject_id) plus typed per-dataset extensions, each stored under its own name. This moves it to timenet_connectors.metadata so both connectors use one copy. It is still a prototype of the shared metadata proposed on TimeNet #433, kept in the connectors package until the core design is agreed. Nothing in it is specific to one dataset, so it can move to timenet.types unchanged. The API nav in zensical.toml is regenerated for the new module. That also adds four entries main was already missing: timenet.json, timenet_connectors.sources, timenet_connectors.splitting and timenet_connectors.time_axes. Validation: make check, make lint-fix, make check-ci and make test (1607 passed, 1 skipped).
A dataset now declares its Record metadata as one subclass of RecordMetadata that adds its own typed fields, and a Record holds one such object. to_json writes it as one flat JSON object, and XRecordMetadata.from_json reads it back as that class. - Removes MetadataExtension, extend, extension() and the kinds argument. - A subclass that redefines a shared field such as subject_id is rejected when it is defined. - from_json raises TimeFValidationError on an unknown key or another version; core_of reads only the shared fields. Validation: make check, make check-ci, make test (2146 passed, 1 skipped) and make test-connectors (every connector on this branch passes).
Every connector reads its metadata with its own class's from_json, so nothing called core_of.
osbai
force-pushed
the
feature/enginemt-qa-connector
branch
from
October 30, 2026 13:25
6d68323 to
935e507
Compare
RecordMetadata.to_json dumped with exclude_none=True, which drops every None field, including fields of nested models. Metadata with a legitimate None then fails to read back (a required nullable field is reported missing) or reads back wrong (a None that overrides a non-None default comes back as the default, without an error). to_json now writes every field, None as null, so from_json returns the values that were stored. A test covers a required nullable field, a None over a non-None default, and a None inside a nested model. Capture24, the only connector upstack built on RecordMetadata, always sets its fields, so its stored metadata only gains an explicit null where a field is unset.
This adds pandalin98/enginemt-qa: question answering over simulated N-CMAPSS aero-engine cycles in four tasks (Wang et al., ITFormer, ICML 2025). The data comes from the Hugging Face release of the authors, pinned at ea2051a, the revision that their own channel audit checked. The connector uses the flat record metadata of #593, and the same module layout as RATs40K, CAPTURE-24 and TRACE: release, keys, questions, qa, signals, records, tasks and loader, and also model_input. The design follows the code of the authors at ITFormer-ICML25 b80142f (dataset/dataset.py, utils/metrics.py, inference.py) and the paper. The dataset holds: - Records: one for each HDF5 row of seq_data, with 33 lazy float64 signals, channel_00 to channel_32, on a 1 Hz axis, 600 values each. The release names no channel, and the audit of the authors says not to guess, so the names are indices. The values are z-scored, as published. The 1 Hz rate is the "10-minute (600 points)" of the paper. Metadata holds the row_id that the QAs name. By default, the connector builds the 7,755 rows that a QA names. EngineMTQAConnector.ALL_ROWS = True builds all 118,921 rows, which the encoder pretraining of the authors reads. - Tasks: one for each published QA pair, one for each stage, under the file split of the authors: Understanding (AnswerTask, 1 cycle), Perception (ClassificationTask, 1 cycle), Reasoning (ClassificationTask, 10 cycles) and Decision-Making (AnswerTask, 10 cycles). There are 142,273 tasks: 100,296 train and 41,977 test. - Closed questions: each of the nine closed questions has one fixed text and one option list. The model of the authors predicts letters, and closed_question_metrics scores them as a set. So the targets are the published letters. The fault cause can hold several, such as ("a", "b", "e"). One vocabulary annotation for each option list maps each letter to its option. - Inputs with 10 cycles: the 10 Records, in published order. The model of the authors reads the first 60 steps of each, joined. One scope cannot cut 10 inputs, so these tasks state steps_per_cycle: 60, and model_input.authors_series rebuilds the (600, 33) array of the authors. - Prompt and metadata: the published question, with its <ts> marker. The task metadata holds source_line and qa_offset, as the authors name them. The data keeps these issues visible: - 1,769 Perception answers are empty: fault cause 1,054 train and 422 test, health state 215 train and 78 test. They are missing labels, not "no fault". So these pairs get no task, and their count is pinned for each question and file. - 2,689 of the 3,384 test lists of 10 ids are also train lists, with the same Reasoning answers, and 3,223 rows are in both files. The connector keeps the split of the authors, and the README says so. - The paper says 32 channels and 11k or 110k pairs. The release has 33 channels and 144,042 pairs. The README lists these differences. check_layout makes sure that the HDF5 file holds exactly seq_data and data_ID, with the published shape and dtype, and the ids 1 to 118,921. Each QA line is validated strictly: exact keys, name repeating id, 10 different ids in each list, alternating turns, stage against format and row count, one <ts> marker, closed questions by their exact text, and answers as different letters of their options. The build checks the pinned counts of each file (lines, pairs for each stage, empty answers for each question) and the 7,755 rows that a QA names. A signal reads its row from the HDF5 file once, for all 33 channels. This adds h5py to the requirements of the connector and to the dev group, as wfdb and huggingface-hub are, so that ty resolves it. On the real release, at the pinned digests, a full build through run_pipeline stores 7,755 Records and 142,273 tasks (69,902 answer, 72,371 classification) in 903 MB, in about one minute. Read back through TimeNet(...).load, 50 random Records equal their HDF5 rows, and the authors_series of 200 random tasks equal the slicing of the authors. With ALL_ROWS = True, a full build stores 118,921 Records, ids 1 to 118,921, and the same 142,273 tasks in 13 GB, in 12 minutes. 50 random Records read back equal to their HDF5 rows. The pinned counts match the README and the audit of the authors: 101,565 train and 42,477 test pairs, for each stage. This commit builds its tasks on the DatasetTask classes of the first version of DatasetLoader. The next commit moves them to the current DatasetLoader, and gives the validation for both commits.
DatasetLoader no longer has DatasetTask. A loader now yields TimeNet tasks, and gives each task of the authors its own method. This moves EngineMT-QA to that design. Two stages make AnswerTasks and two make ClassificationTasks, so the methods are what keeps the stages apart. - tasks.py: the classes UnderstandingTask, PerceptionTask, ReasoningTask and DecisionTask become the functions open_task and closed_task. stage_task picks one of them with Stage.is_closed. The content of the tasks does not change. - loader.py: EngineMTQADatasetLoader gets the methods understanding_tasks, perception_tasks, reasoning_tasks and decision_tasks. tasks() keeps the file order, in which the stages alternate, so the stream and the task ids do not change. TASK_TYPES lists AnswerTask and ClassificationTask. - tests: the test that selected tasks by kind and split now uses the four methods. It also makes sure that tasks() holds exactly the tasks of the four methods. - README.md: the task table names the four loader methods. Validation: make lint-fix, make check, make check-ci and make test (2144 passed, 1 skipped). make test-connectors passes for each connector in this commit, with the 26 EngineMT-QA tests. It also reports failures for folders that hold only __pycache__ from other branches. The full build of the real release was not run again.
osbai
force-pushed
the
feature/enginemt-qa-connector
branch
from
October 30, 2026 14:32
935e507 to
93cb7e3
Compare
osbai
force-pushed
the
pr593
branch
2 times, most recently
from
November 1, 2026 00:12
3d2354f to
fa22d7b
Compare
…main Replace the EngineMT-QA connector of this PR with the single-file version, built on main without the DatasetLoader stack. It flags the repeated question-answer pairs of the release, and the README and card describe the release as it is. This commit merges main at 4818e37, so the PR now targets main.
Contributor
|
Superseded by #801, which includes this work rebased on main (with osbai as co-author). It gives the multi-row questions a joined input Record and leaves out the train pairs that repeat a test pair. Closing this one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
pandalin98/enginemt-qa: question answering over simulated N-CMAPSSaero-engine cycles in four tasks (Wang et al., ITFormer, ICML 2025), from
the authors' Hugging Face release pinned at
ea2051a, the revision theirown channel audit checked. Built on the
DatasetLoaderandDatasetTaskbases and the flat record metadata of #593, in the stage layout RATs40K,
CAPTURE-24 and TRACE use: release, keys, questions, qa, signals, records,
tasks, loader, plus model_input.
The design follows the authors' code at ITFormer-ICML25
b80142f(
dataset/dataset.py,utils/metrics.py,inference.py) and the paper.What the dataset holds
seq_data, with 33 lazy float64signals
channel_00tochannel_32on a 1 Hz axis, 600 values each.The release names no channel and the authors' audit says not to guess,
so the names are indices. The values are z-scored, as published. The
1 Hz rate is the paper's "10-minute (600 points)". Metadata holds the
row_idthat QAs name. By default the 7,755 rows a QA names;EngineMTQAConnector.ALL_ROWS = Truebuilds all 118,921, which theauthors' encoder pretraining reads.
authors' file split: Understanding (
AnswerTask, 1 cycle),Perception (
ClassificationTask, 1 cycle), Reasoning(
ClassificationTask, 10 cycles), Decision-Making (AnswerTask,10 cycles). 142,273 tasks: 100,296 train, 41,977 test.
text and option list, and the authors' model predicts letters that
closed_question_metricsscores as a set. So targets are thepublished letters (the fault cause may hold several, such as
("a", "b", "e")), and one vocabulary annotation per option list mapseach letter to its option.
model reads the first 60 steps of each, joined; one scope can't cut 10
inputs, so tasks state
steps_per_cycle: 60andmodel_input.authors_seriesrebuilds the authors' (600, 33) array.<ts>included. Thetask metadata holds the authors'
source_lineandqa_offset.Data issues, kept visible
test; health state 215 and 78). They are missing labels, not "no
fault", so those pairs get no task, and their count is pinned per
question and file.
Reasoning answers, and 3,223 rows are in both files. The authors'
split is kept and the README says so.
channels and 144,042 pairs. The README lists these against the paper.
Reading
check_layoutchecks the HDF5 holds exactlyseq_dataanddata_ID,of the published shape and dtype, with ids 1 to 118,921.
namerepeatingid,10 distinct ids per list, alternating turns, stage against format and
row count, one
<ts>, closed questions by exact text, answers asdistinct letters of their options.
answers per question) and the 7,755 referenced rows.
Adds
h5pyto the connector's requirements and to the dev group, aswfdbandhuggingface-hubare, sotyresolves it.Validation
make check,make lint-fix,make check-ci(hooks and thePython 3.11
typass),make test(2146 passed, 1 skipped) andmake test-connectors(every connector, the 26 enginemt-qa testsincluded), in a clean worktree of this commit.
run_pipelinestores 7,755 Records and 142,273 tasks (69,902 answer,72,371 classification) in 903 MB, in about a minute. Read back
through
TimeNet(...).load, 50 random Records equal their HDF5 rowsand 200 random tasks'
authors_seriesequal the authors' slicing.ALL_ROWS = True: a full build stores 118,921 Records, ids 1 to118,921, and the same 142,273 tasks in 13 GB, in 12 minutes; 50
random Records read back equal their HDF5 rows.
and 42,477 test pairs, per stage.