Skip to content

feat(connectors): add EngineMT-QA aero-engine question answering dataset - #629

Closed
osbai wants to merge 10 commits into
mainfrom
feature/enginemt-qa-connector
Closed

osbai wants to merge 10 commits into
mainfrom
feature/enginemt-qa-connector

Conversation

@osbai

@osbai osbai commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Adds pandalin98/enginemt-qa: question answering over simulated N-CMAPSS
aero-engine cycles in four tasks (Wang et al., ITFormer, ICML 2025), from
the authors' Hugging Face release pinned at ea2051a, the revision their
own channel audit checked. Built on the DatasetLoader and DatasetTask
bases and the flat record metadata of #593, in the stage layout RATs40K,
CAPTURE-24 and TRACE use: release, keys, questions, qa, signals, records,
tasks, loader, plus model_input.

The design follows the authors' code at ITFormer-ICML25 b80142f
(dataset/dataset.py, utils/metrics.py, inference.py) and the paper.

What the dataset holds

  • Records: one per HDF5 row of seq_data, with 33 lazy float64
    signals channel_00 to channel_32 on a 1 Hz axis, 600 values each.
    The release names no channel and the authors' audit says not to guess,
    so the names are indices. The values are z-scored, as published. The
    1 Hz rate is the paper's "10-minute (600 points)". Metadata holds the
    row_id that QAs name. By default the 7,755 rows a QA names;
    EngineMTQAConnector.ALL_ROWS = True builds all 118,921, which the
    authors' encoder pretraining reads.
  • Tasks: one per published QA pair, one kind per stage, under the
    authors' file split: Understanding (AnswerTask, 1 cycle),
    Perception (ClassificationTask, 1 cycle), Reasoning
    (ClassificationTask, 10 cycles), Decision-Making (AnswerTask,
    10 cycles). 142,273 tasks: 100,296 train, 41,977 test.
  • Closed questions: the nine closed questions each have one fixed
    text and option list, and the authors' model predicts letters that
    closed_question_metrics scores as a set. So targets are the
    published letters (the fault cause may hold several, such as
    ("a", "b", "e")), and one vocabulary annotation per option list maps
    each letter to its option.
  • Multi-cycle inputs: the 10 Records in published order. The authors'
    model reads the first 60 steps of each, joined; one scope can't cut 10
    inputs, so tasks state steps_per_cycle: 60 and
    model_input.authors_series rebuilds the authors' (600, 33) array.
  • Prompt and metadata: the published question, <ts> included. The
    task metadata holds the authors' source_line and qa_offset.

Data issues, kept visible

  • 1,769 Perception answers are empty (fault cause 1,054 train and 422
    test; health state 215 and 78). They are missing labels, not "no
    fault", so those pairs get no task, and their count is pinned per
    question and file.
  • 2,689 of the 3,384 test 10-id lists are also train lists with the same
    Reasoning answers, and 3,223 rows are in both files. The authors'
    split is kept and the README says so.
  • The paper says 32 channels and 11k or 110k pairs; the release has 33
    channels and 144,042 pairs. The README lists these against the paper.

Reading

  • check_layout checks the HDF5 holds exactly seq_data and data_ID,
    of the published shape and dtype, with ids 1 to 118,921.
  • Each QA line is validated strictly: exact keys, name repeating id,
    10 distinct ids per list, alternating turns, stage against format and
    row count, one <ts>, closed questions by exact text, answers as
    distinct letters of their options.
  • The build checks pinned counts per file (lines, pairs per stage, empty
    answers per question) and the 7,755 referenced rows.
  • A signal reads its row from the HDF5 file once for all 33 channels.

Adds h5py to the connector's requirements and to the dev group, as
wfdb and huggingface-hub are, so ty resolves it.

Validation

  • Pass: make check, make lint-fix, make check-ci (hooks and the
    Python 3.11 ty pass), make test (2146 passed, 1 skipped) and
    make test-connectors (every connector, the 26 enginemt-qa tests
    included), in a clean worktree of this commit.
  • Real release, at the pinned digests: a full build through
    run_pipeline stores 7,755 Records and 142,273 tasks (69,902 answer,
    72,371 classification) in 903 MB, in about a minute. Read back
    through TimeNet(...).load, 50 random Records equal their HDF5 rows
    and 200 random tasks' authors_series equal the authors' slicing.
  • ALL_ROWS = True: a full build stores 118,921 Records, ids 1 to
    118,921, and the same 142,273 tasks in 13 GB, in 12 minutes; 50
    random Records read back equal their HDF5 rows.
  • The pinned counts match the authors' README and audit: 101,565 train
    and 42,477 test pairs, per stage.

osbai added 6 commits October 29, 2026 13:18
Connectors that read several files or define several kinds of task
have each grown their own loader: TRACE has TraceDatasetLoader, and
RATs40K repeats the same shape. The [TRACE review](https://review.aioniclabs.dev/OpenTSLM/TimeNet/pull/577#diff-comment-PRRC_kwDOTR9sUs8AAAABBJn8LA) asked for a generic
class those loaders inherit, without duplicating what BaseConnector
or TimeFDataset already do. This adds two small bases for that, in
timenet_connectors.bases.loader.

- DatasetTask is one task as the source defines it, such as TRACE's
  retrieval task. It declares its TIMENET_TYPE and its split, and
  builds the TimeNet task on demand with to_timenet_task().
- DatasetLoader[TTask] parses the downloaded files once. Each dataset
  fills in parse_from(), records() and tasks(), plus
  dataset_annotations() if it needs any. The base provides
  tasks_of(kind, split=), timenet_tasks() and populate(dataset).

Where the boundaries are:

- The connector keeps the outside world: its card, its download, its
  pinned source, and the engine's download/convert interface. Its
  convert() becomes parse_from(...) followed by populate(...). The
  download bases (Hugging Face, PhysioNet, EDF, Excel) are unchanged.
- The loader keeps the downloaded files: parsing, Records and source
  tasks. It holds state, so it is its own object rather than more
  methods on a stateless connector.
- TimeNet core is untouched. TimeFDataset still holds, selects and
  stores the TimeNet tasks. The loader's tasks_of() selects source
  task kinds, which a TimeNet type can't tell apart when two kinds
  become the same type.

The base adds no runtime checks of its own. The type checker rejects
a TASK_KINDS entry that isn't a DatasetTask, and populate() declares
the kinds' TimeNet types, so the writer rejects a task of any other
type. populate() streams the tasks, declaring each TimeNet type once.

Shortcuts such as get_train_tasks(), pinned counts, key lookups and a
default convert() are left out. Each joins the base once two
connectors need it. TRACE and RATs40K move onto these bases in
follow-up PRs.

docs/connectors.md gains a "Loaders and source tasks" section. Two
tests cover a toy loader with three task kinds, two of which become
the same TimeNet type.

Validation: make check, make lint-fix, make check-ci and make test
(1606 passed, 1 skipped). make test-connectors passes for every
connector this commit tracks.
The review found the DatasetTask class an extra step. Each task of a
dataset needed its own class that held the Record and the parsed row,
gave a split, and then built the TimeNet task. Simple connectors got
more code, not less. This removes DatasetTask. tasks() now yields
TimeNet tasks, and TASK_TYPES lists their TimeNet types, as TASK_KINDS
listed the kinds before. timenet_tasks() goes, because tasks() already
gives TimeNet tasks.

tasks_of() goes too. When the dataset authors define two tasks of the
same TimeNet type, the loader gives each one its own method, named as
the authors name it, and tasks() yields from each method. Calling that
method selects the tasks of the authors. After populate(),
TimeFDataset.tasks_of and get_split select the tasks, so the base needs
no selection method and no TTask type variable. A stored name for the
task of the authors is left out until a connector needs it.

The module docstring and the docs section now use simple English. The
docs section is now "Converting with DatasetLoader" and matches the
style of the page. The tests use a toy loader with two tasks of the same
TimeNet type and a task without a split.

Validation: make lint-fix, make check, make check-ci and make test (2142
passed, 1 skipped).
TRACE and RATs40K each carried the same RecordMetadata prototype: a
small shared core (subject_id) plus typed per-dataset extensions, each
stored under its own name. This moves it to timenet_connectors.metadata
so both connectors use one copy.

It is still a prototype of the shared metadata proposed on TimeNet #433,
kept in the connectors package until the core design is agreed. Nothing
in it is specific to one dataset, so it can move to timenet.types
unchanged.

The API nav in zensical.toml is regenerated for the new module. That
also adds four entries main was already missing: timenet.json,
timenet_connectors.sources, timenet_connectors.splitting and
timenet_connectors.time_axes.

Validation: make check, make lint-fix, make check-ci and make test
(1607 passed, 1 skipped).
A dataset now declares its Record metadata as one subclass of
RecordMetadata that adds its own typed fields, and a Record holds one
such object. to_json writes it as one flat JSON object, and
XRecordMetadata.from_json reads it back as that class.

- Removes MetadataExtension, extend, extension() and the kinds argument.
- A subclass that redefines a shared field such as subject_id is
  rejected when it is defined.
- from_json raises TimeFValidationError on an unknown key or another
  version; core_of reads only the shared fields.

Validation: make check, make check-ci, make test (2146 passed, 1
skipped) and make test-connectors (every connector on this branch
passes).
Every connector reads its metadata with its own class's from_json, so
nothing called core_of.
@osbai
osbai force-pushed the feature/enginemt-qa-connector branch from 6d68323 to 935e507 Compare October 30, 2026 13:25
osbai added 3 commits October 30, 2026 14:41
RecordMetadata.to_json dumped with exclude_none=True, which drops every None
field, including fields of nested models. Metadata with a legitimate None then
fails to read back (a required nullable field is reported missing) or reads
back wrong (a None that overrides a non-None default comes back as the
default, without an error). to_json now writes every field, None as null, so
from_json returns the values that were stored.

A test covers a required nullable field, a None over a non-None default, and
a None inside a nested model. Capture24, the only connector upstack built on
RecordMetadata, always sets its fields, so its stored metadata only gains an
explicit null where a field is unset.
This adds pandalin98/enginemt-qa: question answering over simulated
N-CMAPSS aero-engine cycles in four tasks (Wang et al., ITFormer, ICML
2025). The data comes from the Hugging Face release of the authors,
pinned at ea2051a, the revision that their own channel audit checked.
The connector uses the flat record metadata of #593, and the same
module layout as RATs40K, CAPTURE-24 and TRACE: release, keys,
questions, qa, signals, records, tasks and loader, and also
model_input.

The design follows the code of the authors at ITFormer-ICML25 b80142f
(dataset/dataset.py, utils/metrics.py, inference.py) and the paper.

The dataset holds:
- Records: one for each HDF5 row of seq_data, with 33 lazy float64
  signals, channel_00 to channel_32, on a 1 Hz axis, 600 values each.
  The release names no channel, and the audit of the authors says not
  to guess, so the names are indices. The values are z-scored, as
  published. The 1 Hz rate is the "10-minute (600 points)" of the paper.
  Metadata holds the row_id that the QAs name. By default, the connector
  builds the 7,755 rows that a QA names. EngineMTQAConnector.ALL_ROWS =
  True builds all 118,921 rows, which the encoder pretraining of the
  authors reads.
- Tasks: one for each published QA pair, one for each stage, under the
  file split of the authors: Understanding (AnswerTask, 1 cycle),
  Perception (ClassificationTask, 1 cycle), Reasoning
  (ClassificationTask, 10 cycles) and Decision-Making (AnswerTask, 10
  cycles). There are 142,273 tasks: 100,296 train and 41,977 test.
- Closed questions: each of the nine closed questions has one fixed text
  and one option list. The model of the authors predicts letters, and
  closed_question_metrics scores them as a set. So the targets are the
  published letters. The fault cause can hold several, such as
  ("a", "b", "e"). One vocabulary annotation for each option list maps
  each letter to its option.
- Inputs with 10 cycles: the 10 Records, in published order. The model
  of the authors reads the first 60 steps of each, joined. One scope
  cannot cut 10 inputs, so these tasks state steps_per_cycle: 60, and
  model_input.authors_series rebuilds the (600, 33) array of the
  authors.
- Prompt and metadata: the published question, with its <ts> marker.
  The task metadata holds source_line and qa_offset, as the authors
  name them.

The data keeps these issues visible:
- 1,769 Perception answers are empty: fault cause 1,054 train and 422
  test, health state 215 train and 78 test. They are missing labels,
  not "no fault". So these pairs get no task, and their count is pinned
  for each question and file.
- 2,689 of the 3,384 test lists of 10 ids are also train lists, with
  the same Reasoning answers, and 3,223 rows are in both files. The
  connector keeps the split of the authors, and the README says so.
- The paper says 32 channels and 11k or 110k pairs. The release has 33
  channels and 144,042 pairs. The README lists these differences.

check_layout makes sure that the HDF5 file holds exactly seq_data and
data_ID, with the published shape and dtype, and the ids 1 to 118,921.
Each QA line is validated strictly: exact keys, name repeating id, 10
different ids in each list, alternating turns, stage against format and
row count, one <ts> marker, closed questions by their exact text, and
answers as different letters of their options. The build checks the
pinned counts of each file (lines, pairs for each stage, empty answers
for each question) and the 7,755 rows that a QA names. A signal reads
its row from the HDF5 file once, for all 33 channels.

This adds h5py to the requirements of the connector and to the dev
group, as wfdb and huggingface-hub are, so that ty resolves it.

On the real release, at the pinned digests, a full build through
run_pipeline stores 7,755 Records and 142,273 tasks (69,902 answer,
72,371 classification) in 903 MB, in about one minute. Read back
through TimeNet(...).load, 50 random Records equal their HDF5 rows, and
the authors_series of 200 random tasks equal the slicing of the
authors. With ALL_ROWS = True, a full build stores 118,921 Records, ids
1 to 118,921, and the same 142,273 tasks in 13 GB, in 12 minutes. 50
random Records read back equal to their HDF5 rows. The pinned counts
match the README and the audit of the authors: 101,565 train and 42,477
test pairs, for each stage.

This commit builds its tasks on the DatasetTask classes of the first
version of DatasetLoader. The next commit moves them to the current
DatasetLoader, and gives the validation for both commits.
DatasetLoader no longer has DatasetTask. A loader now yields TimeNet
tasks, and gives each task of the authors its own method. This moves
EngineMT-QA to that design. Two stages make AnswerTasks and two make
ClassificationTasks, so the methods are what keeps the stages apart.

- tasks.py: the classes UnderstandingTask, PerceptionTask,
  ReasoningTask and DecisionTask become the functions open_task and
  closed_task. stage_task picks one of them with Stage.is_closed. The
  content of the tasks does not change.
- loader.py: EngineMTQADatasetLoader gets the methods
  understanding_tasks, perception_tasks, reasoning_tasks and
  decision_tasks. tasks() keeps the file order, in which the stages
  alternate, so the stream and the task ids do not change. TASK_TYPES
  lists AnswerTask and ClassificationTask.
- tests: the test that selected tasks by kind and split now uses the
  four methods. It also makes sure that tasks() holds exactly the tasks
  of the four methods.
- README.md: the task table names the four loader methods.

Validation: make lint-fix, make check, make check-ci and make test (2144
passed, 1 skipped). make test-connectors passes for each connector in
this commit, with the 26 EngineMT-QA tests. It also reports failures for
folders that hold only __pycache__ from other branches. The full build
of the real release was not run again.
@osbai
osbai force-pushed the feature/enginemt-qa-connector branch from 935e507 to 93cb7e3 Compare October 30, 2026 14:32
@osbai
osbai force-pushed the pr593 branch 2 times, most recently from 3d2354f to fa22d7b Compare November 1, 2026 00:12
…main

Replace the EngineMT-QA connector of this PR with the single-file
version, built on main without the DatasetLoader stack. It flags the
repeated question-answer pairs of the release, and the README and card
describe the release as it is.

This commit merges main at 4818e37, so the PR now targets main.
@karhu

karhu commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Superseded by #801, which includes this work rebased on main (with osbai as co-author). It gives the multi-row questions a joined input Record and leaves out the train pairs that repeat a test pair. Closing this one.

@karhu karhu closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:connectors Changes packages/timenet-connectors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants