Skip to content

feat(connectors): add EngineMT-QA aero-engine question answering dataset - #801

Open
karhu wants to merge 2 commits into
mainfrom
feature/tnet-174-enginemt-qa-dataset-connector
Open

karhu wants to merge 2 commits into
mainfrom
feature/tnet-174-enginemt-qa-dataset-connector

Conversation

@karhu

@karhu karhu commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

EngineMT-QA (Wang et al., ITFormer, ICML 2025) is a question-answering benchmark over simulated aero-engine flights from NASA's N-CMAPSS. The authors publish it on the Hugging Face Hub. One HDF5 file holds 118,921 rows of 600 steps and 33 channels. Two JSON Lines files hold the question-answer pairs. TimeNet has no connector for it.

The release has properties that its card and paper do not state:

  • A row is a 10-minute window of a flight. The paper's "cycle" is a flight, as in N-CMAPSS. Consecutive ids continue each other. When a flight ends inside a row, the release fills the rest of the row with zeros.
  • A Reasoning or Decision-Making question names 10 rows. The authors' code gives the model only the first 60 steps of each row, joined into one 600-step array. The paper does not describe this cut.
  • The train file holds the paper's training set plus 4,039 lines copied from the test file. Without the copies, the train counts equal the paper's Fig. 8 exactly. The paper does not mention the copies.
  • 1,769 Perception answers are empty. The empty fault causes fall on rows whose faulty components (HPC, LPC) have no option among the six.

Solution

The connector pandalin98/enginemt-qa builds 114,312 tasks (72,335 train, 41,977 test), one per labelled pair, under the authors' file split. It leaves out the train pairs that repeat a test pair:

Stage Task type Input
Understanding AnswerTask row Record
Perception ClassificationTask row Record
Reasoning ClassificationTask joined Record
Decision-Making AnswerTask joined Record
  • A row Record holds one row on a 1 Hz axis. 5,945 rows have one. If the flight ended inside the row, the signals stop at the last real step. The metadata key padded_steps gives the number of zero steps that the connector left out.
  • A joined Record holds the first 60 steps of each of 10 rows, in published order, on an ordinal axis. 8,845 distinct lists have one. Padded steps inside a block are null, with the missing-value reason alignment_padding. In the pinned release, no block has padding.
  • Every closed question is a ClassificationTask over a label set of its option letters. The fault cause is multi-label.
  • Tasks carry source_file, source_line and qa_offset as the authors' evaluation names them.

Design decisions

  • Joined Records for multi-row questions. A task scope must resolve to one input Record, so TimeF cannot say "the first 60 steps of each of 10 inputs". A metadata key can say it, but generic loaders such as the torch adapter ignore task metadata. A joined Record gives any training loop the authors' input. The ordinal axis keeps it honest: the 10 blocks come from different flights, so the joined steps have no clock. The cost is about 1.4 GB of copied values. A separate proposal describes per-input scopes. With them, the copy is not necessary.
  • Padding is not stored as data. The values are standardized, so a zero looks like a real reading. A row Record stops at its last real step. A joined Record keeps 600 steps so that its 10 blocks stay aligned, and it marks padding as null. To get the authors' exact array back, a user pads zeros. Conversion reads only the last step of each named row to find the padded rows.
  • Empty answers are missing labels. The pair gets no task, and the annotation unlabelled_pairs lists each one. The authors' current metric raises on an empty reference.
  • Train pairs that repeat a test pair get no task. A flag would make every user learn about the copies to avoid training on test pairs. The connector compares the rows, the question and the answer of each labelled pair, and leaves out 27,961 train pairs: the labelled pairs of the 4,039 copied lines, and 6 pairs of a train-only line about a row that a test line also names. The annotation test_pairs_in_train lists each one. The test split keeps every labelled pair, so it stays comparable with the paper. The multi-row lines of both files still draw on the same 1,900 rows, in other combinations. The README describes this and the 651 lists that mix two engine units, by our reading of channel 32.
  • The prompt is the published question, unchanged. Channel names stay channel_00 to channel_32, because the authors' channel audit names none of them.
  • QA lines go through the validated JsonlDataReader. The domain rules (stage, rows, <ts> marker, closed questions, answer letters) and the pinned counts per file stay in the connector.
  • h5py is in the connector's requirements.txt and in the dev group, as wfdb is, so that ty resolves it.

Validation

  • Pass: make check, the pre-commit hooks on the changed files, and the connector's 42 tests and ty (Python 3.13 and 3.11) in the environment built from its requirements.txt.
  • Real QA files at the pinned digests: the parser accepts every line and finds the pinned counts.
  • Full build on the real release, at the pinned digests: 104 seconds after the download, 1.47 GB. It stored 14,790 Records (5,945 row Records, 8,845 joined Records) and 114,312 tasks (72,335 train, 41,977 test). No train task has the input, prompt and targets of a test task. 317 row Records end in padding. No joined Record has a null. A read-back found the same counts, and 70 row Records and 50 joined Records equal their HDF5 values.

This PR replaces #629.

@karhu karhu added the area:connectors Changes packages/timenet-connectors label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

✅ CI checks passed

Check Status
Quick checks ✅ Passed
Tests (Python 3.13) ✅ Passed
Tests (Python 3.11) ✅ Passed

Updated for 620d3ef · run #8676

The connector pandalin98/enginemt-qa builds one task per labelled
question-answer pair of the EngineMT-QA release on the Hugging Face
Hub. A single-row question reads the Record of its row. A multi-row
question reads one joined Record with the first 60 steps of each of its
10 rows on an ordinal axis. The authors' code gives its model the same
input.

When a flight ends inside a row, the release fills the rest of the row
with zeros. A row Record stops at its last real step. In a joined
Record, the padded steps are null.

The train file copies 4,039 lines of the test file. Tasks that the
other file repeats carry the source issue repeated_across_splits, so
that a user can rebuild the paper's split. A pair with an empty answer
gets no task. The annotation unlabelled_pairs lists each one.

Co-authored-by: Othman Sbai <212133622+osbai@users.noreply.github.com>
@karhu
karhu force-pushed the feature/tnet-174-enginemt-qa-dataset-connector branch from adbfb2b to d6a0d45 Compare November 20, 2026 20:11
… pair

The released train file holds 4,039 byte copies of test lines. The
paper does not mention them. Before, their tasks stayed in the train
split with the source issue repeated_across_splits, so every user had
to know about the flag to avoid training on test pairs.

Now a labelled train pair that has the same rows, question and answer
as a test pair gets no task. That leaves out 27,961 train pairs: the
labelled pairs of the copied lines, and 6 pairs of a train-only line
about a row that a test line also names. The annotation
test_pairs_in_train lists each one. The test split keeps every
labelled pair, and the train split keeps 72,335 tasks.

The tasks no longer have the source_issues metadata key. After the
change, no pair repeats another pair in its own split.

The README now describes the rows from a read of the whole file. A
flight is a median of 16 rows long, and channel 32 is constant within
a flight. Most multi-row lists take about one window from each of 10
consecutive flights. 651 lists mix two runs of channel 32, which are
likely two engine units.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:connectors Changes packages/timenet-connectors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant