Repository navigation
Conversation
karhu
force-pushed
the
feature/m4-caption-connector
branch
3 times, most recently
from
November 17, 2026 18:43
ef62ea1 to
e4d1992
Compare
karhu
marked this pull request as ready for review
November 17, 2026 23:30
karhu
force-pushed
the
feature/m4-caption-connector
branch
from
November 18, 2026 01:17
4153ed2 to
6a4148a
Compare
✅ CI checks passed
Updated for |
karhu
force-pushed
the
feature/m4-caption-connector
branch
from
November 18, 2026 01:23
6a4148a to
f954d90
Compare
RealLast
requested changes
Nov 18, 2026
RealLast
left a comment
Contributor
There was a problem hiding this comment.
Great direction! See a few points for simplification / clean up but otherwise strong
karhu
force-pushed
the
feature/m4-caption-connector
branch
from
November 20, 2026 17:30
e78b1a6 to
3175d5e
Compare
Add opentslm/m4-caption. It converts the M4-Caption release of OpenTSLM: the train part of the 100,000 M4 competition series, each with a caption that GPT-4o wrote. Each series becomes one Record and one AnswerTask without a prompt. The Record keeps the M4 id and the frequency. The values stay float32, because the release holds the M4 values rounded to float32. The series use an ordinal axis, because the release has no timestamps and a month has no fixed length. The release has no split. The OpenTSLM loader splits the series at load time with the datasets library. The connector computes the same 80/10/10 split with NumPy and stops with an error if the split does not match a pinned digest. generate_splits.py computed that digest with the datasets version from the OpenTSLM uv.lock. The M4 organizers state no license, so the card uses NONE. OpenTSLM releases the captions under MIT.
Add a README to the M4-Caption connector. It describes the records, the split, and how the release relates to the M4 data. It also describes future work: an M4 parent dataset that M4-Caption can import its Records from.
Call CsvDataReader.raise_field_size_limit before the connector reads a series file, in place of the removed max_field_chars option.
Call CsvDataReader.set_field_size_limit, the new name of raise_field_size_limit.
The connector computed the OpenTSLM split with NumPy and compared it with a digest from generate_splits.py. That meant two implementations of one split. generate_splits.py now runs the calls of the OpenTSLM loader once and writes the 10,000 validation and 10,000 test series, by M4 id, to splits.json.xz. The file is 19 KB, and the connector reads it with the lzma module of the standard library. This is the frozen-file pattern of sleep-cot. The file gives the same split as the digest did. The README describes the frozen split. It also notes future work on the axis type, if calendar axes (#555) land.
The series cell accepted values that are not numbers. Pydantic turned true into 1.0 and "3.5" into 3.5. An empty list passed the row check and failed later without the file or the row. The series cell now takes strict finite floats and at least one value. An M4Frequency enum now owns the six frequencies: their order, the id letter, and the two release file names. The connector sets the CSV cell limit once in convert, not once for each frequency. The tests check a Boolean value and an empty series, and no longer repeat the pydantic NaN check. The card and the README get small text fixes.
The card said "the release" for the zip archive that OpenTSLM publishes on Polybox. It now introduces that archive once and calls it the Polybox release.
The pinned Polybox release parses fully, so the script does not need to parse each series to mirror the rows that the OpenTSLM loader skips. The split file is unchanged.
Introduce the Polybox archive in one sentence, and drop the sentence about float32 values.
Shorten the M4Frequency docstring, the docstring of generate_splits.py and the M4-Caption card.
The pinned Polybox release holds only exact float32 values, so the connector converts each series to float32 without checking every value again.
The release now assigns the split while it reads the series, so each series carries its own split. One method creates the Record and its caption task together, so convert no longer passes the Record and the split back in.
Each AnswerTask now has the post-prompt of OpenTSLM's M4QADataset, the same text for every series. The card and the README state it. M4CaptionRelease is now M4CaptionReader, and the reader knows the file names of each frequency, not M4Frequency. read_series is easier to read, and the docstrings say that the release archive stores no split and that OpenTSLM's loader makes it.
The split of OpenTSLM's loader is random over all frequencies, so the small frequencies drift: its test split holds 6.5% of the hourly series. generate_splits.py now makes an 80/10/10 split with StratifiedSplitter, stratified by M4 frequency, and writes it to splits.json.xz as before. Validation and test each hold a tenth of every frequency. The dataset no longer has the split of the OpenTSLM paper. The script now runs in the workspace and reads the M4 ids with the new M4CaptionReader.read_m4_ids. Two runs write the same file.
…s in one split For 706 series, the release has an error message of the caption API instead of a caption, such as "Error generating caption: Connection error.". The connector now leaves out these series, so the dataset has 99,294 series. M4 has 90 pairs of series whose histories are equal value for value. The split put 34 pairs across two splits, so a model could train on the exact input of a test series. generate_splits.py now passes a digest of the values as group_by to StratifiedSplitter, so each pair stays in one split. The script reads the series with the new M4CaptionReader.read_captioned_series, so it also leaves out the error captions. Two runs write the same file.
karhu
force-pushed
the
feature/m4-caption-connector
branch
from
November 20, 2026 17:31
3175d5e to
dae9fb0
Compare
Main changed the connector API. A label set carries its name as an annotation, a Hugging Face connector passes its Hub settings as class keywords and reads a mapping from part to path, BaseConnector names the whole download result, and Annotation takes its value by keyword.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
opentslm/m4-caption. It converts the M4-Caption release of OpenTSLM: the train part of the 100,000 M4 competition series, each with a caption that GPT-4o wrote.Each series becomes one Record and one
AnswerTask. The task prompt is the prompt of the OpenTSLM loader (the post-prompt ofM4QADataset), the same for every series. The Record keeps the M4 id and the frequency. The values stay float32, because the release holds the M4 values rounded to float32. The series use an ordinal axis, because the release has no timestamps and a month has no fixed length.For 706 series, the release has an error message of the caption API instead of a caption, such as "Error generating caption: Connection error.". The connector leaves out these 253 daily, 57 monthly and 396 quarterly series, so the dataset has 99,294 series.
The release archive stores no split.
generate_splits.pymakes an 80/10/10 split withStratifiedSplitter, stratified by M4 frequency, and writes the validation and test series tosplits.json.xz. Validation and test each hold a tenth of every frequency. M4 has 90 pairs of series whose histories are equal value for value, so the splitter groups the series by their values, and each pair stays in one split. This is not the split of the OpenTSLM paper. Its loader makes a random split of all frequencies together, and its test split holds only 6.5% of the hourly series. The result of the splitter depends on the version of scikit-learn, so the script ran once. The connector reads the 19 KB file with thelzmamodule of the standard library.The M4 organizers state no license, so the card uses
NONE. OpenTSLM releases the captions under MIT.The connector folder has a README. It describes the records, the split, and how the release relates to the M4 data. It also describes future work: an M4 parent dataset, and calendar axes if #555 lands.
A full
timenet-build build opentslm/m4-captiongives 99,294 records with a 79,436, 9,929 and 9,929 split, and a tenth of each frequency in validation and test. The values and captions read back from the build match the release.