Repository navigation
feat(connectors): add the OpenTSLM M4-Caption connector #790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
bb07f88
feat(connectors): add the OpenTSLM M4-Caption connector
karhu b589063
docs(connectors): describe the M4-Caption connector and a parent dataset
karhu 6217eec
refactor(connectors): raise the CSV cell limit before M4-Caption reads
karhu c6f426d
refactor(connectors): use the renamed CSV cell limit method
karhu 7bea0f0
refactor(connectors): freeze the M4-Caption split in splits.json.xz
karhu 4b69228
fix(connectors): reject non-numeric M4-Caption values
karhu 368899e
docs(connectors): call the M4-Caption source the Polybox release
karhu 5616a3f
refactor(connectors): drop the parse check from generate_splits.py
karhu 7e3be54
docs(connectors): shorten the M4-Caption card
karhu 5531bb5
docs(connectors): shorten M4-Caption docstrings and card
karhu 12ac217
refactor(connectors): drop the float32 check of M4-Caption values
karhu 0a591e4
refactor(connectors): give each M4-Caption series its split
karhu c60d82f
docs(connectors): say why the M4-Caption task has no prompt
karhu b9f8804
refactor(connectors): lower the M4-Caption cell limit to 200,000 char…
karhu 775a372
docs(connectors): tag M4-Caption with demographics
karhu 3c2e707
refactor(connectors): let the CSV reader handle long M4-Caption cells
karhu dffd82e
feat(connectors): give M4-Caption tasks the prompt of OpenTSLM's loader
karhu 4d928c8
feat(connectors): stratify the M4-Caption split by frequency
karhu dae9fb0
feat(connectors): leave out M4-Caption error captions, keep duplicate…
karhu 8bcd9de
Merge remote-tracking branch 'origin/main' into HEAD
masquare 1bdf0cf
refactor(connectors): move opentslm/m4_caption to the new connector API
masquare File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
63 changes: 63 additions & 0 deletions
63
...imenet-connectors/src/timenet_connectors/datasets/opentslm/m4_caption/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # M4-Caption | ||
|
|
||
| M4-Caption pairs each of the 100,000 series of the M4 forecasting competition with one caption that | ||
| GPT-4o wrote. OpenTSLM publishes it as a zip archive of CSV files, one series file and one caption | ||
| file for each M4 frequency. | ||
|
|
||
| For 706 series, the release has an error message of the caption API instead of a caption. There are | ||
| three messages, each after the prefix "Error generating caption: ": "Connection error." (630 | ||
| series), "Error code: 404" (75) and "Request timed out." (1). The connector leaves out these 253 | ||
| daily, 57 monthly and 396 quarterly series, so the dataset has 99,294 series. | ||
|
|
||
| The connector converts each series into one Record and one `AnswerTask`. The Record holds the | ||
| series as one float32 Signal on an ordinal axis, the M4 id in its metadata, and the M4 frequency as | ||
| an annotation. The task target is the caption. The task prompt is the prompt of the OpenTSLM loader | ||
| (the post-prompt of `M4QADataset`), the same for every series. | ||
|
|
||
| ## Split | ||
|
|
||
| The release archive stores no split. `generate_splits.py` makes an 80/10/10 split with | ||
| `StratifiedSplitter`, stratified by M4 frequency. Validation and test each hold a tenth of every | ||
| frequency, so the small hourly and weekly frequencies keep their share in each split. The result | ||
| depends on the version of scikit-learn, so the script ran once. It wrote the 9,929 validation and | ||
| 9,929 test series, by M4 id, to `splits.json.xz`. The connector reads that file and puts every | ||
| other series in the train split. | ||
|
|
||
| M4 has 90 pairs of series whose histories are equal value for value: 19 monthly, 35 quarterly and | ||
| 36 yearly pairs. The two series of a pair have different M4 ids and different M4 test horizons. All | ||
| pairs but one also have different captions. A model that trains on one series of a pair sees the | ||
| exact input of the other. So the script groups the series by their values with the `group_by` | ||
| option of `StratifiedSplitter`, and each pair stays in one split. | ||
|
|
||
| This split is not the split of the OpenTSLM paper. The OpenTSLM loader makes a random split of all | ||
| frequencies together, so its share of the small frequencies drifts. For example, its test split | ||
| holds 6.5% of the hourly series. | ||
|
|
||
| The file is JSON compressed with xz, about 19 KB. Python reads it with the `lzma` module of the | ||
| standard library. To read it yourself, run `xz -dc splits.json.xz`. The file also states the SHA-256 | ||
| of the release archive. If it does not match the pinned archive, the build stops with an error. | ||
|
|
||
| ## Relation to the M4 data | ||
|
|
||
| Each series of the release is the train part of the M4 series with the same id, from the | ||
| `Dataset/Train` files of the M4-methods repository. The release does not hold the test horizon or | ||
| the series information of `M4-info.csv`. Each value is the float32 rounding of the M4 value. The | ||
| largest relative difference is 6e-8. | ||
|
|
||
| ## Future work: an M4 parent dataset | ||
|
|
||
| TimeNet can model this relation with a parent dataset. A connector for M4, for example | ||
| `mcompetitions/m4`, can convert the `Dataset/` folder of M4-methods: the train and test files and | ||
| `M4-info.csv`. M4-Caption can then declare that release under `parents` and import its Records. | ||
| The caption task then names the train part of each series with a `StepInterval` scope. | ||
|
|
||
| With a parent, the 24 million values exist once, and a reader gets the M4 category, horizon and | ||
| test part. The values become the float64 M4 values, not the float32 values that OpenTSLM plotted | ||
| for GPT-4o. | ||
|
|
||
| ## Future work: change axis types | ||
|
|
||
| We currently use an ordinal axis for consistency, as some of the frequencies are only regular in | ||
| calendar terms, e.g. "monthly". If | ||
| [Proposal: calendar axes, value cells and civil-date origins](https://github.com/OpenTSLM/TimeNet/issues/555) | ||
| lands we can reconsider this choice. |
6 changes: 6 additions & 0 deletions
6
packages/timenet-connectors/src/timenet_connectors/datasets/opentslm/m4_caption/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| """M4-Caption: the M4 competition series, each with a caption that GPT-4o wrote.""" | ||
|
|
||
| from timenet_connectors.datasets.opentslm.m4_caption.connector import ( | ||
| CONNECTOR as CONNECTOR, | ||
| M4CaptionConnector as M4CaptionConnector, | ||
| ) |
323 changes: 323 additions & 0 deletions
323
packages/timenet-connectors/src/timenet_connectors/datasets/opentslm/m4_caption/connector.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,323 @@ | ||
| """The M4-Caption connector: M4 competition series, each with a caption that GPT-4o wrote. | ||
|
|
||
| The OpenTSLM release holds the train part of all 100,000 M4 series, one CSV of series and one CSV of | ||
| captions for each of the six M4 frequencies. Each series becomes one Record and one AnswerTask. The | ||
| target is the caption. The connector leaves out the 706 series whose caption is an error message. The release archive stores no split. generate_splits.py made an 80/10/10 split, | ||
| stratified by frequency, and wrote it to splits.json.xz. The connector reads that file. | ||
| """ | ||
|
|
||
| from collections import Counter | ||
| from collections.abc import Iterator, Sequence | ||
| from dataclasses import dataclass | ||
| from enum import StrEnum | ||
| import lzma | ||
| from pathlib import Path | ||
| import re | ||
| from typing import Annotated, Self | ||
|
|
||
| import numpy as np | ||
| from pydantic import AllowInfNan, BaseModel, ConfigDict, Field, Json, StrictFloat, ValidationError, model_validator | ||
|
|
||
| from timenet.connectors import BaseConnector | ||
| from timenet.dataset import Record, Signal, Source, TimeFDataset | ||
| from timenet.dataset.axis import OrdinalAxis | ||
| from timenet.errors import TimeNetConnectorError | ||
| from timenet.types import Annotation, AnswerTask, Split, TimeSeriesSpec | ||
| from timenet_connectors.download import ensure_archive | ||
| from timenet_connectors.sources.readers import CsvDataReader, DataRow | ||
|
|
||
|
|
||
| M4_CAPTION_URL = "https://polybox.ethz.ch/index.php/s/MT3y9WdEebT8wfj/download/M4TimeSeriesCaptionDatasetV02.zip" | ||
| M4_CAPTION_SHA256 = "4acb748d94d7287fe58dbb5c3cf6314ea259010b7720f42506739269dec54edf" | ||
| RELEASE_FOLDER = "M4TimeSeriesCaptionDataset" | ||
| """The folder in the archive that holds the CSV files.""" | ||
| SPLITS_PATH = Path(__file__).with_name("splits.json.xz") | ||
| """The stratified split of the release, as generate_splits.py wrote it.""" | ||
|
|
||
| CAPTION_PROMPT = "Please generate a detailed caption for this time-series, describing it as accurately as possible." | ||
| """The prompt of OpenTSLM's loader, the same for every series: the post-prompt of ``M4QADataset``.""" | ||
|
|
||
| CAPTION_ERROR_PREFIX = "Error generating caption: " | ||
| """The start of an error message of the caption API, which the release has instead of a caption for | ||
| 706 series. The dataset leaves out these series.""" | ||
|
|
||
| # The release states no unit, and the M4 series come from many domains. | ||
| M4_SERIES = TimeSeriesSpec(spec_type="m4_series", name="M4 series", unit_value=None, dtype="float32") | ||
|
|
||
| _SERIES_ID = re.compile(r"series-(?P<m4_id>(?P<letter>[A-Z])[1-9][0-9]*)") | ||
| """A release id: the M4 id, such as M1, after a ``series-`` prefix that OpenTSLM added.""" | ||
|
|
||
| _SeriesValue = Annotated[StrictFloat, AllowInfNan(False)] | ||
| """A finite JSON number. Strict, so that a Boolean or a string in the list is an error, not a number.""" | ||
|
|
||
|
|
||
| class M4Frequency(StrEnum): | ||
|
karhu marked this conversation as resolved.
Resolved
|
||
| """An M4 frequency. The members are in the order that OpenTSLM's loader reads them.""" | ||
|
|
||
| DAILY = "Daily" | ||
| HOURLY = "Hourly" | ||
| MONTHLY = "Monthly" | ||
| QUARTERLY = "Quarterly" | ||
| WEEKLY = "Weekly" | ||
| YEARLY = "Yearly" | ||
|
|
||
| @property | ||
| def id_letter(self) -> str: | ||
| """The letter that starts the M4 id of each series of this frequency, such as M in M1.""" | ||
| return _M4_ID_LETTERS[self] | ||
|
|
||
| def parse_m4_id(self, series_id: str) -> str: | ||
| """Get the M4 id from a release id, and check that it belongs to this frequency. | ||
|
|
||
| Returns: | ||
| The M4 id, such as ``M1``. | ||
|
|
||
| Raises: | ||
| TimeNetConnectorError: If the id is not an M4 series id of this frequency. | ||
| """ | ||
| match = _SERIES_ID.fullmatch(series_id) | ||
| if match is None or match["letter"] != self.id_letter: | ||
| raise TimeNetConnectorError(f"{series_id!r} is not a {self} M4 series id") | ||
| return match["m4_id"] | ||
|
|
||
|
|
||
| _M4_ID_LETTERS = { | ||
| M4Frequency.DAILY: "D", | ||
| M4Frequency.HOURLY: "H", | ||
| M4Frequency.MONTHLY: "M", | ||
| M4Frequency.QUARTERLY: "Q", | ||
| M4Frequency.WEEKLY: "W", | ||
| M4Frequency.YEARLY: "Y", | ||
| } | ||
|
|
||
|
|
||
| class _SeriesRow(DataRow): | ||
| """A row of a series file: the release id and the values as a JSON list.""" | ||
|
|
||
| id: str | ||
| series: Json[Annotated[list[_SeriesValue], Field(min_length=1)]] | ||
|
|
||
|
|
||
| class _CaptionRow(DataRow): | ||
| """A row of a caption file: the release id and the caption.""" | ||
|
|
||
| id: str | ||
| caption: str | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class M4CaptionedSeries: | ||
| """One M4 series of the release, with its frequency and caption, before the split is known.""" | ||
|
|
||
| m4_id: str | ||
| frequency: M4Frequency | ||
| values: np.ndarray | ||
| caption: str | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class M4CaptionSeries: | ||
| """One M4 series of the release, with its frequency, caption and split.""" | ||
|
|
||
| m4_id: str | ||
| frequency: M4Frequency | ||
| values: np.ndarray | ||
| caption: str | ||
| split: Split | ||
|
|
||
| def create_record_and_caption_task(self) -> tuple[Record, AnswerTask]: | ||
| """Create the Record of the series and the task that asks for its caption. | ||
|
|
||
| The Record holds the values, the M4 id and the frequency. The task names the Record as its | ||
| input. Add the Record to the dataset before the task. | ||
|
|
||
| Returns: | ||
| The Record and the task, not yet added to a dataset. | ||
| """ | ||
| signal = Signal.from_values(self.values, spec=M4_SERIES, name="value", time_axis=OrdinalAxis()) | ||
| record = Record( | ||
| sources=[Source(name="M4", signals=[signal])], | ||
| annotations=[Annotation("frequency", value=self.frequency.value)], | ||
| metadata={"m4_id": self.m4_id}, | ||
| ) | ||
| task = AnswerTask(inputs=[record], prompt=CAPTION_PROMPT, targets=self.caption, split=self.split) | ||
| return record, task | ||
|
|
||
|
|
||
| class M4CaptionSplits(BaseModel): | ||
| """The validation and test series of the split, by M4 id. | ||
|
|
||
| The release archive stores no split. generate_splits.py made an 80/10/10 split, stratified by | ||
| frequency, and wrote it to splits.json.xz. Every series that the file does not name is in the train | ||
| split. | ||
| """ | ||
|
|
||
| model_config = ConfigDict(extra="forbid", frozen=True) | ||
|
|
||
| source_sha256: str | ||
| """SHA-256 digest of the release archive that the split was made from.""" | ||
| validation: tuple[str, ...] | ||
| test: tuple[str, ...] | ||
|
|
||
| @model_validator(mode="after") | ||
| def _each_series_in_one_split(self) -> Self: | ||
| repeated = sorted(m4_id for m4_id, count in Counter(self.m4_ids).items() if count > 1) | ||
| if repeated: | ||
| raise ValueError(f"series {repeated} are named more than once") | ||
| return self | ||
|
|
||
| @property | ||
| def m4_ids(self) -> tuple[str, ...]: | ||
| """The validation and test series together.""" | ||
| return (*self.validation, *self.test) | ||
|
|
||
| @classmethod | ||
| def read_from_file(cls, path: Path) -> Self: | ||
| """Read the split from an xz-compressed JSON file. | ||
|
|
||
| Returns: | ||
| The split. | ||
|
|
||
| Raises: | ||
| TimeNetConnectorError: If the file does not hold a valid split. | ||
| """ | ||
| try: | ||
| return cls.model_validate_json(lzma.decompress(path.read_bytes())) | ||
| except (lzma.LZMAError, ValidationError) as exc: | ||
| raise TimeNetConnectorError(f"{path.name} does not hold a valid split: {exc}") from exc | ||
|
|
||
| def assign_split_to_each_series(self, m4_ids: Sequence[str]) -> list[Split]: | ||
| """Assign a split to each series of the pinned release. | ||
|
|
||
| Returns: | ||
| The split of each series, in the order of ``m4_ids``. | ||
|
|
||
| Raises: | ||
| TimeNetConnectorError: If the split was made from another release, or names a series that | ||
| the release does not have. | ||
| """ | ||
| if self.source_sha256 != M4_CAPTION_SHA256: | ||
| raise TimeNetConnectorError( | ||
| f"the split was made from the release {self.source_sha256}, not from the pinned {M4_CAPTION_SHA256}" | ||
| ) | ||
| unknown = sorted(set(self.m4_ids) - set(m4_ids)) | ||
| if unknown: | ||
| raise TimeNetConnectorError(f"the split names series {unknown} that the release does not have") | ||
| split_of = dict.fromkeys(self.validation, Split.VALIDATION) | dict.fromkeys(self.test, Split.TEST) | ||
| return [split_of.get(m4_id, Split.TRAIN) for m4_id in m4_ids] | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class M4CaptionReader: | ||
| """Read the series, the captions and the split of the extracted release.""" | ||
|
|
||
| directory: Path | ||
| splits_path: Path | ||
|
|
||
| def read_series(self) -> list[M4CaptionSeries]: | ||
| """Read every series of the release with its caption and split, in the order of OpenTSLM's loader. | ||
|
|
||
| The loader reads the files of each frequency, and skips ``m4_series.csv`` and ``m4_captions.csv``. | ||
| A malformed file raises a :class:`~timenet.errors.TimeNetConnectorError` that names the file and | ||
| its row. The split comes from the split file, which the method checks against every M4 id. | ||
|
|
||
| Returns: | ||
| Each series with its caption and split. | ||
| """ | ||
| captioned = self.read_captioned_series() | ||
| split_file = M4CaptionSplits.read_from_file(self.splits_path) | ||
| splits = split_file.assign_split_to_each_series([series.m4_id for series in captioned]) | ||
| return [ | ||
|
karhu marked this conversation as resolved.
Resolved
|
||
| M4CaptionSeries( | ||
| m4_id=series.m4_id, | ||
| frequency=series.frequency, | ||
| values=series.values, | ||
| caption=series.caption, | ||
| split=split, | ||
| ) | ||
| for series, split in zip(captioned, splits, strict=True) | ||
| ] | ||
|
|
||
| def read_captioned_series(self) -> list[M4CaptionedSeries]: | ||
| """Read every series of the release with its caption, in the order of :meth:`read_series`. | ||
|
|
||
| The method leaves out each series whose caption is an error message of the caption API. | ||
|
|
||
| Returns: | ||
| Each series with its caption, without the split. | ||
| """ | ||
| captioned: list[M4CaptionedSeries] = [] | ||
| for frequency in M4Frequency: | ||
| captioned.extend(self._read_frequency(frequency)) | ||
| return captioned | ||
|
|
||
| def _read_frequency(self, frequency: M4Frequency) -> Iterator[M4CaptionedSeries]: | ||
| """Read the series of one frequency, paired with the caption in the same row of its caption file. | ||
|
|
||
| Yields: | ||
| Each series with its caption, except a series whose caption is an error message. | ||
|
|
||
| Raises: | ||
| TimeNetConnectorError: If the files do not pair one to one, or a series row is malformed. | ||
| """ | ||
| series_path = self.directory / f"m4_series_{frequency}.csv" | ||
| captions_path = self.directory / f"m4_captions_{frequency}.csv" | ||
| series_rows = list(CsvDataReader(series_path, _SeriesRow, expected_columns=("id", "series")).iter_rows()) | ||
| caption_rows = list(CsvDataReader(captions_path, _CaptionRow, expected_columns=("id", "caption")).iter_rows()) | ||
| if len(series_rows) != len(caption_rows): | ||
| raise TimeNetConnectorError( | ||
| f"{series_path.name} has {len(series_rows)} series rows but {len(caption_rows)} caption rows" | ||
| ) | ||
| for number, (series_row, caption_row) in enumerate(zip(series_rows, caption_rows, strict=True), start=1): | ||
| try: | ||
| m4_id = frequency.parse_m4_id(series_row.id) | ||
| if series_row.id != caption_row.id: | ||
| raise TimeNetConnectorError( | ||
| f"series id {series_row.id!r} does not match caption id {caption_row.id!r}" | ||
| ) | ||
| if caption_row.caption.startswith(CAPTION_ERROR_PREFIX): | ||
| continue | ||
| yield M4CaptionedSeries( | ||
| m4_id=m4_id, | ||
| frequency=frequency, | ||
| values=np.array(series_row.series, dtype=np.float32), | ||
| caption=caption_row.caption, | ||
| ) | ||
| except TimeNetConnectorError as exc: | ||
| raise TimeNetConnectorError(f"{series_path.name} data row {number}: {exc}") from exc | ||
|
|
||
|
|
||
| class M4CaptionConnector(BaseConnector[list[M4CaptionReader]]): | ||
| """Connector for the M4-Caption release of OpenTSLM.""" | ||
|
|
||
| async def download_async(self, cache_dir: Path) -> list[M4CaptionReader]: # ruff: ignore[no-self-use] | ||
| """Download and extract the release archive once. | ||
|
|
||
| Args: | ||
| cache_dir: The directory that holds the extracted files. | ||
|
|
||
| Returns: | ||
| A reader of the extracted release and of the split file of the connector. | ||
| """ | ||
| root = await ensure_archive(M4_CAPTION_URL, cache_dir, filename="m4_caption.zip", sha256=M4_CAPTION_SHA256) | ||
| return [M4CaptionReader(directory=root / RELEASE_FOLDER, splits_path=SPLITS_PATH)] | ||
|
|
||
| def convert(self, releases: list[M4CaptionReader]) -> TimeFDataset: | ||
| """Create one Record and one AnswerTask for each series. | ||
|
|
||
| Args: | ||
| releases: The reader from :meth:`download_async`. | ||
|
|
||
| Returns: | ||
| The populated dataset. | ||
| """ | ||
| (release,) = releases | ||
| dataset = TimeFDataset(metadata=self.metadata()) | ||
| for series in release.read_series(): | ||
| record, task = series.create_record_and_caption_task() | ||
| dataset.add_record(record) | ||
| dataset.add_task(task) | ||
| return dataset | ||
|
|
||
|
|
||
| CONNECTOR = M4CaptionConnector | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.