Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/add-dataset-connector/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ latter). For PhysioNet, inspect the archive layout and a WFDB header (`fs`, `sig
### 3. Pick the base connector
- Hub dataset: subclass `BaseHuggingFaceConnector`, pass the class keywords `hf_repo`,
`source_revision` (a commit), and `source_files`, inherit `download`, and implement `convert`
over the `HubFileRef` list. The base fills `source_url` and `source_revision` in the dataset metadata from `HF_REPO` and `SOURCE_REVISION`.
over the part-to-path mapping. The base fills `source_url` and `source_revision` in the dataset metadata from `HF_REPO` and `SOURCE_REVISION`.
- PhysioNet/WFDB: subclass `BasePhysioNetConnector`, implement `download` (use `ensure_archive` /
`download_files` from `timenet_connectors.download`) and `convert` (use `_read_header` / `_lead_loader`).
- Neither: subclass `BaseConnector[TRaw]` and implement `download` + `convert` yourself.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ this branch. Anchor files:
## The `BaseConnector` contract

`BaseConnector(ABC, Generic[TRaw])` in `timenet.connectors`. `TRaw` is the whole value that `download` hands to
`convert`: one handle dataclass, a list of file refs (`list[HubFileRef]` for Hub connectors), or `None`.
`convert`: one handle dataclass, a list of file refs, a part-to-path mapping (Hub connectors), or `None`.

- `download(self, cache_dir: Path) -> TRaw` (abstract): fetch/discover raw source files, return
a lightweight handle. I/O only, no parsing, idempotent for a given `cache_dir`.
Expand Down Expand Up @@ -71,7 +71,7 @@ tags:
## Base connectors to reuse

### `BaseHuggingFaceConnector` (`bases/huggingface.py`)
`BaseHuggingFaceConnector(BaseConnector[list[HubFileRef]])` downloads files from a Hub dataset at a pinned
`BaseHuggingFaceConnector(BaseConnector[Mapping[str, Path]])` downloads files from a Hub dataset at a pinned
commit. Pass these class keywords:

- `hf_repo`: the Hub repo id.
Expand All @@ -87,10 +87,10 @@ and `get_card()` for a release that is not built, read the card file without the
card sets one of the fields to a different value, the build fails. Do not copy `HF_REPO` or
`SOURCE_REVISION` into dataset annotations.

`download` is inherited and returns one `HubFileRef(path, part)` per file. Declare
`download` is inherited. It returns the local path of each part, in `SOURCE_FILES` order. Declare
`huggingface_hub>=1.1` in the connector's `requirements.txt`, and set `HF_TOKEN` for gated or
private datasets. The base checks the refs before your `convert` runs: they must be nonempty and
name each declared part at most once. Do not call `check_refs` yourself.
private datasets. Implement `convert(self, files: Mapping[str, Path])` and read a file as
`files["train"]`.

To read the Hub's parquet conversion instead of the original files, pin the commit that
`refs/convert/parquet` points to and read the files with `ParquetDataReader` and a connector-owned
Expand Down Expand Up @@ -184,11 +184,13 @@ the bounds of every `Span` the task carries.

```python
import json
from collections.abc import Mapping
from pathlib import Path

from pydantic import ConfigDict, Field
from timenet.dataset import OrdinalAxis, Record, Signal, Source, TimeFDataset
from timenet.types import Annotation, AnswerTask, TimeSeriesSpec, ureg
from timenet_connectors.bases.huggingface import BaseHuggingFaceConnector, HubFileRef
from timenet_connectors.bases.huggingface import BaseHuggingFaceConnector
from timenet_connectors.sources.readers import DataRow, ParquetDataReader


Expand Down Expand Up @@ -217,18 +219,18 @@ class TSQAConnector(
):
"""Connector for the TSQA time-series QA dataset (Hub repo ``ChengsenWang/TSQA``)."""

def convert(self, raw_refs: list[HubFileRef]) -> TimeFDataset:
def convert(self, files: Mapping[str, Path]) -> TimeFDataset:
"""Build one record per row: the parsed series plus its question-and-answer task.

Args:
raw_refs: The parquet files from :meth:`download`.
files: The parquet file of each part.

Returns:
The populated :class:`~timenet.dataset.TimeFDataset`.
"""
dataset = TimeFDataset(metadata=self.metadata())
for ref in raw_refs:
for row in ParquetDataReader(ref.path, TSQARow).iter_rows():
for path in files.values():
for row in ParquetDataReader(path, TSQARow).iter_rows():
series = json.loads(row.series)
signals = series if series and isinstance(series[0], list) else [series]
time_series = [
Expand Down
11 changes: 8 additions & 3 deletions docs/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,10 @@ falls back to throttled text lines. The `--quiet` flag silences all output.
PhysioNet connectors extend `BasePhysioNetConnector` for WFDB record I/O. They use `ensure_archive` to
pull their database archive.

Hugging Face Hub connectors extend `BaseHuggingFaceConnector`. The base downloads the files in
`SOURCE_FILES`. Its `download` returns a mapping from each part name to its local path. The connector
implements `convert(self, files)` and reads a file as `files["train"]`.

## Example connectors

- `timenet/hello-world` is a synthetic, offline reference connector. It needs no network and produces
Expand Down Expand Up @@ -407,7 +411,8 @@ See the [API reference for `timenet.connectors`](api/timenet.connectors.md) for
## Reading source rows

For CSV, JSON arrays, JSON Lines, and Parquet, declare the source fields in a connector-owned
`DataRow` and read each downloaded file during `convert`. This example reads a list of file references:
`DataRow` and read each downloaded file during `convert`. This example is from a Hub connector. Its
`files` maps each part name to its local path:

```python
from pydantic import Field
Expand All @@ -417,8 +422,8 @@ class SourceRow(DataRow):
question: str = Field(alias="Question")
count: int = Field(alias="Count")

for ref in raw_refs:
for row in ParquetDataReader(ref.path, SourceRow).iter_rows():
for path in files.values():
for row in ParquetDataReader(path, SourceRow).iter_rows():
# Convert row.question and row.count into TimeNet objects.
...
```
Expand Down
Original file line number Diff line number Diff line change
@@ -1,50 +1,36 @@
"""A reusable base for connectors that read files from a pinned HuggingFace Hub dataset revision.

Subclasses pass ``hf_repo``, ``source_revision``, and ``source_files`` as class keywords, ship a
``dataset.yaml`` card beside the connector, and implement ``convert()`` over the :class:`HubFileRef` list ``download()``
returns. The base checks that list before the subclass's ``convert()`` runs. It also fills
``source_url`` and ``source_revision`` in the dataset metadata from ``HF_REPO`` and ``SOURCE_REVISION``.
``dataset.yaml`` card beside the connector, and implement ``convert()``. ``download()`` returns a mapping
from each part in ``SOURCE_FILES`` to its local path. ``convert()`` receives that mapping. The base also
fills ``source_url`` and ``source_revision`` in the dataset metadata from ``HF_REPO`` and ``SOURCE_REVISION``.
:mod:`timenet_connectors.sources.huggingface_hub` describes how the files download. For the Hub's
parquet conversion, pin the commit ``refs/convert/parquet`` points to.
"""

from abc import ABC
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
import functools
from collections.abc import Mapping
from pathlib import Path
import re
from types import MappingProxyType
from typing import Any, ClassVar

from timenet.connectors import BaseConnector
from timenet.dataset import TimeFDataset
from timenet.errors import TimeFValidationError, TimeNetConnectorError, TimeNetDownloadError
from timenet.errors import TimeFValidationError, TimeNetDownloadError
from timenet.types import DatasetMetadata
from timenet_connectors.sources.huggingface_hub import hub_snapshot


_COMMIT_SHA = re.compile(r"[0-9a-f]{40}")


@dataclass(frozen=True)
class HubFileRef:
"""A downloaded file (or a pattern's directory) and the part it belongs to."""

path: Path
part: str
"""The connector's own name for this file, such as ``"hhar_v2"``: the key in ``SOURCE_FILES``, which
``convert()`` branches on and builds ids from."""


class BaseHuggingFaceConnector(BaseConnector[list[HubFileRef]], ABC):
class BaseHuggingFaceConnector(BaseConnector[Mapping[str, Path]], ABC):
"""Base class for connectors that read files from a pinned Hub dataset revision.

Subclasses pass ``hf_repo``, ``source_revision``, and ``source_files`` as class keywords, and
implement ``convert()`` over the returned :class:`HubFileRef` list. The base stores the keywords as
:attr:`HF_REPO`, :attr:`SOURCE_REVISION`, and :attr:`SOURCE_FILES`. A subclass that passes no
keywords keeps the values of its parent. The base wraps each subclass's ``convert()`` to run
:meth:`check_refs` first.
implement :meth:`convert`, which receives the local path of each part. The base stores the keywords
as :attr:`HF_REPO`, :attr:`SOURCE_REVISION`, and :attr:`SOURCE_FILES`. A subclass that passes no
keywords keeps the values of its parent.
"""

HF_REPO: ClassVar[str]
Expand Down Expand Up @@ -78,16 +64,6 @@ def __init_subclass__(
)
if not getattr(cls, "SOURCE_FILES", None):
raise TimeFValidationError(f"{cls.__name__} source_files must declare at least one file")
# Every subclass's convert() checks its refs first, so no connector has to call check_refs itself
convert = cls.__dict__.get("convert")
if convert is not None:

@functools.wraps(convert)
def checked(self: BaseHuggingFaceConnector, raw_refs: list[HubFileRef]) -> TimeFDataset:
self.check_refs(raw_refs)
return convert(self, raw_refs)

cls.convert = checked

def metadata(self) -> DatasetMetadata:
"""Return the card, with ``source_url`` and ``source_revision`` from :attr:`HF_REPO` and :attr:`SOURCE_REVISION`.
Expand All @@ -113,59 +89,24 @@ def metadata(self) -> DatasetMetadata:
)
return card.model_copy(update=source)

def check_refs(self, raw_refs: Sequence[HubFileRef]) -> None:
"""Check that ``raw_refs`` is nonempty and names each part once, only parts :attr:`SOURCE_FILES` declares.

Args:
raw_refs: The references ``convert()`` received.

Raises:
TimeFValidationError: If ``raw_refs`` is empty, repeats a part, or names an undeclared part.
"""
parts = [ref.part for ref in raw_refs]
if not parts or len(set(parts)) != len(parts):
raise TimeFValidationError(f"{type(self).__name__} requires nonempty, distinct part references")
unknown = sorted(set(parts) - self.SOURCE_FILES.keys())
if unknown:
raise TimeFValidationError(f"{type(self).__name__} has no source part {', '.join(unknown)}")

def ref_paths(self, raw_refs: Sequence[HubFileRef], *parts: str) -> dict[str, Path]:
"""Map each part in ``raw_refs`` to its local path.

Args:
raw_refs: The references ``convert()`` received.
*parts: The parts the caller needs.

Returns:
The path of each part: a file, or the directory of a pattern part.

Raises:
TimeNetConnectorError: If one of ``parts`` is not in ``raw_refs``.
"""
paths = {ref.part: ref.path for ref in raw_refs}
missing = [part for part in parts if part not in paths]
if missing:
raise TimeNetConnectorError(f"{self.HF_REPO} refs lack {', '.join(missing)}")
return paths

async def download_async(self, cache_dir: Path) -> list[HubFileRef]:
async def download_async(self, cache_dir: Path) -> dict[str, Path]:
"""Download :attr:`SOURCE_FILES` at :attr:`SOURCE_REVISION` concurrently.

Args:
cache_dir: Directory where the connector caches Hub files.

Returns:
One reference per part, in :attr:`SOURCE_FILES` order.
The local path of each part, in :attr:`SOURCE_FILES` order: a file, or the directory of a pattern part.

Raises:
ImportError: If ``huggingface_hub``, declared in the connector's requirements, is missing.
TimeNetDownloadError: If a declared file is absent from the revision or cannot be downloaded.
""" # ruff: ignore[docstring-extraneous-exception] - also raised by hub_snapshot
root = await hub_snapshot(self.HF_REPO, self.SOURCE_REVISION, cache_dir, tuple(self.SOURCE_FILES.values()))
refs = []
files = {}
for part, name in self.SOURCE_FILES.items():
path = root / name if "*" not in name else root / Path(name).parent
if not path.exists():
raise TimeNetDownloadError(f"{self.HF_REPO}@{self.SOURCE_REVISION} has no {name}")
refs.append(HubFileRef(path, part))
return refs
files[part] = path
return files
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,15 @@
letters that the task does not use.
"""

from collections.abc import Mapping
from pathlib import Path
from typing import Self

from pydantic import model_validator

from timenet.dataset import OrdinalAxis, Record, Signal, Source, TimeFDataset
from timenet.types import ClassificationTask, LabelSet, Split, TimeSeriesSpec, ureg
from timenet_connectors.bases.huggingface import BaseHuggingFaceConnector, HubFileRef
from timenet_connectors.bases.huggingface import BaseHuggingFaceConnector
from timenet_connectors.sources.readers import DataRow, ParquetDataReader


Expand Down Expand Up @@ -76,17 +78,17 @@ class TimeSeriesExam1Connector(
):
"""Build TimeSeriesExam from the pinned test parquet file of its Hugging Face release."""

def convert(self, raw_refs: list[HubFileRef]) -> TimeFDataset:
def convert(self, files: Mapping[str, Path]) -> TimeFDataset:
"""Build one Record per series and one classification task per question.

Args:
raw_refs: The downloaded test file.
files: The downloaded test file.

Returns:
The populated :class:`~timenet.dataset.TimeFDataset`.
"""
dataset = TimeFDataset(metadata=self.metadata())
for row in ParquetDataReader(raw_refs[0].path, _SourceRow).iter_rows():
for row in ParquetDataReader(files["test"], _SourceRow).iter_rows():
inputs = [dataset.add_record(_record(row.id, role, values)) for role, values in row.series.items()]
dataset.add_task(
ClassificationTask(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
from timenet.reader import TimeFReader
from timenet.registry import open_local_version
from timenet.types import ClassificationTask, Split, read_value
from timenet_connectors.bases.huggingface import HubFileRef
from timenet_connectors.datasets.autonlab.timeseriesexam1 import TimeSeriesExam1Connector


Expand All @@ -28,7 +27,7 @@ def _convert(tmp_path: Path, rows: list[dict] | None = None) -> TimeFDataset:
if rows is not None:
path = tmp_path / "rows.parquet"
pq.write_table(pa.Table.from_pylist(rows, schema=pq.read_schema(_FIXTURE)), path)
return TimeSeriesExam1Connector().convert([HubFileRef(path, "test")])
return TimeSeriesExam1Connector().convert({"test": path})


def _task_for(dataset: TimeFDataset, row_id: int) -> ClassificationTask:
Expand Down
Loading
Loading