Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,14 @@ this branch. Anchor files:

## The `BaseConnector` contract

`BaseConnector(ABC, Generic[TRaw])` in `timenet.connectors`. `TRaw` is whatever `download` hands to
`convert` (a dict per Hub row, a dataclass ref per PhysioNet record, etc.).

- `download(self, cache_dir: Path) -> list[TRaw]` (abstract): fetch/discover raw source files, return
lightweight refs. I/O only, no parsing, idempotent for a given `cache_dir`.
- `convert(self, raw_refs: list[TRaw]) -> TimeFDataset` (abstract): parse refs into a `TimeFDataset`.
CPU only, no network.
`BaseConnector(ABC, Generic[TRaw])` in `timenet.connectors`. `TRaw` is the whole value that `download` hands to
`convert`: one handle dataclass, a list of file refs (`list[HubFileRef]` for Hub connectors), or `None`.

- `download(self, cache_dir: Path) -> TRaw` (abstract): fetch/discover raw source files, return
a lightweight handle. I/O only, no parsing, idempotent for a given `cache_dir`.
- `convert(self, raw: TRaw, /) -> TimeFDataset` (abstract): parse the download result into a
`TimeFDataset`. CPU only, no network. The parameter is positional-only, so name it as you like
(`source`, `recordings`).
- `metadata(self) -> DatasetMetadata` (**concrete**, do not override): loads and validates the card via
`DatasetMetadata.from_yaml`. By convention the card is `dataset.yaml` beside the connector module;
set the `CARD` class var to point elsewhere. (Some docs call `metadata` abstract; it isn't.)
Expand Down Expand Up @@ -70,7 +71,7 @@ tags:
## Base connectors to reuse

### `BaseHuggingFaceConnector` (`bases/huggingface.py`)
`BaseHuggingFaceConnector(BaseConnector[HubFileRef])` downloads files from a Hub dataset at a pinned
`BaseHuggingFaceConnector(BaseConnector[list[HubFileRef]])` downloads files from a Hub dataset at a pinned
commit. Pass these class keywords:

- `hf_repo`: the Hub repo id.
Expand Down
38 changes: 20 additions & 18 deletions docs/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,19 @@ from pathlib import Path
from timenet.connectors import BaseConnector
from timenet.dataset import TimeFDataset

class MyConnector(BaseConnector[MyRawRef]):
def download(self, cache_dir: Path) -> list[MyRawRef]: ...
def convert(self, raw_refs: list[MyRawRef]) -> TimeFDataset: ...
class MyConnector(BaseConnector[MySource]):
def download(self, cache_dir: Path) -> MySource: ...
def convert(self, source: MySource) -> TimeFDataset: ...
```

The two stages stay separate. This lets the engine drive
`download -> convert -> derive_schema -> store_dataset`.

| Method | Nature | Contract |
| --- | --- | --- |
| `download(cache_dir)` | I/O only | Fetch or find raw files and return lightweight references. The method is idempotent and does not parse data. |
| `convert(raw_refs)` | local work | Parse the references into a `TimeFDataset`. Do not use the network. |
| `compose(raw_refs, context)` | local work | Build a child dataset from the references and the opened parent releases. The default calls `convert(raw_refs)`. |
| `download(cache_dir)` | I/O only | Fetch or find raw files and return a lightweight handle to them. The method is idempotent and does not parse data. |
| `convert(raw)` | local work | Parse the download result into a `TimeFDataset`. Do not use the network. |
| `compose(raw, context)` | local work | Build a child dataset from the download result and the opened parent releases. The default calls `convert(raw)`. |

A connector must implement `download` or `download_async`. It must also implement `convert` or
`compose`. If either pair has no implementation, Python raises `TypeError` during connector
Expand All @@ -63,9 +63,11 @@ writer after conversion.
card. `metadata().dataset_id` must match the connector's built id.

A connector takes no constructor arguments. Configuration comes from environment variables that
`__init__` reads. `TRaw` is the reference type that the connector defines, for example a path, a small
dataclass, or an S3 key. The connector is generic over `TRaw`:
`class MyConnector(BaseConnector[MyRawRef])`.
`__init__` reads. `TRaw` is the type of the value that `download` returns. The connector defines it.
It can be any value, for example one small dataclass, a list of file references, or `None`. The
connector is generic over `TRaw`: `class MyConnector(BaseConnector[MySource])`. The engine passes the
value to `compose` unchanged. The base takes it as a positional argument, so a connector can name the
parameter, for example `source` or `recordings`.

If an assumption about your source data, downloads, or parents does not hold, raise
`TimeNetConnectorError` from `timenet.errors`.
Expand Down Expand Up @@ -269,7 +271,7 @@ helpers from `download_async`:
```python
from timenet_connectors.download import Artifact, ensure_archive, download_files

class MyConnector(BaseConnector[MyRawRef]):
class MyConnector(BaseConnector[MySource]):
async def download_async(self, cache_dir):
await download_files(
[
Expand All @@ -280,7 +282,7 @@ class MyConnector(BaseConnector[MyRawRef]):
headers={"Authorization": "Bearer …"},
)
await ensure_archive("https://host/records.zip", cache_dir)
return [...] # lightweight references into cache_dir
return MySource(...) # a lightweight handle to the files in cache_dir
```

An ambient sink reports progress, through `timenet_connectors.download.progress`. The download helpers
Expand Down Expand Up @@ -318,9 +320,9 @@ A successful build removes the raw downloads of the dataset, at
`timenet.build()`. This helps while you write a connector for a large source, because each rebuild
downloads the source again.

Keeping `download` and `convert` apart makes a connector testable offline. `convert` takes raw
references and does not touch the network. A test can therefore pass small local inputs or synthetic
references directly to `convert()` and skip `download`.
Keeping `download` and `convert` apart makes a connector testable offline. `convert` takes the
download result and does not touch the network. A test can therefore pass a small local input or a
synthetic handle directly to `convert()` and skip `download`.

After the build, you can load and inspect a dataset with the SDK. See `examples/load_tsqa.py`. This
example loads a dataset and calls `describe()` to print its identity, its counts, its columns per
Expand All @@ -337,8 +339,8 @@ parents:
ptbxl: physionet/ptb-xl@1.0.0
```

The connector of a child overrides `compose(raw_refs, context)`. The default `compose` calls
`convert(raw_refs)`. If the card declares parents and the connector does not override `compose`,
The connector of a child overrides `compose(raw, context)`. The default `compose` calls
`convert(raw)`. If the card declares parents and the connector does not override `compose`,
the build raises before it opens a parent or downloads anything.

Before the connector runs, the build prepares each parent release in the local registry, as
Expand All @@ -352,7 +354,7 @@ from timenet.dataset import TimeFDataset
from timenet.types.reference_keys import ParentAlias


def compose(self, raw_refs, context: BuildContext) -> TimeFDataset:
def compose(self, source, context: BuildContext) -> TimeFDataset:
dataset = TimeFDataset(metadata=self.metadata())
ptbxl = context.parents[ParentAlias("ptbxl")]
# Find parent Records by a key of the source in their metadata.
Expand Down Expand Up @@ -405,7 +407,7 @@ See the [API reference for `timenet.connectors`](api/timenet.connectors.md) for
## Reading source rows

For CSV, JSON arrays, JSON Lines, and Parquet, declare the source fields in a connector-owned
`DataRow` and read each downloaded file during `convert`:
`DataRow` and read each downloaded file during `convert`. This example reads a list of file references:

```python
from pydantic import Field
Expand Down
2 changes: 1 addition & 1 deletion docs/data-model/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ database, and Parquet or Zarr Signal values. Reading it never runs connector cod

```mermaid
flowchart LR
D["download()<br/><i>I/O · fetch raw refs</i>"]
D["download()<br/><i>I/O · fetch raw files</i>"]
C["convert()<br/><i>CPU only</i>"]
S["derive_schema()<br/><i>types from data</i>"]
W["store_dataset()<br/><i>TimeFWriter</i>"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ class HubFileRef:
``convert()`` branches on and builds ids from."""


class BaseHuggingFaceConnector(BaseConnector[HubFileRef], ABC):
class BaseHuggingFaceConnector(BaseConnector[list[HubFileRef]], ABC):
"""Base class for connectors that read files from a pinned Hub dataset revision.

Subclasses pass ``hf_repo``, ``source_revision``, and ``source_files`` as class keywords, and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ def read_rows(self) -> Iterator[HarCotRow]:
yield parsed


class HarCotConnector(BaseConnector[HarCotSplitFile]):
class HarCotConnector(BaseConnector[list[HarCotSplitFile]]):
"""Connector for the HAR-CoT release of OpenTSLM."""

async def download_async(self, cache_dir: Path) -> list[HarCotSplitFile]: # ruff: ignore[no-self-use]
Expand All @@ -199,17 +199,17 @@ async def download_async(self, cache_dir: Path) -> list[HarCotSplitFile]: # ruf
root = await ensure_archive(HAR_COT_URL, cache_dir, filename="har_cot.zip", sha256=HAR_COT_SHA256)
return [HarCotSplitFile(split=split, path=root / name) for split, name in SPLIT_FILES.items()]

def convert(self, raw_refs: list[HarCotSplitFile]) -> TimeFDataset:
def convert(self, split_files: list[HarCotSplitFile]) -> TimeFDataset:
"""Create one Record and one ClassificationTask for each window.

Args:
raw_refs: The split files from :meth:`download_async`.
split_files: The split files from :meth:`download_async`.

Returns:
The populated dataset.
"""
dataset = TimeFDataset(metadata=self.metadata())
for split_file in raw_refs:
for split_file in split_files:
for row in split_file.read_rows():
record = dataset.add_record(row.create_record())
dataset.add_task(row.create_classification_task(record, split=split_file.split))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ def read_splits(self, n_rows: int) -> list[Split]:
class SleepCotConnector(BaseConnector[SleepCotReader]):
"""Connector for the SleepEDF-CoT release of OpenTSLM."""

async def download_async(self, cache_dir: Path) -> list[SleepCotReader]: # ruff: ignore[no-self-use]
async def download_async(self, cache_dir: Path) -> SleepCotReader: # ruff: ignore[no-self-use]
"""Download the release CSV.

Args:
Expand All @@ -273,18 +273,17 @@ async def download_async(self, cache_dir: Path) -> list[SleepCotReader]: # ruff
"""
artifact = Artifact(SLEEP_COT_URL, cache_dir / "sleep_cot.csv", sha256=SLEEP_COT_SHA256)
(csv_path,) = await download_files([artifact])
return [SleepCotReader(csv_path=csv_path, splits_path=SPLITS_PATH)]
return SleepCotReader(csv_path=csv_path, splits_path=SPLITS_PATH)

def convert(self, raw_refs: list[SleepCotReader]) -> TimeFDataset:
def convert(self, reader: SleepCotReader) -> TimeFDataset:
"""Create one Record and one ClassificationTask for each window.

Args:
raw_refs: The reader from :meth:`download_async`.
reader: The reader from :meth:`download_async`.

Returns:
The populated dataset.
"""
(reader,) = raw_refs
rows = reader.read_rows()
splits = reader.read_splits(len(rows))
dataset = TimeFDataset(metadata=self.metadata())
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def _write_release(


def test_each_window_becomes_a_record_and_a_classification_task(tmp_path):
tasks = SleepCotConnector().convert([_write_release(tmp_path)]).tasks_of(ClassificationTask)
tasks = SleepCotConnector().convert(_write_release(tmp_path)).tasks_of(ClassificationTask)
assert [task.split for task in tasks] == [Split.TRAIN, Split.VALIDATION, Split.TRAIN, Split.TEST]
for task, row in zip(tasks, _ROWS, strict=True):
(record,) = task.inputs
Expand Down Expand Up @@ -105,7 +105,7 @@ def test_each_window_becomes_a_record_and_a_classification_task(tmp_path):
)
def test_a_split_file_that_does_not_fit_the_release_raises(tmp_path, splits, message):
with pytest.raises(TimeNetConnectorError, match=message):
SleepCotConnector().convert([_write_release(tmp_path, **splits)])
SleepCotConnector().convert(_write_release(tmp_path, **splits))


@pytest.mark.parametrize(
Expand All @@ -121,13 +121,13 @@ def test_a_split_file_that_does_not_fit_the_release_raises(tmp_path, splits, mes
def test_a_malformed_row_raises_with_its_location(tmp_path, change, message):
rows = [*_ROWS[:2], _ROWS[2] | change]
with pytest.raises(TimeNetConnectorError, match=rf"sleep_cot\.csv data row 3: {message}"):
SleepCotConnector().convert([_write_release(tmp_path, rows)])
SleepCotConnector().convert(_write_release(tmp_path, rows))


def test_the_dataset_round_trips_through_storage(tmp_path):
source = tmp_path / "source"
source.mkdir()
dataset = SleepCotConnector().convert([_write_release(source)])
dataset = SleepCotConnector().convert(_write_release(source))
version_dir = dataset.write(path=tmp_path / "registry")
with TimeFReader(open_local_version(version_dir)) as reader:
reader.verify()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,9 @@

@dataclass(frozen=True)
class EcgQaCotSource:
"""A lightweight handle to the fetched sources, so ``compose`` streams rather than holding refs.
"""A lightweight handle to the fetched sources.

``download`` returns one of these instead of a row per question, so the ~230k questions are read
lazily during ``compose`` and the task stream, never materialized as a list.
``compose`` and the task stream read the ~230k questions lazily. Nothing holds them as a list.
"""

answers_path: Path
Expand Down Expand Up @@ -286,7 +285,7 @@ class EcgQaCotConnector(BaseConnector[EcgQaCotSource]):
(Split.TEST, "ecg_qa_cot_test.csv"),
)

async def download_async(self, cache_dir: Path) -> list[EcgQaCotSource]:
async def download_async(self, cache_dir: Path) -> EcgQaCotSource:
"""Fetch the template answers and CoT CSVs and return one lightweight handle.

The template-answers CSV and the CoT archive download concurrently. This method reads no
Expand All @@ -296,7 +295,7 @@ async def download_async(self, cache_dir: Path) -> list[EcgQaCotSource]:
cache_dir: The directory that holds downloaded archives.

Returns:
A single-element list holding the :class:`EcgQaCotSource` handle.
The :class:`EcgQaCotSource` handle.
"""
answers_path = cache_dir / "answers_for_each_template.csv"
# The two HTTP artifacts download concurrently.
Expand All @@ -309,9 +308,9 @@ async def download_async(self, cache_dir: Path) -> list[EcgQaCotSource]:
cot_csvs = tuple(
(split, find_dir_containing(cot_root, csv_name) / csv_name) for split, csv_name in self._COT_CSVS
)
return [EcgQaCotSource(answers_path=answers_path, cot_csvs=cot_csvs)]
return EcgQaCotSource(answers_path=answers_path, cot_csvs=cot_csvs)

def compose(self, raw_refs: list[EcgQaCotSource], context: BuildContext) -> TimeFDataset:
def compose(self, source: EcgQaCotSource, context: BuildContext) -> TimeFDataset:
"""Build one record per recording and stream one :class:`AnswerTask` per CoT row.

A first pass over the CoT CSVs collects each recording's split and the distinct question
Expand All @@ -321,7 +320,7 @@ def compose(self, raw_refs: list[EcgQaCotSource], context: BuildContext) -> Time
themselves stream from :meth:`_iter_tasks`, so the ~230k questions never all live in memory.

Args:
raw_refs: The single-element list from :meth:`download`.
source: The handle from :meth:`download`.
context: The opened PTB-XL parent, under the parent alias ``ptbxl``.

Returns:
Expand All @@ -332,7 +331,6 @@ def compose(self, raw_refs: list[EcgQaCotSource], context: BuildContext) -> Time
name, or if an imported Record has no patient, age, or sex, or has a sex other than
male or female.
"""
source = raw_refs[0]
answers = _load_template_answers(source.answers_path)
dataset = TimeFDataset(metadata=self.metadata())

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ def _store_ptbxl(
for row in sorted(rows, key=lambda row: int(row["ecg_id"]), reverse=True):
filename = f"records500/00000/{int(row['ecg_id']):05d}_hr"
writer.writerow({column: row.get(column, "") for column in _PTBXL_COLUMNS} | {"filename_hr": filename})
parent = PtbXlConnector().convert([PtbXlSource(tmp_path, database)])
parent = PtbXlConnector().convert(PtbXlSource(tmp_path, database))
if change is not None:
ecg_id, key, value = change
(record,) = (record for record in parent.records if record.metadata["ecg_id"] == ecg_id)
Expand Down Expand Up @@ -252,7 +252,7 @@ def _composed(
_store_ptbxl(tmp_path, records_dir, _PTBXL_ROWS if ptbxl_rows is None else ptbxl_rows, root)
connector = EcgQaCotConnector()
with open_parents(connector.metadata(), LocalRegistry(root)) as context:
yield connector.compose([_source(tmp_path, split)], context)
yield connector.compose(_source(tmp_path, split), context)


def test_is_a_connector():
Expand Down Expand Up @@ -383,7 +383,7 @@ def test_an_incomplete_ptbxl_record_raises(tmp_path, records_dir, key, message):
pytest.raises(TimeNetConnectorError, match=f"ecg_id 2: {message}"),
open_parents(connector.metadata(), LocalRegistry(root)) as context,
):
connector.compose([_source(tmp_path)], context)
connector.compose(_source(tmp_path), context)


def test_a_ptbxl_sex_other_than_male_or_female_raises(tmp_path, records_dir):
Expand All @@ -396,7 +396,7 @@ def test_a_ptbxl_sex_other_than_male_or_female_raises(tmp_path, records_dir):
pytest.raises(TimeNetConnectorError, match="ecg_id 2: sex 'unknown' is not male or female"),
open_parents(connector.metadata(), LocalRegistry(root)) as context,
):
connector.compose([_source(tmp_path)], context)
connector.compose(_source(tmp_path), context)


def test_series_values_match_fixture_record(tmp_path, records_dir):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,14 @@ def _parse_patient_id(cells: dict[str, str]) -> str:
class PtbXlConnector(BasePhysioNetConnector[PtbXlSource]):
"""Preserve PTB-XL waveforms and clinical metadata without downstream tasks."""

async def download_async(self, cache_dir: Path) -> list[PtbXlSource]: # ruff: ignore[no-self-use]
async def download_async(self, cache_dir: Path) -> PtbXlSource: # ruff: ignore[no-self-use]
"""Return the extracted pinned PTB-XL release."""
root = find_dir_containing(
await ensure_archive(PTBXL_ZIP_URL, cache_dir, sha256=PTBXL_ZIP_SHA256), "ptbxl_database.csv"
)
return [PtbXlSource(root, root / "ptbxl_database.csv")]
return PtbXlSource(root, root / "ptbxl_database.csv")

def convert(self, raw_refs: list[PtbXlSource]) -> TimeFDataset:
def convert(self, source: PtbXlSource) -> TimeFDataset:
"""Return records without tasks, with lazy WFDB value loaders.

Each record carries its PTB-XL ``ecg_id`` in its metadata. A child dataset selects the
Expand All @@ -141,7 +141,6 @@ def convert(self, raw_refs: list[PtbXlSource]) -> TimeFDataset:
``ecg_id``. For a repeated ``ecg_id``, the message also names the earlier row.
"""
dataset = TimeFDataset(metadata=self.metadata())
source = raw_refs[0]
# The CSV row where each ecg_id first appears.
first_rows: dict[int, int] = {}
for line, source_row in enumerate(CsvDataReader(source.database_csv, _SourceRow).iter_rows(), 2):
Expand Down
Loading
Loading