Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .agents/skills/add-dataset-connector/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,9 @@ repo's files at the commit you will pin (`https://huggingface.co/api/datasets/<r
latter). For PhysioNet, inspect the archive layout and a WFDB header (`fs`, `sig_len`, `sig_name`).

### 3. Pick the base connector
- Hub dataset: subclass `BaseHuggingFaceConnector`, set `HF_REPO`, `SOURCE_REVISION` (a commit), and
`SOURCE_FILES`, inherit `download`, and implement `convert` over the `HubFileRef` list. The base
fills `source_url` and `source_revision` in the dataset metadata from `HF_REPO` and `SOURCE_REVISION`.
- Hub dataset: subclass `BaseHuggingFaceConnector`, pass the class keywords `hf_repo`,
`source_revision` (a commit), and `source_files`, inherit `download`, and implement `convert`
over the `HubFileRef` list. The base fills `source_url` and `source_revision` in the dataset metadata from `HF_REPO` and `SOURCE_REVISION`.
- PhysioNet/WFDB: subclass `BasePhysioNetConnector`, implement `download` (use `ensure_archive` /
`download_files` from `timenet_connectors.download`) and `convert` (use `_read_header` / `_lead_loader`).
- Neither: subclass `BaseConnector[TRaw]` and implement `download` + `convert` yourself.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,11 +71,14 @@ tags:

### `BaseHuggingFaceConnector` (`bases/huggingface.py`)
`BaseHuggingFaceConnector(BaseConnector[HubFileRef])` downloads files from a Hub dataset at a pinned
commit. Set:
commit. Pass these class keywords:

- `HF_REPO`: the Hub repo id.
- `SOURCE_REVISION`: a 40-character commit, not a branch, so every rebuild reads the same files.
- `SOURCE_FILES`: part name -> file path in the repo.
- `hf_repo`: the Hub repo id.
- `source_revision`: a 40-character commit, not a branch, so every rebuild reads the same files.
- `source_files`: part name -> file path in the repo.

The base stores them as `HF_REPO`, `SOURCE_REVISION`, and `SOURCE_FILES`. A missing keyword or a
bad revision fails when Python defines the class.

The base fills `source_url` and `source_revision` in the dataset metadata from `HF_REPO` and
`SOURCE_REVISION`. The card can omit both fields. Keep `source_url` in the card. The docs catalog,
Expand Down Expand Up @@ -123,11 +126,12 @@ Populate a `TimeFDataset` (`from timenet.dataset import Record, Signal, Source,
has gaps, say why on the spec: `missing=MissingValues(reason=MissingReason.SENSOR_GAP)`. The
writer records how many timesteps each signal misses, as `Signal.null_count`. Keep a null a
null: never fill a gap.
- Group signals under a `Source(name=..., signals=...)`. Then
`dataset.add_record(record=Record(sources=(source,)))` registers the record and returns it. A
- Group signals under a `Source("name", signals=[...])`. Then
`dataset.add_record(Record(sources=[source]))` registers the record and returns it. A collection
field such as `sources`, `signals`, or `inputs` takes any iterable and stores a tuple. A
windowed record says so through its axis: `RegularAxis.at_index(...)` moves the origin into the
recording.
- `record.annotate(Annotation(key=..., value=...))` attaches one annotation and returns it.
- `record.annotate(Annotation("key", value=value))` attaches one annotation and returns it.
`record.add_annotations([...])` takes an iterable and returns a tuple. There is one annotation
class, and its `span` gives its shape. No span means the whole record.
`span=TimePoint.seconds(...)` means a time offset. `span=TimeInterval.seconds(...)` means a region.
Expand All @@ -136,8 +140,9 @@ Populate a `TimeFDataset` (`from timenet.dataset import Record, Signal, Source,
its number of characters, pass it as `length=n` and every read checks the result against it.
The writer stores this text in a separate payload file so readers can load it on request. For a
list of strings, use `LazyTextList`, with the number of strings as `length` if you know it.
- `dataset.add_task(task=<Task>(inputs=(record,), ...))` registers one task and returns it.
`dataset.add_tasks(tasks=[...])` takes an iterable and registers the batch all-or-nothing. The task
- `dataset.add_task(<Task>(inputs=[record], ...))` registers one task and returns it. Pass one
target as a bare value: `targets="walking"`.
`dataset.add_tasks([...])` takes an iterable and registers the batch all-or-nothing. The task
names its input records itself. Set `scope` and `from_tasks` on the task too. A batch can derive
from its own members in any order.
- You cannot pass ids. The dataset gives every record, signal, and task a read-only int `id` when
Expand Down Expand Up @@ -178,12 +183,9 @@ the bounds of every `Span` the task carries.

```python
import json
from collections.abc import Mapping
from typing import ClassVar

from pydantic import ConfigDict, Field
from timenet.dataset import Record, Signal, Source, TimeFDataset
from timenet.dataset.axis import OrdinalAxis
from timenet.dataset import OrdinalAxis, Record, Signal, Source, TimeFDataset
from timenet.types import Annotation, AnswerTask, TimeSeriesSpec, ureg
from timenet_connectors.bases.huggingface import BaseHuggingFaceConnector, HubFileRef
from timenet_connectors.sources.readers import DataRow, ParquetDataReader
Expand All @@ -205,13 +207,14 @@ class TSQARow(DataRow):
answer: str = Field(alias="Answer")


class TSQAConnector(BaseHuggingFaceConnector):
"""Connector for the TSQA time-series QA dataset (Hub repo ``ChengsenWang/TSQA``)."""

HF_REPO = "ChengsenWang/TSQA" # the external Hub repo id (keeps its own casing)
class TSQAConnector(
BaseHuggingFaceConnector,
hf_repo="ChengsenWang/TSQA", # the external Hub repo id (keeps its own casing)
# The commit refs/convert/parquet pointed to: the Hub's parquet conversion of the CSV.
SOURCE_REVISION = "22518209e3ee6302be4b4957f2692b3e8de849c3"
SOURCE_FILES: ClassVar[Mapping[str, str]] = {"train": "default/train/0000.parquet"}
source_revision="22518209e3ee6302be4b4957f2692b3e8de849c3",
source_files={"train": "default/train/0000.parquet"},
):
"""Connector for the TSQA time-series QA dataset (Hub repo ``ChengsenWang/TSQA``)."""

def convert(self, raw_refs: list[HubFileRef]) -> TimeFDataset:
"""Build one record per row: the parsed series plus its question-and-answer task.
Expand All @@ -227,27 +230,21 @@ class TSQAConnector(BaseHuggingFaceConnector):
for row in ParquetDataReader(ref.path, TSQARow).iter_rows():
series = json.loads(row.series)
signals = series if series and isinstance(series[0], list) else [series]
time_series = tuple(
time_series = [
Signal.from_values(
values,
spec=_SPEC,
name=f"c{signal}",
time_axis=OrdinalAxis(),
)
for signal, values in enumerate(signals)
)
record = Record(sources=(Source(name="TSQA series", signals=time_series),))
dataset.add_record(record=record)
record.annotate(Annotation(key="task", value=row.task))
]
record = Record(sources=[Source("TSQA series", signals=time_series)])
dataset.add_record(record)
record.annotate(Annotation("task", value=row.task))
if row.label:
record.annotate(Annotation(key="label", value=row.label))
dataset.add_task(
task=AnswerTask(
inputs=(record,),
prompt=row.question,
targets=(row.answer,),
)
)
record.annotate(Annotation("label", value=row.label))
dataset.add_task(AnswerTask(inputs=[record], prompt=row.question, targets=row.answer))
return dataset


Expand All @@ -273,7 +270,7 @@ a local dict keyed by the PTB-XL `ecg_id`. It stores the `ecg_id` and the PTB-XL
come from `ptbxl_database.csv`. Per-question metadata (question_type, template_id, answer_options,
clinical_context) are value-deduped dataset annotations that the tasks reference. The connector
builds the clinical context from `ptbxl_database.csv`. It streams one
`AnswerTask(inputs=(record,), prompt=..., rationale=<CoT>, targets=(<label>,))` per question.
`AnswerTask(inputs=[record], prompt=..., rationale=<CoT>, targets=<label>)` per question.
`_leads_for` reads the WFDB header for `fs`/`sig_len`/`sig_name` and builds one lazy `Signal` per
lead. All tasks for a recording point at its one record. Records do not share a series. See its
`connector.py` for the full pattern.
Expand Down
12 changes: 6 additions & 6 deletions docs/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,12 @@ Preserve official partitions when the source provides them by setting `split` on
directly. No splitter is needed:

```python
from timenet.types.splits import Split
from timenet.types import Split

record = make_record(sample)
dataset.add_record(record=record)
dataset.add_record(record)
task = make_task(sample, record, split=Split.TRAIN)
dataset.add_task(task=task)
dataset.add_task(task)
```

Translate source-specific partition names into `Split` values in the connector.
Expand All @@ -112,7 +112,7 @@ For a dataset without official partitions, explicitly choose a category to balan
key that prevents related samples from crossing partitions:

```python
from timenet.types.splits import Split
from timenet.types import Split
from timenet_connectors.splitting import StratifiedSplitter

def get_category(sample):
Expand Down Expand Up @@ -175,9 +175,9 @@ When constructing TimeNet tasks, copy the partition name onto each task:
for split_name, samples_in_split in splits.items():
for sample in samples_in_split:
record = make_record(sample)
dataset.add_record(record=record)
dataset.add_record(record)
task = make_task(sample, record, split=split_name)
dataset.add_task(task=task)
dataset.add_task(task)
```

Here `make_record` and `make_task` are the connector's own construction functions. A task refers to
Expand Down
10 changes: 6 additions & 4 deletions docs/data-model/annotations.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ condition = Annotation(
occurrence = record.annotate(condition)
```

`key` can be positional: `Annotation("condition", value="healthy")`. All other fields are keyword-only.

`annotate()` returns the attached occurrence. A Task must refer to this occurrence, not the unattached
content object.

Expand All @@ -47,11 +49,11 @@ from typing import Literal
from timenet.types import Annotation

# A constant to reuse
HEALTHY_ANNOTATION = Annotation(key="condition", value="healthy")
HEALTHY_ANNOTATION = Annotation("condition", value="healthy")


def quality_annotation(level: Literal["low", "medium", "high"]) -> Annotation:
return Annotation(key="quality", value=level)
return Annotation("quality", value=level)


record.annotate(HEALTHY_ANNOTATION)
Expand All @@ -69,8 +71,8 @@ row are stored as one row again, but stored text and the same text in memory bec
An Annotation without a `span` describes its complete owner:

```python
record.annotate(Annotation(key="age", value=54, unit="years"))
record.annotate(Annotation(key="condition", value="healthy"))
record.annotate(Annotation("age", value=54, unit="years"))
record.annotate(Annotation("condition", value="healthy"))
```

Use `metadata` for untyped implementation details. Use an Annotation when consumers need a named,
Expand Down
16 changes: 8 additions & 8 deletions docs/data-model/records.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,18 @@ Build the hierarchy from the leaves upward:
from timenet.dataset import Record, Source

accelerometer = Source(
name="Accelerometer",
"Accelerometer",
signals=(acceleration_x, acceleration_y, acceleration_z),
)
imu = Source(
name="IMU",
"IMU",
sources=(accelerometer,),
)
record = Record(
sources=(imu,),
metadata={"site": "lab-a", "run": "run-001"},
)
dataset.add_record(record=record)
dataset.add_record(record)
```

`dataset.add_record()` gives the Record and each of its Signals a read-only int64 `id`. A Source
Expand All @@ -76,7 +76,7 @@ firmware version, or an event on the timeline.
```python
from timenet.types import Annotation

record.annotate(Annotation(key="condition", value="healthy"))
record.annotate(Annotation("condition", value="healthy"))
```

Sources and Signals also support `metadata` and `annotate()`.
Expand All @@ -95,8 +95,8 @@ origin = TimeOrigin(datetime(2026, 1, 1, tzinfo=timezone.utc))
record = Record(
start_time=origin,
sources=(
Source(name="sensor A", signals=(signal_a,)),
Source(name="sensor B", signals=(signal_b,)),
Source("sensor A", signals=[signal_a]),
Source("sensor B", signals=[signal_b]),
),
)
```
Expand Down Expand Up @@ -128,10 +128,10 @@ CONDITION = LabelSet(

task = ClassificationTask(
inputs=(record,),
targets=("healthy",),
targets="healthy",
label_set=CONDITION,
)
dataset.add_task(task=task)
dataset.add_task(task)
```

Use `dataset.tasks_for(record)` to resolve the Task objects for one Record.
29 changes: 15 additions & 14 deletions docs/data-model/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@ Empatica EmbracePlus watch. Its accelerometer records three axes at 64 Hz.
```python
import numpy as np

from timenet.dataset import Record, Signal, Source
from timenet.dataset.axis import RegularAxis
from timenet.dataset import Record, RegularAxis, Signal, Source
from timenet.types import TimeSeriesSpec, ureg

# Stand-in for 60 s of a device export. A connector reads these values.
Expand All @@ -47,7 +46,7 @@ acceleration = TimeSeriesSpec(
)

chest_strap = Source(
name="Chest strap",
"Chest strap",
signals=(
Signal(
name="ECG",
Expand All @@ -59,19 +58,19 @@ chest_strap = Source(
metadata={"model": "Polar H10"},
)
accelerometer = Source(
name="Accelerometer",
signals=tuple(
"Accelerometer",
signals=[
Signal(
name=axis,
spec=acceleration,
time_axis=RegularAxis.from_rate_hz(64),
data=values,
)
for axis, values in zip("xyz", wrist_values)
),
],
)
watch = Source(
name="Wrist watch",
"Wrist watch",
sources=(accelerometer,),
metadata={"model": "Empatica EmbracePlus"},
)
Expand All @@ -92,14 +91,16 @@ print([signal.name for signal in record.signals])
The device model goes in the `metadata` of its Source. The participant and the session go in the
`metadata` of the Record.

All fields are keyword-only:
`name` can be positional, as in `Source("IMU", signals=[...])`. All other fields are keyword-only. The
collection fields accept any iterable, such as a list, and store a tuple. A single item or a string
raises `TimeFValidationError`.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `name` | `str` | required | Human-readable name. It must not be empty. |
| `sources` | `tuple[Source, ...]` | `()` | Direct child Sources. Their order has no meaning. |
| `signals` | `tuple[Signal, ...]` | `()` | Signals that this Source produces directly. |
| `annotations` | `tuple[Annotation, ...]` | `()` | Annotation occurrences on this Source. |
| `sources` | `Iterable[Source]`, stored as a tuple | `()` | Direct child Sources. Their order has no meaning. |
| `signals` | `Iterable[Signal]`, stored as a tuple | `()` | Signals that this Source produces directly. |
| `annotations` | `Iterable[Annotation]`, stored as a tuple | `()` | Annotation occurrences on this Source. |
| `metadata` | JSON-compatible mapping | `{}` | Untyped device details. |

## Nest Sources
Expand All @@ -121,7 +122,7 @@ interval, for example a time when an electrode lost contact with the skin:
```python
from timenet.types import Annotation, TimeInterval

chest_strap.annotate(Annotation(key="firmware", value="3.1.1"))
chest_strap.annotate(Annotation("firmware", value="3.1.1"))
chest_strap.annotate(
Annotation(
key="electrode_contact",
Expand Down Expand Up @@ -166,12 +167,12 @@ dataset = TimeFDataset(
license="CC-BY-4.0",
)
)
dataset.add_record(record=record)
dataset.add_record(record)
night_2 = Record(
sources=(chest_strap,),
metadata={"participant": "P017", "session": "night-2"},
)
dataset.add_record(record=night_2)
dataset.add_record(night_2)
```

```text
Expand Down
Loading