Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .agents/skills/add-dataset-connector/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,10 @@ Add the org namespace `__init__.py` if the org is new. Keep `download` I/O-only
with lazy value loaders (never materialize arrays in `convert`). See the worked example in
`references/connector-anatomy.md`.

Attach annotations when you build the owner: pass `annotations=[Annotation("key", value=value)]` to
`TimeFDataset`, `Record`, `Source`, `Signal`, or the task. Use `annotate()` only when the owner
exists already, or when a task refers to the occurrence that the call returns.

You cannot pass ids. Records, signals, and tasks get a read-only int `id` when you add them to the
dataset. Sources, axes, and annotations have no id. Put upstream identifiers in `Record.metadata`
(for example `{"ecg_id": 17}`) or in an annotation. Refer to other objects by the Python object,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,19 @@ Populate a `TimeFDataset` (`from timenet.dataset import Record, Signal, Source,
field such as `sources`, `signals`, or `inputs` takes any iterable and stores a tuple. A
windowed record says so through its axis: `RegularAxis.at_index(...)` moves the origin into the
recording.
- `record.annotate(Annotation("key", value=value))` attaches one annotation and returns it.
`record.add_annotations([...])` takes an iterable and returns a tuple. There is one annotation
class, and its `span` gives its shape. No span means the whole record.
- Attach annotations when you build the owner. `TimeFDataset`, `Record`, `Source`, `Signal`, and
every task take `annotations=[...]`, a list of `Annotation` objects:
`Record(sources=[source], annotations=[Annotation("key", value=value)])`. `Annotation` takes
only `key` by position. For an annotation that is not always present, build the list first and
append to it.
- Use `owner.annotate(annotation)` only in these cases. The owner exists already, for example a
Record that you import from a parent. A task refers to the annotation in `input_annotations` or
`target_annotations`, so you need the occurrence that the call returns.
`record.add_annotations([...])` does the same for a batch and returns a tuple. A constructor
attaches a new occurrence of each annotation, so one `Annotation` object can go to several
owners. A task can also refer to an occurrence that it takes from the owner, for example
`record.annotations[0]`.
- There is one annotation class, and its `span` gives its shape. No span means the whole record.
`span=TimePoint.seconds(...)` means a time offset. `span=TimeInterval.seconds(...)` means a region.
- For text that the connector reads from a source file, use `LazyText(function)` as the
annotation value. Import it from `timenet.types`. The function returns the string. If you know
Expand Down Expand Up @@ -242,11 +252,11 @@ class TSQAConnector(
)
for signal, values in enumerate(signals)
]
record = Record(sources=[Source("TSQA series", signals=time_series)])
dataset.add_record(record)
record.annotate(Annotation("task", value=row.task))
annotations = [Annotation("task", value=row.task)]
if row.label:
record.annotate(Annotation("label", value=row.label))
annotations.append(Annotation("label", value=row.label))
record = Record(sources=[Source("TSQA series", signals=time_series)], annotations=annotations)
dataset.add_record(record)
dataset.add_task(AnswerTask(inputs=[record], prompt=row.question, targets=row.answer))
return dataset

Expand Down
7 changes: 6 additions & 1 deletion docs/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,12 @@ Each Signal has exactly one owning Source. Do not attach one Signal to several S
TimeNet rejects that ambiguous ownership.

Signals can share immutable `TimeAxis` and `TimeSeriesSpec` objects. Annotation content can also be
reused. Each call to `annotate()` creates a separate occurrence for the selected owner.
reused. Each call to `annotate()`, and each owner that you build with `annotations=[...]`, creates a
separate occurrence for the owner.

Attach annotations when you build the owner: pass `annotations=[...]` to the dataset, Record, Source,
Signal, or Task. Use `annotate()` only for the cases that
[Attach annotations](data-model/annotations.md#attach-annotations) lists.

`Annotation` and the Task classes are final. To reuse an annotation, keep a constant or write a
builder function, as [Reuse content](data-model/annotations.md#reuse-content) shows. To reuse a
Expand Down
43 changes: 39 additions & 4 deletions docs/data-model/annotations.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,50 @@ An `Annotation` stores reusable content. An annotation occurrence attaches that
Task, Record, Source, Signal, or label set.

```python
from timenet.dataset import Record, TimeFDataset
from timenet.types import Annotation

condition = Annotation(
key="condition",
value="healthy",
description="Condition assigned by the source dataset.",
)
occurrence = record.annotate(condition)
record = Record(sources=(source,), annotations=[condition])
```

`key` can be positional: `Annotation("condition", value="healthy")`. All other fields are keyword-only.

## Attach annotations

Pass `annotations=[...]` when you build the owner. A dataset, a Task, a Record, a Source, and a
Signal all take this argument. It is a list, or any other iterable, of `Annotation` objects:

```python
dataset = TimeFDataset(
metadata=metadata,
annotations=[
Annotation("declared_sampling_rate_hz", value=20),
Annotation("collection_site", value="clinic-a"),
],
)
```

The dataset attaches each annotation in order, as `dataset.annotate()` does. A dataset annotation
cannot have a `span`. Every owner attaches a new occurrence of each annotation that you pass, as
`annotate()` does. So you can pass one `Annotation` object to several owners.

Use `annotate()` in these cases:

- The owner exists already, for example a Record that you import from a parent dataset.
- A Task refers to the annotation. Keep the occurrence that `annotate()` returns.

```python
occurrence = record.annotate(condition)
```

`annotate()` returns the attached occurrence. A Task must refer to this occurrence, not the unattached
content object.
content object. For an annotation that you passed to the constructor, take the occurrence from the
owner, for example `record.annotations[0]`.

## Content and occurrence fields

Expand Down Expand Up @@ -71,8 +101,13 @@ row are stored as one row again, but stored text and the same text in memory bec
An Annotation without a `span` describes its complete owner:

```python
record.annotate(Annotation("age", value=54, unit="years"))
record.annotate(Annotation("condition", value="healthy"))
record = Record(
sources=(source,),
annotations=[
Annotation("age", value=54, unit="years"),
Annotation("condition", value="healthy"),
],
)
```

Use `metadata` for untyped implementation details. Use an Annotation when consumers need a named,
Expand Down
9 changes: 7 additions & 2 deletions docs/data-model/records.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,17 @@ Use an [Annotation](annotations.md) for a typed, searchable fact. Examples inclu
firmware version, or an event on the timeline.

```python
from timenet.dataset import Record
from timenet.types import Annotation

record.annotate(Annotation("condition", value="healthy"))
record = Record(
sources=(source,),
annotations=[Annotation("condition", value="healthy")],
)
```

Sources and Signals also support `metadata` and `annotate()`.
Sources and Signals also take `metadata` and `annotations`. To annotate an owner that exists
already, call `annotate()`, as [Attach annotations](annotations.md#attach-annotations) shows.

## Add time information

Expand Down
3 changes: 2 additions & 1 deletion docs/data-model/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,8 @@ Names do not have to be unique. If two Signals below one Source have the same na

## Annotate a Source

`source.annotate(annotation)` attaches one occurrence and returns it. A static annotation
Pass `annotations=[...]` to the constructor to attach annotations when you build the Source.
`source.annotate(annotation)` attaches one occurrence to a Source that exists and returns it. A static annotation
describes the whole device, for example its firmware version. A timed annotation covers an
interval, for example a time when an electrode lost contact with the skin:

Expand Down
46 changes: 33 additions & 13 deletions docs/timef-dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ An irregular Signal also needs its time offsets. Pass `time_offsets_us` to
`time_offsets_us()`. Regular axes calculate their offsets from their cadence, and ordinal axes
describe order without a clock.

Call `signal.annotate(annotation)` to attach one occurrence to a Signal.
Pass `annotations=[...]` to the constructor to attach annotations when you build the Signal. Call
`signal.annotate(annotation)` to attach an occurrence to a Signal that exists.

## Source

Expand Down Expand Up @@ -110,8 +111,9 @@ with two parents, and a Signal with two owners.

Sources group Signals. All timed Signals in a Record use its common relative origin.

Call `source.annotate(annotation)` for a Source annotation. A selection attaches annotation content
to several Signals in one operation:
Pass `annotations=[...]` to the constructor to attach annotations when you build the Source. Call
`source.annotate(annotation)` to attach an occurrence to a Source that exists. A selection attaches
annotation content to several Signals in one operation:

```python
source.select(signals=(lead_i, lead_ii)).annotate(annotation)
Expand All @@ -128,9 +130,11 @@ annotations and metadata:

```python
from timenet.dataset import Record
from timenet.types import Annotation

record = Record(
sources=(imu,),
annotations=[Annotation("condition", value="healthy")],
metadata={"site": "lab-a", "session": "record-123"},
)
```
Expand All @@ -147,16 +151,19 @@ record = Record(
`record.signals` flattens every Signal in the hierarchy. `walk_sources()` and `walk_signals()` expose
the same deterministic traversal used by the writer.

Call `record.annotate(annotation)` to attach an occurrence. `time_point()` and `time_interval()`
Pass `annotations=[...]` to the constructor to attach annotations when you build the Record. Call
`record.annotate(annotation)` to attach an occurrence to a Record that exists. `time_point()` and `time_interval()`
convert wall-clock values against the Record origin. They require a known absolute timestamp.

Store subject identifiers as a list-valued record annotation. Omit it for a record with no subjects:

```python
from timenet.dataset import Record
from timenet.types import Annotation

record.annotate(
Annotation("subject_ids", value=["patient-1", "patient-2"])
record = Record(
sources=(imu,),
annotations=[Annotation("subject_ids", value=["patient-1", "patient-2"])],
)
```

Expand Down Expand Up @@ -197,12 +204,19 @@ Register complete Records independently of Tasks. This supports datasets with or

```python
from timenet.dataset import TimeFDataset
from timenet.types import Annotation

dataset = TimeFDataset(metadata=metadata)
dataset = TimeFDataset(
metadata=metadata,
annotations=[Annotation("source_revision", value="4f2a9c1")],
)
dataset.add_record(record)
dataset.add_task(task)
```

`annotations` is optional. It takes an iterable of `Annotation` objects and attaches each one to the
dataset in order, as `dataset.annotate()` does. A dataset annotation cannot have a `span`.

`add_record()` validates hierarchy ownership. It rejects a Record that is already in a dataset,
and a Source or Signal that another Record owns. It then gives the Record and its Signals their IDs.
`add_task()` validates referenced objects, the answer representation, and temporal scopes. A
Expand All @@ -213,17 +227,23 @@ inserts that stream once without retaining every Task in memory.

## Annotations

The same `Annotation` type can annotate a Dataset, Task, Record, Source, Signal, or label set. Annotation
content is immutable and reusable. Every call to `annotate()` creates a separate occurrence, so a
long payload can be stored once while its occurrences carry independent placement, confidence, and
provenance. `Annotation` is final: reuse content through a constant, or a builder function for
content that varies, as [Annotations](data-model/annotations.md#reuse-content) shows.
The same `Annotation` type can annotate a Dataset, Task, Record, Source, Signal, or label set. Each of
them takes `annotations=[...]` at construction. This is the usual way to attach annotations. Use
`annotate()` when the owner exists already, or when a Task refers to the occurrence that the call
returns.

Annotation content is immutable and reusable. Every call to `annotate()` and every constructor creates
a separate occurrence, so a long payload can be stored once while its occurrences carry independent
placement, confidence, and provenance. `Annotation` is final: reuse content through a constant, or a
builder function for content that varies, as [Annotations](data-model/annotations.md#reuse-content)
shows.

```python
from timenet.dataset import Record
from timenet.types import Annotation, TimePoint

male = Annotation("patient_sex", value="male")
record.annotate(male)
record = Record(sources=(imu,), annotations=[male])

lead_i.annotate(
Annotation(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -286,14 +286,18 @@ def convert(self, files: Mapping[str, Path]) -> TimeFDataset:
Raises:
TimeFValidationError: If source rows violate the SensorCaps mapping.
"""
dataset = TimeFDataset(metadata=self.metadata())
for key, value in (
("declared_sampling_rate_hz", DECLARED_SAMPLING_RATE_HZ),
("source_sampling_rate_hz", _SOURCE_SAMPLING_RATE_HZ),
("source_window_samples", _SOURCE_WINDOW_SAMPLES),
("time_axis_reconstruction", "round(linspace(0, 119, 60)) / 20 seconds relative to window start"),
):
dataset.annotate(Annotation(key, value=value))
dataset = TimeFDataset(
metadata=self.metadata(),
annotations=[
Annotation("declared_sampling_rate_hz", value=DECLARED_SAMPLING_RATE_HZ),
Annotation("source_sampling_rate_hz", value=_SOURCE_SAMPLING_RATE_HZ),
Annotation("source_window_samples", value=_SOURCE_WINDOW_SAMPLES),
Annotation(
"time_axis_reconstruction",
value="round(linspace(0, 119, 60)) / 20 seconds relative to window start",
),
],
)
records: dict[tuple[str, int], Record] = {}
for part, path in files.items():
count = 0
Expand Down Expand Up @@ -326,7 +330,6 @@ def _make_record(self, ref: _RowRef, row: _ParsedRow) -> Record:
)
for spec, name, sensor, dimension in channels
]
record = Record(sources=[Source("IMU", signals=signals)])
source = ref.part.rsplit("_", 1)[0]
metadata: dict[str, str | int] = {
"source_dataset": source,
Expand All @@ -337,9 +340,10 @@ def _make_record(self, ref: _RowRef, row: _ParsedRow) -> Record:
}
if row.location is not None:
metadata["sensor_location"] = row.location
for key, value in metadata.items():
record.annotate(Annotation(key, value=value))
return record
return Record(
sources=[Source("IMU", signals=signals)],
annotations=[Annotation(key, value=value) for key, value in metadata.items()],
)


CONNECTOR = SensorCapsConnector
Original file line number Diff line number Diff line change
Expand Up @@ -94,11 +94,10 @@ def _make_record(sample: TSQASample) -> Record:
time_axis=OrdinalAxis(),
)
signals.append(signal)
record = Record(sources=[Source("TSQA series", signals=signals)])
record.annotate(Annotation("task", value=sample.category))
annotations = [Annotation("task", value=sample.category)]
if sample.label:
record.annotate(Annotation("label", value=sample.label))
return record
annotations.append(Annotation("label", value=sample.label))
return Record(sources=[Source("TSQA series", signals=signals)], annotations=annotations)

@staticmethod
def _make_task(sample: TSQASample, record: Record, *, split: Split) -> AnswerTask:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -220,17 +220,14 @@ def create_record(self) -> Record:
if label in groups
]
sources.append(Source(label or UNTAGGED, signals=signals))
record = Record(
return Record(
metadata={"source_record_id": f"arfbench-{self.metric}"},
start_time=TimeOrigin(origin_us),
sources=sources,
)
record.add_annotations(
[
annotations=[
Annotation("metric_id", value=self.metric),
Annotation("incident_id", value=self.metric.partition("_")[0]),
Annotation("query_name", value=self.finest.read_query_name()),
Annotation("finest_interval_s", value=self.finest.interval_s, unit="second"),
]
],
)
return record
Original file line number Diff line number Diff line change
Expand Up @@ -166,10 +166,13 @@ def convert(self, source: PtbXlSource) -> TimeFDataset:
)
for position, name in enumerate(header.sig_name)
]
record = dataset.add_record(
Record(metadata={"ecg_id": identity}, sources=[Source("PTB-XL ECG", signals=signals)])
dataset.add_record(
Record(
metadata={"ecg_id": identity},
sources=[Source("PTB-XL ECG", signals=signals)],
annotations=annotations,
)
)
record.add_annotations(annotations)
except (KeyError, ValueError, SyntaxError, OSError, TimeNetConnectorError) as exc:
raise TimeNetConnectorError(
f"invalid PTB-XL row {line} (ecg_id {row.get('ecg_id')}) in {source.database_csv}: {exc}"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -299,10 +299,10 @@ def convert(self, source: SleepEdfxSource) -> TimeFDataset:
record = Record(
sources=[Source("Sleep-EDF polysomnograph", signals=series)],
time_span=TimeInterval.micros(0, session_end),
annotations=[Annotation("subject_ids", value=[recording.subject_id])],
metadata={"recording_id": recording.recording_id},
)
dataset.add_record(record)
record.annotate(Annotation("subject_ids", value=[recording.subject_id]))
annotations.attach(recording.recording_id, series, sleep_stages)
record.add_annotations(recording_metadata)

Expand Down
Loading