Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions benchmarks/end_to_end/corpus.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
DatasetMetadata,
Domain,
ForecastingTask,
Label,
LabelSet,
License,
TimeInterval,
Expand All @@ -31,10 +30,10 @@
from timenet.types.wire import ValueDtype


CONDITION = LabelSet(name="condition", labels=(Label(name="healthy"), Label(name="outer-race-fault")))
RHYTHM = LabelSet(name="rhythm", labels=(Label(name="normal-sinus"), Label(name="atrial-fibrillation")))
SLEEP_STAGE = LabelSet(name="sleep-stage", labels=tuple(Label(name=stage) for stage in ("W", "N1", "N2", "N3", "REM")))
MEAN_SIGN = LabelSet(name="mean-sign", labels=(Label(name="above-zero"), Label(name="below-zero")))
CONDITION = LabelSet(["healthy", "outer-race-fault"])
RHYTHM = LabelSet(["normal-sinus", "atrial-fibrillation"])
SLEEP_STAGE = LabelSet(["W", "N1", "N2", "N3", "REM"])
MEAN_SIGN = LabelSet(["above-zero", "below-zero"])


@dataclass(frozen=True)
Expand Down
21 changes: 20 additions & 1 deletion docs/data-model/annotations.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ tags:
# Annotations

An `Annotation` stores reusable content. An annotation occurrence attaches that content to a Dataset,
Task, Record, Source, or Signal.
Task, Record, Source, Signal, or label set.

```python
from timenet.types import Annotation
Expand Down Expand Up @@ -116,6 +116,25 @@ a separate occurrence with shared content. Other Signals remain unchanged.
Time spans use whole microseconds on the Record timeline. Step spans are reserved for
Tasks on ordinal Signals.

## Label set annotations

A `LabelSet` takes annotations like the other owners. Pass them to the constructor, or call
`annotate()` later. Use them for a name, a description, or other context. No key is required.

```python
from timenet.types import Annotation, LabelSet

SLEEP_STAGE = LabelSet(
["W", "N1", "R"],
annotations=[Annotation("name", value="sleep_stage")],
)
SLEEP_STAGE.annotate(Annotation(key="description", value="Sleep stage of one epoch."))
```

A label set annotation has no span. Annotations are part of the identity of a set. Two sets with the
same labels and different annotations are two sets, and TimeF stores each of them. TimeF stores sets
with the same labels and the same annotations once.

## Valid values

An annotation value can be a string, integer, float, Boolean, or list of strings. For example, a
Expand Down
7 changes: 2 additions & 5 deletions docs/data-model/records.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,9 @@ A [Task](tasks.md) refers directly to its input Records. Registering a Task does
Records.

```python
from timenet.types import ClassificationTask, Label, LabelSet
from timenet.types import ClassificationTask, LabelSet

CONDITION = LabelSet(
name="condition",
labels=(Label(name="healthy"), Label(name="faulty")),
)
CONDITION = LabelSet(["healthy", "faulty"])

task = ClassificationTask(
inputs=(record,),
Expand Down
52 changes: 30 additions & 22 deletions docs/data-model/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ A Task describes one modeling problem. It holds ordered input Records and one re
expected output.

```python
from timenet.types import ClassificationTask, Label, LabelSet
from timenet.types import ClassificationTask, LabelSet

CONDITION = LabelSet(name="condition", labels=(Label(name="healthy"), Label(name="faulty")))
CONDITION = LabelSet(["healthy", "faulty"])

task = ClassificationTask(
inputs=(record,),
Expand Down Expand Up @@ -85,12 +85,9 @@ A dataset that ships a partition marks each Task with its `split`. Records carry
own: one Record may serve Tasks of several splits.

```python
from timenet.types import ClassificationTask, Label, LabelSet, Split
from timenet.types import ClassificationTask, LabelSet, Split

ACTIVITY = LabelSet(
name="activity",
labels=(Label(name="walking"), Label(name="sitting")),
)
ACTIVITY = LabelSet(["walking", "sitting"])

dataset.add_task(
ClassificationTask(
Expand Down Expand Up @@ -224,26 +221,20 @@ dataset.add_task(age_question(record, 64))
```python
from timenet.types import Label, LabelSet

CONDITION = LabelSet(
name="condition",
labels=(Label(name="healthy"), Label(name="faulty", description="A bearing fault.")),
)
FINDINGS = LabelSet(
name="findings",
labels=(Label(name="AF"), Label(name="PVC"), Label(name="LBBB")),
multi_label=True,
)
CONDITION = LabelSet(["healthy", Label(name="faulty", description="A bearing fault.")])
FINDINGS = LabelSet(["AF", "PVC", "LBBB"], multi_label=True)

ClassificationTask(inputs=(record,), targets="healthy", label_set=CONDITION)
ClassificationTask(inputs=(record,), targets=("PVC", "AF"), label_set=FINDINGS)
```

A `LabelSet` defines the allowed labels for a classification target. The `multi_label` flag controls
how many labels a target can contain.
A `LabelSet` defines the allowed labels for a classification target. Like other owners, it can carry
[annotations](annotations.md#label-set-annotations). The `multi_label` flag controls how many labels
a target can contain.

If `multi_label=False`, the default, a target must contain exactly one label. If `multi_label=True`,
a target contains all labels that apply. Omitted labels do not apply. An empty target means that no
label applies. The order of labels does not matter.
label applies.

When the dataset adds or streams a task, it makes sure that the target follows the rules of its
label set. Each label must be in the set. No label can repeat. A target with `multi_label=False`
Expand All @@ -252,9 +243,26 @@ must contain exactly one label.
Labels come from the targets stored directly on the task or from the text of its target
annotations.

Tasks can share the same label set object. Each name identifies one label set within a dataset.
If two different sets have the same name, `add_task()`, the task stream, and the writer raise an
error. TimeF stores each set once. The reader gives tasks that use the same set a shared object.
A label set has no name. TimeF stores equal sets once. For this, two sets are equal if they have the
same labels, label descriptions, `multi_label`, `ordered`, and annotations. The order of labels
matters only for an ordered set. The order of annotations does not matter. `==` follows the same rule.

Annotations are equal if all their fields are equal, the value included. So these two sets are two
stored sets, each with its own id:

```python
LabelSet(["yes", "no"], annotations=[Annotation("name", value="afib_present")])
LabelSet(["yes", "no"], annotations=[Annotation("name", value="signal_noisy")])
```

By default the order of labels does not matter. Use `ordered=True` when it has a meaning, for example
for ordinal labels such as `low`, `medium` and `high`, or for answer options that a model sees in a
fixed order. Then each order is its own set, and the stored set keeps it. An unordered set keeps the
label order of the first equal set that the writer meets. Pass the labels as a list or a tuple. A
label set rejects a Python set, because its order changes from one run to the next.

The stored set gets one integer id, `label_set_id`. In one read, the reader gives tasks that use the
same set a shared object.

### Answer and rationale

Expand Down
6 changes: 3 additions & 3 deletions docs/timef-dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ inserts that stream once without retaining every Task in memory.

## Annotations

The same `Annotation` type can annotate a Dataset, Task, Record, Source, or Signal. Annotation
The same `Annotation` type can annotate a Dataset, Task, Record, Source, Signal, or label set. Annotation
content is immutable and reusable. Every call to `annotate()` creates a separate occurrence, so a
long payload can be stored once while its occurrences carry independent placement, confidence, and
provenance. `Annotation` is final: reuse content through a constant, or a builder function for
Expand All @@ -238,8 +238,8 @@ lead_i.annotate(

`derive_schema()` collects the time-series specifications, annotation descriptors, and concrete Task
types found in the dataset. Annotation descriptors come from every owner: the Dataset, Records,
Sources, Signals, and Tasks. The Tasks of a task stream are not read here. The writer adds their
keys to the schema while it writes them. It does not load Signal values.
Sources, Signals, Tasks, and label sets. The Tasks of a task stream are not read here. The writer
adds their keys to the schema while it writes them. It does not load Signal values.

Write a version with the declarative helper:

Expand Down
30 changes: 18 additions & 12 deletions docs/timef-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,13 +171,14 @@ The `annotation_occurrences` table records where the content applies. It points
the content through `content_id`. It also carries the span as `span_type`, `start_us`, and
`end_us`, plus provenance, confidence, and occurrence metadata.

An occurrence can annotate a Dataset, Task, Record, Source, or Signal. It names its owner with
`object_type` and `object_id`. For a Record, Signal, or Task, `object_id` is the entity ID of the
owner. For a Source, it is the storage ID of the Source. `object_id` is `NULL` exactly when the
An occurrence can annotate a Dataset, Task, Record, Source, Signal, or label set. It names its owner with
`object_type` and `object_id`. For a Record, Signal, Task, or label set, `object_id` is the ID of the
owner. For a Source, it is the storage ID of that row. `object_id` is `NULL` exactly when the
owner is the Dataset. IDs are unique only within one kind, so the type is part of the reference.

A read of every Record or every Task rejects an occurrence with an unknown owner type or a missing
owner. It does not drop that occurrence.
A read of every Record or every Task rejects an occurrence with an unknown owner type. It also
rejects an occurrence whose Record, Source, Signal, or Task owner is missing. It does not drop that
occurrence. Only `audit_control_database()` reports an occurrence whose label set is missing.

One content row can apply to many objects without copying a long value. Each attachment still has
its own `occurrence_id` and temporal placement.
Expand All @@ -188,15 +189,20 @@ its own `occurrence_id` and temporal placement.
Python subclass. Four typed columns store the scope: `scope_type`, `scope_start`, `scope_end`, and
`scope_signal_ids`.

Configuration for each task type uses the columns `label_set`, `unit`, `target_name`, and `mode`.
Configuration for each task type uses the columns `label_set_id`, `unit`, `target_name`, and `mode`.
A column that a task type does not use is `NULL`. SQL can filter these columns without parsing JSON.

Every classification task has a label set, and `tasks.label_set` stores its name. Each row in
`label_sets` stores one set with the fields `name`, `labels`, and `multi_label`. The name is unique
within a release.

The `labels` field holds one `{name, description}` entry per label, in label order. If a label has
no description, its `description` is `NULL`.
Every classification task has a label set, and `tasks.label_set_id` stores its ID. Each row in
`label_sets` stores one set with the fields `label_set_id`, `labels`, `multi_label`, and `ordered`.
Sets with the same labels, label descriptions, `multi_label`, `ordered`, and annotations are one set,
so a release stores them once. If `ordered` is true, the label order is part of the set, so the same
labels in another order are a different set. The same labels with different annotations are also a
different set. The writer numbers the sets in the order that tasks first use them.
The annotations of a set are rows of `annotation_occurrences` with `object_type` `LabelSet` and the
`label_set_id` as `object_id`.

The `labels` field holds one `{name, description}` entry per label, in the label order of the first
equal set that the writer meets. If a label has no description, its `description` is `NULL`.

Each inline target is one row in `task_targets`. Its `target_kind` names the typed value column
that is set. A target stores entity IDs directly: `record_id` for a Record, `signal_id` for a
Expand Down
4 changes: 2 additions & 2 deletions docs/timef-reader.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,8 +112,8 @@ Tasks have four useful read paths:
The Record, Signal, and Task columns hold the ids that the dataset assigned, for example
`task_id`, `input_record_ids`, `from_task_ids`, and `span_signal_ids`. `occurrence_id` and
`content_id` are storage row numbers of this version. The next write numbers them again. In
`read_annotation_table()`, `object_id` is a Record, Signal, or Task id, a storage row number for
a Source, or `NULL` for the dataset.
`read_annotation_table()`, `object_id` is a Record, Signal, or Task id, a label set id, a storage
row number for a Source, or `NULL` for the dataset.

One call returns one consistent set of objects. A Record that two tasks use is one object, and the
Signals on one irregular axis share one handle to its time offsets. `read_tasks(inputs=records)`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,11 @@
import csv
from dataclasses import dataclass
from enum import Enum
import hashlib
import json
from typing import TYPE_CHECKING, Self, TypeVar

from timenet.errors import TimeNetConnectorError
from timenet.types import Annotation, ClassificationTask, InputModality, Label, LabelSet, Split
from timenet.types import Annotation, ClassificationTask, InputModality, LabelSet, Split


if TYPE_CHECKING:
Expand Down Expand Up @@ -108,15 +107,6 @@ def _parse_answer_options(cell: str) -> tuple[str, ...]:
return tuple(options)


def _label_set_name(options: tuple[str, ...]) -> str:
"""Name a label set by its options, so that equal option lists share one set.

Returns:
``options_`` and the first 8 hex digits of the SHA-256 of the options as a JSON list.
"""
return "options_" + hashlib.sha256(json.dumps(options).encode()).hexdigest()[:8]


@dataclass(frozen=True)
class ARFBenchQuestion:
"""One row of the question table: a multiple-choice question about one or two metrics."""
Expand Down Expand Up @@ -216,31 +206,29 @@ def create_chart_task(self, chart_records: Mapping[str, Record]) -> Classificati
def _create_task(self, inputs: list[Record], modality: InputModality) -> ClassificationTask:
"""Create a test task that asks for the correct answer from the options of the question.

The options are the label set of the task. The set is named after a hash of the options, so
questions with the same options share one set. The row index in the metadata pairs the
series task with the chart task of ``datadog/arfbench-charts``. The interpolation flags
describe what an engineer saw. This method does not interpolate.
The options are the label set of the task. The set is ordered, because the order of the options is
part of the question. Questions with the same options in the same order share one set. The row
index in the metadata pairs the series task with the chart task of ``datadog/arfbench-charts``.
The interpolation flags describe what an engineer saw. This method does not interpolate.

Returns:
The task, not yet added to a dataset.
"""
task = ClassificationTask(
return ClassificationTask(
inputs=inputs,
prompt=self.question,
targets=self.correct_answer,
input_modalities=frozenset({InputModality.TEXT, modality}),
split=Split.TEST,
metadata={"question_index": self.index},
label_set=LabelSet(
name=_label_set_name(self.answer_options),
labels=tuple(Label(name=option) for option in self.answer_options),
),
label_set=LabelSet(self.answer_options, ordered=True),
annotations=[
Annotation("task_category", value=self.task_category.value),
Annotation("difficulty", value=self.difficulty.value),
Annotation("interpolate_1", value=self.interpolate_1),
Annotation("interpolate_2", value=self.interpolate_2),
],
)
task.annotate(Annotation("task_category", value=self.task_category.value))
task.annotate(Annotation("difficulty", value=self.difficulty.value))
task.annotate(Annotation("interpolate_1", value=self.interpolate_1))
task.annotate(Annotation("interpolate_2", value=self.interpolate_2))
return task


def read_questions(qa_csv: Path) -> Iterator[ARFBenchQuestion]:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,16 +107,20 @@ def test_each_question_is_a_classification_task_over_its_options(release):
}


def test_questions_with_equal_options_share_a_label_set(release):
def test_questions_with_equal_options_in_equal_order_share_a_label_set(release):
up_down = {**PRESENCE_ROW, "Unnamed: 0": "2", "options_str": '["Up", "Down"]', "correct_answer": "Up"}
write_qa_csv(_path(release, "questions").parent, [*ROWS, up_down])
presence, correlation, up_down_task = ARFBenchConnector().convert(release).get_all()
no_yes = {**PRESENCE_ROW, "Unnamed: 0": "3", "options_str": '["No", "Yes"]', "correct_answer": "No"}
write_qa_csv(_path(release, "questions").parent, [*ROWS, up_down, no_yes])
presence, correlation, up_down_task, no_yes_task = ARFBenchConnector().convert(release).get_all()

assert isinstance(presence, ClassificationTask) and isinstance(correlation, ClassificationTask)
assert isinstance(up_down_task, ClassificationTask)
assert isinstance(up_down_task, ClassificationTask) and isinstance(no_yes_task, ClassificationTask)
assert presence.label_set.ordered
assert presence.label_set == correlation.label_set
assert presence.label_set.name.startswith("options_")
assert up_down_task.label_set.name != presence.label_set.name
assert presence.label_set.annotations == ()
assert up_down_task.label_set != presence.label_set
assert no_yes_task.label_set != presence.label_set
assert no_yes_task.label_set.names == ("No", "Yes")


def test_the_dataset_round_trips_through_parquet_with_its_missing_timesteps(release, tmp_path):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from timenet.connectors import BaseConnector
from timenet.dataset import Record, RegularAxis, Signal, Source, TimeFDataset
from timenet.errors import TimeNetConnectorError
from timenet.types import ClassificationTask, Label, LabelSet, Split, TimeSeriesSpec, ureg
from timenet.types import Annotation, ClassificationTask, Label, LabelSet, Split, TimeSeriesSpec, ureg
from timenet_connectors.download import ensure_archive
from timenet_connectors.sources.readers import CsvDataReader, DataRow

Expand All @@ -35,17 +35,17 @@

# The label order is the order of the OpenTSLM training prompt.
ACTIVITY = LabelSet(
name="activity",
labels=(
Label(name="biking"),
Label(name="lying"),
Label(name="running"),
Label(name="sitting"),
Label(name="standing"),
Label(name="walking"),
[
"biking",
"lying",
"running",
"sitting",
"standing",
"walking",
Label(name="walking_down", description="Walking down stairs."),
Label(name="walking_up", description="Walking up stairs."),
),
],
annotations=[Annotation("name", value="activity")],
)
# The release states no unit. A resting window has a magnitude of about 9.8, which is gravity in m/s².
ACCELERATION = TimeSeriesSpec(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,15 @@
# The release writes the stage codes of the Rechtschaffen and Kales manual. The prompts and the
# rationales write the names in the descriptions.
SLEEP_STAGE = LabelSet(
name="sleep_stage",
labels=(
[
Label(name="W", description="Wake"),
Label(name="N1", description="Non-REM stage 1"),
Label(name="N2", description="Non-REM stage 2"),
Label(name="N3", description="Non-REM stage 3"),
Label(name="N4", description="Non-REM stage 4"),
Label(name="REM", description="REM sleep"),
),
],
annotations=[Annotation("name", value="sleep_stage")],
)
# OpenTSLM converted the microvolts of Sleep-EDF to volts and kept every second sample of 100 Hz.
EEG = TimeSeriesSpec(spec_type="eeg", name="EEG", unit_value=ureg.volt, dtype="float64")
Expand Down
Loading