Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions docs/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,17 +99,15 @@ call:
the connectors stay out.

So a default build lands where `TimeNet()` reads. `local_registry` takes a local directory, a
`file://` URI or a `LocalRegistry`. A remote registry or a `LocalRegistry` with `read_only=True`
raises `TimeFValidationError`, because a build writes into its local registry. To get the release
to such a registry, build it locally and publish it with `timenet.publish()`.
`file://` URI or a `LocalRegistry`. A build rejects a remote registry or a `LocalRegistry` with
`read_only=True`, because a build writes into its local registry. To get the release to such a
registry, build it locally and publish it with `timenet.publish()`.

`sources` lists, in order, where the parents and payload sources that the local registry lacks
come from (see [Prepare parent releases](#prepare-parent-releases)). The order of the list is the
read order: a release that two sources hold comes from the first one.

`force=True` builds the release again, even if the local registry holds it. With
`open_access_only=True`, the build refuses a release without open access with
`TimeNetAccessError` before any step runs.
`force=True` builds the release again, even if the local registry holds it.

```python
import timenet
Expand All @@ -131,17 +129,17 @@ A published version cannot change. If `dest` already holds the same release, `pu
nothing.


A remote registry holds only releases with access declared `open`. Before
`timenet.publish()` copies anything to such a registry, it checks the card and the lineage of each
release.
A remote registry service decides which releases it accepts. The hosted registry accepts only
releases with access declared `open`. An `s3://` registry has no service. The owner of the
bucket decides through the bucket permissions who can publish to it.

## Clean code provenance for remote registries

The manifest of a release records the git commit of the TimeNet code that built it, and whether that
code had local changes (see [`build_env.git`](manifest.md)). A remote registry holds only releases
built from committed code. Before `timenet.publish()` copies anything to such a registry, it checks
the manifest of each release that it copies. If a manifest records local changes, `publish()` raises
`TimeNetBuildError` and copies nothing.
the manifest of each release that it copies. If a manifest records local changes, `publish()`
refuses the release and copies nothing.

The local registry accepts any release, so a build with local changes still lands there. To publish
it, commit the changes, build the release again with `force=True`, and publish it.
Expand Down
7 changes: 4 additions & 3 deletions docs/catalog/licensing.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,9 @@ agreement and sign in before you pull a record. TimeNet does not remove that ste

## Redistribution

The hosted TimeNet registry serves built datasets only when the dataset and every ancestor in its
lineage declare open access. For every other dataset, a connector downloads the raw source data
from the original source and builds the dataset on your machine. In both cases, the data keeps the
The hosted TimeNet registry accepts a built dataset only when its card declares open access. The
registry service checks the card of the dataset itself, not the cards of the ancestors in its
lineage. For every other dataset, a connector downloads the raw source data from the original
source and builds the dataset on your machine. In both cases, the data keeps the
license that its source grants. If you redistribute a converted dataset, keep its original license
with it.
4 changes: 2 additions & 2 deletions docs/cli/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ nothing. If the local registry holds another release under that version, the com
the build, and the hint names `--force`, which compares a fresh build with the published release.
While `--force` builds the release again in the local registry. It cannot overwrite a published
version. If the new build wrote the same files, the publish copies nothing. If it wrote other files,
the publish raises `TimeNetVersionConflictError`, and the change needs a new version.
the publish fails, and the change needs a new version.

A remote destination holds only releases built from committed TimeNet code, as
[Clean code provenance for remote registries](../build.md#clean-code-provenance-for-remote-registries)
Expand Down Expand Up @@ -108,7 +108,7 @@ builds, and publishes nothing.
| `2` | The command or one of its options is invalid, for example an unknown dataset ID, or a registry flag that names no registry. |

If an isolated build cannot make its environment, for example because `uv` is missing, the command
raises `TimeNetIsolationError`.
fails.

Unexpected programming errors keep their traceback.

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/timenet.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,4 @@ timenet download chengsenwang/tsqa # newest of the first source
timenet info chengsenwang/tsqa@latest # highest of all sources
```

If you pin a version that is not committed, the tool exits with a `TimeNetDatasetNotFoundError`.
If you pin a version that is not committed, the tool exits with an error.
30 changes: 13 additions & 17 deletions docs/client.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,12 +120,11 @@ does not find an older version that matches. A
It reads only for a row whose card matches the other filters, and only until `limit` rows match. A
connector card has no manifest, so it never matches these two filters.

The calls ask the sources one after another, in order. A registry without a catalog, such as an
S3 registry, is left out with a `CatalogWarning`. If no source has a catalog, the calls raise
`TimeNetNoCatalogError`, and its hint names `open('<id>@<version>')`, the read of an exact release.
If two sources list a dataset at the same version, the row of the earlier source stays. A listed
version can be newer than the version that `open()` reads without a version. To query one registry
alone, open it and ask it:
The calls ask the sources one after another, in order. A registry without a catalog, such as an S3
registry, is left out with a `CatalogWarning`. If no source has a catalog, the calls fail with a
hint that names `open('<id>@<version>')`, the read of an exact release. If two sources list a
dataset at the same version, the row of the earlier source stays. A listed version can be newer than
the version that `open()` reads without a version. To query one registry alone, open it and ask it:

```python
from timenet.registry import open_registry
Expand All @@ -145,11 +144,9 @@ nothing.
| `get_manifest(dataset_id, version=None)` | The [manifest](manifest.md) of the release. |
| `get_card(dataset_id, version=None)` | The card of the release. |

`get_manifest()` reads only the manifest, and checks no other file. A registry can list a
release without serving its files, for example a credentialed release on the hosted registry. Then
`get_manifest()` returns the listed manifest, which shows the access level and the access page of
each card. A release gets its manifest when it is built. So for a release that only
`InstalledConnectors()` provides, `get_manifest()` raises `TimeNetNotBuiltError`.
`get_manifest()` reads only the manifest, and checks no other file. A release gets its manifest
when it is built. So for a release that only `InstalledConnectors()` provides, `get_manifest()`
raises `TimeNetNotBuiltError`.

`get_card()` returns the card of the release. The card comes from the manifest, or from the
installed connector for a release that is not built yet. `TimeNetNotBuiltError` holds the release
Expand Down Expand Up @@ -265,14 +262,13 @@ local registry lacks or holds damaged. Each one comes from the first other regis
it, so they can come from different registries. If the local registry holds a complete copy of the
release, the call copies only the payload sources that it lacks.

A download never builds. For a release that only `InstalledConnectors()` provides, it raises
`TimeNetNotBuiltError`.
A download never builds. For a release that only `InstalledConnectors()` provides, it fails.

By default, the call uses a complete release in the local registry that matches the requested
dataset ID and version. If the local release is missing or damaged (a listed file is missing or has
another size), or you set `force=True`, the call copies the release from the next lookup source
that has it. If the local registry holds that version with other files or another card, the copy
raises `TimeNetVersionConflictError`.
fails.

`progress=` takes a function that receives the progress events of `timenet.progress`. For each
release that the call copies, the function receives `StepStarted`, then `BytesMoved` events as the
Expand Down Expand Up @@ -310,9 +306,9 @@ that an installed connector built. `source` names the registry, or the connector
version. To read it, call `store.read_origin(release)` on the `LocalRegistry`. It returns an
`Origin` from `timenet.registry.origin`, whose `kind` is an `OriginKind`.

A record that is not valid raises `TimeNetOriginError`, which names its first problem. To replace
the release and its record, get the release again: call `download(release, force=True)` if it came
from a registry, or `timenet.build(release, force=True)` if it was built.
`read_origin()` rejects a record that is not valid, and the error names its first problem. To
replace the release and its record, get the release again: call `download(release, force=True)` if
it came from a registry, or `timenet.build(release, force=True)` if it was built.

## Pin a version

Expand Down
27 changes: 13 additions & 14 deletions docs/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,9 @@ samples. It contains no custom allocation or imbalance-scoring algorithm.

Nonempty input needs at least as many distinct groups as folds. Scikit-learn rejects inputs where
every category has fewer samples than folds; a rare category can produce a warning when other
categories have enough samples. Invalid inputs raise `TimeFValidationError`. Empty input returns
empty configured partitions. Sample counts and category proportions remain approximate when groups
have unequal sizes; inspect `splits.counts` and category distributions before publishing.
categories have enough samples. The splitter rejects invalid inputs. Empty input returns empty
configured partitions. Sample counts and category proportions remain approximate when groups have
unequal sizes; inspect `splits.counts` and category distributions before publishing.

The same ordered input, callbacks, seed, and scikit-learn version reproduce the assignment; changing
input order or adding samples can change it. No global random state is modified. The connectors
Expand Down Expand Up @@ -233,13 +233,12 @@ automatically (needed only for gated or private sources). Downloaded source file
[Settings](client.md#settings)).

A credentialed dataset (a PhysioNet DUA-gated one, for example) declares `access: credentialed`
and an `access_url` on its card. TimeNet never hosts such data, so it is build-your-own: get access
at the `access_url`, set the provider credential in the environment (the way `HF_TOKEN` already
works), and run `timenet-build build <id>` yourself. If no other source of the client provides a
credentialed dataset, an `open`, `load`, or `download` from a hosted registry raises
`TimeNetAccessError` instead of serving bytes.
The error names each `access_url` that the dataset needs, with the release that sets it. A child
dataset can need the access pages of its ancestors.
and an `access_url` on its card. The hosted registry does not serve such data, so it is
build-your-own: get access at the `access_url`, set the provider credential in
the environment (the way `HF_TOKEN` already works), and run `timenet-build build <id>` yourself.
If no other source of the client provides a credentialed dataset, an `open`, `load`, or
`download` from a hosted registry fails instead of serving bytes. The `access_url` is on the card
of the connector.

## Downloading artifacts

Expand All @@ -261,10 +260,10 @@ atomically, through a `.part` temporary file. S3 downloads use boto3 and stay sy
already parallelizes the transfer of a single object. Both backends skip an existing destination.

Pin every source file outside the Hub. Use a versioned or commit-pinned URL, never a moving branch
such as `master`, and pass its `sha256`. A mismatch raises `TimeFFormatError`. A cached file is used
again only while it matches its pin. `ensure_archive` also takes a `filename` override, for URLs
whose path has no usable name, for example a trailing slash or a `/download` suffix. Call these
helpers from `download_async`:
such as `master`, and pass its `sha256`. A mismatch fails the download. A cached file is used again
only while it matches its pin. `ensure_archive` also takes a `filename` override, for URLs whose
path has no usable name, for example a trailing slash or a `/download` suffix. Call these helpers
from `download_async`:

```python
from timenet_connectors.download import Artifact, ensure_archive, download_files
Expand Down
16 changes: 6 additions & 10 deletions docs/data-model/time-series.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,7 @@ numpy_values = temperature.to_numpy()
window = temperature.read_steps(0, 2)
```

`to_arrow()` preserves nulls. `to_numpy()` raises `TimeFValidationError` when the loaded values
contain nulls.
`to_arrow()` preserves nulls. `to_numpy()` fails when the loaded values contain nulls.

For data with nulls, Arrow provides the validity mask:

Expand All @@ -82,9 +81,8 @@ values = temperature.to_arrow()
present = values.is_valid()
```

The mask marks present timesteps. The PyTorch adapter raises `TimeFValidationError` for actual nulls,
even when the spec has a `missing` record. Consumers must handle missing data before tensor
conversion.
The mask marks present timesteps. The PyTorch adapter rejects actual nulls, even when the spec has
a `missing` record. Consumers must handle missing data before tensor conversion.

## Cut a window

Expand All @@ -103,8 +101,7 @@ Which Signals a span applies to and which values it holds are separate questions

By default the span must lie inside the Signal: a `TimeInterval` inside the window `span_us`
reports, and a `StepInterval` that ends at or before `n_values`. A span that reaches past either
end raises `TimeFValidationError`. Pass `clip=True` to keep only the part of the span inside the
Signal instead.
end raises an error. Pass `clip=True` to keep only the part of the span inside the Signal instead.

`overlaps_with()` tells you whether a `TimeInterval` or `TimePoint` holds at least one value of a
Signal, so you can skip the Signals it misses, such as other segments of one recording. It uses the
Expand All @@ -113,9 +110,8 @@ between two values, or after the last one, holds no value. On an irregular Signa
`overlaps_with()` reads the time offsets, unless the span lies outside the window `span_us`
reports.

An ordinal Signal has no timeline. `overlaps_with()` and `slice()` raise `TimeFValidationError` for
a time span on it, so test `is_ordinal` first and scope an ordinal Signal in steps. `slice()` also
raises for a point span.
An ordinal Signal has no timeline. `overlaps_with()` and `slice()` reject a time span on it, so test
`is_ordinal` first and scope an ordinal Signal in steps. `slice()` also rejects a point span.

To cut every Signal of a Record that an interval selects, ask `is_selected_by()` first, and cut
with `clip=True`, because an interval can overlap a Signal only in part:
Expand Down
3 changes: 1 addition & 2 deletions docs/manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,7 @@ early development, and no TimeNet reads them.
raises Pydantic's `ValidationError` with its native field locations and error details.

Registries read a stored manifest with `read_manifest(path)` or `parse_manifest(text, where)` from
`timenet.manifest`. A manifest that is not valid raises `TimeNetManifestError`. A manifest of an
older or a newer unsupported TimeF format raises `TimeFFormatVersionError` instead.
`timenet.manifest`.

### Serialization notes

Expand Down
Loading