diff --git a/docs/build.md b/docs/build.md index ba13ee693..e3dbb98b7 100644 --- a/docs/build.md +++ b/docs/build.md @@ -99,17 +99,15 @@ call: the connectors stay out. So a default build lands where `TimeNet()` reads. `local_registry` takes a local directory, a -`file://` URI or a `LocalRegistry`. A remote registry or a `LocalRegistry` with `read_only=True` -raises `TimeFValidationError`, because a build writes into its local registry. To get the release -to such a registry, build it locally and publish it with `timenet.publish()`. +`file://` URI or a `LocalRegistry`. A build rejects a remote registry or a `LocalRegistry` with +`read_only=True`, because a build writes into its local registry. To get the release to such a +registry, build it locally and publish it with `timenet.publish()`. `sources` lists, in order, where the parents and payload sources that the local registry lacks come from (see [Prepare parent releases](#prepare-parent-releases)). The order of the list is the read order: a release that two sources hold comes from the first one. -`force=True` builds the release again, even if the local registry holds it. With -`open_access_only=True`, the build refuses a release without open access with -`TimeNetAccessError` before any step runs. +`force=True` builds the release again, even if the local registry holds it. ```python import timenet @@ -131,17 +129,17 @@ A published version cannot change. If `dest` already holds the same release, `pu nothing. -A remote registry holds only releases with access declared `open`. Before -`timenet.publish()` copies anything to such a registry, it checks the card and the lineage of each -release. +A remote registry service decides which releases it accepts. The hosted registry accepts only +releases with access declared `open`. An `s3://` registry has no service. The owner of the +bucket decides through the bucket permissions who can publish to it. ## Clean code provenance for remote registries The manifest of a release records the git commit of the TimeNet code that built it, and whether that code had local changes (see [`build_env.git`](manifest.md)). A remote registry holds only releases built from committed code. Before `timenet.publish()` copies anything to such a registry, it checks -the manifest of each release that it copies. If a manifest records local changes, `publish()` raises -`TimeNetBuildError` and copies nothing. +the manifest of each release that it copies. If a manifest records local changes, `publish()` +refuses the release and copies nothing. The local registry accepts any release, so a build with local changes still lands there. To publish it, commit the changes, build the release again with `force=True`, and publish it. diff --git a/docs/catalog/licensing.md b/docs/catalog/licensing.md index a6d5b2246..91e344448 100644 --- a/docs/catalog/licensing.md +++ b/docs/catalog/licensing.md @@ -38,8 +38,9 @@ agreement and sign in before you pull a record. TimeNet does not remove that ste ## Redistribution -The hosted TimeNet registry serves built datasets only when the dataset and every ancestor in its -lineage declare open access. For every other dataset, a connector downloads the raw source data -from the original source and builds the dataset on your machine. In both cases, the data keeps the +The hosted TimeNet registry accepts a built dataset only when its card declares open access. The +registry service checks the card of the dataset itself, not the cards of the ancestors in its +lineage. For every other dataset, a connector downloads the raw source data from the original +source and builds the dataset on your machine. In both cases, the data keeps the license that its source grants. If you redistribute a converted dataset, keep its original license with it. diff --git a/docs/cli/build.md b/docs/cli/build.md index 8eee5b6dc..5d0d4d731 100644 --- a/docs/cli/build.md +++ b/docs/cli/build.md @@ -68,7 +68,7 @@ nothing. If the local registry holds another release under that version, the com the build, and the hint names `--force`, which compares a fresh build with the published release. While `--force` builds the release again in the local registry. It cannot overwrite a published version. If the new build wrote the same files, the publish copies nothing. If it wrote other files, -the publish raises `TimeNetVersionConflictError`, and the change needs a new version. +the publish fails, and the change needs a new version. A remote destination holds only releases built from committed TimeNet code, as [Clean code provenance for remote registries](../build.md#clean-code-provenance-for-remote-registries) @@ -108,7 +108,7 @@ builds, and publishes nothing. | `2` | The command or one of its options is invalid, for example an unknown dataset ID, or a registry flag that names no registry. | If an isolated build cannot make its environment, for example because `uv` is missing, the command -raises `TimeNetIsolationError`. +fails. Unexpected programming errors keep their traceback. diff --git a/docs/cli/timenet.md b/docs/cli/timenet.md index 74a595783..ecddd2b12 100644 --- a/docs/cli/timenet.md +++ b/docs/cli/timenet.md @@ -107,4 +107,4 @@ timenet download chengsenwang/tsqa # newest of the first source timenet info chengsenwang/tsqa@latest # highest of all sources ``` -If you pin a version that is not committed, the tool exits with a `TimeNetDatasetNotFoundError`. +If you pin a version that is not committed, the tool exits with an error. diff --git a/docs/client.md b/docs/client.md index a934b9683..556a5183c 100644 --- a/docs/client.md +++ b/docs/client.md @@ -120,12 +120,11 @@ does not find an older version that matches. A It reads only for a row whose card matches the other filters, and only until `limit` rows match. A connector card has no manifest, so it never matches these two filters. -The calls ask the sources one after another, in order. A registry without a catalog, such as an -S3 registry, is left out with a `CatalogWarning`. If no source has a catalog, the calls raise -`TimeNetNoCatalogError`, and its hint names `open('@')`, the read of an exact release. -If two sources list a dataset at the same version, the row of the earlier source stays. A listed -version can be newer than the version that `open()` reads without a version. To query one registry -alone, open it and ask it: +The calls ask the sources one after another, in order. A registry without a catalog, such as an S3 +registry, is left out with a `CatalogWarning`. If no source has a catalog, the calls fail with a +hint that names `open('@')`, the read of an exact release. If two sources list a +dataset at the same version, the row of the earlier source stays. A listed version can be newer than +the version that `open()` reads without a version. To query one registry alone, open it and ask it: ```python from timenet.registry import open_registry @@ -145,11 +144,9 @@ nothing. | `get_manifest(dataset_id, version=None)` | The [manifest](manifest.md) of the release. | | `get_card(dataset_id, version=None)` | The card of the release. | -`get_manifest()` reads only the manifest, and checks no other file. A registry can list a -release without serving its files, for example a credentialed release on the hosted registry. Then -`get_manifest()` returns the listed manifest, which shows the access level and the access page of -each card. A release gets its manifest when it is built. So for a release that only -`InstalledConnectors()` provides, `get_manifest()` raises `TimeNetNotBuiltError`. +`get_manifest()` reads only the manifest, and checks no other file. A release gets its manifest +when it is built. So for a release that only `InstalledConnectors()` provides, `get_manifest()` +raises `TimeNetNotBuiltError`. `get_card()` returns the card of the release. The card comes from the manifest, or from the installed connector for a release that is not built yet. `TimeNetNotBuiltError` holds the release @@ -265,14 +262,13 @@ local registry lacks or holds damaged. Each one comes from the first other regis it, so they can come from different registries. If the local registry holds a complete copy of the release, the call copies only the payload sources that it lacks. -A download never builds. For a release that only `InstalledConnectors()` provides, it raises -`TimeNetNotBuiltError`. +A download never builds. For a release that only `InstalledConnectors()` provides, it fails. By default, the call uses a complete release in the local registry that matches the requested dataset ID and version. If the local release is missing or damaged (a listed file is missing or has another size), or you set `force=True`, the call copies the release from the next lookup source that has it. If the local registry holds that version with other files or another card, the copy -raises `TimeNetVersionConflictError`. +fails. `progress=` takes a function that receives the progress events of `timenet.progress`. For each release that the call copies, the function receives `StepStarted`, then `BytesMoved` events as the @@ -310,9 +306,9 @@ that an installed connector built. `source` names the registry, or the connector version. To read it, call `store.read_origin(release)` on the `LocalRegistry`. It returns an `Origin` from `timenet.registry.origin`, whose `kind` is an `OriginKind`. -A record that is not valid raises `TimeNetOriginError`, which names its first problem. To replace -the release and its record, get the release again: call `download(release, force=True)` if it came -from a registry, or `timenet.build(release, force=True)` if it was built. +`read_origin()` rejects a record that is not valid, and the error names its first problem. To +replace the release and its record, get the release again: call `download(release, force=True)` if +it came from a registry, or `timenet.build(release, force=True)` if it was built. ## Pin a version diff --git a/docs/connectors.md b/docs/connectors.md index 84559bda1..be5251051 100644 --- a/docs/connectors.md +++ b/docs/connectors.md @@ -160,9 +160,9 @@ samples. It contains no custom allocation or imbalance-scoring algorithm. Nonempty input needs at least as many distinct groups as folds. Scikit-learn rejects inputs where every category has fewer samples than folds; a rare category can produce a warning when other -categories have enough samples. Invalid inputs raise `TimeFValidationError`. Empty input returns -empty configured partitions. Sample counts and category proportions remain approximate when groups -have unequal sizes; inspect `splits.counts` and category distributions before publishing. +categories have enough samples. The splitter rejects invalid inputs. Empty input returns empty +configured partitions. Sample counts and category proportions remain approximate when groups have +unequal sizes; inspect `splits.counts` and category distributions before publishing. The same ordered input, callbacks, seed, and scikit-learn version reproduce the assignment; changing input order or adding samples can change it. No global random state is modified. The connectors @@ -233,13 +233,12 @@ automatically (needed only for gated or private sources). Downloaded source file [Settings](client.md#settings)). A credentialed dataset (a PhysioNet DUA-gated one, for example) declares `access: credentialed` -and an `access_url` on its card. TimeNet never hosts such data, so it is build-your-own: get access -at the `access_url`, set the provider credential in the environment (the way `HF_TOKEN` already -works), and run `timenet-build build ` yourself. If no other source of the client provides a -credentialed dataset, an `open`, `load`, or `download` from a hosted registry raises -`TimeNetAccessError` instead of serving bytes. -The error names each `access_url` that the dataset needs, with the release that sets it. A child -dataset can need the access pages of its ancestors. +and an `access_url` on its card. The hosted registry does not serve such data, so it is +build-your-own: get access at the `access_url`, set the provider credential in +the environment (the way `HF_TOKEN` already works), and run `timenet-build build ` yourself. +If no other source of the client provides a credentialed dataset, an `open`, `load`, or +`download` from a hosted registry fails instead of serving bytes. The `access_url` is on the card +of the connector. ## Downloading artifacts @@ -261,10 +260,10 @@ atomically, through a `.part` temporary file. S3 downloads use boto3 and stay sy already parallelizes the transfer of a single object. Both backends skip an existing destination. Pin every source file outside the Hub. Use a versioned or commit-pinned URL, never a moving branch -such as `master`, and pass its `sha256`. A mismatch raises `TimeFFormatError`. A cached file is used -again only while it matches its pin. `ensure_archive` also takes a `filename` override, for URLs -whose path has no usable name, for example a trailing slash or a `/download` suffix. Call these -helpers from `download_async`: +such as `master`, and pass its `sha256`. A mismatch fails the download. A cached file is used again +only while it matches its pin. `ensure_archive` also takes a `filename` override, for URLs whose +path has no usable name, for example a trailing slash or a `/download` suffix. Call these helpers +from `download_async`: ```python from timenet_connectors.download import Artifact, ensure_archive, download_files diff --git a/docs/data-model/time-series.md b/docs/data-model/time-series.md index 9db374cb4..42dc86ca0 100644 --- a/docs/data-model/time-series.md +++ b/docs/data-model/time-series.md @@ -72,8 +72,7 @@ numpy_values = temperature.to_numpy() window = temperature.read_steps(0, 2) ``` -`to_arrow()` preserves nulls. `to_numpy()` raises `TimeFValidationError` when the loaded values -contain nulls. +`to_arrow()` preserves nulls. `to_numpy()` fails when the loaded values contain nulls. For data with nulls, Arrow provides the validity mask: @@ -82,9 +81,8 @@ values = temperature.to_arrow() present = values.is_valid() ``` -The mask marks present timesteps. The PyTorch adapter raises `TimeFValidationError` for actual nulls, -even when the spec has a `missing` record. Consumers must handle missing data before tensor -conversion. +The mask marks present timesteps. The PyTorch adapter rejects actual nulls, even when the spec has +a `missing` record. Consumers must handle missing data before tensor conversion. ## Cut a window @@ -103,8 +101,7 @@ Which Signals a span applies to and which values it holds are separate questions By default the span must lie inside the Signal: a `TimeInterval` inside the window `span_us` reports, and a `StepInterval` that ends at or before `n_values`. A span that reaches past either -end raises `TimeFValidationError`. Pass `clip=True` to keep only the part of the span inside the -Signal instead. +end raises an error. Pass `clip=True` to keep only the part of the span inside the Signal instead. `overlaps_with()` tells you whether a `TimeInterval` or `TimePoint` holds at least one value of a Signal, so you can skip the Signals it misses, such as other segments of one recording. It uses the @@ -113,9 +110,8 @@ between two values, or after the last one, holds no value. On an irregular Signa `overlaps_with()` reads the time offsets, unless the span lies outside the window `span_us` reports. -An ordinal Signal has no timeline. `overlaps_with()` and `slice()` raise `TimeFValidationError` for -a time span on it, so test `is_ordinal` first and scope an ordinal Signal in steps. `slice()` also -raises for a point span. +An ordinal Signal has no timeline. `overlaps_with()` and `slice()` reject a time span on it, so test +`is_ordinal` first and scope an ordinal Signal in steps. `slice()` also rejects a point span. To cut every Signal of a Record that an interval selects, ask `is_selected_by()` first, and cut with `clip=True`, because an interval can overlap a Signal only in part: diff --git a/docs/manifest.md b/docs/manifest.md index 8179f4fef..1c3a00c9a 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -114,8 +114,7 @@ early development, and no TimeNet reads them. raises Pydantic's `ValidationError` with its native field locations and error details. Registries read a stored manifest with `read_manifest(path)` or `parse_manifest(text, where)` from -`timenet.manifest`. A manifest that is not valid raises `TimeNetManifestError`. A manifest of an -older or a newer unsupported TimeF format raises `TimeFFormatVersionError` instead. +`timenet.manifest`. ### Serialization notes diff --git a/docs/registry.md b/docs/registry.md index 0f3d298da..20a81bb9b 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -54,7 +54,8 @@ which is how you point at a private or a dev deployment. The HTTP transport authenticates with a bearer token read from `$TIMENET_TOKEN`. When it is unset the SDK talks to the registry anonymously (no `Authorization` header), which is enough for public reads. -Set it to reach private datasets or to write: +Set it to reach private datasets or to write. The service decides which releases a token can read, +and which releases it accepts: ```bash export TIMENET_TOKEN=your-token @@ -87,7 +88,6 @@ Every backend implements this contract. | `get_manifest(dataset_id, version=None)` | A dataset's [manifest](manifest.md) (latest if `version` is `None`). Raises `TimeNetDatasetNotFoundError` for an unknown id/version. Raises `TimeNetManifestError` if the stored manifest is not valid, `TimeFFormatVersionError` if it has an unsupported older or newer TimeF format, or `TimeFFormatError` if its own id disagrees with the directory it was loaded from. | | `open_file(dataset_id, version, relpath)` | A file of a dataset version, opened for seekable binary reading (an object store must return a range-capable handle, not a forward-only stream). | | `location` | Where the registry is: the base URL of a remote registry, the `s3://bucket/prefix` root of an S3 registry, or the resolved root directory of a local registry. Each backend must define it. | -| `requires_open_access` | Whether the registry holds and serves only releases with open access. `False` only for `LocalRegistry`. | | `find_manifest(release)` | The manifest of an exact release, or `None` if the registry does not hold it. Raises `TimeFFormatError` if the stored manifest names another release. | | `download_release(reference, directory, *, force=False, manifest=None, progress_cb=None)` | Download the files of one release, without its payload sources, into a local directory. A failed download leaves no partial copy, and it keeps the copy that was there before. | | `open_release(reference, *, manifest=None)` | A `DatasetVersion` of one exact release without its payload sources: the parsed manifest and a handle to the files. Pass `manifest` if you already read it from this registry. A local registry raises `TimeNetFileError` for a missing or resized file. | @@ -104,8 +104,7 @@ over the versioned REST contract (`GET /api/v1/datasets`, `/api/v1/datasets/{id} `.../download/{relpath}`); `S3Registry` serves the same layout under an `s3://bucket/prefix` root. The S3 backend has no catalog: its `has_catalog` is `False`, and `list_datasets` and `search` raise `NotImplementedError`. `TimeNet.list()` and `TimeNet.search()` leave a registry without a catalog -out, with a `CatalogWarning` that names it. If no source of the list has a catalog, they raise -`TimeNetNoCatalogError`. +out, with a `CatalogWarning` that names it. If no source of the list has a catalog, they fail. A dataset id is an `org/name` pair, for example `chengsenwang/tsqa`. The id nests one level deep on disk, at `/chengsenwang/tsqa//`. Use lowercase ids to avoid casing clashes on @@ -122,7 +121,7 @@ names each one and pins the files it reads from it (see [Opening a version](timef-reader.md#opening-a-version) for the checks and their errors. - `TimeNet.download()` and `copy_release()` copy complete copies of the version and of each payload source, and check each source against the pins. If a downloaded file has another size or checksum - than its manifest records, the copy raises `TimeNetFileError` and keeps no partial copy. + than its manifest records, the copy fails and keeps no partial copy. - `put_dataset` refuses a version if the registry does not hold one of its payload sources. ### Remote reads @@ -153,15 +152,17 @@ A source gives one of these answers: | `FoundInRegistry(version, registry)` | The registry serves the release. | | `FoundInConnector(builder)` | An installed connector builds the release. `version` is the version that its card declares. | | `NotFound()` | The source does not know the release. | -| `NotServed(version, access, requirements)` | The source lists the release, but does not serve its files. `access` is the effective access of the release, and `requirements` names each card that is not open, with its access page. `access_error(dataset_id)` returns the `TimeNetAccessError` that names each page. | +| `NotServed(version, access, requirements, reason)` | The source lists the release, but does not serve its files to this caller. `access` is the access level of the release and `requirements` names the cards with their access pages, or `None` and `()` if the source does not say. `reason` is the reason that the source gives, or `None`. `access_error(dataset_id)` returns the matching `TimeNetAccessError`. | -The type `Found` covers `FoundInRegistry` and `FoundInConnector`. `FoundInRegistry` and -`NotServed` also carry the `manifest` that the source read, or `None`, so that you do not need to -fetch it again. +The type `Found` covers `FoundInRegistry` and `FoundInConnector`. `FoundInRegistry` also carries +the `manifest` that the source read, or `None`, so that you do not need to fetch it again. -A `LocalRegistry` serves every release that it holds, so it never answers `NotServed`. A registry -that is not local serves the files of a release only if its effective access is `open`, and -answers `NotServed` otherwise. +A registry does not check the access level of a release. A `LocalRegistry` and an `S3Registry` +serve every release that they hold, so they never answer `NotServed`. For an `S3Registry`, the +owner of the bucket decides through the bucket permissions who can read and write releases. A +`RemoteRegistry` asks its service. The service decides which releases a caller can read, and a +refusal becomes `NotServed` with the reason that the service gives. The service does not send the +access level or the access page, so `access` is `None`. `InstalledConnectors()` answers `Found` only for the version that the card of a connector declares, because a connector builds only that version. For any other request, it answers `NotFound`. @@ -212,7 +213,7 @@ so you can publish to a local directory, a remote registry, or an S3 bucket: | `requires_clean_code_provenance` | Whether the registry refuses releases built from locally modified TimeNet code. `False` only for `LocalRegistry`. Otherwise `put_dataset` and `put_release` raise `TimeNetBuildError` for a release whose manifest records local changes. | `LocalRegistry`, `RemoteRegistry` and `S3Registry` take `read_only=True`. A read-only registry -refuses each write with `TimeNetRegistryError`. +refuses each write. `put_dataset` writes the dataset with a [`TimeFWriter`](timef-writer.md) and checks each payload source of the new release in the registry that receives it. In a local registry, `force=True` @@ -238,16 +239,16 @@ release = registry.put_dataset(dataset) Every backend is a `WritableRegistry`. As a result, this type alone does not show if a backend is a directory the engine can write to or a remote stub. `local_registry_path(uri)` gives this -information. It returns the directory that a `file://` URI or a plain path names. It raises -`TimeNetRegistryError` for a remote scheme or an empty string. +information. It returns the directory that a `file://` URI or a plain path names. It rejects a +remote scheme or an empty string. ### Remote publishing `RemoteRegistry.put_dataset` compiles the dataset locally and then sends the files to the service. The data does not pass through the API. The service gives a presigned URL for each file, and the SDK uploads the file to the object store directly. You need a writer token in `$TIMENET_TOKEN`. The -service refuses an anonymous or read-only caller. `RemoteRegistry.put_release` runs steps 2 to 4 -for a release directory that already exists. +service refuses an anonymous or read-only caller. The service also decides which releases it +accepts. `RemoteRegistry.put_release` runs steps 2 to 4 for a release directory that already exists. 1. Compile into a temporary staging directory with a [`TimeFWriter`](timef-writer.md), deriving the schema first if the dataset has none. An already-committed version returns early. The writer @@ -294,8 +295,8 @@ registry.search(DatasetFilter.of(domain=Domain.CARDIOLOGY, tag="ecg", limit=10)) A `DatasetFilter` from `timenet.types` holds the filters and the limit. Each filter is a tuple, and an empty tuple sets no restriction. `DatasetFilter.of()` takes one value, a list, or `None` for each filter. The registry combines the filters with AND logic. `limit` caps the number of -results, with a default of 100. `limit=0` returns no results, and a negative limit raises -`TimeFValidationError`. +results, with a default of 100. `limit=0` returns no results. `DatasetFilter` rejects a negative +limit. `DatasetFilter.select()` does the filtering. It takes `ListedRelease` rows, each a card and the registry that lists it, and keeps their order. It checks the card first. A `task` or diff --git a/docs/timef-dataset.md b/docs/timef-dataset.md index 50493bc82..bcba92e39 100644 --- a/docs/timef-dataset.md +++ b/docs/timef-dataset.md @@ -380,8 +380,8 @@ copy, they stay removed. A new task copy that derives from a task copy in the ch copy. The import refuses a `TSCorrespondenceTask` without `candidate_records`, because the candidates of -such a task are all Records of the parent. For a child that streams its tasks, `tasks=` raises -`TimeFValidationError`. +such a task are all Records of the parent. For a child that streams its tasks, the import rejects +`tasks=`. Each call reads the selected tasks, the tasks that they derive from, and the Records that they reference in one batch, before the other Records. This read loads no stored values either. `load=` @@ -421,13 +421,13 @@ now, by the rules in [Signals](#signals), and did not keep before. Other annotat the child stay as they are. So an import of a whole Record completes a partial copy. The import checks a copy that it adds to, as for a new Record. For example, a new Signal must lie -inside the `time_span` of the copy. If a check fails, the import raises `TimeFValidationError` and -restores the copy as it was. +inside the `time_span` of the copy. If a check fails, the import fails and restores the copy as it +was. A Signal that an import adds goes below the copy of its Source. The import finds that copy through the Signals that it imported below the Source before, or else by the name of the Source. Sibling Sources have distinct names. If the child moved or renamed Sources of the copy, the import can find -another Source, or it raises `TimeFValidationError`. +another Source, or it fails. Do not assign `Source.signals` or `Source.sources` on a copy. A later import can then attach an annotation that the copy already holds a second time. To remove a Signal from a copy, use diff --git a/docs/timef-reader.md b/docs/timef-reader.md index a22afcc54..fcc55a2a9 100644 --- a/docs/timef-reader.md +++ b/docs/timef-reader.md @@ -43,7 +43,7 @@ Catalog connections close before the process forks, because DuckDB cannot share fork. Each one opens again on its next query, in the parent and in a forked data-loader worker alike, so a task stream that was in progress when the process forked continues. Forking while other threads use readers is not supported: a read that the fork's closing cuts short in another thread -raises `TimeFReaderError`. Use the `spawn` or `forkserver` start method there. +fails. Use the `spawn` or `forkserver` start method there. ## What loads when @@ -198,12 +198,12 @@ The reader translates the parent's ID to the local copy through the catalog, the from whichever release stores them. An ID that names nothing, an unknown alias or parent, and a stored object of another kind, such as -`get(AnswerTask, ...)` on a classification task, raise `TimeFNotFoundError`, which subclasses both -`TimeFValidationError` and `LookupError`. An edit such as `remove_records` can remove the copy of a -parent object. The parent ID of that object then names nothing, because the edit drops that import. -`get_many` raises it, naming every missing ID, before it returns anything. `Source`, `Annotation`, and any other kind not listed above, an ID -that is not a positive int64, a single string passed to `get_many`, and a `parent` that is neither a -`ParentAlias` nor a `DatasetRef` raise `TimeFValidationError`. +`get(AnswerTask, ...)` on a classification task, fail as not found. An edit such as +`remove_records` can remove the copy of a parent object. The parent ID of that object then names +nothing, because the edit drops that import. `get_many` fails before it returns anything, and the +error names every missing ID. The reader rejects `Source`, `Annotation`, and any other kind not +listed above, an ID that is not a positive int64, a single string passed to `get_many`, and a +`parent` that is neither a `ParentAlias` nor a `DatasetRef`. ## Loading values at read time @@ -345,10 +345,6 @@ releases that source reuses. | A payload source differs from its pins | `TimeFFormatError` | The source and the first differing file | | A file is missing or has another size | `TimeNetFileError` | The file, its expected size, and its actual size | -An invalid manifest raises `TimeNetManifestError`. A manifest or a `control.duckdb` with a format -version that this TimeNet does not read raises `TimeFFormatVersionError`. A missing manifest raises -`FileNotFoundError`. - ### Where payload sources are found Payload sources resolve relative to a registry root. By default, `open_local_version()` takes the @@ -359,7 +355,7 @@ sources. A version without payload sources opens from any directory. A version with payload sources that is not stored as `///` needs `sources_root=`, and opening it without one -raises `TimeFValidationError`. +fails. ## Errors and integrity @@ -368,19 +364,17 @@ chunk. Opening does not look for remote files. A missing remote `control.duckdb` read of Records or tasks. A missing remote values file fails when a Signal reads its chunk. Errors include the relevant file or table context. -A catalog query raises `TimeFFormatError` when the catalog lacks a table, column, or type that the -read expects, and `TimeFReaderError` when the reader's connection closed while the read ran, through -`close()` or a fork in another thread. Other DuckDB errors, such as I/O failures, running out of -memory, or an interrupt, propagate unchanged. +A catalog query fails when the catalog lacks a table, column, or type that the read expects, or +when the reader's connection closed while the read ran, through `close()` or a fork in another +thread. Other DuckDB errors, such as I/O failures, running out of memory, or an interrupt, +propagate unchanged. -A request for an object, ID, or parent alias that the dataset does not have raises -`TimeFNotFoundError`, a `TimeFValidationError`. A catalog that disagrees with its manifest raises -`TimeFFormatError`. For example, the catalog names no parent release for a parent alias of the -manifest, or it names another release. +A read fails when the catalog disagrees with its manifest. For example, the catalog names no parent +release for a parent alias of the manifest, or it names another release. Call `verify()` to hash every file before reading data: the version's own files, and the files it pins on each payload source. If a file is missing or does not match its manifest record or pin, -`verify()` raises `TimeNetFileError`, a `TimeFFormatError`. +`verify()` fails. ## Inherited storage diff --git a/docs/timef-writer.md b/docs/timef-writer.md index d7547c912..3da238e6e 100644 --- a/docs/timef-writer.md +++ b/docs/timef-writer.md @@ -83,15 +83,13 @@ local registry over `root`. The writer fetches the manifest of the source from t compares the record of each reused file. It refuses the write, before it writes the remaining values and with nothing published, when: -- the registry does not hold the source, which raises `TimeNetDatasetNotFoundError` naming the source - and the registry; +- the registry does not hold the source; - the registry's source records a reused file differently, as after a forced replace, or because the - registry holds another release under the same dataset ID and version, which raises - `TimeFFormatError` naming the source and the first differing file; or -- the source reads payloads of the release being written, which would make each the other's source - and raises `TimeFValidationError`. + registry holds another release under the same dataset ID and version; or +- the source reads payloads of the release being written, which would make each the other's source. -These are the errors that [opening](timef-reader.md#opening-a-version) raises for a missing or +The error names the source, and the first differing file when there is one. +[Opening](timef-reader.md#opening-a-version) a release fails in the same way for a missing or differing payload source. ## Values settings @@ -129,10 +127,10 @@ The writer fails before publication when it finds an invalid state, including: - a payload source that reads payloads of the output release; or - chunk rows that do not cover a Signal exactly. -These failures raise TimeNet validation errors. Where possible, the errors include object IDs and -relationship fields. The writer checks most failures while it streams rows. It uses the ID sets -that it owns. It does not keep the ID of every task, so after the last task batch it runs one query -on `tasks` to find a repeated task ID. +Where possible, the errors for these failures include object IDs and relationship fields. The +writer checks most failures while it streams rows. It uses the ID sets that it owns. It does not +keep the ID of every task, so after the last task batch it runs one query on `tasks` to find a +repeated task ID. Each Record and Signal that a task refers to, as an input, a target, or in a span, must be the same object that the dataset registered under that ID. Thus a Record from another dataset fails, diff --git a/docs/types.md b/docs/types.md index c3383b744..1f6f36dc0 100644 --- a/docs/types.md +++ b/docs/types.md @@ -36,7 +36,7 @@ Records, Signals, and Tasks have a read-only, positive int64 `id` (`EntityId`). object to a dataset, the dataset sets the ID. When the reader reads the object back, the reader sets it. You cannot set an ID. The dataset ID is the exception and stays an `org/name` string. -If an object is not in a dataset, reading its `id` raises `TimeFValidationError`. +If an object is not in a dataset, reading its `id` fails. IDs carry no meaning. Store upstream identifiers in `metadata` or in an annotation. To refer to another object, use the object itself. For example, a span holds Signal objects. diff --git a/docs/usage.md b/docs/usage.md index c5001face..43a36f3dd 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -108,10 +108,9 @@ with client.open_torch("timenet/hello-world") as view: targets = item["tasks"][0].targets ``` -The adapter preserves the Signal dtype and per-timestep shape. It raises `TimeFValidationError` -when values contain nulls, even when the spec's `missing` record allows them. Consumers must -handle missing data before conversion to PyTorch. Each mask in `series_masks` contains only true -values. +The adapter preserves the Signal dtype and per-timestep shape. It fails when values contain nulls, +even when the spec's `missing` record allows them. Consumers must handle missing data before +conversion to PyTorch. Each mask in `series_masks` contains only true values. Tasks and annotations remain Python objects. Signal lengths can also vary. Supply a model-specific `transform` or `collate_fn` when the default PyTorch batching cannot combine them. diff --git a/packages/timenet-connectors/src/timenet_connectors/builder/cli.py b/packages/timenet-connectors/src/timenet_connectors/builder/cli.py index b18703c55..15b0b1ee0 100644 --- a/packages/timenet-connectors/src/timenet_connectors/builder/cli.py +++ b/packages/timenet-connectors/src/timenet_connectors/builder/cli.py @@ -102,8 +102,7 @@ def build( TimeNetMissingParentError: If a parent is missing. The hint of the error names the flags. TimeNetFileError: If a copy of a parent in the local registry is damaged, or a copied file does not match its manifest record. The hint of the error names the flags. - TimeNetAccessError: If ``--publish-to`` is remote and gets a release without access declared - ``open``. + TimeNetAccessError: If a registry lists a parent but does not serve it to this caller. TimeNetVersionConflictError: If the copy to ``--publish-to`` finds other files of the version, or of a payload source of it, in that registry. Nothing is copied then. """ # ruff: ignore[docstring-extraneous-exception, escape-sequence-in-docstring] - raised by the build and the publish; the form feed ends the help text @@ -127,15 +126,7 @@ def build( if not dry_run: typer.echo(str(wanted.version)) return - # A destination that requires open access holds open releases only. So for such a destination, the - # plan refuses another release before any download. - plan = plan_release_build( - str(wanted), - target_registry=local, - parents_from=sources, - force=force, - open_access_only=dest is not None and dest.requires_open_access, - ) + plan = plan_release_build(str(wanted), target_registry=local, parents_from=sources, force=force) if dry_run: lines = _plan_lines(plan) if dest is not None: diff --git a/packages/timenet-connectors/tests/test_build_plan.py b/packages/timenet-connectors/tests/test_build_plan.py index f1939b1c3..6787ff927 100644 --- a/packages/timenet-connectors/tests/test_build_plan.py +++ b/packages/timenet-connectors/tests/test_build_plan.py @@ -35,8 +35,7 @@ from timenet.sources import InstalledConnectors from timenet.testing import install_backends, make_dataset from timenet.types import Access, TimeSeriesSpec, ureg -from timenet.types.access import AccessRequirement -from timenet.types.parent_dataset_card import ParentDatasetCard +from timenet.types.access import AccessRequirement, access_requirements from timenet.types.reference_keys import ParentAlias, SourceKey from timenet.types.references import DatasetRef from timenet.values_backends import ValuesBackend @@ -434,7 +433,10 @@ def test_a_damaged_parent_is_copied_again_but_never_built_over(tmp_path, monkeyp class _Hosted(WritableRegistry): - """A registry that is not local, over a local directory.""" + """A registry that is not local, over a local directory. + + Like a registry service for an anonymous caller, it lists a release whose card is not open without serving it. + """ def __init__(self, root): self._local = LocalRegistry(root) @@ -451,7 +453,12 @@ def list_datasets(self): return self._local.list_datasets() def get_manifest(self, dataset_id, version=None): - return self._local.get_manifest(dataset_id, version) + manifest = self._local.get_manifest(dataset_id, version) + card = manifest.metadata + if card.access is not Access.OPEN: + release = DatasetRef(dataset_id=card.dataset_id, version=card.dataset_version) + raise TimeNetAccessError(release, card.access, access_requirements(card)) + return manifest def open_file(self, dataset_id, version, relpath): return self._local.open_file(dataset_id, version, relpath) @@ -469,35 +476,6 @@ def put_release(self, from_directory): _GATED = {"access": Access.CREDENTIALED, "access_url": "https://example.org/dua"} -def test_open_access_only_refuses_a_parent_to_build_whose_card_is_not_open(tmp_path, monkeypatch): - cards = _install_cards(monkeypatch, child={"parent": PARENT}, parent={}) - cards[PARENT.dataset_id] = cards[PARENT.dataset_id].model_copy(update=_GATED) - target_registry = LocalRegistry(tmp_path / "store") - - with pytest.raises(TimeNetAccessError) as caught: - _parent_steps(CHILD.dataset_id, target_registry, parents_from=[InstalledConnectors()], open_access_only=True) - - error = caught.value - requirement = AccessRequirement(PARENT, Access.CREDENTIALED, _GATED["access_url"]) - assert (error.dataset, error.access, error.requirements) == (PARENT, Access.CREDENTIALED, (requirement,)) - - -def test_open_access_only_names_the_access_page_of_a_held_parent_from_its_lineage(tmp_path, monkeypatch): - cards = _install_cards(monkeypatch, child={"parent": PARENT}, parent={}) - target_registry = LocalRegistry(tmp_path / "store") - # The card of the held parent stays open. Its lineage holds GRAND, which is credentialed. - held = TimeFDataset(metadata=cards[PARENT.dataset_id]) - grand = make_dataset().metadata.model_copy(update=_GATED) - held._import_state.lineage[str(GRAND)] = ParentDatasetCard(metadata=grand) - target_registry.put_dataset(held) - - with pytest.raises(TimeNetAccessError) as caught: - _parent_steps(CHILD.dataset_id, target_registry, parents_from=[InstalledConnectors()], open_access_only=True) - - assert (caught.value.dataset, caught.value.access) == (PARENT, Access.CREDENTIALED) - assert caught.value.requirements == (AccessRequirement(GRAND, Access.CREDENTIALED, _GATED["access_url"]),) - - def _store_from_card(dataset_id: str, root: Path, *, force: bool) -> Path: """Store a dataset from its card and return its version directory. @@ -620,7 +598,7 @@ def child(command, env): assert local_registry.exists(GRAND.dataset_id, str(GRAND.version)) -@pytest.mark.parametrize("case", ["missing", "missing-buildable", "damaged", "not-open", "not-served"]) +@pytest.mark.parametrize("case", ["missing", "missing-buildable", "damaged", "not-served"]) def test_the_cli_hint_names_its_flags(tmp_path, monkeypatch, case): cards = _install_cards(monkeypatch, child={"parent": PARENT}, parent={}) monkeypatch.setattr(cli_module, "known_ids", lambda: list(cards)) @@ -636,10 +614,6 @@ def test_the_cli_hint_names_its_flags(tmp_path, monkeypatch, case): local_registry.put_dataset(TimeFDataset(metadata=cards[PARENT.dataset_id])) (release_dir(local_registry.root, PARENT) / "control.duckdb").write_bytes(b"cut short") options = ["--remote-registry", str(tmp_path / "remote registry")] - if case == "not-open": - cards[CHILD.dataset_id] = cards[CHILD.dataset_id].model_copy(update=_GATED) - monkeypatch.setattr(hints_module, "open_registry", lambda url, **_: _Hosted(tmp_path / "hosted")) - options = ["--publish-to", "s3://bucket/registry"] if case == "not-served": # The remote registry lists the parent without serving its files, and the list has no connectors. hosted = _Hosted(tmp_path / "hosted") @@ -662,11 +636,6 @@ def test_the_cli_hint_names_its_flags(tmp_path, monkeypatch, case): ), # No registry provides the parent, so the hint names no download. "damaged": (TimeNetFileError, f"Build it again with `timenet-build build test/parent{flags} --force`."), - "not-open": ( - TimeNetAccessError, - "_Hosted serves open releases only, so --publish-to takes open releases only. " - "Build it into the local registry without --publish-to instead.", - ), "not-served": (TimeNetAccessError, "Build it locally from its installed connector with --auto-build."), } error, hint = hints[case] @@ -692,25 +661,6 @@ def _publish_to(tmp_path, monkeypatch, cards) -> tuple[_Hosted, LocalRegistry, l return remote, LocalRegistry(tmp_path / "store"), calls -@pytest.mark.parametrize("gated", ["child", "parent"]) -def test_publish_to_refuses_a_credentialed_release_before_any_download(tmp_path, monkeypatch, gated): - cards = _install_cards(monkeypatch, child={"parent": PARENT}, parent={}) - gated_ref = CHILD if gated == "child" else PARENT - cards[gated_ref.dataset_id] = cards[gated_ref.dataset_id].model_copy(update=_GATED) - origin = LocalRegistry(tmp_path / "origin") - origin.put_dataset(TimeFDataset(metadata=cards[PARENT.dataset_id])) - _, local_registry, calls = _publish_to(tmp_path, monkeypatch, cards) - command = ["build", CHILD.dataset_id, "--publish-to", "s3://bucket/registry"] - - result = runner.invoke(app, [*command, "--remote-registry", str(origin.root), "--no-auto-build"]) - - # The build into the local registry gets the access checks of the destination. - assert isinstance(result.exception, TimeNetAccessError), result.output - assert result.exception.dataset == gated_ref - assert calls == [] - assert not local_registry.exists(PARENT.dataset_id, str(PARENT.version)) - - def test_publish_to_builds_into_the_local_registry_and_publishes(tmp_path, monkeypatch): cards = _install_cards(monkeypatch, child={"parent": PARENT}, parent={}) origin = LocalRegistry(tmp_path / "origin") diff --git a/packages/timenet/src/timenet/_hints.py b/packages/timenet/src/timenet/_hints.py index 14063ef05..c12080506 100644 --- a/packages/timenet/src/timenet/_hints.py +++ b/packages/timenet/src/timenet/_hints.py @@ -179,17 +179,6 @@ def leave_unreachable(self) -> str: def run_in_this_interpreter(self) -> str: """Return the flag or the variable that builds in this interpreter.""" - @abstractmethod - def publish_to_open_only(self, dest: WritableRegistry) -> str | None: - """Return the hint for a publish target that serves open releases only, or ``None``. - - Args: - dest: The registry that the call publishes to. - - Returns: - The hint, or ``None`` if the front end has no such target. - """ - class SdkWording(HintWording): """The parts of a hint for the SDK: calls and keywords.""" @@ -237,11 +226,6 @@ def leave_unreachable(self) -> str: def run_in_this_interpreter(self) -> str: return "TIMENET_ISOLATION=off" - @override - def publish_to_open_only(self, dest: WritableRegistry) -> str | None: - # The SDK names the target of a publish only in publish(), which has a hint of its own. - return None - @dataclass(frozen=True) class CliWording(HintWording): @@ -317,13 +301,6 @@ def leave_unreachable(self) -> str: def run_in_this_interpreter(self) -> str: return "--no-isolation" - @override - def publish_to_open_only(self, dest: WritableRegistry) -> str | None: - return ( - f"{dest} serves open releases only, so --publish-to takes open releases only. Build it into the local " - "registry without --publish-to instead." - ) - def _flags(self, remote_registry: str | None = None) -> str: """Return the registry flags of the command, for a command that it suggests.""" remote = remote_registry or (None if self.remote_registry is None else shlex.quote(self.remote_registry)) @@ -470,14 +447,7 @@ def _version(error: TimeNetVersionUnavailableError, facts: Facts, wording: HintW def _access(release: DatasetRef, facts: Facts, wording: HintWording) -> str: - """Return a hint that names the local build of a release that no source serves, or a registry for it.""" - dest = facts.dest - if dest is not None and dest.requires_open_access: - if facts.request is Operation.PUBLISH: - return f"{dest} serves open releases only. Publish it to a registry of your own, such as a local directory." - refused = wording.publish_to_open_only(dest) - if refused is not None: - return refused + """Return a hint that names the local build of a release that no source serves.""" build = wording.build(release.dataset_id, facts) card = builders.find_card(release.dataset_id) if card is None or card.dataset_version != release.version: diff --git a/packages/timenet/src/timenet/client.py b/packages/timenet/src/timenet/client.py index 89defd20e..a3d7eebaa 100644 --- a/packages/timenet/src/timenet/client.py +++ b/packages/timenet/src/timenet/client.py @@ -255,7 +255,7 @@ def get_manifest(self, dataset_id: str, version: str | None = None) -> Manifest: TimeNetNotBuiltError: If the release needs a build to create its manifest. TimeNetDatasetNotFoundError: If no source provides the release. TimeNetVersionUnavailableError: If the version is unavailable and a connector declares another version. - TimeNetAccessError: If a registry only lists the release and provides no manifest. + TimeNetAccessError: If a registry lists the release but does not serve it. TimeNetInvalidCardError: If the connector's card is unreadable or invalid. TimeFValidationError: If ``dataset_id`` and ``version`` both give a version. """ # ruff: ignore[docstring-extraneous-exception] (raised by the resolution of the release) @@ -281,7 +281,7 @@ def get_card(self, dataset_id: str, version: str | None = None) -> DatasetMetada Raises: TimeNetDatasetNotFoundError: If no source provides the release. TimeNetVersionUnavailableError: If the version is unavailable and a connector declares another version. - TimeNetAccessError: If a registry only lists the release and provides no manifest. + TimeNetAccessError: If a registry lists the release but does not serve it. TimeNetInvalidCardError: If the connector's card is unreadable or invalid. TimeFValidationError: If ``dataset_id`` and ``version`` both give a version. """ # ruff: ignore[docstring-extraneous-exception] (raised by the resolution of the release) @@ -365,7 +365,7 @@ def download( Raises: TimeFValidationError: If the sources have no writable local registry. - TimeNetAccessError: If a required release is only listed or lacks open access in a remote registry. + TimeNetAccessError: If a registry lists a required release but does not serve it. TimeNetNotBuiltError: If only installed connectors provide the release. TimeNetDatasetNotFoundError: If no registry other than the local registry provides the release. TimeNetVersionUnavailableError: If the version is missing and its connector builds another version. @@ -502,16 +502,14 @@ def _manifest_or_connector(self, dataset_id: str, version: str | None) -> Manife Raises: TimeNetDatasetNotFoundError: If no source provides the release. - TimeNetAccessError: If a registry only lists the release and provides no manifest. + TimeNetAccessError: If a registry lists the release but does not serve it. """ # ruff: ignore[docstring-extraneous-exception] (raised by raise_not_found) lookups = Lookups() answer = lookups.resolve(self._sources, dataset_id, version) if isinstance(answer, NotFound): raise_not_found(dataset_id, version, self._sources) if isinstance(answer, NotServed): - if answer.manifest is None: - raise answer.access_error(dataset_id) - return answer.manifest + raise answer.access_error(dataset_id) if isinstance(answer, FoundInConnector): return answer manifest = lookups.manifest(answer.registry, DatasetRef(dataset_id=dataset_id, version=answer.version)) diff --git a/packages/timenet/src/timenet/coordinator/build.py b/packages/timenet/src/timenet/coordinator/build.py index 43a2df98d..ab104fea1 100644 --- a/packages/timenet/src/timenet/coordinator/build.py +++ b/packages/timenet/src/timenet/coordinator/build.py @@ -20,7 +20,6 @@ def plan_release_build( target_registry: LocalRegistry, parents_from: Sequence[ReleaseSource] = (), force: bool = False, - open_access_only: bool = False, ) -> Plan: """Plan a release build in a local registry. @@ -32,7 +31,6 @@ def plan_release_build( target_registry: The local registry to write to. parents_from: Sources to copy or build missing dependencies from, in preference order. force: Plan a forced rebuild and its parents even if the requested release already exists. - open_access_only: Require ``access: open`` for every needed release. Returns: A plan with steps in execution order. @@ -49,7 +47,6 @@ def plan_release_build( target_registry, parents_from=parents_from, force_build=force, - open_access_only=open_access_only, ) return Plan(tuple(steps)) diff --git a/packages/timenet/src/timenet/coordinator/plan.py b/packages/timenet/src/timenet/coordinator/plan.py index 83975a146..38e7bffd9 100644 --- a/packages/timenet/src/timenet/coordinator/plan.py +++ b/packages/timenet/src/timenet/coordinator/plan.py @@ -30,13 +30,11 @@ FoundInConnector, FoundInRegistry, InstalledConnectors, - NotFound, NotServed, ReleaseLookup, ReleaseSource, resolve_release, ) -from timenet.types.access import require_open_access from timenet.types.references import DatasetRef from timenet.types.version import Version @@ -180,7 +178,7 @@ def manifest(self, registry: BaseRegistry, reference: DatasetRef) -> Manifest | key = (id(registry), reference) if key not in self.manifests: answer = self.answers.get(key) - if isinstance(answer, FoundInRegistry | NotServed) and answer.manifest is not None: + if isinstance(answer, FoundInRegistry) and answer.manifest is not None: validate_manifest_reference(answer.manifest, reference) self.manifests[key] = answer.manifest else: @@ -206,8 +204,6 @@ class _Graph: """The local registry for copies and builds, or ``None`` for a read that copies and builds nothing.""" sources: Sequence[ReleaseSource] """The sources in preference order. A read reads the target registry like any other source.""" - open_only: bool - """True if each release that the walk visits must declare ``access: open``.""" builds: bool = True """Whether the plan allows builds from installed connectors.""" releases: dict[DatasetRef, _Release] = field(default_factory=dict) @@ -337,8 +333,6 @@ def local(self, reference: DatasetRef, needed_by: DatasetRef | None, role: Role, raise TimeFValidationError(f"{reference} must be in a local registry, but the read has none") held, damage = self.in_target(reference) if held is not None and not force: - if self.open_only: - require_open_access(held.metadata, held.lineage) sources = list(held.payload_sources.values()) if role == "dataset" else [] self.releases[reference] = _Release(role, True, None, tuple(source.release for source in sources)) for source in sources: @@ -385,20 +379,14 @@ def copy(self, reference: DatasetRef, role: Role, registry: BaseRegistry, *, for manifest = self.lookups.manifest(registry, reference) if manifest is None: raise TimeNetDatasetNotFoundError(f"{registry} no longer holds {reference}") - if self.open_only: - require_open_access(manifest.metadata, manifest.lineage) sources = CopySources() if role == "dataset": sources = find_payload_sources( reference, manifest, - registry, - find=lambda source: self.listed(registry, source), + find=lambda source: self.served(registry, source), held=lambda source: self.in_target(source)[0], ) - if self.open_only: - for found in (*sources.fetched.values(), *sources.held.values()): - require_open_access(found.metadata, found.lineage) step = CopyStep(reference, registry, with_payload_sources=role == "dataset", force=force) # The node holds its final dependencies before the walk visits them. So a release of the walk # that depends on it closes a cycle and does not look it up again. @@ -415,8 +403,6 @@ def build(self, reference: DatasetRef, role: Role, builder: InstalledBuilder) -> builder: The installed connector that builds the release. """ card = builder.card - if self.open_only: - require_open_access(card) parents = tuple(card.parents.values()) self.releases[reference] = _Release(role, True, BuildStep(reference, builder), parents) # A build opens each of its parents with the payload sources of that parent. @@ -444,13 +430,13 @@ def missing( described = describe_searched_sources(searched, reference) return TimeNetMissingParentError(reference, needed_by, described, payload_source=role == "payload-source") - def listed(self, registry: BaseRegistry, reference: DatasetRef) -> Manifest | None: - """Get a release's manifest if the registry holds the release, also if it does not serve it. + def served(self, registry: BaseRegistry, reference: DatasetRef) -> Manifest | None: + """Get a release's manifest if the registry serves the release. Returns: - The manifest, or ``None`` if the registry does not hold the release. + The manifest, or ``None`` if the registry lacks the release or does not serve it to this caller. """ - if isinstance(self.lookups.ask(registry, reference), NotFound): + if not isinstance(self.lookups.ask(registry, reference), FoundInRegistry): return None return self.lookups.manifest(registry, reference) @@ -598,7 +584,6 @@ def plan_steps( *, parents_from: Sequence[ReleaseSource], force_build: bool, - open_access_only: bool, ) -> tuple[DatasetRef, list[Step]]: """Plan the steps to build a dataset and its dependencies. @@ -609,23 +594,17 @@ def plan_steps( target_registry: The destination registry. parents_from: The sources to search for missing releases, in order. force_build: Rebuild the requested release, even if the target registry holds it. - open_access_only: Require open access for every release. Returns: The requested release and its steps, with its own build last if needed. """ card = builder.card - if open_access_only: - require_open_access(card) root = builder.release # The graph leaves the target registry out of the sources of its copies and builds. - graph = _Graph(target_registry, parents_from, open_only=open_access_only) + graph = _Graph(target_registry, parents_from) # If the target registry already holds the requested release, a reader needs only its payload sources to # open it. Its declared parents matter only to a rebuild. held, damage = (None, None) if force_build else graph.in_target(root) - if held is not None and open_access_only: - # The installed card has no lineage, so the check above does not see an ancestor that is not open. - require_open_access(held.metadata, held.lineage) # The graph holds the requested release before the walk starts. So a release that depends on it # closes a cycle, and the walk does not look it up again. if held is None: @@ -674,7 +653,7 @@ def plan_download( TimeFFormatError: If a payload source differs from the files that a release pins. """ # ruff: ignore[docstring-extraneous-exception] - raised by the walk and as values others = [source for source in sources if not is_same_registry(source, target_registry)] - graph = _Graph(target_registry, others, open_only=False, builds=False, lookups=lookups or Lookups()) + graph = _Graph(target_registry, others, builds=False, lookups=lookups or Lookups()) graph.visit(release, None, needed_in_target=True, force=force) graph.check() return Plan(tuple(graph.create_ordered_steps())) diff --git a/packages/timenet/src/timenet/coordinator/read.py b/packages/timenet/src/timenet/coordinator/read.py index ae55bf2ad..917cf8fbe 100644 --- a/packages/timenet/src/timenet/coordinator/read.py +++ b/packages/timenet/src/timenet/coordinator/read.py @@ -122,7 +122,7 @@ def _walk( Raises: TimeNetDatasetNotFoundError: If no source provides the release. """ # ruff: ignore[docstring-extraneous-exception] - raised by raise_not_found - graph = _Graph(target_registry, sources, open_only=False) + graph = _Graph(target_registry, sources) answer = graph.lookups.resolve(sources, dataset_id, version) if isinstance(answer, NotFound): if not is_latest(version): diff --git a/packages/timenet/src/timenet/errors.py b/packages/timenet/src/timenet/errors.py index e92fabf27..95a33d8e3 100644 --- a/packages/timenet/src/timenet/errors.py +++ b/packages/timenet/src/timenet/errors.py @@ -157,24 +157,31 @@ def _message(self) -> str: class TimeNetAccessError(TimeNetError): - """A dataset needs credentialed or restricted access, so only the user's own registry may hold its data. + """A registry does not serve a dataset release to this caller. Attributes: dataset: The release. - access: Its effective access level: the strictest level of its card and of the cards in its - lineage. See :func:`~timenet.types.access.effective_access`. - requirements: Each card, of the release itself or of an ancestor, whose access is not open, - with its access level and its ``access_url``. A user needs access at each of them. See - :func:`~timenet.types.access.access_requirements`. + access: The access level of the release, or ``None`` if the registry does not say. + requirements: The cards that are not open access, with their ``access_url``. + reason: The reason that the registry gives, or ``None``. """ - def __init__(self, dataset: "DatasetRef", access: "Access", requirements: "tuple[AccessRequirement, ...]") -> None: - super().__init__(dataset, access, requirements) + def __init__( + self, + dataset: "DatasetRef", + access: "Access | None", + requirements: "tuple[AccessRequirement, ...]" = (), + reason: str | None = None, + ) -> None: + super().__init__(dataset, access, requirements, reason) self.dataset = dataset self.access = access self.requirements = requirements + self.reason = reason def _message(self) -> str: + if self.access is None: + return f"{self.dataset} is not served to this caller: {self.reason}" message = f"{self.dataset} is {self.access.value}." if self.requirements: pages = [f"{item.access_url} for {item.release}" for item in self.requirements] diff --git a/packages/timenet/src/timenet/producer.py b/packages/timenet/src/timenet/producer.py index a1f2e8355..6ed2f4a1d 100644 --- a/packages/timenet/src/timenet/producer.py +++ b/packages/timenet/src/timenet/producer.py @@ -27,7 +27,6 @@ def build( # ruff: ignore[too-many-arguments] local_registry: str | Path | LocalRegistry | None = None, sources: Sequence[ReleaseSource | str | Path] | None = None, force: bool = False, - open_access_only: bool = False, keep_cache: bool = False, progress: ProgressCallback | None = None, ) -> Path: @@ -43,7 +42,6 @@ def build( # ruff: ignore[too-many-arguments] sources: Sources, registry URLs, or paths for missing releases, or ``None`` for :func:`~timenet.client.default_sources` with ``local_registry`` first. force: Build the release again, even if the local registry holds it. - open_access_only: Require open access for every release before running the plan. keep_cache: Keep the raw downloads in ``TIMENET_CACHE`` after the build. progress: A callback for copy and build progress events. @@ -56,7 +54,7 @@ def build( # ruff: ignore[too-many-arguments] TimeNetDatasetNotFoundError: If no installed connector builds the dataset. TimeNetVersionUnavailableError: If the requested version differs from the connector's version. TimeNetMissingParentError: If no source provides a required parent or payload source. - TimeNetAccessError: If no source grants access to a parent or a release violates ``open_access_only``. + TimeNetAccessError: If a source lists a parent but no source serves it. TimeNetVersionConflictError: If a release to copy has other files than the same version in the local registry. TimeNetFileError: If a stored release is damaged and no registry in ``sources`` provides it. TimeNetBuildError: If a build fails. @@ -72,9 +70,7 @@ def build( # ruff: ignore[too-many-arguments] with explain(Facts([target, *parents], target, request=Operation.BUILD, wanted=wanted)): # A connector builds only the version of its card, so "latest" asks for that version. builder = declared(wanted.dataset_id, None if is_latest(wanted.version) else wanted.version) - release, steps = plan_steps( - builder, target, parents_from=parents, force_build=force, open_access_only=open_access_only - ) + release, steps = plan_steps(builder, target, parents_from=parents, force_build=force) run_plan(Plan(tuple(steps)), target, options=BuildOptions(keep_cache=keep_cache, progress=progress)) return release_dir(target.root, release) @@ -104,7 +100,7 @@ def publish( Raises: TimeNetRegistryError: If ``dest`` or ``source`` fails to open as a registry, or ``dest`` is read-only. TimeNetDatasetNotFoundError: If ``source`` lacks the release. - TimeNetAccessError: If a registry withholds files or requires open access that a copied release lacks. + TimeNetAccessError: If the source registry does not serve a release to this caller. TimeNetBuildError: If ``dest`` is not local and a copied release records TimeNet code with local changes. TimeNetVersionConflictError: If ``dest`` holds different files for the release or a required payload source. TimeNetMissingParentError: If a required payload source is missing from ``source`` and incomplete in ``dest``. @@ -142,7 +138,7 @@ def _publish_from( Raises: TimeNetDatasetNotFoundError: If ``origin`` lacks the release. - TimeNetAccessError: If a registry withholds files or requires open access that a copied release lacks. + TimeNetAccessError: If the source registry does not serve a release to this caller. TimeNetBuildError: If ``dest`` is not local and a copied release records TimeNet code with local changes. TimeNetVersionConflictError: If ``dest`` holds different files for the release or a required payload source. TimeNetMissingParentError: If a required payload source is missing from ``origin`` and incomplete in ``dest``. diff --git a/packages/timenet/src/timenet/registry/_copy.py b/packages/timenet/src/timenet/registry/_copy.py index 844303d5b..c899d267d 100644 --- a/packages/timenet/src/timenet/registry/_copy.py +++ b/packages/timenet/src/timenet/registry/_copy.py @@ -8,7 +8,12 @@ from pathlib import Path import tempfile -from timenet.errors import TimeFValidationError, TimeNetMissingParentError, TimeNetVersionConflictError +from timenet.errors import ( + TimeFValidationError, + TimeNetAccessError, + TimeNetMissingParentError, + TimeNetVersionConflictError, +) from timenet.manifest import Manifest from timenet.progress import BytesMoved, ProgressCallback, StepFinished, StepStarted from timenet.registry._paths import release_dir @@ -16,7 +21,6 @@ from timenet.registry.dependencies import find_payload_sources, validate_manifest_reference from timenet.registry.local import LocalRegistry from timenet.registry.writable import WritableRegistry -from timenet.types.access import require_open_access from timenet.types.references import DatasetRef @@ -49,7 +53,7 @@ def copy_release( # ruff: ignore[too-many-arguments] TimeNetRegistryError: If ``dest`` is read-only. TimeFValidationError: If ``force`` is set and ``dest`` is not local. TimeNetVersionConflictError: If ``dest`` holds the same release with other files or another card. - TimeNetAccessError: If a release lacks open access and either registry requires it. + TimeNetAccessError: If ``source`` does not serve a release to this caller. TimeNetBuildError: If ``dest`` is not local and a copied release records TimeNet code with local changes. TimeNetDatasetNotFoundError: If ``source`` lacks the release. TimeNetMissingParentError: If neither registry holds a complete payload source. @@ -67,7 +71,6 @@ def copy_release( # ruff: ignore[too-many-arguments] for ref, manifest in manifests.items() if _needs_copy(reference, ref, manifest, source, dest) or (force and ref == reference) ] - _check_access(manifests, set(pending), source, dest) for ref in pending: dest.require_clean_code_provenance(manifests[ref]) if not pending: @@ -145,29 +148,6 @@ def report(count: int) -> None: return report -def _check_access( - manifests: dict[DatasetRef, Manifest], fetched: set[DatasetRef], source: BaseRegistry, dest: WritableRegistry -) -> None: - """Check release access against the source and destination requirements. - - Check all releases against ``dest`` and fetched releases against ``source``. - - Args: - manifests: Release references mapped to their manifests. - fetched: The releases that the copy fetches from ``source``. - source: The registry to copy from. - dest: The registry to copy to. - - Raises: - TimeNetAccessError: If a release or its lineage lacks open access where a registry requires it. - """ # ruff: ignore[docstring-extraneous-exception] - raised by require_open_access - for reference, manifest in manifests.items(): - dest_needs_open = dest.requires_open_access - source_needs_open = reference in fetched and source.requires_open_access - if dest_needs_open or source_needs_open: - require_open_access(manifest.metadata, manifest.lineage) - - def _manifests( reference: DatasetRef, source: BaseRegistry, dest: WritableRegistry, *, with_payload_sources: bool ) -> dict[DatasetRef, Manifest]: @@ -185,21 +165,27 @@ def _manifests( Manifests keyed by release, with the payload sources before the release. Raises: - TimeNetAccessError: If ``source`` lists a payload source without serving it and ``dest`` lacks a complete copy. + TimeNetAccessError: If ``source`` does not serve the release or a payload source to this caller, and + ``dest`` lacks a complete copy of that payload source. TimeNetMissingParentError: If neither ``source`` nor ``dest`` holds a complete payload source. - """ # ruff: ignore[docstring-extraneous-exception] - raised by require_open_access + """ # ruff: ignore[docstring-extraneous-exception] - raised by get_manifest manifest = source.get_manifest(reference.dataset_id, str(reference.version)) validate_manifest_reference(manifest, reference) if not with_payload_sources: return {reference: manifest} - sources = find_payload_sources( - reference, manifest, source, find=source.find_manifest, held=lambda ref: _complete_copy(ref, dest) - ) + refused: dict[DatasetRef, TimeNetAccessError] = {} + + def served(payload_source: DatasetRef) -> Manifest | None: + try: + return source.find_manifest(payload_source) + except TimeNetAccessError as refusal: + refused[payload_source] = refusal + return None + + sources = find_payload_sources(reference, manifest, find=served, held=lambda ref: _complete_copy(ref, dest)) for missing in sources.missing: - listed = source.find_manifest(missing) - if listed is not None: - # The source lists the payload source, but does not serve its files. - require_open_access(listed.metadata, listed.lineage) + if missing in refused: + raise refused[missing] raise TimeNetMissingParentError(missing, reference, f"{source} or {dest}", payload_source=True) return {**sources.fetched, **sources.held, reference: manifest} diff --git a/packages/timenet/src/timenet/registry/base.py b/packages/timenet/src/timenet/registry/base.py index 870a89128..d3e1fa469 100644 --- a/packages/timenet/src/timenet/registry/base.py +++ b/packages/timenet/src/timenet/registry/base.py @@ -13,7 +13,7 @@ from typing import BinaryIO, ClassVar from timenet._fs import staged_directory -from timenet.errors import TimeNetDatasetNotFoundError, TimeNetFileError, TimeNetRegistryError +from timenet.errors import TimeNetAccessError, TimeNetDatasetNotFoundError, TimeNetFileError, TimeNetRegistryError from timenet.format.checksums import file_checksum from timenet.format.constants import MANIFEST_FILE from timenet.manifest import FilePart, Manifest @@ -22,8 +22,7 @@ from timenet.registry.shared import complete_copy from timenet.registry.version import DatasetVersion from timenet.sources import FoundInRegistry, NotFound, NotServed, ReleaseLookup -from timenet.types import Access, DatasetFilter, DatasetMetadata, ListedRelease, Version -from timenet.types.access import access_requirements, effective_access +from timenet.types import DatasetFilter, DatasetMetadata, ListedRelease, Version from timenet.types.references import DatasetRef @@ -70,11 +69,6 @@ def check_writable(self, write: str) -> None: f"{self} has read_only=True, so it refuses {write}. Open it without read_only=True to write to it." ) - @property - def requires_open_access(self) -> bool: - """Whether stored or served releases must declare open access.""" - return True - @abstractmethod def list_datasets(self) -> list[DatasetMetadata]: """Return the metadata of every dataset, using the latest version. The result is sorted by dataset id. @@ -156,25 +150,23 @@ def open_release(self, reference: DatasetRef, *, manifest: Manifest | None = Non def resolve(self, dataset_id: str, version: Version | None) -> ReleaseLookup: """Check whether the registry serves a dataset release. - If :attr:`requires_open_access` is true, the registry serves a release only if its card and each card in its - lineage declare open access. + The registry does not check the access level of the release. Args: dataset_id: The dataset ID. version: An exact version, or ``None`` for the latest version. Returns: - ``FoundInRegistry`` if served, ``NotServed`` if access is restricted, or ``NotFound`` if absent. + ``FoundInRegistry`` if served, ``NotFound`` if absent, or ``NotServed`` if the registry refuses this + caller. """ try: manifest = self.get_manifest(dataset_id, None if version is None else str(version)) except TimeNetDatasetNotFoundError: return NotFound() - held = manifest.metadata.dataset_version - access = effective_access(manifest.metadata, manifest.lineage) - if access is Access.OPEN or not self.requires_open_access: - return FoundInRegistry(held, self, manifest) - return NotServed(held, access, access_requirements(manifest.metadata, manifest.lineage), manifest) + except TimeNetAccessError as error: + return NotServed(error.dataset.version, error.access, error.requirements, error.reason) + return FoundInRegistry(manifest.metadata.dataset_version, self, manifest) def download_release( self, diff --git a/packages/timenet/src/timenet/registry/dependencies.py b/packages/timenet/src/timenet/registry/dependencies.py index 50d743bfb..100f0f2e7 100644 --- a/packages/timenet/src/timenet/registry/dependencies.py +++ b/packages/timenet/src/timenet/registry/dependencies.py @@ -26,7 +26,6 @@ from timenet.manifest import FilePart, Manifest, PayloadSource, read_manifest from timenet.registry._paths import release_dir from timenet.registry.version import DatasetVersion -from timenet.types.access import Access, effective_access from timenet.types.reference_keys import SourceKey from timenet.types.references import DatasetRef @@ -158,7 +157,6 @@ class CopySources: def find_payload_sources( release: DatasetRef, manifest: Manifest, - registry: "BaseRegistry", *, find: Callable[[DatasetRef], Manifest | None], held: Callable[[DatasetRef], Manifest | None], @@ -170,8 +168,7 @@ def find_payload_sources( Args: release: The release to copy. manifest: The manifest of ``release``. - registry: The registry to copy from. - find: Return the manifest of a release in ``registry``, or ``None`` if it lacks the release. + find: Return the manifest of a release that the registry to copy from serves, or ``None``. held: Return the manifest of a complete copy in the destination, or ``None``. Returns: @@ -184,7 +181,7 @@ def find_payload_sources( for pinned in manifest.payload_sources.values(): source = pinned.release found = find(source) - if found is not None and _serves(registry, found): + if found is not None: sources.fetched[source] = found elif (found := held(source)) is not None: sources.held[source] = found @@ -195,22 +192,6 @@ def find_payload_sources( return sources -def _serves(registry: "BaseRegistry", manifest: Manifest) -> bool: - """Check whether a registry serves the files of a release that it holds. - - A registry that requires open access serves a release only if its card and each card in its lineage are open. - - Args: - registry: The registry that holds the release. - manifest: The manifest of the release in ``registry``. - - Returns: - ``True`` if the registry serves the files. - """ - open_access = effective_access(manifest.metadata, manifest.lineage) is Access.OPEN - return open_access or not registry.requires_open_access - - def check_pins(release: DatasetRef, pinned: PayloadSource, manifest: Manifest) -> None: """Check that a payload source still records each pinned file as the release pins it. diff --git a/packages/timenet/src/timenet/registry/local.py b/packages/timenet/src/timenet/registry/local.py index c21dd0014..52ddf2410 100644 --- a/packages/timenet/src/timenet/registry/local.py +++ b/packages/timenet/src/timenet/registry/local.py @@ -78,15 +78,6 @@ def requires_clean_code_provenance(self) -> bool: """ return False - @property - def requires_open_access(self) -> bool: - """Whether local releases must declare open access. - - Returns: - ``False``, since a local registry accepts any access level. - """ - return False - def list_datasets(self) -> list[DatasetMetadata]: """Return the latest-version metadata of every dataset, sorted by id. diff --git a/packages/timenet/src/timenet/registry/remote/_download.py b/packages/timenet/src/timenet/registry/remote/_download.py index 572b5601b..323c7a8c4 100644 --- a/packages/timenet/src/timenet/registry/remote/_download.py +++ b/packages/timenet/src/timenet/registry/remote/_download.py @@ -133,7 +133,7 @@ async def _resolve(client: httpx.AsyncClient, headers: dict, dataset_id: str, ve """ # ruff: ignore[docstring-extraneous-exception] - raised by _redirect_target path = _download_path(dataset_id, version, relpath) response = await _send_with_retry_async(lambda: client.get(path, headers=headers, follow_redirects=False)) - return _redirect_target(response, path) + return _redirect_target(response, path, anonymous="Authorization" not in headers) async def _stream_one( # ruff: ignore[too-many-arguments] diff --git a/packages/timenet/src/timenet/registry/remote/_http.py b/packages/timenet/src/timenet/registry/remote/_http.py index 936f5b2ee..c521b45b0 100644 --- a/packages/timenet/src/timenet/registry/remote/_http.py +++ b/packages/timenet/src/timenet/registry/remote/_http.py @@ -154,15 +154,38 @@ def timenet_user_agent() -> str: return f"timenet/{version}" -def _raise_for_status(response: httpx.Response) -> None: +class RegistryForbiddenError(TimeNetRegistryError): + """The registry service refused the request (403).""" + + +def _refusal_reason(response: httpx.Response) -> str: + """Return the reason that the service gives for a 401 or a 403. + + Args: + response: The error response. + + Returns: + The whole ``detail`` text of a JSON body, otherwise the start of the body. + """ + try: + detail = response.json()["detail"] + except (ValueError, KeyError, TypeError): + detail = None + return detail if isinstance(detail, str) else response.text[:200] + + +def _raise_for_status(response: httpx.Response, *, anonymous: bool = False) -> None: """Map an error response onto a TimeNet error. Args: response: The response to check. + anonymous: The request went to the registry API without a token. Then a 401 or a 403 says how to set one. Raises: TimeNetDatasetNotFoundError: On 404. - TimeNetRegistryError: On any other 4xx/5xx, with the status and a body snippet. + RegistryForbiddenError: On 403, with the reason that the service gives. + TimeNetRegistryError: On any other 4xx/5xx, with the status and a body snippet. A 401 carries the reason + that the service gives. """ if not response.is_error: return @@ -172,7 +195,14 @@ def _raise_for_status(response: httpx.Response) -> None: if response.status_code == httpx.codes.TOO_MANY_REQUESTS: retry = response.headers.get("retry-after", "?") raise TimeNetRegistryError(f"registry rate-limited (429), retry after {retry}s: {body}") - raise TimeNetRegistryError(f"registry request to {response.request.url} failed ({response.status_code}): {body}") + failed = f"registry request to {response.request.url} failed ({response.status_code})" + hint = ". Set TIMENET_TOKEN or pass token= to RemoteRegistry." if anonymous else "" + refused = f"{failed}: {_refusal_reason(response)}{hint}" + if response.status_code == httpx.codes.FORBIDDEN: + raise RegistryForbiddenError(refused) + if response.status_code == httpx.codes.UNAUTHORIZED: + raise TimeNetRegistryError(refused) + raise TimeNetRegistryError(f"{failed}: {body}") def _download_path(dataset_id: str, version: str, relpath: str) -> str: @@ -189,12 +219,13 @@ def _download_path(dataset_id: str, version: str, relpath: str) -> str: return f"{API_PREFIX}/datasets/{dataset_id}/{version}/download/{relpath}" -def _redirect_target(response: httpx.Response, path: str) -> str: +def _redirect_target(response: httpx.Response, path: str, *, anonymous: bool) -> str: """Return a download response's redirect ``Location``, or raise if it did not redirect. Args: response: The download response fetched with ``follow_redirects=False``. path: The requested path, for the error message. + anonymous: The request carried no token. Returns: The presigned URL from the ``Location`` header. @@ -204,7 +235,7 @@ def _redirect_target(response: httpx.Response, path: str) -> str: """ if response.is_redirect: return response.headers["location"] - _raise_for_status(response) + _raise_for_status(response, anonymous=anonymous) raise TimeNetRegistryError(f"expected a redirect from {path}, got {response.status_code}") @@ -280,7 +311,7 @@ def get_text(self, path: str) -> str: """ with _reaching(self._base_url): response = _send_with_retry(lambda: self._client.get(f"{API_PREFIX}{path}", headers=self.api_headers())) - _raise_for_status(response) + _raise_for_status(response, anonymous=not self._token) return response.text def post(self, path: str, *, content: bytes | None = None, json: Any = None) -> httpx.Response: @@ -298,7 +329,7 @@ def post(self, path: str, *, content: bytes | None = None, json: Any = None) -> response = _send_with_retry( lambda: self._client.post(f"{API_PREFIX}{path}", content=content, json=json, headers=self.api_headers()) ) - _raise_for_status(response) + _raise_for_status(response, anonymous=not self._token) return response def resolve_presigned(self, dataset_id: str, version: str, relpath: str) -> str: @@ -320,7 +351,7 @@ def resolve_presigned(self, dataset_id: str, version: str, relpath: str) -> str: response = _send_with_retry( lambda: self._client.get(path, headers=self.api_headers(), follow_redirects=False) ) - return _redirect_target(response, path) + return _redirect_target(response, path, anonymous=not self._token) def stream_to(self, url: str, sink: BinaryIO) -> None: """Stream a presigned URL's bytes into a binary sink (no auth header). diff --git a/packages/timenet/src/timenet/registry/remote/registry.py b/packages/timenet/src/timenet/registry/remote/registry.py index e3385bd7f..6671b50c6 100644 --- a/packages/timenet/src/timenet/registry/remote/registry.py +++ b/packages/timenet/src/timenet/registry/remote/registry.py @@ -13,12 +13,12 @@ import pyarrow.fs as pafs from timenet.config import settings -from timenet.errors import TimeNetRegistryError +from timenet.errors import TimeNetAccessError, TimeNetRegistryError from timenet.format.constants import MANIFEST_FILE from timenet.manifest import Manifest, parse_manifest from timenet.registry.lazy_read import LazyReadRegistry from timenet.registry.remote._download import ByteCountCallback, download_version_files -from timenet.registry.remote._http import RegistryHttpClient, _reaching +from timenet.registry.remote._http import RegistryForbiddenError, RegistryHttpClient, _reaching from timenet.registry.remote.filesystem import RegistryFileSystem from timenet.registry.remote.models import ( DatasetDetailModel, @@ -89,9 +89,14 @@ def get_manifest(self, dataset_id: str, version: str | None = None) -> Manifest: Raises: TimeNetDatasetNotFoundError: If the dataset id or version is unknown. TimeFFormatError: If the served manifest is not valid. + TimeNetAccessError: If the service does not serve the release to this caller. """ # ruff: ignore[docstring-extraneous-exception] - raised by the HTTP client and parse_manifest resolved = self._resolve_latest(dataset_id) if is_latest(version) else version - text = self._http.get_text(f"/datasets/{dataset_id}/{resolved}/manifest") + try: + text = self._http.get_text(f"/datasets/{dataset_id}/{resolved}/manifest") + except RegistryForbiddenError as refusal: + release = DatasetRef.parse(f"{dataset_id}@{resolved}") + raise TimeNetAccessError(release, None, reason=str(refusal)) from None return parse_manifest(text, f"the manifest of {dataset_id}@{resolved} in {self}") def open_file(self, dataset_id: str, version: str, relpath: str) -> BinaryIO: diff --git a/packages/timenet/src/timenet/sources.py b/packages/timenet/src/timenet/sources.py index 9f847e3d0..ab423d678 100644 --- a/packages/timenet/src/timenet/sources.py +++ b/packages/timenet/src/timenet/sources.py @@ -69,15 +69,15 @@ class NotServed: Attributes: version: The exact version that the source lists. - access: The strictest access level of the release's card and the cards in its lineage. - requirements: Each card that is not open access, with its access level and access page. - manifest: The release manifest, or ``None``. + access: The access level of the release, or ``None`` if the source does not say. + requirements: The cards that are not open access, with their access level and access page. + reason: The reason that the source gives, or ``None``. """ version: Version - access: Access - requirements: tuple[AccessRequirement, ...] - manifest: "Manifest | None" = field(default=None, compare=False, repr=False) + access: Access | None + requirements: tuple[AccessRequirement, ...] = () + reason: str | None = None def access_error(self, dataset_id: str) -> TimeNetAccessError: """Create an error with the release's access requirements. @@ -89,7 +89,7 @@ def access_error(self, dataset_id: str) -> TimeNetAccessError: An error with the release, its access level, and its access pages. """ release = DatasetRef(dataset_id=dataset_id, version=self.version) - return TimeNetAccessError(release, self.access, self.requirements) + return TimeNetAccessError(release, self.access, self.requirements, self.reason) ReleaseLookup: TypeAlias = Found | NotFound | NotServed diff --git a/packages/timenet/src/timenet/types/access.py b/packages/timenet/src/timenet/types/access.py index 9af350801..19788fcac 100644 --- a/packages/timenet/src/timenet/types/access.py +++ b/packages/timenet/src/timenet/types/access.py @@ -9,7 +9,6 @@ from enum import StrEnum, unique from typing import TYPE_CHECKING -from timenet.errors import TimeNetAccessError from timenet.types.references import DatasetRef @@ -86,28 +85,3 @@ def effective_access(metadata: "DatasetMetadata", lineage: "Iterable[ParentDatas """ cards = (metadata, *(ancestor.metadata for ancestor in lineage)) return max((card.access for card in cards), key=_STRICTNESS.__getitem__) - - -def require_open_access(metadata: "DatasetMetadata", lineage: "Iterable[ParentDatasetCard]" = ()) -> None: - """Reject a release that is not open, for a registry that serves its bytes to others. - - Credentialed and restricted data cannot be redistributed. Only the user's own registry, which - holds what the user built, may hold such a release. The check uses the effective access level - of the release, so a child whose card says open but whose parent is credentialed is not open. - - Args: - metadata: The release's card or manifest metadata. - lineage: The lineage of a manifest or a dataset. A card that is not built yet has none, so - its own level and requirement are the only ones. - - Raises: - TimeNetAccessError: If the effective access of the release is credentialed or restricted. - The error names the release, its effective access level, and each requirement that - :func:`access_requirements` finds. - """ - lineage = tuple(lineage) - access = effective_access(metadata, lineage) - if access is Access.OPEN: - return - release = DatasetRef(dataset_id=metadata.dataset_id, version=metadata.dataset_version) - raise TimeNetAccessError(release, access, access_requirements(metadata, lineage)) diff --git a/packages/timenet/tests/_fake_registry.py b/packages/timenet/tests/_fake_registry.py index 21eb3a3a3..f5375bd94 100644 --- a/packages/timenet/tests/_fake_registry.py +++ b/packages/timenet/tests/_fake_registry.py @@ -16,6 +16,10 @@ from timenet.errors import TimeNetDatasetNotFoundError from timenet.manifest import Manifest from timenet.registry import LocalRegistry +from timenet.types import Access + + +_GATED = {"detail": "dataset is gated; a token grant is required"} def _summary(manifest: Manifest) -> dict: @@ -28,6 +32,8 @@ def _summary(manifest: Manifest) -> dict: "license": md.license.value, "domains": [d.value for d in md.domains], "tags": list(md.tags), + "access": md.access.value, + "access_url": md.access_url, } @@ -40,7 +46,9 @@ def _detail(manifest: Manifest) -> dict: } -def build_fake(version_dir: Path, *, token: str | None = None, blob_base: str = "http://blob.local"): +def build_fake( + version_dir: Path, *, token: str | None = None, blob_base: str = "http://blob.local", gate: bool = False +): """Serve one on-disk version through a MockTransport; return (transport, recorded_requests). The version_dir is ``<...>///``; its ``manifest.json`` drives every response. @@ -51,6 +59,9 @@ def build_fake(version_dir: Path, *, token: str | None = None, blob_base: str = The default blob host never contacts the network. It reads existing files from the temporary fixture release; this download handler does not create files. + + With ``gate``, the manifest, files and download routes of a release whose card is not open answer 403, + as the service does for an anonymous caller. """ version_dir = Path(version_dir) manifest = Manifest.model_validate_json((version_dir / "manifest.json").read_text()) @@ -72,6 +83,8 @@ def handler(request: httpx.Request) -> httpx.Response: # ruff: ignore[too-many- return httpx.Response(200, json={"datasets": [_summary(manifest)]}) if path == base: return httpx.Response(200, json=_detail(manifest)) + if gate and manifest.metadata.access is not Access.OPEN and path.startswith(f"{base}/{version}/"): + return httpx.Response(403, json=_GATED) if path == f"{base}/{version}/manifest": return httpx.Response( 200, @@ -190,7 +203,7 @@ def handler(request: httpx.Request) -> httpx.Response: relpath = request.url.path.removeprefix("/upload/") state["store"][relpath] = request.content return httpx.Response(200) - if request.headers.get("authorization") != f"Bearer {token}": + if request.method != "GET" and request.headers.get("authorization") != f"Bearer {token}": return httpx.Response(403, json={"detail": "write scope required"}) if path.endswith("/publish"): state["manifest"] = json.loads(request.content) @@ -241,7 +254,7 @@ def _blob(request: httpx.Request, root: Path) -> httpx.Response: # ruff: ignore[too-many-return-statements] -def _service_read(local: LocalRegistry, root: Path, parts: list[str]) -> httpx.Response: +def _service_read(local: LocalRegistry, root: Path, parts: list[str], gate: bool) -> httpx.Response: """Serve the anonymous read endpoints from a LocalRegistry over ``root``.""" if not parts: # GET /api/v1/datasets summaries = [_summary(local.get_manifest(md.dataset_id)) for md in local.list_datasets()] @@ -253,8 +266,10 @@ def _service_read(local: LocalRegistry, root: Path, parts: list[str]) -> httpx.R except TimeNetDatasetNotFoundError as exc: return httpx.Response(404, json={"detail": str(exc)}) version, action = parts[2], parts[3] if len(parts) > 3 else "" + path = root / dataset_id / version / "manifest.json" + if gate and path.exists() and Manifest.model_validate_json(path.read_text()).metadata.access is not Access.OPEN: + return httpx.Response(403, json=_GATED) if action == "manifest": - path = root / dataset_id / version / "manifest.json" if not path.exists(): return httpx.Response(404, json={"detail": f"no manifest for {dataset_id}@{version}"}) return httpx.Response(200, content=path.read_bytes(), headers={"content-type": "application/json"}) @@ -283,7 +298,7 @@ def _service_write(root: Path, manifests: dict, parts: list[str], request: httpx return httpx.Response(404, json={"detail": action}) -def build_service_fake(root: Path, *, token: str | None = None): +def build_service_fake(root: Path, *, token: str | None = None, gate: bool = False): """Emulate the whole registry API over a real on-disk tree at ``root``; return (transport, requests). Reads (list, detail, manifest, presigned download) are anonymous and served by a @@ -292,6 +307,8 @@ def build_service_fake(root: Path, *, token: str | None = None): downloads redirect to a ``blob.local`` host this same transport serves (GET with Range), and uploads PUT their bytes to that host. This is enough to drive a :class:`RemoteRegistry` end to end, publish included, without a network. + + With ``gate``, the manifest and download routes of a release whose card is not open answer 403. """ root = Path(root) local = LocalRegistry(root) @@ -304,7 +321,7 @@ def handler(request: httpx.Request) -> httpx.Response: return _blob(request, root) parts = [p for p in request.url.path.removeprefix("/api/v1/datasets").split("/") if p] if request.method == "GET": - return _service_read(local, root, parts) + return _service_read(local, root, parts, gate) if token is not None and request.headers.get("authorization") != f"Bearer {token}": return httpx.Response(403, json={"detail": "write scope required"}) return _service_write(root, manifests, parts, request) diff --git a/packages/timenet/tests/coordinator/test_build.py b/packages/timenet/tests/coordinator/test_build.py index 188b688f2..ea9748699 100644 --- a/packages/timenet/tests/coordinator/test_build.py +++ b/packages/timenet/tests/coordinator/test_build.py @@ -8,7 +8,7 @@ from timenet.coordinator.build import plan_release_build, run_plan from timenet.coordinator.plan import BuildStep, CopyStep, Plan from timenet.dataset import TimeFDataset -from timenet.errors import TimeNetAccessError, TimeNetRegistryError, TimeNetVersionUnavailableError +from timenet.errors import TimeNetRegistryError, TimeNetVersionUnavailableError from timenet.progress import BytesMoved, StepFinished, StepStarted from timenet.reader import TimeFReader from timenet.registry import LocalRegistry, RemoteRegistry, open_local_version @@ -16,7 +16,6 @@ from timenet.registry.origin import OriginKind from timenet.sources import InstalledConnectors from timenet.testing import install_backends, make_dataset -from timenet.types import Access from timenet.types.reference_keys import ParentAlias, SourceKey from timenet.types.references import DatasetRef @@ -25,7 +24,6 @@ PARENT_REF = DatasetRef(dataset_id=make_dataset().metadata.dataset_id, version=make_dataset().metadata.dataset_version) CHILD_REF = DatasetRef.parse("a/child@1.0.0") GRANDCHILD_REF = DatasetRef.parse("a/grandchild@1.0.0") -_GATED = {"access": Access.CREDENTIALED, "access_url": "https://example.org/dua"} def _build(release: DatasetRef, **fields) -> BuildStep: @@ -276,54 +274,6 @@ def test_a_damaged_copy_of_the_requested_release_gets_a_forced_build(target_regi assert target_registry.damage(REF_AB) is None -def test_plan_release_build_with_open_access_only_refuses_a_credentialed_card(target_registry, backends): - fake_credentialed = _Backend(_card(REF_AB, **_GATED)) - backends.append(fake_credentialed) - - with pytest.raises(TimeNetAccessError) as caught: - plan_release_build("a/b", target_registry=target_registry, open_access_only=True) - - assert caught.value.dataset == REF_AB - # Without the flag, the target registry takes a release that is not open. - run_plan(plan_release_build("a/b", target_registry=target_registry), target_registry) - assert fake_credentialed.calls == [(REF_AB, False, {})] - - -def test_plan_release_build_with_open_access_only_refuses_a_held_release_with_a_credentialed_ancestor( - target_registry, backends -): - # The card of the child is open. The child keeps a Record of the open release a/b, which the target registry - # lacks. It reads nothing from the credentialed parent, which is only in its lineage. - child_card = _card(CHILD_REF, parents={ParentAlias("parent"): PARENT_REF, ParentAlias("open"): REF_AB}) - other, child = _Backend(_card(REF_AB)), _Backend(child_card) - backends.extend([other, child]) - target_registry.put_dataset(_demo(make_dataset().metadata.model_copy(update=_GATED))) - target_registry.put_dataset(_demo(_card(REF_AB))) - dataset = TimeFDataset(metadata=child_card) - with ( - TimeFReader(open_local_version(release_dir(target_registry.root, PARENT_REF))) as gated_reader, - TimeFReader(open_local_version(release_dir(target_registry.root, REF_AB))) as open_reader, - ): - dataset.import_from(gated_reader, alias=ParentAlias("parent")) - dataset.import_from(open_reader, alias=ParentAlias("open"), records=[make_dataset().records[0].id]) - target_registry.put_dataset(dataset) - shutil.rmtree(target_registry.root / PARENT_REF.dataset_id) - shutil.rmtree(release_dir(target_registry.root, REF_AB)) - held = target_registry.get_manifest(CHILD_REF.dataset_id) - assert [card.source for card in held.lineage] == [PARENT_REF, REF_AB] - assert list(held.payload_sources) == [SourceKey(str(REF_AB))] - - with pytest.raises(TimeNetAccessError) as caught: - plan_release_build( - CHILD_REF.dataset_id, - target_registry=target_registry, - parents_from=[InstalledConnectors()], - open_access_only=True, - ) - - assert caught.value.dataset == CHILD_REF - - def test_a_plan_runs_only_through_run_plan(target_registry, fake_parent, fake_child): plan = plan_release_build( CHILD_REF.dataset_id, target_registry=target_registry, parents_from=[InstalledConnectors()] diff --git a/packages/timenet/tests/coordinator/test_read.py b/packages/timenet/tests/coordinator/test_read.py index 7da5b4bf0..d73ded091 100644 --- a/packages/timenet/tests/coordinator/test_read.py +++ b/packages/timenet/tests/coordinator/test_read.py @@ -27,7 +27,6 @@ from timenet.sources import InstalledConnectors, ReleaseSource from timenet.testing import install_backends, make_dataset from timenet.types import Access, TimeSeriesSpec, ureg -from timenet.types.access import AccessRequirement from timenet.types.reference_keys import ParentAlias, SourceKey from timenet.types.references import DatasetRef @@ -123,10 +122,10 @@ def _child_reading(target_registry, *cards, child_release=CHILD): def _gated_remote(tmp_path) -> RemoteRegistry: - """Return a remote registry that lists DEMO as credentialed, so it does not serve its files.""" + """Return a remote registry whose service lists DEMO as credentialed and refuses to serve it.""" origin = LocalRegistry(tmp_path / "origin") origin.put_dataset(_demo(make_dataset().metadata.model_copy(update=_GATED))) - transport, _ = build_fake(release_dir(origin.root, DEMO)) + transport, _ = build_fake(release_dir(origin.root, DEMO), gate=True) return RemoteRegistry("http://api.local", transport=transport) @@ -286,7 +285,8 @@ def test_pins_that_the_target_registry_copy_of_a_payload_source_breaks_fail_the_ def test_a_release_that_a_source_lists_without_serving_raises_the_access_error(tmp_path): with pytest.raises(TimeNetAccessError) as caught: read_release(DEMO.dataset_id, None, sources=[_gated_remote(tmp_path)], target_registry=None) - assert caught.value.requirements == (AccessRequirement(DEMO, Access.CREDENTIALED, _GATED["access_url"]),) + assert caught.value.dataset == DEMO + assert "a token grant is required" in str(caught.value) def test_an_exact_version_that_the_connector_does_not_declare_raises_version_unavailable(tmp_path, monkeypatch): diff --git a/packages/timenet/tests/registry/test_copy.py b/packages/timenet/tests/registry/test_copy.py index 99965a417..06f5d8970 100644 --- a/packages/timenet/tests/registry/test_copy.py +++ b/packages/timenet/tests/registry/test_copy.py @@ -144,11 +144,14 @@ def _other_build(dataset_id: str) -> TimeFDataset: return dataset -def _remote(tmp_path: Path, name: str, *, token: str | None = None) -> tuple[RemoteRegistry, Path]: - """Return a remote registry over the service fake, and the directory that the fake serves.""" +def _remote(tmp_path: Path, name: str, *, token: str | None = None, gate: bool = False) -> tuple[RemoteRegistry, Path]: + """Return a remote registry over the service fake, and the directory that the fake serves. + + With ``gate``, the service refuses each release whose card is not open. + """ root = tmp_path / name root.mkdir(exist_ok=True) - transport, _ = build_service_fake(root, token=token) + transport, _ = build_service_fake(root, token=token, gate=gate) registry = RemoteRegistry(f"http://{name}.local", token=token, transport=transport) return registry, root @@ -290,18 +293,6 @@ def test_copy_release_refuses_a_payload_source_that_only_dest_holds_with_other_f assert dest.get_manifest(*parent) == before -def test_copy_release_checks_the_access_of_a_payload_source_that_only_dest_holds(tmp_path): - root = tmp_path / "source" - _write_parent_and_child(root, credentialed=parent_ref) - dest, dest_root = _remote(tmp_path, "dest", token="tok_rw") - # the service holds the parent, whose card declares credentialed access - shutil.copytree(root / parent_ref.dataset_id, dest_root / parent_ref.dataset_id) - with pytest.raises(TimeNetAccessError) as raised: - copy_release(child_ref, _without_parent(LocalRegistry(root)), dest) - assert raised.value.dataset == parent_ref - assert not dest.exists(*child) - - def test_copy_release_refuses_a_payload_source_that_no_registry_holds(source, dest): origin = _without_parent(source) with pytest.raises(TimeNetMissingParentError) as raised: @@ -512,18 +503,6 @@ def replace(self, target): assert complete_copy(directory, manifest) -@pytest.mark.parametrize("credentialed", [child_ref, parent_ref], ids=["release", "payload-source"]) -def test_copy_release_refuses_a_release_that_is_not_open_for_a_registry_that_is_not_local(credentialed, tmp_path): - root = tmp_path / "source" - _write_parent_and_child(root, credentialed=credentialed) - dest, _ = _remote(tmp_path, "dest", token="tok_rw") - with pytest.raises(TimeNetAccessError) as raised: - copy_release(child_ref, LocalRegistry(root), dest) - assert raised.value.dataset == credentialed - assert not dest.exists(*child) - assert not dest.exists(*parent) - - @pytest.mark.parametrize("credentialed", [child_ref, parent_ref], ids=["release", "payload-source"]) def test_copy_release_copies_a_release_that_is_not_open_between_local_registries(credentialed, tmp_path): root = tmp_path / "source" @@ -536,15 +515,43 @@ def test_copy_release_copies_a_release_that_is_not_open_between_local_registries assert dest.exists(*parent) -def test_copy_release_refuses_to_fetch_a_release_that_is_not_open_from_a_registry_that_is_not_local(tmp_path): - remote_source, source_root = _remote(tmp_path, "source") - _write_parent_and_child(source_root, credentialed=child_ref) +@pytest.mark.parametrize("credentialed", [child_ref, parent_ref], ids=["release", "payload-source"]) +def test_copy_release_raises_the_access_error_for_a_release_that_the_source_does_not_serve(credentialed, tmp_path): + remote_source, source_root = _remote(tmp_path, "source", gate=True) + _write_parent_and_child(source_root, credentialed=credentialed) dest = LocalRegistry(tmp_path / "dest") with pytest.raises(TimeNetAccessError) as raised: copy_release(child_ref, remote_source, dest) - assert raised.value.dataset == child_ref + assert raised.value.dataset == credentialed assert not dest.exists(*child) - assert not dest.exists(*parent) # the open parent is not copied either: every check runs first + assert not dest.exists(*parent) # the other release is not copied either: every check runs first + + +def test_copy_release_takes_a_payload_source_that_the_source_does_not_serve_from_dest(tmp_path): + remote_source, source_root = _remote(tmp_path, "source", gate=True) + _write_parent_and_child(source_root, credentialed=parent_ref) + dest = LocalRegistry(tmp_path / "dest") + copy_release(parent_ref, LocalRegistry(source_root), dest) + assert copy_release(child_ref, remote_source, dest) == (child_ref,) + assert dest.damage(child_ref) is None + + +@pytest.mark.parametrize("remote", ["dest", "source"]) +@pytest.mark.parametrize("credentialed", [child_ref, parent_ref], ids=["release", "payload-source"]) +def test_copy_release_copies_a_release_that_is_not_open_to_and_from_a_remote_registry(remote, credentialed, tmp_path): + source: WritableRegistry + dest: WritableRegistry + if remote == "dest": + source = LocalRegistry(tmp_path / "source") + _write_parent_and_child(source.root, credentialed=credentialed) + dest, _ = _remote(tmp_path, "dest", token="tok_rw") + else: + source, source_root = _remote(tmp_path, "source") + _write_parent_and_child(source_root, credentialed=credentialed) + dest = LocalRegistry(tmp_path / "dest") + copy_release(child_ref, source, dest) + assert dest.get_manifest(*child) == source.get_manifest(*child) + assert dest.get_manifest(*parent) == source.get_manifest(*parent) def _record_local_changes(root: Path, reference: DatasetRef) -> None: diff --git a/packages/timenet/tests/registry/test_http_client.py b/packages/timenet/tests/registry/test_http_client.py index 968da16ba..b782e6444 100644 --- a/packages/timenet/tests/registry/test_http_client.py +++ b/packages/timenet/tests/registry/test_http_client.py @@ -65,6 +65,53 @@ def handler(request): _client(handler).get_text("/datasets") +def test_a_403_keeps_the_whole_detail(): + detail = "this registry does not accept the release. " * 10 + + def handler(request): + return httpx.Response(403, json={"detail": detail}) + + with pytest.raises(TimeNetRegistryError) as caught: + _client(handler, token="tok_abc").get_text("/datasets") + + assert len(detail) > 200 + assert str(caught.value) == f"registry request to http://api.local/api/v1/datasets failed (403): {detail}" + + +def test_a_401_names_the_token(): + def handler(request): + return httpx.Response(401, json={"detail": "an API key is required"}) + + with pytest.raises(TimeNetRegistryError) as caught: + _client(handler).get_text("/datasets") + + assert str(caught.value) == ( + "registry request to http://api.local/api/v1/datasets failed (401): an API key is required. " + "Set TIMENET_TOKEN or pass token= to RemoteRegistry." + ) + + +@pytest.mark.parametrize("status", [401, 403]) +def test_a_refusal_names_the_token_only_if_the_client_sent_none(status): + def handler(request): + return httpx.Response(status, json={"detail": "refused"}) + + hint = "Set TIMENET_TOKEN or pass token= to RemoteRegistry." + with pytest.raises(TimeNetRegistryError, match=rf"\({status}\): refused\. {hint}$"): + _client(handler).post("/datasets/o/n/1.0.0/publish") + with pytest.raises(TimeNetRegistryError, match=rf"\({status}\): refused$"): + _client(handler, token="tok_abc").post("/datasets/o/n/1.0.0/publish") + + +@pytest.mark.parametrize("status", [401, 403]) +def test_a_refusal_of_the_storage_does_not_name_the_token(status): + def handler(request): + return httpx.Response(status, text="signature expired") + + with pytest.raises(TimeNetRegistryError, match=rf"\({status}\): signature expired$"): + _client(handler).stream_to("http://blob.local/file", io.BytesIO()) + + def test_retries_a_429_then_succeeds(monkeypatch): calls = {"n": 0} diff --git a/packages/timenet/tests/registry/test_remote.py b/packages/timenet/tests/registry/test_remote.py index d8072482b..7ff2205c8 100644 --- a/packages/timenet/tests/registry/test_remote.py +++ b/packages/timenet/tests/registry/test_remote.py @@ -27,7 +27,6 @@ from timenet.sources import FoundInRegistry, NotServed from timenet.testing import assert_datasets_equal, make_dataset from timenet.types import Access, DatasetFilter, DatasetSchema, Domain, License, Version -from timenet.types.access import AccessRequirement from timenet.types.parent_dataset_card import ParentDatasetCard from timenet.types.reference_keys import ParentAlias from timenet.types.references import DatasetRef @@ -608,40 +607,92 @@ def _gated_release(origin: Path, page: str) -> tuple[str, DatasetRef]: @pytest.mark.parametrize("entry_point", ["open", "download"]) -def test_a_read_or_a_download_checks_access_before_it_reads_from_a_remote_registry(tmp_path, entry_point): - dataset_id, release = _gated_release(tmp_path / "origin", "ancestor") - transport, requests = build_fake(next((tmp_path / "origin").rglob("manifest.json")).parent) +def test_a_read_or_a_download_raises_the_access_error_if_the_service_refuses_the_release(tmp_path, entry_point): + dataset_id, release = _gated_release(tmp_path / "origin", "own") + transport, requests = build_fake(next((tmp_path / "origin").rglob("manifest.json")).parent, gate=True) remote = RemoteRegistry("http://api.local", transport=transport) client = TimeNet(sources=[LocalRegistry(tmp_path / "local"), remote]) with pytest.raises(TimeNetAccessError) as caught: getattr(client, entry_point)(dataset_id) - assert caught.value.requirements == (AccessRequirement(release, Access.CREDENTIALED, "https://example.org/dua"),) + assert caught.value.dataset == release + assert caught.value.access is None + assert "dataset is gated; a token grant is required. Set TIMENET_TOKEN" in str(caught.value) assert not any("/download/" in request.url.path or request.url.host == "blob.local" for request in requests) assert not (tmp_path / "local").exists() -def test_a_remote_registry_lists_but_does_not_serve_a_release_that_is_not_open(tmp_path): - # The card of the release is open. A card in its lineage is not. - dataset_id, release = _gated_release(tmp_path / "origin", "ancestor") - transport, requests = build_fake(next((tmp_path / "origin").rglob("manifest.json")).parent) +def test_a_remote_registry_answers_not_served_for_a_release_that_the_service_refuses(tmp_path): + dataset_id, release = _gated_release(tmp_path / "origin", "own") + transport, requests = build_fake(next((tmp_path / "origin").rglob("manifest.json")).parent, gate=True) remote = RemoteRegistry("http://api.local", transport=transport) - requirement = AccessRequirement(release, Access.CREDENTIALED, "https://example.org/dua") answer = remote.resolve(dataset_id, None) - assert answer == NotServed(Version(1, 0, 0), Access.CREDENTIALED, (requirement,)) - # The answer keeps the manifest that the lookup fetched. assert isinstance(answer, NotServed) - assert answer.manifest is not None - assert answer.manifest.metadata.dataset_id == dataset_id + assert (answer.version, answer.access, answer.requirements) == (release.version, None, ()) + assert answer.reason is not None + assert "dataset is gated; a token grant is required. Set TIMENET_TOKEN" in answer.reason assert not any("/download/" in request.url.path or request.url.host == "blob.local" for request in requests) # A local registry serves every release that it holds. local = LocalRegistry(tmp_path / "origin") assert local.resolve(dataset_id, None) == FoundInRegistry(Version(1, 0, 0), local) +@pytest.mark.parametrize("page", ["own", "ancestor"]) +def test_a_remote_registry_serves_a_release_that_is_not_open_if_the_service_serves_it(tmp_path, page): + dataset_id, _ = _gated_release(tmp_path / "origin", page) + transport, _ = build_fake(next((tmp_path / "origin").rglob("manifest.json")).parent) + remote = RemoteRegistry("http://api.local", transport=transport) + local = LocalRegistry(tmp_path / "local") + + assert remote.resolve(dataset_id, None) == FoundInRegistry(Version(1, 0, 0), remote) + TimeNet(sources=[local, remote]).download(dataset_id) + + assert local.damage(DatasetRef.parse(f"{dataset_id}@1.0.0")) is None + + +def test_a_gated_release_with_an_open_card_is_not_served(version_dir, tmp_path): + directory, manifest = version_dir + served, _ = build_fake(directory) + + def handler(request): + if request.url.path.endswith("/manifest"): + return httpx.Response(403, json={"detail": "dataset is gated; a token grant is required"}) + return served.handle_request(request) + + remote = RemoteRegistry("http://api.local", transport=httpx.MockTransport(handler)) + + answer = remote.resolve(manifest.dataset_id, None) + + assert isinstance(answer, NotServed) + assert answer.reason is not None + assert "dataset is gated; a token grant is required. Set TIMENET_TOKEN" in answer.reason + + +def test_put_release_raises_the_refusal_of_the_service(version_dir, tmp_path): + directory, _ = version_dir + detail = ( + "dataset access is credentialed; this registry does not accept non-open datasets " + "(set ALLOW_CREDENTIALED_DATASETS=true to allow)" + ) + requests: list[httpx.Request] = [] + + def handler(request): + requests.append(request) + return httpx.Response(403, json={"detail": detail}) + + remote = RemoteRegistry("http://api.local", token="tok_rw", transport=httpx.MockTransport(handler)) + + with pytest.raises(TimeNetRegistryError) as caught: + remote.put_release(directory) + + assert str(caught.value).endswith(f"/publish failed (403): {detail}") + # The publish request is the first write, so the refusal comes before any upload. + assert [request.method for request in requests if request.method != "GET"] == ["POST"] + + def test_resolve_keeps_the_manifest_that_it_fetched(version_dir, tmp_path): registry, requests = _remote(version_dir, tmp_path) _, manifest = version_dir diff --git a/packages/timenet/tests/registry/test_s3.py b/packages/timenet/tests/registry/test_s3.py index 0efe995e4..a39fe97a9 100644 --- a/packages/timenet/tests/registry/test_s3.py +++ b/packages/timenet/tests/registry/test_s3.py @@ -15,8 +15,9 @@ ) from timenet.reader import TimeFReader # ruff: ignore[module-import-not-at-top-of-file] from timenet.registry import LocalRegistry, S3Registry, open_local_version # ruff: ignore[module-import-not-at-top-of-file] +from timenet.sources import FoundInRegistry # ruff: ignore[module-import-not-at-top-of-file] from timenet.testing import assert_datasets_equal, make_dataset # ruff: ignore[module-import-not-at-top-of-file] -from timenet.types import DatasetFilter, DatasetSchema, Version # ruff: ignore[module-import-not-at-top-of-file] +from timenet.types import Access, DatasetFilter, DatasetSchema, Version # ruff: ignore[module-import-not-at-top-of-file] from timenet.types.references import DatasetRef # ruff: ignore[module-import-not-at-top-of-file] from timenet.values_backends import ValuesBackend # ruff: ignore[module-import-not-at-top-of-file] @@ -153,3 +154,11 @@ def test_the_latest_version_over_an_unreachable_bucket_names_the_version_in_the_ "Ask for timenet/hello-world@1.0.0 to read the copy in the local registry, or leave the unreachable " "registry out of the sources." ) + + +def test_an_s3_registry_holds_and_serves_a_release_that_is_not_open(s3_root): + registry = S3Registry(s3_root) + gated = make_dataset(access=Access.CREDENTIALED, access_url="https://example.org/dua") + reference = registry.put_dataset(gated) + assert registry.resolve(reference.dataset_id, None) == FoundInRegistry(reference.version, registry) + assert_datasets_equal(gated, TimeNet(sources=[registry]).load(reference.dataset_id)) diff --git a/packages/timenet/tests/test_client.py b/packages/timenet/tests/test_client.py index 0fc57cba4..361d56785 100644 --- a/packages/timenet/tests/test_client.py +++ b/packages/timenet/tests/test_client.py @@ -1,7 +1,6 @@ from pathlib import Path import re import shutil -from types import SimpleNamespace from _demo_child import _child_of_the_demo from _fake_registry import build_service_fake @@ -33,13 +32,11 @@ from timenet.manifest.files import FilePart from timenet.progress import BytesMoved, ProgressEvent, StepFinished, StepStarted from timenet.reader import Loading, LoadPlan, TimeFReader -from timenet.registry import TIMENET_REGISTRY_URL, LocalRegistry, RemoteRegistry, open_local_version +from timenet.registry import TIMENET_REGISTRY_URL, LocalRegistry, RemoteRegistry, copy_release, open_local_version from timenet.registry.origin import OriginKind from timenet.sources import InstalledConnectors from timenet.testing import assert_datasets_equal, install_backends, make_dataset from timenet.types import Access, Annotation, AnswerTask, DatasetMetadata, Domain, License, Version -from timenet.types.access import AccessRequirement -from timenet.types.parent_dataset_card import ParentDatasetCard from timenet.types.reference_keys import ParentAlias from timenet.types.references import DatasetRef from timenet.writer import TimeFWriter @@ -525,8 +522,8 @@ def test_get_manifest_of_a_release_that_is_not_built_raises_with_the_card_of_the assert not (tmp_path / "store").exists() -def test_get_manifest_returns_the_listed_manifest_of_a_release_that_is_not_served(tmp_path): - # The open card of test/child reads a credentialed parent, test/a. +def test_a_payload_source_that_the_service_refuses_comes_from_the_local_registry(tmp_path): + # The open card of test/child reads a credentialed parent, test/a. The service refuses test/a. hosted = LocalRegistry(tmp_path / "hosted") parent = make_dataset(dataset_id="test/a", access=Access.CREDENTIALED, access_url="https://a.example") reference = hosted.put_dataset(parent) @@ -536,20 +533,23 @@ def test_get_manifest_returns_the_listed_manifest_of_a_release_that_is_not_serve ) ) with TimeNet(sources=[hosted]).open("test/a") as reader: - child.import_from(reader) + child.import_from(reader, records=[parent.records[0].id]) hosted.put_dataset(child) - transport, _ = build_service_fake(tmp_path / "hosted") + transport, _ = build_service_fake(tmp_path / "hosted", gate=True) remote = RemoteRegistry("http://api.local", transport=transport) - client = TimeNet(sources=[remote]) + store = LocalRegistry(tmp_path / "store") + client = TimeNet(sources=[store, remote]) - manifest = client.get_manifest("test/child") + for read in (client.open, client.download): + with pytest.raises(TimeNetAccessError) as caught: + read("test/child") + assert caught.value.dataset == reference + assert not store.exists("test/child", "1.0.0") - # The hosted registry lists the release without serving its files, so the manifest keeps its access pages. - assert [card.metadata.dataset_id for card in manifest.lineage] == ["test/a"] - assert manifest.lineage[0].metadata.access_url == "https://a.example" - assert client.get_card("test/child") == manifest.metadata - with pytest.raises(TimeNetAccessError): - client.open("test/child") + # The user builds or copies test/a into the local registry. Then the service provides only test/child. + copy_release(reference, hosted, store) + client.download("test/child") + assert store.damage(DatasetRef.parse("test/child@1.0.0")) is None def test_get_card_reads_the_card_from_the_manifest_of_a_registry(tmp_path, monkeypatch): @@ -1160,14 +1160,13 @@ def _card(dataset_id, access, access_url=None): ) -def _hosted(tmp_path, metadata, lineage=()): - """Return a registry that is not local and serves one manifest, so the build-your-own gate applies.""" - - class _Hosted(RemoteRegistry): - def get_manifest(self, dataset_id, version=None): - return SimpleNamespace(metadata=metadata, lineage=lineage) - - return _Hosted("http://api.local", transport=httpx.MockTransport(lambda request: httpx.Response(404))) +def _hosted(tmp_path, metadata): + """Return a remote registry whose service lists one release and refuses it.""" + dataset = make_dataset() + dataset._metadata = metadata + LocalRegistry(tmp_path / "hosted").put_dataset(dataset) + transport, _ = build_service_fake(tmp_path / "hosted", gate=True) + return RemoteRegistry("http://api.local", transport=transport) @pytest.mark.parametrize( @@ -1192,27 +1191,18 @@ def test_the_access_hint_names_the_package_that_builds_the_release(tmp_path, mon ) -def test_the_hint_for_a_hosted_child_names_the_access_page_of_each_ancestor(tmp_path, monkeypatch): - monkeypatch.setattr(_Builder, "IDS", ("org/child",)) +def test_the_hint_for_a_refused_release_names_the_sources_that_a_local_build_needs(tmp_path, monkeypatch): + monkeypatch.setattr(_Builder, "IDS", ("org/gated",)) install_backends(monkeypatch, _Builder([])) - # The card of the child stays open and has no access page. Its parents set the access. - parents = ( - _card("org/a", Access.CREDENTIALED, "https://a.example"), - _card("org/b", Access.RESTRICTED, "https://b.example"), - ) - lineage = tuple(ParentDatasetCard(metadata=card) for card in parents) - client = TimeNet(sources=[_hosted(tmp_path, _card("org/child", Access.OPEN), lineage)]) + client = TimeNet(sources=[_hosted(tmp_path, _card("org/gated", Access.RESTRICTED, "https://b.example"))]) with pytest.raises(TimeNetAccessError) as caught: - client.open("org/child") + client.open("org/gated") - assert caught.value.requirements == ( - AccessRequirement(DatasetRef.parse("org/a@1.0.0"), Access.CREDENTIALED, "https://a.example"), - AccessRequirement(DatasetRef.parse("org/b@1.0.0"), Access.RESTRICTED, "https://b.example"), - ) assert str(caught.value) == ( - "org/child@1.0.0 is restricted. " - "Get access at https://a.example for org/a@1.0.0 and at https://b.example for org/b@1.0.0. " + "org/gated@1.0.0 is not served to this caller: registry request to " + "http://api.local/api/v1/datasets/org/gated/1.0.0/manifest failed (403): dataset is gated; a token grant is " + "required. Set TIMENET_TOKEN or pass token= to RemoteRegistry. " # A client without a store cannot build, so the hint names the store too. "Add LocalRegistry() and InstalledConnectors() to the sources to build it locally." ) diff --git a/packages/timenet/tests/test_producer.py b/packages/timenet/tests/test_producer.py index 3daccba02..3a0a277cc 100644 --- a/packages/timenet/tests/test_producer.py +++ b/packages/timenet/tests/test_producer.py @@ -1,8 +1,8 @@ import shutil import subprocess import sys -from types import SimpleNamespace +from _fake_registry import build_service_fake import httpx import pytest @@ -124,17 +124,6 @@ def test_auto_build_false_leaves_the_connectors_out_of_the_default_parents(tmp_p assert [release for release, _ in cards] == [PARENT, CHILD] -def test_open_access_only_refuses_a_gated_parent_before_any_step(tmp_path, cards): - origin, store = LocalRegistry(tmp_path / "origin"), tmp_path / "store" - gated = _card(PARENT, access=Access.CREDENTIALED, access_url="https://physionet.example/dua") - origin.put_dataset(_demo(gated)) - - with pytest.raises(TimeNetAccessError, match="credentialed"): - timenet.build(CHILD.dataset_id, local_registry=store, sources=[origin], open_access_only=True) - assert cards == [] - assert not store.exists() - - def test_timenet_exports_the_client_and_its_sources(): assert (timenet.TimeNet, timenet.LocalRegistry, timenet.InstalledConnectors) == ( TimeNet, @@ -280,12 +269,9 @@ def test_publish_of_a_listed_release_that_the_source_does_not_serve_raises_the_a card = _card( DatasetRef.parse("org/gated@1.0.0"), access=Access.CREDENTIALED, access_url="https://physionet.example/dua" ) - - class _Hosted(RemoteRegistry): - def get_manifest(self, dataset_id, version=None): - return SimpleNamespace(metadata=card, lineage=()) - - hosted = _Hosted("http://api.local", transport=httpx.MockTransport(lambda request: httpx.Response(404))) + LocalRegistry(tmp_path / "hosted").put_dataset(_demo(card)) + transport, _ = build_service_fake(tmp_path / "hosted", gate=True) + hosted = RemoteRegistry("http://api.local", transport=transport) with pytest.raises(TimeNetAccessError) as caught: timenet.publish("org/gated", dest=tmp_path / "dest", source=hosted) assert caught.value.dataset == DatasetRef.parse("org/gated@1.0.0")