Skip to content

feat(connectors): take the Hub source of a dataset from its connector - #786

Merged
masquare merged 3 commits into
mainfrom
feature/hf-source-from-connector
Nov 20, 2026
Merged

masquare merged 3 commits into
mainfrom
feature/hf-source-from-connector

Conversation

@masquare

@masquare masquare commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

BaseHuggingFaceConnector.metadata() now fills source_url and source_revision from HF_REPO and SOURCE_REVISION. The dataset card can omit both fields. If the card sets one, it must match the connector, or the build fails.

SensorCaps and TSAQA no longer copy HF_REPO and SOURCE_REVISION into dataset annotations (source_repository, source_revision). Nothing read them, and the manifest now records both in the metadata fields meant for them. Before this, only these two connectors recorded the Hub commit. TSQA and the two ARFBench connectors recorded it nowhere.

The cards keep source_url. The docs catalog (scripts/gen_dataset_docs.py) and get_card() read the card file, but they do not import the connector. An isolated build imports the connector only in the child process. Without this line, the catalog loses the source link.

Dataset versions

All five Hugging Face datasets now write source_revision in their manifest. SensorCaps and TSAQA also write two fewer annotations. chengsenwang/tsqa@1.2.0, datadog/arfbench@1.0.0 and tsaqa/tsaqa-benchmark@1.0.0 are already in registry-dev. A rebuild at the same version gives a different manifest. If we rebuild them, they need a dataset_version bump. I left the versions unchanged in this PR.

Test plan

  • New tests in test_huggingface_hub.py: the fields are filled from the connector, a card that repeats the same values passes, and a card with a different source_url or source_revision fails.
  • make check, make check-ci, make test and make test-connectors (all 12 connectors) pass.

@masquare
masquare marked this pull request as ready for review November 15, 2026 14:07
@masquare
masquare requested a review from karhu November 20, 2026 06:53
BaseHuggingFaceConnector.metadata() fills source_url and source_revision from HF_REPO and SOURCE_REVISION. The card can omit both fields; a card value that differs fails. SensorCaps and TSAQA no longer copy the repo and revision into dataset annotations.
@masquare
masquare merged commit 6bf4059 into main Nov 20, 2026
4 checks passed
@masquare
masquare deleted the feature/hf-source-from-connector branch November 20, 2026 18:21
masquare added a commit that referenced this pull request Nov 20, 2026
…#821)

Shorten the calls that a connector writes to build records, sources and
tasks. The types check the same things as before. This builds on #786
below it, and the PRs above use the short forms.

## What changes

- `add_record`, `add_task` and `add_tasks` take their argument
positionally: `dataset.add_record(record)`.
- The collection fields of `Record`, `Source`, `Signal` and the tasks
accept any iterable and store a tuple, so a connector passes a list:
`Record(sources=[...])`, `inputs=[...]`. They reject a string, a mapping
and a pydantic model such as a single `Label`, with a clear error. The
type of such a field is written as `TupleField[X]`.
- A task accepts one bare target: `targets="normal"` is the same as
`targets=("normal",)`. A mapping or bytes value is rejected.
- `Annotation` takes `key` positionally, and `Source` takes `name`:
`Annotation("age", value=64)`, `Source("IMU", signals=[...])`. The
annotation value is keyword-only.
- A Hugging Face connector passes its settings as class keywords: `class
TSQAConnector(BaseHuggingFaceConnector, hf_repo=...,
source_revision=..., source_files={...})`. The base checks them when the
class is defined and stores `SOURCE_FILES` read-only. No ruff rule is
relaxed.
- Connectors and docs import from the package roots.

The keyword forms still work, so the tests keep them. The connectors,
the docs and the connector-authoring skill use the new forms.

No call that worked before needs a change, except an `Annotation` value
given by position: write `Annotation("age", value=64)`.

## Before and after

```python
# before
dataset.add_record(record=Record(sources=(Source(name="IMU", signals=(signal,)),)))
Annotation(key="age", value=64)
ClassificationTask(inputs=(record,), targets=("normal",), label_set=label_set)

# after
dataset.add_record(Record(sources=[Source("IMU", signals=[signal])]))
Annotation("age", value=64)
ClassificationTask(inputs=[record], targets="normal", label_set=label_set)
```

## Where the logic is

- `types/_fields.py`: `as_tuple`, the shared converter of the collection
fields.
- `types/tasks.py`: the bare target.
- `types/annotations.py`, `dataset/source.py`: the positional fields.

The rest is connectors, docs and tests.

## Checks

- `make check` and `make test-connectors` pass.
- The core suite passes per directory.
- The registry and torch tests pass outside the sandbox.

---------

Co-authored-by: masquare <11543761+masquare@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants