Repository navigation
feat(connectors): take the Hub source of a dataset from its connector - #786
Merged
Merged
Conversation
masquare
marked this pull request as ready for review
November 15, 2026 14:07
BaseHuggingFaceConnector.metadata() fills source_url and source_revision from HF_REPO and SOURCE_REVISION. The card can omit both fields; a card value that differs fails. SensorCaps and TSAQA no longer copy the repo and revision into dataset annotations.
masquare
force-pushed
the
feature/hf-source-from-connector
branch
from
November 20, 2026 07:25
c38f3e1 to
cd223ef
Compare
karhu
approved these changes
Nov 20, 2026
masquare
added a commit
that referenced
this pull request
Nov 20, 2026
…#821) Shorten the calls that a connector writes to build records, sources and tasks. The types check the same things as before. This builds on #786 below it, and the PRs above use the short forms. ## What changes - `add_record`, `add_task` and `add_tasks` take their argument positionally: `dataset.add_record(record)`. - The collection fields of `Record`, `Source`, `Signal` and the tasks accept any iterable and store a tuple, so a connector passes a list: `Record(sources=[...])`, `inputs=[...]`. They reject a string, a mapping and a pydantic model such as a single `Label`, with a clear error. The type of such a field is written as `TupleField[X]`. - A task accepts one bare target: `targets="normal"` is the same as `targets=("normal",)`. A mapping or bytes value is rejected. - `Annotation` takes `key` positionally, and `Source` takes `name`: `Annotation("age", value=64)`, `Source("IMU", signals=[...])`. The annotation value is keyword-only. - A Hugging Face connector passes its settings as class keywords: `class TSQAConnector(BaseHuggingFaceConnector, hf_repo=..., source_revision=..., source_files={...})`. The base checks them when the class is defined and stores `SOURCE_FILES` read-only. No ruff rule is relaxed. - Connectors and docs import from the package roots. The keyword forms still work, so the tests keep them. The connectors, the docs and the connector-authoring skill use the new forms. No call that worked before needs a change, except an `Annotation` value given by position: write `Annotation("age", value=64)`. ## Before and after ```python # before dataset.add_record(record=Record(sources=(Source(name="IMU", signals=(signal,)),))) Annotation(key="age", value=64) ClassificationTask(inputs=(record,), targets=("normal",), label_set=label_set) # after dataset.add_record(Record(sources=[Source("IMU", signals=[signal])])) Annotation("age", value=64) ClassificationTask(inputs=[record], targets="normal", label_set=label_set) ``` ## Where the logic is - `types/_fields.py`: `as_tuple`, the shared converter of the collection fields. - `types/tasks.py`: the bare target. - `types/annotations.py`, `dataset/source.py`: the positional fields. The rest is connectors, docs and tests. ## Checks - `make check` and `make test-connectors` pass. - The core suite passes per directory. - The registry and torch tests pass outside the sandbox. --------- Co-authored-by: masquare <11543761+masquare@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BaseHuggingFaceConnector.metadata()now fillssource_urlandsource_revisionfromHF_REPOandSOURCE_REVISION. The dataset card can omit both fields. If the card sets one, it must match the connector, or the build fails.SensorCaps and TSAQA no longer copy
HF_REPOandSOURCE_REVISIONinto dataset annotations (source_repository,source_revision). Nothing read them, and the manifest now records both in the metadata fields meant for them. Before this, only these two connectors recorded the Hub commit. TSQA and the two ARFBench connectors recorded it nowhere.The cards keep
source_url. The docs catalog (scripts/gen_dataset_docs.py) andget_card()read the card file, but they do not import the connector. An isolated build imports the connector only in the child process. Without this line, the catalog loses the source link.Dataset versions
All five Hugging Face datasets now write
source_revisionin their manifest. SensorCaps and TSAQA also write two fewer annotations.chengsenwang/tsqa@1.2.0,datadog/arfbench@1.0.0andtsaqa/tsaqa-benchmark@1.0.0are already in registry-dev. A rebuild at the same version gives a different manifest. If we rebuild them, they need adataset_versionbump. I left the versions unchanged in this PR.Test plan
test_huggingface_hub.py: the fields are filled from the connector, a card that repeats the same values passes, and a card with a differentsource_urlorsource_revisionfails.make check,make check-ci,make testandmake test-connectors(all 12 connectors) pass.