Skip to content

feat(types): simplify the calls that build records, sources and tasks - #821

Merged
masquare merged 6 commits into
feature/hf-source-from-connectorfrom
feature/connector-api-ergonomics
Nov 20, 2026
Merged

masquare merged 6 commits into
feature/hf-source-from-connectorfrom
feature/connector-api-ergonomics

Conversation

@masquare

@masquare masquare commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Shorten the calls that a connector writes to build records, sources and tasks. The types check the same things as before. This builds on #786 below it, and the PRs above use the short forms.

What changes

  • add_record, add_task and add_tasks take their argument positionally: dataset.add_record(record).
  • The collection fields of Record, Source, Signal and the tasks accept any iterable and store a tuple, so a connector passes a list: Record(sources=[...]), inputs=[...]. They reject a string, a mapping and a pydantic model such as a single Label, with a clear error. The type of such a field is written as TupleField[X].
  • A task accepts one bare target: targets="normal" is the same as targets=("normal",). A mapping or bytes value is rejected.
  • Annotation takes key positionally, and Source takes name: Annotation("age", value=64), Source("IMU", signals=[...]). The annotation value is keyword-only.
  • A Hugging Face connector passes its settings as class keywords: class TSQAConnector(BaseHuggingFaceConnector, hf_repo=..., source_revision=..., source_files={...}). The base checks them when the class is defined and stores SOURCE_FILES read-only. No ruff rule is relaxed.
  • Connectors and docs import from the package roots.

The keyword forms still work, so the tests keep them. The connectors, the docs and the connector-authoring skill use the new forms.

No call that worked before needs a change, except an Annotation value given by position: write Annotation("age", value=64).

Before and after

# before
dataset.add_record(record=Record(sources=(Source(name="IMU", signals=(signal,)),)))
Annotation(key="age", value=64)
ClassificationTask(inputs=(record,), targets=("normal",), label_set=label_set)

# after
dataset.add_record(Record(sources=[Source("IMU", signals=[signal])]))
Annotation("age", value=64)
ClassificationTask(inputs=[record], targets="normal", label_set=label_set)

Where the logic is

  • types/_fields.py: as_tuple, the shared converter of the collection fields.
  • types/tasks.py: the bare target.
  • types/annotations.py, dataset/source.py: the positional fields.

The rest is connectors, docs and tests.

Checks

  • make check and make test-connectors pass.
  • The core suite passes per directory.
  • The registry and torch tests pass outside the sandbox.

@masquare
masquare added this pull request to stack #824 November 19, 2026 16:18
@masquare
masquare force-pushed the feature/connector-api-ergonomics branch from ae756b9 to 8d3fd9f Compare November 19, 2026 17:34
@masquare
masquare removed this pull request from stack #824 November 19, 2026 17:34
@masquare
masquare added this pull request to stack #826 November 19, 2026 17:34
@masquare
masquare requested a review from karhu November 19, 2026 17:51

@karhu karhu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like it 💪 one suggestion for the BaseHuggingFaceConnector 's SOURCE_FILES change and one more question here, regarding "!" in the PR. Can't tell what's breaking here. Either remove it or add an explicit "BREAKING CHANGES:" section to the PR description

Comment thread pyproject.toml Resolved
- add_record, add_task and add_tasks take their argument positionally.
- The collection fields of Record, Source and Task accept any iterable
  and store a tuple. They reject a string, a mapping and a single item.
- A task accepts one bare target, such as targets="normal".
- Annotation takes key and value positionally, and Source takes name.
- A Hugging Face connector sets SOURCE_FILES without an annotation.
- Connectors and docs import from the package roots.

The connectors and the docs use the new forms.
…ld type

- A mapping or bytes value as targets is one bare value, so the task
  rejects it like the tuple form does.
- TupleField names the type of a collection field once.
- hello_world uses the list forms everywhere.
Annotation takes only key positionally: Annotation("age", value=64).
A single item that can be iterated, such as a Label, passes the string
and mapping checks. With item_type, as_tuple rejects it.
…em type

A pydantic model can be iterated, so a single Label passed where a list
is expected went through. as_tuple now rejects every pydantic model, so
each collection field gets this check and the item_type argument goes.
A Hugging Face connector passes hf_repo, source_revision and
source_files as class keywords. The base checks them when the class is
defined and stores SOURCE_FILES read-only. A subclass without keywords
keeps the values of its parent. The ruff ignore for mutable class
defaults in connector modules goes away.
@masquare
masquare force-pushed the feature/connector-api-ergonomics branch from abf02ae to c1ef22b Compare November 20, 2026 07:25
@masquare
masquare removed this pull request from stack #826 November 20, 2026 07:25
@masquare
masquare changed the base branch from main to feature/hf-source-from-connector November 20, 2026 07:25
@masquare
masquare added this pull request to stack #829 November 20, 2026 07:25
@masquare masquare changed the title feat(types)!: simplify the calls that build records, sources and tasks feat(types): simplify the calls that build records, sources and tasks Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

thx for the suggestion, implemented in new commit

@masquare
masquare merged commit b87039e into main Nov 20, 2026
4 checks passed
@masquare
masquare deleted the feature/connector-api-ergonomics branch November 20, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants