Skip to content

refactor(connectors)!: make the Hugging Face download result a mapping from part to path - #823

Merged
masquare merged 2 commits into
refactor/connector-download-resultfrom
refactor/huggingface-convert-files
Nov 20, 2026
Merged

masquare merged 2 commits into
refactor/connector-download-resultfrom
refactor/huggingface-convert-files

Conversation

@masquare

@masquare masquare commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

A Hugging Face connector now gets a mapping from each part name to its local path. The mapping is the download result itself, which the PR below makes possible.

What changes

  • BaseHuggingFaceConnector is a BaseConnector[Mapping[str, Path]]. download returns the local path of each part in SOURCE_FILES, in the declared order.
  • A connector implements the normal convert(self, files) and reads files["questions"].
  • HubFileRef, check_refs and ref_paths are removed, and so is the rewrite of convert in __init_subclass__.
  • tsqa, sensorcaps, tsaqa_benchmark, arfbench, arfbench_charts and TimeSeriesExam use the new form.
# before
def convert(self, raw_refs: list[HubFileRef]) -> TimeFDataset:
    paths = self.ref_paths(raw_refs, "questions", "series")
    for question in read_questions(paths["questions"]): ...

# after
def convert(self, files: Mapping[str, Path]) -> TimeFDataset:
    for question in read_questions(files["questions"]): ...

The first commit adds an intermediate convert_files method. The second commit removes it again in favour of the form above.

Behaviour change

The base no longer checks the input of convert for empty, repeated or undeclared parts. download always returns every declared part, so this affects only tests that build the mapping by hand. A missing part there raises a plain KeyError. TSAQA loses its own check that every split file is present, for the same reason.

Checks

  • make check and make test-connectors pass.
  • The core, registry and torch tests pass.
  • A real build of the TimeSeriesExam connector succeeds.

@masquare
masquare added this pull request to stack #825 November 19, 2026 16:18
@masquare
masquare removed this pull request from stack #825 November 19, 2026 17:34
@masquare
masquare force-pushed the refactor/huggingface-convert-files branch from ccc77b7 to ced7561 Compare November 19, 2026 17:34
@masquare
masquare added this pull request to stack #826 November 19, 2026 17:34
@masquare masquare changed the title refactor(connectors)!: give Hugging Face connectors a mapping from part to path refactor(connectors)!: make the Hugging Face download result a mapping from part to path Nov 19, 2026
@masquare
masquare force-pushed the refactor/huggingface-convert-files branch from ced7561 to 704613e Compare November 19, 2026 17:42
@masquare
masquare force-pushed the refactor/huggingface-convert-files branch from 704613e to 9b41ca1 Compare November 19, 2026 17:48
@masquare
masquare requested a review from karhu November 19, 2026 18:54
@masquare
masquare marked this pull request as ready for review November 19, 2026 18:54

@karhu karhu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add GH labels, and update the PR description / merge-commit message, otherwise LGTM

@masquare
masquare force-pushed the refactor/huggingface-convert-files branch from 9b41ca1 to 6d6b352 Compare November 20, 2026 07:25
@masquare
masquare removed this pull request from stack #826 November 20, 2026 07:25
@masquare
masquare added this pull request to stack #829 November 20, 2026 07:25
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

✅ CI checks passed

Check Status
Quick checks ✅ Passed
Tests (Python 3.13) ✅ Passed
Tests (Python 3.11) ✅ Passed

Updated for 79b4915 · run #8644

…rt to path

BaseHuggingFaceConnector implements convert once. It checks the refs
and calls convert_files with a mapping from each part name to its local
path. A connector reads files["questions"] and no longer calls
ref_paths, which is removed.
…ing itself

download_async returns the mapping from each part name to its local
path, and a connector implements the normal convert(files). HubFileRef,
check_refs and convert_files are removed. TimeSeriesExam uses the same
form.
@masquare
masquare force-pushed the refactor/huggingface-convert-files branch from 6d6b352 to 79b4915 Compare November 20, 2026 15:31
@masquare
masquare merged commit 96f2123 into main Nov 20, 2026
4 of 8 checks passed
@masquare
masquare deleted the refactor/huggingface-convert-files branch November 20, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants