Skip to content

Restore ARTE crawling with EMAC and player v2 - #1183

Open
derreisende77 wants to merge 1 commit into
mediathekview:developfrom
derreisende77:hotfix/1181_arte-emac-crawler
Open

derreisende77 wants to merge 1 commit into
mediathekview:developfrom
derreisende77:hotfix/1181_arte-emac-crawler

Conversation

@derreisende77

Copy link
Copy Markdown
Contributor

ARTE crawls no longer discover film entries through the retired OPA catalogue API. Replace the active OPA pipeline with EMAC catalogue discovery and player-v2 HLS resolution.

Fixes #1181.

Changes

  • Browse each configured sender's ALL_VIDEOS feeds, follow bounded pagination, and expand collections into episodes.
  • Deduplicate programme references and apply the existing history filter before requesting player details. Resolve remaining programmes with at most four virtual-thread workers per sender, using the configured rate limiter.
  • Preserve the existing ARTE.DE/FR/EN/ES/IT/PL sender mapping and programme identity namespace. Populate metadata, audio-language information, audio descriptions, and catalogue-language subtitles.
  • Keep ARTE's published default HLS URLs; validate additional synthesized version URLs before including them. Leave HLS file sizes unknown instead of issuing size probes.
  • Isolate malformed catalogue entries and ordinary socket/call timeouts while propagating actual cancellation and closing transport/worker resources.
  • Remove unused OPA tasks, DTOs, parsers, version mapping, and their obsolete tests and fixtures. Most deleted lines are retired JSON fixtures; new tests exercise the active crawler path.
  • Add a source-based English design document with operational limits and maintenance guidance.

Uses Java 25 and existing MServer libraries; no dependency changes.

Validation

  • Java 25: ./mvnw -q clean test — 493 tests passed, no failures/errors; one opt-in live test skipped.
  • ./mvnw -q -Dtest=ArteLiveTest -Darte.live=true test — all six sender checks passed; 18 film entries and their HLS playlists verified.
  • Design document rendered with Asciidoctor with warnings treated as failures; source links and whitespace checks passed.

The live checks use one page per feed and resolve the first three discovered programmes per sender. They verify playlist availability, not a complete catalogue crawl or full media downloads. Discovery remains bounded by the configured maximumSubpages for each feed.

Replace the retired OPA crawl pipeline with catalogue discovery, early history filtering and bounded Java 25 player resolution. Preserve catalogue sender identities and handle HLS variants, subtitles, timeouts and cancellation.

Remove unused OPA code and fixtures, add regression and opt-in live tests, and document the crawler design.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant