Skip to content

Set a Wikimedia-compliant User-Agent in stream_file - #353

Merged
benoit74 merged 1 commit into
openzim:mainfrom
pawank925:issue-214-user-agent
Oct 16, 2026
Merged

benoit74 merged 1 commit into
openzim:mainfrom
pawank925:issue-214-user-agent

Conversation

@pawank925

Copy link
Copy Markdown
Contributor

Fix #214

When stream_file downloads from upload.wikimedia.org, it now sets a Wikimedia-compliant User-Agent (zimscraperlib/<version> (<contact>)) unless the caller already provided one (whatever the casing). This makes scrapers compliant with the Wikimedia User-Agent policy without each of them having to remember it.

Behavior for all other hosts is left untouched, so existing callers keep sending no User-Agent by default.

Scrapers that want to advertise their own identity (e.g. myscraper/1.0 (contact) zimscraperlib/x.y.z) can still pass it through headers, like maps2zim currently does with its own wrapper. Happy to add a dedicated parameter or a module-level configuration if you'd prefer that instead of relying on headers.

Changes:

  • constants.py: new WIKIMEDIA_UPLOAD_HOST constant.
  • download.py: two small helpers (is_wikimedia_upload_url, has_user_agent_header) and the default User-Agent applied for upload.wikimedia.org.
  • tests: unit tests for the helpers and for the stream_file behavior (default UA applied, custom UA preserved, extra headers merged).
  • CHANGELOG entry.

@codecov

codecov Bot commented Oct 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (93709f8) to head (ad49f2c).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #353   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           42        42           
  Lines         2720      2728    +8     
  Branches       404       405    +1     
=========================================
+ Hits          2720      2728    +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@benoit74 benoit74 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@benoit74
benoit74 merged commit 1285116 into openzim:main Oct 16, 2026
10 checks passed
@pawank925
pawank925 deleted the issue-214-user-agent branch October 16, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pass proper user-agent in stream_file when host is upload.wikimedia.org

2 participants