Skip to content

Add fuzzy rule to rewrite URLs of lesbases.anct.gouv.fr (Fixes #334) - #352

Open
Sriram-Nambiar wants to merge 1 commit into
openzim:mainfrom
Sriram-Nambiar:fuzzy-rule-lesbases-anct-gouv
Open

Sriram-Nambiar wants to merge 1 commit into
openzim:mainfrom
Sriram-Nambiar:fuzzy-rule-lesbases-anct-gouv

Conversation

@Sriram-Nambiar

@Sriram-Nambiar Sriram-Nambiar commented Oct 15, 2026 •

Copy link
Copy Markdown

Fixes #334
Related to openzim/zim-requests#1205

Summary of Changes:

  • Added fuzzy rule for lesbases.anct.gouv.fr in rules/rules.yaml to normalize Next.js _rsc query parameters to a fixed value (_rsc=1).
  • Added test cases covering root path, subpaths, www prefix, and query parameter variations.
  • Added changelog entry in CHANGELOG.md under [Unreleased] -> Added.

@benoit74

Copy link
Copy Markdown
Collaborator

@Sriram-Nambiar can you please share a working ZIM of lesbases.anct.gouv.fr? few pages is enough, no need to ZIM the whole website, just confirm the regex is working fine and they did not changed something which will cause it to fail again

@Sriram-Nambiar

Sriram-Nambiar commented Oct 15, 2026 •

Copy link
Copy Markdown
Author

Hi @benoit74 I generated a small sample ZIM and tested it

When clicking on links like /connexion, it unfortunately displays raw React JSON on a black screen instead of the webpage (screenshot attached). Next.js returns text/x-component for requests with ?_rsc=.... When _rsc is stripped by the fuzzy rule, warc2zim saves that component payload over the HTML page.

I've attached
lesbases_test_2026-10.zim.zip
the sample ZIM (.zip, ~6 MB) and the screenshot.
image

@benoit74

Copy link
Copy Markdown
Collaborator

Thank you very much! Do I get it correctly that JS code is inspecting the content-type header returned to take a decision?

@benoit74

Copy link
Copy Markdown
Collaborator

I also see that the main page keeps loading assets ... should we tweak the fuzzy-rule so that _rsc is still set but has a fixed value? Did you achieved to understand why this value is changing (like the ZIM does not produces the same value than the online website) and what it is used for?

@Sriram-Nambiar

Sriram-Nambiar commented Oct 16, 2026 •

Copy link
Copy Markdown
Author

Yes, you're correct! Next.js analyzes the Content-Type header of background fetches and determines whether they're of type React Server Components (text/x-component) or HTML (text/html). That's because the _rsc parameter is Next.js's cache-buster hash for those component payloads, and as such varies between builds and router states.

Your suggestion of keeping _rsc with a fixed value (_rsc=1) worked nicely! I tested it on a sample ZIM with 7 pages, and both the HTML and component files are saved separately now without overwriting each other, so /connexion loads cleanly.

I've attached a screenshot and the sample ZIM below. If this looks good to you, should I go ahead and push the updated rule to this PR?
lesbases_test_2026-10.zip

image

@benoit74

Copy link
Copy Markdown
Collaborator

I still see continuous web requests on home page in the new ZIM ... as if Next.js considered some pages were not fine and needed to be downloaded again

@Sriram-Nambiar

Sriram-Nambiar commented Oct 16, 2026 •

Copy link
Copy Markdown
Author

The home page was sending multiple web requests because the prefetch target (/rechercher/tout/ressources) was only an incomplete stub in the 7 page crawl so Next.js kept retrying it. When served with the full payload Nextjs caches it on the first try and stops making any further requests.....so the loop is purely due to the shallow test crawl. The _rsc=1 rule itself is working as expected
image

@benoit74

Copy link
Copy Markdown
Collaborator

Nice, makes sense.

Please go ahead and push the updated rule (please simply amend your commit and rebase on latest main commit)

@Sriram-Nambiar
Sriram-Nambiar force-pushed the fuzzy-rule-lesbases-anct-gouv branch from 49bd5be to 077a542 Compare October 16, 2026 15:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add fuzzy rule to rewrite URLs of lesbases.anct.gouv.fr

2 participants