Skip to content

fix: retry ministry scrape on transient network failure - #677

Merged
DerekRoberts merged 1 commit into
mainfrom
fix/retry-ministry-scrape
Oct 14, 2026
Merged

DerekRoberts merged 1 commit into
mainfrom
fix/retry-ministry-scrape

Conversation

@DerekRoberts

@DerekRoberts DerekRoberts commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

The scheduled ministry validation job intermittently fails due to unhandled socket hang ups (ECONNRESET) when scraping www2.gov.bc.ca.

  • Adds retry logic (3 attempts with exponential backoff) and a 30s timeout to getMinistriesHtml in schema/script/index.js
  • Sets a descriptive User-Agent header for scraping www2.gov.bc.ca
  • Reads schema/bcgovpubcode.json directly from the local workspace instead of making an unneeded remote fetch to raw.githubusercontent.com
  • Adds unit tests covering schema loading, successful HTML fetching, retry on transient failure, and exhaustion of retries

Closes #676


Thanks for the PR!

Deployments, as required, will be available below:

Please create PRs in draft mode. Mark as ready to enable:

After merge, new images are deployed in:

@DerekRoberts DerekRoberts self-assigned this Oct 14, 2026
@DerekRoberts DerekRoberts moved this to New in DevOps (NR) Oct 14, 2026
@DerekRoberts
DerekRoberts marked this pull request as ready for review October 14, 2026 00:36
Copilot AI balanced review requested due to automatic review settings October 14, 2026 00:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@DerekRoberts DerekRoberts moved this from New to Active in DevOps (NR) Oct 14, 2026
@DerekRoberts
DerekRoberts merged commit 8dcf802 into main Oct 14, 2026
29 checks passed
@DerekRoberts
DerekRoberts deleted the fix/retry-ministry-scrape branch October 14, 2026 01:27
@github-project-automation github-project-automation Bot moved this from Active to Done in DevOps (NR) Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix: failure in Schedule Jobs

2 participants