Skip to content

feat: let destination types classify failures and set their own retry policy - #1174

Open
alexluong wants to merge 2 commits into
mainfrom
feat/delivery-error-classes
Open

alexluong wants to merge 2 commits into
mainfrom
feat/delivery-error-classes

Conversation

@alexluong

@alexluong alexluong commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Let a destination type decide whether a failed delivery is retried and counted toward auto-disable, and let it bring its own retry schedule.

Today every failed attempt is retried on the one global schedule and counts toward auto-disable, whatever the cause. A destination type can know better: a failure may be permanent for that request, or say nothing about the endpoint's health. This PR lets the provider that saw the failure say so: the publish error carries the classification, and deliverymq and the alert evaluator act on it. The MCP destination (#1181) is the first user.

No existing destination type sets any of this, so delivery behavior doesn't change.

Changes

  1. Publish errors classify themselves. destregistry.ErrDestinationPublishAttempt gains NonRetryable (no automatic retry is scheduled, and exhausted_retries fires right away) and SkipFailureCount (the failure neither increments nor resets the consecutive-failure count). Zero values keep today's behavior. The attempt is still logged as failed, attempt.failed is still emitted, and a manual retry still works.
  2. A destination type can bring its own retry policy. A provider can implement the optional destregistry.RetryPolicyProvider; deliverymq resolves it per delivery and falls back to the global policy. The resolved limit drives scheduling, the attempt_max log field and the exhausted_retries alert, so all three agree.

🤖 Generated with Claude Code

@alexluong alexluong changed the title feat: stop retrying webhook 410 and 413, use the Standard Webhooks retry schedule in standard mode feat: no retries on webhook 410/413, Standard Webhooks retry schedule in standard mode Oct 27, 2026
@alexluong
alexluong force-pushed the test/fake-clock-deliverymq branch from 77f2699 to 2dcedc4 Compare October 29, 2026 12:23
Base automatically changed from test/fake-clock-deliverymq to main October 29, 2026 12:32
@alexluong
alexluong force-pushed the feat/delivery-error-classes branch from 5c9de0c to cbdc99f Compare October 29, 2026 19:13
@alexluong alexluong changed the title feat: no retries on webhook 410/413, Standard Webhooks retry schedule in standard mode feat: no retries on webhook 410/413, 410 disables the destination, Standard Webhooks retry schedule in standard mode Oct 29, 2026
@alexbouchardd

Copy link
Copy Markdown
Contributor

Webhook 410 and 413 are not retried automatically, in both webhook modes. They still count toward auto-disable.

A webhook 410 disables the destination and emits alert.destination.disabled with reason: "endpoint_gone" (the existing reason is "consecutive_failure"). A burst of 410s disables once and alerts once, and a late attempt doesn't disable a destination the tenant has enabled again since.

I don't know about this; there are 2 main issues I see here. First, the decision to use Standard Webhook is currently positioned as a signature decision, not a behavioural one.

Second, there's no way to guarantee that people use those codes as intended, and it's not common knowledge that this would be the behaviour on a webhook call.

Both of these seem very questionable to me

@alexluong
alexluong force-pushed the feat/delivery-error-classes branch from 76b8c20 to 5d84830 Compare October 30, 2026 10:23
@alexluong alexluong changed the title feat: no retries on webhook 410/413, 410 disables the destination, Standard Webhooks retry schedule in standard mode feat: let destination types classify failures and set their own retry policy Oct 30, 2026
@alexluong
alexluong force-pushed the feat/delivery-error-classes branch from 5d84830 to 006d6df Compare October 30, 2026 10:29
@alexluong

Copy link
Copy Markdown
Collaborator Author

@alexbouchardd makes sense, I've updated the PR to just support the mechanisms that will be used by MCP Events without changing the existing webhook destination behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants