Skip to content

fix: wait for running scheduler tasks on shutdown - #1185

Open
alexluong wants to merge 1 commit into
mainfrom
fix/scheduler-drain-on-shutdown
Open

alexluong wants to merge 1 commit into
mainfrom
fix/scheduler-drain-on-shutdown

Conversation

@alexluong

@alexluong alexluong commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

When the retry scheduler stops, Monitor returns as soon as its context is cancelled, while tasks it already received keep running on that cancelled context. This happens on every restart with retries in flight:

  • A task cancelled before it publishes fails and backs off like any failed retry, so the retry is delayed and the receive counts toward the dead-letter limit.
  • A task that published but stops before deleting its message is received again after the visibility timeout, and the retry is delivered twice.

Monitor now waits for running tasks before it returns:

  1. Tasks run on a context that shutdown doesn't cancel (context.WithoutCancel, so logger and trace values carry over), so a received task finishes and deletes its message. This matches how internal/consumer shuts down.
  2. The wait is bounded by a shutdown grace, 5 s by default (WithShutdownGrace). Tasks still running after it are cancelled and handled like any failed task, as today.

Follow-up from the review on #1162.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexluong
alexluong force-pushed the fix/scheduler-drain-on-shutdown branch from ce66f12 to b4ba021 Compare October 30, 2026 14:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant