Skip to content

test: run deliverymq retry tests on a fake clock - #1163

Merged
alexluong merged 3 commits into
mainfrom
test/fake-clock-deliverymq
Oct 29, 2026
Merged

alexluong merged 3 commits into
mainfrom
test/fake-clock-deliverymq

Conversation

@alexluong

@alexluong alexluong commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Run the deliverymq retry tests on a fake clock.

deliverymq was the slowest package in the short suite: its tests ran one after another, each waiting out rsmq visibility timeouts and context deadlines. Everything now runs inside a synctest bubble (in-memory deliverymq, the retry scheduler on testutil.NewSynctestRedis, Monitor, mempubsub), with one shared teardown, stopRetryStack. The tests also run in parallel (t.Parallel); none of them touch package-level state.

The package now runs in under 1 s.

The conversion surfaced three test bugs:

  1. TestRetryScheduler_EventFetchSuccess was flaky: it could assert after the 2nd publish but before the 2nd log entry was written (also a data race on the entries).
  2. TestDeliveryMQRetry_RetryMaxCount published a task without Attempt (0); real tasks start at 1, so it made 4 attempts. The old check asserted as soon as it saw 3 and never waited for the 4th. Now Attempt: 1; setting it back to 0 fails with 4 vs 3.
  3. mockEventGetter.records was written and read without its mutex.

Timing choices:

  • WithRetryNotReadyDelay(time.Second): the 10 s default outlasted the test windows. It only shortens the wait; the tested behavior is the same.
  • RetryMaxCount and EligibleForRetryTrue use a 2 s backoff instead of 50 ms. Under ~1 s a retry is due at once and can run before the current attempt is logged. The lookup accepts attempts up to 1 s older than the retry's attempt time (retryLookupSkew), so it finds the previous attempt and publishes the current attempt number a second time. That's a race in the retry path; 2 s keeps these tests off it and costs nothing on the fake clock.
  • "No further attempts" checks advance the fake clock through the full window before asserting. EventNotFound and EventFetchError also assert the lookup ran.
  • The mempubsub consumer receives on context.Background() and stops through Shutdown: stopping it through its context leaks gocloud's receive goroutine, which fails the bubble.

The package was stress-tested with repeated runs, plain and with -race.

🤖 Generated with Claude Code

Base automatically changed from test/fake-clock-redis to main October 29, 2026 12:21
@alexluong
alexluong force-pushed the test/fake-clock-deliverymq branch from 77f2699 to 2dcedc4 Compare October 29, 2026 12:23
@alexluong
alexluong merged commit bdc2e4c into main Oct 29, 2026
1 check passed
@alexluong
alexluong deleted the test/fake-clock-deliverymq branch October 29, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants