Skip to content

perf(tests): Reuse one Relay container per module in Relay tests - #131668

Draft
sentry-junior[bot] wants to merge 7 commits into
masterfrom
ci/relay-reuse-container-per-module
Draft

sentry-junior[bot] wants to merge 7 commits into
masterfrom
ci/relay-reuse-container-per-module

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Refs DI-2898

relay_server was function-scoped, so every Relay integration test started and stopped its own Relay container. Container startup is a large share of the slowest Relay files (lang/java/test_plugin.py, lang/javascript/test_plugin.py), since neither has rejection tests that wait out timeouts.

What changes

  • Each module and xdist worker gets one Relay container. relay_server stays function-scoped but only starts a container when the module's Relay fails its readiness probe. The first test in a module starts it, later tests reuse it, and a Relay that dies mid-module gets replaced. relay_server_setup still removes the container when the module ends.
  • Relay is started from a test, not from module setup. At startup it fetches its global config, and serving that reads options from the database, which module setup can't access.
  • relay_test_settings() and the relay.endpoint-fetch-config.enabled override now stay active for the whole module. Relay fetches its global config once per container (cache.global_config_fetch_interval: 3600), and the fixture restarts a Relay that comes up without one.
  • The test Relay authenticates with SENTRY_RELAY_STATIC_AUTH, read from template/credentials.json. Its normal registration is a Relay row, which the first TransactionTestCase flush deletes, so later tests would get "Unknown relay".

Isolation

  • Relay caches project configs by public key. Every test creates its own project keys with random public keys, so no test can get a config Relay cached for another. That includes test_sdk.py, which reuses project id 1 but gets a new key each time.
  • Per-worker Kafka topics, Redis DB and container names don't change.
  • Apart from relay.endpoint-fetch-config.enabled, no Relay test overrides an option that goes into the global config.

Overlap with #131170

#131170 (merged) changes src/sentry/testutils/relay.py and two Relay test files, and this PR doesn't touch any of them. Its outcome checks filter on each test's own org, project and key IDs, so late outcomes from an earlier test on the shared Relay can't match.

Status: parked

  • Benefit: a modest cut in summed Relay/Symbolicator test time per full run.
  • No critical-path gain: the slowest shard is unchanged, because the full-run planner spreads these files across many shards.
  • Risks: more shards need a pytest rerun than on master, and which tests rerun is unknown. lang/javascript/test_plugin.py is slower than on master.
  • Cost: module-scoped test settings, static Relay auth, a restart loop for a Relay without a global config, and a 1h global config refresh.
  • Not pursued: putting these files on one shard. Image pulls and container starts are a small share of runner time.
  • The OptionsStore race fix is in fix(options): Read the silo-dependent model once in OptionsStore #131740.

via @joshuarli.

--

View Session [Sentry]

relay_server started a new Relay container for every test. Keep one per module instead: a test starts it only when the module's Relay is not ready, so the first test starts it and later tests reuse it. Each test still gets its own project keys, which is what Relay caches configs by.

Authenticate the test Relay with static auth, since its database registration would be flushed after the first test, and keep the test settings and option overrides for the module, since Relay keeps fetching its global config while it runs.

Refs DI-2898

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
@linear-code

linear-code Bot commented Nov 18, 2026

Copy link
Copy Markdown

DI-2898

@sentry-junior sentry-junior Bot added the Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling label Nov 18, 2026
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 18, 2026
…ainer-per-module

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
sentry-junior Bot and others added 5 commits November 18, 2026 21:48
A reused Relay fetches its global config when it starts, inside a test.
Its 10s refetch could land in the next test's setup, before that test has
database access, and the endpoint returned a 500.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Temporary measurement for this PR; reverted before review.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
This reverts commit bf08579.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
A reused Relay fetches its global config only at startup and holds
envelopes until it has one. If that fetch fails, start another Relay
instead of letting every test in the module time out.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
OptionsStore.model follows the current silo mode. get_store and
get_last_update_channel read it for the query and again for the except
clause, so a silo mode switch on another thread in between let
Option.DoesNotExist escape as an error. Tests hit this when a live
server request ran while the test switched silo mode.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>

This branch was successfully deployed

1 active (outdated) deployment
Preview — 9b422cd0 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant