docs: [Sample Tests 40] Describe the in-module end-to-end tests instead of the system tests

.claude/skills/test/SKILL.md

@@ -1,13 +1,14 @@
 ---
 name: test
-description: Run tests for a specific SDK module. Use when asked to "run tests", "test module", "run unit tests", "run system tests", "run e2e tests", or test a specific class. Auto-detects unit vs system tests. Supports interactive mode.
+description: Run tests for a specific SDK module. Use when asked to "run tests", "test module", "run unit tests", "run integration tests", "run e2e tests", or test a specific class. Auto-detects unit tests vs integration test suites. Supports interactive mode.
 allowed-tools: Bash, Read, Glob, AskUserQuestion
 argument-hint: [interactive] <module-name-or-file-path> [test-class-filter]
 ---
 
 # Run Tests
 
-Run tests for a specific module. Auto-detects whether to run unit tests or system tests.
+Run tests for a specific module. Auto-detects whether to run unit tests or an integration test
+suite.
 
 ## Step 0: Check for Interactive Mode
 
@@ -19,7 +20,7 @@ In interactive mode, use AskUserQuestion at decision points as described in the
 
 The argument can be either:
 - A **file path** (e.g., `@sentry/src/test/java/io/sentry/ScopesTest.kt`)
-- A **module name** (e.g., `sentry-android-core`, `sentry-samples-spring-boot-4`)
+- A **module name** (e.g., `sentry-android-core`, `sentry-spring-boot-4`)
 - A **module name + test filter** (e.g., `sentry ScopesTest`)
 
 Extract the module name and optional test class filter from the argument.
@@ -30,10 +31,13 @@ Extract the module name and optional test class filter from the argument.
 
 | Signal | Test Type |
 |--------|-----------|
-| Path contains `sentry-samples/` | System test |
-| Module name starts with `sentry-samples-` | System test |
+| Path contains `/src/integrationTest*/` or `/src/agentFixture/` | Integration test suite |
+| Path contains `sentry-samples/`, or module name starts with `sentry-samples-` | None: samples have no tests |
 | Everything else | Unit test |
 
+For a sample, tell the user that samples have no tests, and that the end-to-end tests for what
+the sample shows live in the module it uses (see the `e2e_tests` rule).
+
 ## Step 3a: Run Unit Tests
 
 Determine the Gradle test task:
@@ -60,31 +64,23 @@ Without a filter:
 ./gradlew ':<module>:<task>' --info
 ```
 
-## Step 3b: Run System Tests
+## Step 3b: Run an Integration Test Suite
 
-System tests require the Python-based test runner which manages a mock Sentry server and sample app lifecycle.
+The suite's task is named after its source set directory: `src/integrationTestWebflux/...` runs
+with `integrationTestWebflux`. The OpenTelemetry agent suite runs in two tasks, one per init mode,
+so for `src/integrationTestOtelAgent/` or `src/agentFixture/` run both:
 
-1. Ensure the Python venv exists:
 ```bash
-test -d .venv || make setupPython
+./gradlew ':<module>:integrationTestOtelAgent' ':<module>:integrationTestOtelAgentWithoutAutoInit'
 ```
 
-2. Extract the sample module name. For file paths like `sentry-samples/<sample-module>/src/...`, the sample module is the directory name (e.g., `sentry-samples-spring`).
+Otherwise, with or without a test class filter:
 
-3. Run the system test:
 ```bash
-.venv/bin/python test/system-test-runner.py test --module <sample-module>
+./gradlew ':<module>:<suite>' --tests="*<filter>*"
 ```
 
-This starts the mock Sentry server, starts the sample app (Spring Boot/Tomcat/CLI), runs tests via `./gradlew :sentry-samples:<sample-module>:systemTest`, and cleans up afterwards.
-
-To run **every** system test instead of one module, use the Makefile targets — they also create the
-venv for you:
-
-```bash
-make systemTest             # all system tests (--all)
-make systemTestInteractive  # pick the setups to run (--interactive)
-```
+To run every Spring integration suite at once, use `./gradlew integrationTests`.
 
 ## Step 4: Report Results
 

.cursor/rules/e2e_tests.mdc

@@ -3,26 +3,83 @@ alwaysApply: false
 description: Java SDK End to End Tests
 ---
 
-# Java SDK End to End Tests (System Tests)
+# Java SDK End to End Tests
 
-The samples in the `sentry-samples` directory are used to run end to end tests against them.
+End-to-end tests live in the module whose code they test, not in `sentry-samples`. Samples have no
+tests. A test runs the code the way an app would and asserts what reaches Sentry over the real HTTP
+transport.
 
-There is a python script (`system-test-runner.py`) that can be used to run one (using `--module SAMPLE_NAME`) or all (using `--all`) system tests.
+## MockSentryServer
 
-The script has an interactive mode (`-i`) which allows selection of test setups to execute, whether to run the tests or just prepare infrastructure for testing from IDE.
+`MockSentryServer` (`sentry-test-support`) is an in-process HTTP server on a free port. Point the
+SDK at `server.dsn`, then:
 
-The tests run a mock Sentry server via `system-test-sentry-server.py`. Any system under test will then have a DSN set that reflects this local mock server like `http://502f25099c204a2fbf4cb16edc5975d1@localhost:8000/0`.
-By using this local DSN, the system under test sends events to the local mock server.
-The tests can then use `TestHelper` to assert envelopes that were received by the mock server.
-`TestHelper` uses HTTP requests to retrieve the JSON payload of the received events and deserialize them back to objects for easier assertion.
-Tests can then assert events, transactions, logs etc. similar to how they would appear in `beforeSend` and similar callbacks.
+- `awaitEnvelope(timeoutMillis) { predicate }` polls for a matching envelope and fails with what it
+  received on timeout.
+- `assertNoEnvelope(millis) { predicate }` checks that nothing matching arrives.
+- `MockSentryServer { ... }` takes a hook that runs before each check while waiting, for example to
+  flush an OpenTelemetry tracer provider.
 
-`TestHelper` has a lot of helper methods for asserting, e.g. by span name, log body etc.
+`EnvelopeMatchers.kt` has the accessors and matchers: `event()`, `transaction()`, `logs()`,
+`metrics()`, `profileChunk()`, `hasSpanWithOp`, `hasFlag`, `hasLogWithAttribute`, `hasMetric` and
+so on.
 
-The end to end tests either expect the system under test to either be running on a server or call `java -jar` to execute a CLI system under test.
+Call `Sentry.flush()` before awaiting when Sentry runs in the test's JVM: logs and metrics are
+batched. Envelopes that arrive late can outlive a server's `reset()`, so tests that share a server
+match on values only they produce.
 
-For Spring Boot, we spin up the Spring Boot server. The tests then send requests to that server and assert what is sent to Sentry.
+## Where the tests are
 
-End to end tests are also executed on CI using a matrix build, as defined in `.github/workflows/system-tests-backend.yml`.
+- **Core and plain integrations** use their module's `src/test`. Examples: `SentryEndToEndTest`
+  (`sentry`), `SentryAppenderIntegrationTest` (`sentry-logback`, `sentry-log4j2`, loading the
+  appender from XML), `SentryHandlerIntegrationTest` (`sentry-jul`),
+  `ContinuousProfilerEndToEndTest` (`sentry-async-profiler`), `SentryKafkaEndToEndTest`
+  (`sentry-kafka`), `AgentlessEndToEndTest` and `OtlpEndToEndTest` (`sentry-opentelemetry-*`).
+- **Spring Boot** (`sentry-spring-boot`, `sentry-spring-boot-jakarta`, `sentry-spring-boot-4`) has
+  one JVM test suite per flavor, because a flavor is a classpath: Sentry's auto-configuration and
+  OpenTelemetry mode switch on which classes are present.
+  - `integrationTest`: servlet app.
+  - `integrationTestWebflux`: reactive app.
+  - `integrationTestOtelNoAgent`: `sentry-opentelemetry-agentless-spring`.
+  - `integrationTestOtelAgent` and `integrationTestOtelAgentWithoutAutoInit`: the built agent.
+  - `integrationTestOtlp` (Boot 4): `sentry-opentelemetry-otlp-spring`.
+  - `integrationTestLog4j2` (Boot 3, 4): Log4j2 instead of Logback, for the auto-configured
+    Log4j2 appender.
 
-Some of the samples are tested in multiple ways, e.g. with OpenTelemetry Agent auto init turned on and off.
+  Each suite has a base class with `@SpringBootTest(RANDOM_PORT)` and a `@DynamicPropertySource`
+  that sets `sentry.dsn`. Subclasses share one context and one server.
+- **Plain Spring** (`sentry-spring`, `sentry-spring-jakarta`, `sentry-spring-7`) has an
+  `integrationTest` suite that starts the app in an embedded Tomcat, the way a WAR starts. Spring
+  Boot stays off its classpath, because `SentrySpringServletContainerInitializer` behaves
+  differently with it.
+
+## The OpenTelemetry agent
+
+The agent must not run in the test's JVM, because it would instrument the test's own HTTP client
+and server. The Spring Boot modules have an `agentFixture` source set with the app. `AgentApp`
+starts it once per test JVM in a child process, with `ChildJvmApp` and the agent jar passed through
+`ChildJvmTestArguments`. The suite's task runs with the agent initializing Sentry, and
+`integrationTestOtelAgentWithoutAutoInit` runs the same tests with the Spring Boot starter doing
+it. Gradle runs the two in parallel.
+
+## Running them
+
+- The `spring-boot-{2,3,4}-matrix.yml` workflows run the suites against each supported Spring Boot
+  version, by setting the version in `gradle/libs.versions.toml`. The suites aren't part of
+  `check`, so a new suite only runs on CI once it's added to its matrix.
+- `./gradlew integrationTests` runs every Spring suite.
+
+## Gotchas
+
+- Don't use the context's `TestRestTemplate`. It comes from the `RestTemplateBuilder` that Sentry
+  customizes, and that the OpenTelemetry starter instruments, so it adds or replaces trace headers
+  on the requests a test sends. Build one outside the context.
+- Sentry exports OpenTelemetry spans in batches every 5 seconds. In-process, flush the tracer
+  provider from the server's hook. With the agent this isn't possible, so each test waits for a
+  batch and negative checks wait at least 6 seconds.
+- Sentry is global, so a second live Spring context in an OpenTelemetry suite takes it over. Keep
+  one live context per JVM (`@DirtiesContext` when a test needs its own).
+- The async profiler rejects a traces path with a `.` anywhere in its absolute form. Put the path
+  in `java.io.tmpdir`.
+- Kafka uses an in-JVM broker (`@EmbeddedKafka` or spring-kafka-test's broker classes), so no
+  Docker is needed.

.cursor/rules/new_module.mdc

@@ -69,8 +69,8 @@ static {
 addIntegrationToSdkVersion("{IntegrationName}");
 ```
 
-7. If adding a JVM sample, add E2E (system) tests, following the structure we have in the existing JVM examples.
-   The test should then be added to `test/system-test-runner.py` and `.github/workflows/system-tests-backend.yml`.
+7. Samples have no tests. Add end-to-end coverage of what the sample shows to the module it uses,
+   against `MockSentryServer`, as described in the `e2e_tests` rule.
 
 `sentry-bom` and the root `build.gradle.kts` need no change — they iterate over subprojects.
 

AGENTS.md

@@ -20,7 +20,7 @@ rule file in `.cursor/rules/`:
 | `continuous_profiling_jvm` | `sentry-async-profiler`, `IContinuousProfiler`, `ProfileChunk`, JFR files, `ProfileLifecycle` |
 | `opentelemetry` | `sentry-opentelemetry-*`, agent vs agentless, span processing, sampling, context propagation |
 | `new_module` | Adding a new integration or sample module |
-| `e2e_tests` | System tests, sample applications, `system-test-runner.py`, mock Sentry server |
+| `e2e_tests` | End-to-end tests, `MockSentryServer`, integration test suites, the OpenTelemetry agent fixture, `integrationTests` |
 
 Rules can be combined — a tracing scope issue may need both `scopes` and `opentelemetry`.
 There is no rule for Android profiling yet; read the `sentry-android-core` profiling code
@@ -55,7 +55,7 @@ The project uses **Gradle** with Kotlin DSL. Key build files:
 ```
 
 To run tests, use the `test` skill rather than composing the Gradle invocation by hand — it
-resolves the per-module test task and the unit-test vs system-test split for you.
+resolves the per-module test task, including integration test suites, for you.
 
 ### Android-Specific Commands
 ```bash
@@ -109,8 +109,7 @@ The repository is organized into multiple modules:
 - **Other**: `sentry-spotlight`, `sentry-kotlin-extensions`, `sentry-android-distribution`
 
 ### Utility Modules
-- **`sentry-test-support`** - Shared test utilities
-- **`sentry-system-test-support`** - System testing infrastructure
+- **`sentry-test-support`** - Shared test utilities, including `MockSentryServer` and envelope matchers
 - **`sentry-samples`** - Example applications
 - **`sentry-bom`** - Bill of Materials for dependency management
 
@@ -158,7 +157,7 @@ PR description why the broad catch is necessary.
 ### Testing Requirements
 - Write comprehensive unit tests for new features
 - Android modules require both unit tests and instrumented tests where applicable
-- System tests validate end-to-end functionality with sample applications
+- End-to-end tests run in the module they test, against `MockSentryServer`. Samples have no tests
 - **Assertions**: For new unit tests, prefer [Google Truth](https://truth.dev/) (`com.google.common.truth.Truth.assertThat`) over `kotlin.test`/JUnit assertions for its readable, fluent API. Keep using `kotlin.test` for test structure (`@Test`, `assertFailsWith`). See `sentry/src/test/java/io/sentry/DsnTest.kt` for the style. Don't rewrite existing `kotlin.test` assertions solely to switch libraries.
 - Truth is wired into the `sentry` module. When adding Truth-based tests to another module, add `testImplementation(libs.google.truth)` to that module's `build.gradle.kts`.