Skip to content

feat(divan): export benchmark tracing spans over OTLP behind an otel feature - #197

Draft
lvaroqui wants to merge 4 commits into
mainfrom
cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat
Draft

lvaroqui wants to merge 4 commits into
mainfrom
cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat

Conversation

@lvaroqui

Copy link
Copy Markdown
Contributor

Add an opt-in otel feature to codspeed and codspeed-divan-compat that exports the tracing spans and events emitted by benchmarks over OTLP/HTTP.

With the feature enabled, divan::main() installs a global tracing subscriber when CODSPEED_OTEL_ENDPOINT is set, and flushes the spans when the run ends. CODSPEED_OTEL_FILTER takes EnvFilter directives to select what is exported, and defaults to info. Nothing changes when the variable is unset or the feature is off; all the OpenTelemetry dependencies are optional.

Benchmarks that already set their own global subscriber are left untouched. They can add divan::otel::layer() to it instead, as documented in the README.

Worth a careful look:

  • Spans emitted by the exporter's own HTTP stack (hyper, reqwest, opentelemetry_*, …) are always filtered out. Exporting them would create more spans, in an endless loop.
  • The OpenTelemetry service name is the bench executable name, with the hash cargo appends to it stripped.

benches/otel_example.rs emits nested spans and events, and a debug span that the default filter drops. Point it at any OTLP/HTTP receiver to check the output:

CODSPEED_OTEL_ENDPOINT=http://localhost:4318 cargo bench -p codspeed-divan-compat --features otel --bench otel_example

Closes COD-3862

…feature

Add an opt-in `otel` feature to `codspeed` and `codspeed-divan-compat`. When
`CODSPEED_OTEL_ENDPOINT` is set, `divan::main()` installs a global `tracing`
subscriber that exports the spans and events emitted by benchmarks over
OTLP/HTTP, filtered by `CODSPEED_OTEL_FILTER` (defaults to `info`), and
flushes them when the run ends. Spans from the exporter's own HTTP stack are
always filtered out to avoid an export loop.

Benchmarks that already set a global subscriber are left untouched; they can
compose `divan::otel::layer()` into their own subscriber instead.

Closes COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codspeed

codspeed Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 18 benchmarks

⚠️ 59 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ 9 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

ℹ️ 49 benchmarks have no flame graph
  • The flamegraph could not be generated because too much time was spent in runtime internals, making the visualization not useful for performance analysis. (1 benchmark)
  • The flamegraph could not be generated because no execution data was recorded when the benchmark was run. Ensure that the benchmark has not been optimized out. (48 benchmarks)
⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 18 improved benchmarks
❌ 18 regressed benchmarks
✅ 558 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation find_highest_set_bit[0] 172.3 ns 282 ns -38.91%
❌ WallTime graph_coloring[5] 1.1 µs 1.4 µs -25.65%
❌ WallTime hamiltonian_cycle[6] 1.1 µs 1.5 µs -25.41%
❌ Simulation rem 178.1 ns 232.3 ns -23.32%
❌ Simulation div 178.1 ns 232.3 ns -23.32%
❌ WallTime recursive[5] 34 ns 44 ns -22.73%
❌ Simulation iterative[5] 234.1 ns 289.6 ns -19.18%
❌ Simulation find_highest_set_bit[42] 229.3 ns 283.5 ns -19.11%
❌ Simulation find_highest_set_bit[1024] 229.3 ns 283.5 ns -19.11%
❌ Simulation find_highest_set_bit[255] 229.3 ns 283.5 ns -19.11%
❌ Simulation find_highest_set_bit[65535] 229.3 ns 283.5 ns -19.11%
❌ Simulation iterative[10] 245.4 ns 300.9 ns -18.46%
❌ Simulation iterative[20] 267.9 ns 323.5 ns -17.17%
❌ Simulation iterative[30] 290.4 ns 346 ns -16.06%
❌ WallTime sort 118.3 µs 139.2 µs -15.02%
❌ Simulation process_items_with_counter 1.8 µs 1.9 µs -5.67%
❌ WallTime add_two_integers[(255, 255)] 19 ns 20 ns -5%
❌ WallTime add_two_integers[(42, 13)] 19 ns 20 ns -5%
⚡ WallTime bench_array1[4] 2 ns 1 ns ×2
⚡ WallTime iter_batched_ref_large_input 13 ns 7 ns +85.71%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3862-featrust-otel-cargo-feature-in-codspeed-divan-compat (6c3f81a) with main (5aaf570)

Open in CodSpeed

lvaroqui and others added 3 commits October 14, 2026 12:32
…sions

`reqwest` pulls `idna_adapter` 1.2.2 and the `icu_*` 2.x crates, which require
rustc 1.88. Resolve them with the MSRV-aware fallback resolver so the lockfile
builds on rustc 1.85.

`sharded-slab` declares `lazy_static = "1"` but fails to build with 1.0.x, which
breaks the minimal-versions check. Raise the floor with an optional direct
`lazy_static = "1.1"` dependency enabled by the `otel` feature.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spans exported with the otel feature carried nothing tying them to a
benchmark, so a suite with several benchmarks could not be told apart.

Add `benchmark_scope::enter`/`exit`, which the integrations call around
each benchmark, outside the measured section. A span processor sets the
`codspeed.benchmark.uri` attribute on every span started inside the
scope. The scope is process-wide, so spans of threaded benchmarks are
labelled too. Without the otel feature, both calls do nothing.

The divan walltime harness now computes the benchmark URI before running
it instead of after.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The batch span processor exports on a 5s timer and whenever 512 spans
are queued, so exports could run inside the measured section, and it
dropped every span past a 2048-span queue.

Replace it with a processor holding the ended spans in memory, exported
when the benchmark scope exits, after the measured section and before
the benchmark is reported as executed, and on shutdown. Exports are
sent in requests of 4096 spans. Past 1,000,000 held spans, spans are
dropped with a warning, to bound the memory of a benchmark emitting
spans on every iteration.

Refs COD-3862
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant