opentelemetry-sdk: enforce metric cardinality limits by herin049 · Pull Request #5809 · open-telemetry/opentelemetry-python

herin049 · 2026-10-22T03:13:09Z

Description

Implements the metrics cardinality limit spec in the SDK and wires it through declarative configuration. Previously nothing bounded the number of attribute sets per metric stream, and opentelemetry-configuration parsed cardinality_limits and aggregation_cardinality_limit but ignored them.

Note: Updating metrics with delta temporality now require acquiring the lock, however this does not seems to have any noticeable impact on performance.

Benchmark	main median (ns)	PR median (ns)	Δ median
`counter_add[0-delta]`	1541	1542	+0.1%
`counter_add[1-delta]`	1958	1834	-6.3%
`counter_add[3-delta]`	2334	2333	-0.0%
`counter_add[5-delta]`	2792	2750	-1.5%
`counter_add[10-delta]`	3958	3833	-3.2%
`counter_add[0-cumulative]`	1500	1459	-2.7%
`counter_add[1-cumulative]`	1833	1750	-4.5%
`counter_add[3-cumulative]`	2333	2209	-5.3%
`counter_add[5-cumulative]`	2792	2666	-4.5%
`counter_add[10-cumulative]`	3958	3750	-5.3%
`up_down_counter_add[0]`	1459	1417	-2.9%
`up_down_counter_add[1]`	1792	1750	-2.3%
`up_down_counter_add[3]`	2292	2209	-3.6%
`up_down_counter_add[5]`	2791	2666	-4.5%
`up_down_counter_add[10]`	3917	3709	-5.3%
`counter_add_with_meter_configurator_rules[None]`	125	88	-29.8%
`counter_add_with_meter_configurator_rules[0]`	1459	1458	-0.1%
`counter_add_with_meter_configurator_rules[1]`	1438	1458	+1.4%
`counter_add_with_meter_configurator_rules[10]`	1459	1427	-2.2%
`counter_add_with_meter_configurator_rules[50]`	1500	1458	-2.8%

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

I am a human and not a bot
I have run tox -e precommit
I have run tox -e typecheck
I have run the relevant tests and linting tox environments
This requires the following PR in contrib:

opentelemetry-pr-dashboard · 2026-10-22T03:19:15Z

Pull request dashboard status

Waiting on reviewers · refreshed 2026-10-25 03:37 UTC

Review the latest changes.

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Copilot

🟡 Changes recommended

Asynchronous collection handling can lose attributes and emit negative counter deltas, while plugin pull readers silently ignore configured limits.

3 open issues

What changed in this PR

This PR adds metric cardinality limits to the SDK and connects them to declarative configuration and the Prometheus reader.

Changes:

Enforces a default limit of 2000 points with overflow aggregation and configurable reader/view limits.
Adds collection-cycle handling for synchronous delta metrics.
Adds SDK, configuration, Prometheus, and Docker scenario coverage.

File	Description
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limits_view/scenario.yaml	Checks view precedence and attribute filtering.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limits_view/run.py	Records measurements for view-limit checks.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limits_view/otel-config.yaml	Configures reader and view limits.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limits_reader/scenario.yaml	Checks per-instrument limits and fallback.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limits_reader/run.py	Records counter and histogram measurements.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limits_reader/otel-config.yaml	Configures default and counter limits.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limit_delta/scenario.yaml	Checks limits across delta collections.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limit_delta/run.py	Records two separated measurement batches.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limit_delta/otel-config.yaml	Configures delta export and cardinality limits.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limit_default/scenario.yaml	Checks the default limit and overflow.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limit_default/run.py	Records measurements exceeding the default limit.
tests/opentelemetry-docker-tests/tests/declconfig/scenarios/metric_cardinality_limit_default/otel-config.yaml	Configures export without explicit limits.
opentelemetry-sdk/tests/metrics/test_view.py	Tests view-limit validation.
opentelemetry-sdk/tests/metrics/test_measurement_consumer.py	Updates the reader mock for cardinality settings.
opentelemetry-sdk/tests/metrics/test_in_memory_metric_reader.py	Tests reader defaults and validation.
opentelemetry-sdk/tests/metrics/integration_test/test_cardinality_limit.py	Tests overflow, precedence, temporality, and concurrency.
opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/__init__.py	Exposes cardinality configuration types.
opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/view.py	Adds and validates view-level limits.
opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/metric_reader_storage.py	Resolves limits and collection-reset behavior.
opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/measurement_consumer.py	Passes reader limits to storage.
opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/export/__init__.py	Adds reader limit parameters and validation.
opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/_view_instrument_match.py	Enforces limits and routes overflow measurements.
opentelemetry-configuration/tests/test_meter_provider.py	Tests declarative reader and view settings.
opentelemetry-configuration/src/opentelemetry/configuration/_meter_provider.py	Maps declarative limits to SDK settings.
exporter/opentelemetry-exporter-prometheus/tests/test_prometheus_exporter.py	Tests Prometheus overflow output.
exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/__init__.py	Accepts and forwards cardinality limits.
.changelog/5809.added	Documents the feature and configuration options.

_{Review effort: Balanced}

ocelotl · 2026-10-23T00:51:41Z

@herin049 please resolve conflicts ✌️

road-cycling · 2026-10-23T22:22:03Z

Following, thanks for adding this @herin049.

hectorhdzg · 2026-10-23T23:30:56Z

+
+        # One slot is reserved for the overflow aggregation, so the number of
+        # exported points never exceeds the cardinality limit.
+        if len(self._attributes_aggregation) >= self._cardinality_limit - 1:


With a limit of 100 the 100th distinct attribute set already goes to overflow, because the check is len >= limit - 1. The spec says overflow must not happen when the number of distinct sets is less than or equal to the limit, so I think this should be len >= limit. Can we add a test with exactly limit distinct sets and no overflow?

I should have made this more clear in the PR, but I was split on how to do this. Currently Go, Java and JavaScript allow limit - 1 distinct attribute streams with the overflow stream whereas Rust, C++ and .NET allow limit distinct attribute streams with the overflow stream.

Honestly, I think the difference here is minor, but I do agree that it's probably better to allow limit distinct attribute stream with the overflow stream even though technically this means there are limit + 1 distinct streams.

We could implement this in a way where we only overflow when there is an attempt to create a new metric stream while the cardinality is at its limit. We could then just update the attribute set of the last inserted metric stream to contain the overflow attributes and effectively make this our overflow bucket. However, adding this logic would convolute everything even further, so I'd prefer if we just stick with the simpler approach above and accept that we technically exceed the limit by 1.

Can we add a test with exactly limit distinct sets and no overflow?

This should already be included in the Docker test scenarios I added. I've updated these scenarios to validate that overflow only starts beyond limit attribute streams instead of limit - 1 (as it was previously).

hectorhdzg · 2026-10-23T23:31:50Z

        aggr_key = _hash_attributes(attributes)

-        if aggr_key not in self._attributes_aggregation:
+        if self._reset_on_collect:


For sync delta instruments every measurement now takes the lock, so threads recording to the same stream serialize. Before it was a lock-free lookup in the common case, and since this is the hot path it could be worth looking at. Would there be a way to handle the dict swap in collect without locking on every record?

Yes, I did realize this. Unfortunately making this lock free is a bit tricky because of this scenario.

Caller acquires reference to stream with attributes {x=a, y=b}

Collect is called and exports the contents from stream with attributes {x=a, y=b} (stream is removed from lookup dict)

Update to stream with attributes {x=a, y=b} completes.

Stream is GC'd and data is lost from this increment.

If we had atomics, we could probably make this work, but I don't currently have a simplish implementation that achieves this. We can maybe create a tracking issue to try to make this lock free again. Let me also run a few benchmarks just to see how bad this regression is.

Here are some benchmarks, they don't seem to have any measurable impact for delta metrics.

Benchmark main median (ns) PR median (ns) Δ median

counter_add[0-delta] 1541 1542 +0.1%

counter_add[1-delta] 1958 1834 -6.3%

counter_add[3-delta] 2334 2333 -0.0%

counter_add[5-delta] 2792 2750 -1.5%

counter_add[10-delta] 3958 3833 -3.2%

counter_add[0-cumulative] 1500 1459 -2.7%

counter_add[1-cumulative] 1833 1750 -4.5%

counter_add[3-cumulative] 2333 2209 -5.3%

counter_add[5-cumulative] 2792 2666 -4.5%

counter_add[10-cumulative] 3958 3750 -5.3%

up_down_counter_add[0] 1459 1417 -2.9%

up_down_counter_add[1] 1792 1750 -2.3%

up_down_counter_add[3] 2292 2209 -3.6%

up_down_counter_add[5] 2791 2666 -4.5%

up_down_counter_add[10] 3917 3709 -5.3%

counter_add_with_meter_configurator_rules[None] 125 88 -29.8%

counter_add_with_meter_configurator_rules[0] 1459 1458 -0.1%

counter_add_with_meter_configurator_rules[1] 1438 1458 +1.4%

counter_add_with_meter_configurator_rules[10] 1459 1427 -2.2%

counter_add_with_meter_configurator_rules[50] 1500 1458 -2.8%

herin049 requested a review from a team as a code owner October 22, 2026 03:13

otelbot-python Bot added this to Python PR digest Oct 22, 2026

herin049 requested a balanced review from Copilot October 22, 2026 03:15

Copilot started reviewing on behalf of herin049 October 22, 2026 03:15

View session

opentelemetry-pr-dashboard Bot mentioned this pull request Oct 22, 2026

Pull Request Dashboard #5550

Open

Copilot AI reviewed Oct 22, 2026

View reviewed changes

tammy-baylis-swi moved this to Ready for review in Python PR digest Oct 22, 2026

hectorhdzg reviewed Oct 23, 2026

View reviewed changes

Comment thread opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/metric_reader_storage.py Outdated

hectorhdzg reviewed Oct 23, 2026

View reviewed changes

herin049 added 4 commits October 24, 2026 00:02

opentelemetry-sdk: enforce metric cardinality limits

1686c2d

add changelog fragment

324269c

fix docs errors

afb99b6

add warning log for plugins

e2e47b8

herin049 force-pushed the feat/metrics-cardinality-limit branch from aa01663 to e2e47b8 Compare October 24, 2026 04:02

address PR comments

89b0c42

herin049 requested a review from hectorhdzg October 25, 2026 03:37

Navigation Menu

Repository navigation

opentelemetry-sdk: enforce metric cardinality limits - #5809

opentelemetry-sdk: enforce metric cardinality limits#5809
herin049 wants to merge 5 commits into
open-telemetry:mainfrom
herin049:feat/metrics-cardinality-limit

herin049 commented Oct 22, 2026 •

edited

Loading

Uh oh!

opentelemetry-pr-dashboard Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocelotl commented Oct 23, 2026

Uh oh!

road-cycling commented Oct 23, 2026

Uh oh!

Uh oh!

hectorhdzg Oct 23, 2026

Uh oh!

herin049 Oct 24, 2026

Uh oh!

herin049 Oct 24, 2026

Uh oh!

hectorhdzg Oct 23, 2026

Uh oh!

herin049 Oct 24, 2026 •

edited

Loading

Uh oh!

herin049 Oct 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

herin049 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist:

Uh oh!

opentelemetry-pr-dashboard Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ocelotl commented Oct 23, 2026

Uh oh!

road-cycling commented Oct 23, 2026

Uh oh!

Uh oh!

hectorhdzg Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

herin049 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

herin049 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

hectorhdzg Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

herin049 Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

herin049 Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

herin049 commented Oct 22, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Oct 22, 2026 •

edited

Loading

herin049 Oct 24, 2026 •

edited

Loading