[dodjdnh]

Summary

Add Provider Request Rate and Consumer Request Rate links to Dubbo RPC spans in Jaeger. In Dubbo Admin's Grafana Trace Explore view, users can expand a span and open its Prometheus query through Grafana's native Trace-to-Metrics support, retaining the trace in Split Explore.

Correlation uses application, interface, and method and queries existing Dubbo request counters. It adds no Admin API, Vue page, spanmetrics pipeline, or RPC metric. Results are aggregate request rates for the RPC identity in the selected window, not metrics exclusive to one trace.

Motivation

Metric and Trace views currently require users to switch manually and re-enter the RPC identity and time range.

Some older dubbo-go versions also emit rpc.service as [group/]interface[:version], while Prometheus uses a plain interface label. Direct mapping may return no matching metrics. This PR adds normalized correlation attributes without changing the original trace identity.

Changes

1. Supply metric-correlation identity without overwriting original trace fields

• Run a transform after k8sattributes in the existing Collector trace pipeline, restricted to rpc.system=apache_dubbo spans meeting the relevant field conditions.
• Copy resource service.name into span attribute dubbo.application for mapping to application_name.
• Derive dubbo.metric.interface from nonempty rpc.service, removing legacy ServiceKey group/version components while preserving plain Java/Go interface names.
• Preserve original rpc.service and resource identity. Retain parsed group/version separately; explicit dubbo.group / dubbo.version attributes take precedence.
• Do not add these attributes to non-Dubbo spans or retroactively modify traces already stored in Jaeger.

2. Use Grafana's native span-to-Prometheus correlation

• Assign the stable Prometheus datasource UID dubbo-admin-prometheus and reference it from Jaeger's tracesToMetrics configuration.
• Preserve the existing Jaeger datasource UID, endpoint, type, and access settings.

Span attribute	Prometheus label
dubbo.application	application_name
dubbo.metric.interface	interface
rpc.method	method

Provide two links:

# Provider Request Rate
sum by (application_name, interface, method) (
  rate(dubbo_provider_requests_total{$__tags}[5m])
)

# Consumer Request Rate
sum by (application_name, interface, method) (
  rate(dubbo_consumer_requests_total{$__tags}[5m])
)

• Write $$__tags in provisioning so Grafana's environment-variable interpolation preserves runtime $__tags for selected-span matchers.
• Use spanStartTimeShift="-5m" and spanEndTimeShift="5m": span start minus five minutes through span end plus five minutes. Do not use the unsupported +5m interval.
• Offer both links without inferring provider/consumer side from span.kind.
• Retain group/version as span metadata, but do not include them in the current selector or isolate individual instances.

3. Add repeatable checks and deployment guidance

• A static contract script checks pipeline wiring, normalization rules, and datasource mappings.
• An isolated Collector script validates the release configuration and executes its actual OTTL transform with Collector Contrib 0.112.0. Cases include plain interfaces, legacy ServiceKeys, explicit attribute precedence, missing identity, and non-Dubbo spans.
• Documentation explains explicit Collector/Grafana restarts and verification through datasource APIs and fresh spans. Updating the ConfigMap/Secret alone does not prove runtime configuration has loaded.

Verification

The static checks below were rerun on the rebased branch. Runtime/browser PASS entries are results recorded in the supplied verification draft for the earlier implementation; they were not rerun after this rebase and are not a claim of fresh end-to-end acceptance of the rebased branch.

Area	Check	Result
Unit	No Go/Vue implementation changes or corresponding new unit tests	N/A; does not replace configuration/runtime verification
Static	Only Collector transform, datasource provisioning, two check scripts, and documentation in the feature diff	PASS
Static	git diff --check upstream/develop HEAD	PASS
Static	pwsh -File scripts/verify-trace-metric-servicekey.ps1	PASS
Static	Datasource embedded JSON parsing	PASS
Integration	Text merge simulation against latest upstream and each of #1587 / #1591	PASS; not a combined runtime test
Static	Full release configuration validation with Collector Contrib 0.112.0	NOT VERIFIED on the rebased branch
Isolated Runtime	Actual OTTL execution and original-field preservation	Previously recorded PASS; not rerun after rebase
Runtime	Jaeger/Prometheus datasource readback and health	Previously recorded PASS; not rerun after rebase
Runtime	Fresh Java RPC, Jaeger span, real counters, and generated PromQL	Previously recorded PASS; not rerun after rebase
Runtime	Legacy Go RPC with nonempty group/version, normalization, and real metric matching	Previously recorded PASS; not rerun after rebase
Browser	Admin iframe span links, Split Explore, selectors, and time window	Previously recorded PASS; not rerun after rebase
Regression	Existing Metric/Trace views and non-Dubbo traces	Previously recorded PASS; not rerun after rebase

Earlier human proof used Grafana 12.1.0, Jaeger 1.70, and real Dubbo Java 3.2.16 to verify native span links, RPC-label queries, and Split Explore inside the Admin iframe. This does not establish support for every SDK/version combination.

Review Focus

1. Whether Collector-level enrichment is the appropriate compatibility boundary for plain and legacy RPC identities without requiring SDK upgrades, and whether resource service.name matches metric application_name in target deployments.
2. Whether legacy ServiceKey parsing, explicit group/version precedence, and original-field preservation are correct; incomplete identity must not silently produce overly broad queries.
3. Whether three-label aggregation and two unconditional links provide clear semantics without implying trace-exclusive, per-instance, group/version, or automatic-side correlation.
4. Whether provisioning preserves $__tags and the intended span-relative time range at runtime.
5. The effect of assigning a stable Prometheus UID on existing custom references, and the explicit configuration reload/restart requirements.

Compatibility / Impact

• No Admin backend API, Vue component, or existing Dashboard JSON changes. Links are supplied by the native Grafana Jaeger datasource.
• Legacy dubbo-go v3.1.1/v3.3.2 ServiceKeys are compatibility targets, not a claim of runtime acceptance for all Go/Java versions.
• New attributes apply only to spans passing through the updated pipeline. Generate fresh RPCs for verification.
• Existing counters must be scraped. Queries may be empty when the selected side has no counter, is not scraped, or has insufficient samples.
• Jaeger UID is preserved. Existing installations must check custom references before adopting the stable Prometheus UID; do not overwrite environment-specific settings blindly.
• No automatic rollout is added. Existing deployments must explicitly reload configuration and inspect fresh spans and Grafana API readback rather than relying only on Pod readiness.
• The verification target is Grafana 12.1.0, Jaeger 1.70, Collector Contrib 0.112.0, and existing Prometheus; other combinations are not implicitly verified.

Impact:

• Docs
• Installation
• User Experience
• Dubboctl
• Console
• Core Component

Scope

Included

• Dubbo RPC span correlation identity and legacy dubbo-go ServiceKey normalization.
• Native Jaeger-to-Prometheus request-rate links, time shifts, and datasource provisioning.
• Static/isolated checks and deployment verification documentation.

Not Included

• Log observability or Trace-to-Log / Log-to-Trace navigation.
• Java/Go query compatibility changes in standalone Metric dashboards.
• SDK changes, new RPC metrics, histograms, or success/error/latency-percentile links.
• Tempo, spanmetrics, exemplars, service graphs, or a new frontend navigation implementation.
• Instance/group/version isolation or automatic provider/consumer selection.
• Automatic cluster updates or migration of custom datasource UIDs.

Related Issue

Fixes #1606.

Relates to #1524.

Dependency

Built on the Collector infrastructure merged in #1584. Rebased onto upstream develop at f1c3f3a; this PR contains only its own two commits and does not depend on #1587 or #1591.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[dodjdnh]

e2e测试
1.证明otel成功注入span标签

2.Trace → Metric 不仅能跳转，还能带有真实的promel语句查询到真实的 Prometheus 指标数据，consumer和provider关联均通过

3.+-5min metric查询窗口实现正确

[dodjdnh]

4.go旧版带servicekey测试，正常拆分group，version，接口，并且trace->metric关联查询成功

[AlexStocks]

结论：方向正确，可以合入

PR 只改 5 个文件（+419/-1），changed_files=5 与 Diff 一致，没有 Go 代码路径变化，风险集中在 Collector 配置与文档的准确性上。

已核对

otelCollector-configuration.yaml 的 transform/dubbo_application 与文档描述一致，处理顺序正确：

1. 从 rpc.service 复制出 dubbo.metric.interface；
2. replace_pattern(..., "^[^/]+/", "") 去掉 [group/] 前缀；
3. 当 dubbo.version 缺失且 interface 以 :version 结尾时，先取出 dubbo.metric.version，再从 interface 剥掉版本；
4. dubbo.metric.group 先按 ^[^/]+/.+ 提取，再用显式的 dubbo.group / dubbo.version 覆盖。

这与 scripts/verify-trace-metric-collector.py 里的 11 个 fixture（含 COLON_IN_GROUP、EXPLICIT_PRECEDENCE、NO_RESOURCE_NAME）能一一对应；脚本还会断言 pipeline 顺序必须是 k8sattributes, transform/dubbo_application，这个检查很有价值。

P3 · 验证脚本没有 CI 执行路径

verify-trace-metric-collector.py 需要本机存在 otelcol-contrib 0.112.0 原生可执行文件，.github/workflows 里没有任何作业会运行它；Windows 脚本版本同理。也就是说，这份 419 行的配置改动目前只有作者本地验证。

建议后续加一个 job：用 otel/opentelemetry-collector-contrib:0.112.0 容器跑该脚本（脚本本身已支持 --collector 指定二进制）。否则 Collector 升版时这里会在无人察觉的情况下失效。

P3 · 两套脚本重复维护

Python 版（178 行）与 Windows 版维护同一份 fixture 矩阵。当前两份内容一致，但后续任何规则调整都需要改两处，容易漂移。若 Windows 版本只是为了本地验证，建议让它调用 Python 版本，或把 fixture 抽成一个 JSON 文件由两者共用。

P3 · error_mode: ignore 会掩盖配置错误

transform/dubbo_application 使用 error_mode: ignore。运行时这能避免一条坏 span 影响整批，但也会让 OTTL 语句里的拼写错误静默通过。配合上面的 CI 检查会更稳妥。

[dodjdnh]

结论：方向正确，可以合入

PR 只改 5 个文件（+419/-1），changed_files=5 与 Diff 一致，没有 Go 代码路径变化，风险集中在 Collector 配置与文档的准确性上。

已核对

otelCollector-configuration.yaml 的 transform/dubbo_application 与文档描述一致，处理顺序正确：

1. 从 rpc.service 复制出 dubbo.metric.interface；
2. 移除 [group/] 前缀；
3. 当 dubbo.version 缺失且 interface 以 :version 结尾时，先取出 dubbo.metric.version，再从 interface 剥掉版本；
4. dubbo.metric.group 先按 ^[^/]+/.+ 提取，再用显式的 dubbo.group / dubbo.version 覆盖。

这与 scripts/verify-trace-metric-collector.py 里的 11 个 fixture（含 COLON_IN_GROUP、EXPLICIT_PRECEDENCE、NO_RESOURCE_NAME）能一一对应；脚本还会断言 pipeline 顺序必须是 k8sattributes, transform/dubbo_application，这个检查很有价值。

P3 · 验证脚本没有 CI 执行路径

verify-trace-metric-collector.py 需要本机存在 otelcol-contrib 0.112.0 原生可执行文件，.github/workflows 里没有任何作业会运行它；Windows 脚本版本同理。也就是说，这份 419 行的配置改动目前只有作者本地验证。

建议后续加一个 job：用 otel/opentelemetry-collector-contrib:0.112.0 容器跑该脚本（脚本本身已支持 --collector 指定二进制）。否则 Collector 升版时这里会在无人察觉的情况下失效。

P3 · 两套脚本重复维护

Python 版（178 行）与 Windows 版维护同一份 fixture 矩阵。当前两份内容一致，但后续任何规则调整都需要改两处，容易漂移。若 Windows 版本只是为了本地验证，建议让它调用 Python 版本，或把 fixture 抽成一个 JSON 文件由两者共用。

P3 · error_mode: ignore 会掩盖配置错误

transform/dubbo_application 使用 error_mode: ignore。运行时这能避免一条坏 span 影响整批，但也会让 OTTL 语句里的拼写错误静默通过。配合上面的 CI 检查会更稳妥。

明白