Skip to content

fix(telemetry): count a turn by its segment, not by its invoke_agent spans - #3324

Open
QuentinBisson wants to merge 3 commits into
kagent-dev:mainfrom
giantswarm:upstream/telemetry-count-turns-by-segment-v2
Open

QuentinBisson wants to merge 3 commits into
kagent-dev:mainfrom
giantswarm:upstream/telemetry-count-turns-by-segment-v2

Conversation

@QuentinBisson

@QuentinBisson QuentinBisson commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Recent Claude Code releases (2.1.308 included; 2.1.285 does not) mark their claude_code.interaction span with gen_ai.operation.name=invoke_agent. The span sits beneath the invoke_agent span the harness opens (Claude Code reaches it through TRACEPARENT), so the tracing e2e (go/core/test/e2e/tracing_test.go:417-426) sees two invoke_agent spans in one turn and fails. #3321 bumps Claude Code to 2.1.308, which hits it.

Change

The e2e requires exactly one invoke_agent span outside the native harness process, whose spans the reference collector strips by kagent.genai.producer=runtime, requires its initial segment, and requires Claude Code's interaction span directly beneath it. The contract and telemetry docs now say that Claude Code's span is a child of kagent's invoke_agent, and that the reference collector already removes its operation through the kagent.genai.producer=runtime strip. Consumers of the reference pipeline therefore see one invocation per turn.

…spans

Claude Code 2.1.302 marks its claude_code.interaction span with
gen_ai.operation.name=invoke_agent; 2.1.285 emits the same span without
it. The span is a child of the invoke_agent span kagent opens for the
harness, which it reaches through TRACEPARENT, so a Claude turn's trace
now holds two invoke_agent spans and TestE2ECompletedChatFlushesTraces
failed on every run of the Claude Code bump with "invoke_agent spans =
2, want exactly one".

kagent keeps its own span for a native coding agent: only it carries the
segment, the A2A identity, the capture and the duration metric. The
contract states the case: kagent opens invoke_agent for a runtime whose
own spans carry no invocation segment, a native agent may still mark a
span of its own invoke_agent beneath it, and a consumer counts turns by
kagent.invocation.segment, as the request-span notes already say. The
test selects the invocation by operation and segment and requires Claude
Code's interaction span under it.

The trace diagnostic lists the expected trace's spans before the other
traces' request spans, and with their operation: the parallel cases'
request spans filled its twelve entries and hid the trace that failed.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
@QuentinBisson
QuentinBisson requested a review from a team as a code owner October 22, 2026 16:03
@github-actions github-actions Bot added the bug Something isn't working label Oct 22, 2026
…'s native strip

kagent also opens invoke_agent for LangGraph, CrewAI and BYO runtimes, not only
native harnesses. Name the existing mechanism that keeps claude_code.interaction
from counting as a second invocation: the reference collector removes the GenAI
operation from kagent.genai.producer=runtime spans.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 22, 2026
QuentinBisson added a commit to giantswarm/kagent-upstream that referenced this pull request Oct 23, 2026
#881)

U1 is kagent-dev#3324, U6 is kagent-dev#3327, the webhook end state of the
controller binary is kagent-dev#3326, the egress ask is agent-substrate/substrate#3137
and the labels use case sits on substrate#186. kagent-dev#3299 took the Session create
namespace at all three sites. The session label and the Dex display rows go
back to to-file: the baggage coverage PR and the +15/-2 UI PR are the filings.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
Comment thread go/core/test/e2e/tracing_test.go Outdated
// compiler owns this identity, so the assertion holds without any
// Invocations are selected by operation and segment, the way the
// contract tells a consumer to count turns: a native runtime's own
// invoke_agent span, such as Claude Code's interaction span, carries

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we still assert no stray unsegmented invoke_agent spans?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, the segment selector lost that guard. The test now takes every invoke_agent span outside the native harness process (the ones carrying kagent.genai.producer=runtime, which the reference collector strips), requires exactly one, and checks its segment. Can you take another look?

…r keeps

Selecting the invocation by segment let a second invoke_agent span pass
unseen. Select every invoke_agent span outside the native harness
process, whose GenAI operation the reference collector removes by
kagent.genai.producer=runtime, require exactly one, and require its
segment.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants