Repository navigation
feat(mcp): Record side quest next-call fingerprints so taken quests can be measured - #2393
Draft
sentry-junior[bot] wants to merge 1 commit into
Draft
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
…an be measured Each shown side quest records fingerprints of the next calls it offered as app.side_quest.<id>.next, and every tool call records its own fingerprint as app.tool_call.fingerprint (on the inner span for execute_sentry_tool). Analytics can then join a later call from the same user to the quest that offered it, with no server-side state. Co-Authored-By: Christina Long <60594860+Christinarlong@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We record when a side quest is shown, but not whether the agent took it. This adds what a funnel needs, for every quest that has
nextcalls, with no server-side state:nextcalls asapp.side_quest.<id>.next, a comma-separated list of fingerprints. Only calls the session can make are recorded, which is the same list the agent sees.app.tool_call.fingerprint. Forexecute_sentry_tool, it goes on the child span of the tool it runs, not on the wrapper.A fingerprint is a 32-bit FNV-1a hash of the tool name and its arguments, sorted by key. It leaves out scope keys (organization, project, region) and empty values. The server fills scope keys from constraints or the default org, so an agent may or may not pass them, and including them would miss real matches.
Measuring it: "taken" means a later span from the same
user.id, within a window (say 60 min), whose fingerprint is in an earlier span's.nextlist. Remote spans are sampled at 0.3. Afaict each tool call is its own trace, so both sides are sampled independently. Start from sampled spans that showed the quest, count how many have a sampled matching follow-up, and divide that share by 0.3. stdio samples at 1.0, so its pairs are complete.Not covered:
get_sentry_resourcewith aurlorissueIdalias instead ofresourceId, or the agent going straight toupdate_issueon a related issue. Those count as not taken, so the rate is a lower bound.related_issues_openlists up to 3 issues indetails, butnext(and so.next) holds only the first 2.Open decision: this uses span attributes only and does the matching in analytics (DATA-672). An exact, unsampled
app.side_quest.takenmetric would need the server to remember offered fingerprints per user: an in-memory map for stdio, andMCP_CACHEKV for remote. That is left out on purpose and can come later if the sampled estimate is too noisy.via @Christinarlong.
--
View Session [Sentry]