Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,22 @@ Count package acquisition and successful instruction delivery with the exact sou

## Requirements and scenarios

- FR-1: A recognized remote package acquisition emits one `gen_ai_skill_download` observation on completion, including validated disk-cache restoration and failure. Invalid cache followed by download is one attempt, with `cache_hit=false`.
- FR-2: A successful instruction response emits one `gen_ai_skill_load`. Two successful `load_skill` calls in one invocation count twice even when ADK fetches once. Unknown skills, read failures and cancellation never count as successful loads.
- FR-1: A recognized remote package acquisition emits one `gen_ai_skill_download_count` observation on completion, including validated disk-cache restoration and failure. Invalid cache followed by download is one attempt, with `skill.download.cache_hit=false`.
- FR-2: A successful instruction response emits one `gen_ai_skill_load_count`. Two successful `load_skill` calls in one invocation count twice even when ADK fetches once. Unknown skills, read failures and cancellation never count as successful loads.
- FR-3: Cloud identity survives conversion/cache. Sandbox Python distinguishes built-ins, SkillHub slugs, explicit AgentKit IDs and both space kinds. Go retains its existing four-source support. Unattributable local skills are not guessed to be built-ins.
- FR-4: Use existing process-global OTEL providers; sandbox installs a periodic OTLP metric exporter only when no provider exists. Trace and metric disable switches are independent. Export failures do not affect conversations; shutdown flush is bounded.
- FR-5: New telemetry never records archive URLs, credentials, instruction bodies, paths or raw exception messages. Existing content telemetry policy remains unchanged.
- FR-6: Python and Go use the same labels and source semantics for supported sources; prefetch produces downloads only. Existing installed packages/session links are retained.

The current schema follows technical document revision 2419, read on 2026-10-21, plus the subsequent user-requested unit change to `time`: both counters use the `_count` suffix and unit `time`, with `skill_name` added to identity. Outcome is only `skill.download.outcome`, no longer a metric label. Spans are named `download_skill {skill_name}` / `load_skill {skill_name}` and use `skill.consumer` and `skill.download.cache_hit`. Name provenance and empty-value rules are in the component contract. Preserve fixed Counter instruments; fields and record methods do not need a `gen_ai` prefix.

## Design and affected consumers

Runtime: add a small `veadk/skills/telemetry.py` recorder, instrument the existing materializer, preserve cloud identity under reserved ADK frontmatter metadata, and extend the existing VeADK tool tracing callback for successful native ADK `LoadSkillTool` responses. No replacement toolset or new global cache. Runtime load telemetry follows the existing VeADK tracing-hook activation; metrics are recorded independently of span sampling once the hook is installed.

Sandbox: implement a local recorder using already-installed OTEL packages, because the sandbox's released SDK must work independently of unpublished VeADK changes. Instrument `playground_skills/packages.py`, SkillHub reconciliation, and the existing `ObservedSkillsTool`. Persist the installed SkillHub slug alongside its package so a failed replacement cannot relabel an older built-in. The Go package store records downloads; catalog metadata carries identity into the existing tracked tool wrapper. Built-ins produce no download events. Configure metric export in existing telemetry setup and flush with existing shutdown handling.

Counters count acquisition attempts after identity resolution. Metadata-list failures without a resolved skill ID cannot produce a per-skill download. Successful acquisition means a validated usable package, not just HTTP success. Python worker-thread downloads may finish after awaiting-task cancellation; count their actual acquisition result, never a cancelled load. Go context cancellation is an error outcome. Metrics contain only contract dimensions; `consumer`, `cache_hit` and errors are span attributes. Span duration covers acquisition, including validation/installation. Cache hit rates derived from spans are subject to trace sampling.
Counters count acquisition attempts after identity resolution. Metadata-list failures without a resolved skill ID cannot produce a per-skill download. Successful acquisition means a validated usable package, not just HTTP success. Python worker-thread downloads may finish after awaiting-task cancellation; count their actual acquisition result, never a cancelled load. Go context cancellation is an error outcome. Metrics contain only contract dimensions; `skill.consumer`, `skill.download.cache_hit` and errors are span attributes. Span duration covers acquisition, including validation/installation. Cache hit rates derived from spans are subject to trace sampling.

## Tasks, tests and acceptance

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,22 @@

## 需求与场景

- FR-1:已识别来源的远端技能包获取结束时记录一次 `gen_ai_skill_download`,覆盖验证成功的磁盘缓存恢复和失败。无效缓存后重新下载属于一次获取,`cache_hit=false`。
- FR-2:成功返回指令时记录一次 `gen_ai_skill_load`。同一 invocation 两次成功 `load_skill` 计两次,即使 ADK 只获取一次。未知技能、读取失败和取消不计成功加载。
- FR-1:已识别来源的远端技能包获取结束时记录一次 `gen_ai_skill_download_count`,覆盖验证成功的磁盘缓存恢复和失败。无效缓存后重新下载属于一次获取,`skill.download.cache_hit=false`。
- FR-2:成功返回指令时记录一次 `gen_ai_skill_load_count`。同一 invocation 两次成功 `load_skill` 计两次,即使 ADK 只获取一次。未知技能、读取失败和取消不计成功加载。
- FR-3:云端身份在转换/缓存后保留。Sandbox Python 区分内置技能、SkillHub slug、显式 AgentKit ID 和两类空间;Go 保留既有四类来源支持。来源不明的本地技能不猜测成内置技能。
- FR-4:使用既有进程级 OTEL provider;Sandbox 仅在未配置 provider 时安装周期 OTLP metric exporter。trace 与 metric 禁用开关独立。导出失败不影响对话;退出 flush 有时间上限。
- FR-5:新增遥测不记录包 URL、凭据、指令正文、路径或原始异常消息。既有内容遥测策略保持不变。
- FR-6:Python 与 Go 对已支持的来源使用相同标签和来源语义;预取只产生下载数据。保留既有安装包和会话链接。

当前数据结构以 2026-10-21 读取的技术文档 revision 2419 及随后用户要求的单位 `time` 调整为准:两个 counter 使用 `_count` 后缀和单位 `time`,身份新增 `skill_name`;结果仅放在 `skill.download.outcome`,不再作为 metric 标签。Span 命名为 `download_skill {skill_name}` / `load_skill {skill_name}`,并使用 `skill.consumer` 和 `skill.download.cache_hit`。具体名称来源与空值规则见组件契约。保留固定 Counter 实现;字段和 record 方法名无需 `gen_ai` 前缀。

## 设计与受影响消费者

Runtime:增加小型 `veadk/skills/telemetry.py` 记录模块,在既有 materializer 采集;将云端身份保存在 ADK frontmatter 的保留 metadata 中;在既有 VeADK 工具 tracing 回调中处理原生 ADK `LoadSkillTool` 的成功响应。不替换 toolset、不增加全局缓存。Runtime 加载采集沿用既有 VeADK tracing hook 的启用方式;hook 启用后,计数不依赖 Span 采样。

Sandbox:使用已经安装的 OTEL 包实现本地记录模块,确保已发布 SDK 无需依赖尚未发布的 VeADK 改动。在 `playground_skills/packages.py`、SkillHub 安装和既有 `ObservedSkillsTool` 采集。随安装包保存 SkillHub slug,避免替换失败时把旧内置技能标记成新的 SkillHub 技能。Go 包存储负责下载记录;目录元数据把身份传递给既有 tracked tool。内置技能不产生下载事件。在现有遥测初始化中配置 metric 导出,并复用退出 flush。

counter 从身份解析后的技能包获取开始计数。尚无技能 ID 的元数据列表失败无法产生按技能归属的下载。获取成功表示技能包通过验证且可用,不只是 HTTP 成功。Python 工作线程可能在等待任务取消后完成下载;按实际获取结果计数,但不计取消的加载。Go context 取消计入错误结果。metric 仅含契约规定的维度;`consumer`、`cache_hit` 和错误放在 Span。Span 耗时覆盖获取、验证和安装。通过 Span 计算的缓存命中率受 trace 采样影响。
counter 从身份解析后的技能包获取开始计数。尚无技能 ID 的元数据列表失败无法产生按技能归属的下载。获取成功表示技能包通过验证且可用,不只是 HTTP 成功。Python 工作线程可能在等待任务取消后完成下载;按实际获取结果计数,但不计取消的加载。Go context 取消计入错误结果。metric 仅含契约规定的维度;`skill.consumer`、`skill.download.cache_hit` 和错误放在 Span。Span 耗时覆盖获取、验证和安装。通过 Span 计算的缓存命中率受 trace 采样影响。

## 任务、测试与验收

Expand Down
26 changes: 14 additions & 12 deletions specs/skill-observability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,29 @@

[中文版](README.zh.md)

Component: `skill-observability`; status: implemented (local verification; full regression limitations in PRD); revised 2026-10-16.
Component: `skill-observability`; status: implemented (local verification; full regression limitations in PRD); revised 2026-10-21.
Change: [skill-observability](../../prd-spec/features/skill-observability/2026-10-16-skill-observability.md).

## Ownership and interfaces

VeADK owns Runtime package acquisition and ADK instruction-load instrumentation in `veadk/skills/` and `veadk/tracing/telemetry/telemetry.py`. `actb-mono/sandboxes/agentkit-skill` owns Python/Go Sandbox installation, tool adaptation and exporter lifecycle. The maintained wire schema below is shared; the sandbox is independently released and does not import an unpublished SDK recorder. Existing tools/signatures/results and metrics remain compatible. Discovery, resource/script execution, issue reporting and business success are not owned here.
VeADK owns Runtime package acquisition and ADK instruction-load instrumentation in `veadk/skills/` and `veadk/tracing/telemetry/telemetry.py`. `actb-mono/sandboxes/agentkit-skill` owns Python/Go Sandbox installation, tool adaptation and exporter lifecycle. The maintained wire schema below is shared; the sandbox is independently released and does not import an unpublished SDK recorder. Existing tools/signatures/results remain compatible. Skill metrics use the new names and unit below; consumers must update queries. Old metric names are not emitted in parallel. Discovery, resource/script execution, issue reporting and business success are not owned here.

## Contracts

- CON-1: `gen_ai_skill_download` is an OTEL counter, unit `count`; dimensions are exactly `skill_source`, `skill_id`, `skill_version`, `skill_space_id`, `outcome`. `outcome` is `success` or `error` (including cancellation). One acquisition completion, including cache restore, adds one. No event for mere discovery or local symlink creation.
- CON-2: `gen_ai_skill_load` is an OTEL counter, unit `count`; dimensions are exactly the four identity fields above. One successful instruction response adds one. ADK invocation-cache hits still count. Errors, cancellation, forwarding and resource/script access do not.
- CON-3: Download Span `skill_download` has `skill.source`, `skill.id`, `skill.version`, `skill.space_id`, `outcome`, `cache_hit`, `consumer`. Failures add `error.type` and a safe static `error.description`, with ERROR status. Successful load annotates the existing tool Span with the four dotted identity fields and `consumer`. Consumer is `runtime` or `skill_sandbox`. Span duration is total acquisition time. Never add consumer/cache/error text to metric labels.
- CON-1: `gen_ai_skill_download_count` is an OTEL counter, unit `time`; dimensions are exactly `skill_source`, `skill_id`, `skill_name`, `skill_version`, `skill_space_id`. One acquisition completion, including cache restore and failure, adds one. No event for mere discovery or local symlink creation; outcome is recorded only on the Span.
- CON-2: `gen_ai_skill_load_count` is an OTEL counter, unit `time`; dimensions are exactly the five identity fields above. One successful instruction response adds one. ADK invocation-cache hits still count. Errors, cancellation, forwarding and resource/script access do not.
- CON-3: Download Span `download_skill {skill_name}` has `skill.source`, `skill.id`, `skill.name`, `skill.version`, `skill.space_id`, `skill.consumer`, `skill.download.outcome`, `skill.download.cache_hit`. Outcome is `success` or `error` (including cancellation); failures add `error.type` and a safe static `error.description`, with ERROR status. Successful loading renames the existing tool Span to `load_skill {skill_name}` and adds the five dotted identity fields and `skill.consumer`, without creating a duplicate Span. `skill.consumer` is `runtime` or `skill_sandbox`. Span duration covers the actual operation. Consumer/cache/outcome/error text never becomes a metric label.
- CON-4: Identity follows the table. Missing version/space is the empty string, never a synthesized cache hash. Remote skills without a trustworthy ID and unclassified local skills are excluded. Registry-owned metadata overrides archive metadata for telemetry identity. SkillHub installation records the slug with the installed package, not just the current desired configuration.

| skill_source | skill_id | skill_version | skill_space_id |
| --- | --- | --- | --- |
| builtin_skill | name | empty | empty |
| skillhub_skill | slug | empty | empty |
| skillhub_skill_space | s-* | sv-* when supplied | sp-* |
| agentkit_skill | s-* | vN | empty |
| agentkit_skill_space | s-* | vN when supplied or present in the legacy package path | ss-* |
`skill_name` uses the name resolved by the current source: metadata for Runtime/skill spaces, the loaded name for local skills, and the resolved installation name for SkillHub installation. Explicit TOS acquisition first propagates the version API `Name`; when absent, successful acquisition or cache restoration reads the validated package name. Failure before a name is available uses the empty string, never an ID, slug or path as a substitute. Reading a name for telemetry must not change the acquisition result.

| skill_source | skill_id | skill_name | skill_version | skill_space_id |
| --- | --- | --- | --- | --- |
| builtin_skill | name | name | empty | empty |
| skillhub_skill | slug | resolved name or empty | empty | empty |
| skillhub_skill_space | s-* | resolved name or empty | sv-* when supplied | sp-* |
| agentkit_skill | s-* | resolved name or empty | vN | empty |
| agentkit_skill_space | s-* | resolved name or empty | vN when supplied or present in the legacy package path | ss-* |

Python supports all five rows. The pinned Sandbox Go SDK still rejects `sp-*`; the user approved retaining that limitation. Local SkillHub receipts are shared across Python/Go selection.

Expand Down
Loading
Loading