Repository navigation
Description
Problem
Pull request #836 merged the Go port of skill token measurement: agentrepo skill-tokens (internal/agentrepo/skilltokens.go:12) over internal/tokens/measure.go:64. It left activation to a follow-up. The skill-tokens task still runs bash tools/skill-tokens.sh (mise.toml:487), so the shell backend selection, manifest partitioning, file discovery, record assembly and budget reporting (tools/skill-tokens.sh:69, tools/skill-tokens.sh:110, tools/skill-tokens.sh:139, tools/skill-tokens.sh:166, tools/skill-tokens.sh:203, tools/skill-tokens.sh:224) are still what the repository runs, on origin/main at 44be762a34eec4a832e1ed608c77ebf188357719. The Go command has no caller.
The remaining work is to activate and verify that command through the task, with parity fixes only where a check finds a difference from the frozen shell.
This is the token-measurement split of #332. Part of #588.
Acceptance criteria
- AC1: Default and explicit skill selection, including missing-manifest skips, equals the frozen command's selection.
- AC2: Each skill selects the same counting backend, model and encoding under the existing defaults and environment precedence.
- AC3: Codex skill measurement remains local and never invokes the Anthropic counting backend.
AC4: Manifest, frontmatter, description, body and bundled-file counts equal the frozen command's for identical backend responses.Merged into AC6, whose record bytes contain every count.AC5: Bundled-file selection includes tracked and nonignored untracked text and excludes manifests, token records, ignored files and non-text files as the frozen command does.Merged into AC6, whose record rows list the bundled files.- AC6: Each skill's token-record bytes after an invocation equal the frozen command's for the same input and backend results. That covers manifest, frontmatter, description, body and bundled-file counts, and bundled-file selection: tracked and nonignored untracked text is included; manifests, token records, ignored files and non-text files are excluded.
- AC7: Failed counting, discovery, measurement or total calculation returns nonzero, including a producer that prints plausible data before failing.
- AC8: The body-budget threshold remains advisory, with the same warning/report transcript and a successful status when all measurements succeed.
AC9: Existing entry points end as thin shell wrappers that probe the Go subcommand without side effects and fall back to frozen shell only when that capability is absent.Dropped: the design no longer edits the shell file, and the dispatch moves into the task (AC13, AC14).AC10: A supported Go failure never invokes frozen shell or repeats a counting request.Dropped: the task never reaches frozen shell. The no-repeat half is covered by AC13.- AC11: Non-functional: release N-2 invocations, environment settings and record consumers remain compatible without raising the required binary floor.
- AC12: Non-functional: identical input text and backend responses produce the same records on supported Linux, macOS and Windows environments.
- AC13: The
skill-tokensmise task runs the checkout'sagentrepo skill-tokenssource (go run ./cmd/agentrepo skill-tokens) with the same arguments. No shell file is on its path, noagentrepobinary found onPATHis trusted, and a failure triggers no repeated counting request. - AC14:
tools/skill-tokens.shand its tests are byte-identical toorigin/main, and the change contains no changed.shor.bashfile. - AC15: The shell file's retirement is tracked by chore(skill-tokens): retire the frozen shell fallback #951, which is amended so it deletes
tools/skill-tokens.shand its frozen-comparison fixtures after this pull request merges, with no capability-probe wrapper and no N-2 window. - AC16: Non-functional: the pull request declares its consumer impact (
Compatibility: noneunder Risk, with the reason that no consumer calls the shell file) and passesmise run check,mise run cover-go-checkandmise run cover-py-check.
Scope and non-goals
Preserve backend classification for packaged/deployed Codex paths versus other skill paths; model/encoding defaults and overrides; one measured Claude request envelope per invocation; manifest partitioning and folded-description extraction; byte sizes; sorted bundled rows; relative recorded paths; backend-specific encoding metadata; deterministic JSON without timestamps; budget-at-or-above comparisons; skip text, summaries and final guidance. Source evidence: tools/skill-tokens.sh:44, tools/skill-tokens.sh:114, tools/skill-tokens.sh:147, tools/skill-tokens.sh:166, tools/skill-tokens.sh:203, tools/skill-tokens.sh:240 and tools/skill-tokens.sh:266.
The Go engine and its unit tests exist from #836. This change reuses them and adds only the task change, parity and failure fixtures that compare against the frozen shell, and focused corrections for any difference those fixtures find. Port or cover every behavior exercised by tests/scripts/skill-tokens-backends.bats:29 through tests/scripts/skill-tokens-backends.bats:86. Preserve the failure boundary before writing a skill record when that skill's counts, file discovery or byte measurements fail (tests/scripts/skill-tokens-backends.bats:39, tests/scripts/skill-tokens-backends.bats:52, tests/scripts/skill-tokens-backends.bats:65). A later skill's failure does not roll back earlier completed records; the final-total regression asserts failure status, not whole-invocation rollback.
Non-goals: changing tokenization, models, encodings, authentication or provider contracts; adding another counting service; imposing a hard body limit; generalizing frontmatter syntax; making the entire invocation transactional; changing skill generation and deployment; or deleting the shell file (#951).
Approach
Change the skill-tokens task body at mise.toml:487 to go run ./cmd/agentrepo skill-tokens, the way other tasks start Go from the checkout (mise.toml:290 and mise.toml:604). Running the checkout's source means the task uses the reviewed code, never an older agentrepo on PATH, and raises no installed-binary floor (AC11). Don't route through the generic run task: it executes Bash and a shell helper (mise.toml:196, mise.toml:200), which would violate AC13. Leave tools/skill-tokens.sh and its tests untouched: the shell-allowlist gate checks every command of any changed .sh file (cmd/shell-allowlist/changed.go:30), and moving the body into that file failed it.
No consumer calls tools/skill-tokens.sh directly: the only caller is mise.toml:487, and the remaining occurrences are tests, the generated_by provenance string, documentation and comments. tools/AGENTS.md marks tools/ internal. This was checked across the tracked tree only, not external clones. The shell file therefore needs no capability-probe wrapper or N-2 window, and #951 deletes it after this merges.
Start with failed-count and failed-discovery checks, then compare the Go command with frozen shell using deterministic backend responses. Preserve the generation order (mise.toml:466, mise.toml:477). Leave code structure to the pull request.
Superseded pull requests
The new pull request replaces #937 and #987. Both edit the shell file into a wrapper with a legacy fallback, so neither implements this design. The new pull request's description says so, begins a fresh review count, and does not inherit approvals or findings from either. After it opens, #937 and #987 are closed as superseded, each with a comment naming the new pull request. Neither keeps a Closes #810 reference.
Dependencies
Native blocked-by edges on this issue: none to set. The live blockedBy list is empty, and that is correct.
- feat(tokens): add deterministic skill measurement #836 delivered the Go command and is merged.
- refactor(release): remove obsolete dispatch orchestration #542 is dropped as a dependency. It removes obsolete release orchestration, and this change touches no release interface. The prose line and the delivery chain through Ship the Go binaries as release assets installed through mise's github backend #333, feat(distribution): migrate consumer tool pins to release assets #544, feat(release): provide configurable shared release code and tasks #382, feat(release): distribute the reusable workflow and release skills #545, build(release): adopt the shared release path in the first consumer #546 and Release on every merge that has no release dependencies #359 are removed with it.
- Blocker to set on chore(skill-tokens): retire the frozen shell fallback #951: refactor(tools): move skill token measurement to Go #810, as a native blocked-by edge, and its text is amended per AC15. Its current "blocked by the N-2 window" prose and wrapper requirement no longer apply.
#332 and #588 track this split. No unpublished sibling draft is needed to begin.
Size
M. One pull request for task activation, parity and failure fixtures, and any focused corrections. It reuses the #836 implementation and stays under 800 code lines.
Verify plan
- AC7: first reproduce each failure case at
tests/scripts/skill-tokens-backends.bats:39,tests/scripts/skill-tokens-backends.bats:52,tests/scripts/skill-tokens-backends.bats:65,tests/scripts/skill-tokens-backends.bats:74andtests/scripts/skill-tokens-backends.bats:86. Cover silent failure, plausible output followed by failure and invalid counts; assert status and the existing per-record preservation boundary, without imposing invocation-wide rollback. - AC1: compare default discovery, explicit directories, trailing separators and absent manifests with frozen shell.
- AC2, AC3: backend/model/encoding fixtures cover default values, every override and precedence combination, relative/absolute input spellings, packaged and deployed Codex paths and non-Codex paths. An Anthropic invocation trap remains untouched for Codex, extending
tests/scripts/skill-tokens-backends.bats:29. - AC6 (counts): deterministic responses cover manifest sections, folded descriptions, missing frontmatter, empty and UTF-8 text, multiple Claude skills and envelope subtraction; compare each count and the single baseline request.
- AC6 (selection): tracked, untracked, ignored, empty, binary and text fixtures compare the sorted bundled set, skip diagnostics and byte sizes.
- AC6: compare exact record bytes, keys, optional encoding metadata, relative paths and totals; repeat unchanged input and assert no content drift. A two-skill failure fixture confirms the first completed record remains.
- AC8: counts just below, equal to and above the budget compare warnings, per-skill summaries, final guidance and successful status.
- AC13: read the task definition and run it against a stub backend with an older stand-in
agentrepofirst onPATH; assert the checkout's source runs, the arguments arrive, no.shfile runs, and a failure after partial output fails the task with one recorded backend request. - AC14:
git diff --stat origin/mainlists no.shor.bashpath, andmise run lint-shell-allowlistpasses. - AC15: read chore(skill-tokens): retire the frozen shell fallback #951 after amendment: its body has no wrapper or probe requirement, names the deleted paths, and the live
blockedByof chore(skill-tokens): retire the frozen shell fallback #951 lists refactor(tools): move skill token measurement to Go #810. - AC16: the description carries the compatibility line under Risk;
mise run check,mise run cover-go-checkandmise run cover-py-checkpass on the reviewed head. - AC11: run the changed task and token-record readers against N-2 record fixtures.
- AC12: run the Go corpus on Linux, macOS and Windows. All verification uses local deterministic backend stubs, without real API counting or authentication changes. Run focused tests, generation/deployment consistency checks,
mise run lintandmise run teston the reviewed head. - Definition of done: D1 each AC is verified at the merged head, with the commands and results cited in the pull request's Verification. D2 an independent review approves the exact merged head, and every finding has a recorded disposition. D3 required checks pass with no gate loosened. D4 the leak scans the repository specifies are clean, and every post made for this work (comments, descriptions) passes the per-post leak review before it is posted. D5 every commit is signed. D6 the documentation matches, including the
tools/skill-tokens.shrow indocs/script-inventory.md, or the description saysNo doc impactwith the reason. D7 every commit carries anAssisted-bytrailer and the pull request carries theprovenance:agent-assistedlabel. D8 the change stays within the pull request size limits. D9 follow-up work, including deferred review findings, is filed as issues, each with a Problem section and numbered acceptance criteria and linking this issue; the pull request's Related section lists them, including chore(skill-tokens): retire the frozen shell fallback #951 as amended, and any code that defers work links its issue. D10 the merge closes this issue through a closing reference, the project board shows the state Done, and a finalStatus:comment names the merge commit.
Risks
An incorrect envelope or backend selection corrupts committed token records; nondeterministic metadata creates persistent drift. A task left on the old shell path would leave the port unused. go run compiles on each invocation, which adds startup time to regen-generated. Exact record comparisons and invocation-count assertions cover the first boundaries. Undo: revert the task change.
Operator authority
None.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Work type
Projects
- StatusShow more project fieldsDone
Activity
tbhb-agent commentedon Oct 26, 2026
Refinement: READY — skill token measurement port to Go has clear, testable requirements and verified source boundaries
Model: gemini-4-argon-high
Checked
tbhb-dev/devtools#810(0 comments) and referenced issues#332,#588,#542,#333,#544,#382,#545,#546, and#359.origin/main(a0347a715fab580ece45edfcb3d13509620459a8):tools/skill-tokens.sh:69(count_claudebackend invocation;count_codexfollows attools/skill-tokens.sh:87).tools/skill-tokens.sh:110(prepare_backendsingle-request Claude envelope baseline measurement attools/skill-tokens.sh:110-120).tools/skill-tokens.sh:139(skill directory loopfor dir in "${dirs[@]}", trailing slash normalization, missingSKILL.mdskip tostderr, and path-based backend classification attools/skill-tokens.sh:139-158).tools/skill-tokens.sh:166(embedded Python manifest partitioning intofrontmatter,body, and foldeddescriptionattools/skill-tokens.sh:166-177).tools/skill-tokens.sh:203(git -c core.quotePath=false ls-files --cached --others --exclude-standard -- "$dir" | sortdiscovery,SKILL.md/tokens.jsonexclusion,grep -Iq .non-text skip, per-file net count, and byte measurement attools/skill-tokens.sh:203-219).tools/skill-tokens.sh:224(embedded Pythontokens.jsonassembly with relative paths, sorted keys, 2-space indentation, optionaltokenizer, and no timestamp attools/skill-tokens.sh:224-264, followed by advisory body-budget warning and summary output attools/skill-tokens.sh:266-284).AC1–AC12):AC1–AC10are observable, testable, single-outcome behavioral requirements free of prescribed internal Go package layout or implementation details.AC11andAC12state the non-functional requirements for release N-2 compatibility (docs/compatibility.md:1-5) and cross-platform record determinism across Linux, macOS, and Windows.origin/main:tools/skill-tokens.sh:44(BODY_BUDGETdefault and model/encoding environment variables attools/skill-tokens.sh:44-49),tools/skill-tokens.sh:114(baseline measurement),tools/skill-tokens.sh:147(Codex vs. Claude path matching),tools/skill-tokens.sh:166(manifest split),tools/skill-tokens.sh:203(bundled file discovery),tools/skill-tokens.sh:240(timestamp-free JSON record structure), andtools/skill-tokens.sh:266(-gebudget check and summary formatting).origin/main:tests/scripts/skill-tokens-backends.bats:29(Codex local counting without touching Anthropic trap),tests/scripts/skill-tokens-backends.bats:39(bundled tokenizer failure writes no record),tests/scripts/skill-tokens-backends.bats:52(git ls-filesdiscovery failure writes no record),tests/scripts/skill-tokens-backends.bats:65(wcbyte measurement failure writes no record),tests/scripts/skill-tokens-backends.bats:74(Claude count failure propagation throughnet), andtests/scripts/skill-tokens-backends.bats:86(failed bundled total calculation returns nonzero after record write).mise.toml:466([tasks.regen-generated], tested intests/tasks/regen-generated.bats:35-42) andmise.toml:477([tasks.skill-tokens]).#542,#333,#544,#382,#545,#546,#359,#332, and#588exist and are open; prerequisite (#542following the#333–#546and#359chain) and tracking epics (#332,#588) are accurately classified.Mfits a single pull request covering the Go measurement port, record compatibility, failure coverage, and thin entry point wrapper.AC1–AC12) andD1–D10to concrete, offline-capable verification checks using deterministic backend stubs.None.Verified against.agents/agent-skills.conf:4-9(golden_paths,constitutional_paths,gate_config_paths,owner_paths) and.github/CODEOWNERS:1-13: portingtools/skill-tokens.shto Go while keepingmise.toml:466andmise.toml:477unchanged does not touch protected owner/golden paths or loosen any gate.Gaps
None.
Split
None needed; Size
Mfits in one pull request.Definition of done
The definition of done for #810 (Feature). Tick each item and give its evidence. All ticked boxes below indicate planning coverage in the issue's acceptance criteria and verify plan, not completed implementation work.
## Verify planspecifies concrete checks forAC1throughAC12and explicitly requiresD1verification at the merged head with commands and results cited in the pull request'sVerificationsection.## Verify plan(Definition of done: D2) requires independent review approval of the exact merged head and a recorded disposition for every finding.## Verify planrequiresmise run lint,mise run test, and generation/deployment consistency checks on the reviewed head, andDefinition of done: D3requires required checks to pass with no gate loosened.check-leaks.shwith the repository's private-strings list, and the leak review of each post.## Verify plan(Definition of done: D4) requires the repository's leak scans to be clean and every post (comments, descriptions) to pass per-post leak review before posting.## Verify plan(Definition of done: D5) requires every commit to be signed.docs/, CHANGELOG or help text), or the Verification section says "No doc impact" and why.## Verify plan(Definition of done: D6) requires documentation to match or the pull request description to stateNo doc impactwith the reason.provenance:agent-assistedlabel.## Verify plan(Definition of done: D7) requires every commit to carry anAssisted-bytrailer and the pull request to carry theprovenance:agent-assistedlabel.## Sizescopes the work to oneMpull request and## Verify plan(Definition of done: D8) requires staying within pull request size limits.## Scope and non-goalsnotes fallback removal follows separately underD9, and## Verify plan(Definition of done: D9) requires filing follow-up work and deferred review findings as issues with aProblemsection and numbered acceptance criteria linked underRelated.Status:comment names the merge commit.## Verify plan(Definition of done: D10) requires closing the issue via a closing reference on merge, the project board showingDone, and a finalStatus:comment naming the merge commit.Round: 1 of 3
feat(tools): port skill token measurement to Go
fix(tokens): preserve frozen discovery and summary output
feat(tokens): add deterministic skill measurement
feat(tokens): add deterministic skill measurement
refactor(tokens): use Go measurement from the shell entry point
tbhb-dev-reviews commentedon Oct 31, 2026
Refinement: NOT READY — option 2 is viable, but dependencies and remaining scope need reconciliation
Checked
44be762a34eec4a832e1ed608c77ebf188357719.mise.toml:487; direct Go task precedents aremise.toml:290andmise.toml:604. The allowlist selects changed shell files atcmd/shell-allowlist/changed.go:30. That filter does not authorize shell logic inside TOML. The generic run task executes Bash and a shell helper atmise.toml:196andmise.toml:200; routing through it would violate AC13. Direct Go dispatch avoids that conflict.mise.toml:487; other occurrences are tests, provenance, documentation or comments.tools/AGENTS.md:3identifies these tools as internal. This supports the body's answer within this repository, without claiming knowledge of arbitrary external clones.tools/skill-tokens.sh:69,tools/skill-tokens.sh:110,tools/skill-tokens.sh:139,tools/skill-tokens.sh:166,tools/skill-tokens.sh:203,tools/skill-tokens.sh:224andtools/skill-tokens.sh:266, and the failure cases attests/scripts/skill-tokens-backends.bats:39,tests/scripts/skill-tokens-backends.bats:52,tests/scripts/skill-tokens-backends.bats:65,tests/scripts/skill-tokens-backends.bats:74andtests/scripts/skill-tokens-backends.bats:86.Gaps
internal/tokens/measure.go:64andinternal/agentrepo/skilltokens.go:12. feat(tokens): add deterministic skill measurement #836 explicitly left caller activation to a follow-up. Reframe Problem/Approach/Size around activating and verifying that command, with focused parity fixes only as needed. State how the task uses the checkout's command without raising the installed binary floor (AC11), executing shell files (AC13) or trusting an older PATH binary. Direct Go task precedents provide a viable route. Require the replacement statement and fresh review count in the new PR text, and explicitly disposition the two open superseded PRs.mise run check,mise run cover-go-checkandmise run cover-py-check; lint/test alone do not cover them. Include the task's consumer-impact declaration or the reason for no consumer impact.Split
No further implementation split is needed. #836 already delivered the Go engine. Task activation, argument/failure integration checks and bounded parity corrections fit one PR below 800 code lines. Shell deletion remains a separately tracked follow-up. Retain #588's freeze and do not edit shell to implement dispatch.
Definition of done
Checked boxes indicate explicit planning coverage in the ACs/Verify plan, not completed implementation. The gaps above still prevent readiness.
Round: 2 of 3
tbhb-dev-reviews commentedon Oct 31, 2026
Refinement: READY — activate the delivered Go command through the task with frozen-shell parity checks
Checked
mise.toml:487. The delivered command atinternal/agentrepo/skilltokens.go:12delegates tointernal/tokens/measure.go:64; feat(tokens): add deterministic skill measurement #836 explicitly left activation undone. Neither open wrapper PR implements the revised direct-task design. The remaining work is not already delivered.tools/skill-tokens.sh:69,tools/skill-tokens.sh:110,tools/skill-tokens.sh:139,tools/skill-tokens.sh:166,tools/skill-tokens.sh:203,tools/skill-tokens.sh:224andtools/skill-tokens.sh:266, alongside the Go implementation. Failure witnesses remain attests/scripts/skill-tokens-backends.bats:39,tests/scripts/skill-tokens-backends.bats:52,tests/scripts/skill-tokens-backends.bats:65,tests/scripts/skill-tokens-backends.bats:74andtests/scripts/skill-tokens-backends.bats:86.mise.toml:290andmise.toml:604. It excludes the generic shell-backed task atmise.toml:196andmise.toml:200. No new shell dispatch, probe or fallback is proposed. The shell and its existing tests remain frozen; retirement is separate under chore(skill-tokens): retire the frozen shell fallback #951.tools/AGENTS.md:3: other references are tests, provenance, documentation or comments. This supports the compatibility rationale without claiming knowledge of external clones. The N-2 requirement is preserved without raising the installed binary floor (docs/compatibility.md:3)..agents/agent-skills.conf:8), so normal policy checks still apply, but this issue proposes no gate loosening or protected-path policy edit.Gaps
None blocking. Round 2's dependency, remaining-scope and cleanup/completion gaps are addressed in the revised body. The #951 amendment/native edge and superseded-PR dispositions remain implementation completion checks, not claims of actions already performed.
Split
No further split needed. #836 already delivered the engine and command. Task activation, deterministic parity/failure fixtures and focused corrections fit one M pull request under 800 code lines. Shell deletion remains #951; no shell logic is added here.
Definition of done
Ticks indicate independently checked planning coverage in the current ACs and Verify plan, not completed implementation.
Repository additions are also planned: AC16 requires check and both coverage floors; the Approach preserves generation/token/deployment order; AC16 supplies the compatibility declaration and no-consumer-call rationale. Carry the consumer-impact explanation into PR Verification as well as Risk.
Round: 3 of 3
feat(tools): run skill-tokens from the Go command
feat(tools): run skill-tokens from the Go command