Skip to content

refactor(leaks): port working-tree and explicit-file scans to Go #815

Description

@tbhb-agent

Problem

Part of #588. This is the working-tree and explicit-file slice of #329.

At origin/main (a0347a715fab580ece45edfcb3d13509620459a8), private-list parsing and matching remain shell (.devtools/scripts/check-leaks.sh:97, .devtools/scripts/check-leaks.sh:134), as do working-tree selection (.devtools/scripts/check-leaks.sh:289) and skip policy (.devtools/scripts/leak-skip.sh:4). Move this complete user-visible scan boundary to tested Go without changing which values or files it checks.

Acceptance criteria

  1. AC1: Every exported *_PRIVATE_STRINGS list retains its current matching results, entry labels and warnings, including comma separation, trimming, case handling, empty entries and entries shorter than four characters.
  2. AC2: Built-in and repository pattern checks retain their current matches and rule labels, including address boundaries, documentation-domain exceptions and accepted expression semantics.
  3. AC3: Path allowances suppress patterns only; private literals remain detectable, and the built-in exception for the shipped leak test remains limited to that exact path.
  4. AC4: A no-argument scan checks tracked and non-ignored untracked working-tree files from the repository root, including binary bytes, link targets as text and paths, with the same selection and results as the frozen shell.
  5. AC5: Working-tree skip expressions exclude matching paths before their content is opened, and invalid or unanchored expressions fail with status 2 without disclosing the expression.
  6. AC6: Explicit file arguments check exactly the supplied files with the frozen shell's current-directory semantics, including invocation outside a repository; missing or unreadable inputs never pass.
  7. AC7: Findings retain their stdout location-and-rule format and stderr summary; sensitive locations retain opaque identifiers, and matched values, sensitive arguments and malformed configuration never appear on either stream.
  8. AC8: The result remains 0 for complete clean scans, 1 for findings and 2 for misuse or incomplete reads.
  9. AC9: Non-functional: existing consumer entry points end as thin shell wrappers that probe Go subcommand support without writes or consuming input; absent binary or capability uses the frozen shell unchanged, while supported execution runs once and preserves its streams and status without fallback after failure.
  10. AC10: Non-functional: the migrated behavior works on Linux, macOS and Windows without new consumer configuration or a higher minimum binary version.
  11. AC11: Non-functional: no temporary matched content remains after success or failure.
  12. AC12: When no private literals are loaded, the current notice is retained, including its existing local-file hint and GitHub Actions annotation.

Scope and non-goals

In scope: matching, configuration and diagnostics used by these two modes, their task entry points, corresponding Go and wrapper coverage, and consumer help and adoption documentation for their unchanged source, limitations, fallback and rollback. Evidence includes .devtools/scripts/check-leaks.sh:87, .devtools/scripts/check-leaks.sh:99, .devtools/scripts/check-leaks.sh:189, .devtools/scripts/check-leaks.sh:249, .devtools/scripts/check-leaks.sh:808, .devtools/scripts/check-leaks.sh:828 and .devtools/tasks/leaks.toml:2.

Behavior coverage: .devtools/scripts/check-leaks.bats:70 through the configuration and file cases at .devtools/scripts/check-leaks.bats:466; path, disclosure, task-argument and read-failure cases at .devtools/scripts/check-leaks.bats:516, .devtools/scripts/check-leaks.bats:554, .devtools/scripts/check-leaks.bats:606, .devtools/scripts/check-leaks.bats:644, .devtools/scripts/check-leaks.bats:699, .devtools/scripts/check-leaks.bats:709, .devtools/scripts/check-leaks.bats:1429 and .devtools/scripts/check-leaks.bats:1464; skip and basename cases at .devtools/scripts/check-leaks.bats:1508.

Overlap: AC1 and AC12 absorb the exported-list and no-list behavior retained by #501. Its new source discovery, precedence, configured-source refusal and source notices remain in #501, including its provisioning documentation and operator provisioning. This port does not claim to fix or close #501. No #551 scan-selection requirement is absorbed: AC7 preserves private-scanner disclosure behavior only; #551's gitleaks modes, callers and coverage decision stay there.

Non-goals: staged, range, publication and push producers, hook installation and gitleaks orchestration; broadening detection to transformed or split values, comma-containing entries or literals shorter than four characters; recursive submodule scanning. These are the existing detection boundaries at .devtools/scripts/check-leaks.sh:53 and .devtools/scripts/check-leaks.sh:289. Other scan modes retain their frozen behavior. Removing the compatibility fallback is deferred until the N-2 window permits it, with a dedicated follow-up under D9.

Approach

Move the scoped behavior to tested Go, with failing cases first. The mechanism is the implementer's choice within these outcomes. Preserve the shell as a frozen compatibility path; add no new scanning logic to it.

Dependencies

Blocked by #753 before activating replacement enforcement; the implementation PR must cite its merged protection evidence. #329 tracks the containing migration. #501 and #551 retain the exclusions stated above. #661 governs N-2 compatibility; #333 supplies binary distribution without making it a new consumer prerequisite.

Size

M. One pull request for two scan modes and their shared matching contract.

Verify plan

Start with failing Go tests for an omitted file, a disclosed sensitive location and an incomplete read; then port every scoped Bats behavior before reducing its shell coverage.

  • AC1: table cases for multiple exported lists in discovery order, whitespace, empty and short entries, mixed ASCII case and non-ASCII bytes; compare results, entry numbering, warnings and streams with the pinned shell.
  • AC2, AC3: positive and boundary-negative cases for every built-in category, repository expressions and invalid expressions; test the exact built-in exception, allowed versus ordinary paths and literals under both. Compare frozen results, including expression case semantics.
  • AC4, AC5: tracked, untracked, ignored and deleted files; binary and empty files; symlinks without following their targets; paths with spaces, tabs, newlines and backslashes; a nested invocation; and skipped unreadable content. Invalid skip fixtures must return 2 without expression bytes.
  • AC6: several explicit files, relative and absolute file arguments, a spaced filename, an unrelated leaky file that must remain unchecked, a missing or unreadable file, a directory, symlinks to readable and missing targets, and execution outside Git; compare selection, status and streams. Explicit symlink arguments retain their existing target-content behavior, distinct from tree mode's link-text scan. Confirm skip configuration is not consulted for explicit files.
  • AC7, AC8: compare stdout and stderr separately for clean, finding, misuse and injected read/configuration/discovery failures, including partial producer output. Search both streams for every synthetic private value and escaped form.
  • AC12: exercise absent, empty and short-only lists, with and without the local settings file and Actions annotation environment; compare each notice and stream with the frozen shell.
  • AC9: actual task and script entry points with missing binary, older binary lacking capability, supported success, findings, and output-then-failure. Assert probe side effects are absent, one execution, exact argument boundaries and no fallback after supported execution. An operational probe failure must refuse rather than claim absent capability. Unmigrated modes must retain their frozen transcript.
  • AC10: native Go tests on Linux, macOS and Windows; shell comparisons where Bash is available; unchanged N-2 consumer fixtures. Record before/after process timings over the same fixed 100-file fixture, with 20 warm runs and build time excluded; investigate any regression before acceptance.
  • AC11: inspect the fixture temporary area after clean scans, findings and each injected failure; no matched content may remain. As D6 evidence, compare help/adoption text with the actual entry points and verify the documented rollback in a disposable fixture.
  • Run the branch binary through mise run test-leaks, then mise run check, fresh Go and Python coverage and mise run cover-go-check and mise run cover-py-check. Verify build(agents): protect replacement Go enforcement paths #753's protection applies to the replacement enforcement and command registration. Any changed skill sources require ordered generation, token measurement, deployment and mise run check-generated-skills.
  • Definition of done: D1 each AC is verified at the merged head, with the commands and results cited in the pull request's Verification. D2 an independent review approves the exact merged head, and every finding has a recorded disposition. D3 required checks pass with no gate loosened. D4 the leak scans the repository specifies are clean, and every post made for this work (comments, descriptions) passes the per-post leak review before it is posted. D5 every commit is signed. D6 the documentation matches, or the description says No doc impact with the reason. D7 every commit carries an Assisted-by trailer and the pull request carries the provenance:agent-assisted label. D8 the change stays within the pull request size limits. D9 follow-up work, including deferred review findings, is filed as issues, each with a Problem section and numbered acceptance criteria and linking this issue; the pull request's Related section lists them, and any code that defers work links its issue. D10 the merge closes this issue through a closing reference, the project board shows the state Done, and a final Status: comment names the merge commit.

Risks

A changed expression dialect or path allowance can weaken scanning. Differential cases must preserve accepted configuration, including quirks. The fallback deliberately retains #501's present limitation for older binaries. Undo: revert this PR to restore the original entry points and frozen behavior.

Operator authority

standing rule names the operator: the replacement enforcement must have #753's protection before activation; satisfy its exact-head owner-review requirements without changing approval authority.

Activity

  1. added
    change:featPR adds or changes a feature (Conventional Commit type feat)
    on Oct 26, 2026
  2. tbhb-agent commented on Oct 26, 2026

    @tbhb-agent
    ContributorAuthor

    Refinement: READY — working-tree and explicit-file leak-scan Go port is well-scoped, testable, and properly bounded
    Model: gemini-4-argon-high

    Checked

    • Verified in an independent clone that origin identifies tbhb-dev/devtools and origin/main resolves to a0347a715fab580ece45edfcb3d13509620459a8, matching the commit cited in the Problem section of #815.
    • Read the live #815 issue body and comments (0 comments), as well as the issues referenced by #815 and its parent context (#277, #329, #333, #501, #551, #573, #588, #661, #753).
    • Verified every path:line reference in #815 against origin/main:
      • .devtools/scripts/check-leaks.sh:53 (existing detection boundaries: transformed/split values, literals under four characters, comma-containing entries)
      • .devtools/scripts/check-leaks.sh:87 (on_exit cleanup of $tmp and incomplete-read exit-2 reporting)
      • .devtools/scripts/check-leaks.sh:97 and .devtools/scripts/check-leaks.sh:99 (loaded=0, add_needle, and exported *_PRIVATE_STRINGS parsing at lines 97–127)
      • .devtools/scripts/check-leaks.sh:134 and .devtools/scripts/check-leaks.sh:189 (scan() and check() matching private literals, built-in patterns, .leak-patterns, .leak-allow, and opaque() path sanitization at lines 134–247)
      • .devtools/scripts/check-leaks.sh:249 (esc(), has_nul(), emit_body(), emit_path(), emit_file())
      • .devtools/scripts/check-leaks.sh:289 (emit_tree() working-tree selection via git -c diff.ignoreSubmodules=none ls-files -z --cached --others --exclude-standard and symlink target handling)
      • .devtools/scripts/check-leaks.sh:808 and .devtools/scripts/check-leaks.sh:828 (.leak-allow and .leak-patterns loading, no-private-values notice and GITHUB_ACTIONS annotation at lines 814–826, ERE validation at lines 828–840, and tree/files dispatch at lines 842–860)
      • .devtools/scripts/leak-skip.sh:4 (load_leak_skip() anchor and grep -E validation of .leak-skip)
      • .devtools/tasks/leaks.toml:2 (["devtools:lint-leaks"] task entry point)
      • .devtools/scripts/check-leaks.bats:70 through .devtools/scripts/check-leaks.bats:466, plus .devtools/scripts/check-leaks.bats:516, .devtools/scripts/check-leaks.bats:554, .devtools/scripts/check-leaks.bats:606, .devtools/scripts/check-leaks.bats:644, .devtools/scripts/check-leaks.bats:699, .devtools/scripts/check-leaks.bats:709, .devtools/scripts/check-leaks.bats:1429, .devtools/scripts/check-leaks.bats:1464, and .devtools/scripts/check-leaks.bats:1508 (all match the stated Bats test blocks).
    • Checked Acceptance criteria AC1–AC12: each criterion is observable, testable, states a single outcome, includes non-functional requirements where applicable (AC9 wrapper probing/fallback, AC10 cross-platform/N-2 compatibility, AC11 temporary-content cleanup), and avoids prescribing internal package layout, file structure, or library choices.
    • Checked #501 and #551/#573 non-goal boundaries:
      • #501 (open, refinement:ready): #815 preserves only the existing exported *_PRIVATE_STRINGS parsing and no-list notice (AC1, AC12), while explicitly leaving #501's worktree source discovery (mise.local.toml via git common dir and private_strings_ref), precedence, configured-source refusal, source notices, and operator provisioning in #501.
      • #551 (open, refinement:ready) and #573 (open PR): #815 explicitly excludes gitleaks orchestration and #551's local/CI mode selection, callers, and coverage reduction decision; AC7 covers only check-leaks.sh private-scanner disclosure behavior.
    • Checked tbhb-dev/devtools#277: #277 is open on GitHub (currently without refinement:ready following its Round 1 NOT READY comment) and concerns .devtools/scripts/install-pre-push.sh shared-worktree hook installation (#329). Hook installation is explicitly a non-goal of #815, which touches only the working-tree and explicit-file modes of .devtools/scripts/check-leaks.sh and .devtools/scripts/leak-skip.sh, so #815 is independent of #277.
    • Checked Dependencies (#753, #329, #588, #501, #551, #661, #333): all named issues exist, states and roles match live GitHub data (#753, #333, #501, #551 are open with refinement:ready; #329 and #588 are open parent issues; #661 is closed/completed), and #753 is properly classified as blocking activation of replacement enforcement.
    • Checked Operator authority against #753 and .agents/agent-skills.conf:4,8,9: the Operator authority section accurately states that replacement enforcement must have #753's protection before activation and must satisfy its exact-head owner-review requirements without changing approval authority.
    • Checked Verify plan, Risks, and Size: every AC ID (AC1–AC12) has a concrete verification check that proves the outcome, Risks names the regression hazards and a clear rollback/undo path, and Size (M, one PR) fits the two scan modes and their shared matching contract.

    Gaps

    None

    Split

    No split is needed. Porting the shared literal/pattern matcher together with the working-tree (tree) and explicit-file (files) scan modes, while leaving staged, range, publication, push, and hook-installation slices separate under #329, forms a single coherent M-sized pull request.

    Definition of done

    The definition of done for #815 (Feature). Tick each item and give its evidence.

    A ticked box below indicates planning coverage in the issue's acceptance criteria and verification plan, not completed implementation work.

    • D1 The author verified each AC on the merged head, and the Verification section of the pull request cites each AC ID with its evidence: the command and its result, or a link.
      Evidence: Verify plan maps AC1–AC12 to explicit test and differential checks and states under Definition of done that "D1 each AC is verified at the merged head, with the commands and results cited in the pull request's Verification."
    • D2 An independent reviewer approved the exact merged head, and the author resolved each finding or filed it as a follow-up.
      Evidence: Verify plan states "D2 an independent review approves the exact merged head, and every finding has a recorded disposition."
    • D3 The required checks pass on that head, and any loosened gate has the owner's approval of that exact head.
      Evidence: Verify plan requires running mise run test-leaks, mise run check, mise run cover-go-check, mise run cover-py-check, and mise run check-generated-skills (when skill sources change), verifying #753's protection applies, and states "D3 required checks pass with no gate loosened"; Operator authority requires satisfying exact-head owner-review requirements.
    • D4 The leak scans come back clean: gitleaks, devtools' check-leaks.sh with the repository's private-strings list, and the leak review of each post.
      Evidence: Verify plan requires mise run test-leaks, mise run check, and states "D4 the leak scans the repository specifies are clean, and every post made for this work (comments, descriptions) passes the per-post leak review before it is posted."
    • D5 The author signed each commit, the squash included.
      Evidence: Verify plan states "D5 every commit is signed."
    • D6 The docs match the change (README, docs/, CHANGELOG or help text), or the Verification section says "No doc impact" and why.
      Evidence: Scope includes "consumer help and adoption documentation for their unchanged source, limitations, fallback and rollback," and Verify plan requires comparing help/adoption text with actual entry points, verifying documented rollback in a disposable fixture as D6 evidence, and states "D6 the documentation matches, or the description says No doc impact with the reason."
    • D7 The commits and the squash have Assisted-by trailers, and the pull request has the provenance:agent-assisted label.
      Evidence: Verify plan states "D7 every commit carries an Assisted-by trailer and the pull request carries the provenance:agent-assisted label."
    • D8 The pull request stays within the size limits, or the owner approved its exact head over the hard limit.
      Evidence: Size is scoped to M (one pull request for two scan modes and their shared matching contract), and Verify plan states "D8 the change stays within the pull request size limits."
    • D9 The author files follow-ups and deferred findings as issues with at least a Problem and numbered AC and lists them under the Related section of the pull request. Any work left in the code links its issue.
      Evidence: Scope and non-goals states "Removing the compatibility fallback is deferred until the N-2 window permits it, with a dedicated follow-up under D9," and Verify plan states "D9 follow-up work, including deferred review findings, is filed as issues, each with a Problem section and numbered acceptance criteria and linking this issue; the pull request's Related section lists them, and any code that defers work links its issue."
    • D10 The merge closes the issue through a closing reference and the board shows Done. The final Status: comment names the merge commit.
      Evidence: Verify plan states "D10 the merge closes this issue through a closing reference, the project board shows the state Done, and a final Status: comment names the merge commit."

    Round: 1 of 3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    change:featPR adds or changes a feature (Conventional Commit type feat)refinement:readyPassed the Definition of Ready; may be picked up

    Fields

    Priority

    None yet

    Work type

    None yet

    Projects

    • Status
      Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions