Repository navigation
Redact secrets written inside indexed files - #262
Merged
Merged
Conversation
The walk already refuses credential files by name, but a token pasted into a source file, a README or a shell script reached graph.json, the lexical and dense indexes and rerank prompts. Every text read for extraction now passes through secrets_scanner's bundled rules first; each secret becomes [REDACTED_SECRET], keeping its newlines so line numbers hold, and inline gitleaks:allow markers are not honoured. Compiling all ~1,000 rules costs 1.8 s per process, which a reader's inline update would pay on every edit. The scanner already skips a rule whose keywords a text lacks, so a run builds one scanner from only the rules its texts' keywords name: a one-file update takes 0.14 s in all. A test pins that the subset finds what the full ruleset finds on every .rs and .md file in this repository. The regex crates build optimised in the dev profile, since every test that builds a store compiles the subset.
Texts are scanned and extracted in 64 MiB batches instead of all at once, and an empty batch builds no scanner, so a no-op update stays free of the rules. GRAMMAR moves to 7 so a store written before redaction is re-read once rather than keeping the secrets of files that never change. The redaction count is the markers written, not the findings, and the docs name the paths the rules' allowlist skips.
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.env,*.pem, …), but a secret written inside an indexed file — a token in a source file, a key exported in a shell script, a private key pasted into a README — reachedgraph.json, the BM25 and dense indexes, and--rerankprompts. Every text read for extraction now goes through secrets_scanner's bundled (gitleaks-compatible) rules first. Each secret becomes[REDACTED_SECRET], newlines inside it are kept so line numbers hold, and inlinegitleaks:allowmarkers are not honoured.updatelogs per file how many were redacted.updatetakes 0.14 s in all (release build), and a test pins that this subset finds exactly what the full ruleset finds on every.rs/.mdfile in this repository. Texts go through in 64 MiB batches, and a no-op update builds no scanner.GRAMMAR6 → 7, so stores written before this change are re-read once instead of keeping the secrets of files that never change. The regex crates build optimised in the dev profile. Debug builds were 7× slower at compiling the rules, which pushedtests/reader_budget.rsfrom 3.5 s to 52.7 s.Known gap:
zerograph familiesprints the first line where each ID prefix appears without redacting it. That output goes to the terminal, not the store; left for a follow-up.Test plan
cargo test: 1,546 lib tests and every integration binary passcargo clippy --all-targets -- -D warningsis clean.zerograph/. The test fails with redaction disabled..rs/.mdfile in the repo--no-dense build5.08 s, one-fileupdate0.14 s