Repository navigation
fix(search): recover camelCase symbols from prose words - #4710
Closed
ScriptedAlchemy wants to merge 14 commits into
Closed
ScriptedAlchemy wants to merge 14 commits into
ScriptedAlchemy wants to merge 14 commits into
Conversation
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Keep whole and subtoken channels independently selectable and charge only new document IDs under the shared candidate cap. Prune provably oversized postings from stored frequencies before decoding and check cancellation within bounded posting batches. Cover rare whole-word recall, overlap, common-source pruning, and mid-stream cancellation.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Do not prune a common source when the first oversized source may already cover all of its documents. Share one lower-bound guard between runtime admission and tests, and cover complete overlap and truly disjoint oversized channels. Keep test imports at module scope.
The shared admission helper owns the cap after source admission was consolidated; avoid a stale query-reader import.
Apply formatting from the failing hosted rustfmt_check_2024 gate. Preserve admission behavior, budgets and all assertions.
The exact-head Linux search evaluation finds train-020's digest-mismatch anchor at rank 5 after the candidate union fix. Three other train misses remain, and protected recall remains 8/8 without forbidden hits. Pin the measured ordered train fallback receipt and explicitly guard the recovered query. Preserve the validation receipt, corpus/workload identity, and all resource and conceptual-miss acceptance checks.
Use the validated shared fixture repair from #4731 after this branch's native Linux tests failed in the ignored-dependency publication race. The fixture now relies only on its canonical fresh serving-generation wait instead of missing a completed publication broadcast. Keep the stronger readiness assertion, its deadline and every admission regression. The matching source change passed native Linux in #4731; this branch's separate search quality regressions and full CI remain unresolved.
Multiword prose also searched each plain word as a subtoken, and subtoken postings split every chunk token, so a word the text already held whole scored twice. That reordered ordinary prose results and lost validation-018 and validation-019. Prose words now earn subtoken credit only when the row does not hold them whole and at least two of them occur only inside identifiers. This restores both validation queries and also recovers validation-017. train-020 returns to the expected-miss list: a trace showed its recovery came from stop-word fragments, so the assertion guarding it is removed. Both query-fallback receipts are re-pinned from the measured rankings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Only subtract overlapping whole-term evidence from a subtoken score when the whole-word field is admitted by the request. A Subtoken whitelist or a BodyText exclusion must not erase otherwise matching prose rows. Cover both filtered routes against real sealed lexical postings. Keep the existing unfiltered ranking and the recovered camelCase search regressions. Rust Bazel validation is required on this new branch head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multiword prose could miss camelCase symbols because unsplit words were absent from Subtoken queries. This PR admits content words through the canonical identifier-subtoken grammar while preserving single-token queries, exact search, authorization boundaries, and bounded candidate admission.
Validation of
4b73fa1742on native Linux through Bazel and the remote fleet:train-020(DigestMismatch) at rank 5, restoresvalidation-018at rank 4 andvalidation-019at rank 1, and recoversvalidation-017at rank 6.Evidence: query suites, direct evaluation, evaluator suites. These private links require tailnet access.
Fresh full CI and review remain required; this focused validation is not a claim that every repository check has passed. Keep #4451 open until its complete acceptance journey is demonstrated.