Repository navigation
Keep a search's order for its later pages, so walking the catalog is no longer quadratic - #106
Merged
Merged
Conversation
added 2 commits
November 18, 2026 17:14
Paging tools.search ranked every row its targets hold again for each page (tool-search-store page()), so walking a catalog of n tools in pages of k cost n²/k on the product's one thread. On Nov 18 three walks with neither query nor namespace over 11,467 tools held a self-hosted product at full CPU for minutes. A page past the first now ranks once, keeps the order (eight searches, least recently used dropped) and reads later pages' rows by id, in a statement that also checks every ranked listing still has its digest; a listing written since is ranked again in one statement, as before. The first page, which most searches end at, is unchanged. Paging, offsets and totals keep their contract. mcp-search-walk walks 8,000 tools twice: SQL across 55 pages falls from 1,848 ms to 153 ms, and a later page from ~34 ms to ~3 ms.
A kept order held each item as a [row id, target, ordinals] array: about 5 MB per order at 11,467 tools on V8 and 28 MB at 60,000, eight of them with no byte cap. An item is now three Int32 cells (target, row position in its listing, end of its other merged targets), about 140 KB per order at 11,467 tools. Kept orders are bounded at 32 searches and 6 MB together, 2 MB each; a larger order is not kept and its pages rank as before. A miss reads its page from the same statement that ranks, so it no longer makes a second read. A new order evicts only a search unread for 30 s: while every kept search is in use, a new one's pages rank as before, so more concurrent walks than the cap cost what they did before, instead of evicting each other on every page.
shuv1337
added a commit
that referenced
this pull request
Nov 19, 2026
…search in use (#107) * Stop the store walk at any value it has walked, not only leaves A shared value of many small records, such as a meta object of 1,000 records attached to every row, was walked record by record on every store: about 500 steps each, 10 million for 20,000 rows, so the program stopped at the 5,000,000-step budget. The store walk now remembers every object whose whole graph it checked. A leaf stays remembered until an object is stored into it; any other object is remembered for the current epoch, and storing an object into a remembered object that another remembered object holds starts a new epoch. A differential fuzz of 3.9 million stores against a fresh reachability check agrees on all of them and fails at once when the epoch is not advanced. mcp-execute-step-budget: 20,000 rows sharing a 1,000-record meta now complete in 283 ms (TimeoutExceeded at 1,317 ms before); storing a row into that shared value is still rejected as a cycle; runaways still stop with TimeoutExceeded. * Never evict a search still being paged, and keep oversized searches apart Kept search orders (#106) had three gaps: - room was checked before the ranking await and eviction after it was strict LRU, so new searches ranking at once evicted searches still in use; an order is now admitted after the await, evicting only searches unread for 30 s, or not kept; - idle time read the wall clock; it now reads the monotonic clock; - a search too large to keep was remembered in the same cache with its key's bytes, and a key past 6 MB evicted every kept search; such searches are now remembered in their own bounded map, and a key too large to keep is never remembered. mcp-search-kept-orders covers the first and third: both assertions fail on the previous store and pass on this one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Paging
tools.searchre-ranked every row its targets hold on every page (tool-search-store.tspage()). So walking a catalog of n tools in pages of k cost n²/k on the product's single thread. On Nov 18 (3:10–3:24 PM PT), three concurrent walks with neither a query nor a namespace, over 11,467 tools, ran at 0.8–5 s per page. That held the self-hosted product isolate at ~90–100% CPU in bursts.The fix keeps the paging contract (
offset,limit,remaining,nextand totals are unchanged) and makes a walk cost one ranking plus n row reads:listing:position), and the end of its other merged targets in a separate Int32 array. That's 12 B per item, about 140 KB per order at 11,467 tools and 720 KB at 60,000.below) plus terms. Callers with different scopes never share an order.sdk.tools.searchspans carryexecutor.search.order, which isranked,orderedorkept.I considered capping listings that have no query at 200 items and dropped the idea: it would break the documented "page through a large catalog" contract, and it doesn't help a single large namespace.
Verification
All runs were on a Linux x64 dev host with Node 24.22.2 and Bun.
bun run check: passes (format, lint, typecheck of 9 projects, protocols/telemetry/API-errors checks, no-unit-tests, boundary).New scenario
mcp-search-walk.spec.ts: "An MCP search without a query pages through 8,000 tools ranking them once, each later page reading only its own rows". It deploys 16 apps × 500 tools and walks the whole catalog withtools.search({ limit: 150 })plusnext, twice, in one execute each (55 pages).It asserts that every tool is listed once.
It asserts the order sequence: first walk
ranked, ordered, kept…; second walkranked, kept….It asserts that the median kept page's SQL time is under half the first page's ranking SQL.
Before and after, from the same scenario (SQL time summed from
sql.executespans under eachsdk.tools.search; "after" is head 08fabf9):The before run fails the new scenario's assertions; the after run passes.
Memory, with PR Reviewer's PoC on the real store and PGlite, 16 listings. "Held" is the memory freed when the store and its kept orders are dropped, as heap plus external, since Int32Array bytes are external on V8.
Node 24 (V8):
Before this commit: 5.39 MB per order and 43 MB for 8 at 11,467 tools; 28 MB per order and 224 MB for 8 at 60,000.
Bun:
process.memoryUsage()didn't show the freed memory after dropping the store, so I report growth while the orders were kept, after warm-up:Before this commit: 12.5 MB per order and 100 MB for 8 at 11,467 tools.
The reviewer's original
mem.ts, which readsheapUsedwith no warm-up, reports 0.05 MB per order on Node, because it doesn't count the external typed arrays. On Bun it reports a flat 80.5 MB at both 11,467 and 60,000 tools. I read that as first-use warm-up of the ranking-with-order statement: it doesn't grow with the catalog, and with a warm-up it doesn't show.Thrash (PR Reviewer's
thrash.ts: 8,000 tools, walkers in lockstep, 6 pages each, ms per page):At 40 walkers, the 8 walkers the cache has no room for rank each page as before, and nothing is evicted while in use.
Correctness, PR Reviewer's PoCs on this head:
Self-host scenarios: on f5995c3, one combined run of 48 scenarios, 43 passed. The 5 failures were the 4
mcp-catalog-scalescenarios described below and the first draft of mcp-search-walk. Passing files: mcp-search-walk, mcp-search-statements, search-statements, mcp-search-concise, mcp-catalog, catalog-agent-search, mcp-execute-reach, mcp-execute-lazy, mcp-execute-concurrency, mcp-execute-failures, and Cold MCP discovery (8,549 OpenAPI tools).Local:
MCP search resolves namespace-relative dotted pathsandA broad MCP tool search returns concise pagespass (2/2), as in fork-ci.mcp-catalog-scale.spec.tson self-host: 3/7 pass. The same 4 fail identically on unpatched f482d19 on this host (TraceUnsettledon "over 7,000 tools" and "beside stalled apps", plus "still aged" and "not yet reported" on the two aged-rows scenarios). They are pre-existing and environmental, and not caused by this change: those scenarios only search page 0, whose statement is unchanged.Not run: the Cloud target and the packaged workerd Docker image. fork-ci doesn't run the new scenario, and I made no workflow edits.
bun run e2e:check-descriptionfails on this repo before reaching this PR's block. It reports scheduled scenariocloudOAuthProxyRefusalwith no CI job, which is unrelated and pre-existing; fork-ci has noselectjob.Re-run on 08fabf9:
bun run checkpasses;Behavior
No API or contract change. The only new state is up to 32 kept search orders in the store's memory, together at most 6 MB. A miss costs one statement, the ranking plus that order, with no second read.
Deployment
Not deployed. Shipping it to prod needs a product restart, which needs Lauren's sign-off.