Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
4 tasks done
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #9253 +/- ##
==========================================
+ Coverage 86.14% 86.52% +0.37%
==========================================
Files 87 88 +1
Lines 6164 6411 +247
Branches 369 385 +16
==========================================
+ Hits 5310 5547 +237
- Misses 850 860 +10
Partials 4 4 ☔ View full report in Codecov by Harness. |
Contributor
📦 Build Size ComparisonSummary
Changes🔄 Modified Routes (4)
|
ovflowd
force-pushed
the
perf/rehype-shiki-single-pass
branch
from
October 31, 2026 13:25
43e0fc2 to
991e49c
Compare
ovflowd
changed the title
ovflowd
marked this pull request as ready for review
October 31, 2026 13:29
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The cached ruleStack continuation path lacks a split-snippet test comparing each theme’s tokens with uncached Shiki, leaving carried-over theme metadata unverified.
1 open issue
What changed in this PR
This PR reduces repeated grammar tokenization in @node-core/rehype-shiki by sharing cached tokens across named themes while preserving Shiki’s rendering pipeline.
Changes:
- Adds
withTokenizationCachefor multi-theme highlighting, with fallbacks for unsupported options and grammars. - Enables caching in the highlighter and selects default themes by name.
- Adds output-parity and cache-behavior tests, plus a minor-release changeset.
| File | Description |
|---|---|
packages/rehype-shiki/src/tokenization-cache.mjs |
Implements shared tokenization and per-theme metadata caching. |
packages/rehype-shiki/src/highlighter.mjs |
Integrates caching and uses named default themes. |
packages/rehype-shiki/src/__tests__/tokenization-cache.test.mjs |
Tests output parity, cache reuse, and fallback behavior across regex engines. |
packages/rehype-shiki/src/__tests__/highlighter.test.mjs |
Updates mocks and assertions for named default themes. |
.changeset/rehype-shiki-tokenization-cache.md |
Documents the new cache wrapper for a minor release. |
Review effort: Balanced
| } | ||
| } | ||
|
|
||
| return { tokens: Uint32Array.from(binary), ruleStack, stoppedEarly }; |
Shiki tokenizes code once for each theme it's highlighted in: every theme's pass runs the grammar over every line again, though matching a grammar doesn't depend on the theme, only the styles the matched scopes get do. Highlighting in a light and a dark theme, as doc-kit and the site do, tokenizes everything twice. `withTokenizationCache`, in the new `tokenization-cache` module, wraps a highlighter so that with two or more themes given by name each line's tokens are cached for all of them: the line is tokenized once, and each theme's token metadata is computed from the scopes found, as vscode-textmate computes it, and cached by scope path. Everything after that is Shiki's own code, so the output doesn't change: it's byte-identical on the 3,357 code blocks of Node.js's API docs with both regex engines, and in full doc-kit builds. Highlighting them takes 32% less time on Oniguruma and 43% less on the JavaScript engine. Theme objects, a single theme and `includeExplanation` take Shiki's usual path, as does a grammar with token types of its own. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
The default themes were passed as theme objects, which Shiki normalizes, and parses into a new vscode-textmate theme, on every highlight, and whose tokens can't be cached for both. They're registered with the highlighter anyway, so they're now given by name, in a `DEFAULT_THEME_OPTIONS` constant. The output is the same. Given by name, a theme's own color replacements no longer apply to the code block's foreground and background, but the default themes' only replacement (`#616e88`) is neither. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
ovflowd
force-pushed
the
perf/rehype-shiki-single-pass
branch
from
October 31, 2026 13:36
991e49c to
61e78d9
Compare
avivkeller
reviewed
Oct 31, 2026
avivkeller
left a comment
avivkeller
left a comment
Member
There was a problem hiding this comment.
I don't really understand this, but I reviewed what I can reasonably understand
Member
There was a problem hiding this comment.
Can we rename this tokenizationCache.mjs? Or just cache.mjs?
Comment on lines
+9
to
+24
| /** | ||
| * A vscode-textmate grammar, with the internals its tokens' styles are | ||
| * computed from, which the interface Shiki types it with leaves out. | ||
| * | ||
| * @typedef {Object} Grammar | ||
| * @property {string} name | ||
| * @property {(line: string, stack: import('shiki/textmate').StateStack | null, timeLimit?: number) => import('shiki/textmate').ITokenizeLineResult} tokenizeLine | ||
| * @property {(scopeName: string) => { languageId: number, tokenType: number }} getMetadataForScope | ||
| * @property {{ getDefaults: () => Style, themeMatch: (scopePath: ScopePath) => Style | null }} themeProvider | ||
| * @property {{ matchesAlways: boolean, matchesNever: boolean, match: (scopes: Array<string>) => boolean } | null} balancedBracketSelectors | ||
| * @property {{ getDefaultAttributes: () => { languageId: number, tokenType: number } }} _basicScopeAttributesProvider | ||
| * @property {Array<unknown>} [_tokenTypeMatchers] | ||
| * | ||
| * @typedef {{ fontStyle: number, foregroundId: number, backgroundId: number }} Style | ||
| * @typedef {{ scopeName: string, parent: ScopePath | null }} ScopePath | ||
| */ |
Member
There was a problem hiding this comment.
Do we need to explicitly type this? Doesn't Shiki provide the type? If not, can we use unknown?
Comment on lines
+30
to
+48
| /** | ||
| * Gets a map's value for a key, setting it first if it has none. | ||
| * | ||
| * @template K, V | ||
| * @param {Map<K, V> | (K extends object ? WeakMap<K, V> : never)} map | ||
| * @param {K} key | ||
| * @param {() => V} create | ||
| * @returns {V} | ||
| */ | ||
| const getOrSet = (map, key, create) => { | ||
| let value = map.get(key); | ||
|
|
||
| if (value === undefined) { | ||
| value = create(); | ||
| map.set(key, value); | ||
| } | ||
|
|
||
| return value; | ||
| }; |
Member
There was a problem hiding this comment.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR makes
@node-core/rehype-shikicache each line's tokens for all the themes code is highlighted in, instead of tokenizing it again for each.Shiki highlights in several themes by running
codeToTokensBaseonce per theme, and each run goes through the grammar over every line again (grammar.tokenizeLine2). Matching a grammar doesn't depend on the theme, only the styles the matched scopes get do, so the default light and dark highlighting (and doc-kit's) tokenizes everything twice.tokenization-cachemodule exportswithTokenizationCache(shiki). With two or more themes given by name, it hands@shikijs/core'scodeToHast,codeToHtml,codeToTokensandcodeToTokensWithThemesa primitive whose grammar keeps two caches: each line's tokens for the length of one highlight, by the state the line starts in, and each scope path's metadata per grammar and theme, across highlights. A theme's metadata is computed from the scopes found the way vscode-textmate does: itsmergeAttributesover the scope path, the bracket bit, and merging equal neighbours. Everything after that is Shiki's own code. Theme objects, a single theme,includeExplanationand grammars with token types of their own take Shiki's usual path.createHighlighterwraps its highlighter with it.DEFAULT_THEME_OPTIONSconstant. As objects, Shiki normalizes them and parses a new vscode-textmate theme on every highlight, and their tokens can't be cached for both. Given by name, a theme's own color replacements no longer apply to the code block's foreground and background, but the defaults' only replacement (#616e88) is neither.The grammar state of a cached highlight only continues highlights made the same way, which the JSDoc says. Nothing here reads grammar state.
Validation
doc/api(3,357 of them) highlights byte-identically to Shiki's own output, on Oniguruma and on the JavaScript regex engine. Highlighting them all takes 1,873ms instead of 2,755ms on Oniguruma (-32%), and 3,745ms instead of 6,541ms on the JavaScript engine (-43%).tokenization-cache.test.mjscomparescodeToHtml,codeToHast,codeToTokensandcodeToTokensWithThemeswith Shiki's for JavaScript, TypeScript, Markdown (with embedded code), HTML, C, shell and JSON on both engines. It also checks that each line is tokenized once for three themes, that theme objects, a single theme and an unsupported grammar take the usual path, and that a line stopped at the time limit runspreprocesshooks once.tscand ESLint pass and Prettier is clean, on each commit too. A minor changeset releases it.Related Issues
Refs: nodejs/doc-kit#1205, which highlights with this package and gets the speedup once it's released.
Check List
pnpm formatto ensure the code follows the style guide.pnpm testto check if all tests are passing.pnpm buildto check if the website builds without errors.