perf: stream dependencies between generators

.changeset/highlight-fewer-grammars.md

@@ -0,0 +1,5 @@
+---
+'@doc-kit/core': minor
+---
+
+Code is highlighted with fewer grammars by default, as each one takes its own memory on every thread: JavaScript (`js`, `mjs`, `cjs`, `javascript`) with TypeScript's grammar, which already highlights types, and C++ (`cpp`, `c++`) with C's. The Shiki plugin's `langAlias` option takes precedence over these defaults, so `{ cpp: 'cpp', 'c++': 'cpp' }` brings back C++'s own grammar, and aliases now keep working once a grammar with the same aliases is registered. A highlighter's `resolveLanguage` now returns the language code is highlighted as, as its Shiki instance names it (`typescript` for `mjs`): highlight with what it returns. Transformers see that language too.

.changeset/hold-less-in-memory.md

@@ -0,0 +1,7 @@
+---
+'@doc-kit/core': minor
+'@doc-kit/cli': patch
+'@doc-kit/generator-react': patch
+---
+
+Builds hold less in memory. The nodes of `metadata`'s entries no longer keep their `position` in the source, so neither does the output of `json-simple`, now about a third of its size. Each chunk of a streamed output is let go once every generator reading it has read that chunk, rather than once they have all read all of it, and `runGenerators(configuration, { results: false })` keeps none of the requested generators' output, which the CLI now uses. A page made of other pages' content, such as Node.js's `all.html`, renders one part at a time, rather than building every part's elements first.

.changeset/html-all.md

@@ -0,0 +1,7 @@
+---
+'@doc-kit/core': minor
+'@doc-kit/generator-react': minor
+'@node-core/doc-kit': minor
+---
+
+`all.html` moves out of the `html` generator into a new Node.js generator, `html-all` (`@node-core/doc-kit`), and `html`'s `generateAllPage` option is removed. `html-all` is spliced between `jsx-ast` and `html`, and adds `all.html`, every module page's content on one page, as a page made of theirs. `html` renders such a page, given with `parts`, the pages it is made of, from their compiled content. Builds that want `all.html` add the `html-all` target.

.changeset/six-threads.md

@@ -0,0 +1,5 @@
+---
+'@doc-kit/core': minor
+---
+
+Builds use up to 6 threads by default, instead of 4: with generators streaming into each other, more threads now pay off. On the Node.js API docs, the site build is ~15% faster at 6 threads than at 4, for ~20% more memory, while 8 threads were no faster than 6. `--threads` still sets any count.

.changeset/stream-dependencies.md

@@ -0,0 +1,6 @@
+---
+'@doc-kit/core': minor
+'@doc-kit/generator-react': minor
+---
+
+Generators can start on their dependency's first chunk instead of waiting for all of it: with `streamsInput: true`, a generator receives its dependency's output as an async iterable of chunks (with one thread, still whole), and `worker.stream` accepts one. `worker.stream` now keeps only a few chunks per thread in flight, started in the order they were submitted, so generators running at once share the worker pool. `metadata`, `section-pages`, `jsx-ast` and `html` now stream their input, so with more than one thread, parsing, metadata, page building and rendering overlap: `html` bundles as soon as every page's metadata is known, and renders pages as they are built. Streaming leaves the generated files unchanged, but `jsx-ast` now yields pages in the order their entries arrive, not the order the site lists them, and, once every page has arrived, a `{ manifest }` item listing their heads in that order. To list the pages as the site does, skip the `{ manifest }` item and sort them with `headingSortFn`, now exported from `@doc-kit/generator-react/jsx-ast/utils/getSortedHeadNodes`, as `pages.toSorted((a, b) => headingSortFn(a.data, b.data))`, keeping the synthetic pages (`data.synthetic`) last. Bundler adapters' `compile` can now be called before and while `buildServer` and `buildClient` run.

.github/workflows/ci.yml

@@ -77,7 +77,7 @@ jobs:
           path: node
 
       - name: Build Node.js API docs
-        run: node packages/cli/bin/cli.mjs generate -t web -i "./node/doc/api/assert.md" -o out --config-file ./e2e/doc-kit.config.mjs
+        run: node packages/cli/bin/cli.mjs generate -t web -t html-all -i "./node/doc/api/assert.md" -o out --config-file ./e2e/doc-kit.config.mjs
 
       - name: Install Playwright browsers
         run: pnpm exec playwright install --with-deps

docs/configuration.md

@@ -134,7 +134,7 @@ these for that generator alone, except `markdown`, which it adds to.
 Top-level, alongside `target` and `global`:
 
 - `threads` {number} Worker threads used for generation. Defaults to your
-  CPU count. Each worker's heap is limited to 512MB; to raise the limit, run
+  CPU count, up to 6. Each worker's heap is limited to 512MB; to raise the limit, run
   Node.js with `--max-old-space-size` (directly or through `NODE_OPTIONS`).
 - `chunkSize` {number} Items processed per worker thread. **Default:** `10`.
 
@@ -219,12 +219,20 @@ bundles. To configure it, list it with options in
 dependencies, so your configuration file can resolve it):
 
 - `langs` {Array} More languages: grammars, or modules default-exporting them.
-- `langAlias` {Object} Aliases of languages, such as `{ conf: 'ini' }`.
+- `langAlias` {Object} Aliases of languages, such as `{ conf: 'ini' }`. An
+  alias leads to the language it names, or on to the one that language is an
+  alias of. By default, JavaScript (`javascript`, `js`, `mjs`, `cjs`) is
+  highlighted with TypeScript's grammar, and C++ (`cpp`, `c++`) with C's:
+  every grammar takes its own memory on each thread, and those highlight that
+  code too. Aliases take precedence over these defaults:
+  `{ cpp: 'cpp', 'c++': 'cpp' }` highlights C++ with its own grammar, and a
+  grammar in `langs` for one of these languages needs an alias to it.
 - `themes` {Object} The `light` and `dark` themes: names of themes Shiki
   bundles, modules default-exporting themes, or themes.
 - `transformers` {Array} Modules default-exporting
   [Shiki transformers](https://shiki.style/guide/transformers), or lists of
-  them.
+  them. They see the language code is highlighted as (`options.lang`), such as
+  `typescript` for `mjs`.
 
 These modules are paths relative to the working directory, or URLs, such as
 the ones `import.meta.resolve()` gives:

docs/creating-generators.md

@@ -330,9 +330,64 @@ export async function* generate(input, worker) {
 ### Benefits of Streaming
 
 - **Reduced memory usage** - Process data in chunks
-- **Earlier downstream starts** - Next generator can begin before this one finishes
+- **Earlier downstream starts** - A generator that streams its input (see below)
+  begins on the first chunk, before this one finishes
 - **Better parallelism** - Multiple generators can work simultaneously
 
+### Streaming Input
+
+By default, a generator receives its dependency's output once it is complete,
+collected into one array. With `streamsInput: true`, it receives it as it is
+produced instead: `input` is an async iterable of the dependency's chunks, and
+if the dependency streams its results too, the generator starts as soon as the
+dependency does.
+
+```mjs displayName="index.mjs"
+import { generate, processChunk } from './generate.mjs';
+
+export default {
+  name: 'per-chunk-generator',
+
+  description: 'Processes each chunk of metadata as it arrives',
+
+  dependsOn: '@doc-kit/core/metadata',
+
+  hasParallelProcessor: true,
+
+  streamsInput: true,
+
+  generate,
+  processChunk,
+};
+```
+
+`worker.stream` takes the chunks as they arrive, starting each one's work while
+the next ones are still being produced:
+
+```mjs displayName="generate.mjs"
+export async function* generate(input, worker) {
+  for await (const chunkResult of worker.stream(input, {})) {
+    yield chunkResult;
+  }
+}
+```
+
+- A dependency that does not stream arrives as a single chunk, once it is
+  complete.
+- With one thread, nothing would run alongside, so the input arrives whole, as
+  it does without `streamsInput`: accept both, as `worker.stream` does.
+- With more than one thread, `input` can be iterated only once, and passing it
+  to `worker.stream` iterates it. If you need it twice, collect it into an
+  array first.
+- The chunks are the dependency's own, in the order it yields them. `metadata`'s
+  hold whole files, but chunks need not hold whole modules: a generator that
+  needs all of a module's entries at once should regroup them with
+  `groupWholeModules` from `@doc-kit/core/utils/generators.mjs`.
+- Work that needs the whole input, such as sorting or writing one file, has to
+  wait for the last chunk.
+- Other generators reading the same dependency are not affected: each gets
+  every chunk, from the first.
+
 ### Non-Streaming Generators
 
 Some generators must collect all input before processing.
@@ -526,7 +581,9 @@ returning the plugin, instead of a default export. The Shiki plugin does, and
 returns the highlighter of a generator's Shiki plugin, to highlight code as its
 pipeline does. It registers Shiki's bundled languages as code in them is
 highlighted, so before using its `shiki` instance directly, pass the language
-through its `resolveLanguage`.
+through its `resolveLanguage`, and highlight with the language it returns,
+which may not be the one the code names (`mjs` gives `typescript`, and an
+unknown language `text`).
 
 ## File Output
 

docs/generators.md

@@ -48,6 +48,7 @@ Outputs consumed by the Node.js project's own build and release processes.
 | [`man-page`](./generators/man-page.md)         | The `node.1` manual page.                           |
 | [`api-links`](./generators/api-links.md)       | A map of API symbols to their source locations.     |
 | [`addon-verify`](./generators/addon-verify.md) | Extracted addon code samples, arranged for testing. |
+| [`html-all`](./generators/html-all.md)         | `all.html`, every module page on one page.          |
 
 ## Pipeline stages
 

packages/cli/bin/commands/generate.mjs

@@ -64,6 +64,6 @@ export default generate.action(
     const config = await setConfig(opts);
     assertRunnableOptions(config);
 
-    await runGenerators(config);
+    await runGenerators(config, { results: false });
   })
 );

packages/cli/bin/commands/preview.mjs

@@ -62,7 +62,7 @@ export default preview.action(
       const config = await setConfig(opts);
       assertRunnableOptions(config);
 
-      await runGenerators(config);
+      await runGenerators(config, { results: false });
 
       // With the default coming from the configuration, this is always set
       output = config.global.output;