Repository navigation
HermitCrab analysis: port the edge-segment prefilter (time-only, 0 steps) #564
Description
Port ledger row 22's edge-segment prefilter to master, only if the owner wants a time-only gain. HermitCrab analysis spends 70-91% of its events on attempts that do not unapply; each attempt runs a full FST traversal even when the word's edge segment cannot match. The prefilter makes each attempt cheaper but removes no step, so under the owner's step metric its value is 0.
Problem (steps and cost)
Census: not-unapplied attempts are 70-91% of analysis events. Ledger row 7 ("count is not cost") and row 30 (time-only work is outside the owner's metric) apply. The archived record (origin/perf/hc-optimization-archive, docs/optimizations/edge-segment-prefilter.md) measured parity on 45 Sena words and 464 conformance words, FST traversals -87% and arc checks -54%, trace events unchanged. Steps saved: 0%. Arc checks and traversals are different units and are not steps.
Proposed change
Re-apply the archived prefilter to current master in AnalysisAffixProcessRule (check the first or last segment of the word against the literal edge of the affix pattern before running the traversal). No behaviour change; no new parses.
Soundness
Argued in the archive record; it was sound on the parity sets above. Must be re-argued against current master and against #561 or #562 if they land, since they touch the same rules.
Memory bound
Per-rule edge segment sets sized to the grammar, bytes to KB. No cache.
Expected value (low)
0% in steps by definition. Worth doing only as a wall-time improvement and only if the owner decides time-only work is acceptable. This issue is filed so that decision is recorded, not as a recommendation.
Measurement plan
Same-binary A/B on the 45 Sena words and the conformance suite: verdicts and trace events identical, report traversals and arc checks beside a step count that must not change.
Kill criterion
Close if the owner rules time-only work out of scope (ledger row 30), or if any trace event or parse differs.
Dependencies
None. Interacts with #561 in that it removes whole partial subtrees before their attempts run, which reduces the attempts this prefilter would make cheaper.
Related
- PR HermitCrab optimization round record: what was tried, and a faster package that is not mergeable #490: HermitCrab optimization round record (ledger rows cited above are in it).
- PR Add a conformance suite for HermitCrab, with its adequacy argument #480: HermitCrab conformance suite. Conformance per-fixture verdicts must be identical with and without the change.
- Units: deterministic analysis events and synthesis steps, never ms. "% less work" = (off - on) / off per grammar, never pooled. Census figures are ceilings unless stated. Ranges marked "est." are estimates that the shadow phase replaces with measurements.
- Owner constraint: no memo or cache; any new structure must be grammar-sized and at most a few MB.
🤖 Generated with Claude Code
Activity
johnml1135 commentedon Oct 30, 2026
The edge-segment prefilter was implemented and measured in the #490 optimization ledger round, which is now closed. The record gives deterministic counts for 45 Sena words with the prefilter off and on: FST traversals 12,376,416 to 1,617,662, arc checks 34,170,777 to 15,583,242, traversal instances 15,642,175 to 2,429,643. Analysis sets matched on all 45 Sena words and on 464 conformance-fixture words (33 grammars). It also records how the ledger's earlier ~10% cap came from matcher self-time and was wrong.
Record: https://github.com/sillsdev/machine/blob/2eba2f502338705a4d529d7034cecb6303db7f35/docs/optimizations/edge-segment-prefilter.md
Standalone wall-clock re-measure: https://github.com/sillsdev/machine/blob/2eba2f502338705a4d529d7034cecb6303db7f35/docs/hermitcrab-perf-2026-09-disposition.md
From #490 (closed).
johnml1135 commentedon Oct 30, 2026
Claude-drafted comment. Follow-up: a rule-level index on the same edge predicate, which skips a rule before visiting it, is filed as 599 with PanGloss measurements (sillsdev/PanGloss#95: Cebuano −11.8%, Aweti −4.9%, Sena −3.2%, parses identical).
johnml1135 commentedon Oct 31, 2026
Claude-drafted. Numbers update, since PR #490 is closed as a record: this issue's body says time-only (0 steps). PR #574 (R5 port) leaves the step counter unchanged, but cuts matcher work by 47.5% on Aweti, 19.6% on Mbugwe, 19.6% on Sena, 12.8% on Amharic and 6.5% on Indonesian. On 45 Sena words, the prefilter removes 87% of FST traversals (12,376,416 → 1,617,662; ledger row 22). The indexed variant is #599. PanGloss's version of it (PanGloss#95) does count work steps, and with #600's cache cut Cebuano work by 12–13% and Sena by 24.1%.