diff --git a/docs/hermes-base-verification.md b/docs/hermes-base-verification.md index a475307..bd9396e 100644 --- a/docs/hermes-base-verification.md +++ b/docs/hermes-base-verification.md @@ -18,7 +18,7 @@ |---|---| | `Bytecode File Information` 等头部 | 文本头不比较；raw 核对从二进制比较运行时选项、全局函数索引、segment ID、模块与函数源码表 | | `Global String Table` | 不比较死字符串；有效 HBC 的字符串按二进制表恢复完整内容，而非截断或转义后的显示文本 | -| `Array Buffer` / `Object Key Buffer` / `Object Value Buffer`（v98：`Literal Value Buffer` / `Object Key Buffer`） | 文本段本身**不比较**（见下）；无法读二进制缓冲区时整段文本仅用于诊断，不能据此判定等价 | +| `Array Buffer` / `Object Key Buffer` / `Object Value Buffer`（v98/v99：`Literal Value Buffer` / `Object Key Buffer`） | 文本段本身**不比较**（见下）；无法读二进制缓冲区时整段文本仅用于诊断，不能据此判定等价 | | `Function<…>` / `NCFunction<…>` / `Constructor<…>` 各函数 | 头文本必须相同；正文逐行经 `normalizeDisassemblyLine` 归一化后比较；**按函数分段**，错位不级联 | | `Debug *` / `Textified callees table` | 排掉（只服务调试器/性能工具） | @@ -26,14 +26,14 @@ 两种布局： -| | v87–96（`split`） | v98（`shaped`） | +| | v87–96（`split`） | v98/v99（`shaped`） | |---|---|---| | 缓冲区 | `arrayBuffer` / `objKeyBuffer` / `objValueBuffer` | `literalValueBuffer`（数组元素与对象值共用）/ `objKeyBuffer` / `objShapeTable`（8 字节项：keyOffset u32、numProps u32） | | 数组 | `NewArrayWithBuffer rX, sizeHint, count, offset` | 同左，offset 指向值缓冲区 | | 对象 | `NewObjectWithBuffer rX, sizeHint, count, keyOffset, valueOffset` | `NewObjectWithBuffer rX, shapeIndex, valueOffset`；`NewObjectWithBufferAndParent rX, rParent, shapeIndex, valueOffset` | | tag 类型 6 | 1 字节字符串 id | `undefined`（无值字节）；字符串 id 只用 2/4 字节 | -v98 的 shape 索引和 offset 一样只用于定位（delta 可能重排 shape 表），归一化后只留 shape 指向的键与值；`AndParent` 的父对象寄存器原样保留比较。 +v98/v99 的 shape 索引和 offset 一样只用于定位（delta 可能重排 shape 表），归一化后只留 shape 指向的键与值；`AndParent` 的父对象寄存器原样保留比较。 为什么不能按 dump 的整段文本比：Hermes 的缓冲区构建器会**重叠/去重**序列化后的字面量——一个字面量的最后一个值字节可以同时是下一个字面量的 tag 字节（模糊测试实测：`61 52 | cd 09 b3 05 11`，前一段以 `[String 82]` 结尾，后一条指令的 offset 正指向 `52`）。顺序解析整段缓冲区（hermesc 的 dump 就是这么打印的）从这里开始失步，之后的条目全是噪声；delta 构建的 id 宽度不同，重叠位置也不同，于是两段"噪声"在某处不一致就被判为差异。2026-09-10 的 20 轮冒烟模糊测试里 3 次误杀全部源于此，改按指令比较后全部等价。无法读二进制缓冲区（文件结构不识别）时两侧一起比较整段文本，仅辅助诊断；即使文本相等也返回 `dump-failed` 并回退 plain，不能以丢失 offset/count 的文本确认等价。结果里 `literals: 'buffer'` 标明这一点。 @@ -43,7 +43,7 @@ v98 的 shape 索引和 offset 一样只用于定位（delta 可能重排 shape **原始操作数核对**：`hermes-raw.ts` 从 raw dump 读取指令起点和操作数类型，并检查操作数与 HBC 字节一致、指令覆盖完整函数体。字符串从 small/overflow string table 与 string storage 按完整 ASCII/UTF-16 code unit 解码；BigInt、正则和 double 读取真实字节（保留 `-0` 和尾部精度）；函数引用保留索引，与顺序对齐的函数表共同检查，同名函数不能互换。地址映射为目标指令序号；整数和字符串 switch 从二进制恢复 case 值与目的地。函数运行时 flags 和参数/寄存器等字段也参与比较，剔除的仅是物理地址、debug presence 与 compact/overflow 表示。 -这不是一个可以忽略所有新指令的通用语义证明器。已验证的函数头范围为 HBC v87–96，以及 `hermes-compiler@250829098.0.16/.17` 的 v98 快照；升级布局或引入新的字符串/shape 引用指令时，需要复核 `hermes-raw.ts` 的解码规则及测试，不可仅扩展 diff-transform 的布局表。 +这不是一个可以忽略所有新指令的通用语义证明器。已验证的函数头范围为 HBC v87–96、`hermes-compiler@250829098.0.16/.17` 的 v98 快照，以及 `hermes-compiler@260318099.0.4` 的 v99 快照；升级布局或引入新的字符串/shape 引用指令时，需要复核 `hermes-raw.ts` 的解码规则及测试，不可仅扩展 diff-transform 的布局表。 结果三态：`equivalent` / `different`（带第一处差异：函数、行号、两侧内容，或缓冲区条目）/ `dump-failed`（无法解析完整语义数据、dump 进程失败/超时/提前结束；带原因或 stderr 末行）。后两种都放弃 base，但日志分开。 @@ -51,6 +51,8 @@ v98 的 shape 索引和 offset 一样只用于定位（delta 可能重排 shape **v98 函数头快照边界**：当前大头按 37 字节、flags 位于 `[36]` 读取，小头 cache 位域为 6/1/1，绑定 `250829098` 稳定快照。上游 [7193d4485b](https://github.com/facebook/hermes/commit/7193d4485beeb87cd7a3b6ca8b6b5d97a1a433c4) 删除 `NumCacheNewObject` 后，仍报 v98 的构建曾使用 36 字节 / flags `[35]`、cache 位域 7/1。`hbcTransform` 的两套 v98 文件头布局不能识别这次**函数头**变化；本轮不宣称支持该后续快照，也不能仅凭文件头或版本号选择它。更换编译器时必须补对应的大头、小头和真实 Metro 测试；未审核的 v98 构建应使用 `--hermesBase none`。 +**v99 函数头**：`hermes-compiler@260318099.0.4`（RN 0.88）是上述删除之后的首次版本号升级，文件头与段布局同晚期 v98（`hbcTransform` 共用一条布局），函数头一律是删除后的形态：大头 36 字节、flags 位于 `[35]`，小头 cache 位域 7/1。v99 新增的 typed 模式 `NewTypedObjectWithBuffer` 的字面量操作数尚未解码，遇到即 `dump-failed` 回退 plain。 + pretty 输出本身会截断长字符串与 BigInt、用函数名替代函数索引，并可能把 `-0` 显示为 `0`，因此不再以 pretty 相等作为最终结论。新增归一化规则时必须同时添加真实 HBC 负例，确保没有把语义差异折叠掉。 ## 3. 已完成与待办 @@ -70,7 +72,7 @@ SELECT hermesBaseOutcome, COUNT(*) FROM versions 旧版服务端忽略未知字段，CLI 先于服务端发布也安全。 -### 3.2 指令级缓冲区偏移 —— 已完成（v87–96、v98） +### 3.2 指令级缓冲区偏移 —— 已完成（v87–96、v98、v99） 见 §2"字面量按指令比较"。v98 最初按 v96 的 tag 语义解码，模糊测试（300 轮，RN 0.87 的 hermes-compiler）报出 259 次对象值不一致——全部源于 tag 类型 6 在 v98 表示 `undefined` 而非 1 字节字符串 id；修正后 v98 两个种子共 800 轮：0 次差异、75 处植入差异全部抓到。 @@ -90,9 +92,10 @@ raw 核对保留函数引用索引，与出现顺序对齐的函数体及二进 - **资源开销**：base/plain 编译并发完成后，执行 pretty 和 raw 两遍 dump；每遍两个进程，raw 不增加编译。二进制元数据读取目前持有两份 HBC、完整字符串映射及字面量缓冲区，反汇编仅保留当前函数；这是完整数据核对的额外内存和时间开销。不要以删除验证数据来优化内存，可后续改为按段读取或降低并行度。 - **进程期限**：版本探测默认 30 秒（`PUSHY_HERMES_PROBE_TIMEOUT_MS`），完整校验两遍合计默认 120 秒（`PUSHY_HERMES_VERIFY_TIMEOUT_MS`），单个编译/源码映射子进程默认 300 秒（`PUSHY_HERMES_COMPILE_TIMEOUT_MS`）。环境变量单位均为毫秒，必须是 1–2147483647 的整数，否则用默认值。超时终止子进程，优化失败回退 plain；真正的 plain 编译或最终 sourcemap 失败仍使构建失败。校验函数还接受 `AbortSignal`；base 下载任务的取消传播尚未统一。 - **源码映射竞态**：推测执行的 base sourcemap 合成任务启动时立即观察拒绝，之后再根据最终采用哪份字节码决定抛出错误还是为 plain 重做合成。 -- **CI**：`hermes-hbc-96` / `hermes-hbc-98` / `hermes-hbc-98-patch17` 分别安装固定的 `react-native@0.77.3` / `hermes-compiler@250829098.0.16` / `.17`。除小程序回归与 50 轮固定种子 fuzz 外，三组都执行真实 Metro bundle 的自比和 base/plain 比较；v98 测试断言产物确实包含零长度函数。缺编译器或指定的 fixture 缺失/哈希不符会失败，不静默跳过。 +- **CI**：`hermes-hbc-96` / `hermes-hbc-98` / `hermes-hbc-98-patch17` / `hermes-hbc-99` 分别安装固定的 `react-native@0.77.3` / `hermes-compiler@250829098.0.16` / `.17` / `hermes-compiler@260318099.0.4`。除小程序回归与 50 轮固定种子 fuzz 外，四组都执行真实 Metro bundle 的自比和 base/plain 比较；v98 测试断言产物确实包含零长度函数。缺编译器或指定的 fixture 缺失/哈希不符会失败，不静默跳过。 - **真实 Metro 数据**：[hbc-diff-benchmark](https://github.com/sunnylqm/hbc-diff-benchmark/tree/e6a870a1c26c4b64c7860d7e1aa575707d22ad88) 的 `base.jsbundle` 和 `s3-medium-feature.jsbundle`（MIT，保留其 LICENSE；来源与生成步骤见该仓库 `fixtures/GENERATION.md`）。CI 固定提交和 SHA256，不运行下载的 JS，仅交给固定编译器。离线运行：`HERMESC= HERMES_METRO_FIXTURES= bun test tests/hermes-metro.test.ts`；普通单元测试不联网下载。 - **调试输出异常回归**：`tests/hermes-blockers.test.ts` 在独立进程覆盖带 `dumpTo` 的超时、运行中取消、预取消、ENOENT，同时检查返回标记、退出状态和晚到的异步错误。错误处理立即 unpipe 全部目的地并结束 debug 文件，避免 `finish()` 等待一个只会在后续 `kill()` 中结束的流；Node 18 CI 也执行相同场景。 +- **不要删掉 pretty 这一遍**：raw 指令归一化不覆盖异常处理表等信息，pretty 对比仍是必要的一层（PR #85 评审结论）。 - **`hbcdump`/`hbc-diff`**：Hermes 仓库自带的工具，RN 的 hermesc 不随附；如果将来 hermes-compiler 包里带上，可替代文本 dump 解析。 ## 4. 明确接受的剩余风险 @@ -104,6 +107,6 @@ raw 核对保留函数引用索引，与出现顺序对齐的函数体及二进 1. 看日志里的 detail：函数 + 行号 + 两侧内容，或缓冲区条目，或 dump 失败原因。 2. 两行只差宽度后缀/数字 id/偏移：先核对解析后的完整值与控制流目标。只有证明是表示差异后才能补归一化规则，不能直接删除操作数；同时添加负例与 fuzz。 3. 指令数量/顺序/opcode 不同，或某条 `New*WithBuffer` 解析后的字面量不同（detail 形如 `NewArrayWithBuffer r2 size=44 n=44 entry 7: [String "a"] vs [String "b"]`）→ 真差异，回退正确，向 Hermes 上游报告。detail 带 `undecodable` 说明无法解码，归为 `dump-failed`；可能是损坏的 HBC，也可能是验证器尚不支持的布局，不能直接断言上游错误。 - 若日志/元数据里 `literals` 为 `buffer`（文件结构不识别），缓冲区是按整段文本比的，`Array Buffer entry N` 形式的差异可能是 §2 的重叠误杀，需人工用二进制解码核对。 + detail 形如 `Array Buffer entry N: … vs …` 时，说明文件结构未被识别、缓冲区是按整段 dump 文本比的（比较结果的 `literals` 为 `buffer`，该字段不进日志），差异可能是 §2 的重叠误杀，需人工用二进制解码核对。 4. `dump-failed` → 校验无法完成；检查解析原因、编译器版本、超时与 stderr。它不是等价证据，也不必然是与字节码无关的进程错误。 5. 需要完整现场：`PUSHY_HERMES_BASE_DEBUG=1 pushy bundle …`，两份反汇编在中间目录旁 `hermes-base-dump-{base,plain}.txt`。 diff --git a/docs/hermes-review-follow-up.md b/docs/hermes-review-follow-up.md deleted file mode 100644 index 604207b..0000000 --- a/docs/hermes-review-follow-up.md +++ /dev/null @@ -1,52 +0,0 @@ -# PR #85 review follow-up - -## Changes - -- `CacheNewObject` now resolves its shape index into the ordered, fully decoded - property keys in the pretty pass. Both registers and the per-function cache - index remain part of the comparison. Unknown shapes, undecodable keys and - unsupported operands fail closed. With no binary resolver, the shape is only - folded for diagnostics; the existing binary-data and raw-audit gates still - forbid text-only equivalence. -- Fuzz success requires a positive integer round count, all requested positive - comparisons, zero compilation/comparison failures, at least one effective - planted difference, and no missed or unbuildable negatives. Optimized-away - differences do not count. The last round also attempts a negative, so runs - shorter than ten rounds are not structurally unable to exercise rejection. -- New regressions cover shape-index relocation, key changes, key order, - register/cache preservation, malformed/missing references, text-only fallback, - and fuzz runs with zero or incomplete useful coverage. -- Enforcing that gate exposed a pre-existing generator defect in the HBC 96 - CI run (seed 96, round 41): the quoted-string regex matched a gap after an - escaped quote and inserted `~` before a quoted property name. String mutation - and negative planting now use real string-token boundaries from the already - installed Babel parser and JSON-encode replacements. The minimized regression - also checks comments, regexps, templates, escapes and UTF-16 source offsets. - The failure gate remains strict; the generator is fixed rather than skipping - the failing round or changing the seed. - -## Scope and evidence - -The CacheNewObject regression uses synthetic instruction bytes and binary literal -sections to exercise the production pretty and raw normalizers. It does not claim -that a source-level fixture was compiled into CacheNewObject by a real hermesc. -The existing pinned HBC 96/98 compiler CI remains required before merging. - -Unknown-opcode semantic coverage remains a separate follow-up, not a verified -current-compiler false-acceptance bug. Before accepting additional compiler -snapshots, audit their complete opcode/operand schemas and referenced sections. -Do not remove the whole pretty pass: information such as exception-handler tables -is not currently covered by the raw instruction normalizer alone. - -Local validation in the authoring environment is limited to Node.js execution of -transpiled helpers and synthetic fixtures; Bun and hermesc are not installed. -The PR description records the subsequent CI state separately. - -## CodeRabbit argument-validation follow-up - -Only an omitted `--rounds` flag selects the default of 200. A flag without a -value, an empty value, or a following option is rejected with exit code 2 before -creating the output directory or invoking the compiler. Four CLI regressions -use an executable compiler fixture with a call marker to verify those side -effects do not occur; the valid-count control verifies the marker does work. -Argument-reading, compilation and run-entry functions now document their contracts. diff --git a/src/bundle-runner.ts b/src/bundle-runner.ts index 323a647..b97629a 100644 --- a/src/bundle-runner.ts +++ b/src/bundle-runner.ts @@ -1358,7 +1358,6 @@ async function selectHermesBase( try { const base = await resolveHermesBase({ option: request.option, - hermesCommand, bytecodeVersion, appId: request.appId, cacheMaxMb: request.cacheMaxMb, diff --git a/src/utils/hbcTransform.ts b/src/utils/hbcTransform.ts index 43f205f..d89d0e9 100644 --- a/src/utils/hbcTransform.ts +++ b/src/utils/hbcTransform.ts @@ -219,16 +219,10 @@ export const HBC_LAYOUTS: HbcLayout[] = [ // 多一个 numStringSwitchImms 槽(hermes static_h 在不 bump 版本号的 // 情况下插入,同一版本号 98 因此存在两种布局,依结构校验区分)。 // 段布局与早期 v98 相同。 - minVersion: 98, - maxVersion: 98, - headerFields: V98_LATE_HEADER_FIELDS, - sections: V98_SECTIONS, - }, - { - // Hermes V1 v99(hermes-compiler 260318099,RN 0.88+):文件头与段布局 + // Hermes V1 v99(hermes-compiler 260318099,RN 0.88+)文件头与段布局 // 同晚期 v98,只改了函数头内部位域(去 NumCacheNewObject)与操作码表, // 二者都不在本变换触及的范围内。 - minVersion: 99, + minVersion: 98, maxVersion: 99, headerFields: V98_LATE_HEADER_FIELDS, sections: V98_SECTIONS, @@ -240,27 +234,9 @@ export const HBC_LAYOUTS: HbcLayout[] = [ // arrayBufferSize 更名 literalValueBufferSize(槽位不变)。 minVersion: 98, maxVersion: 98, - headerFields: [ - 'fileLength', - 'globalCodeIndex', - 'functionCount', - 'stringKindCount', - 'identifierCount', - 'stringCount', - 'overflowStringCount', - 'stringStorageSize', - 'bigIntCount', - 'bigIntStorageSize', - 'regExpCount', - 'regExpStorageSize', - 'literalValueBufferSize', - 'objKeyBufferSize', - 'objShapeTableCount', - 'segmentID', - 'cjsModuleCount', - 'functionSourceCount', - 'debugInfoOffset', - ], + headerFields: V98_LATE_HEADER_FIELDS.filter( + (field) => field !== 'numStringSwitchImms', + ), sections: V98_SECTIONS, }, ]; @@ -398,7 +374,11 @@ export type HbcSectionRange = { start: number; size: number }; export function resolveHbcSections( header: Buffer, fileLength: number, -): { version: number; sections: Map } | null { +): { + version: number; + layout: HbcLayout; + sections: Map; +} | null { const version = getHbcVersion(header); if (version === null) return null; for (const layout of findLayouts(version)) { @@ -406,6 +386,7 @@ export function resolveHbcSections( if (resolved) { return { version, + layout, sections: new Map( resolved.map((s) => [s.desc.name, { start: s.start, size: s.size }]), ), @@ -455,7 +436,7 @@ export function transformHbc(buf: Buffer, inverse = false): Buffer | null { } /** 返回该 buffer 结构上匹配的布局(候选中第一个校验通过者)。 */ -export function pickLayout(buf: Buffer): HbcLayout | null { +function pickLayout(buf: Buffer): HbcLayout | null { const version = getHbcVersion(buf); if (version === null) return null; for (const layout of findLayouts(version)) { @@ -610,6 +591,25 @@ export type HbcTransformMeta = { layout: HbcLayoutWire; }; +/** 两侧共同的 HBC 版本;任一侧不是 HBC 或版本不同时为 null */ +function sharedHbcVersion(oldHeader: Buffer, newHeader: Buffer): number | null { + const version = getHbcVersion(newHeader); + return version !== null && getHbcVersion(oldHeader) === version + ? version + : null; +} + +function transformMeta( + hbcVersion: number, + layout: HbcLayout, +): HbcTransformMeta { + return { + v: HBC_TRANSFORM_VERSION, + hbcVersion, + layout: compileLayoutToWire(layout), + }; +} + /** * 若 old/new 均为同版本且存在两侧都通过结构校验的候选布局,返回变换结果、 * 选中布局与写入 __diff.json 的元数据;否则返回 null。 @@ -623,25 +623,13 @@ export function tryTransformPair( layout: HbcLayout; meta: HbcTransformMeta; } | null { - const oldVersion = getHbcVersion(oldBuf); - const newVersion = getHbcVersion(newBuf); - if (oldVersion === null || newVersion === null || oldVersion !== newVersion) { - return null; - } - for (const layout of findLayouts(newVersion)) { + const version = sharedHbcVersion(oldBuf, newBuf); + if (version === null) return null; + for (const layout of findLayouts(version)) { const tOld = transformHbcWithLayout(oldBuf, layout, false); const tNew = transformHbcWithLayout(newBuf, layout, false); if (!tOld || !tNew) continue; - return { - tOld, - tNew, - layout, - meta: { - v: HBC_TRANSFORM_VERSION, - hbcVersion: newVersion, - layout: compileLayoutToWire(layout), - }, - }; + return { tOld, tNew, layout, meta: transformMeta(version, layout) }; } return null; } @@ -662,12 +650,9 @@ export async function tryTransformPairFiles( readHbcFileHeader(newPath), ]); if (!oldInfo || !newInfo) return null; - const oldVersion = getHbcVersion(oldInfo.header); - const newVersion = getHbcVersion(newInfo.header); - if (oldVersion === null || newVersion === null || oldVersion !== newVersion) { - return null; - } - for (const layout of findLayouts(newVersion)) { + const version = sharedHbcVersion(oldInfo.header, newInfo.header); + if (version === null) return null; + for (const layout of findLayouts(version)) { if ( !resolveSectionsFromHeader(oldInfo.header, oldInfo.size, layout) || !resolveSectionsFromHeader(newInfo.header, newInfo.size, layout) @@ -679,14 +664,7 @@ export async function tryTransformPairFiles( transformHbcFile(newPath, tNewPath, layout, false), ]); if (!tOld || !tNew) continue; - return { - layout, - meta: { - v: HBC_TRANSFORM_VERSION, - hbcVersion: newVersion, - layout: compileLayoutToWire(layout), - }, - }; + return { layout, meta: transformMeta(version, layout) }; } return null; } diff --git a/src/utils/hermes-base.ts b/src/utils/hermes-base.ts index 60c5812..0a45d16 100644 --- a/src/utils/hermes-base.ts +++ b/src/utils/hermes-base.ts @@ -2,7 +2,7 @@ // // Hermes re-sorts its string table by identifier frequency on every compile, so // any source change renumbers most string IDs and the bytecode diff explodes -// (see hermes-base-bytecode-方案.md). Compiling with the previous HBC of the +// (see docs/hermes-base-verification.md). Compiling with the previous HBC of the // same app as `-base-bytecode` keeps old IDs stable and shrinks patches 5-30×. // This module resolves that base: gating by hermesc build, HBC version probe, // server lookup, a sha256-named local cache, download + verification, and the @@ -35,9 +35,14 @@ import { normalizeCachedObjectInstruction } from './hermes-cached-object'; import { type LiteralBuffers, LiteralResolver, - readLiteralBuffers, + literalBuffersFromBytes, } from './hermes-literals'; -import { auditRawHermesBytecode, readHermesSemanticData } from './hermes-raw'; +import { + auditRawHermesBytecode, + foldWidthSuffix, + hermesSemanticData, + isFunctionHeader, +} from './hermes-raw'; import { hermesTimeout } from './hermes-timeout'; import { t } from './i18n'; import { webFetch } from './runtime'; @@ -194,48 +199,32 @@ export function classifyHermesCommand(hermesCommand: string): { }; } const build = /^(\d{9})\./.exec(version); - if (build) { - const ok = Number(build[1]) >= MIN_HERMES_COMPILER_BUILD; - return { - allowed: ok, - kind: 'hermes-compiler', - version, - reason: ok - ? undefined - : `hermes-compiler ${version} predates the delta-mode fix`, - }; - } const commitly = /^0\.(\d+)\.\d+-commitly-(\d{8})/.exec(version); - if (commitly) { - const ok = + const stable = /^0\.(\d+)\.\d+$/.exec(version); + let ok: boolean; + if (build) { + ok = Number(build[1]) >= MIN_HERMES_COMPILER_BUILD; + } else if (commitly) { + ok = Number(commitly[1]) >= 14 && Number(commitly[2]) >= MIN_HERMES_COMPILER_COMMITLY_DATE; + } else if (stable) { + ok = Number(stable[1]) >= 14; + } else { return { - allowed: ok, - kind: 'hermes-compiler', - version, - reason: ok - ? undefined - : `hermes-compiler ${version} predates the delta-mode fix`, - }; - } - const stable = /^0\.(\d+)\.\d+$/.exec(version); - if (stable) { - const ok = Number(stable[1]) >= 14; - return { - allowed: ok, + allowed: false, kind: 'hermes-compiler', version, - reason: ok - ? undefined - : `hermes-compiler ${version} predates the delta-mode fix`, + reason: `unrecognized hermes-compiler version ${version}`, }; } return { - allowed: false, + allowed: ok, kind: 'hermes-compiler', version, - reason: `unrecognized hermes-compiler version ${version}`, + reason: ok + ? undefined + : `hermes-compiler ${version} predates the delta-mode fix`, }; } return { @@ -415,6 +404,9 @@ export async function extractBundleFromArchive( export const DEFAULT_CACHE_MAX_BYTES = 500 * 1024 * 1024; export const DEFAULT_CACHE_MAX_FILES = 20; +/** cache entries are named by the sha256 of their bundle */ +const isBundleHash = (name: string) => /^[0-9a-f]{64}$/.test(name); + export function cacheDir(): string { return process.env.PUSHY_CACHE_DIR || path.join(tempDir, 'cache'); } @@ -436,7 +428,7 @@ export function cacheLimits(overrideMaxMb?: number) { /** Returns the cached bundle path when present and its sha256 still matches. */ export async function cacheLookup(bundleHash: string): Promise { - if (!/^[0-9a-f]{64}$/.test(bundleHash)) return null; + if (!isBundleHash(bundleHash)) return null; const file = path.join(cacheDir(), bundleHash); if (!(await pathExists(file))) return null; if ((await sha256File(file).catch(() => '')) !== bundleHash) { @@ -482,7 +474,7 @@ export async function enforceCacheLimits( const staleTmpCutoff = Date.now() - STALE_CACHE_TMP_MS; for (const name of await fs.promises.readdir(dir)) { const isTmp = isCacheTmpName(name); - if (!isTmp && !/^[0-9a-f]{64}$/.test(name)) continue; + if (!isTmp && !isBundleHash(name)) continue; const stat = await fs.promises.stat(path.join(dir, name)).catch(() => null); if (!stat?.isFile()) continue; if (isTmp) { @@ -516,7 +508,7 @@ export async function cleanCache(): Promise { if (!(await pathExists(dir))) return 0; let bundles = 0; for (const name of await fs.promises.readdir(dir)) { - const isBundle = /^[0-9a-f]{64}$/.test(name); + const isBundle = isBundleHash(name); if (!isBundle && !isCacheTmpName(name)) continue; await remove(path.join(dir, name)); if (isBundle) bundles += 1; @@ -534,7 +526,7 @@ export async function cacheStats(): Promise<{ let bytes = 0; if (await pathExists(dir)) { for (const name of await fs.promises.readdir(dir)) { - if (!/^[0-9a-f]{64}$/.test(name)) continue; + if (!isBundleHash(name)) continue; const stat = await fs.promises .stat(path.join(dir, name)) .catch(() => null); @@ -706,17 +698,18 @@ export async function fetchBaseBundle( } return { bundle, bundleHash }; }; - const fromFullResponse = async ( - response: Response, - ): Promise => { - await saveResponse(response, archive); - return { - ...verified(await extractBundleFromArchive(archive)), - transport: 'full', - }; - }; + const fromArchive = async (): Promise => ({ + ...verified(await extractBundleFromArchive(archive)), + transport: 'full', + }); const skip = (reason: string) => log(t('hermesBaseRangeFallback', { reason })); + // a 200 to a Range request already carries the whole archive + const rangeIgnored = async (response: Response) => { + skip('server ignores Range'); + await saveResponse(response, archive); + return fromArchive(); + }; // harmony .app nests the bundle in a second zip, so a location inside the // outer archive is never reported for it @@ -745,8 +738,7 @@ export async function fetchBaseBundle( }; } catch (error: any) { if (error instanceof RangeUnsupportedError) { - skip('server ignores Range'); - return fromFullResponse(error.response); + return rangeIgnored(error.response); } // not a transport failure: the directory read below settles it if (!(error instanceof StaleHeadError)) { @@ -759,8 +751,7 @@ export async function fetchBaseBundle( try { const remote = await openRemoteZip(record.url); if (remote.kind === 'full') { - skip('server ignores Range'); - return fromFullResponse(remote.response); + return rangeIgnored(remote.response); } const bundle = await readRemoteZipEntry( remote.zipFile, @@ -786,8 +777,7 @@ export async function fetchBaseBundle( }; } catch (error: any) { if (error instanceof RangeUnsupportedError) { - skip('server ignores Range'); - return fromFullResponse(error.response); + return rangeIgnored(error.response); } // the real entry was read and is not the recorded one: final if (error instanceof PermanentBaseError) throw error; @@ -796,15 +786,11 @@ export async function fetchBaseBundle( } await downloadToFile(record.url, archive); - return { - ...verified(await extractBundleFromArchive(archive)), - transport: 'full', - }; + return fromArchive(); } export interface ResolveHermesBaseParams { option: HermesBaseOption; - hermesCommand: string; bytecodeVersion: number; appId?: string; cacheMaxMb?: number; @@ -879,12 +865,6 @@ export async function resolveHermesBase( ); return null; } - const source: HermesBaseSelection['source'] = - record?.artifactType && record.artifactType !== 'ppk' - ? 'native-package' - : record?.bytecodeVersion == null - ? 'latest-version' - : 'download'; if (!record) { log(t('hermesBaseNone', { reason: 'no published version yet' })); return null; @@ -903,20 +883,20 @@ export async function resolveHermesBase( // 1. cache by bundleHash if (record.bundleHash) { const hit = await cacheLookup(record.bundleHash); - if (hit && record.bytecodeVersion == null) { - // the server does not know the HBC version (legacy version, native - // package): the download path checks it, the cache path must too - const version = await readFileHbcVersion(hit); - if (version !== bytecodeVersion) { - log( - t('hermesBaseNone', { - reason: `cached base is HBC ${version ?? 'n/a'}, need ${bytecodeVersion}`, - }), - ); - return null; - } - } if (hit) { + if (record.bytecodeVersion == null) { + // the server does not know the HBC version (legacy version, native + // package): the download path checks it, the cache path must too + const version = await readFileHbcVersion(hit); + if (version !== bytecodeVersion) { + log( + t('hermesBaseNone', { + reason: `cached base is HBC ${version ?? 'n/a'}, need ${bytecodeVersion}`, + }), + ); + return null; + } + } log( t('hermesBaseUsing', { source: `cache ${record.bundleHash.slice(0, 12)}${record.versionId == null ? '' : ` (version ${record.versionId})`}`, @@ -934,17 +914,20 @@ export async function resolveHermesBase( } } // 2. download, extract, verify, cache — transport errors get a second try + const artifactType: BaseArtifactType = ['ppk', 'apk', 'ipa', 'app'].includes( + record.artifactType ?? '', + ) + ? (record.artifactType as BaseArtifactType) + : 'ppk'; + const source: HermesBaseSelection['source'] = + record.artifactType && record.artifactType !== 'ppk' + ? 'native-package' + : record.bytecodeVersion == null + ? 'latest-version' + : 'download'; for (let attempt = 1; attempt <= 2; attempt++) { const dir = tmpDir(); await ensureDir(dir); - const artifactType: BaseArtifactType = [ - 'ppk', - 'apk', - 'ipa', - 'app', - ].includes(record.artifactType ?? '') - ? (record.artifactType as BaseArtifactType) - : 'ppk'; const archive = path.join( dir, `base-${process.pid}-${Date.now()}-${attempt}.${artifactType}`, @@ -1274,15 +1257,6 @@ function isLetter(code: number): boolean { } /** strip one operand-width suffix (LongIndex, Long, Short) off an opcode */ -function foldWidthSuffix(opcode: string): string { - for (const suffix of ['LongIndex', 'Long', 'Short']) { - if (opcode.length > suffix.length && opcode.endsWith(suffix)) { - return opcode.slice(0, -suffix.length); - } - } - return opcode; -} - /** Split a readable stream into lines without readline (Bun's readline async iterator misbehaves when pulled manually). */ async function* streamLines( stream: NodeJS.ReadableStream, @@ -1319,7 +1293,7 @@ export interface HermesEquivalenceResult { functions: number; /** * `instruction`: literals decoded from the binary buffers at each - * New*WithBuffer instruction (HBC v87–96 and v98); `buffer`: the dumped + * New*WithBuffer instruction (HBC v87–96, v98 and v99); `buffer`: the dumped * buffers compared as a whole (unknown layout) */ literals?: 'instruction' | 'buffer'; @@ -1356,10 +1330,6 @@ const DUMP_STDERR_KEEP = 4 * 1024; /** a diagnostic line is cut to this many characters */ const DETAIL_LINE_MAX = 120; -function isFunctionHeader(line: string): boolean { - return /^(?:Function|NCFunction|Constructor) null), - readLiteralBuffers(plain).catch(() => null), - readHermesSemanticData(withBase).catch(() => null), - readHermesSemanticData(plain).catch(() => null), - ]); + // Each file is read once; its literal buffers and semantic data both come + // from those bytes. Only when both sides' buffers can be read do the + // instructions compare decoded literals; otherwise both fall back to the + // dumped buffers as a whole, so the two sides always normalize the same way. + const [bytesA, bytesB] = await Promise.all( + [withBase, plain].map((file) => + fs.promises.readFile(file).catch(() => null), + ), + ); + const orNull = (read: () => T): T | null => { + try { + return read(); + } catch { + return null; + } + }; + const buffersA = bytesA && orNull(() => literalBuffersFromBytes(bytesA)); + const buffersB = bytesB && orNull(() => literalBuffersFromBytes(bytesB)); + const dataA = bytesA && orNull(() => hermesSemanticData(bytesA)); + const dataB = bytesB && orNull(() => hermesSemanticData(bytesB)); const binary = buffersA && buffersB && buffersA.version === buffersB.version; const literals = binary ? 'instruction' : 'buffer'; const a = new DumpReader( diff --git a/src/utils/hermes-literals.ts b/src/utils/hermes-literals.ts index 6bbb8cf..bfec1bd 100644 --- a/src/utils/hermes-literals.ts +++ b/src/utils/hermes-literals.ts @@ -24,14 +24,14 @@ * - HBC v87–96 (`split`): arrayBuffer, objKeyBuffer, objValueBuffer. * `NewArrayWithBuffer dst, sizeHint, count, arrayOffset`; * `NewObjectWithBuffer dst, sizeHint, count, keyOffset, valueOffset`. - * - HBC v98 (`shaped`): literalValueBuffer (array elements and object values + * - HBC v98 and v99 (`shaped`): literalValueBuffer (array elements and object values * alike), objKeyBuffer, objShapeTable (8-byte entries: keyOffset u32, * numProps u32). `NewArrayWithBuffer dst, sizeHint, count, valueOffset`; * `NewObjectWithBuffer dst, shapeIndex, valueOffset` and * `NewObjectWithBufferAndParent dst, parent, shapeIndex, valueOffset`. */ -import { open as openFile } from 'node:fs/promises'; -import { HBC_HEADER_SIZE, resolveHbcSections } from './hbcTransform'; +import { readFile } from 'node:fs/promises'; +import { resolveHbcSections } from './hbcTransform'; const TAG_TYPE_MASK = 0x70; const TAG_LENGTH_CONTINUES = 0x80; @@ -75,57 +75,38 @@ export type LiteralBuffers = const SHAPE_ENTRY_SIZE = 8; /** - * Read only the literal buffers of an HBC file (header + three byte ranges, - * never the whole file). Null for a file whose layout is unknown: the caller - * then falls back to comparing the dumped buffers as a whole. + * The literal buffers of an HBC file's bytes, as views into them. Null for a + * file whose layout is unknown: the caller then falls back to comparing the + * dumped buffers as a whole. */ +export function literalBuffersFromBytes(bytes: Buffer): LiteralBuffers | null { + const resolved = resolveHbcSections(bytes, bytes.length); + if (!resolved) return null; + // resolveHbcSections keeps every section inside the file + const views = (names: string[]) => { + const ranges = names.map((name) => resolved.sections.get(name)); + return ranges.every(Boolean) + ? ranges.map((r) => bytes.subarray(r!.start, r!.start + r!.size)) + : null; + }; + const { version } = resolved; + if (resolved.sections.has('objShapeTable')) { + const bufs = views(['literalValueBuffer', 'objKeyBuffer', 'objShapeTable']); + if (!bufs) return null; + const [values, objectKeys, shapes] = bufs; + return { layout: 'shaped', version, values, objectKeys, shapes }; + } + const bufs = views(['arrayBuffer', 'objKeyBuffer', 'objValueBuffer']); + if (!bufs) return null; + const [array, objectKeys, objectValues] = bufs; + return { layout: 'split', version, array, objectKeys, objectValues }; +} + +/** {@link literalBuffersFromBytes} for a file. */ export async function readLiteralBuffers( file: string, ): Promise { - const handle = await openFile(file, 'r'); - try { - const { size } = await handle.stat(); - if (size < HBC_HEADER_SIZE) return null; - const header = Buffer.alloc(HBC_HEADER_SIZE); - const read = await handle.read(header, 0, HBC_HEADER_SIZE, 0); - if (read.bytesRead !== HBC_HEADER_SIZE) return null; - const resolved = resolveHbcSections(header, size); - if (!resolved) return null; - const readRanges = async (names: string[]) => { - const ranges = names.map((name) => resolved.sections.get(name)); - if (ranges.some((r) => !r)) return null; - const bufs = await Promise.all( - ranges.map(async (r) => { - const buf = Buffer.alloc(r!.size); - if (r!.size === 0) return buf; - const { bytesRead } = await handle.read(buf, 0, r!.size, r!.start); - return bytesRead === r!.size ? buf : null; - }), - ); - return bufs.some((b) => !b) ? null : (bufs as Buffer[]); - }; - const { version } = resolved; - if (resolved.sections.has('objShapeTable')) { - const bufs = await readRanges([ - 'literalValueBuffer', - 'objKeyBuffer', - 'objShapeTable', - ]); - if (!bufs) return null; - const [values, objectKeys, shapes] = bufs; - return { layout: 'shaped', version, values, objectKeys, shapes }; - } - const bufs = await readRanges([ - 'arrayBuffer', - 'objKeyBuffer', - 'objValueBuffer', - ]); - if (!bufs) return null; - const [array, objectKeys, objectValues] = bufs; - return { layout: 'split', version, array, objectKeys, objectValues }; - } finally { - await handle.close(); - } + return literalBuffersFromBytes(await readFile(file)); } /** diff --git a/src/utils/hermes-raw.ts b/src/utils/hermes-raw.ts index 7832adc..26cb59e 100644 --- a/src/utils/hermes-raw.ts +++ b/src/utils/hermes-raw.ts @@ -11,7 +11,7 @@ import { readFile } from 'node:fs/promises'; import { StringDecoder } from 'node:string_decoder'; import { spawn } from 'child_process'; -import { pickLayout, resolveHbcSections } from './hbcTransform'; +import { resolveHbcSections } from './hbcTransform'; import { type LiteralBuffers, LiteralResolver } from './hermes-literals'; export class UnverifiableHermesBytecode extends Error {} @@ -58,7 +58,11 @@ export interface HermesSemanticData { export async function readHermesSemanticData( file: string, ): Promise { - const bytes = await readFile(file); + return hermesSemanticData(await readFile(file)); +} + +/** {@link readHermesSemanticData} for HBC bytes already in memory. */ +export function hermesSemanticData(bytes: Buffer): HermesSemanticData { const resolved = requireValue( resolveHbcSections(bytes, bytes.length), 'unsupported or unreadable HBC layout', @@ -74,7 +78,7 @@ export async function readHermesSemanticData( ) { throw new UnverifiableHermesBytecode('unsupported semantic HBC version'); } - const layout = requireValue(pickLayout(bytes), 'unknown HBC layout'); + const { layout } = resolved; const section = (name: string) => { const r = requireValue(resolved.sections.get(name), `missing ${name}`); return checkedSlice(bytes, r.start, r.size); @@ -271,8 +275,19 @@ interface RawInstruction { } /** Remove encoding-width suffixes; operand values remain part of the audit. */ -function foldWidth(opcode: string): string { - return opcode.replace(/(?:LongIndex|Long|Short)$/, ''); +/** The opcode without its operand-width suffix (`GetByIdLong` → `GetById`). */ +export function foldWidthSuffix(opcode: string): string { + for (const suffix of ['LongIndex', 'Long', 'Short']) { + if (opcode.length > suffix.length && opcode.endsWith(suffix)) { + return opcode.slice(0, -suffix.length); + } + } + return opcode; +} + +/** The first line of a function in either dump. */ +export function isFunctionHeader(line: string): boolean { + return /^(?:Function|NCFunction|Constructor) o.value); const refs = STRING_OPERANDS[op] ?? []; const operands: unknown[] = inst.operands.map((operand, i) => { @@ -561,38 +576,22 @@ export async function auditRawHermesBytecode( const functions = async function* (side: number): AsyncGenerator { let lines: string[] | null = null; let index = 0; + const normalize = (body: string[]) => + normalizeRawHermesFunction(body, data[side], index++, buffers[side]); for await (const line of linesOf(children[side].stdout!)) { - if (/^(?:Function|NCFunction|Constructor) { removeSync(dir); }); const common = { - hermesCommand: 'hermesc', bytecodeVersion: 98, log: (m: string) => logs.push(m), }; @@ -431,8 +430,6 @@ describe('resolveHermesBase over HTTP Range', () => { ))!; const selection = await resolveHermesBase({ option: 'auto', - hermesCommand: - '/x/node_modules/react-native/sdks/hermesc/osx-bin/hermesc', bytecodeVersion: 98, appId: '1', log: (m) => messages.push(m), @@ -491,8 +488,6 @@ describe('resolveHermesBase over HTTP Range', () => { try { const selection = await resolveHermesBase({ option: 'auto', - hermesCommand: - '/x/node_modules/react-native/sdks/hermesc/osx-bin/hermesc', bytecodeVersion: 98, appId: '1', log: (m) => messages.push(m), @@ -533,8 +528,6 @@ describe('resolveHermesBase over HTTP Range', () => { }; const params = { option: 'auto', - hermesCommand: - '/x/node_modules/react-native/sdks/hermesc/osx-bin/hermesc', appId: '1', log: (m: string) => messages.push(m), fetchBase: async () => record,