Skip to content

fix(csv-parse): refresh trim state after BOM detection - #537

Open
nk-batman21 wants to merge 1 commit into
adaltas:masterfrom
nk-batman21:fix/parse-bom-trim-state
Open

nk-batman21 wants to merge 1 commit into
adaltas:masterfrom
nk-batman21:fix/parse-bom-trim-state

Conversation

@nk-batman21

Copy link
Copy Markdown

Fixes #536.

When a UTF-16LE BOM changes the encoding, trimming still uses buffers prepared for the initial encoding. For example, parsing Buffer.from("\ufeff a , b ", "utf16le") with { bom: true, trim: true } returns [["愀", "戀"]] instead of [["a", "b"]].

Refresh the encoding-dependent trim buffers and lookahead sizes after BOM detection, preserving the existing parser state and byte offsets. Add sync and byte-sized streaming regressions for spaces and nonbreaking spaces with trim and ltrim.

Validation on Node.js 24.18.0:

  • All eight new cases fail before the fix and pass afterward; 370 additional encoding, Unicode whitespace, header and quoted-field controls pass.
  • Linux workspace build and csv-parse TypeScript check pass. Parser tests: 626 passing, three pending, excluding the unchanged group_columns_by_name validation assertion.
  • Changed-file ESLint, Prettier and git diff --check pass.

The full test command still fails that existing assertion: it expects "an boolean" while the source now returns "a boolean" after #525. The same failure was verified with pristine production code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

csv-parse corrupts UTF-16LE fields when BOM detection is combined with trim

1 participant