Skip to content

Integrate #367: opt-in byte strings with compatible I/O - #369

Draft
akeit0 wants to merge 12 commits into
mainfrom
codex/byte-strings-pr367-fixes
Draft

akeit0 wants to merge 12 commits into
mainfrom
codex/byte-strings-pr367-fixes

Conversation

@akeit0

@akeit0 akeit0 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Adds opt-in byte strings through LuaPlatform.DefaultByteStrings and LuaStringMode.Bytes: Lua strings carry bytes, file and stream I/O preserve raw data, and string classes/case conversion follow ASCII C-locale rules. UTF-16 remains the default, and existing main public APIs remain compatible.

The three original commits from #367 retain their authorship and hashes. This branch also includes the main fixes from #368.

  • Keep C# source text, chunk names, paths and module-loader names as normal .NET strings. Convert at Lua boundaries; retain raw-byte loading for encoded source.
  • Expose the string representation through LuaStringMode. Preserve existing constructors and stream-factory signatures; add mode-aware overloads.
  • Use state.Load(module) for modules and share file-loading paths. Fix the recursive unbuffered Unity asset overload.
  • Add mode-aware UnityStandardIO/DebugLogStream overloads so byte-mode UTF-8 text displays correctly in Unity logs. Preserve the existing constructors and UTF-16 defaults.
  • Preserve raw I/O, CR handling, temporary-file mode and host-error conversion; avoid temporary encoding buffers in byte I/O.
  • Encode UTF-8 directly into the byte-string char representation. Reuse ASCII strings and source spans; use one pooled char buffer for non-ASCII source loading. Preserve BOM handling, binary-chunk rejection and .NET surrogate replacement behavior.
  • Run the 19 enabled official Lua scripts in both modes. Preserve the original files.lua bytes and byte-mode UTF-8 BOM checks; keep UTF-16 adaptations in that mode's harness. Vendored scripts are unchanged.

Validation on Windows, .NET 8.0.32:

  • 713 runtime tests passed, excluding the existing ExpectedFailure category, including 38 official-script runs. Existing unsupported reader loading, bytecode dumping and process execution remain excluded by the harness.
  • 69 source-generator tests and 76 focused tests against the netstandard2.1 assembly passed.
  • All four library targets build with zero warnings/errors; formatting and public API compatibility checks pass. A client compiled against main runs with the preserved constructors.
  • The converter oracle checks all UTF-16 code units, sampled supplementary scalars, malformed surrogates and ASCII-copy boundaries against Encoding.UTF8.
  • BenchmarkDotNet, .NET 8 x64, 4K chars: standalone conversion time falls 94% for ASCII, 12-13% for sparse Unicode/Japanese and 3% for mixed text. ASCII conversion allocates nothing; other inputs allocate about one-third less. Exploratory whole-script loading improves about 8% for ASCII, while ordinary Unicode gains are small. The reproducible converter benchmark is included.
  • Unity 6000.5.5f1 on Windows: 11/11 checks passed in Editor Mono and 11/11 in a native IL2CPP player; the native build succeeded and the player exited 0. Coverage includes real Lua assets, buffered/unbuffered execution, Unicode Resources modules and filenames, binary I/O, converter surrogate/ASCII-copy boundaries, and split UTF-8 writes to normal/error logs.

Draft pending maintainer approval. Use a merge commit when approved to preserve the contributor's commits and indirect PR merge status.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants