[billythekidz]

Adds an opt-in mode in which every char of a Lua string holds one byte (0-255), so string behavior matches the reference C implementation for UTF-8 data. The default (UTF-16) is unchanged. The README documents the UTF-16 representation as intentional, so nothing changes unless a host sets ByteStrings = true.

Motivation: with UTF-8 text from files, string.len("あいうえお") is 5 instead of 15, #, sub, byte, find and format("%5s") count UTF-16 code units, and string.upper / %a change non-ASCII characters. Scripts written for C Lua (or LuaJIT / xLua) that handle UTF-8 as bytes then behave differently. Hosts can work around it by converting to Latin-1 themselves, but I measured that this still differs (pattern classes, file contents on disk, file names, stdout), see "Verification".

What changes with ByteStrings = true

• LuaStream / Utf8Reader: bytes map 1:1 to chars on read and write. The BOM is not skipped on reads (io.read, io.lines). io.read("*n") skips only ASCII whitespace.
• LuaState.Load(ReadOnlySpan<byte>): only a UTF-8 BOM is skipped, the rest is kept byte by byte (so LoadFile, dofile and require keep UTF-8 literals as bytes).
• MatchState: %a %c %d %g %l %p %s %u %w %x use the C locale (ASCII only).
• string.upper / lower: ASCII only. string.char rejects values above 255. string.format("%c") takes the low 8 bits.
• File names, os.getenv, os.tmpname and the text of I/O and OS errors cross the OS boundary as UTF-8: Lua strings are decoded to .NET strings on the way in and encoded on the way out. Without this a non-ASCII file name would be created with mojibake.

API

• LuaPlatform.ByteStrings (init property, default false) and LuaPlatform.DefaultByteStrings.
• Optional byteStrings parameter on FileSystem, ConsoleStandardIO and LuaStream, and an ILuaStream.CreateFromStream overload.
• Lua.LuaByteString helper (FromString, ToString, FromBytes, ToBytes) and an internal LuaByteEncoding.
• Binary compatibility: FileSystem(string?), LuaStream(mode, stream) and ConsoleStandardIO() gained an optional parameter. That is source compatible but a binary break for already compiled callers. I can add explicit overloads instead if you prefer.

Verification

• dotnet test tests/Lua.Tests/Lua.Tests.csproj -c Release --filter "TestCategory!=ExpectedFailure": 642 passed (610 on main + 32 new), 0 failed. Lua.SourceGenerator.Tests: 69 passed.
• Expected values in ByteStringTests come from Lua 5.2.4, and the test scripts also run as is on Lua 5.2.4.
• Differential run against a Lua 5.2.4 build (about 100 checks, 12+ files written to disk compared with cmp, directory listing, stdout and stdin compared byte by byte): identical in byte mode. The same probe on main with a host Latin-1 workaround still differs on many checks, 11 of the 12 files and stdout.
• Official conformance files strings.lua, literals.lua, sort.lua and errors.lua pass in byte mode.
• Mutation check: reverting any of nine parts of the change (including the path and environment conversions) makes at least one new test fail.
• dotnet csharpier check . is clean. src/Lua builds for net10.0, net8.0, net6.0 and netstandard2.1.

Not covered

• Only run on macOS (arm64) with the .NET 8 runtime. Not run on Unity (Mono / IL2CPP) or on a netstandard2.1 runtime, only compiled.
• Custom ILuaFileSystem / ILuaStandardIO implementations must read and write bytes themselves.
• string.dump is still unsupported. UTF-16/32 BOM files are treated as plain bytes in this mode (as in C Lua).
• Characters above U+00FF in a string handed to a byte-mode state become ? when written; use LuaByteString.FromString to pass .NET text in.

Related

Unrelated bugs found while building the differential tests are sent separately: #364 (os.rename), #365 (w+ on a missing file), #366 (gsub replacement strings, %f). This PR does not depend on them.

This is a larger design change than the other three. If you would rather discuss the approach first, or want it split (platform flag, I/O, pattern classes), I am happy to rework it.