Integrate #367: opt-in byte strings with compatible I/O

README.md

@@ -673,7 +673,7 @@ Lua-CSharp is designed with integration into .NET in mind, so there are several
 
 ### Character Encoding
 
-The character encoding used in Lua-CSharp is UTF-16. Since standard Lua assumes a single-byte character encoding, string behavior differs significantly.
+By default, Lua-CSharp uses UTF-16. Since standard Lua assumes a single-byte character encoding, string behavior differs significantly. See [String modes](docs/string-modes.md) for UTF-16 and byte-string behavior and C# conversions.
 
 For example, in regular Lua, the following code outputs `15`, but in Lua-CSharp, it outputs `5`.
 
@@ -682,7 +682,53 @@ local l = string.len("あいうえお")
 print(l)
 ```
 
-All string library functions handle strings as UTF-16.
+In the default mode, string library functions handle strings as UTF-16.
+
+#### Byte strings (opt-in)
+
+If you need byte-exact strings like the reference C implementation (for example `#"ế"` is `3`, `string.sub` cuts inside a UTF-8 sequence, and files are read and written as raw bytes), use `LuaStringMode.Bytes`. `LuaPlatform.StringMode` defaults to `LuaStringMode.Utf16`. `LuaPlatform.DefaultByteStrings` provides byte strings and byte-oriented file and standard I/O:
+
+```cs
+using Lua;
+using Lua.Platforms;
+using Lua.Standard;
+
+var state = LuaState.Create(LuaPlatform.DefaultByteStrings);
+state.OpenStandardLibraries();
+
+// Every char of a Lua string holds one byte, so convert .NET strings at the boundary.
+state.Environment["name"] = LuaByteString.FromString("Tiến Lên"); // UTF-8 bytes, #name == 11
+var results = await state.DoStringAsync("return #name, name:upper()");
+var upper = LuaByteString.ToString(results[1].Read<string>());   // back to a .NET string
+```
+
+In this mode `string.upper`, `string.lower` and the pattern classes (`%a`, `%w`, `%l`, ...) follow the C locale (ASCII only), `string.char` rejects values above 255, and UTF-8 source files are loaded byte by byte (a UTF-8 BOM is skipped). The default stays UTF-16.
+
+Choose one representation for a state and configure it before executing scripts. Setting `StringMode = LuaStringMode.Bytes` on an otherwise unchanged platform does not switch its streams to raw bytes. Use matching modes for the platform and its streams. When configuring a file root, keep that root as a normal .NET path:
+
+```cs
+var platform = LuaPlatform.DefaultByteStrings with
+{
+    FileSystem = new Lua.IO.FileSystem(scriptsDirectory, LuaStringMode.Bytes)
+};
+using var byteState = LuaState.Create(platform);
+byteState.OpenStandardLibraries();
+
+// C# source and filename APIs accept normal .NET text.
+var values = await byteState.DoStringAsync("return 'こんにちは'");
+var greeting = LuaByteString.ToString(values[0].Read<string>());
+
+// Raw packets can contain NUL, invalid UTF-8 and every value from 0 to 255.
+byteState.Environment["packet"] = LuaByteString.FromBytes(new byte[] { 0xFF, 0, 0x80 });
+var packet = await byteState.DoStringAsync("return packet:reverse()");
+byte[] reversed = LuaByteString.ToBytes(packet[0].Read<string>());
+```
+
+Use `FromString`/`ToString` for string values and `FromBytes`/`ToBytes` for binary data. This also applies to non-ASCII table keys and the arguments/results of C# functions and generated bindings; value interop does not convert strings automatically. C# source APIs (`DoStringAsync`, `Load` with character source or a `LuaModule`) accept normal .NET text and encode it as UTF-8 in byte mode. Chunk names and paths passed to `LoadFileAsync`/`DoFileAsync` are also .NET text. Byte source passed to `Load`/`ExecuteAsync` and source read from files are preserved directly; their explicit chunk names are .NET text. Lua's `load`, `loadfile`, `dofile` and `require` continue to use Lua byte strings without re-encoding their source.
+
+Paths and names passed to `ILuaFileSystem`, `ILuaFileLoader` and `ILuaModuleLoader` are normal .NET strings in both modes. Text modules are treated as .NET text and encoded to UTF-8 in byte mode; byte modules remain bytes. Unity text assets use the same conversion. Custom file systems and standard-I/O providers must implement the selected stream representation themselves; use `ILuaStream.CreateFromStream(stream, mode, LuaStringMode.Bytes)` for raw streams.
+
+Files, temporary files and redirected standard I/O preserve bytes, including a BOM read as data. Raw line reads stop at LF and retain CR; `*L` also retains the LF. Invalid UTF-8 is replaced when decoding with `ToString`, so keep binary data in byte strings or byte arrays. Passing unconverted .NET characters above U+00FF to raw output writes `?` for those characters.
 
 ### Garbage Collection
 

docs/string-modes.md

@@ -0,0 +1,124 @@
+# String modes
+
+## Choose a mode
+
+`LuaStringMode.Utf16` is the default and uses ordinary .NET UTF-16 strings. Use `LuaStringMode.Bytes` for Lua byte semantics or binary data; each C# `char` then holds one byte (0–255).
+
+```csharp
+using Lua;
+using Lua.Platforms;
+using Lua.Standard;
+
+using var textState = LuaState.Create(); // Utf16
+using var byteState = LuaState.Create(LuaPlatform.DefaultByteStrings);
+textState.OpenStandardLibraries();
+byteState.OpenStandardLibraries();
+```
+
+Keep the mode fixed for a state's lifetime. Changing its platform does not convert existing strings, table keys or compiled chunks. Values shared between states with different modes need explicit conversion.
+
+## Lua string operations
+
+Byte mode follows Lua 5.2's byte-string model: strings can contain arbitrary bytes, and lengths, substring positions and pattern matching operate on bytes. Character classes and case conversion follow ASCII C-locale rules.
+
+For UTF-8 source text:
+
+| Operation | `Utf16` | `Bytes` |
+| --- | --- | --- |
+| `#"ế"` | 1 | 3 |
+| `#"😀"` | 2 | 4 |
+| `string.len`, `string.sub`, `string.byte` | UTF-16 code units | Bytes |
+| Case conversion and pattern classes | .NET character rules | ASCII C-locale rules |
+
+For example, this script returns different results in each mode:
+
+```lua
+local s = "é"
+return #s, string.byte(s, 1), s:match("^%a+$") ~= nil
+-- Utf16: 1, 233, true
+-- Bytes: 2, 195, false
+```
+
+Lengths and slices do not count user-perceived characters. UTF-16 slicing can split a surrogate pair; byte slicing can split a UTF-8 sequence. Byte positions from Lua cannot be reused as indices into decoded .NET text. Use .NET text-element APIs on decoded text for user-perceived-character editing.
+
+In byte mode, `string.char` accepts only 0–255. Non-ASCII letters remain unchanged by `string.upper`/`string.lower` and do not match `%a`. Decode text before applying Unicode case conversion or classification in C#.
+
+Neither mode normalizes Unicode. For example, C# `"é"` and `"e\u0301"` remain different strings and table keys.
+
+This describes string semantics. Existing Lua-CSharp limitations still apply; for example, `string.dump` is unsupported. C# values still use `string` and require the conversions below, with matching stream configuration.
+
+## Passing values between C# and Lua
+
+UTF-16 values can use .NET strings directly. In byte mode, convert values according to their contents:
+
+| Conversion | API |
+| --- | --- |
+| .NET text → Lua byte string | `LuaByteString.FromString(text)` |
+| UTF-8 or binary bytes → Lua byte string | `LuaByteString.FromBytes(bytes)` |
+| Lua byte string → .NET text | `LuaByteString.ToString(value)` |
+| Lua byte string → raw bytes | `LuaByteString.ToBytes(value)` |
+
+C# UTF-8 literals (`u8`, C# 11+) can supply bytes directly. `FromString("é")` and `FromBytes("é"u8)` produce the same bytes, `C3 A9`.
+
+```csharp
+byteState.Environment["greeting"] = LuaByteString.FromBytes("こんにちは"u8);
+var values = await byteState.DoStringAsync("return greeting");
+string greeting = LuaByteString.ToString(values[0].Read<string>());
+
+byteState.Environment["packet"] = LuaByteString.FromBytes(new byte[] { 0xFF, 0, 0x80 });
+var packets = await byteState.DoStringAsync("return packet");
+byte[] packet = LuaByteString.ToBytes(packets[0].Read<string>());
+```
+
+### Convert text once
+
+Both representations use C# `string`; the type does not identify the representation. A string read from a byte-mode state is already a byte string. Calling `FromString` on it again changes non-ASCII data.
+
+Value conversion is explicit for table keys, C# function arguments/results and generated bindings too. ASCII-only testing can hide missing conversions: directly assigning .NET `"é"` in byte mode gives one byte, `E9`, rather than the UTF-8 bytes `C3 A9`.
+
+### Preserve binary data
+
+Use `FromBytes`/`ToBytes` for NUL bytes, invalid UTF-8 and other binary data. `FromString` replaces unpaired UTF-16 surrogates; `ToString` replaces invalid UTF-8 sequences with U+FFFD. These replacements lose the original data.
+
+Valid byte strings contain only chars up to U+00FF. `ToBytes` and raw output replace higher chars with `?`.
+
+## Loading source and naming files
+
+Source-loading APIs handle conversion for you:
+
+```csharp
+// .NET text: encoded as UTF-8 in byte mode.
+var textChunk = byteState.Load("return 'é'", "example");
+
+// Already encoded UTF-8: pass bytes directly.
+var utf8Chunk = byteState.Load("return 'é'"u8, "example");
+```
+
+- `DoStringAsync`, character-source `Load` and text `LuaModule` sources accept normal .NET text. Applying `FromString` first would encode the source twice in byte mode.
+- In byte mode, byte-source `Load`, byte modules and source files preserve source bytes, except for an initial UTF-8 BOM. Lua's `load` uses strings in the state's representation without encoding them again.
+- C# chunk names and paths are normal .NET strings in both modes, including names passed to `ILuaFileSystem`, `ILuaFileLoader` and `ILuaModuleLoader`.
+
+## Configuring file and standard I/O
+
+`LuaPlatform.DefaultByteStrings` configures matching byte-mode file and standard I/O. Changing `StringMode` alone does not reconfigure streams. Custom providers must use the same representation as the state; wrap raw streams with `ILuaStream.CreateFromStream(stream, openMode, LuaStringMode.Bytes)`.
+
+For Unity, use `new Lua.Unity.UnityStandardIO(LuaStringMode.Bytes)` with a byte-mode state. It decodes UTF-8 for display through `Debug.Log`/`Debug.LogError`; use raw streams to preserve binary output. A supplied input stream must also use byte strings.
+
+A file root remains a normal .NET path:
+
+```csharp
+var platform = LuaPlatform.DefaultByteStrings with
+{
+    FileSystem = new Lua.IO.FileSystem("scripts", LuaStringMode.Bytes)
+};
+```
+
+| I/O behavior | `Utf16` | `Bytes` |
+| --- | --- | --- |
+| File and standard stream contents | UTF-8 text decoded to/from UTF-16 | Raw bytes |
+| `file:read(n)` | Up to `n` UTF-16 code units | Up to `n` bytes |
+| `*l` reading `hello\r\n` | `hello` | `hello\r` |
+| `*L` reading `hello\r\n` | `hello\r\n` | `hello\r\n` |
+| Initial UTF-8 BOM read as data | Skipped | Preserved |
+
+The `b` flag in `io.open(path, "rb")` does not select byte-string mode. Built-in streams use their configured `LuaStringMode` for both `"r"` and `"rb"`. Byte-mode line reads stop at LF; remove a trailing CR explicitly when the application expects a line without either newline character.

sandbox/Benchmark/ByteStringIoBenchmark.cs

@@ -0,0 +1,50 @@
+using BenchmarkDotNet.Attributes;
+using Lua;
+using Lua.IO;
+
+[Config(typeof(BenchmarkConfig))]
+public class ByteStringIoBenchmark
+{
+    [Params(LuaStringMode.Utf16, LuaStringMode.Bytes)]
+    public LuaStringMode StringMode { get; set; }
+
+    [Params(1, 4096)]
+    public int Size { get; set; }
+
+    MemoryStream input = default!;
+    MemoryStream output = default!;
+    LuaStream reader = default!;
+    LuaStream writer = default!;
+    string text = default!;
+
+    [GlobalSetup]
+    public void Setup()
+    {
+        text = new string('a', Size);
+        input = new MemoryStream(Enumerable.Repeat((byte)'a', Size).ToArray());
+        output = new MemoryStream(Size);
+        reader = new LuaStream(LuaFileOpenMode.Read, input, StringMode);
+        writer = new LuaStream(LuaFileOpenMode.Write, output, StringMode);
+    }
+
+    [GlobalCleanup]
+    public void Cleanup()
+    {
+        reader.Dispose();
+        writer.Dispose();
+    }
+
+    [Benchmark]
+    public string? Read()
+    {
+        reader.Seek(SeekOrigin.Begin, 0);
+        return reader.ReadAsync(Size, default).GetAwaiter().GetResult();
+    }
+
+    [Benchmark]
+    public void Write()
+    {
+        output.Position = 0;
+        writer.WriteAsync(text.AsMemory(), default).GetAwaiter().GetResult();
+    }
+}

sandbox/Benchmark/Utf8CharConversionBenchmark.cs

@@ -0,0 +1,37 @@
+using System.Text;
+using BenchmarkDotNet.Attributes;
+using Lua;
+
+[MemoryDiagnoser]
+[ShortRunJob]
+public class Utf8CharConversionBenchmark
+{
+    [Params("Ascii", "SparseUnicode", "Japanese", "Mixed", "InvalidUtf16")]
+    public string Corpus { get; set; } = "Ascii";
+
+    [Params(4096)]
+    public int Length { get; set; }
+
+    string source = default!;
+
+    [GlobalSetup]
+    public void Setup()
+    {
+        var unit = Corpus switch
+        {
+            "Ascii" => "hello world 1234 ",
+            "SparseUnicode" => new string('a', Length - 1) + "ế",
+            "Japanese" => "こんにちは世界",
+            "Mixed" => "aế😀漢字 ",
+            "InvalidUtf16" => "a\uD800b\uDC00c",
+            _ => throw new ArgumentOutOfRangeException(nameof(Corpus)),
+        };
+        source = string.Concat(Enumerable.Repeat(unit, Length / unit.Length + 1))[..Length];
+    }
+
+    [Benchmark(Baseline = true)]
+    public string ViaBytes() => LuaByteString.FromBytes(Encoding.UTF8.GetBytes(source));
+
+    [Benchmark]
+    public string DirectChars() => LuaByteString.FromString(source);
+}

src/Lua.Unity/Assets/Lua.Unity/Runtime/LuaThreadAssetExtensions.cs

@@ -21,10 +21,7 @@ public static ValueTask<int> ExecuteAsync(
             }
 
             var module = luaAssetBase.GetModule(name);
-            var closure =
-                module.Type == LuaModuleType.Bytes
-                    ? state.Load(module.ReadBytes(), module.Name)
-                    : state.Load(module.ReadText(), module.Name);
+            var closure = state.Load(module);
             return state.ExecuteAsync(closure, buffer, cancellationToken);
         }
 
@@ -35,7 +32,13 @@ public static ValueTask<LuaValue[]> ExecuteAsync(
             CancellationToken cancellationToken = default
         )
         {
-            return state.ExecuteAsync(luaAssetBase, name, cancellationToken);
+            if (luaAssetBase == null)
+            {
+                throw new ArgumentNullException(nameof(luaAssetBase));
+            }
+
+            var module = luaAssetBase.GetModule(name);
+            return state.ExecuteAsync(state.Load(module), cancellationToken);
         }
     }
 }

src/Lua.Unity/Assets/Lua.Unity/Runtime/UnityStandardIO.cs

@@ -9,20 +9,18 @@ namespace Lua.Unity
     public class UnityStandardIO : ILuaStandardIO
     {
         public UnityStandardIO(ILuaStream input = null)
+            : this(LuaStringMode.Utf16, input) { }
+
+        public UnityStandardIO(LuaStringMode stringMode, ILuaStream input = null)
         {
-            if (input != null)
-            {
-                Input = input;
-            }
-            else
-            {
-                Input = new DummyInputStream();
-            }
+            Input = input ?? new DummyInputStream();
+            Output = new DebugLogStream(false, stringMode);
+            Error = new DebugLogStream(true, stringMode);
         }
 
         public ILuaStream Input { get; }
-        public ILuaStream Output { get; } = new DebugLogStream(false);
-        public ILuaStream Error { get; } = new DebugLogStream(true);
+        public ILuaStream Output { get; }
+        public ILuaStream Error { get; }
     }
 
     internal class DummyInputStream : ILuaStream
@@ -36,15 +34,20 @@ public void Dispose() { }
     public class DebugLogStream : ILuaStream
     {
         public DebugLogStream(bool isError = false)
+            : this(isError, LuaStringMode.Utf16) { }
+
+        public DebugLogStream(bool isError, LuaStringMode stringMode)
         {
             IsError = isError;
+            this.stringMode = stringMode;
         }
 
         public bool IsError { get; } = false;
         public bool IsOpen { get; } = true;
         public LuaFileOpenMode Mode => LuaFileOpenMode.Write;
 
         private readonly StringBuilder stringBuilder = new();
+        private readonly LuaStringMode stringMode;
 
         ValueTask ILuaStream.WriteAsync(
             ReadOnlyMemory<char> content,
@@ -60,6 +63,8 @@ ValueTask ILuaStream.FlushAsync(CancellationToken cancellationToken)
             if (stringBuilder.Length > 0)
             {
                 var message = stringBuilder.ToString();
+                if (stringMode == LuaStringMode.Bytes)
+                    message = LuaByteString.ToString(message);
                 if (IsError)
                     UnityEngine.Debug.LogError(message);
                 else

src/Lua/CodeAnalysis/Compilation/BomUtility.cs

@@ -12,6 +12,11 @@ static class BomUtility
 
     static ReadOnlySpan<byte> BomUtf32Little => [0xFF, 0xFE, 0x00, 0x00];
 
+    public static ReadOnlySpan<byte> SkipUtf8Bom(ReadOnlySpan<byte> text)
+    {
+        return text.StartsWith(BomUtf8) ? text.Slice(BomUtf8.Length) : text;
+    }
+
     /// <summary>
     ///  Removes the BOM from the beginning of the text and returns the encoding.
     ///  Supported encodings are UTF-8, UTF-16 (little and big endian), and UTF-32 (little endian).

src/Lua/Exceptions.cs

@@ -2,6 +2,7 @@
 using Lua.CodeAnalysis;
 using Lua.CodeAnalysis.Syntax;
 using Lua.Internal;
+using Lua.Platforms;
 using Lua.Runtime;
 
 namespace Lua;
@@ -490,9 +491,13 @@ internal void Forget()
         State = null;
     }
 
-    internal string MinimalMessage()
+    internal string MinimalMessage(LuaPlatform platform)
     {
-        var message = InnerException?.ToString() ?? ErrorObject.ToString();
+        var message = InnerException is { } inner
+            ? platform.StringMode == LuaStringMode.Bytes
+                ? platform.GetErrorMessage(inner)
+                : inner.ToString()
+            : ErrorObject.ToString();
         if (level <= 0)
         {
             return message;