Skip to content

Sixel fixes for VT DOOM: decoder crash, dropped frames, aspect ratio, shared registers, GC pressure - #177

Merged
tomlm merged 3 commits into
mainfrom
fix-sixel-canvas-growth
Oct 14, 2026
Merged

tomlm merged 3 commits into
mainfrom
fix-sixel-canvas-growth

Conversation

@tomlm

@tomlm tomlm commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

VT DOOM (https://github.com/j4james/vtdoom) showed a black screen in Avalonia.Terminal and works in Windows Terminal. Diagnosed by running it headless through Porta.Pty into XTerm.NET and into a headless TerminalView, then at the user's real window size.

doom

What was wrong, in the order found

  1. Decoder crash → black screen. Growing the canvas for a taller band doubled the width too, overshot MaxSixelPixels, and the retry at "exactly the size asked for" took the new band's narrower width; copying the old rows ran off the end. The exception escaped Terminal.Write and ended the host's read loop, so ConPTY blocked and VT DOOM stopped on its first frame. Only the short dimension doubles now, and the retry never cuts into drawn pixels.
  2. Aspect ratio ignored → picture a fifth of its height. VT DOOM scales vertically with the raster attribute's Pan;Pad ("5;1). Each sixel row is now Pan/Pad pixels tall (capped at 16). P1 stays ignored, as in xterm.
  3. Private colour registers → a scatter of colours on black. VT DOOM sends its palette only when it changes (29 frames in 496). Registers are now shared by default, as on a VT340 and in Windows Terminal; CSI ? 1070 h still selects private registers, DECRQM reports it, and RIS now also resets the shared registers.
  4. Dropped frames → freezes, and a permanent freeze after a key press. The doubled canvas (4,096 wide for a 2,188-wide frame) was what got tested against the budget. At 307×75 cells such frames were discarded — data arriving, nothing drawn; once the menu appeared, every frame. The budget retry now shrinks to the drawn extent.
  5. GC pressure → input arriving in bursts. ~25 full collections a second while VT DOOM ran:
    • the decoder allocated a canvas per picture, per growth, plus an exact copy → now a per-terminal SixelCanvas with two reused buffers, in-place compaction, starting at the last picture's size;
    • SixelStrip held a row in one array past the large-object threshold, and a host capturing frames made every repaint copy it → strips are now blocks of 16 cells, allocated on paint and copied on write per block.

Measured: full collections 373 → a handful in 15 s; VT DOOM at 307×75 places every frame (150/151, the last one in flight) through Escape and Enter.

API change (3.0 is at rc001)

SixelStrip.Pixels is replaced by CopyPixelsTo(Span<byte> destination, int destinationStride, int columns), which hands out an ordinary contiguous BGRA image. The block storage behind it is internal; nothing outside depends on it. README's 3.0 section and drawing guide updated. Avalonia.Terminal's matching change is in its own PR.

Behaviour changes worth noting for 3.0: shared colour registers by default; the aspect ratio is honoured.

Tests

2,312 pass. New: the growth crash (fails without the fix), a picture inside the budget surviving the canvas doubling past it, aspect ratio (3:1, square == none, capped), shared/private/RIS colour registers, a reused canvas not showing the previous picture. Strip tests read pixels through CopyPixelsTo; the budget test works from real strip sizes.

🤖 Generated with Claude Code

tomlm and others added 2 commits October 14, 2026 13:46
… colour registers

VT DOOM showed a black screen. Three things, found by running it headless
through Porta.Pty into XTerm.NET:

- A wide picture growing downwards near the pixel budget crashed the
  decoder. Growing for a taller band doubled the width too, overshot the
  budget, and the retry at "exactly the size asked for" took the new
  band's narrower width, so copying the old rows ran off the end of the
  new buffer. The exception escaped Terminal.Write and ended the host's
  read loop, so ConPTY blocked and the program behind it stopped on its
  first frame. Only the short dimension doubles now, and the retry never
  shrinks the canvas.
- The raster attribute's aspect ratio (Pan;Pad) was ignored. VT DOOM
  scales vertically with it, so the game drew at a fifth of its height.
  Each sixel row is now Pan/Pad pixels tall, capped at 16; P1 stays
  ignored, as in xterm.
- Colour registers are shared between images by default, as on a VT340
  and in Windows Terminal, instead of xterm's private default. VT DOOM
  sends its palette only when it changes (29 frames in 496); the rest drew
  in the sixteen default colours on black. CSI ? 1070 h still selects
  private registers, and RIS now resets the shared ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… strips in blocks

VT DOOM ran, but froze for seconds at a time and stopped for good after a
key press, and input arrived in bursts.

- Frames were discarded. The canvas doubles while a picture of unknown
  size is drawn, so a 2,188-pixel-wide frame sat on a 4,096-wide canvas;
  growing it taller tested that capacity against MaxSixelPixels, failed,
  and the retry kept the whole capacity and failed again. At 307x75 cells
  frames that grew that way were dropped: data arriving, nothing drawn,
  and after the menu appeared, every frame. The retry now shrinks to what
  has been drawn, and the growth copies only that.
- The decoder allocated a fresh canvas per picture, again per growth, and
  once more for the exactly sized copy -- several large-object-heap arrays
  per frame, two dozen full collections a second. A per-terminal
  SixelCanvas now holds two buffers reused from picture to picture (a
  Sixel picture is discarded once painted), the result is compacted in
  place, and the canvas starts at the last picture's size, so frames of
  one size never grow. SixelPlaced subscribers get a copy.
- SixelStrip held a line's pixels in one array, past the large-object
  threshold at an ordinary width; a host capturing frames shares every
  line, and the next paint copied the whole row. Strips are now blocks of
  16 cells, allocated when painted and copied on write one block at a
  time. SixelStrip.Pixels is replaced by CopyPixelsTo(destination, stride,
  columns); README updated.

Full collections while VT DOOM runs: 373 in 15 s before, a handful after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BlockColumns is private, and the README describes CopyPixelsTo rather
than how a strip stores its pixels: CopyPixelsTo hands out an ordinary
contiguous image, and nothing outside depends on the block size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Perf comparison — this change, against its base

3 run(s) of each side, alternating on one machine. Allocation is a count and is gated exactly. Time is a measurement, so its gate is derived from the spread this job just observed in itself rather than fixed in advance.

corpus bytes/char gen0/Mchar ns/char Δ time noise gate
scroll-ascii 0.00 → 0.00 0.00 → 0.00 2.74 → 2.72 -0.8% ±1% 4%
sgr-churn 0.00 → 0.00 0.00 → 0.00 7.26 → 7.33 +0.9% ±3% 9%
truecolor 0.00 → 0.00 0.00 → 0.00 7.70 → 7.62 -1.0% ±2% 6%
alt-redraw 0.00 → 0.00 0.00 → 0.00 11.53 → 11.71 +1.6% ±1% 4%
unicode 7.66 → 7.66 0.45 → 0.45 28.08 → 28.07 -0.0% ±3% 9%
flood 0.00 → 0.00 0.00 → 0.00 82.61 → 82.72 +0.1% ±1% 4%

Each corpus is gated at max(4%, 3 × its own noise). A wide noise column means this runner was busy and the timing half of the table should be read as advisory; the allocation half is exact either way.

assemblies measured
  • base: XTerm.NET 3.0.0.0 mvid:8a972284-ba2c-4595-bf7a-75a1ded0c392
  • head: XTerm.NET 3.0.0.0 mvid:80cc7062-6296-4011-878d-4ee399031774

Perf comparison — cumulative, everything since 2.0.7

3 run(s) of each side, alternating on one machine. Allocation is a count and is gated exactly. Time is a measurement, so its gate is derived from the spread this job just observed in itself rather than fixed in advance.

corpus bytes/char gen0/Mchar ns/char Δ time noise gate
scroll-ascii 0.00 → 0.00 0.00 → 0.00 2.72 → 2.72 +0.0% ±1% 5%
sgr-churn 0.00 → 0.00 0.00 → 0.00 7.25 → 7.33 +1.1% ±3% 9%
truecolor 0.00 → 0.00 0.00 → 0.00 7.62 → 7.62 +0.1% ±0% 5%
alt-redraw 0.00 → 0.00 0.00 → 0.00 11.54 → 11.71 +1.5% ±1% 5%
unicode 7.66 → 7.66 0.45 → 0.45 27.94 → 28.07 +0.4% ±1% 5%
flood 0.00 → 0.00 0.00 → 0.00 82.15 → 82.72 +0.7% ±1% 5%

Each corpus is gated at max(5%, 3 × its own noise). A wide noise column means this runner was busy and the timing half of the table should be read as advisory; the allocation half is exact either way.

assemblies measured
  • base: XTerm.NET 2.0.7.0 mvid:2609fe33-3671-4fb3-9e45-c3e43c231eb3
  • head: XTerm.NET 3.0.0.0 mvid:80cc7062-6296-4011-878d-4ee399031774

@tomlm
tomlm merged commit 453ed95 into main Oct 14, 2026
4 checks passed
@tomlm
tomlm deleted the fix-sixel-canvas-growth branch October 14, 2026 21:36
tomlm added a commit to tomlm/Iciclecreek.Avalonia.Terminal that referenced this pull request Oct 14, 2026
* Update XTerm.NET to 2.0.7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Example: font size menu and per-frame timings

A Font Size menu (and Ctrl+Plus/Minus) to tell cell count from pixel area,
and a fixed-height status panel with Avalonia's own frame timings -- UI
render, layout, compositor update and compositor paint -- plus the read
loop's throughput and time in Terminal.Write. Render > Repaint test redraws
the whole terminal for 120 frames and reports what each frame cost on each
thread. Render also toggles the Skia renderer and Avalonia's timing graphs;
EXAMPLE_SOFTWARE=1 and EXAMPLE_GPU_CACHE_MB change how Avalonia renders.

The read-loop counters are internal to TerminalView, visible to Example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Draw pictures in the Skia renderer, and Sixel as one image per line (4.1.0)

The Skia renderer handed every row holding a picture back to the classic
renderer, text and all. A SixelBuffer app has pictures on every row, so the
whole screen went classic: thousands of recorded draws a frame.

- Rows with pictures in front of the text (Sixel, Kitty z >= 0) stay on the
  Skia path. A Kitty placement's rows are drawn as one draw clipped to all of
  them, ordered by z then placement serial -- every row's own order. The
  plan is the classic path's TryPlanImageBlit, from its own runs.
- Pictures are uploaded once as GPU textures, released on the render thread.
  Painting tracked total picture size (~5 ms/MB) before this.
- Picture bitmaps on the classic path are immutable.
- Sixel comes from XTerm.NET 3.0's per-line SixelStrip: one draw per row out
  to the last painted cell, over the text and under Kitty pictures in front,
  uploaded again only when its version moves. Double-width rows draw it
  inside their transform.

ControlCatalog in Sixel mode at 307x77: 593 ms per frame on 2.0.7, 0.6 ms
now. Skia and classic output are pixel-identical in the headless harness.

Requires XTerm.NET 3.0.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Draw text on the classic path and pictures on a Skia picture layer

Measured at full screen (307x75, 2540x1231) the two renderers each won
half: the classic path drew text in 1 ms against the Skia text layer's 5,
because it replays each line's shaped glyph runs, while the Skia layer drew
pictures from GPU textures -- Sixel 0.7 ms against 3.7-5.5, 741 Kitty
pictures 35 ms against 250. A larger GPU cache did not close that gap.

So text always goes through the classic path, the one Consolonia sees, and
the pictures in front of it go to a PictureLayer: one custom draw operation
per frame, planned on the UI thread from the classic path's own runs and
TryPlanImageBlit, drawing every row's Sixel strip and then each Kitty
placement once from GPU textures. A placement's rows are clipped to one
rectangle when they stack into one (nearly always) and row by row when not;
the path clip they had cost more than the draws it saved. Where Avalonia
has no Skia canvas the layer reports it and pictures go back to DrawImage.

The Skia text renderer is removed (snapshot builder, text layer, font
cache). UseSkiaRenderer stays on all three controls as an obsolete no-op.

Now: text 0.46 ms, Sixel 0.6 ms, Kitty (741 pictures) 14.7 ms per frame.
The picture layer and the DrawImage path are pixel-identical headless.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* README: pictures go through the picture layer, text through DrawingContext

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reference XTerm.NET 3.0.0-rc001, the published prerelease

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Upload Sixel strips through SixelStrip.CopyPixelsTo

XTerm.NET now holds a strip in blocks of 16 cells (tomlm/XTerm.NET#177),
so there is no Pixels array to read. Both upload paths -- the Skia picture
layer's texture and the DrawImage fallback's bitmap -- copy the strip out
with CopyPixelsTo into a pooled buffer, so an upload per frame is not a
large allocation per frame, and only up to the last painted cell, which is
all a row draws.

Requires the XTerm.NET release containing #177.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reference XTerm.NET 3.0.0-rc002, which has SixelStrip.CopyPixelsTo

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant