Fix halfwidth katakana voiced and semi-voiced sound marks in width calculations by tats-u · Pull Request #89 · unicode-rs/unicode-width

tats-u · 2026-05-26T14:57:39Z

Halfwidth katakana (semi-)voiced sound marks U+FF9E & U+FF9F are the only Grapheme Extenders that belong to Letter (Lm). They are typical edge cases when you cosider grapheme clusters.

They should be counted as 1 (their EAW is H), not 0.

English transtion of initial Prompt (Copilot + GPT-5.4 high):

Add test_halfwidth_katakana:
ﾊﾟｸﾞ→4 characters

English translation of additional handling prompt after discovering that the added test fails (because GPT started flailing around trying to fix it):

The half-width Kana voiced mark test failed because they're Grapheme Extenders despite being Letters. The current spec sucks, so please fix it together

"pug dog" was hand-written. The rest part of that comment was completed by Copilot line completion.

…lculations

Co-authored-by: Jules Bertholet <jules.bertholet@gmail.com>

tats-u · 2026-06-30T11:29:43Z

@Manishearth Do you agree with the current change? Is there anything that prevents this PR from merged?

Manishearth · 2026-06-30T18:18:08Z

Just busy and haven't been able to get around to it

tats-u · 2026-07-01T03:32:24Z

That's too bad. Please do it when you have the chance.

tats-u · 2026-07-02T08:41:36Z

Thank you for the merge.

tats-u · 2026-10-07T12:03:12Z

Since the Unicode version has been updated, isn't it about time to release a new version?

…rks one cell wide (#717) ## Summary Renovate's #711 bumps unicode-width to 0.2.3, and every Rust and Nix job then fails `testing::tests::buffer_rows::read_an_offset_area_and_skip_the_cells_wide_symbols_cover`, which reads `["界ｶﾞa", " "]` where it expects `["界ｶﾞa ", " "]`. The new width table is correct, but ratatui double counts with it. ratatui-core 0.1.2 adds a cell to each halfwidth katakana sound mark (U+FF9E, U+FF9F) in `CellWidth` (ratatui/ratatui#2499) because unicode-width used to count them as zero-width, and unicode-width 0.2.3 counts them as one cell itself (unicode-rs/unicode-width#89). Together they lay `ｶﾞ` out over three cells while terminals draw it in two, so ratatui reserves a cell after each mark that the terminal never draws. This PR holds unicode-width at 0.2.2, where ratatui paints each mark one cell wide. The hold has a cost: `ox`'s own measures, such as `wrap.rs` and the `ox --list` padding, still count each mark as zero cells, as on `main`, one cell less than ratatui and the terminal draw. unicode-width 0.2.3 would fix `ox --list`, which writes to the terminal directly, but not the TUI, since ratatui would then paint one cell more than `wrap.rs` measures. Both gaps close once a fixed ratatui lets the bound go. - **Bound:** `Cargo.toml` requires `unicode-width = ">=0.2, <0.2.3"`, so `cargo update` and lock file maintenance stay on 0.2.2. ratatui-core 0.1.2 is its latest release, ratatui's `main` still adds the cell, and its manifest accepts any unicode-width from 0.2.0, so no ratatui update fixes the combination yet. - **Renovate:** Renovate widens a range with a `<` bound by default, and a local dry run proposed `>=0.2, <0.2.4` in the Rust dependencies group. A `packageRules` entry with `allowedVersions: "<0.2.3"` stops that, so Renovate's rebase of #711 leaves only xattr 1.7.0. - **Upstream:** No ratatui issue or PR reports the double count yet. #718 tracks dropping the bound once a ratatui-core release counts each mark once. ## Test plan - [x] With #711's lock on `main` (unicode-width 0.2.3, xattr 1.7.0), `cargo test -p ox-tui --lib testing::tests` fails with the same `["界ｶﾞa", " "]` as CI - [x] With the bound, `cargo update -p unicode-width` locks nothing, and a full `cargo update` changes only xattr 1.6.1 → 1.7.0, on which `cargo test --workspace` passes - [x] `renovate-config-validator --strict --no-global .github/renovate.json` passes, and `renovate --platform=local --dry-run=lookup --enabled-managers=cargo` proposes no unicode-width update and keeps xattr 1.7.0 in `renovate/rust-dependencies` - [x] `cargo fmt --check`, `cargo build`, `cargo clippy --all-targets`, `cargo clippy`, and `cargo doc --no-deps` pass - [x] `cargo llvm-cov`: 3543 tests pass with 99.17% line coverage - [x] `cargo deny check` passes: advisories, bans, licenses, and sources ok - [x] `pnpm format`, `pnpm lint`, and `pnpm spellcheck` pass - [x] CI passes on every job at 64f1c61, including Rust and Nix on every platform, Lint, MSRV, and the release builds

…rows (#41) ## Why The latest-versions sweep left termpane on `unicode-width` 0.2.2 and an older generated CI. Moving to the latest stable crates exposed one width change and left two review rows open. ## What changes - **Toolchain, dependencies, Renovate, CI** (b000637, c95befd): Rust 1.99.1, the latest crates, a Renovate config, and the workflows regenerated with velnor-actions 0.1.73. - **`unicode-width` 0.2.3** (`fix(width)`): 0.2.3 gives the halfwidth katakana voiced marks U+FF9E and U+FF9F a width of one (unicode-rs/unicode-width#89, in 0.2.3; 0.2.2 gave them zero because they are `Grapheme_Extend`). `display_width` added a column per mark to compensate, which now counts a lone mark twice, and the grid no longer joins the mark to its base. termpane follows the new widths: xterm's `wcwidth.c` lists neither mark as combining and not as wide, so each mark takes its own cell. The compensation is removed, the dependency floor is `0.2.3` so a 0.2.2 resolution cannot bring the join back, and the three cluster-growth tests that used the mark as their only example of growth now use U+2601 U+FE0F. The per-cell cap (base plus 16) is untouched. - **DECSTR** (`fix(grid)`): follows xterm (`charproc.c`, `ReallyReset`, DECSTR branch): auto-wrap is restored, the cursor and a deferred wrap stay where they are, and the saved cursor is homed. It stays safe for the cap and the earlier security fixes: the phantom column is always paired with its deferred wrap, so a printable is never dropped while marks join the last cell, and a join checks the cap whatever the cursor state. If `?7l` was in force, the parked column is not a wrap, so the cursor goes back onto the last column, as in xterm. One divergence stays and is recorded in the README: xterm keeps its deferred-wrap flag beside the homed saved cursor, which termpane's saved-column encoding cannot hold. ## Tests Every new or changed test fails with its change reverted and passes with it (checked three ways: the width compensation restored, the old DECSTR body, and the lock back on 0.2.2). The retargeted growth tests do not depend on the width change, so they pass on both. Gate: fmt, clippy `-D warnings`, nextest in both modes (206 and 325 tests), doc, deny, and `cargo build --benches`, all clean. ## Required check The branch protection ruleset `protect-main` requires `Required` and `DCO`. The regenerated `ci.yml` still emits a job named `Required` (`required-gate`, plus a separate `Required verdict` job), so no ruleset change is expected. If CI reports the check as `Velnor / Required` instead, branch protection needs to change to match; this PR does not touch branch protection. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

Fix halfwidth katakana voiced and semi-voiced sound marks in width ca…

c01a001

…lculations

Manishearth approved these changes May 26, 2026

View reviewed changes

Jules-Bertholet reviewed May 27, 2026

View reviewed changes

Comment thread src/lib.rs Resolved

Move definition location

e0f8a13

tats-u requested a review from Jules-Bertholet May 27, 2026 23:57

Jules-Bertholet reviewed May 28, 2026

View reviewed changes

Comment thread src/lib.rs Resolved

Tweak definition

fcc0e2d

Co-authored-by: Jules Bertholet <jules.bertholet@gmail.com>

tats-u requested a review from Jules-Bertholet May 28, 2026 03:33

Jules-Bertholet approved these changes May 28, 2026

View reviewed changes

tats-u requested a review from Manishearth June 8, 2026 00:34

Manishearth approved these changes Jul 2, 2026

View reviewed changes

Manishearth merged commit c7b4d42 into unicode-rs:master Jul 2, 2026

tats-u deleted the fix-half-voiced branch July 2, 2026 08:41

This was referenced Oct 28, 2026

chore(deps): hold unicode-width below 0.2.3 so ratatui keeps sound marks one cell wide hakula139/oxide-code#717

Merged

Drop the unicode-width bound once ratatui counts halfwidth sound marks once hakula139/oxide-code#718

Open

donbeave mentioned this pull request Nov 5, 2026

chore: move to the latest stable dependencies and address the review rows tailrocks/termpane#41

Merged

Navigation Menu

Repository navigation

Fix halfwidth katakana voiced and semi-voiced sound marks in width calculations - #89

Fix halfwidth katakana voiced and semi-voiced sound marks in width calculations#89
Manishearth merged 3 commits into
unicode-rs:masterfrom
tats-u:fix-half-voiced

tats-u commented May 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

tats-u commented Jun 30, 2026

Uh oh!

Manishearth commented Jun 30, 2026

Uh oh!

tats-u commented Jul 1, 2026

Uh oh!

tats-u commented Jul 2, 2026

Uh oh!

tats-u commented Oct 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tats-u commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tats-u commented Jun 30, 2026

Uh oh!

Manishearth commented Jun 30, 2026

Uh oh!

tats-u commented Jul 1, 2026

Uh oh!

tats-u commented Jul 2, 2026

Uh oh!

tats-u commented Oct 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tats-u commented May 26, 2026 •

edited

Loading