Some Devanagari including Viramas return wrong width

[kommander]

Hi! Great stuff. Tim aka Rockorager pointed out uucode in a conversation. I am currently playing with zg.

क्‍ष - This indic scripture is rendered as two cells by all terminals I tested (ghostty, kitty, alacritty, vscode),
but both zg and uucode give me width 1, which makes my rendering end up with ghost cells.

Any idea why this particular one (there might be otheres) returns 1 but is rendered as 2 by terminals?

[jacobsandlund]

Hi @kommander, glad you got pointed here! Issue #1, woohoo! Turns out I didn't have notifications configured properly (now I do).

I took a look at that string with this:

const str = "क्‍ष";
var break_state: uucode.grapheme.BreakState = .default;
var it = uucode.utf8.Iterator.init(str);
var cp = it.next();
std.debug.print("cp: {x:0>4}, wcwidth: {d}\n", .{ cp.?, get(.wcwidth, cp.?)  });
var last_cp = cp;
cp = it.next();
var isBreak = uucode.grapheme.isBreak(last_cp.?, cp.?, &break_state);
std.debug.print("cp: {x:0>4}, wcwidth: {d}, break: {}\n", .{ cp.?, get(.wcwidth, cp.?), isBreak });
last_cp = cp;
cp = it.next();
isBreak = uucode.grapheme.isBreak(last_cp.?, cp.?, &break_state);
std.debug.print("cp: {x:0>4}, wcwidth: {d}, break: {}\n", .{ cp.?, get(.wcwidth, cp.?), isBreak });
last_cp = cp;
cp = it.next();
isBreak = uucode.grapheme.isBreak(last_cp.?, cp.?, &break_state);
std.debug.print("cp: {x:0>4}, wcwidth: {d}, break: {}\n", .{ cp.?, get(.wcwidth, cp.?), isBreak });

and that gives me:

cp: 0915, wcwidth: 1
cp: 094d, wcwidth: 0, break: false
cp: 200d, wcwidth: 0, break: false
cp: 0937, wcwidth: 1, break: false

What's going on is that wcwidth gives only the "width" of a code point, but the actual resulting width of the grapheme (in this case all 4 code points) is likely to be more complicated. Note that the U+200D there is the zero width joiner, and in fact it looks like this string is on that wikipedia page.

The wcwidth is part of the built-in extensions, that I've nested under src/x and required using the advanced configuration to pull in, but initially I even wanted this to be in a separate uucode.x repo/library, but the zig build integration got too complicated (and there's a bug to figure out).

The reason this is an extension is that it's not part of the Unicode standard. See the source code at wcwidth.zig to see where I'm basing it from.

which makes my rendering end up with ghost cells.

Back to your issue, though, for your use case, are you computing grapheme clusters before continuing any rendering? If so, you might be able to add up the wcwidth for each code point and that might get you a better value? I'm not actually sure how terminals such as Ghostty are handling this, but I am curious to look at some point.

[jacobsandlund]

Oh @kommander I see, you're using Tim's code here: https://github.com/sst/opentui/pull/90/files#diff-957e7436e16569e080cb465f4caebbbd00fa2710550b7e790f1874fd62010033, and that's using zg strWidth. Looks like that says // Only adding width of first non-zero-width code point. in the zg source, so indeed that'll still return 1 instead of adding the 1 + 1. It looks like ziglyph is doing the same thing in its strWidth. I know Ghostty doesn't use strWidth, so it might be combining the code point widths into a wide grapheme some other way, and I haven't gotten to the bottom of it, yet.

Like I mentioned, this is all outside of the Unicode standard, sadly.

I'll leave this open for now, as I'm still getting up to speed on terminal use cases, and if I add some equivalent to strWidth (as an extension), then I'd like to have a better idea of what's standard there.

[kommander]

Thanks for looking into this! I wasn't aware that was out of standard. Maybe worth looking into what Ghostty does internally to end up with 2 cells for it. They have an esoteric c++ implementation they use currently, couldn't figure it out, that's how Tim ended up pointing me here.

Most things work for now switching between unicode and wcwidth like libvaxis does, so that's okay. Would love to get to the bottom of this at some point though.

[jacobsandlund]

I'd definitely love to get to the bottom of this, and also fully understand what Ghostty is doing here. I'll circle back to this eventually.

In the meantime, I was thinking of adding this:

var it = uucode.grapheme.Iterator.initUtf8(text); // tentative API, still an open issue: https://github.com/jacobsandlund/uucode/issues/5
var g = it.next()
uucode.x.grapheme.unverifiedWcwidth(g); // hopefully 2 for `क्‍ष`

For now that grapheme.unverifiedWcwidth(g) method would:

• take the width of the first non-zero-width code point (as per ziglyph and zg),
• but, if there's a ZWJ (U+200D), then make that a width of 2
  • the use of ZWJ in some of these scripts pretty much means combining two or more glyphs into one, so that's probably safer to assume that it's going to be wide.

I'd like to get a better idea of what to do here, but maybe that will be useful enough to do for now?

[kommander]

Sorry for my late answers, I don't get github notifications.

That could very much help! I hope Tim can give some insights into Ghostty.

[jacobsandlund]

sweet, i'll make sure to have at least something for this in by a 0.1.0 release or shortly after

[jacobsandlund]

Hey @kommander, I just added a grapheme.Iterator and then a x.grapheme.unverifiedWcwidth in 577f0d0. I hope that works decently well if you give that a try!

I'll leave this issue open until I've gotten a chance to take a deeper look, and see if this algorithm is the most reasonable thing to do (given it's non-standard), or if Ghostty does something better.

[jacobsandlund]

Once I get to this, this looks like a good resource: https://www.jeffquast.com/post/ucs-detect-test-results/

[kommander]

I am working on text rendering again and will give this a go in the coming week.