UTF-8 Offsets into Encoding as VAL_INDEX() Invalidated Via Other References

[hostilefork]

Good to see you're moving to "UTF-8 Everywhere"...I've found it to be undeniably the right answer. No regrets!

However: you are using byte offsets inside Cells as the VAL_INDEX(). This means that offset can be at a place that is no longer a valid codepoint boundary, if the data is modified through another reference.

In your utf8-squash build:

>> str: "abc"
== "abc"

>> str2: next str
== "bc"

>> change str "😺"
== "bc"

>> str
== "😺bc"

>> str2
==
[process exited with code 3221225477 (0xc0000005)]

This means that there'd have to be some kind of reconciliation of VAL_INDEX() when you notice you're on a continuation byte. I guess the options are to just sync up to an arbitrary new position, or perhaps raise an error...

Admittedly, Rebol's invariants have never been great when you modify a series reference... in terms of what happens to the other references. But I think the small anchor of being able to conceptually model a series index living in a Cell as at least "no worse than an integer" offers some benefit.

So if you are curious: Ren-C stores the character index in the strings, and bears the cost of calculating the offset from that. In order to make this tolerable, there are "Bookmarks" on the string (currently at most one at a time) which map indices for offsets. There are optimizations (all ASCII strings can do O(1) jumps, short strings don't pay for the bookmarks, etc.)

https://github.com/metaeducation/ren-c/blob/45806b2fb6f56e8aaefcf718d77cae2c6a1241fb/src/core/types/t-string.c#L60

[Oldes]

Thanks... I agree that this must be solved... as I consider mentioned coding pattern as a bad one (and not really much useful), I will probably just throw an error, when string will start with a continuation byte.

Fixing an index to a position of the next valid code point is probably not possible, because I can imagine, that one may modify the string with more than one char and then the position would not be as expected.

Keeping an index in chars instead of bytes would require another big rewrite, which I don't want to do at this moment. Your concept of bookmarks sounds too complicated to me.

[hostilefork]

I consider mentioned coding pattern as a bad one (and not really much useful)

Is this bad?

c: take back str
pick str 1

From my point of view: storing an integer position inside a cell will always be too simplistic to provide any "magic". So what I decided, was that the user's best model of what it is would be just a pairing of an integer and the array data (as a convenience, so they do not have to pass them separately).

So I feel a user should expect the same behavior as if they passed the array around with the integer, and used that integer to do PICK-ing etc.

; UPDATE: This is not a proposed new syntax, I am using <<...>> to show the conceptual
; contents that I believe the user should be able to think of as what they can take for granted.

>> str: "abc"
== <<"abc" 0>>

>> str2: next str
== <<"abc" 1>>

>> change <<"abc" 0>> "😺"
== <<"😺bc" 0>>

>> str
== <<"😺bc" 0>>

>> str2
== <<"😺bc" 1>>

>> first str  ; pick <<"😺bc" 0>> 1 => pick "😺bc" (0 + 1)
== #"😺"

>> first str2  ; pick <<"😺bc" 1>> 1 => pick "😺bc" (1 + 1)
== #"b"

The "it's exactly like if you had a separate integer" philosophy answers a lot of questions, about how boundedness should be handled and other things. You are only implementing one code path, which is the one you would have to write for PICK and such anyway.

[Oldes]

I don't see anything bad on c: take back str or pick str 1.
I consider bad using multiple references to the same string and modifying these, like in your bug example.

Your code does not look like Rebol anymore. But I understand what do you mean... I just don't want to spend another week or two on some code that I don't consider useful enough.

But I will fix the crash of course.

[Oldes]

Maybe one day I can use indexes of chars, because as I said.. I'm aware that current fix is not good enough. But I have also other work to do.

[hostilefork]

I don't see anything bad on c: take back str or pick str 1.

Just to make sure we're on the same page (or clarify for any other readers): this breaks in the same way.

Let's say your string has a 1-byte first character, and a 4-byte second character. Then say str is at character position 2, which you would encode as a VAL_INDEX() of 1 in str.

When you do take back str you didn't change the offset that str held. It's still at VAL_INDEX() 1.

Now that points into the middle of a codepoint, when you would do the PICK.

I consider bad using multiple references to the same string and modifying these, like in your bug example.

Whether you think one is bad and one is good or not, they both require the same guarantee.

There's really no difference in the expectations, of the "multiple reference" case of str and str2 and the take/pick case on the "single reference" of str. Because str is effectively multiple references (one before the take, one after).

If one works, the other should too.

Your code does not look like Rebol anymore.

Note I was not proposing a new syntax, I was using non-syntax to "lay bare" the conceptual contents of what was there.

[Oldes]

Ok.. I see the problem... it would be easier if you would use code examples, like:

str: next "b🙂"
;== "🙂"

take back str
;== #"b"

str
;== "" but should be "🙂"

It looks that I must re-think the implementation:/

[Oldes]

Unfortunately there are still more serious issues than invalidated indexes, but related, like:

str: "b🙂" foreach c str [probe c]
#"b"
#"ð"
#"^(9F)"
#"^(99)"
#"^(82)"

So the implementation is far to be considered as complete... not a wonder why I was hesitating to start with this move:/

[hostilefork]

It is a non-trivial change to make (I did it in 2019)

How I did it: Ren-C has a separate designation for validated UTF-8 data, besides just Byte*:

https://github.com/metaeducation/ren-c/blob/master/src/include/structs/struct-char.h

In the C build, that does just become Byte*. But in C++ validating builds, it is a class that prohibits byte-level access (e.g. raw addition to get new pointers, or dereferencing bytes). So you can only use the UTF-8 operations on it. They can be optimized in terms of getting next and previous character by skipping over continuation bytes, because this type is only used on previously validated UTF-8.

Such tools have been very useful in making sure the codebase handles strings correctly, and are a pattern for how the C++ build is applied more generally to solve what would be hard cases to verify manually.

[Oldes]

I appreciate your interest, but I'm not following your code. It just confuses me, because you changed everything from the original Carl's sources. There is probably not a single line which would not be changed. So I rather do it my way.

[Oldes]

Problem is, that in my code there is like 370 appearances of the VAL_INDEX define. If I would use it like position in codepoints for UTF-8 encoded strings and bytes elsewhere, I would need to handle it correctly in all these places. I don't believe that the change of using UTF-8 everywhere would be so positive move. Actually it could be slower than the original width-switching implementation:/

Not to mention that in many places there are patterns like: if (VAL_INDEX(val) >= VAL_TAIL(val)) return 0; I cannot have index in codepoints and tail in bytes:/