Add isAsciiDigit to Rune

[johnnovak]

Summary

Some util functions that are defined for chars are missing for Runes.

isAsciiDigit is one of them.

Description

func isAsciiDigit*(r: Rune): bool =
  ord(r) >= ord('0') and ord(r) <= ord('9')

Alternatives

No response

Examples

No response

Backwards Compatibility

No response

Links

No response

I think that such a function might be a bit too misleading for unicode, a proper Unicode implementation will be harder to do, but is already done in e.g. https://github.com/nitely/nim-unicodedb:

assert utmDigit in Rune(0x00B2).unicodeTypes()

https://github.com/nitely/nim-unicodedb?tab=readme-ov-file#types

[johnnovak]

I think that such a function might be a bit too misleading for unicode, a proper Unicode implementation will be harder to do, but is already done in e.g. https://github.com/nitely/nim-unicodedb:

assert utmDigit in Rune(0x00B2).unicodeTypes()

https://github.com/nitely/nim-unicodedb?tab=readme-ov-file#types

Oh right. Wasn't aware of these extra Unicode numeric runes.

[johnnovak]

Ok, so I've looked into it, and I still think my implementation is fine for 99.999% use cases. Sure, there are all sorts of other interesting Unicode digit code points, but how often do people use these... extremely rarely, I'd say.

In real life, the 0 to 9 ASCII chars are enough.

https://www.fileformat.info/info/unicode/category/Nd/list.htm

[Alogani]

Hello,

Maybe this implementation ?

const allZeros = [48, 1632, 1776, 1984, 2406, 2534, 2662, 2790, 2918, 3046, 3174, 3302, 3430, 3558, 3664, 3792, 3872, 4160, 4240, 6112, 6160, 6470, 6608, 6784, 6800, 6992, 7088, 7232, 7248, 42528, 43216, 43264, 43472, 43504, 43600, 44016, 65296, 4170, 4307, 4358, 4367, 4371, 4381, 4399, 4421, 4429, 4453, 4460, 4467, 4494, 4501, 4549, 4565, 4570, 4597, 5798, 5804, 5813, 7548, 7549, 7550, 7550, 7551, 7700, 7727, 7759, 7829, 8127]
# List of zero converted from https://www.fileformat.info/info/unicode/category/Nd/list.htm

func toDigitImpl(r: Rune): int =
    let codePoint = ord(r)
    for z in allZeros:
        # not a binary search, because first runes are more common
        if codePoint >= z:
            return codePoint - z
    return -1

func isDigit*(r: Rune): bool =
    let digit = r.toDigitImpl()
    digit in {0..9}

func toDigit*(r: Rune): int =
    result = r.toDigitImpl()
    if result notin {0..9}:
        raise newException(RangeDefect, "rune is not a valid digit")

It should need some unit tests case

[tersec]

Ok, so I've looked into it, and I still think my implementation is fine for 99.999% use cases. Sure, there are all sorts of other interesting Unicode digit code points, but how often do people use these... extremely rarely, I'd say.

In real life, the 0 to 9 ASCII chars are enough.

https://www.fileformat.info/info/unicode/category/Nd/list.htm

Nim's standard library is already full of random jank which comes from this approach. Please do this correctly if it's done at all.

Not just some "in real life ..." approach dismissive of what you regard as irrelevant edge cases.

And, if that's too difficult, that's maybe an indication that what you're looking for is conceptually nontrivial, and not just to be casually tossed into the Nim stdlib.

[johnnovak]

Dunno man. I'd personally take a 99.99% solution than no solution, or having to install a library for the 0.01% use case that I'll never need. I guess this is an approach thing, we are all a bit different 😄

Having full support for all the obscure and historical languages and their codepoints seems like a specialised, niche use case to me, not the standard use case.

Thinking about standard use cases vs niche use cases might be a good guide for the stdlib.

I've looked at your Unicode library. It's nice and super handy for someone who needs that level of support. But personally I don't want to pull it in if I only need a simple isDigit on Rune that will work with all languages in actual use.

[SolitudeSF]

If its tailored for your use case why does it have to be in everyone's stdlib?

[johnnovak]

If its tailored for your use case why does it have to be in everyone's stdlib?

Because it's practical and 99.99% of people find it useful, and most people haven't even heard of those obscure languages with those alternative Unicode codepoints, neither will come near them in their lifetime?

Good enough reason? 😆

To be more productive, it's more fruitful to think in terms of levels of support. Something similar to what's described here:
https://cldr.unicode.org/index/language-support-levels

I'd imagine supporting most major languages in actual use is a good practical level of support. Which my isDigit does. Then supporting historical and obscure languages, at the expense of added complexity, extra databases, libraries, and whatnot, is the full support, but I don't think many people ever need that level.

[Alogani]

At first, I didn't see the point of adding johnnovak's isDigit to Rune, as it is :

• really simple for anyone with this case to code, if you use utf8, there is a probability you have heard of codepoints (or maybe not ?)
• not well adapted to rune, because why not use isDigit for strings (which do the same, but imply a conversion from rune) ?
• deceptive, because the user could think it checks all unicode digits (if he heards of)

But I do agree with johnnovak that :

• No one wants to rewrite the wheel (or download dozens of nimble packages) because simple things are not present in the stdlib
• Most people wants a solution, not always the perfect solution
• Proposing only perfect solution/specialized has some drawbacks : time to implement, more bugs, generally more complex to use, slower

So I think a isDigit implementation should be added for rune in std/unicode, either the one from johnnovak with a proper documentation/proper name about its limitation (really important !), either the one I propose (but which may be too specialized for stdlib).

So I would :

• not add my implementation of isDigit that checks all codepoints
• but I would add johnnovak but renaming it into isAsciiDigit(r: Rune): bool !

[tersec]

Dunno man. I'd personally take a 99.99% solution than no solution, or having to install a library for the 0.01% use case that I'll never need.

This is a false dichotomy. Implementing isAsciiDigit for Rune neither has no solution, nor requires installing a library, as @Alogani alludes to. One (but not the only, or necessarily best) approach is:

import std/[sequtils, strutils, unicode]
func isAsciiDigit(x: Rune): bool = ($x).allIt(it.isDigit)
for digit in Digits:
  doAssert ($digit).runeAt(0).isAsciiDigit
for letter in Letters:
  doAssert not ($letter).runeAt(0).isAsciiDigit

allIt is questionable for this purpose; one can/should exploit UTF-8 encoding more closely. And, this should work for multi-byte UTF-8 encodings too, because of how UTF-8 works. The high bit will be set, et cetera.

This also addresses

No one wants to rewrite the wheel (or download dozens of nimble packages) because simple things are not present in the stdlib

What "dozens of nimble packages" do you refer to here?

This is the ASCII case, of course. But since you've already indicated that's all you care about, literally one line will do it, plus the tests of which I provided some non-exhaustive examples, without having to create an attractive nuisance in the stdlib. One might note that nothing else in std/unicode restricts itself to ASCII. The point of the module is to support Unicode. That means Unicode casing, Unicode spaces/whitespace, Unicode alpha/letters, et cetera.

Thinking about standard use cases vs niche use cases might be a good guide for the stdlib.

You're already planning on using this functionality in #23462 it looks like? Which is also supposed to go in the stdlib. So it wouldn't just be this function, but this natural sort with this ASCII-digits constraints. This illustrates the point, even: you're already, right now, planning on propagating this brokenness to other parts of the stdlib.

I've looked at your Unicode library. It's nice and super handy for someone who needs that level of support. But personally I don't want to pull it in if I only need a simple isDigit on Rune that will work with all languages in actual use.

I don't understand the logic here. Aside from more of the dismissal of other languages ("in actual use"), you're proposing, I think, to add

func isDigit*(r: Rune): bool =
  ord(r) >= ord('0') and ord(r) <= ord('9')

in this very issue, to, presumably (?) the std/unicode library. Which one wouldn't be able to access if you aren't willing to "pull it in".

The difference is only, do you use the $/toUTF8 or other approach (many creative approaches exist) to access Runes from std/unicode, or do you use this hypothetical isDigit. Both require that you would "pull it in".

[johnnovak]

Everybody uses standard Arabic ASCII numerals these days, come on. Anyway, you guys do what you think it's best, I don't particularly care anymore 😄

A sane way to do natural sorting is to restrict yourself to standard Arabic numerals in the natural sort algo. There's no committee that will tell you this is wrong or anything... It's perfectly fine.

[Alogani]

I do agree with some points of tersec. Especially even the corner cases are always important and "isDigit" should not be deceptive for those corner cases.

But:

• I don't see the problem with Add a natural sort inside std/library #23462 if naturalSort relies on std/lib isDigit implementation, so it woult act accordingly to how we decide isdigit to looks like. Or you would want naturalSortRuneAscii, naturalSortRuneWithAlgebric, etc ?... Or you prefer no code is either added to std/lib, and if someone needs somethings, he just takes some days and hours to do anything ?
• using a nimble package has always a learning curve, and some basics or common functions should be in stdlib. The std/lib begins to have some good tools, but is still lacking in some regards (i don't agree that std/lib should be minimum and everything as possible should be externalised in nimble, but this is another subject)
• isDigit is one of those common functions
• i also see no problem of adding isAsciiDigit to runes. Why is adding more code to stdlib is bad if that code is documented, tested, and explicit ? Otherwise, the std/lib will never grow if we battle for such a "simple" function

But to say who is wrong or true, let's look at what the other languages do. Like python (yes so original), who uses utf8 strings as default :
https://github.com/python/cpython/blob/ab0e60101637b7cf47f3cc95813996791e7118c4/Objects/unicodeobject.c#L2365

Spoiler : all zeros are checked in python :

"\u0660".isdigit() == True
"\u06F1".isdigit() == True

So i repropose my implementation : #23465 (comment)
Or the implementation of someone else. But I still think a kind of isDigit for runes shall be added to std/lib

[johnnovak]

Yeah, and it's a philosophy / approach thing too. I always thought of Nim as a "sweet spot" language. Look at the Java standard libraries; they're far more "complete" than any Nim std lib. But they are huge, with a huge and complicated API surface.

Nim for me always has been 80% of the benefits with 20% of the complexity. Having said that, these ultra-detailed and specialised libs have their place too as separate packages for those who really need them.

[johnnovak]

The difference is only, do you use the $/toUTF8 or other approach (many creative approaches exist) to access Runes from std/unicode, or do you use this hypothetical isDigit. Both require that you would "pull it in".

Wait, is there a plan to move the various Nim std libs out into their own projects that you must then install via Nimble? I'm not too thrilled of that prospect if it's true. I like a stdlib to be slow moving like in most languages (NodeJS does not count 😄). Having to install libs just to be able to use UTF8 is a bit backwards...