Regarding retrieval of Unihan property in ICU4X

[opnuub]

Hi, I'll like to ask about how can I retrieve the kRSUnicode property for CJK characters (Unihan) in Rust for reimplementation of the AdaBoost zh-hant/hk/tw word segmentation parser in unicode_seg. Thank you.

@sffc

[sffc]

CC @Manishearth, is this a property that we could add easily?

[Manishearth]

I believe icuexportdata doesn't do unihan right now.

kRSUnicode is a numeric-valued or maybe string-valued property, fwiw. We may wish to have a different data structure for unihan data since a lot of it won't benefit from CPT.

(so no, not super easily)

[sffc]

Do you know what ICU4C functions to call in order to get kRSUnicode?

[Manishearth]

No, ICU4C doesn't expose them. It does in some small cases use Unihan data to compute things like collation tables, but it does not really have broad unihan support.

The reason is that the data is massive and highly specialized. In theory ICU4X could ship this based on Unihan.zip, but I'm inclined to suggest people parse these files themselves.

[sffc]

ok, do you have a suggestion on how to mechanically get the kRSUnicode property into ICU4X? Should we use icuexportdata or should we read unihan.zip directly in datagen?