GiNZA >= 5.1 cannot process long (over 49149 bytes) texts #242
Description
This is essentially due to sudachi.rs's limitation but texts over 49149 bytes cannot be processed by GiNZA >= 5.1.
According to the sudachi.rs's code, the maximum text length (in bytes) is defined as u16::MAX / 4 * 3 (= 49149), so if a given text is longer than this size in bytes, sudachipy (sudachi.rs) raises an InputTooLong error.
Here is the related lines in the sudachi.rs's repo, which might help.
- https://github.com/WorksApplications/sudachi.rs/blob/c7d20b22c68bb3f6585351847ae91bc9c7a61ec5/sudachi/src/input_text/buffer/mod.rs#L32
- https://github.com/WorksApplications/sudachi.rs/blob/c7d20b22c68bb3f6585351847ae91bc9c7a61ec5/sudachi/src/input_text/buffer/mod.rs#L125
- https://github.com/WorksApplications/sudachi.rs/blob/c7d20b22c68bb3f6585351847ae91bc9c7a61ec5/sudachi/src/input_text/buffer/mod.rs#L244
(I personally asked sudachi's developpers about this limitation and they gave me a feedback that the max length (u16::MAX / 4 * 3) is chosen for performance.)