Request to enable BM25 search for Japanese in formal container release

[caozhejiong]

BM25 search for Japanese requires Lindera tokenizer with the dictionray of ipadic.
But this feature is disabled on Mar 27 by changing the conditional compilation of the following file:
internal/core/thirdparty/tantivy/tantivy-binding/Cargo.toml

default = ["lindera-ipadic", "lindera-ko-dic", "lindera-cc-cedict"] is changed into default = [].

The merge request is: enhance: not build lindera features by default and support make milvus with tantivy features #40813
#40813

Why the BM25 search for Japanese is removed? Is it for making unit test code passed?

Do you have any release plan to enable this feature again in a formal release version?
The current document of v2.5.x states this feature is supported but the released container images do not support it.

[liliu-z]

/assign @aoiasd

[aoiasd]

Because Lindera changes their dictionary download path and abolish the old path cause milvus which use low version lindera comlilation failure.
In order to prevent Lindera changes dictionary download path again and cause Milvus to be completely uncompileable in some old version, we remove all Lindera dictionary feature in default.
If you want use some Lindera dictionary could make milvus with TANTIVY_FEATURES args like:
make milvus TANTIVY_FEATURES=lindera-ipadic,lindera-ko-dic

And we are exploring other solutions to make it possible to support by default.