Data Assertion Issue from Tantivy while Building Scalar Index

[rohitreddy1698]

Hi,

I have a collection where I have multiple fields of type Array<VarChar(65535)>[1024] , I have added a scalar index : Inverted Index on this field and while the index build has started I see errors of following type on the Index Node :

[2025/05/12 06:29:17.248 +00:00] [WARN] [indexcgowrapper/helper.go:71] ["failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575\n\n"] [2025/05/12 06:29:17.248 +00:00] [WARN] [indexnode/task_index.go:319] ["failed to build index"] [clusterID=by-dev] [buildID=457240409305302803] [collection=457240409193279491] [segmentID=457240409300685969] [currentIndexVersion=6] [error="failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575\n\n: segcore error[segcoreCode=2001]"] [errorVerbose="failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575: segcore error[segcoreCode=2001]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrSegcore\n | \t/workspace/source/pkg/util/merr/utils.go:1025\n | github.com/milvus-io/milvus/internal/util/indexcgowrapper.HandleCStatus\n | \t/workspace/source/internal/util/indexcgowrapper/helper.go:78\n | github.com/milvus-io/milvus/internal/util/indexcgowrapper.CreateIndex\n | \t/workspace/source/internal/util/indexcgowrapper/index.go:114\n | github.com/milvus-io/milvus/internal/indexnode.(*indexBuildTask).Execute\n | \t/workspace/source/internal/indexnode/task_index.go:314\n | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask.func1\n | \t/workspace/source/internal/indexnode/task_scheduler.go:222\n | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask\n | \t/workspace/source/internal/indexnode/task_scheduler.go:235\n | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).indexBuildLoop.func1\n | \t/workspace/source/internal/indexnode/task_scheduler.go:262\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575\nWraps: (3) segcore error[segcoreCode=2001]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [2025/05/12 06:29:17.248 +00:00] [WARN] [indexnode/task_scheduler.go:236] ["process task failed"] [error="failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575\n\n: segcore error[segcoreCode=2001]"] [errorVerbose="failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575: segcore error[segcoreCode=2001]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/pkg/util/merr.WrapErrSegcore\n | \t/workspace/source/pkg/util/merr/utils.go:1025\n | github.com/milvus-io/milvus/internal/util/indexcgowrapper.HandleCStatus\n | \t/workspace/source/internal/util/indexcgowrapper/helper.go:78\n | github.com/milvus-io/milvus/internal/util/indexcgowrapper.CreateIndex\n | \t/workspace/source/internal/util/indexcgowrapper/index.go:114\n | github.com/milvus-io/milvus/internal/indexnode.(*indexBuildTask).Execute\n | \t/workspace/source/internal/indexnode/task_index.go:314\n | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask.func1\n | \t/workspace/source/internal/indexnode/task_scheduler.go:222\n | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask\n | \t/workspace/source/internal/indexnode/task_scheduler.go:235\n | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).indexBuildLoop.func1\n | \t/workspace/source/internal/indexnode/task_scheduler.go:262\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) failed to create index, C Runtime Exception: Assert \"(IsStringDataType(array_column[i].get_element_type()))\" at /workspace/source/internal/core/src/index/InvertedIndexTantivy.cpp:575\nWraps: (3) segcore error[segcoreCode=2001]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

1. Is element type validation performed for all the elements in the array during ingestion ?

2. Why is the DataType assertion failing because from my understanding Milvus strictly imposes the schema. So this error must have been thrown during ingestion and only and this should not be possible.

Thanks,
Rohit Mothe

[yhmo]

Which version of your milvus?
In the document, Array with VARCHAR element type supports INVERTED index.
https://milvus.io/docs/index-explained.md

[rohitreddy1698]

We are using 2.5.7

And yes we are able to add the index with the VARCHAR and INT64 element types for Field Type Array.
And the Indexing task is successfully added as well to the index queue, but while the actual index build , since this is a scalar index which runs on Tantivy that is where the Data Type Assertion is failing.

Is element type validation performed for all the elements in the array during ingestion ? - Can you confirm on this ?

Regards,
Rohit Mothe

[xiaofan-luan]

We are using 2.5.7

And yes we are able to add the index with the VARCHAR and INT64 element types for Field Type Array. And the Indexing task is successfully added as well to the index queue, but while the actual index build , since this is a scalar index which runs on Tantivy that is where the Data Type Assertion is failing.

Is element type validation performed for all the elements in the array during ingestion ? - Can you confirm on this ?

Regards, Rohit Mothe

I think this indicates that your array has elements that is not a string.

how did you write this data in? through bulk import or SDK?

can you build a case for this?
@yanliang567

[xiaofan-luan]

I'm assuming we have all the check and QA has test on it.

Also if you can offer the file of this segment this could be super helpful

[rohitreddy1698]

Hi @xiaofan-luan , @yanliang567
I'm assuming we have all the check and QA has test on it. - Got it.

Also if you can offer the file of this segment this could be super helpful - unfortunately we do not have that anymore since that collection has been dropped and cleaned up.

I think this indicates that your array has elements that is not a string. - Yes that's what I was getting at, but if ingest time validation is performed ( which from my understanding is ) it should not have gotten in at all.

how did you write this data in? through bulk import or SDK? - Definitely through the SDK, not bulk import.

which sdk and version of sdk did you run? - v.2.5.0

the max length for a varchar element is 65535, and Milvus refuses to create a collection with varchar max length =65536. - apologies for this confusion, yes max length was 65535, I typed in 65536 by mistake.

Thanks,

[yanliang567]

@rohitreddy1698 which sdk and version of sdk did you run?
the max length for a varchar element is 65535, and Milvus refuses to create a collection with varchar max length =65536.

[rohitreddy1698]

Hi @yanliang567 ,

Were you able to find out anything on this ? Thank you !

[yanliang567]

no luck for now. does this issue still reproduce to you recently? do you have any code snippet to reproduce it?

[rohitreddy1698]

Hello @yanliang567 ,

No unfortunately I am unsure of how to reproduce it. I have tried a few methods but have unable to, and recently as well I faced the same. But one follow up question is this is blocking all other indexing tasks and all the index nodes have the stuck index failing tasks for these they are busy retrying until I manually drop the index, restart the index nodes and then indexing for other collections/indexes proceed ?

Regards,
Rohit Mothe

[yanliang567]

Mmm...is it possible to share some pieces of data so that we could try to reproduce it with them?

[xiaofan-luan]

If we get the data from the segment with dirty data data that would be really helpful.

you can send to my email james.luan@zilliz.com

[xiaofan-luan]

(IsStringDataType(array_column[i].get_element_type(

from the log

(IsStringDataType(array_column[i].get_element_type(

is seems that the array column is not a string, and you are trying to create string index over it.

We can not do precheck on it, becasue check all data in string need to scan the whole collection.

you need to guarantee this by yourself.

[xiaofan-luan]

or you want to cast all the data into certain type?

@sunby might be able to help with the cast feature.

[sunby]

Or do you want to cast all the data into a certain type?

@sunby might be able to help with the cast feature.

I tried to reproduce the issue using pymilvus 2.5.0 but failed. The milvus array type should not be cast; it seems the data is incorrect.

[xiaofan-luan]

Or do you want to cast all the data into a certain type?
@sunby might be able to help with the cast feature.

I tried to reproduce the issue using pymilvus 2.5.0 but failed. The milvus array type should not be cast; it seems the data is incorrect.

problem is how does the wrong data get in

[rohitreddy1698]

Hi @xiaofan-luan ,

Thank you so much for following up on this.

So did you got a way to verify the corrupted data ? - NO

problem is how does the wrong data get in - Exactly the reason why I asked the two original questions :

Is element type validation performed for all the elements in the array during ingestion ?

Why is the DataType assertion failing because from my understanding Milvus strictly imposes the schema. So this error must have been thrown during ingestion and only and this should not be possible.

The issue is thrown during runtime , what this means is when the indexing tasks are started by tantivy for building scalar indexes.
If it throws Data Type Assertion Issue it led me to doubt if element type validation is performed during ingest time.

If I am able to reproduce this or find more details around this, will surely share with you.

Thanks and Regards,
Rohit Mothe

[xiaofan-luan]

we should have strong check on insert path, @sunby is working on checking that.

as long as it is not Json field and dynamic field, this should be verified