Getting "grpc: received message larger than max (76596666 vs. 67108864)" error when using Hybrid search

[Preet1234-ux]

I am using hybrid search for the retrieval process for my application . I have defined the field as :

fields = [
FieldSchema(
name=pk_field,
dtype=DataType.VARCHAR,
is_primary=True,
auto_id=True,
max_length=100,
),
FieldSchema(name=dense_field, dtype=DataType.FLOAT_VECTOR, dim=dense_dim),
FieldSchema(name=sparse_field, dtype=DataType.SPARSE_FLOAT_VECTOR),
FieldSchema(name=text_field, dtype=DataType.VARCHAR, max_length=4194304),
]

I have large dataset for which I want to store embeddings in milvus DB . But I am getting below error :

**raise InactiveRpcError(state) # pytype: disable=not-instantiable grpc.channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "grpc: received message larger than max (76596666 vs. 67108864)" debug_error_string = "UNKNOWN:Error received from peer ipv4:20.199.108.204:19530 {grpc_message:"grpc: received message larger than max (76596666 vs. 67108864)", grpc_status:8, created_time:"2024-08-16T07:06:03.282491587+00:00"}"

If I try to increase the max_length , then I am getting below error :

raise MilvusException(status.code, status.reason, status.error_code)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1100, message=the maximum length specified for a VarChar should be in (0, 65535]: invalid parameter)>

Can you provide me solution to store all the documents in milvus DB ?

[yhmo]

FieldSchema(name=text_field, dtype=DataType.VARCHAR, max_length=4194304) ------------------------- In FieldSchema, the max_length defines the max bytes for each row in the text_field. The max_length value is not allowed to exceed 65535. So, 4194304 is an illegal value. Milvus is not a good place to store long-length text content.

grpc: received message larger than max (76596666 vs. 67108864) --------------------------------- This is grpc transfer limit. The milvus server has limited the size of each grpc request to 64MB. You are trying to insert a batch of data more than 73MB, which exceeds the max value. You can split the batch into smaller batches.

[xiaofan-luan]

I am using hybrid search for the retrieval process for my application . I have defined the field as :

fields = [ FieldSchema( name=pk_field, dtype=DataType.VARCHAR, is_primary=True, auto_id=True, max_length=100, ), FieldSchema(name=dense_field, dtype=DataType.FLOAT_VECTOR, dim=dense_dim), FieldSchema(name=sparse_field, dtype=DataType.SPARSE_FLOAT_VECTOR), FieldSchema(name=text_field, dtype=DataType.VARCHAR, max_length=4194304), ]

I have large dataset for which I want to store embeddings in milvus DB . But I am getting below error :

**raise _InactiveRpcError(state) # pytype: disable=not-instantiable grpc._channel.InactiveRpcError: <InactiveRpcError of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "grpc: received message larger than max (76596666 vs. 67108864)" debug_error_string = "UNKNOWN:Error received from peer ipv4:20.199.108.204:19530 {grpc_message:"grpc: received message larger than max (76596666 vs. 67108864)", grpc_status:8, created_time:"2024-08-16T07:06:03.282491587+00:00"}"

If I try to increase the max_length , then I am getting below error :

raise MilvusException(status.code, status.reason, status.error_code) pymilvus.exceptions.MilvusException: <MilvusException: (code=1100, message=the maximum length specified for a VarChar should be in (0, 65535]: invalid parameter)>

Can you provide me solution to store all the documents in milvus DB ?

are u using any batch insertion or a entity execed 65536? I would expect to put any data beyond this size to another storage like S3

[songohannyc]

@xiaofan-luan First, I agreed that batch insertion is the right thing to do.

Just out of curiosity, is it possible to configure this 64MB payload limit. Is it a soft limit or hard limit?
If it's a soft limit, could you point out how one could add the configurations? Thank you for your help.

[xiaofan-luan]

@xiaofan-luan First, I agreed that batch insertion is the right thing to do.

Just out of curiosity, is it possible to configure this 64MB payload limit. Is it a soft limit or hard limit? If it's a soft limit, could you point out how one could add the configurations? Thank you for your help.

it is a grpc limit and you can change by

Proxy
grpc:
serverMaxSendSize: 268435456 # The maximum size of each RPC request that the proxy can send, unit: byte
serverMaxRecvSize: 67108864 # The maximum size of each RPC request that the proxy can receive, unit: byte
clientMaxSendSize: 268435456 # The maximum size of each RPC request that the clients on proxy can send, unit: byte
clientMaxRecvSize: 67108864 # The maximum size of each RPC request that the clients on proxy can receive, unit: byte

there is many other config you might need to change, for example the kafka write size.