Support for Pulsar's partitioned topic?

[yyfrankyy]

Currently, the pulsar consumer uses the Seek op to recover the consuming process, but Pulsar has no support for the Consumer concept (moved to a Reader, apache/pulsar#7518 ). instead, Milvus maintains its own implementation of partition in many places.

In our own cases, we enabled auto-partition in a pulsar cluster scope, then a seek failure occurred,

[2022/11/08 05:25:28.929 +00:00] [WARN] [datanode/flow_graph_manager.go:61] ["new data sync service fail"] [vChannelName=milvus-book2vec-rootcoord-dml_3_437225530014826497v0] [error="failed to seek, error All attempts results:\nattempt #1:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #2:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #3:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #4:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #5:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #6:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #7:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #8:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #9:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #10:for partition topic, seek command should perform on the individual partitions: SeekFailed\nattempt #11:for partition topic, seek command should perform on the individual partitions: SeekFailed\n"]

Due to this error, datanode will not able to release its disposal process, then it came with another ERROR message:

time="2022-11-08T05:25:28Z" level=error msg="[Failed to create consumer]" consumerID=47 error="server error: ConsumerBusy: Exclusive consumer is already connected" name=lumxy subscription=milvus-book2vec-dataNode-56-437225530014826497 topic="persistent://public/default/milvus-book2vec-rootcoord-dml_3-partition-10"

I personally patched the CreateProducer in pulsar_client.go, create a NonPartitionedTopic by pulsarctl before the topic was created automatically, which fixes this issue.

Will milvus consider supporting partitioned topic of Pulsar, by using a Reader interface instead of a Consumer.

[xiaofan-luan]

Milvus will use only one partition topic because we want to make sure all messages are in order.
If one topic throughput is not enough, you can simply use more shards instead.(Each shard will physically allocated to one topic)

[yyfrankyy]

Understand, my point is that this could make it a little bit complex to integrate into some working pulsar cluster (w/ auto partitioned-topic creation).

[yyfrankyy]

BTW, while I patch CreateProducer and Subscribe methods in pulsar_client.go using pulsarctl, there is another pulsar-related configuration that the document never mentioned is pulsar.webport, the default goes to 80 (which different from the pulsar default web service port, 8080), and it's configurable. ^_^