请去除semaphore的限制,以提高Data Node的compaction并行度。 #39024
xiaobingxia-at
started this conversation in
Ideas & Feature requests
Replies: 2 comments 1 reply
-
你看到的这行代码是控制单个datanode里的最大的并行任务数,目前虽然hard code为10,但对于多个datanode的集群已经够用,你如果有10个datanode,那么整个集群并行的任务数量可以达到10x10=100 |
Beta Was this translation helpful? Give feedback.
1 reply
-
这个semaphore应该是早期的约束。确实可以去掉。 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
之前在询问如何提高data node的compaction任务并发度,被告知用slot机制。但是详细读代码后: https://github.com/milvus-io/milvus/blob/master/internal/datanode/compaction/executor.go#L147
发现data node的slot机制只能控制task queue的大小,并不能控制task执行的并发度。这个并发度是由一个size为10的semaphore控制的。也就是说,最多允许10个compaction task在运行。
从comment上也看出这个机制是要被deprecated。麻烦请把这个sempphore限制去除由slot机制控制并发度,或者我们能config semaphore的size。谢谢!
Beta Was this translation helpful? Give feedback.
All reactions