feat: Ensemble async callback execution (rework) #438
Conversation
| } | ||
|
|
||
| // Case 1: Sequence batching is enabled | ||
| // Case 2: Dynamic batching is disabled and there is only one instance group |
There was a problem hiding this comment.
I don't understand why the order needs to be preserved in this case.
There was a problem hiding this comment.
In gRPC streaming case, the client would expect the responses order match the requests order.
There was a problem hiding this comment.
I don’t think the scheduler needs to care whether the request is received via gRPC streaming or not. That is outer layer requirement. Scheduler only care whether itself needs to preserve ordering at model instance levels (i.e. whether preserve ordering is set / sequence batching is used)
There was a problem hiding this comment.
Yes. GRPC stream is just one example where response order is guaranteed to be the same as request.
In case of one model instance, we don't want to use asynchronous callbacks, which violates the response order.
| // Case 3: Dynamic batching is enabled and preserve_ordering is true | ||
| // Case 4: Model transaction policy is decoupled (breaks RequestTracker | ||
| // lifecycle) | ||
| // Note: Although decoupled models do not preserve the order of |
There was a problem hiding this comment.
decoupled models "should preserve" the order of response
There was a problem hiding this comment.
A decoupled model/backend may also send responses out-of-order relative to the order that the request batches are executed.
I found this from
https://github.com/triton-inference-server/server/blob/main/docs/user_guide/decoupled_models.md#decoupled-backends-and-models
There was a problem hiding this comment.
That is referring to responses among different requests within the same batch, the responses of the same request is still preserved. i.e. batch can contain req1, req2, and respond in req2res1, req1res1, req1res2, req2res2
|
|
||
| // Attempt to enqueue the callback. If all workers are busy and queue is at | ||
| // capacity, execute the callback immediately in current thread. | ||
| if (pool->TaskQueueSize() < pool->Size()) { |
There was a problem hiding this comment.
Is this correct? Size() returns the number of worker and TaskQueueSize() returns the number of "pending" task. You can consider the workers are busy when TaskQueueSize() > 0, because pool->TaskQueueSize() == pool->Size() actually means the # of pending requests equals to the # of workers, right?
There was a problem hiding this comment.
Correct. But consider a case where N busy workers are almost finishing. Then as long as TaskQueueSize <= N, the pending tasks can execute almost immediately. The maximum of N is 8.
In fact, I did compare if (pool->TaskQueueSize() == 0) vs if (pool->TaskQueueSize() < pool->Size()), and the latter one yielded higher throughput, indicating a small wait time is better than synchronus execution on average.
| } | ||
|
|
||
| // Case 1: Sequence batching is enabled | ||
| // Case 2: Dynamic batching is disabled and there is only one instance group |
There was a problem hiding this comment.
I don’t think the scheduler needs to care whether the request is received via gRPC streaming or not. That is outer layer requirement. Scheduler only care whether itself needs to preserve ordering at model instance levels (i.e. whether preserve ordering is set / sequence batching is used)
| // Case 3: Dynamic batching is enabled and preserve_ordering is true | ||
| // Case 4: Model transaction policy is decoupled (breaks RequestTracker | ||
| // lifecycle) | ||
| // Note: Although decoupled models do not preserve the order of |
There was a problem hiding this comment.
That is referring to responses among different requests within the same batch, the responses of the same request is still preserved. i.e. batch can contain req1, req2, and respond in req2res1, req1res1, req1res2, req2res2
What does the PR do?
Reduce e2e latency in ensemble model by executing callbacks asynchronously at the end of each ensemble step. Excluding models that require responses to have the same order of requests.
Improvement: maximum throughput of sample ensemble model increased from 39k infer/sec to 50k infer/sec.
Checklist
<commit_type>: <Title>Commit Type:
Check the conventional commit type
box here and add the label to the github PR.
Related PRs:
triton-inference-server/common#133
Previous PR: #429
Where should the reviewer start?
Reviewer should start from the second commit.
Attention to the
preserve_responses_orderlogic.Test plan:
L0_simple_ensemble
L0_sequence_batcher
L0_backend_python
28454142
Caveats:
Background
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)