fix: Revert async execution of ensemble model ResponseComplete callback

[yinggeh]

What does the PR do?

Fixes L0_backend_python--base and L0_sequence_batcher--base.

Initially ensemble asynchronous callback feature was introduced in #429 in order to reduce latency in overheads added in the ensemble pipeline.

Asynchrous ResponseComplete didn't work in decoupled model because the order of responses cannot be guarenteed. In rare case, TRITONSERVER_RESPONSE_COMPLETE_FINAL is sent before other responses and causes segfault.

Even though I disable async call of ResponseComplete in decoupled model, it's still failing one test case in L0_sequence_batcher. So I ended up reverting async ResponseComplete callback feature. But async RequestComplete callback is working fine.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• fix

Related PRs:

Where should the reviewer start?

Test plan:

• CI Pipeline ID:
  27706423

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

[yinggeh]

Decrease thread pool size because there will be fewer callbacks running asynchronusly.

[oandreeva-nv]

why?

[GuanLuo]

Do you even need this constant?

[yinggeh]

Yes. Because RequestComplete callback is still leveraging thread pool to run asynchrously, which will benefit the ensemble model throughput.

[yinggeh]

core/src/ensemble_scheduler/ensemble_scheduler.cc

Lines 608 to 632 in 729422e

	void
	EnsembleContext::RequestComplete(
	TRITONSERVER_InferenceRequest* request, const uint32_t flags, void* userp)
	{
	auto request_tracker = reinterpret_cast<RequestTracker*>(userp);
	auto pool = request_tracker->CallbackPool();
	auto fn = [request, flags, request_tracker]() {
	if ((flags & TRITONSERVER_REQUEST_RELEASE_ALL) != 0) {
	LOG_TRITONSERVER_ERROR(
	TRITONSERVER_InferenceRequestDelete(request),
	"deleting ensemble inference request");
	if (request_tracker->DecrementCounter()) {
	delete request_tracker;
	}
	}
	};
	// Attempt to enqueue the callback. If all workers are busy and queue is at
	// capacity, execute the callback immediately.
	if (pool->TaskQueueSize() < pool->Size()) {
	pool->Enqueue(fn);
	} else {
	fn();
	}
	}

[oandreeva-nv]

how was this number picked and why this number?

[yinggeh]

The previous pool size 8 was explained here #429 (comment).
In a non-decoupled ensemble model, this PR will reduce half of async callbacks. Thus I reduce pool size by half to 4.

[oandreeva-nv]

My concern is how general those experiments were?

[yinggeh]

Could you elaborate? If thread pool queue is full, new task will execute synchronously like normal to avoid delay. See

core/src/ensemble_scheduler/ensemble_scheduler.cc

Lines 625 to 632 in 729422e

	// Attempt to enqueue the callback. If all workers are busy and queue is at
	// capacity, execute the callback immediately.
	if (pool->TaskQueueSize() < pool->Size()) {
	pool->Enqueue(fn);
	} else {
	fn();
	}
	}

[GuanLuo]

So I ended up reverting async ResponseComplete callback feature. But async RequestComplete callback is working fine.

Why RequestComplete is also made async at the first place?

[yinggeh]

So I ended up reverting async ResponseComplete callback feature. But async RequestComplete callback is working fine.

Why RequestComplete is also made async at the first place?

Explanation with visualization

From the chart, this composing model "preprocessing" backend thread takes 62 usec to complete, including 24 usec EnsembleContext::ResponseComplete and 7 usec EnsembleContext::RequestComplete.

Maximum throughput of sample ensemble_model comparison

Both synchronous (no optimization): 39794.5 infer/sec
Synchronous EnsembleContext::ResponseComplete and asynchronous EnsembleContext::RequestComplete (this PR): 47017.2 infer/sec