BLS scripting: executing / submitting multiple requests at once in a single-shot?

[vadimkantorov]

Does BLS support submitting multiple InferenceRequest's at once as an API? (to save maybe some gRPC roundtrips and networking API-related thread blocking) Or is there no point in this?

And also, is there support in InferenceClient of submitting multiple requests to the same model as a single gRPC roundtrip packing them into a single gRPC request to reduce overhead?

At least from DX/UX standpoint, this would be a useful regime to support at least in frontend methods IMO... and could be useful for text models: e.g. BERT-based classification of multiple short strings

Thanks!

[vadimkantorov]

Currently I'm constructing individual InferenceRequest and .exec() or async_exec()'ing them one by one, but it would be very nice to submit a handful of these to the same model