Replies: 1 comment
-
Allowing the output buffer to come from the caller is no easy task however. How should we carry it? As a part of Dict? Possible but it's not clean. Dicts will have to be treated as I/O arguments of sorts. Maybe an optional additional argument to ops with buffers... or even a buffer factory? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
@pminev observed that we seem to be doing excessive copying of binary data in
ac::Dict
.Currently it's stored in
std::vector<uint8_t>
typedef-ed asac::Blob
.While blobs can be moved in and out of
Dict
the most obvious problem is that if the user produces data which is notstd::vector<uint8_t>
, but saystd::vector<float>
, they will have to copy it into theDict
(copying further down the line, when the inference code consumes the data can be avoided as there we can control the lifetime of theDict
itself).This become a non-trivial problem if the data is large. I'd say that our current use case of audio is just on the edge of size which is annoying to have to copy. If we ever do video inputs, this copy would really be frustrating.
Luckily the blob type is a customization point of
nlohmann::json
and we can place practically anything there and adapt it through a specialization ofbinary_t::container_type
.To me allowing straight-up refs as binary data is a definite no. We can't impose the lifetime management of the data to the user.
A relatively easy solution which would allow us to move any type of data into a
Dict
would be to useitlib::pod_vector
or other type-punnable or recastable containers that may exist. This however is a miniscule improvement as it would still impose the use ofitlib::pod_vector
(or something else concrete) as a container. Moreover it would allow data to be moved inside theDict
but if the user wants to keep the data for longer (seems like a common use case as they would likely want to display the input) they will still have to copy it.A more powerful solution would be to design our own buffer container type which can hold multiple things through
variant
or evenany
and use ashared_ptr
of it as a Blob. This is a decent solution if C++ is the only target language.However it is not. We expect buffers which come from Java, Swift, and many other languages. For a complete solution here, we need to adapt further. We need to work with external ref counts.
So, we would have to create our own customization point for blobs which allow them to take and adapt pretty much anything to a buffer (dynamic polymorphism, yay).
Edit:
Two more things come to mind:
Dict
is mutable so given that we have CoW, we should allow the output buffer to come from the caller.Beta Was this translation helpful? Give feedback.
All reactions