Support for strided reads/writes?

[Jacobfaib]

I currently have data stored in a collection of std::mdspans which may or may not cover a dense region in memory. Since I can't guarantee the memory region is dense (and pread() and pwrite() don't seem to allow me to pass the strides on to kvikio) I make use of an intermediate buffer:

auto tmp_buffer = std::unique_ptr<T[]>{new T[my_mdspan.size()]};

copy_into_1d_buffer(my_mdspan, tmp_buffer);

kvikio::FileHandle{...}.pwrite(tmp_buffer.get(), ...);

Or visually:

    mdspan.          ->  tmp_buffer
[ x x x x o o o o ].    [ x x x ... x x x ]
[ x x x x o o o o ].
[ x x x x o o o o ].
[ x x x x o o o o ].
[ x x x x o o o o ].

where x is the data I want to copy, and o are the (potential) holes in the buffer straddled by the mdspan.

Is there explicit support for strided reads/writes? Or if not, perhaps a "blessed" workaround for doing this more efficiently? I can't imagine the temporary buffer strategy is the best way to go here.

[kingcrimsontianyu]

Thanks for raising this issue.

Your workaround seems a reasonable approach. I think profiling is the best way to tell if the overhead of extra memory copies is a real concern or not compared to the I/O time. To make memory copies faster, consider using threads.

Vectorized I/O is a suitable candidate for the strided pattern, but KvikIO doesn't have a multithreaded implementation yet. There are single-threaded system call preadv/preadv2 that you could tinker with but I doubt if it could go faster than the memory copy + KvikIO pread approach.