Skip to content

Feature Request: Seekable Buffer ("SeekBuf") and cursor/iterator support #657

Open
@dylanplecki

Description

@dylanplecki

(note that this is a somewhat large feature request with significant functionality additions)

While using the bytes crate, I noticed a use case that is not currently covered under the existing Buf type. Specifically, when returning non-contiguous chunks of memory, there is no ability to "look ahead" of the current chunk without also advancing the buffer and losing access to the current chunk. This is somewhat possible using the chunks_vectored method on Buf, but only in std environments and without a guarantee that it's implemented by the buffer (i.e. the default impl only allows partial access to the buffer).

In a network programming sense, it may be useful to scan further than the first chunk without consuming the original chunk of data, i.e. when parsing an HTTP header across a ring buffer (like VecDeque).

One way to implement this using the currently available interfaces is to have a Buf type that also implements Clone. The user could then simply advance a new, cloned buffer without advancing the original. But with this, the user is unaware whether Clone would clone the underlying buffer (an expensive operation) or just a cheap reference to a buffer. It is also not clear whether mutations to the new clone would alter the original (i.e. via an Arc). Buf does not provide any way to distinguish between an owned memory region and a reference to a memory region, so there is no way for an existing Buf to be iterated past the first chunk without possible mutations or copies for all existing implementations.

Any change to Buf to make it able to seek to any arbitrary point in the buffer would likely be backward-incompatible, since current implementations of Buf only provide access to the first chunk. So I propose adding a new parent type, SeekBuf, that also implements Buf but adds two new additional methods: chunk_from to get a chunk beginning at some offset in the buffer, and chunk_to to get a chunk ending at some offset in the buffer. Any type that implements Buf can also implement SeekBuf if it supports these features.

When a buffer type implements SeekBuf, we can also use a new BufCursor type to iterate and seek within the buffer without consuming its contents. This new cursor type can implement Iterator, Buf, and SeekBuf to provide full immutable access to the underlying buffer memory, useful for parsing, lookahead, search operations, etc. It can also be used to provide highly-tuned operations across any memory buffer, such as selective SIMD operations that provide search-like behavior (i.e. in HTTP parsing).

SeekBuf will be strictly a superset of the functionality of Buf and will not be automatically implemented for any type which already implements Buf, thus requiring opt-in support from consumers of this crate.

I have already prepared a PR for this feature and will attach it to this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions