Skip to content

Adds as_slice and friends to basic slice and vec iterators #1255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

lucascool12
Copy link

This commit adds the methods as_slice, as_mut_slice, into_slice and AsRef<[T]> on the appropriate iterators for slice iterators and vec iterators.
This allows custom producers to use these iterators to decide on custom split points based on the remaining data in the slice.

These impls are based on the std impls on std::slice::Iter, std::slice::IterMut, and std::vec::IntoIter.

This commit adds the methods `as_slice`, `as_mut_slice`, `into_slice`
and `AsRef<[T]>` on the appropriate iterators for slice iterators and
vec iterators.
This allows custom producers to use these iterators to decide on
custom split points based on the remaining data in the slice.
@cuviper
Copy link
Member

cuviper commented Jul 21, 2025

This allows custom producers to use these iterators to decide on custom split points based on the remaining data in the slice.

Do you have an example of code where you would like to use this?

Unlike an Iterator where actual work is done, ParallelIterator is kind of a passive starting point, and nothing is actually split until we convert to producers and consumers. I'm not sure rayon::slice::Iter et al. are helpful to custom code, because even with as_slice you can't really do anything with them yet. I usually have custom code just hold the direct slice and work with that.

These impls are based on the std impls on std::slice::Iter, std::slice::IterMut, and std::vec::IntoIter.

That's good, we like to match precedent from the standard library -- although IterMut::as_mut_slice isn't stable yet.

@lucascool12
Copy link
Author

Unlike an Iterator where actual work is done, ParallelIterator is kind of a passive starting point, and nothing is actually split until we convert to producers and consumers. I'm not sure rayon::slice::Iter et al. are helpful to custom code, because even with as_slice you can't really do anything with them yet. I usually have custom code just hold the direct slice and work with that.

I only just noticed this, so this doesn't actually help me accomplish what I'm trying to do.

I'm trying to create a parallel iterator of a join like iterator that works like this (assumes both iterators are sorted):

impl<I, J> Iterator for Pairs<I, J>
where
    I: Iterator<Item = u64>,
    J: Iterator<Item = u64>,
{
    type Item = (Option<u64>, Option<u64>);

    fn next(&mut self) -> Option<Self::Item> {
        match (self.left.peek(), self.right.peek()) {
            (None, None) => None,
            (Some(_), None) => Some((self.left.next(), None)),
            (None, Some(_)) => Some((None, self.right.next())),
            (Some(left), Some(right)) => match left.cmp(&right) {
                Ordering::Equal => Some((self.left.next(), self.right.next())),
                Ordering::Less => Some((self.left.next(), None)),
                Ordering::Greater => Some((None, self.right.next())),
            },
        }
    }
}

After thinking about it for a little while, I noticed that if you split one of the two iterators you must split the other iterator at a specific point, otherwise you produce duplicates.
This specific point is not super easy to find, you have to binary search for the first value higher than the last value in the first splitted iterator.
Knowing all the code that uses this uses iterators over slices, or stuff that acts like slices (IntoIter from vec), I can just use the as_slice method from them to search for this split point.
Well that is what I thought when I was theory crafting a solution.

But actually doing all of this with rayon seems impossible currently. Since the producer types of slices and vecs are all private so even if they have an as_slice method I'll never be able to call it, unless I'm missing something.
So I'm not really sure how to solve this specific problem with rayon, unless I copy and paste the producers (and then also the iterators) so I have them available but that is not really a solution.

Any help is greatly appreciated.

@cuviper
Copy link
Member

cuviper commented Jul 23, 2025

I can't think of a way to do that with generic ParallelIterator or IndexedParallelIterator, but if you start with the actual slices then I think you could use rayon::iter::split to implement this, which can be hidden in a proper impl like:

impl ParallelIterator for ParallelPairs {
    type Item = (Option<u64>, Option<u64>);

    fn drive_unindexed<C>(self, consumer: C) -> C::Result
    where
        C: UnindexedConsumer<Self::Item>
    {
        rayon::iter::split(self, |pairs| todo!("find your split points and do it"))
            .flat_map_iter(|pairs| todo!("convert into the serial Pairs"))
            .drive_unindexed(consumer)
    }
}

@lucascool12
Copy link
Author

Thanks for the suggestion, I still wanted it to be generic over anything that contains a slice, such as Vec.
So I experimented a little bit and found a solution. It does require a couple new methods and making producers public, but I'll create a separate issue for the required changes.

Regarding this PR, as you said this doesn't really help my use case at all, regardless if anyone thinks this would be beneficial I'll gladly continue working on this PR until it is ready to be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants