Understanding rayon workflow and trait implementations

[maeldroem]

Hello!

I'm having some trouble understanding how to implement rayon for my crate. ParallelIterator and IndexedParallelIterator is fine, but other traits like UnindexedProducer still puzzle me a bit.

The plumbing README sure helps understanding what are producers/consumers, and what are ProducerCallbacks, but how those structures interact with each other is still an issue for me.

I've tried looking up reverse dependencies of rayon to check how other crates have implemented rayon, but most of them either don't implement rayon but use it, or it seems that some crates are just going by feeling over some implementations. It may be wrong, and this may seem that way due to how confused I currently am, but anyways.

My goal would be to understand more of how the high-level API of rayon works so that I can create some graphics and add documentation to the crate so that we can increase the overall understanding for other developers that may have the same questions as me :)

Anyways, here's my current list of questions:

1. For IndexedParallelIterator::with_producer(), where does the callback come from? and what does it call back to?
2. For UnindexedProducer::fold_with(), why is it the responsability of UnindexedProducer to use a given Folder? What should implementors do with this method?
3. Related to question 2, is the return value of fold_with() a full version of the given Folder?
4. How does rayon deal with iterators that have a Peekable iterator of their base?
5. What is the role of a Folder when rayon processes a chain of iterators? When does it come into action?
6. What is the role of a Reducer when rayon processes a chain of iterators? When does it come into action? Also, its documentation says to "see the plumbing README for further details", but the readme doesn't contain information about Reducers.
7. Since an iterator can conserve a state that may affect its future iterations (for example, an iterator that has a logic which can output more than 1 element on each iteration will most likely have a queue of results that it will try to empty before resuming its consumption of its base iterator), how does rayon deal with such cases? It seems that Reducer is made to resolve such cases, but where does it come into action?
8. Related to question 7, when an iterator requires a special processing when reduced/folded, where do we implement those key processes? when implementing ParallelIterator and IndexedParallelIterator, the only required method that touches to consumers is drive() (or drive_unindexed()), which often consist of simply calling bridge() or bridge_unindexed() and passing the consumer argument to those functions.
9. (Lesser priority) Why does the Producer trait has a required method for converting the implementor into an iterator? it seems to clash with std's IntoIterator trait with added trait bounds to what the returned iterator has to satisfy. Why are those extra trait bounds required? and why not instead use IntoIterator as trait bound on Self with the additional (double-ended iterator, exact length iterator) trait bounds be added on what the IntoIter associated type of IntoIterator may be?

If you have any questions about my questions, don't hesitate! I want to be sure I understand how rayon works as it seems additional documentation could be very helpful to other developers wanting to make their custom iterators parallel and work with rayon's capabilities.

Thank you in advance for your answers and your time 😃