ToSpan for Linq?

[WhiteBlackGoose]

Hello.

I think it might be useful to have method ToSpan for Linq. It will not add a copy overhead like there is in .ToArray(), as it will be a span of the list's internal array. So essentially it would be

ienumerable.ToSpan()

<=>

CollectionMarshal.AsSpan(ienumerable.ToList())

So it behaves similarly to ToArray(), but fits high-perf scenarios better.

What do you think?

[Clockwork-Muse]

Span doesn't have its own internal storage; it's simply a way of abstracting over (several different ways to access) memory.
LINQ has no "saved" internal working state (and most of the ones built when an output operator is called aren't arrays or Lists, anyways), so there's nothing to create a Span over.
If you're in a high performance scenario where using a Span results in a measurable difference, it's unlikely you'd be calling (most of the) LINQ operators on a given collection, anyways.

CollectionMarshal.AsSpan(someList) isn't a copy operation, it returns a view of the underlying backing array (which is why the documentation of the method says you shouldn't modify the list while the Span is in use). Given that information, does that method actually work for your purposes?

[WhiteBlackGoose]

Spans can also only represent contiguous blocks of memory, but an arbitrary LINQ chain can be non-linear, so creating a Span from that will require copying that data anyway, so you haven't saved anything at all.

I have saved a copy. ToArray makes two copies (or actually n + n + n/2 + n/4 ...), and ToSpan would do one (n + n/2 + n/4 ...).

To clarify more, consider something like Enumerable.Range(0, 10). This doesn't create an array or list - it create an IEnumerable implementation that stores the start and the end. Even when you create an IEnumerator - such as when enumerating via foreach - it doesn't create a collection of any sort, and instead the enumerator increments the "current" number. In order to have a collection of some sort - at all - you need to realize it via ToArray() or ToList() (or one of the other types not relevant here).
At that point, getting a Span over the output collection will be a non-copy operation, but the calling code will need to retain ownership of the output collection to keep it in memory.

There's no contradiction again. At some point, for instance, to make your sequence indexable, you will need to fill with it a continuous piece of memory. That's what ToList and ToArray do. ToList and ToArray exist for a reason, and I'm saiyng that ToSpan might be more useful in some cases than both ToArray and ToList. Neither ToArray nor ToList are lazy, both of them do make a copy. I wasn't talking about any other linq methods.

[Joe4evr]

ToArray makes two copies (or actually n + n + n/2 + n/4 ...), and ToSpan would do one (n + n/2 + n/4 ...).

Wat.

Neither ToArray nor ToList are lazy, both of them do make a copy. I wasn't talking about any other linq methods.

But your hypothesized ToSpan cannot be lazy in the first place, it requires the collection to be materialized by way of either ToList or ToArray, so you gain nothing from this function in the first place because both those can be converted to spans easily enough (arrays implicitly/explicitly and lists via CollectionsMarshal).

[WhiteBlackGoose]

Of course it cannot be lazy, I'm responding to you mentioning lazy methods. All of ToArray, ToList, ToSpan are non-lazy.

gain nothing from this function in the first place because both those can be converted to spans easily enough (arrays implicitly/explicitly and lists via CollectionsMarshal

Yes, it can be done very easily by adding a single line code. See the second reply in this discussion (from svick), I haven't justified the presence of the method in BCL. But in our dialog I was trying to explain that the method itself does make sense for some scenarios.

[Clockwork-Muse]

ToArray makes two copies (or actually n + n + n/2 + n/4 ...), and ToSpan would do one (n + n/2 + n/4 ...).

Wat.

.... I think the attempt here is to remove the need to - at least in the array case - shorten the array to the exact number of elements found in the case of over-allocation.
This is not a compelling case, because ToList() almost certainly doesn't trim extra elements.

[Joe4evr]

I think the attempt here is to remove the need to - at least in the array case - shorten the array to the exact number of elements found in the case of over-allocation.

But in many cases, ToArray() can allocate an exact-sized array immediately by way of the IIListProvider optimization. No over-allocation, only a single copy of all the elements.

[svick]

What makes you think that the cases where the (usually not so great) performance of LINQ is acceptable, but the fairly small difference between ToList() and ToSpan() would be significant are so common that it makes sense to add this to .Net?

(I realize that I am somewhat repeating the previous discussion, but I though explicitly asking like this is still worthwhile.)

Especially consider that if it does often make a difference in your codebase, you can easily write this one-liner extension method yourself.

[WhiteBlackGoose]

Nothing makes me think so, that's why I put it on discussion instead of filing a proposal, to see if it's interesting for others.