Proposal: Query Conclusions for LINQ Query Syntax

[tuespetre]

This is closely related to #333, but given the length of that issue (even just within the first post), the amount of deliberation therein, and the differences between the ideas therein and those within my proposal, I felt it would be appropriate to submit this proposal as a separate issue.

The motivation for this issue is the same as that of #333: how can we 'stay within' query syntax while gracefully making use of various non-language-integrated query operators, all while avoiding the need to introduce new syntax for each of the said operators?

Limitations of query expression syntax today

Suppose you had three sequences of objects, each sequence being of a different element type, and you needed to obtain a projection from each of these sequences, concatenate the resulting projected sequences, filter the concatenated sequence, project that sequence, take elements from the projected sequence while a certain condition was satisfied, and produce a list of distinct results.

Such a query can be written using method syntax in a single statement like so:

// Method syntax
var result
    = aseq
        .Select(a => a.MethodA())
        .Concat(bseq.Select(b => b.MethodB()))
        .Concat(cseq.Select(c => c.MethodC()))
        .Where(i => i.HasValue)
        .Select(i => i.Value)
        .TakeWhile(i => i < 6)
        .Distinct()
        .ToList();

However, in order to write such an expression using query syntax in a single statement, something like the following must be written:

var result
    = (from _ in Enumerable.Repeat(1, 1)
       let aq = from a in aseq select a.MethodA()
       let bq = from b in bseq select b.MethodB()
       let cq = from c in cseq select c.MethodC()
       let concatenated = aq.Concat(bq).Concat(cq)
       from i in concatenated 
       where i.HasValue 
       select i.Value).TakeWhile(i => i < 6).Distinct().ToList();

This poses a number of issues:

• Readability, in the opinion of some including myself, is hurt by not being able to 'stay within' query syntax and having to abuse Enumerable.Repeat to begin the query expression.

• The (ab)use of let clauses in this manner results in unnecessary allocations for the anonymous 'transparent identifier' classes, which could have a considerable effect on performance under other circumstances (nested query expressions, etc.)

• The root of the query is an IEnumerable and not an IQueryable (and for nested query expressions, query providers typically will not translate Enumerable.Repeat), so if aseq, bseq, and cseq are IQueryables, so the author would likely end up splitting up this single statement by moving aq, bq, and cq out into standalone variable declarations and writing the rest of the query in method syntax anyways.

Query conclusions

Today, the query expression syntax offers a construct called the 'query continuation', which effectively takes the resulting elements of a select or group clause, re-introduces them under the guise of a new range variable, and begins a new scope with no access to range variables defined prior to the query continuation.

In order to address the sticking points from the previous example of the limitations of LINQ query expression syntax, I am proposing a new but similar construct called a 'query conclusion'. A query conclusion 'concludes' a query expression, meaning no other clauses or continuations may follow it. It starts a new scope and creates a new identifier similar to a query continuation, although the identifier that is introduced refers to the IEnumerable<> or IQueryable<> result of the preceding select or group clause.

The following example demonstrates the usage of query conclusions as applied to the previous example:

var result = from a in aseq select a.MethodA() yield into aq 
             do from b in bseq select b.MethodB() yield into bq 
             do from c in cseq select c.MethodC() yield into cq 
             do from x in aq.Concat(bq).Concat(cq) select x yield into concatenated 
             do from i in concatenated where i.HasValue select i.Value yield into f
             do f.TakeWhile(i => i < 6).Distinct().ToList();

As can be seen in this example, the syntax is highly composable and can effectively be used to avoid declaring unneeded local variables and let clauses, while allowing code authors to lift nested query expressions out of parentheses and even reorder them as desired to bring clarity to the overall query expression. The syntax should also behave well with existing code formatting standards, such as the alignment of query clauses with the beginning from clause:

var result = from a in aseq 
             select a.MethodA() yield into aq 
             do from b in bseq 
                select b.MethodB() yield into bq 
                do from c in cseq 
                   select c.MethodC() yield into cq 
                   do from x in aq.Concat(bq).Concat(cq) 
                      select x yield into concatenated 
                      do from i in concatenated
                         where i.HasValue 
                         select i.Value yield into f
                         do f.TakeWhile(i => i < 6).Distinct().ToList();

Proposed changes to the formal query expression syntax

query_body
-    : query_body_clauses? select_or_group_clause query_continuation?
+    : query_body_clauses? select_or_group_clause query_continuation_or_conclusion?
    ;

+query_continuation_or_conclusion
+    : query_continuation
+    | query_conclusion
+    ;

+query_conclusion
+    : 'yield' 'into' identifier 'do' expression
+    ;

Notes regarding the proposed syntax for query conclusions:

• By leveraging the existing yield, into, and do keywords, no new keywords are required.

• Given the existing syntax for query continuations and the existing syntax for yield return, the proposed syntax offers some familiarity and seems to flow naturally.

• A query conclusion is said to have an identifier and an expression.

• A query conclusion's identifier is in scope within the entirety of the query conclusion's expression.

• At compile time, all references to a query conclusion's identifier are 'expanded' or 'replaced' with the translated version of the query expression it concluded. This is performed on an in-order basis in relation to the rest of the query expression.

• At compile time, a query conclusion is translated into an immediately-invoked lambda function following the signature of Func<TArg,TResult>, where the argument passed in is the containing query expression.

Appendix: compilation example

var before = from a in aseq select a.MethodA() yield into aq 
             do from b in bseq select b.MethodB() yield into bq 
             do from c in cseq select c.MethodC() yield into cq 
             do from x in aq.Concat(bq).Concat(cq) select x yield into concatenated 
             do from i in concatenated where i.HasValue select i.Value yield into f
             do f.TakeWhile(i => i < 6).Distinct().ToList();
                      
var after = 
    ((Func<IEnumerable<A>,List<int>>)(IEnumerable<A> aq) => 
        ((Func<IEnumerable<B>,List<int>>)(IEnumerable<B> bq) => 
            ((Func<IEnumerable<C>,List<int>>)(IEnumerable<C> cq) => 
                ((Func<IEnumerable<int?>,List<int>>)(IEnumerable<int?> concatenated) => 
                    ((Func<IEnumerable<int>,List<int>>)(IEnumerable<int> f) => f.TakeWhile(i => i < 6).Distinct().ToList())
                    (concatenated.Where(i => i.HasValue))
                )(aq.Concat(bq).Concat(cq))
            )(cseq.Select(c => c.MethodC()))
        )(bseq.Select(b => b.MethodB()))
    )(aseq.Select(a => a.MethodA()));

[svick]

I don't understand the point of introducing CQEs. What's wrong with just creating some local variables before the query? I.e. this code with CQEs:

var result = (using aq = from a in aseq select a.MethodA()
              using bq = from b in bseq select b.MethodB()
              using cq = from c in cseq select c.MethodC()
              using concatenated = aq.Concat(bq).Concat(cq)
              from i in concatenated 
              where i.HasValue 
              select i.Value).TakeWhile(i => i < 6).Distinct().ToList();

could instead be written as:

var aq = from a in aseq select a.MethodA();
var bq = from b in bseq select b.MethodB();
var cq = from c in cseq select c.MethodC();
var concatenated = aq.Concat(bq).Concat(cq);

var result = (from i in concatenated 
              where i.HasValue 
              select i.Value).TakeWhile(i => i < 6).Distinct().ToList();

[tuespetre]

Apart from the aesthetic appeal of containing such expressions within a query expression when they have no purpose outside of it, a CQE clause's expression is free to reference range variables, CQE identifiers, and conclusion identifiers preceding it in the query expression tree, which allows scenarios within nested query expressions that could not be reproduced by moving the expressions out into a standalone variable.

Of course, I care more about the aesthetic component. The point of query expressions is to be able to express a query, is it not? 😉

The primary motivation is to offer a more palatable experience for using query expression syntax with operators such as Concat, Intersect, Union, Except, Zip, and SequenceEqual, or any custom operators that combine two or more sequences. Everything past that is gravy.

[scalablecory]

you can use locals or let to capture a query already -- no workarounds necessary. I don't see how adding this new construct would improve on these.

[Joe4evr]

OP wants to rescue the princess himself.

But yeah, as scalablecory said, a little creative use of the black magic that is let will already do what you're asking for.

[svick]

@tuespetre

a CQE clause's expression is free to reference range variables, CQE identifiers, and conclusion identifiers preceding it in the query expression tree, which allows scenarios within nested query expressions that could not be reproduced by moving the expressions out into a standalone variable

Do you have an example showing that?

[tuespetre]

@svick I hate it when people ask me to show my work. 😉 You're right of course, I missed the fact that by the time a CQE clause could appear in a nested query expression, a let clause in the outer query expression could have achieved the same result, barring details such as the timing and number of evaluations of multiple references to the CQE vs the let clause due to the difference in semantics (expansion vs projection.)

I still find the idea of the construct to be convenient, because as @Joe4evr said, I'm all about rescuing me some princesses 💪 🍄 🏰 💃

Does anyone who has commented so far have any opinion on the other half of the proposal?

[miloush]

I like the into f do f.Whatever() syntax more than the floating unqualified methods in #333

[tuespetre]

@svick @Joe4evr @scalablecory

After sleeping on it I'm realizing that what I wanted to accomplish with CQE clauses could just as easily be accomplished with query conclusions as well, so this:

var result = (using aq = from a in aseq select a.MethodA()
              using bq = from b in bseq select b.MethodB()
              using cq = from c in cseq select c.MethodC()
              using concatenated = aq.Concat(bq).Concat(cq)
              from i in concatenated 
              where i.HasValue 
              select i.Value).TakeWhile(i => i < 6).Distinct().ToList();

Can be written like this:

var result = from a in aseq 
             select a.MethodA() yield into aq 
             do from b in bseq 
                select b.MethodB() yield into bq 
                do from c in cseq 
                   select c.MethodC() yield into cq 
                   do from x in aq.Concat(bq).Concat(cq) 
                      select x yield into concatenated 
                      do from i in concatenated
                         where i.HasValue 
                         select i.Value yield into f
                         do f.TakeWhile(i => i < 6).Distinct().ToList();

Which could be stylized in various ways, including:

var result = from a in aseq select a.MethodA() yield into aq 
             do from b in bseq select b.MethodB() yield into bq 
             do from c in cseq select c.MethodC() yield into cq 
             do from x in aq.Concat(bq).Concat(cq) select x yield into concatenated 
             do from i in concatenated where i.HasValue select i.Value yield into f
             do f.TakeWhile(i => i < 6).Distinct().ToList();

With that in mind, I agree 100% that CQEs are unnecessary. I am going to modify the proposal to remove them.

[tuespetre]

Here is a prototype: https://github.com/tuespetre/roslyn/tree/query-conclusions

[aluanhaddad]

This would add significant value and it would encourage some people to use query comprehension syntax more often. Some developers really are stubbornly insistent on writing GroupJoins using method syntax simply because they call ToList on the result.

There are a lot of other reasons to want this as well.

I'm not a huge fan of the proposed syntax but just having a syntax for this would be great.