Collect sections in calls and emit them as document symbols

[lionel-]

Branched from #866.
Addresses posit-dev/positron#8402.

You can now add sections in function calls too, between arguments.

The implementation is shared with the section logic for {} blocks. This works a little differently than in RStudio: The sections are nested in the call and can't affect the section hierarchy outside the call. If you have:

## level 2 ----

call(
  # level 1 ----
)

call({
  # level 1 ----
})

The "level 1" sections in the block and call do not close the top-level "level 2" section. This difference is necessary because the LSP outline is more powerful and includes syntactic elements like function definitions and assigned variables in the outline. Allowing sections to close from inside a nested element would make things complicated and I'd argue respecting the nesting of the code makes sense from a user pespective too.

QA Notes

Sections in calls:

# level 1 ----

list(
  ## foo ----
  1,
  2, ## bar ----
  3,
  4
  ## baz ----
)

## level 2 ----

list(
  # foo ----
  1,
  2, # bar ----
  3,
  4
  # baz ----
)

now work the same way as sections in blocks:

# level 1 ----

list({
  ## foo ----
  1
  2 ## bar ----
  3
  4
  ## baz ----
})

## level 2 ----

list({
  # foo ----
  1
  2 # bar ----
  3
  4
  # baz ----
})

Screen.Recording.2025-07-07.at.15.31.07.mov

[DavisVaughan]

Blocking so we can at least discuss my comment above together before merging

[DavisVaughan]

So I guess you view a function call as a kind of "reset" of the nesting level right?

I know you mentioned an example like this in your description, but explicitly compare positron and rstudio here:

At first glance I thought RStudio made more sense here, as it seems to only rely on the number of # to determine how the Outline works.

But then I did this, and I think I like how Positron works here more

It's like you're saying:

"The call() implicitly starts an H2 ## nesting level under level 2, but when you write sections in call(), you can "start over" with a single #"

I don't know how to precisely describe that feature. It kind of feels right, but it's also fairly weird to see # level 1-a and ## this show up at the same level in the outline, because they have a differing number of leading #. It makes it kind of hard to predict?

I also wonder if differing from RStudio too much here is going to make the outline annoying when coming from RStudio with a big {targets} script...

[DavisVaughan]

Actually I take it back that I like Positron more in the second example.

I think having # level 1-a show up under # level 2 is too confusing.

I really think the number of # should drive the nesting level, as that is very easily predictable (possibly it is the only thing driving the nesting level?)

i.e. I think letting the number of # drive the nesting level would lean into the principle of least surprise here.

It also seems like it would align with RStudio more.

[juliasilge]

I also think we should have behavior more similar to RStudio for this.

[lionel-]

I could maybe see how it's technically possible to close nesting syntax from a section header, at the expense of making the implementation convoluted and confusing.

But let's think through the UX implications of this. We could be arbitrarily deep in a syntax tree through layers of calls, function definitions, and blocks. E.g. deep in an R6 method, inside an if branch inside lapply. Then someone adds a section down there that starts with # by mistake and all of a sudden the rest of the tree, including all following R6 methods, is pulled at top level? This feels confusing and hard to use to me. It really goes against the nesting nature of syntax and tree invariants.

[DavisVaughan]

I think your argument will be more convincing if you can provide a video in RStudio of what would confuse you, because I'm having trouble following

[DavisVaughan]

Ok @juliasilge, @lionel- and I talked about this quite a bit and we now think we have a good mental model that describes how this should work.

@lionel- if you could find some way to encode some of the theory into docs for this, that could be helpful for us in the future when thinking about this

RStudio is not the best reference guide

We thought we understood how RStudio works, but we actually don't, and we don't think this behavior is right. It's very odd for # bar to be under ## foo

# alone are not enough

I briefly mentioned above that we should let the number of # fully drive the outline, but I no longer believe that based on what @lionel- is trying to convey here #867 (comment)

If we truly believed that, then consider this example

# top level ----

class <- R6::R6Class(
  'class',
  public = list(
    initialize = function() {
      'initialize'
    },
    # middle ----
    foo = function() {
      # inner ----
      1
    },
    bar = function() {
      
    }
  )
)

The outline we'd build based on that principle would either look like this if we didn't collect any structure from the code elements themselves, which is pretty weak for a code outline...

- top level
- middle
- inner

...or like this if we did, but this feels deeply broken. # middle should be fully contained under the class <- part of the outline, but it "escapes" up to top level using this principle. That's not right.

- top level
  - class
    - initialize
-  middle
  - foo
- inner
  - bar  

RStudio doesn't handle this particularly well, again suggesting it is not great as a reference here. It has the "escaping" problem referenced above.

Defining sub-documents

So starting from first principles, here's what we've come up with. In markdown we have a strong notion of how # H1, ## H2, etc interact with each other. When you add code elements into the picture, we need a little bit more terminology to understand how it should work - it's simply not something that exists in native markdown.

Imagine the whole file as the root Document. Within this document, # H1 and friends work as you'd expect.

When you do one of the following, you enter a totally new nested Document:

• Start a { scope, i.e. function() { or expr({
• Enter a function call, i.e. fn(

Everything from { -> } or ( -> ) should be treated as its own standalone md file that gets nested into the parent Document. The only thing this scope inherits from the parent Document is the indent level to start at, i.e. here:

# top level ----

list(
  # section ----
  fn()
)

# top level 2 ----

• The file starts a Document, the indent level starts at 0
• # top level --- goes in the outline at indent level 0 and starts a scope where the indent level is 1
• list( starts a new Document inheriting indent level 1
• # section --- goes in the outline at indent level 1 and starts a scope where the indent level is 2
• At the ), the # section --- scope closes and we drop to indent level 1, also the inner Document ends
• At # top level 2 ---, the # top level --- scope closes and we drop to indent level 0, the # top level 2 --- goes in the outline at indent level 0, and starts a scope at indent level 1

That reasoning allows the R6 example to make sense as well

Each function call or { you see starts a new Document which inherits the starting indent level from the parent, but the actual usage of # H1 and friends is scoped to just that Document.

In practice, with {targets}

Theory aside, I'm mostly convinced this will work quite well with existing targets scripts, which is really the most important thing here.

I think targets users would do one of two things:

1. They have a top level # section, followed by ## sections inside their list() call, which makes the nesting work with RStudio. This also works out of the box with Positron with this PR.

2. They don't have a top level # section, and instead have # sections inside their list() call. That also works out of the box with this PR.

The cool thing this PR lets you do is also drop the ## when you have a top level section if you understand that the call to list( opens a new Document that inherits the parent nest level but otherwise functions as a standalone md file. I think this is cool!

[lionel-]

Thanks for this very clear summary @DavisVaughan!

Key things:

• In Ark, each level of syntax nesting creates a new "document" with independent sections
• Ark has always been doing this without complaints from users, we're just extending the notion to calls
• RStudio actually does something similar to this in some cases, albeit in a buggy way

The cool thing this PR lets you do is also drop the ## when you have a top level section if you understand that the call to list( opens a new Document that inherits the parent nest level but otherwise functions as a standalone md file. I think this is cool!

I'll add that this is a big ergonomic advantage: While you're deep in the code, you can add sections without having to worry about the current markdown structure. I.e. let's say we have the hypothetical implementation where headers close sections across the syntax tree. In this example I'm adding nested section and need to double check the current heading structure to avoid resetting the outline. I see I need to add a level 2 section to nest it within the level 1 header:

# Level 1 ----

...

imagine_a_deep_nesting({
  ## I'm adding a level 2 section here ----
  ...
})

If I then change the top-level structure and forget to adjust headers in nested elements, I'll unexpectedly get the confusing reset of outline in the middle of my code:

# Level 1 ----

## New Level 2 ----

### New Level 3 ----
...

imagine_a_deep_nesting({
  ## Oh no! My section is now closing level 3 ----
  ...
})

With the approach implemented here, where each level of syntax nesting creates a new "document", I can start a new hierarchy and have it consistently nested, no matter what actual header hierarchy is at top-level. I can copy-paste the code anywhere and the outline will still work the same.

# Level 1 ----

## Level 2 ----

imagine_a_deep_nesting({
  # This section is nested in top-level headers no matter what ----
  ...
})

[juliasilge]

Thank you for the detailed explanation on this! ❤️

I agree that the targets use case is the most important one to make sure we get right here, based on how people are talking about using this feature. I'm a little uncomfortable straying from the RStudio behavior given that we (RStudio/Posit) basically made up this feature so if there is any norm to align with, that would be it, but I am happy to defer here as long as we get those targets users supported.