Segfault in nimlangserver when `-d:nimOptIter` is defined

[jmgomez]

Description

Download and build latest https://github.com/nim-lang/langserver
Remove the lock file and the requires from the nimlangserver.nimble so it uses the latest Nim to build.
Run nimble build
Open a nim project with the vscode extension (make sure the built nimlangserver is the one used)

It will crash with the following (you can see the output in the Nim Language Server tab or Nim Lsp if you are running a newer version)

Works fine with 2.0.8 and with 2.0.9

Traceback (most recent call last)
/Volumes/Store/Nim/lib/system/deepcopy.nim(190) genericDeepCopy
/Volumes/Store/Nim/lib/system/deepcopy.nim(132) genericDeepCopyAux
/Volumes/Store/Nim/lib/system/deepcopy.nim(72) genericDeepCopyAux
/Volumes/Store/Nim/lib/system/deepcopy.nim(68) genericDeepCopyAux
/Volumes/Store/Nim/lib/system/deepcopy.nim(153) genericDeepCopyAux
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

Nim Version

Nim Compiler Version 2.1.99 [MacOSX: arm64]
Compiled at 2024-09-10
Copyright (c) 2006-2024 by Andreas Rumpf

git hash: baec1955b5453ec71fc12355142dd9a813fa02fb

Current Output

No response

Expected Output

No response

Known Workarounds

No response

Additional Information

No response

[metagn]

When trying to reproduce I didn't get a stacktrace, only "out of memory".

[jmgomez]

hmm In what OS are you testing? Do you get that same issue in version-2.0 too?

There is proc raiseOutOfMem() {.noinline.} = maybe you could add a stacktrace there

[metagn]

hmm In what OS are you testing? Do you get that same issue in version-2.0 too?

Windows; no; with refc/arc it gives the segfault exit code (as opposed to orc). The above stacktrace should only be reachable in refc from my understanding since usrToCell is not declared otherwise.

I'm out of my depth here, I thought maybe a type graph change or cycle was the cause, I also don't know where a deep copy could be happening

[Araq]

@ringabout @metagn Please bisect it.

[jmgomez]

Windows; no; with refc/arc it gives the segfault exit code (as opposed to orc). The above stacktrace should only be reachable in refc from my understanding since usrToCell is not declared otherwise.

I got the stacktrace in MacOs. Yes, refc is used

[metagn]

Currently got as far as:

started with good 185e06c, bad a93c5d7
(1 commit good after disabling effect errors)
good 3203111 after disabling effect errors
good c23d6a3
good f126838

[metagn]

Bisected to 05df263 (commit history was messed up the first time), @yglukhov any idea what the problem could be?

[metagn]

The type that deepcopy is iterating over during the segfault is tyRef. #23787 changes the creation of the environment ref type here. This might not have anything to do with it, but I tried to stress the added condition by mixing proc/iterator closures in the same owner context. I noticed this behavior difference:

proc foo() =
  let x = 123
  iterator bar2(): int {.closure.} =
    yield x
  proc bar() =
    let z = bar2
    for y in z(): # just doing bar2() gives param not in env: x
      echo y
  bar()

foo()

The output of this is now 1 when it used to be 0. Both are invalid, it should be 123. This is the only difference I've found that gives a runtime difference versus failing to compile. There is no problem when a closure iterator calls a closure proc, only when a closure proc calls a closure iterator.

Changing foo to be a closure iterator crashes the compiler with all permutations now, but is different before #23787. This used to give a codegen error, now segfaults when getEnvParam calls getEnvTypeForOwner:

iterator foo(): int {.closure.} =
  let x = 123
  iterator bar2(): int {.closure.} =
    yield x
  proc bar() =
    let z = bar2
    for y in z():
      echo y
  bar()
  yield x

for x in foo(): echo x

And this used to work and give the correct output but now also segfaults:

iterator foo(): int {.closure.} =
  let x = 123
  proc bar() =
    echo x
  iterator bar2(): int {.closure.} =
    bar()
    yield x
  for y in bar2():
    yield y

for x in foo(): echo x

I wouldn't even know where to start to try and minimize the segfault in nimlangserver, but the first closure proc/iterator mixing example seems realistic to encounter in async code. It was broken before too but maybe it zeroing made it more managable.