SN 0.5.6 -> 0.5.7 -> 0.5.8

[durban]

Notes so far:

• There is something weird going on with exception stacktraces; I haven't been able to minimize so far. UPDATE: seems to work fine in CI, so possibly something is broken in my local environment.
• ioAppTestsNative fails on macos-14 and ubuntu-22.04-arm

[durban]

Okay, I think I can turn on and off the (probably) segfaults with a scala-native config flag 😀

generateFunctionSourcePositions(true) // ioAppTestsNative fails in CI, and locally I get incorrect stacktraces
generateFunctionSourcePositions(false) // neither of those things happens

I can see, that stacktraces are computed in a different way if that flag is true (and this seems new in 0.5.7). My best guess is that that different way sometimes segfaults 🤷‍♂️ (and also causes incorrect stacktraces for me locally). Also, this seems related: scala-native/scala-native/pull/4094

[armanbilge]

Green CI, nice work!!

[durban]

Current status:

• generateFunctionSourcePositions(true) makes ioAppTestsNative fail on ubuntu-22.04-arm (I think, although lately I haven't seen it) and macos-14 (definitely). I believe both of those are ARM64. With false, everything is green.
• (true also causes incorrect exception stacktraces for me locally, but I also can't minimize this.)
• The CI failures are usually "segfault-like" (i.e., SIGSEGV or SIGBUS), although sometimes just exit code 1 (I'm not sure what's that about).
• -fcxx-exceptions / -fno-cxx-exceptions doesn't seem to make a difference either way.
• If I disable tracing in CE (i.e., hardwire stackTracingMode = "none"), everything is green, even with generateFunctionSourcePositions(true). (Of course, to do this, I had to disable a few tests which need tracing to work. This was just an experiment, I'll obviously rollback this.)

Based on these, I can't definitively prove, but I suspect this could be a scala-native issue. I can see, that stacktraces are calculated in a different way in 0.5.7 if generateFunctionSourcePositions is true. My best guess is that that different way sometimes segfaults 🤷‍♂️. (Also, this seems related: scala-native/scala-native#4094.)

[durban]

I think it's scala-native/scala-native/issues/4366 (more details there).

[djspiewak]

Okay so I think what we're going to do is release 3.7 (or at least, an RC) against 0.5.6, just to unblock the ecosystem and move the train forward. Excellent digging btw, @durban.

[durban]

Good news, everyone! It seems we can avoid scala-native/scala-native/issues/4366 by turning on the SN optimizer (see details there, tl;dr: alwaysinline isn't).

Apparently the optimizer being on is the default, and funnily we probably turned it off to make multithreading work easier. I don't know why would anyone want to turn it off (okay, I obviously know...), but we might just say in the release notes: "don't do that"?

[durban]

Also, I can now confirm that my fix for the SN 0.5.7 segfaults (scala-native/scala-native#4301) was indeed correct. (Due to the new segfaults in 0.5.8 that wasn't certain until now. But now I've tried (#4447), and 0.5.7 also segfaults with the optimizer on.)

[lolgab]

true is the default, so you can remove the line altogether.

Suggested change

.withOptimize(true)

[durban]

As long as stuff is broken with false, I'd prefer to be explicit. Also, to remove that line would mean removing the explanatory comment (you've reviewed an outdated commit).

[djspiewak]

Somehow I missed the fact that turning the optimizer on fixes this. Great news! @durban should we unwip this and merge the upgrade to 0.5.8?

[durban]

Yeah, I'm gonna merge 3.x into this branch, to be sure. If CI is green, I think it's good to go.

[djspiewak]

🎉 All that work for 6 lines lol

[kubukoz]

should this be recommended somewhere in a "scala native" docs page?

[durban]

Uhh... thanks, that's an outdated comment. We can use true in 0.5.8, I've fixed it. I'm gonna remove that false and the comment...

[kubukoz]

hell yeee

[kubukoz]

oops just saw the failing test

[durban]

There is a failure I don't remember seeing before 😠
https://github.com/typelevel/cats-effect/actions/runs/16481182824/job/46595663830#step:22:1633

[djspiewak]

Annoying. We'll have to be sure to experiment downstream (fs2 and http4s in particular) with whether this same configuration is required.

[durban]

To summarize: I thought I've fixed things (in SN) to work with generateFunctionSourcePositions(true), and I've definitely fixed something (scala-native/scala-native#4301). However, now there is a "new" error with that option. This is not a segfault, but a ClassCastException.

Interestingly, with generateFunctionSourcePositions(false), the error disappears. So the situation is that we need both withOptimize(true) and generateFunctionSourcePositions(false) to have a green CI.

We might document that as a workaround, and leave it at that. But I'm bothered by the details of this "new" error:

Error: Exception in thread "main" java.lang.ClassCastException: scala.scalanative.runtime.dwarf.DWARF$SubprogramDIE cannot be cast to scala.scalanative.runtime.Array
	at scala.scalanative.runtime.package$.throwClassCast(package.scala:189)
	at scala.scalanative.unsafe.package$UnsafeRichArray$.at$extension(package.scala:150)
	at java.lang.String.hashCode(String.scala:392)
	at scala.runtime.Statics$.anyHash(Statics.scala:76)
	at scala.runtime.Statics.anyHash(Statics.scala:76)
	at scala.collection.mutable.HashMap.scala$collection$mutable$HashMap$$computeHash(HashMap.scala:79)
	at scala.collection.mutable.HashMap.findNode(HashMap.scala:85)
	at scala.collection.mutable.HashMap.get(HashMap.scala:434)
	at catseffect.examples.NativeRunner$.main(examplesplatform.scala:61)
	at catseffect.examples.NativeRunner.main(examplesplatform.scala:61)
	at <none>.main(unknown:2)
	at <none>.(Unknown Source)
	at <none>.__libc_start_main(Unknown Source)
	at <none>._start(Unknown Source)

Two things:

1. I really don't see how is this related generateFunctionSourcePositions. Okay, there are no DWARF$SubprogramDIE instances created if that is false, but other objects could be, so:
2. This very much looks like either a GC bug, or someone incorrectly not having a GC-discoverable reference to something they're still using. I've tried to look into it, and couldn't find any evidence of the latter. So it's concerning. And the generateFunctionSourcePositions could be a complete coincidence, and not an actual workaround. I'm really not sure.

(In any case, I really don't think this is CE's fault.)