GC segfault in julia v1.11.5

[MasonProtter]

I encountered a segfault today that appears to happen in the GC but only on julia v1.11, I observe no problem on v1.10.9 or v1.12-beta3.

Here is an rr trace of the crash: https://julialang-dumps.s3.amazonaws.com/reports/2025-05-30T15-04-51-MasonProtter.tar.zst

A MWE to trigger the crash yourself would be to clone https://github.com/Neuroblox/Neuroblox.jl master branch, activate its environment and then run the script Neuroblox/examples/NMDA_gc_mwe.jl

Here is a (non-rr) stacktrace I get when I run the MWE (I was unable to make build julia v1.11.5 to give a gdb backtrace):

julia> include("examples/NMDA_gc_mwe.jl");

[231663] signal 11 (1): Segmentation fault
in expression starting at /home/masonp/Nextcloud/neuroblox/Neuroblox/examples/NMDA_gc_mwe.jl:70
gc_mark_obj8 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:2113
gc_mark_outrefs at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:2882 [inlined]
gc_mark_and_steal at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:2987
gc_mark_loop_parallel at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:3127
gc_mark_loop at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:3146 [inlined]
_jl_gc_collect at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:3532
ijl_gc_collect at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:3893
maybe_collect at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:926 [inlined]
jl_gc_pool_alloc_inner at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:1319
ijl_gc_pool_alloc_instrumented at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/gc.c:1377
ht_keyindex at /home/masonp/.julia/packages/OrderedCollections/EqSuR/src/ordered_dict.jl:250
getindex at /home/masonp/.julia/packages/OrderedCollections/EqSuR/src/ordered_dict.jl:386 [inlined]
has_connection at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:42
unknown function (ip: 0x78ef3c7a3acd)
#56 at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:210
iterate at ./generator.jl:48 [inlined]
collect_to! at ./array.jl:849
collect_to_with_first! at ./array.jl:827
unknown function (ip: 0x78ef3c7ae782)
collect at ./array.jl:801
unknown function (ip: 0x78ef3c7aa442)
map at ./abstractarray.jl:3399
unknown function (ip: 0x78ef3c7a8e36)
call_composed at ./operators.jl:1054 [inlined]
call_composed at ./operators.jl:1053 [inlined]
#_#113 at ./operators.jl:1050
ComposedFunction at ./operators.jl:1050 [inlined]
#55 at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:204 [inlined]
iterate at ./generator.jl:48 [inlined]
collect_to! at ./array.jl:849
collect_to_with_first! at ./array.jl:827
unknown function (ip: 0x78ef3c7ad8e2)
collect at ./array.jl:801
unknown function (ip: 0x78ef3c7a8d22)
map at ./abstractarray.jl:3399 [inlined]
call_composed at ./operators.jl:1054 [inlined]
call_composed at ./operators.jl:1053 [inlined]
call_composed at ./operators.jl:1053
unknown function (ip: 0x78ef3c7a80a3)
#_#113 at ./operators.jl:1050 [inlined]
ComposedFunction at ./operators.jl:1050 [inlined]
#54 at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:203 [inlined]
iterate at ./generator.jl:48 [inlined]
collect at ./array.jl:791
unknown function (ip: 0x78ef3c7a7f62)
map at ./abstractarray.jl:3399 [inlined]
call_composed at ./operators.jl:1054 [inlined]
call_composed at ./operators.jl:1053 [inlined]
call_composed at ./operators.jl:1053
unknown function (ip: 0x78ef3c7a7c13)
#_#113 at ./operators.jl:1050
ComposedFunction at ./operators.jl:1050 [inlined]
#make_connection_matrices#51 at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:202
make_connection_matrices at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:192 [inlined]
PartitionedGraphSystem at /home/masonp/.julia/packages/GraphDynamics/IVrFE/src/graph_system.jl:163
#system_from_graph#520 at /home/masonp/Nextcloud/neuroblox/Neuroblox/src/Neurographs.jl:302 [inlined]
system_from_graph at /home/masonp/Nextcloud/neuroblox/Neuroblox/src/Neurographs.jl:298 [inlined]
system_from_graph at /home/masonp/Nextcloud/neuroblox/Neuroblox/src/Neurographs.jl:298
unknown function (ip: 0x78ef3c7833e6)
top-level scope at /home/masonp/Nextcloud/neuroblox/Neuroblox/examples/NMDA_gc_mwe.jl:71
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:430 [inlined]
include_string at ./loading.jl:2734
_include at ./loading.jl:2794
include at ./sysimg.jl:38
unknown function (ip: 0x78ef3c6ff232)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
do_call at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:666
jl_interpret_toplevel_thunk at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:824
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:430 [inlined]
eval_user_input at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:261
repl_backend_loop at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:368
#start_repl_backend#59 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:343
start_repl_backend at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:340
#run_repl#76 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:500
run_repl at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:486
jfptr_run_repl_10123.1 at /home/masonp/.julia/juliaup/julia-1.11.5+0.x64.linux.gnu/share/julia/compiled/v1.11/REPL/u0gqU_4x0TT.so (unknown line)
#1150 at ./client.jl:446
jfptr_YY.1150_14797.1 at /home/masonp/.julia/juliaup/julia-1.11.5+0.x64.linux.gnu/share/julia/compiled/v1.11/REPL/u0gqU_4x0TT.so (unknown line)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_f__call_latest at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/builtins.c:875
#invokelatest#2 at ./essentials.jl:1055 [inlined]
invokelatest at ./essentials.jl:1052 [inlined]
run_main_repl at ./client.jl:430
repl_main at ./client.jl:567 [inlined]
_start at ./client.jl:541
jfptr__start_73430.1 at /home/masonp/.julia/juliaup/julia-1.11.5+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
true_main at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/jlapi.c:1059
main at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/cli/loader_exe.c:58
unknown function (ip: 0x78ef6d05f6b4)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 153434091 (Pool: 153432645; Big: 1446); GC: 72
zsh: segmentation fault (core dumped)  julia +1.11 --project=.

[giordano]

I was unable to make build julia v1.11.5

Why?

[MasonProtter]

It errored during building for me with a big list of LLVM errors.

[giordano]

That's very vague. Are you talking about #58478? If so, see #58478 (comment) for a workaround.

[MasonProtter]

I didn't spend much time on it because it wasn't the pressing issue to me, and I thought an rr trace would be more useful anyways.

Looks like #58478 is probably the same issue. If you need a GDB trace in order to help with this, I can try the workaround there.

[giordano]

Sometimes segfaults are code paths which are doing something unexpected which would be caught by assertions (disabled in the release builds), so doing an assert build may be helpful.

[MasonProtter]

Okay, I made an assert build and ran the script again. It hit the same segfault with no assertions shown. I'm now makeing a debug build, and will post a gdb backtrace when it's done running.

[MasonProtter]

Here's what I get from gdb:

Thread 12 "julia" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc0bff6c0 (LWP 78755)]
0x00007ffff72d0072 in gc_mark_obj8 (ptls=0x7fffa8000900, obj8_parent=0x7fff2df753c0 "\344\003", 
    obj8_begin=0x7fff32a8bf1e <jl_system_image_data+16978654> "", obj8_end=0x7fff32a8bf1f <jl_system_image_data+16978655> "", nptr=4)
    at /home/masonp/julia-debug/src/gc.c:2120
2120                    nptr |= !gc_old(o->header);
(gdb) bt
#0  0x00007ffff72d0072 in gc_mark_obj8 (ptls=0x7fffa8000900, obj8_parent=0x7fff2df753c0 "\344\003", 
    obj8_begin=0x7fff32a8bf1e <jl_system_image_data+16978654> "", obj8_end=0x7fff32a8bf1f <jl_system_image_data+16978655> "", nptr=4)
    at /home/masonp/julia-debug/src/gc.c:2120
#1  0x00007ffff72d49c8 in gc_mark_outrefs (ptls=0x7fffa8000900, mq=0x7fffa8001780, _new_obj=0x7fff2df753c0, meta_updated=0)
    at /home/masonp/julia-debug/src/gc.c:2882
#2  gc_mark_and_steal (ptls=0x7fffa8000900) at /home/masonp/julia-debug/src/gc.c:2987
#3  0x00007ffff72d5388 in gc_mark_loop_parallel (ptls=0x7fffa8000900, master=0) at /home/masonp/julia-debug/src/gc.c:3135
#4  0x00007ffff72c559b in jl_parallel_gc_threadfun (arg=0x5555556e0450) at /home/masonp/julia-debug/src/scheduler.c:151
#5  0x00007ffff7e2f7eb in start_thread (arg=<optimized out>) at pthread_create.c:448
#6  0x00007ffff7eb318c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[giordano]

CC @d-netto who seems to have been the last one to touch the code around there

julia/src/gc.c

Line 2120 in 760b2e5

nptr |= !gc_old(o->header);

[d-netto]

This code looks fine. It's just de-referencing o to read some metadata in the header.

Perhaps you just got a garbage pointer in o?

[MasonProtter]

Perhaps visible through the rr-trace I posted?

[Victor-Zane]

Well Mason first i want to thank you for the really detailed report. That rr trace and the MWE are super helpful for diagnosing these kinds of tricky bugs. It's awesome you put that together.
A segfault in the GC that's specific to a Julia v1.11.x version (and doesn't show up in v1.10.9 or v1.12-beta3) is a classic sign of a regression that might have crept in and then got fixed. Given it's working in v1.12-beta3, there's a good chance the fix is already on the master branch.

Here are a couple of solutions i would advise you to try:

First you should search Julia's GitHub Issues: You might want to check the JuliaLang/julia issues page( link:https://www.google.com/url?sa=E&q=https%3A%2F%2Fgithub.com%2FJuliaLang%2Fjulia%2Fissues )to see if someone else has already run into this. You could try searching for terms like "GC segfault 1.11", "gc_mark_obj8 crash", or similar. If there's an existing issue, it might give more context or even link to the pull request that fixed it.

The second and simplest solution (that is the same you've already discovered) would be sticking with v1.10.9 or using v1.12-beta3 for now to avoid this particular crash.
It's a bit of a pain when you hit these issues, but detailed reports like yours are how they get squashed. Good luck!