LLVM segfault in Julia-1.6.x

[wbhart]

We are seeing a crash when running the test suite of our pure Julia package AbstractAlgebra.jl at current master (c00a5bde2f1e972aa36a3eefc7c5a250155f0ce4).

The issue there only occurs on Julia-1.6.2, not 1.6.1, but see further down in the thread, where we reduce it to an example which occurs in 1.6.x.

julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen 7 4800HS with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, znver2)

We are seeing it across a wide variety of platforms, e.g. Ubuntu 20.04 on WSL with a Ryzen 2 laptop, for a specific example. But it occurs on our Gentoo servers, MacOS and so on. Further information about those machines can be found in our issue tracker at Nemocas/AbstractAlgebra.jl#1002

We do CI of our package against Julia nightlies and I don't recall seeing this issue there.

signal (11): Segmentation fault
in expression starting at /home/wbhart/.julia/dev/AbstractAlgebra/test/generic/RationalFunctionField-test.jl:97
_ZN4llvm11Instruction15eraseFromParentEv at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
finalize at /buildworker/worker/package_linux64/build/src/llvm-alloc-opt.cpp:380 [inlined]
runOnFunction at /buildworker/worker/package_linux64/build/src/llvm-alloc-opt.cpp:1518
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
operator() at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:613
_ZN4llvm3orc14IRCompileLayer4emitENS0_29MaterializationResponsibilityENS0_16ThreadSafeModuleE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeENS0_29MaterializationResponsibilityE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession26materializeOnCurrentThreadESt10unique_ptrINS0_19MaterializationUnitESt14default_deleteIS3_EENS0_29MaterializationResponsibilityE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZNSt17_Function_handlerIFvSt10unique_ptrIN4llvm3orc19MaterializationUnitESt14default_deleteIS3_EENS2_29MaterializationResponsibilityEEPS8_E9_M_invokeERKSt9_Any_dataOS6_OS7_ at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17runOutstandingMUsEv at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_EENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_EENSN_IS6_SV_EEEEEE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISK_EEEENSL_IS5_EENS_6detail12DenseMapPairIS5_SN_EEEEEE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /home/wbhart/julia-1.6.2/bin/../lib/julia/libLLVM-11jl.so (unknown line)
addModule at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:788
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1067
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1111
jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:1133 [inlined]
_jl_compile_codeinst at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:154
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:352
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1970
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:2236 [inlined]
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2229 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:490
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1116
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1170
include at ./client.jl:444
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1116
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1170
include at ./client.jl:444
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1116
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1170
include at ./client.jl:444
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1116
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
_include at ./loading.jl:1170
include at ./client.jl:444
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:115
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:204
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:155 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:562
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:670
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:877
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:825
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
exec_options at ./client.jl:261
_start at ./client.jl:485
jfptr__start_34281.clone_1 at /home/wbhart/julia-1.6.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:560
repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:51
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/wbhart/julia-1.6.2/bin/julia (unknown line)
Allocations: 967661496 (Pool: 967421861; Big: 239635); GC: 1014
ERROR: Package AbstractAlgebra errored during testing (received signal: 11)

The line of test code in question is simply a testset line, rather than an actual test within the testset, so doesn't yield helpful information at this point.

I have tried looking for any type piracy that might be an issue on our side that could contribute to killing the compiler, but haven't been able to find any.

I am working to reduce this to a minimum example, however, AbstractAlgebra is a huge package, so that may take quite some days.

I have not yet built a debug version of Julia to see if that sheds any more light, but I can do so if that is likely to be more helpful.

[fingolfin]

I have reduced the crash to the following:

using AbstractAlgebra
R, x = RationalFunctionField(QQ, "x")
x * denominator(x)

It happens only in Julia 1.6.2; not in 1.6.1, nor in 1.7.0-beta3. But since 1.6 is likely going to be the LTS, getting this fixed would be kinda important to us.

I'll see if I can bisect the crash.

[KristofferC]

You can also check with #41554

I'll see if I can bisect the crash.

Should be fairly quick since it is just the commits between 1.6.1 and 1.6.2.

[fingolfin]

I'll do both. Indeed, it's just 32 commits. Also, @wbhart is looking into turning my reduced example into one that is even simpler, possible pure Julia. We'll see.

[vtjnash]

You also might want to try with LLVM_ASSERTIONS=1 FORCE_ASSERTIONS=1 build, since it looks like we might be generating invalid IR, and that will enable many checks for it

[fingolfin]

That crash also appears in the backports-release-1.6 branch from #41554. Still am bisecting. Will now also try @vtjnash's suggestion.

[fingolfin]

I tried backports-release-1.6 with a Make.user file containing this:

LLVM_ASSERTIONS := 1
FORCE_ASSERTIONS := 1

Is that the right way to do it, @vtjnash ?

Next I did make clean && make but in the end, nothing different happened. The backtrace still starts like this:

signal (11): Segmentation fault: 11
in expression starting at REPL[4]:1
_ZN4llvm11Instruction15eraseFromParentEv at /Users/mhorn/Projekte/Julia/julia.spielwiese/usr/lib/libLLVM.dylib (unknown line)
_ZN12_GLOBAL__N_18AllocOpt13runOnFunctionERN4llvm8FunctionE at /Users/mhorn/Projekte/Julia/julia.spielwiese/usr/lib/libjulia-internal.1.dylib (unknown line)
...

Here's the output of Apple's crash reporter, where the C++ symbols are demangled:

0   libLLVM.dylib                 	0x000000010a28db9b llvm::Instruction::eraseFromParent() + 59
1   libjulia-internal.1.dylib     	0x0000000109aebb49 (anonymous namespace)::AllocOpt::runOnFunction(llvm::Function&) + 601 (llvm-alloc-opt.cpp:379)
2   libLLVM.dylib                 	0x000000010a2bc1a0 llvm::FPPassManager::runOnFunction(llvm::Function&) + 1072
3   libLLVM.dylib                 	0x000000010a2c2a28 llvm::FPPassManager::runOnModule(llvm::Module&) + 72
4   libLLVM.dylib                 	0x000000010a2bc782 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 914
5   libjulia-internal.1.dylib     	0x0000000109aa4ee5 JuliaOJIT::CompilerT::operator()(llvm::Module&) + 1877 (jitlayers.cpp:613)
6   libLLVM.dylib                 	0x000000010bb8f83c decltype(auto) llvm::orc::ThreadSafeModule::withModuleDo<llvm::orc::IRCompileLayer::IRCompiler&>(llvm::orc::IRCompileLayer::IRCompiler&&&) + 92
7   libLLVM.dylib                 	0x000000010bb8f53c llvm::orc::IRCompileLayer::emit(llvm::orc::MaterializationResponsibility, llvm::orc::ThreadSafeModule) + 92
8   libLLVM.dylib                 	0x000000010bb99b65 llvm::orc::BasicIRLayerMaterializationUnit::materialize(llvm::orc::MaterializationResponsibility) + 725
9   libLLVM.dylib                 	0x000000010bb65c07 llvm::orc::ExecutionSession::materializeOnCurrentThread(std::__1::unique_ptr<llvm::orc::MaterializationUnit, std::__1::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility) + 119
10  libLLVM.dylib                 	0x000000010bb6cb75 void std::__1::__invoke_void_return_wrapper<void>::__call<void (*&)(std::__1::unique_ptr<llvm::orc::MaterializationUnit, std::__1::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility), std::__1::unique_ptr<llvm::orc::MaterializationUnit, std::__1::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility>(void (*&&&)(std::__1::unique_ptr<llvm::orc::MaterializationUnit, std::__1::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility), std::__1::unique_ptr<llvm::orc::MaterializationUnit, std::__1::default_delete<llvm::orc::MaterializationUnit> >&&, llvm::orc::MaterializationResponsibility&&) + 133
11  libLLVM.dylib                 	0x000000010bb5e427 llvm::orc::ExecutionSession::dispatchMaterialization(std::__1::unique_ptr<llvm::orc::MaterializationUnit, std::__1::default_delete<llvm::orc::MaterializationUnit> >, llvm::orc::MaterializationResponsibility) + 215
12  libLLVM.dylib                 	0x000000010bb617fc llvm::orc::ExecutionSession::runOutstandingMUs() + 364
13  libLLVM.dylib                 	0x000000010bb5c382 llvm::orc::ExecutionSession::lookup(llvm::orc::LookupKind, std::__1::vector<std::__1::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::__1::allocator<std::__1::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet, llvm::orc::SymbolState, llvm::unique_function<void (llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, llvm::JITEvaluatedSymbol> > >)>, std::__1::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) + 3570
14  libLLVM.dylib                 	0x000000010bb672b6 llvm::orc::ExecutionSession::lookup(std::__1::vector<std::__1::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::__1::allocator<std::__1::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolLookupSet const&, llvm::orc::LookupKind, llvm::orc::SymbolState, std::__1::function<void (llvm::DenseMap<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> >, llvm::DenseMapInfo<llvm::orc::JITDylib*>, llvm::detail::DenseMapPair<llvm::orc::JITDylib*, llvm::DenseSet<llvm::orc::SymbolStringPtr, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr> > > > const&)>) + 646
15  libLLVM.dylib                 	0x000000010bb67880 llvm::orc::ExecutionSession::lookup(std::__1::vector<std::__1::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::__1::allocator<std::__1::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) + 432
16  libLLVM.dylib                 	0x000000010bb67d64 llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib*>, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) + 276
17  libLLVM.dylib                 	0x000000010bb67e67 llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib*>, llvm::StringRef, llvm::orc::SymbolState) + 151
18  libjulia-internal.1.dylib     	0x0000000109aa74e4 JuliaOJIT::addModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >) + 2116 (jitlayers.cpp:788)
19  libjulia-internal.1.dylib     	0x0000000109aa9d00 jl_add_to_ee(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >&, llvm::StringMap<std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >*, llvm::MallocAllocator>&, llvm::DenseMap<llvm::Module*, int, llvm::DenseMapInfo<llvm::Module*>, llvm::detail::DenseMapPair<llvm::Module*, int> >&, std::__1::vector<std::__1::vector<std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >*, std::__1::allocator<std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >*> >, std::__1::allocator<std::__1::vector<std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >*, std::__1::allocator<std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module> >*> > > >&, int) + 2832 (memory:2649)
20  libjulia-internal.1.dylib     	0x0000000109aa3823 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long) + 2243 (memory:2637)
21  libjulia-internal.1.dylib     	0x0000000109aa2deb jl_generate_fptr + 523 (jitlayers.cpp:353)
22  libjulia-internal.1.dylib     	0x0000000109a1de51 jl_compile_method_internal + 257 (gf.c:1970)
23  libjulia-internal.1.dylib     	0x0000000109a20718 _jl_invoke + 136 (gf.c:2229)
24  libjulia-internal.1.dylib     	0x0000000109a20c21 jl_apply_generic + 1153 (gf.c:2419)
25  libjulia-internal.1.dylib     	0x0000000109a3b49e do_call + 222
26  libjulia-internal.1.dylib     	0x0000000109a3972c eval_body + 1820
27  libjulia-internal.1.dylib     	0x0000000109a39d3a jl_interpret_toplevel_thunk + 282 (interpreter.c:671)
28  libjulia-internal.1.dylib     	0x0000000109a583f0 jl_toplevel_eval_flex + 5120 (toplevel.c:877)
29  libjulia-internal.1.dylib     	0x0000000109a5825d jl_toplevel_eval_flex + 4717 (toplevel.c:825)
30  libjulia-internal.1.dylib     	0x0000000109a595a5 jl_toplevel_eval_in + 149 (toplevel.c:886)
31  sys.dylib                     	0x0000000115914545 japi1_eval_user_input_53881 + 1045 (REPL.jl:139)
32  sys.dylib                     	0x0000000115914ad4 japi1_repl_backend_loop_54654 + 916 (REPL.jl:200)
33  sys.dylib                     	0x0000000115914bac japi1_start_repl_backend_54652 + 108 (REPL.jl:186)
34  sys.dylib                     	0x00000001157fae59 japi1_run_repl_52820 + 25 (REPL.jl:305)
35  libjulia-internal.1.dylib     	0x0000000109a20c21 jl_apply_generic + 1153 (gf.c:2419)
36  sys.dylib                     	0x0000000115665adc julia_#874_45436 + 1532 (client.jl:387)
37  sys.dylib                     	0x0000000115665b69 jfptr_YY.874_45437 + 9
38  libjulia-internal.1.dylib     	0x0000000109a20c21 jl_apply_generic + 1153 (gf.c:2419)
39  libjulia-internal.1.dylib     	0x0000000109a2f519 jl_f__call_latest + 73 (builtins.c:715)
40  sys.dylib                     	0x00000001156c61a6 julia_run_main_repl_31056 + 2198 (client.jl:372)
41  sys.dylib                     	0x00000001156d7175 julia_exec_options_25063 + 23829
42  sys.dylib                     	0x00000001156d9dc1 julia__start_23005 + 593 (client.jl:485)

[fingolfin]

Bisecting points at commit eac11b9 by @vtjnash which is a backport of commit 399ec04 from PR #40487

[fingolfin]

Hmm, it seems setting LLVM_ASSERTIONS this way does not work. I've now just edited its default value in Make.inc and will try again with that.

[vtjnash]

I am not sure I can craft a test for this, but does this patch fix it? From just looking at the code, it seems like it is needed (@yuyichao):

diff --git a/src/llvm-alloc-opt.cpp b/src/llvm-alloc-opt.cpp
index ec7060bd10..65f36a0737 100644
--- a/src/llvm-alloc-opt.cpp
+++ b/src/llvm-alloc-opt.cpp
@@ -318,10 +318,15 @@ void Optimizer::initialize()
 
 void Optimizer::optimizeAll()
 {
+    size_t last_deleted = 0;
     while (!worklist.empty()) {
+        while (last_deleted < removed.size())
+            removed[last_deleted++]->removeFromParent();
         auto item = worklist.pop_back_val();
         auto orig = item.first;
         size_t sz = item.second;
+        if (orig->getParent() == nullptr)
+            continue; // already processed (and deleted) this
         checkInst(orig);
         if (use_info.escaped) {
             if (use_info.hastypeof)
@@ -363,6 +368,8 @@ void Optimizer::optimizeAll()
         // The object has no fields with mix reference access
         moveToStack(orig, sz, has_ref);
     }
+    while (last_deleted < removed.size())
+        removed[last_deleted++]->removeFromParent();
 }
 
 bool Optimizer::finalize()
@@ -370,7 +377,7 @@ bool Optimizer::finalize()
     if (removed.empty())
         return false;
     for (auto inst: removed)
-        inst->eraseFromParent();
+        delete inst;
     return true;
 }
 

[fingolfin]

@vtjnash still getting a crash with that patch, albeit a slightly different one:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libLLVM.dylib                 	0x000000010452ba51 llvm::Instruction::removeFromParent() + 33
1   libjulia-internal.1.dylib     	0x0000000101b70250 (anonymous namespace)::Optimizer::optimizeAll() + 17184 (SmallVector.h:66)
2   libjulia-internal.1.dylib     	0x0000000101b6bdd8 (anonymous namespace)::AllocOpt::runOnFunction(llvm::Function&) + 488 (SmallVector.h:69)
3   libLLVM.dylib                 	0x000000010455170c llvm::FPPassManager::runOnFunction(llvm::Function&) + 1116
4   libLLVM.dylib                 	0x0000000104557243 llvm::FPPassManager::runOnModule(llvm::Module&) + 67
5   libLLVM.dylib                 	0x0000000104551c99 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 825
6   libjulia-internal.1.dylib     	0x0000000101b26dc2 JuliaOJIT::CompilerT::operator()(llvm::Module&) + 1762 (jitlayers.cpp:613)
7   libLLVM.dylib                 	0x0000000105a41d32 llvm::orc::IRCompileLayer::emit(llvm::orc::MaterializationResponsibility, llvm::orc::ThreadSafeModule) + 114
8   libLLVM.dylib                 	0x0000000105a49fb9 llvm::orc::BasicIRLayerMaterializationUnit::materialize(llvm::orc::MaterializationResponsibility) + 393
...

[wbhart]

I don't know whether this segfault is related to our original one, but one reduction of our code that does segfault is given at:

https://github.com/wbhart/AbstractAlgebra.jl/tree/rat_crash3

If I pull all of those files into a single file, the segfault goes away, so unfortunately it's not as minimal an example as one would like. Of course the code is totally nonsense and invalid, but I guess that it may still expose a real bug.

To get it to trigger, one does the following at the REPL (with -O2):

using AbstractAlgebra

R = RationalFunctionField(QQ, "")

There is a partway minimised example that exhibits our original bug at:

https://github.com/wbhart/AbstractAlgebra.jl/tree/rat_crash3

I will try to reduce this further.

[wbhart]

I have reduced the example as far as I can. You can find it at:

https://github.com/wbhart/AbstractAlgebra.jl/tree/rat_crash4

Unfortunately it goes away if all the files are merged into a single file. There is one more or less nonsense piece of code which I haven't been able to remove successfully. But this now exhibits our original issue, I think.

To trigger it, type the following at the REPL with the branch above dev'd:

using AbstractAlgebra
a, b = AbstractAlgebra.RationalFunctionField(AbstractAlgebra.QQ, "")
b*0