CeresSolver_jll leads to `libcholmod.so.3` not found -- julia 1.10+

[avik-pal]

julia> using CeresSolver_jll
ERROR: InitError: could not load library "/home/avikpal/.julia/artifacts/d137110a8d2540ce7c5d544b481ce9aff7a5594a/lib/libceres.so"
libcholmod.so.3: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:116
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/pG9bm/src/products/library_generators.jl:63 [inlined]
  [4] __init__()
    @ CeresSolver_jll ~/.julia/packages/CeresSolver_jll/nq3AI/src/wrappers/x86_64-linux-gnu-cxx11.jl:12
  [5] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1134
  [6] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1122
  [7] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1067
  [8] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1581
  [9] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1938
 [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [11] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [12] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [14] macro expansion
    @ ./loading.jl:1790 [inlined]
 [15] macro expansion
    @ ./lock.jl:267 [inlined]
 [16] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [17] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [18] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [19] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [20] top-level scope
    @ ~/.julia/packages/Infiltrator/TNlCu/src/Infiltrator.jl:798
during initialization of module CeresSolver_jll

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × 11th Gen Intel(R) Core(TM) i5-11400 @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, rocketlake)
Threads: 12 default, 0 interactive, 6 GC (on 12 virtual cores)

This is in a fresh environment. Works fine in 1.9.

[giordano]

Probably need to rebuild with whatever version of suitesparse is in julia v1.10.

[imciner2]

Julia 1.10 uses Suitesparse 7.2.1, which has libcholmod.so.4. Note that Suitesparse 7.3.0+ changed that to libcholmod.so.5, and Julia 1.11 has Suitesparse 7.6.0 in it, so you will need to have very strict compat bounds for the Julia version to fix this, probably.

[ViralBShah]

Another reason why we should excise SparseArrays and SuiteSparse from stdlib (but it has other repercussions naturally). Most likely, just building an updated CeresSolver_jll with a recent Julia 1.10 minimum bound should fix this.

[avik-pal]

How do I trigger a rebuild?

[imciner2]

Another reason why we should excise SparseArrays and SuiteSparse from stdlib (but it has other repercussions naturally).

Excising SuiteSparse should be more doable at least - it is already somewhat optional since there is the GPL-free build mode, I thought. SparseArrays is still useful without SuiteSparse I think though (e.g. to at least provide the base types that other sparse environments like Cuda.jl extend).

How do I trigger a rebuild?

@avik-pal you need to make a change to the build_tarballs.jl file for Ceres in this repo. We need to be careful with this though due to those compatibility issues mentioned earlier. You will need to add a version bound to the SuiteSparse dependency of the package to make it get SuiteSparse 7.2.1, and increment the version number I believe (@giordano can confirm, but I think we need a version number change if we change compat bounds). Then we will also need to do this again with SuiteSparse 7.6.0 to make it then also work on Julia 1.11.

[giordano]

Yes, changing existing compat bounds*** always requires*** the version of the package (and no, removing the compat bounds doesn't count as a good solution, that only brings runtime errors), because Pkg (and hence the registry) can't distinguish different compat bounds within same major.minor.patch version numbers.

[ViralBShah]

My recommendation would be to skip Julia 1.10, and just build the latest version of Ceres (2.2) for Julia 1.11.

[ViralBShah]

It seems the existing Ceres just could not build properly with SuiteSparse 7 based on the comment in there. Is it still necessary to bump the Ceres version? My understanding is that we have to bump versions when we change SuiteSparse versions because that has an implicit dependency on Julia itself.

Yggdrasil/C/CeresSolver/build_tarballs.jl

Line 48 in 8566c7a

# Hard code the version now as the latest v7.0.1 does not get recognized

[giordano]

Yes, but sadly that's not a compat bound, so the registry doesn't know about it, otherwise it wouldn't have allowed installing the package to start with.

Aside, it's unfortunate that for historic reasons the second positional argument of Dependency is the build version but isn't recorded as compat bound, I think it doesn't make any sense to specify the build version but not the compat: if you have to specify the build version then you're doing it because only certain versions are compatible, and so specifying the compat should also be necessary. I tried to make the compat argument mandatory with JuliaPackaging/BinaryBuilderBase.jl#314, but got into troubles with Pkg internals.

[avik-pal]

My recommendation would be to skip Julia 1.10, and just build the latest version of Ceres (2.2) for Julia 1.11.

I triggered a build with 1.10 #8491 for the time being to check if it builds with newer suitesparse

[amontoison]

@imciner2
I got a similar issue with the package qr_mumps.
With SuiteSparse 7.x.x they changed the SONAME (libamd.so.2 -> libamd.so.3).
Can you confirm if the SONAME of all libraries in SuiteSparse are updated when Tim Davis changes the major version?
I don't want to set a compat entry on SuiteSparse_jll.jl if the SONAME is not always modified between two major releases

[imciner2]

The SO information for the libraries aren't tied together, unfortunately, so they can change even inside the major version (see my comment about cholmod changing SO version between 7.2 and 7.3).

If we were going to redesign the packaging of SuiteSparse, I almost think it would make more sense to split each library into its own JLL with its own version and then keep track of them that way, so then packages can declare the version dependencies only on the library they need. Of course that either requires better automation on our end to extract those versions, or more work during updates.

[avik-pal]

This is now fixed. Thanks everyone!