Extension module is missing from the cache

[jishnub]

In JuliaApproximation/ApproxFunBase.jl#506, I am trying to move Statistics and SparseArrays to weak deps loaded through extensions, and I test downstream packages against this PR. However, I obtained the following warning during precompilation of the downstream package ApproxFunOrthogonalPolynomials:

┌ ApproxFunBaseSparseArraysExt [2448fcf3-777e-5e57-899e-a100cfbe46a1]
│  ┌ Warning: Module ApproxFunBaseSparseArraysExt with build ID ffffffff-ffff-ffff-0000-0056a0f5e447 is missing from the cache.
│  │ This may mean ApproxFunBaseSparseArraysExt [2448fcf3-777e-5e57-899e-a100cfbe46a1] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1793
│  ┌ Error: Error during loading of extension ApproxFunBaseSparseArraysExt of ApproxFunBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1797
│  │      [2] _require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1660
│  │      [3] _require_prelocked(uuidkey::Base.PkgId)
│  │        @ Base ./loading.jl:1658
│  │      [4] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1255
│  │      [5] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1290
│  │      [6] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1124
│  │      [7] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1667
│  │      [8] macro expansion
│  │        @ ./loading.jl:1648 [inlined]
│  │      [9] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [10] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1611
│  │     [11] include
│  │        @ ./Base.jl:457 [inlined]
│  │     [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2049
│  │     [13] top-level scope
│  │        @ stdin:3
│  │     [14] eval
│  │        @ ./boot.jl:370 [inlined]
│  │     [15] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:1903
│  │     [16] include_string
│  │        @ ./loading.jl:1913 [inlined]
│  │     [17] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:305
│  │     [18] _start()
│  │        @ Base ./client.jl:522
│  └ @ Base loading.jl:1261
└ 

and a couple of other similar warnings (one for Statistics and one for ApproxFunOrthogonalPolynomials). I am unsure about what leads to this warning, although it might be related to the fact that the package is deved while testing.

In case it matters, this is the CI script that is used to initiate the test

[IanButterworth]

Seems like #3571 which is fixed on master and should be fixed in 1.9.4 (it's in the Pkg bump in JuliaLang/julia#50977)

[jishnub]

This is not fixed on v1.9.4 or v1.10.0, but will possibly be resolved by JuliaLang/julia#52841. Reopening until that is merged.

[IanButterworth]

That is a different issue to the one above, which doesn't appear have multiple exts that each depend on packages in the sysimage.

If the issue is only happening once you add the above packages to the sysimage, then the other issue is also enough to track this

[jishnub]

Can confirm that this is fixed by JuliaLang/julia#52841

[jishnub]

Oddly, this doesn't seem to be resolved yet, and I am being able to replicate the issue on v1.10.2. The message is

┌ ApproxFunBase → ApproxFunBaseSparseArraysExt
│  ┌ Warning: Module ApproxFunBaseSparseArraysExt with build ID ffffffff-ffff-ffff-0000-003639e1cc73 is missing from the cache.
│  │ This may mean ApproxFunBaseSparseArraysExt [2448fcf3-777e-5e57-899e-a100cfbe46a1] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1948
│  ┌ Error: Error during loading of extension ApproxFunBaseSparseArraysExt of ApproxFunBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1952
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1812
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:1803 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:1802 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1295
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1330
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1164
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1819
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1803
│  │     [14] macro expansion
│  │        @ ./loading.jl:1790 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1753
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1746
│  │     [20] include
│  │        @ ./Base.jl:495 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2222
│  │     [22] top-level scope
│  │        @ stdin:3
│  │     [23] eval
│  │        @ ./boot.jl:385 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2076
│  │     [25] include_string
│  │        @ ./loading.jl:2086 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:316
│  │     [27] _start()
│  │        @ Base ./client.jl:552
│  └ @ Base loading.jl:1301
└  
┌ ApproxFunBase → ApproxFunBaseStatisticsExt
│  ┌ Warning: Module ApproxFunBaseStatisticsExt with build ID ffffffff-ffff-ffff-0000-00377536ceb1 is missing from the cache.
│  │ This may mean ApproxFunBaseStatisticsExt [06f40816-ae97-5315-8130-9e560ff8d968] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1948
│  ┌ Error: Error during loading of extension ApproxFunBaseStatisticsExt of ApproxFunBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1952
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1812
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:1803 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:1802 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1295
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1330
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1164
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1819
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1803
│  │     [14] macro expansion
│  │        @ ./loading.jl:1790 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1753
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1746
│  │     [20] include
│  │        @ ./Base.jl:495 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2222
│  │     [22] top-level scope
│  │        @ stdin:3
│  │     [23] eval
│  │        @ ./boot.jl:385 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2076
│  │     [25] include_string
│  │        @ ./loading.jl:2086 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:316
│  │     [27] _start()
│  │        @ Base ./client.jl:552
│  └ @ Base loading.jl:1301
└  

I obtain a similar warning on v1.11.0-beta1.

I have uploaded the project and manifest files at https://github.com/jishnub/ApproxFunBase.jl/tree/extensionserror, so it should hopefully be easy to reproduce.

[jishnub]

Update: using julia v1.11.0-rc2, I find

┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│   Base.PkgId(Base.UUID("2448fcf3-777e-5e57-899e-a100cfbe46a1"), "ApproxFunBaseSparseArraysExt")
│   Base.PkgId(Base.UUID("06f40816-ae97-5315-8130-9e560ff8d968"), "ApproxFunBaseStatisticsExt")
└ @ Base.Precompilation precompilation.jl:552
Precompiling project...
  56 dependencies successfully precompiled in 52 seconds. 37 already precompiled. 2 skipped due to circular dependency.

I wonder if this is because some of the dependencies have SparseArrays and Statistics as hard dependencies? So the chain might go ApproxFunBase -> DSP -> Statistics -> ApproxFunBaseStatisticsExt -> ApproxFunBase, therefore creating a cycle?