Omniscape fails on specific row / column: "median of empty array is undefined"

[slamander]

Cross-posting from gitter: there have two mentions of Omniscape failing on a specific row / column.

In my case, I receive this error: ERROR: ArgumentError: median of an empty array is undefined, Float64[] . The run will fail at different stages--sometimes quite far into a run, other times immediately--but always at the same row-column combination. I've tried to diagnose the error by changing the raster inputs or adjusting model parameters (e.g., solver, block size, r_cutoff [for source = resistance], and running conditional and non-conditional criteria).

Here's my error print-out. Let me know if additional information is needed.

Progress:   9%|█████                                             |  ETA: 2:57:07
Omniscape failed on the moving window centered on row 325 column 5198
ERROR: TaskFailedException
Stacktrace:
 [1] wait
   @ .\task.jl:322 [inlined]
 [2] threading_run(func::Function)
   @ Base.Threads .\threadingconstructs.jl:34
 [3] macro expansion
   @ .\threadingconstructs.jl:93 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [6] top-level scope
   @ REPL[5]:1

    nested task error: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] macro expansion
       @ C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [3] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48

    caused by: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] median!(v::Vector{Float64})
       @ Statistics C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Statistics\src\Statistics.jl:791
     [2] median
       @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Statistics\src\Statistics.jl:836 [inlined]
     [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:231
     [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:172
     [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\utils.jl:308
     [6] macro expansion
       @ C:\Users\jbaecher\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
     [7] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [8] (::Omniscape.var"#161#threadsfor_fun#11"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48

[vlandau]

Sorry for the delay in getting back to you on this. Is this still giving you problems?

It does make sense that the timing of the error being thrown is random since the order of the moving window solves is randomized for more accurate ETA estimates in the progress bar.

If possible, would you mind sharing your input files (and the associate ini file)? I'll try to take a look in the coming couple of weeks.

[slamander]

@vlandau, and my apologies for taking even longer to get back. Yes, this is still giving me issues, across several different input files. Here's the input data and ini file: https://drive.google.com/drive/folders/1kC4FVzw2HjlXVbLLot4a6LU0EPu-E-7s?usp=sharing.

[slamander]

By the way, here's the problem cells from two failed runs. The value of all of these cells are NA.
And here's the stack trace from one of them:

Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
 [6] top-level scope
   @ /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_All.jl:7

    nested task error: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [3] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
    
    caused by: ArgumentError: median of an empty array is undefined, Float64[]
    Stacktrace:
     [1] median!(v::Vector{Float64})
       @ Statistics /apps/julia/1.8.2/share/julia/stdlib/v1.8/Statistics/src/Statistics.jl:810
     [2] median
       @ /apps/julia/1.8.2/share/julia/stdlib/v1.8/Statistics/src/Statistics.jl:855 [inlined]
     [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:231
     [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:172
     [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:308
     [6] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
     [7] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [8] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [9] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_All.jl:7
Omniscape failed on the moving window centered on row 145 column 2049
Fri Mar 31 10:46:47 EDT 2023

[Daviser95]

Hi guys, I'm having the exact same error launching a Europe-wide Omniscape scan. The error appeared after adding a conditional layer in the .ini file. I've already checked that the conditional layer has the same size and resolution as the resistance layer. I tried to change some parameters (radius and block size) but the error keeps appearing.

Omniscape failed on the moving window centered on row 3323 column 2228
ERROR: TaskFailedException

nested task error: ArgumentError: median of an empty array is undefined, Float64[]
Stacktrace:
 [1] macro expansion
   @ C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
 [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
   @ Omniscape .\threadingconstructs.jl:163
 [3] #161#threadsfor_fun
   @ .\threadingconstructs.jl:130 [inlined]
 [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
   @ Base.Threads .\threadingconstructs.jl:108

caused by: ArgumentError: median of an empty array is undefined, Float64[]
Stacktrace:
 [1] median!(v::Vector{Float64})
   @ Statistics C:\Users\utente\AppData\Local\Programs\Julia-1.9.0\share\julia\stdlib\v1.9\Statistics\src\Statistics.jl:803
 [2] median
   @ C:\Users\utente\AppData\Local\Programs\Julia-1.9.0\share\julia\stdlib\v1.9\Statistics\src\Statistics.jl:848 [inlined]
 [3] source_target_match!(source_subset::Matrix{Union{Missing, Float64}}, n_conditions::Int64, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, ylower::Int64, yupper::Int64, xlower::Int64, xupper::Int64, ylower_buffered::Int64, yupper_buffered::Int64, xlower_buffered::Int64, xupper_buffered::Int64)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:219
 [4] get_source(source_array::Matrix{Union{Missing, Float64}}, arguments::Dict{String, Int64}, conditional::Bool, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, target::Omniscape.Target)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:172
 [5] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
   @ Omniscape C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\utils.jl:308
 [6] macro expansion
   @ C:\Users\utente\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
 [7] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
   @ Omniscape .\threadingconstructs.jl:163
 [8] #161#threadsfor_fun
   @ .\threadingconstructs.jl:130 [inlined]
 [9] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
   @ Base.Threads .\threadingconstructs.jl:108

Stacktrace:
[1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
@ Base.Threads .\threadingconstructs.jl:120
[2] macro expansion
@ .\threadingconstructs.jl:168 [inlined]
[3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
@ Omniscape C:\Users\utente.julia\packages\Omniscape\9gHf2\src\main.jl:257
[4] run_omniscape(path::String)
@ Omniscape C:\Users\utente.julia\packages\Omniscape\9gHf2\src\main.jl:536
[5] top-level scope
@ REPL[6]:1

[vlandau]

My best guess is at the column/row of one of your inputs, the resulting slice/chunks of the array is just empty (all NaN values). Would you be able to clip out a section (square, with side lengths equal to 2×radius + 1) of each of your input rasters centered on the failing row/column and share it with me?

[Daviser95]

Hi @vlandau and thanks for replying, you're probably right. I share the clipped input rasters with your options. However, if this is the problem, I don't know why using only the resistance raster doesn't cause problems, while this error only occurs when using a second layer as a condition.
Moreover, I noticed that the resolution of the two layers is not the same, while the coordinate system and the dimensions are, but I don't know if this difference can cause the error.

[ashton314]

Hi there! I'm a PhD student and I'm building a tool to help developers automatically track down where NaNs come from in computations, and I'd like to try it out on the problem you're having here to see if I can help. Would someone mind showing me how to fire it all off to reproduce the problems that you're having here? I think I have the .ini and the .tiff files already—just wondering how to set it up properly.

[Daviser95]

Hi @ashton314! If you already have .ini and .tiff file, I think you just need to donwload Omniscape on Julia and then run the .ini file. You can get information on this page https://docs.circuitscape.org/Omniscape.jl/stable/usage/#Running-Omniscape

[slamander]

@vlandau I also thought there could be NaNs on the failing pixels, but after investigating the failures I found no NaNs. Does it make sense that identical jobs fail at different locations/durations? Sometimes the job will near completion (>90%), but others will fail early--all failing on different pixels. Unlike @Daviser95, some jobs will fail whether or not I have conditional criteria (it's not very predictable). Here's an output from the last job I ran without conditional criteria:

Stacktrace:
 [1] wait
   @ ./task.jl:345 [inlined]
 [2] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads ./threadingconstructs.jl:38
 [3] macro expansion
   @ ./threadingconstructs.jl:89 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:257
 [5] run_omniscape(path::String)
   @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/main.jl:536
 [6] top-level scope
   @ /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_Americas.jl:7

    nested task error: 
Progress:  56%|████████████████████████████�                     |  ETA: 6:19:09�[K
Progress:  56%|████████████████████████████�                     |  ETA: 6:19:09�[KAssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape ./threadingconstructs.jl:84
     [3] #161#threadsfor_fun
       @ ./threadingconstructs.jl:51 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads ./threadingconstructs.jl:30
    
    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SuiteSparse.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/core.jl:621
      [2] macro expansion
        @ ./timing.jl:382 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/raster/advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/raster/advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape ~/.julia/packages/Circuitscape/33lUW/src/utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape ~/.julia/packages/Omniscape/9gHf2/src/utils.jl:332
      [7] macro expansion
        @ ~/.julia/packages/Omniscape/9gHf2/src/main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape ./threadingconstructs.jl:84
      [9] #161#threadsfor_fun
        @ ./threadingconstructs.jl:51 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads ./threadingconstructs.jl:30
in expression starting at /blue/scheffers/jbaecher/global_connectivity/julia_scripts/hpg_Americas.jl:7

Progress:  56%|████████████████████████████�                     |  ETA: 6:19:08�[KOmniscape failed on the moving window centered on row 2542 column 7157
Omniscape failed on the moving window centered on row 1424 column 1983
Omniscape failed on the moving window centered on row 1593 column 4102
Wed Jul 12 21:26:43 EDT 2023

[vlandau]

It does make sense that it would occur at different times because the order in which the windows are solved is random (this helps the ETA on the progress bar to be more accurate). I assume in your case there are multiple moving window locations that will cause failure, but which one you run into first is random due to the random solve order.

[geomenke]

I regularly get the same issue as @slamander and @Daviser95. Omniscape will fail on specific row / columns. These are not No Data or Nan pixels however. I have clipped out a small area centered on these pixels and run OS on the subset and it finishes successfully. What are the potential reasons for this type of error?
Here is the stacktrace of the error.

moving window centered on row 6461 column 8703
Progress:  92%|███████████████████████████████████████████████   |  ETA: 4:55:16ERROR: TaskFailedException

    nested task error: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
     [1] macro expansion
       @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:278 [inlined]
     [2] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:194
     [3] #161#threadsfor_fun
       @ .\threadingconstructs.jl:161 [inlined]
     [4] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
       @ Base.Threads .\threadingconstructs.jl:139

    caused by: AssertionError: norm(matrix * lhs[:, i] .- rhs[:, i]) / norm(rhs[:, i]) < 1.0e-6
    Stacktrace:
      [1] solve_linear_system(factor::SparseArrays.CHOLMOD.Factor{Float64}, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, rhs::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\core.jl:621
      [2] macro expansion
        @ .\timing.jl:393 [inlined]
      [3] multiple_solve(s::Circuitscape.CholmodSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:320
      [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.CholmodSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\raster\advanced.jl:291
      [5] compute_omniscape_current(conductance::Matrix{Float64}, source::Matrix{Float64}, ground::Matrix{Float64}, cs_cfg::Dict{String, String})
        @ Circuitscape C:\Users\Kurt Menke\.julia\packages\Circuitscape\33lUW\src\utils.jl:529
      [6] solve_target!(target::Omniscape.Target, int_arguments::Dict{String, Int64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, condition_layers::Omniscape.ConditionLayers{Float64, 2}, conditions::Omniscape.Conditions, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
        @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\utils.jl:332
      [7] macro expansion
        @ C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:264 [inlined]
      [8] (::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}})(tid::Int64; onethread::Bool)
        @ Omniscape .\threadingconstructs.jl:194
      [9] #161#threadsfor_fun
        @ .\threadingconstructs.jl:161 [inlined]
     [10] (::Base.Threads.var"#1#2"{Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, Int64})()
        @ Base.Threads .\threadingconstructs.jl:139
Stacktrace:
 [1] threading_run(fun::Omniscape.var"#161#threadsfor_fun#12"{Omniscape.var"#161#threadsfor_fun#11#13"{Int64, ProgressMeter.Progress, Int64, Dict{String, String}, Omniscape.ConditionLayers{Float64, 2}, Omniscape.Conditions, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}}}, static::Bool)
   @ Base.Threads .\threadingconstructs.jl:151
 [2] macro expansion
   @ .\threadingconstructs.jl:199 [inlined]
 [3] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:257
 [4] run_omniscape(path::String)
   @ Omniscape C:\Users\Kurt Menke\.julia\packages\Omniscape\9gHf2\src\main.jl:536
 [5] top-level scope
   @ REPL[3]:1

[vlandau]

@geomenke this is a different error than the empty array error mentioned above (but I see now one of the stack traces had an AssertionError too), coming directly from Circuitscape, but fortunately, we just made a patch release to Circuitscape that may fix that issue. Would you update to Circuitscape 5.13.2 (you'll need Julia 1.9) and try again?

[geomenke]

@vlandau I am already at Julia 1.9.2 so what is the best way to update to Circuitscape 5.13.2? Do I upgrade Omniscape? Can that be done via the command line?

[vlandau]

Oh great! To update, in a Julia terminal, enter the package interface by typing the ] key, then run update Circuitscape and you should see 5.13.2 get installed.

[geomenke]

It did not specifically list Circuitscape as being updated, just things like updating registry etc.
How can I check which Circuitscape version I have?