Bonsai-SLEAP "JIT compilation failed." error after installation

[frr665]

Dear Bonsai Community,

I am attempting to use the latest version of the SLEAP package for Bonsai (0.2.0) on Windows 10. Unfortunately, when I try to run a Bonsai workflow that was previously functional on another computer, I encounter the following error messages:

2023-08-22 10:59:06.638611: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-22 10:59:15.773451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21328 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4090, pci bus id: 0000:01:00.0, compute capability: 8.9
2023-08-22 10:59:16.438130: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2023-08-22 10:59:22.128265: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -37 } dim { size: -38 } dim { size: -39 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -12 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -12 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 4090" frequency: 2565 num_cores: 128 environment { key: "architecture" value: "8.9" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 75497472 shared_memory_size_per_multiprocessor: 102400 memory_size: 22365016064 bandwidth: 1008096000 } outputs { dtype: DT_FLOAT shape { dim { size: -12 } dim { size: -41 } dim { size: -42 } dim { size: 1 } } }
2023-08-22 10:59:25.260362: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8200
2023-08-22 10:59:25.889744: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] INTERNAL: ptxas exited with non-zero error code -1, output:
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
2023-08-22 10:59:26.953522: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
2023-08-22 10:59:27.051380: W tensorflow/core/framework/op_kernel.cc:1768] UNKNOWN: JIT compilation failed.

As stated on the GitHub page of bonsai-sleap, I have installed CUDA 11.3.0, cuDNN 11.3, and libtensorflow-gpu-windows-x86_64-2.10.0.
I copied the .dll files to the Extensions folder of Bonsai using the following approach:

The cuda_dir nvmm/libdevice is located at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3 . I've also tried to add CUDA and cudnn\cuda\bin in the Path environment variables.

What should I do to fix this installation issue ? Any insights or suggestions would be greatly appreciated.

Thank you for your time and assistance !

Best regards,
Hugo

[bruno-f-cruz]

Out of curiosity can you do a pip install of that version of tensorflow? If I remember correctly it should install those DLL s in your system.

[frr665]

I'm not sure if I did what you were expecting, but I got the same kind of error using their code snippet. By the way is it possible to run bonsai in an anaconda virtual environment ?

python -m pip install "tensorflow<2.11"
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement tensorflow<2.11 (from versions: 2.12.0rc0, 2.12.0rc1, 2.12.0, 2.12.1, 2.13.0rc0, 2.13.0rc1, 2.13.0rc2, 2.13.0, 2.14.0rc0)
ERROR: No matching distribution found for tensorflow<2.11

[bruno-f-cruz]

When you install sleap, does it run? It should also install the necessary cuda dependencies if I remember correctly.

[frr665]

Yes I have SLEAP in an anaconda environment and I can open the GUI with sleap-label command line. I haven't tried to run some training or inference yet on this computer. Edit : I can use SLEAP to perform tracking inference on recorded videos.

[farhanaugustine]

Did anyone figure this problem out? I am having the same issue in Bonsai when I try to run a top-down model.

[roomrys]

It is the usage of python's default % operator instead of tf.math.mod in tensorflow >2.7 (seen on 2.9 and 2.10, have not tried 2.8).

Related:

• Use tf.math.mod instead of % #1931

[EleonoraAmbrad]

@roomrys hi, so would you suggest to downgrade to tensorflow <2.7? are there precompiled libraries for it? thanks!

[Mateo-Lopez-Espejo]

hello

Any luck with this issue? I am having the same problem running with bonsai 2.8.5.

[Mateo-Lopez-Espejo]

solved by downgrading tensorflow 2.10 > 2.8
https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-windows-x86_64-2.8.0.zip