[Question]: Linking onnxruntime, shared library hell on MacOs

[rdcm]

Hi,

Right now I'm working on porting a Python service that uses silero-vad and sherpa-onnx to Rust. For model inference in the Rust version of the application, I’m using:

• https://github.com/pykeio/ort
• https://github.com/thewh1teagle/sherpa-rs

Example of how the libraries are included:

sherpa-rs = { version = "0.6.6", features = ["sys", "download-binaries" ] }
ort = { version = "=2.0.0-rc.9", features = ["ndarray", "cuda", "load-dynamic", "download-binaries"] }

At the moment, I’ve run into an issue that could be described as “shared library hell,” as mentioned in the documentation:

1. sherpa-rs copies its own version of libonnxruntime.dylib (version 1.17) into target/debug.

2. ort apparently detects that libonnxruntime.dylib is already loaded and doesn’t try to load its own.

3. At runtime, when I call the ort API, I get the error: "The requested API version [20] is not available, only API versions [1, 17] are supported in this build".

This brings me to a couple of questions:

1. Where does ort get libonnxruntime.dylib from, and why doesn’t it appear in target/debug even when I’m only using ort?
   My current hypothesis is that ort looks it up from somewhere in the cache:
   /Library/Caches/ort.pyke.io/dfbin/x86_64-apple-darwin/{hash}/onnxruntime/lib/libonnxruntime.a.

2. What is the recommended way to configure ort in my case?
   So far, it seems like I need to manually download a matching ONNX Runtime distribution and specify its path using ORT_DYLIB_PATH.

Thanks

[decahedron1]

1. Where does ort get libonnxruntime.dylib from, and why doesn’t it appear in target/debug even when I’m only using ort?

ort does not provide dylibs on macOS, it links to ONNX Runtime statically (in this case, though, it doesn't link at all since load-dynamic is also enabled). The libonnxruntime.dylib is coming from sherpa.

2. What is the recommended way to configure ort in my case?

I'd say just override sherpa's libonnxruntime.dylib with a newer version.

[rdcm]

I'd say just override sherpa's libonnxruntime.dylib with a newer version.

My current solution:

1. Upload latest bin files to libs folder from onnxruntime release page.
2. Copy libonnxruntime.1.21.0.dylib to target/debug via custom build.rs script
3. Enable load-dynamic feature
4. Set ORT_DYLIB_PATH to target/debug/libonnxruntime.1.21.0.dylib

Are there any other solutions to my problem? I would like ort to work with its own version of the library, and sherpa-rs with its own.

[decahedron1]

I don't know if sherpa uses dlopen like ort's load-dynamic does, or if it is using compile-time dynamic linking. If it is, then another solution would be to let ort use its static libraries by disabling load-dynamic, then extracting any logic using sherpa-rs to its own crate which is linked as a dylib (or the other way around, instead splitting out ort logic). That way, the sherpa code & ort code are in completely separate binaries, and the libraries they link to don't interfere.

I would like ort to work with its own version of the library, and sherpa-rs with its own.

Why?

[csukuangfj]

FYI: To build sherpa-onnx with a different version of onnxruntime, please have a look at
k2-fsa/sherpa-onnx#1719

[rdcm]

@csukuangfj

Returning to this discussion.
Is it possible to use something like ORT_DYLIB_PATH in sherpa-onnx?

[csukuangfj]

Sorry, I don't know what ORT_DYLIB_PATH is.

I know about LD_LIBRARY_PATH and DYLD_LIBRARY_PATH and LIBRARY_PATH.

[rdcm]

Is it possible to specify the path to libonnxruntime.so for sherpa-onnx using LD_LIBRARY_PATH?

[csukuangfj]

Do you mean when building sherpa-onnx or when running sherpa-onnx after the build?

[rdcm]

I see several possible approaches to address the libonnxruntime.so version mismatch issue:

1. Use the same ONNX Runtime version across all libraries by rebuilding them from source, as shown here.

2. Dynamically specify the required library version at runtime using an environment variable like ORT_DYLIB_PATH (if supported).

3. Isolate models into separate processes, avoiding symbol conflicts between versions.

4. Load modules manually at runtime using libloading (my current approach), for example:

fn main() {
    _ = dotenv();
    let ort_module = load_module("ORT_MODULE");
    let sherpa_module = load_module("SHERPA_MODULE");

    unsafe {
        (ort_module.run)();
        (sherpa_module.run)();
    }
}

In any case, the ability to explicitly specify the desired libonnxruntime.so version at runtime would be a very helpful feature.

[decahedron1]

Hey @rdcm, sorry but I'm locking this issue as this discussion is no longer related to ort. Feel free to open a new issue if you think there's anything I can do in ort to help. You should also bring up these points in k2-fsa/sherpa-onnx & thewh1teagle/sherpa-rs#106 if you haven't already.