Use it on Windows

[zpge]

Very nice work. The code is well orgnized. But which part should I modify if I want to use it on windows platform?

[ryouchinsa]

Thanks for writing the issue.
This code was for cpu use-case not limited to the macOS.
We updated code to use a cuda device.

On Windows and the ONNX runtime environment, if you can use a gpu device, could you set the device from cpu to cuda:0, for example?

Sam sam;
std::string modelName = "mobile_sam";
if(modelName.find("sam_hq") != std::string::npos){
  sam.changeMode(HQSAM);
}else if(modelName.find("efficientsam") != std::string::npos){
  sam.changeMode(EfficientSAM);
}else if(modelName.find("edge_sam") != std::string::npos){
  sam.changeMode(EdgeSAM);
}
std::string pathEncoder = modelName + "/" + modelName + "_preprocess.onnx";
std::string pathDecoder = modelName + "/" + modelName + ".onnx";
std::string device = "cpu"; // cpu, cuda:0, etc
std::cout<<"loadModel started"<<std::endl;
bool successLoadModel = sam.loadModel(pathEncoder, pathDecoder, std::thread::hardware_concurrency(), device);
if(!successLoadModel){
  std::cout<<"loadModel error"<<std::endl;
  return 1;
}

[zpge]

Thanks for writing the issue. This code was for cpu use-case not limited to the macOS. We updated code to use a cuda device.

On Windows and the ONNX runtime environment, if you can use a gpu device, could you set the device from cpu to cuda:0, for example?

Sam sam;
std::string modelName = "mobile_sam";
if(modelName.find("sam_hq") != std::string::npos){
  sam.changeMode(HQSAM);
}else if(modelName.find("efficientsam") != std::string::npos){
  sam.changeMode(EfficientSAM);
}else if(modelName.find("edge_sam") != std::string::npos){
  sam.changeMode(EdgeSAM);
}
std::string pathEncoder = modelName + "/" + modelName + "_preprocess.onnx";
std::string pathDecoder = modelName + "/" + modelName + ".onnx";
std::string device = "cpu"; // cpu, cuda:0, etc
std::cout<<"loadModel started"<<std::endl;
bool successLoadModel = sam.loadModel(pathEncoder, pathDecoder, std::thread::hardware_concurrency(), device);
if(!successLoadModel){
  std::cout<<"loadModel error"<<std::endl;
  return 1;
}

Thanks for your reply. I do successfully compiled this project both on windows and ubuntu by replacing with the corresponding onnx files. Sometimes it works fine, but occasionally it report the same error:

"Unhandled exception at 0x00007FFB4009F79F (onnxruntime.dll) in sam_cpp_test.exe: 0xC0000005: Access violation reading location 0x000002925FEC3080."

I also compiled this project https://github.com/dinglufe/segment-anything-cpp-wrapper/tree/main. There is no such error. Do you happen to know what is happening?

[ryouchinsa]

Thanks for reporting the problem. Let us ask you some questions to reproduce the problem.
On Windows and Ubuntu, you can build the project.

cmake -S . -B build
cmake --build build

Sometimes when running the execution file, it raises the exception, correct?

./build/sam_cpp_test

[zpge]

On ubuntu, I do excatly use the above commonds. It reports the error: segment fault. I cannot get the detailed informtion.
On windows, I use cmake-gui and visual studio and I get the error mentioned above.

Btw, the error occurs at
sessionEncoder->Run(runOptionsEncoder, inputNames.data(), &inputTensor, 1, outputNames.data(), &outputTensor, 1);

On Windows, I only made very small changes.
In sam.cpp
I add the following lines to convert the path format.

#if _MSC_VER
std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter;
auto wpreModelPath = converter.from_bytes(encoderPath);
auto wsamModelPath = converter.from_bytes(decoderPath);
#else
auto wpreModelPath = encoderPath;
auto wsamModelPath = decoderPath;
#endif

sessionEncoder = std::make_unique<Ort::Session>(env, wpreModelPath.c_str(), sessionOptions[0]);

In sam.h
To export the dll, I changed

class Sam {

to

#if _MSC_VER
class __declspec(dllexport) Sam {
#else
class Sam {

Btw, from this page https://github.com/yformer/EfficientSAM/blob/main/export_to_onnx.py, I found the inputnames and input size is different from your code. Is the onnx file generated from the code in that page? Can you give it acheck?

[ryouchinsa]

Thanks for your detailed feedback.

For EfficientSAM, the developer says that we resize the input image to the size of 1024x1024 for model input.
yformer/EfficientSAM#11 (comment)

For input names, looking at export_onnx_esam_encoder() and export_onnx_esam_decoder(), our code has the same input names.

On Ubuntu, we will test soon and give you our feedback.

[ryouchinsa]

We updated our code so that we checked it works for both gpu and cpu on Ubuntu 22.04. The detailed document how to run sam-cpp-macos on Ubuntu will be updated on the top page.

[zpge]

We updated our code so that we checked it works for both gpu and cpu on Ubuntu 22.04. The detailed document how to run sam-cpp-macos on Ubuntu will be updated on the top page.

Thanks. Do you encounter the same problem? Is it because I used it in a wrong way?

[ryouchinsa]

The error did not happen. But the inference accuracy was bad. Because the preprocessing was not correctly done.

Your detailed feedback could help us.

Btw, the error occurs at
sessionEncoder->Run(runOptionsEncoder, inputNames.data(), &inputTensor, 1, outputNames.data(), &outputTensor, 1);

The function return type "auto" caused the problem.

Before fix.

auto inputTensor = getEncoderInputTensor(image);

auto Sam::getEncoderInputTensor(const cv::Mat& image){
  if(mode == EfficientSAM ||
     mode == EdgeSAM){
    std::vector<float> inputTensorValues(inputShapeEncoder[0] * inputShapeEncoder[1] * inputShapeEncoder[2] * inputShapeEncoder[3]);
    for(int i = 0; i < inputShapeEncoder[2]; i++){
      for(int j = 0; j < inputShapeEncoder[3]; j++){
        int64_t pos = i * inputShapeEncoder[3] + j;
        int64_t size = inputShapeEncoder[2] * inputShapeEncoder[3];
        inputTensorValues[pos + size * 0] = image.at<cv::Vec3b>(i, j)[2] / 255.0;
        inputTensorValues[pos + size * 1] = image.at<cv::Vec3b>(i, j)[1] / 255.0;
        inputTensorValues[pos + size * 2] = image.at<cv::Vec3b>(i, j)[0] / 255.0;
      }
    }
    auto inputTensor = Ort::Value::CreateTensor<float>(memoryInfo, inputTensorValues.data(), inputTensorValues.size(), inputShapeEncoder.data(), inputShapeEncoder.size());
    return inputTensor;
  }
  std::vector<uint8_t> inputTensorValues(inputShapeEncoder[0] * inputShapeEncoder[1] * inputShapeEncoder[2] * inputShapeEncoder[3]);
  for(int i = 0; i < inputShapeEncoder[2]; i++){
    for(int j = 0; j < inputShapeEncoder[3]; j++){
      int64_t pos = i * inputShapeEncoder[3] + j;
      int64_t size = inputShapeEncoder[2] * inputShapeEncoder[3];
      inputTensorValues[pos + size * 0] = image.at<cv::Vec3b>(i, j)[2];
      inputTensorValues[pos + size * 1] = image.at<cv::Vec3b>(i, j)[1];
      inputTensorValues[pos + size * 2] = image.at<cv::Vec3b>(i, j)[0];
    }
  }
  auto inputTensor = Ort::Value::CreateTensor<uint8_t>(memoryInfo, inputTensorValues.data(), inputTensorValues.size(), inputShapeEncoder.data(), inputShapeEncoder.size());
  return inputTensor;
}

After fix.

std::vector<float> inputTensorValuesFloat;
std::vector<uint8_t> inputTensorValuesInt;
bool isInputTensorFloat = (mode == EfficientSAM || mode == EdgeSAM);
if(isInputTensorFloat){
  inputTensorValuesFloat.resize(inputShapeEncoder[0] * inputShapeEncoder[1] * inputShapeEncoder[2] * inputShapeEncoder[3]);
  for(int i = 0; i < inputShapeEncoder[2]; i++){
    for(int j = 0; j < inputShapeEncoder[3]; j++){
      int64_t pos = i * inputShapeEncoder[3] + j;
      int64_t size = inputShapeEncoder[2] * inputShapeEncoder[3];
      inputTensorValuesFloat[pos + size * 0] = image.at<cv::Vec3b>(i, j)[2] / 255.0;
      inputTensorValuesFloat[pos + size * 1] = image.at<cv::Vec3b>(i, j)[1] / 255.0;
      inputTensorValuesFloat[pos + size * 2] = image.at<cv::Vec3b>(i, j)[0] / 255.0;
    }
  }
}else{
  inputTensorValuesInt.resize(inputShapeEncoder[0] * inputShapeEncoder[1] * inputShapeEncoder[2] * inputShapeEncoder[3]);
  for(int i = 0; i < inputShapeEncoder[2]; i++){
    for(int j = 0; j < inputShapeEncoder[3]; j++){
      int64_t pos = i * inputShapeEncoder[3] + j;
      int64_t size = inputShapeEncoder[2] * inputShapeEncoder[3];
      inputTensorValuesInt[pos + size * 0] = image.at<cv::Vec3b>(i, j)[2];
      inputTensorValuesInt[pos + size * 1] = image.at<cv::Vec3b>(i, j)[1];
      inputTensorValuesInt[pos + size * 2] = image.at<cv::Vec3b>(i, j)[0];
    }
  }
}
auto inputTensor = isInputTensorFloat ?
Ort::Value::CreateTensor<float>(memoryInfo, inputTensorValuesFloat.data(), inputTensorValuesFloat.size(), inputShapeEncoder.data(), inputShapeEncoder.size()) :
Ort::Value::CreateTensor<uint8_t>(memoryInfo, inputTensorValuesInt.data(), inputTensorValuesInt.size(), inputShapeEncoder.data(), inputShapeEncoder.size());

[ryouchinsa]

This is how to run on Ubuntu CPU/GPU environment.

Download an ONNX Runtime folder.

• onnxruntime-osx-universal2-1.17.1.tgz for macOS
• onnxruntime-linux-x64-1.17.1.tgz for Ubuntu CPU
• onnxruntime-linux-x64-gpu-1.17.1.tgz for Ubuntu GPU

For Ubuntu, install gflags and opencv through vcpkg.

git clone https://github.com/microsoft/vcpkg.git
./vcpkg/bootstrap-vcpkg.sh
./vcpkg/vcpkg install gflags
./vcpkg/vcpkg install opencv

For Ubuntu GPU, install cuda and cudnn.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-drivers
reboot
nvidia-smi

sudo apt install cuda-toolkit-11-8
vi ~/.bashrc
export PATH="/usr/local/cuda/bin${PATH:+:${PATH}}"
export LD_LIBRARY_PATH="/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
source ~/.bashrc
which nvcc
nvcc --version

apt list libcudnn8 -a
cudnn_version=8.9.7.29
cuda_version=cuda11.8
sudo apt install libcudnn8=${cudnn_version}-1+${cuda_version}
sudo apt install libcudnn8-dev=${cudnn_version}-1+${cuda_version}
sudo apt install libcudnn8-samples=${cudnn_version}-1+${cuda_version}

Build and run.

# macOS
cmake -S . -B build -DONNXRUNTIME_ROOT_DIR=/Users/ryo/Downloads/onnxruntime-osx-universal2-1.17.1
# Ubuntu CPU
cmake -S . -B build -DONNXRUNTIME_ROOT_DIR=/root/onnxruntime-linux-x64-1.17.1 -DCMAKE_TOOLCHAIN_FILE=/root/vcpkg/scripts/buildsystems/vcpkg.cmake
# Ubuntu GPU
cmake -S . -B build -DONNXRUNTIME_ROOT_DIR=/root/onnxruntime-linux-x64-gpu-1.17.1 -DCMAKE_TOOLCHAIN_FILE=/root/vcpkg/scripts/buildsystems/vcpkg.cmake

cmake --build build

# macOS and Ubuntu CPU
./build/sam_cpp_test -encoder="mobile_sam/mobile_sam_preprocess.onnx" -decoder="mobile_sam/mobile_sam.onnx" -image="david-tomaseti-Vw2HZQ1FGjU-unsplash.jpg" -device="cpu"
# Ubuntu GPU
./build/sam_cpp_test -encoder="mobile_sam/mobile_sam_preprocess.onnx" -decoder="mobile_sam/mobile_sam.onnx" -image="david-tomaseti-Vw2HZQ1FGjU-unsplash.jpg" -device="cuda:0"