Pybind11 and python sys.path

[tdavidcl]

Hello,

Sorry if this is a duplicate but I've not managed to find any working solutions to my problem

The issue in my case is that for standard python when I print sys.path I get the system paths and the .local ones. But when i compile an embedded the .local paths are missing for an unknown reason.

I use pybind11 with cmake like so

find_package(pybind11 CONFIG REQUIRED)
...
target_link_libraries(main PUBLIC pybind11::embed)

and i set the python executable manually.

Why is the sys.path different from the standard one ?

Nb:
Without pybind11 with the following cmake config the .local paths are present in sys.path

find_package (Python3 COMPONENTS Interpreter Development )
target_link_libraries(main PUBLIC Python3::Python)

[Lord-KA]

Hi, am am facing the same issue. On ubuntu I have the following in the CMakeLists:

  target_link_libraries(main PUBLIC pybind11::pybind11 pybind11::embed Python3::Python)

but the sys.path inside the embedded interpreter, that I start with py::initialize_interpreter() is lacking .local path to pip installed modules. @tdavidcl, would you mind telling some more details about your fix?

[tdavidcl]

just for reference currently in cmake i'm using to import pybind11 :

set(PYBIND11_FINDPYTHON ON)
add_subdirectory(pybind11)
target_link_libraries(libcore PUBLIC pybind11::embed)

also it is not that important actually, whatever i do with cmake i still get the issue

[Lord-KA]

Simplified reproducer. You are right, set(PYBIND11_FINDPYTHON ON) doesn't make a difference.

[tdavidcl]

actually i can reproduce it without the .py file

CMakeLists.txt :

cmake_minimum_required(VERSION 3.1)

include(FetchContent)

add_executable(main ./main.cpp)

FetchContent_Declare(
  pybind11
  GIT_REPOSITORY https://github.com/pybind/pybind11.git
  GIT_TAG        v2.10.4
)

FetchContent_GetProperties(pybind11)
if(NOT pybind11_POPULATED)
  FetchContent_Populate(pybind11)
  add_subdirectory(./build/_deps/pybind11-src)
endif()

target_link_libraries(main pybind11::embed)

main.cpp

#include <pybind11/stl.h>
#include <pybind11/pybind11.h>
#include <pybind11/embed.h>
#include <pybind11/numpy.h>

namespace py = pybind11;

int main()
{
    py::scoped_interpreter guard{};

    py::exec("import sys\nprint(sys.path)");
}

running gives :

> ./main 
['', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/lib/python3.10/site-packages']

in normal python :

In [1]: import sys

In [2]: print(sys.path)
['/usr/bin', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '', '/home/xxx/.local/lib/python3.10/site-packages', '/usr/lib/python3.10/site-packages']

[tdavidcl]

I've opened a issue with this reproducer
#4654

[Lord-KA]

Thank you!