Docker Installation Fails

[FullMetalNicky]

I tried to build the library with Docker, after pulling the Github code.
git clone --recursive https://github.com/mapillary/OpenSfM
Looks like despite pulling the submodules, the ThirdParty files for gtest are missing, resulting in the following error

44.28 -- Performing Test HAS_FLTO
44.36 -- Performing Test HAS_FLTO - Success
44.36 -- LTO enabled
44.37 -- Configuring done
44.37 CMake Error at CMakeLists.txt:112 (add_library):
44.37 Cannot find source file:
44.37
44.37 testing_main.cc
44.37
44.37 Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
44.37 .hpp .hxx .in .txx
44.37
44.37
44.37 CMake Error at CMakeLists.txt:106 (add_library):
44.37 Cannot find source file:
44.37
44.37 third_party/gtest/gmock_gtest_all.cc
44.37
44.37 Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
44.37 .hpp .hxx .in .txx
44.37
44.37
44.37 CMake Error at CMakeLists.txt:112 (add_library):
44.37 No SOURCES given to target: test_main
44.37
44.37
44.37 CMake Error at CMakeLists.txt:106 (add_library):
44.37 No SOURCES given to target: gtest
44.37
44.37
44.37 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
44.37 No SOURCES given to target: vl
44.37
44.37
44.37 CMake Generate step failed. Build files cannot be regenerated correctly.
44.38 Configuring for python 3.8...
44.38 Traceback (most recent call last):
44.38 File "setup.py", line 59, in
44.38 configure_c_extension()
44.38 File "setup.py", line 43, in configure_c_extension
44.38 subprocess.check_call(cmake_command, cwd="cmake_build")
44.38 File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
44.38 raise CalledProcessError(retcode, cmd)
44.38 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.

Dockerfile:30

29 |
30 | >>> RUN pip3 install -r requirements.txt &&
31 | >>> python3 setup.py build
32 |

ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt && python3 setup.py build" did not complete successfully: exit code: 1
make: *** [Makefile:2: build] Error 1

Please advise on how to solve this. The whole point of providing a Docker is to have a smooth and painless installation...

[MysteryHS]

I'm experiencing the same issue even though I'm not using docker to build the tool.

[DiegoHerrera1890]

Is there any update in this matter?

[alessandro-pistola-me]

The most frequent errors encountered are:

1. pyyaml installation error (cython error with pyyaml version 5.4.1)
2. pybundle error
3. third_party/gtest error

By reviewing the commit history from August 9 (commit ef872b2399cf1cc036d91e950f210a7be33c2745) onward, you’ll notice that all workflows for subsequent commits, like "Docker CI / Build docker and run tests (push)" and "Docker CI / Build docker installing CeresSolver 2 and run tests (push)," fail.

Below is a step-by-step guide explaining how I resolved these issues.

The missing files should be taken from the release, but only those for the third_party folder (missing gtest files), as the release uses outdated dependencies and likely Ceres 2.0.

To begin, create two clones of the latest repository:

• One will serve as the main repository.
• The other will be used to retrieve the missing parts.

In the root of the main repo, run:
`git submodule update --init --recursive
This will load the removed submodules.

In the main repo, remove pyyaml==5.4.1 from requirements.txt and install it manually with the following command:
pip install "cython<3.0.0" && pip install --no-build-isolation pyyaml==5.1

In the second repo, go to the August 9 commit:
git checkout ef872b2399cf1cc036d91e950f210a7be33c2745

From this commit, copy the missing files (gmock_gtest_all.cc, gmock_main.cc, testing_main.cc, and any other required files) into the main repository.
Now you should be able to build the image from the modified dockerfile in which you should install ceres 2.2 insted ceres 2.0
RUN \ curl -L http://ceres-solver.org/ceres-solver-2.2.0.tar.gz | tar xz && \ mkdir ceres-bin && cd ceres-bin && \ cmake ../ceres-solver-2.2.0 && \ make -j3 && \ make install

[DiegoHerrera1890]

I did exactly what you said but now I have this error:

42.59 -- Configuring done
42.60 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
42.60   No SOURCES given to target: vl
42.60 
42.60 
42.60 CMake Generate step failed.  Build files cannot be regenerated correctly.
42.60 Configuring for python 3.8...
42.60 Traceback (most recent call last):
42.60   File "setup.py", line 59, in <module>
42.60     configure_c_extension()
42.60   File "setup.py", line 43, in configure_c_extension
42.60     subprocess.check_call(cmake_command, cwd="cmake_build")
42.60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
42.60     raise CalledProcessError(retcode, cmd)
42.60 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.
------
Dockerfile.ceres2:41
--------------------
  40 |     
  41 | >>> RUN pip3 install -r requirements.txt && \
  42 | >>>     python3 setup.py build
  43 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt &&     python3 setup.py build" did not complete successfully: exit code: 1

It seems to be related to vlfeat...
Do you have any idea how to solve this?

[alessandro-pistola-me]

I did exactly what you said but now I have this error:

42.59 -- Configuring done
42.60 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
42.60   No SOURCES given to target: vl
42.60 
42.60 
42.60 CMake Generate step failed.  Build files cannot be regenerated correctly.
42.60 Configuring for python 3.8...
42.60 Traceback (most recent call last):
42.60   File "setup.py", line 59, in <module>
42.60     configure_c_extension()
42.60   File "setup.py", line 43, in configure_c_extension
42.60     subprocess.check_call(cmake_command, cwd="cmake_build")
42.60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
42.60     raise CalledProcessError(retcode, cmd)
42.60 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.
------
Dockerfile.ceres2:41
--------------------
  40 |     
  41 | >>> RUN pip3 install -r requirements.txt && \
  42 | >>>     python3 setup.py build
  43 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt &&     python3 setup.py build" did not complete successfully: exit code: 1

It seems to be related to vlfeat... Do you have any idea how to solve this?

did you import OpenSfM/opensfm/src/third_party/vlfeat/vl from the "release" repository?

[NyxAither]

I'm having the same issue on windows, not using docker.

[DiegoHerrera1890]

I did exactly what you said but now I have this error:

42.59 -- Configuring done
42.60 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
42.60   No SOURCES given to target: vl
42.60 
42.60 
42.60 CMake Generate step failed.  Build files cannot be regenerated correctly.
42.60 Configuring for python 3.8...
42.60 Traceback (most recent call last):
42.60   File "setup.py", line 59, in <module>
42.60     configure_c_extension()
42.60   File "setup.py", line 43, in configure_c_extension
42.60     subprocess.check_call(cmake_command, cwd="cmake_build")
42.60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
42.60     raise CalledProcessError(retcode, cmd)
42.60 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.
------
Dockerfile.ceres2:41
--------------------
  40 |     
  41 | >>> RUN pip3 install -r requirements.txt && \
  42 | >>>     python3 setup.py build
  43 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt &&     python3 setup.py build" did not complete successfully: exit code: 1

It seems to be related to vlfeat... Do you have any idea how to solve this?

did you import OpenSfM/opensfm/src/third_party/vlfeat/vl from the "release" repository?

I run git submodule update --init --recursive and the OpenSfM/opensfm/src/third_party/vlfeat/vl folder only has three items:

(base) diego@diego:~/OpenSfM/opensfm/src/third_party/vlfeat/vl$ ls
ikmeans_elkan.tc  ikmeans_init.tc  ikmeans_lloyd.tc

[alessandro-pistola-me]

I did exactly what you said but now I have this error:

42.59 -- Configuring done
42.60 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
42.60   No SOURCES given to target: vl
42.60 
42.60 
42.60 CMake Generate step failed.  Build files cannot be regenerated correctly.
42.60 Configuring for python 3.8...
42.60 Traceback (most recent call last):
42.60   File "setup.py", line 59, in <module>
42.60     configure_c_extension()
42.60   File "setup.py", line 43, in configure_c_extension
42.60     subprocess.check_call(cmake_command, cwd="cmake_build")
42.60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
42.60     raise CalledProcessError(retcode, cmd)
42.60 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.
------
Dockerfile.ceres2:41
--------------------
  40 |     
  41 | >>> RUN pip3 install -r requirements.txt && \
  42 | >>>     python3 setup.py build
  43 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt &&     python3 setup.py build" did not complete successfully: exit code: 1

It seems to be related to vlfeat... Do you have any idea how to solve this?

did you import OpenSfM/opensfm/src/third_party/vlfeat/vl from the "release" repository?

I run git submodule update --init --recursive and the OpenSfM/opensfm/src/third_party/vlfeat/vl folder only has three items:

(base) diego@diego:~/OpenSfM/opensfm/src/third_party/vlfeat/vl$ ls
ikmeans_elkan.tc  ikmeans_init.tc  ikmeans_lloyd.tc

Try to import the folder content from the release.
Btw I managed to build the docker image but OpenSfM is still not running, this is the error:
Traceback (most recent call last): File "/code/OpenSfM/bin/opensfm_main.py", line 10, in <module> from opensfm import commands File "/code/OpenSfM/opensfm/__init__.py", line 2, in <module> from opensfm import pybundle ImportError: cannot import name 'pybundle' from partially initialized module 'opensfm' (most likely due to a circular import) (/code/OpenSfM/opensfm/__init__.py)

[DiegoHerrera1890]

You mean copying the files from the release folder?
I did that, but a bunch of new errors arose.

[alessandro-pistola-me]

You mean copying the files from the release folder? I did that, but a bunch of new errors arose.

Then you can try from this commit: ef872b2
But as I told you previously It seems the docker build fails anyway (also in their github workflows)

[dedmu5]

Using git checkout 110619 the error disappears

[KuchikiRukia7]

The most frequent errors encountered are:

1. pyyaml installation error (cython error with pyyaml version 5.4.1)
2. pybundle error
3. third_party/gtest error

By reviewing the commit history from August 9 (commit ef872b2399cf1cc036d91e950f210a7be33c2745) onward, you’ll notice that all workflows for subsequent commits, like "Docker CI / Build docker and run tests (push)" and "Docker CI / Build docker installing CeresSolver 2 and run tests (push)," fail.

Below is a step-by-step guide explaining how I resolved these issues.

The missing files should be taken from the release, but only those for the third_party folder (missing gtest files), as the release uses outdated dependencies and likely Ceres 2.0.

To begin, create two clones of the latest repository:

• One will serve as the main repository.
• The other will be used to retrieve the missing parts.

In the root of the main repo, run: `git submodule update --init --recursive This will load the removed submodules.

In the main repo, remove pyyaml==5.4.1 from requirements.txt and install it manually with the following command: pip install "cython<3.0.0" && pip install --no-build-isolation pyyaml==5.1

In the second repo, go to the August 9 commit: git checkout ef872b2399cf1cc036d91e950f210a7be33c2745

From this commit, copy the missing files (gmock_gtest_all.cc, gmock_main.cc, testing_main.cc, and any other required files) into the main repository. Now you should be able to build the image from the modified dockerfile in which you should install ceres 2.2 insted ceres 2.0 RUN \ curl -L http://ceres-solver.org/ceres-solver-2.2.0.tar.gz | tar xz && \ mkdir ceres-bin && cd ceres-bin && \ cmake ../ceres-solver-2.2.0 && \ make -j3 && \ make install

Hello!Where are the missing files?Copy from the August 9 commit?Or from the sconde repo?Thanks!

[SvetimFM]

I did exactly what you said but now I have this error:

42.59 -- Configuring done
42.60 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
42.60   No SOURCES given to target: vl
42.60 
42.60 
42.60 CMake Generate step failed.  Build files cannot be regenerated correctly.
42.60 Configuring for python 3.8...
42.60 Traceback (most recent call last):
42.60   File "setup.py", line 59, in <module>
42.60     configure_c_extension()
42.60   File "setup.py", line 43, in configure_c_extension
42.60     subprocess.check_call(cmake_command, cwd="cmake_build")
42.60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
42.60     raise CalledProcessError(retcode, cmd)
42.60 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.
------
Dockerfile.ceres2:41
--------------------
  40 |     
  41 | >>> RUN pip3 install -r requirements.txt && \
  42 | >>>     python3 setup.py build
  43 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt &&     python3 setup.py build" did not complete successfully: exit code: 1

It seems to be related to vlfeat... Do you have any idea how to solve this?

did you import OpenSfM/opensfm/src/third_party/vlfeat/vl from the "release" repository?

I run git submodule update --init --recursive and the OpenSfM/opensfm/src/third_party/vlfeat/vl folder only has three items:

(base) diego@diego:~/OpenSfM/opensfm/src/third_party/vlfeat/vl$ ls
ikmeans_elkan.tc  ikmeans_init.tc  ikmeans_lloyd.tc

Try to import the folder content from the release. Btw I managed to build the docker image but OpenSfM is still not running, this is the error: Traceback (most recent call last): File "/code/OpenSfM/bin/opensfm_main.py", line 10, in <module> from opensfm import commands File "/code/OpenSfM/opensfm/__init__.py", line 2, in <module> from opensfm import pybundle ImportError: cannot import name 'pybundle' from partially initialized module 'opensfm' (most likely due to a circular import) (/code/OpenSfM/opensfm/__init__.py)

This is where I find myself too, any paths forward?

[alessandro-pistola-me]

I did exactly what you said but now I have this error:

42.59 -- Configuring done
42.60 CMake Error at third_party/vlfeat/CMakeLists.txt:11 (add_library):
42.60   No SOURCES given to target: vl
42.60 
42.60 
42.60 CMake Generate step failed.  Build files cannot be regenerated correctly.
42.60 Configuring for python 3.8...
42.60 Traceback (most recent call last):
42.60   File "setup.py", line 59, in <module>
42.60     configure_c_extension()
42.60   File "setup.py", line 43, in configure_c_extension
42.60     subprocess.check_call(cmake_command, cwd="cmake_build")
42.60   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
42.60     raise CalledProcessError(retcode, cmd)
42.60 subprocess.CalledProcessError: Command '['cmake', '../opensfm/src', '-DPYTHON_EXECUTABLE=/usr/bin/python3']' returned non-zero exit status 1.
------
Dockerfile.ceres2:41
--------------------
  40 |     
  41 | >>> RUN pip3 install -r requirements.txt && \
  42 | >>>     python3 setup.py build
  43 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt &&     python3 setup.py build" did not complete successfully: exit code: 1

It seems to be related to vlfeat... Do you have any idea how to solve this?

did you import OpenSfM/opensfm/src/third_party/vlfeat/vl from the "release" repository?

I run git submodule update --init --recursive and the OpenSfM/opensfm/src/third_party/vlfeat/vl folder only has three items:

(base) diego@diego:~/OpenSfM/opensfm/src/third_party/vlfeat/vl$ ls
ikmeans_elkan.tc  ikmeans_init.tc  ikmeans_lloyd.tc

Try to import the folder content from the release. Btw I managed to build the docker image but OpenSfM is still not running, this is the error: Traceback (most recent call last): File "/code/OpenSfM/bin/opensfm_main.py", line 10, in <module> from opensfm import commands File "/code/OpenSfM/opensfm/__init__.py", line 2, in <module> from opensfm import pybundle ImportError: cannot import name 'pybundle' from partially initialized module 'opensfm' (most likely due to a circular import) (/code/OpenSfM/opensfm/__init__.py)

This is where I find myself too, any paths forward?

I gave up, now Im using this image https://hub.docker.com/r/dronet/opensfm