Installation problems #5
Description
I've been trying to install Keras and Tensorflow from R following the instructions in the book, but I'm encountering an issue that I can't solve even following the discussion here.
First I installed the libraries on Windows using RStudio and everything worked perfectly. However, I only got it to work on CPU, not GPU. To facilitate the installation of dependencies to use the GPU I followed the instructions in the book and used Windows Subsystem for Linux to install Linux on my computer.
Once I did this, using Visual Studio Code with WSL to run the same code, the installation does not work. The end result is as follows:
Error: Valid installation of TensorFlow not found.
Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10
My full code so far:
install.packages("keras")
Installing package into ‘/home/dberm/R/x86_64-pc-linux-gnu-library/4.1’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/keras_2.11.1.tar.gz'
Content type 'application/x-gzip' length 3527604 bytes (3.4 MB)downloaded 3.4 MB
- installing source package ‘keras’ ...
** package ‘keras’ successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path- DONE (keras)
The downloaded source packages are in
‘/tmp/RtmpDj8V9u/downloaded_packages’
library(reticulate)
virtualenv_create("r-reticulate", python = install_python())
virtualenv: r-reticulate
library(keras)
install_keras(envname = "r-reticulate")
Using virtual environment 'r-reticulate' ...
'/home/dberm/.virtualenvs/r-reticulate/bin/python' -m pip install --upgrade --no-user --ignore-installed 'tensorflow==2.11.' 'tensorflow-hub' 'tensorflow-datasets' 'scipy' 'requests' 'Pillow' 'h5py' 'pandas' 'pydot'
Collecting tensorflow==2.11.
Using cached tensorflow-2.11.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (588.3 MB)
Collecting tensorflow-hub
Using cached tensorflow_hub-0.13.0-py2.py3-none-any.whl (100 kB)
Collecting tensorflow-datasets
Using cached tensorflow_datasets-4.8.3-py3-none-any.whl (5.4 MB)
Collecting scipy
Using cached scipy-1.10.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting Pillow
Using cached Pillow-9.4.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.4 MB)
Collecting h5py
Using cached h5py-3.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
Collecting pandas
Using cached pandas-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
Collecting pydot
Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Collecting astunparse>=1.6.0
Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting opt-einsum>=2.3.2
Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting typing-extensions>=3.6.6
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting google-pasta>=0.1.1
Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting grpcio<2.0,>=1.24.3
Using cached grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
Collecting tensorflow-estimator<2.12,>=2.11.0
Using cached tensorflow_estimator-2.11.0-py2.py3-none-any.whl (439 kB)
Collecting termcolor>=1.1.0
Using cached termcolor-2.2.0-py3-none-any.whl (6.6 kB)
Collecting libclang>=13.0.0
Using cached libclang-15.0.6.1-py2.py3-none-manylinux2010_x86_64.whl (21.5 MB)
Collecting packaging
Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting tensorboard<2.12,>=2.11
Using cached tensorboard-2.11.2-py3-none-any.whl (6.0 MB)
Collecting wrapt>=1.11.0
Using cached wrapt-1.15.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
Collecting flatbuffers>=2.0
Using cached flatbuffers-23.3.3-py2.py3-none-any.whl (26 kB)
Collecting tensorflow-io-gcs-filesystem>=0.23.1
Using cached tensorflow_io_gcs_filesystem-0.31.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)
Collecting numpy>=1.20
Using cached numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting protobuf<3.20,>=3.9.2
Using cached protobuf-3.19.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting absl-py>=1.0.0
Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Collecting keras<2.12,>=2.11.0
Using cached keras-2.11.0-py2.py3-none-any.whl (1.7 MB)
Collecting setuptools
Using cached setuptools-67.6.0-py3-none-any.whl (1.1 MB)
Collecting gast<=0.4.0,>=0.2.1
Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting six>=1.12.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting promise
Using cached promise-2.3.tar.gz (19 kB)
Preparing metadata (setup.py) ... done
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting dm-tree
Using cached dm_tree-0.1.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (153 kB)
Collecting tensorflow-metadata
Using cached tensorflow_metadata-1.12.0-py3-none-any.whl (52 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting etils[enp,epath]>=0.9.0
Using cached etils-1.1.1-py3-none-any.whl (115 kB)
Collecting psutil
Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
Collecting click
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting charset-normalizer<4,>=2
Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting pytz>=2020.1
Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Collecting python-dateutil>=2.8.1
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pyparsing>=2.1.4
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting wheel<1.0,>=0.23.0
Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Collecting zipp
Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting importlib_resources
Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
Collecting markdown>=2.6.8
Using cached Markdown-3.4.2-py3-none-any.whl (93 kB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting google-auth<3,>=1.6.3
Using cached google_auth-2.16.2-py2.py3-none-any.whl (177 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting werkzeug>=1.0.1
Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
Collecting googleapis-common-protos<2,>=1.52.0
Using cached googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting importlib-metadata>=4.4
Using cached importlib_metadata-6.1.0-py3-none-any.whl (21 kB)
Collecting MarkupSafe>=2.1.1
Using cached MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting pyasn1<0.5.0,>=0.4.6
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Building wheels for collected packages: promise
Building wheel for promise (setup.py) ... error
error: subprocess-exited-with-error× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [34 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-c9bisgzm/promise_af6168e0cb2e42adaeca9a8025bee6aa/setup.py", line 28, in
setup(
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/init.py", line 108, in setup
return distutils.core.setup(**attrs)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 172, in setup
ok = dist.parse_command_line()
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 475, in parse_command_line
args = self._parse_command_opts(parser, args)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 1119, in _parse_command_opts
nargs = _Distribution._parse_command_opts(self, parser, args)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 534, in _parse_command_opts
cmd_class = self.get_command_class(command)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 966, in get_command_class
self.cmdclass[command] = cmdclass = ep.load()
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/init.py", line 208, in load
module = import_module(match.group('module'))
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 28, in
from .macosx_libfile import calculate_macosx_platform_tag
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 43, in
import ctypes
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for promise
Running setup.py clean for promise
Failed to build promise
Installing collected packages: tensorboard-plugin-wit, pytz, pyasn1, libclang, flatbuffers, dm-tree, zipp, wrapt, wheel, urllib3, typing-extensions, tqdm, toml, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, setuptools, rsa, pyparsing, pyasn1-modules, psutil, protobuf, Pillow, packaging, oauthlib, numpy, MarkupSafe, keras, idna, grpcio, gast, etils, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tensorflow-hub, scipy, requests, python-dateutil, pydot, promise, opt-einsum, importlib_resources, importlib-metadata, h5py, googleapis-common-protos, google-pasta, google-auth, astunparse, tensorflow-metadata, requests-oauthlib, pandas, markdown, tensorflow-datasets, google-auth-oauthlib, tensorboard, tensorflow
Running setup.py install for promise ... done
DEPRECATION: promise was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 23.1 will enforce this behaviour change. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at Deprecate call tosetup.py installwhen building a wheel failed for source distributions without pyproject.toml pypa/pip#8368
Successfully installed MarkupSafe-2.1.2 Pillow-9.4.0 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 dm-tree-0.1.8 etils-1.1.1 flatbuffers-23.3.3 gast-0.4.0 google-auth-2.16.2 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 googleapis-common-protos-1.59.0 grpcio-1.51.3 h5py-3.8.0 idna-3.4 importlib-metadata-6.1.0 importlib_resources-5.12.0 keras-2.11.0 libclang-15.0.6.1 markdown-3.4.2 numpy-1.24.2 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-23.0 pandas-1.5.3 promise-2.3 protobuf-3.19.6 psutil-5.9.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.2 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 scipy-1.10.1 setuptools-67.6.0 six-1.16.0 tensorboard-2.11.2 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.11.1 tensorflow-datasets-4.8.3 tensorflow-estimator-2.11.0 tensorflow-hub-0.13.0 tensorflow-io-gcs-filesystem-0.31.0 tensorflow-metadata-1.12.0 termcolor-2.2.0 toml-0.10.2 tqdm-4.65.0 typing-extensions-4.5.0 urllib3-1.26.15 werkzeug-2.2.3 wheel-0.40.0 wrapt-1.15.0 zipp-3.15.0Installation complete.
tensorflow::tf_config()
Valid installation of TensorFlow not found.
Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10Python exception encountered:
Traceback (most recent call last):
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/init.py", line 37, in
from tensorflow.python.tools import module_util as _module_util
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/python/init.py", line 24, in
import ctypes
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in
from _ctypes import Union, Structure, Array
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
ModuleNotFoundError: No module named '_ctypes'You can install TensorFlow using the install_tensorflow() function.
If I try to use tensorflow::install_tensorflow() before tensorflow::tf_config() the result is the same. I also tried the instructions from the RStudio website, with the same results:
install.packages("tensorflow")
library(reticulate)
path_to_python <- install_python()
virtualenv_create("r-reticulate", python = path_to_python)
library(tensorflow)
install_tensorflow(envname = "r-reticulate")
install.packages("keras")
library(keras)
install_keras(envname = "r-reticulate")
tensorflow::tf_config()
As far as I can tell by the output, tensorflow::tf_config() is looking for Tensorflow here:
Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10
while Tensorflor should be in envname = "r-reticulate", but I'm not used to work with Python enviroments so I don't know to change this behaviour.
I also read here this comment:
Additionally, you must be running an Arm native build of R, not the x86 build running under Rosetta.
Which I don't know if applies to my case or is just for Intel Mac.
I don't know what else to try as I have been several days stuck here and I lack the required Python knowledge to make any progress. Any help would be appreciated.