GPU issues with REST API

[OMGAmici]

I am not sure what is going wrong here, but lately whenever I try to launch the REST API using GPU, I get this message below.

I have tried doing a fresh clone of the haystack repo, but I still get the same error message.

Command I use from the haystack root directory: docker compose -f docker-compose-gpu.yml up

haystack_model-haystack-api-1   | Error processing line 1 of /usr/local/lib/python3.7/dist-packages/hatch_autorun_farm_haystack.pth:
haystack_model-haystack-api-1   |
haystack_model-haystack-api-1   |   Traceback (most recent call last):
haystack_model-haystack-api-1   |     File "/usr/lib/python3.7/site.py", line 174, in addpackage
haystack_model-haystack-api-1   |       exec(line)
haystack_model-haystack-api-1   |     File "<string>", line 1, in <module>
haystack_model-haystack-api-1   |     File "<string>", line 6, in <module>
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/haystack/__init__.py", line 18, in <module>
haystack_model-haystack-api-1   |       import pandas as pd
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/__init__.py", line 22, in <module>
haystack_model-haystack-api-1   |       from pandas.compat import (
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/compat/__init__.py", line 15, in <module>
haystack_model-haystack-api-1   |       from pandas.compat.numpy import (
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/compat/numpy/__init__.py", line 7, in <module>
haystack_model-haystack-api-1   |       from pandas.util.version import Version
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/util/__init__.py", line 1, in <module>
haystack_model-haystack-api-1   |       from pandas.util._decorators import (  # noqa
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/util/_decorators.py", line 14, in <module>
haystack_model-haystack-api-1   |       from pandas._libs.properties import cache_readonly  # noqa
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/_libs/__init__.py", line 13, in <module>
haystack_model-haystack-api-1   |       from pandas._libs.interval import Interval
haystack_model-haystack-api-1   |     File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
haystack_model-haystack-api-1   |     File "pandas/_libs/hashtable.pyx", line 1, in init pandas._libs.hashtable
haystack_model-haystack-api-1   |     File "pandas/_libs/missing.pyx", line 1, in init pandas._libs.missing
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/pandas/_libs/tslibs/__init__.py", line 31, in <module>
haystack_model-haystack-api-1   |       from pandas._libs.tslibs.conversion import (
haystack_model-haystack-api-1   |     File "pandas/_libs/tslibs/conversion.pyx", line 1, in init pandas._libs.tslibs.conversion
haystack_model-haystack-api-1   |     File "pandas/_libs/tslibs/timezones.pyx", line 14, in init pandas._libs.tslibs.timezones
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/dateutil/tz/__init__.py", line 2, in <module>
haystack_model-haystack-api-1   |       from .tz import *
haystack_model-haystack-api-1   |     File "/usr/local/lib/python3.7/dist-packages/dateutil/tz/tz.py", line 19, in <module>
haystack_model-haystack-api-1   |       import six
haystack_model-haystack-api-1   |   ModuleNotFoundError: No module named 'six'
haystack_model-haystack-api-1   |
haystack_model-haystack-api-1   | Remainder of file ignored
haystack_model-haystack-api-1   | [2022-11-15 21:59:27 +0000] [1] [INFO] Starting gunicorn 20.1.0
haystack_model-haystack-api-1   | [2022-11-15 21:59:27 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
haystack_model-haystack-api-1   | [2022-11-15 21:59:27 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
haystack_model-haystack-api-1   | [2022-11-15 21:59:27 +0000] [21] [INFO] Booting worker with pid: 21
haystack_model-haystack-api-1   | [2022-11-15 21:59:31 +0000] [21] [ERROR] Exception in worker process
haystack_model-haystack-api-1   | Traceback (most recent call last):
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/arbiter.py", line 589, in spawn_worker
haystack_model-haystack-api-1   |     worker.init_process()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/uvicorn/workers.py", line 66, in init_process
haystack_model-haystack-api-1   |     super(UvicornWorker, self).init_process()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/workers/base.py", line 134, in init_process
haystack_model-haystack-api-1   |     self.load_wsgi()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/workers/base.py", line 146, in load_wsgi
haystack_model-haystack-api-1   |     self.wsgi = self.app.wsgi()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
haystack_model-haystack-api-1   |     self.callable = self.load()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/app/wsgiapp.py", line 58, in load
haystack_model-haystack-api-1   |     return self.load_wsgiapp()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
haystack_model-haystack-api-1   |     return util.import_app(self.app_uri)
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/gunicorn/util.py", line 359, in import_app
haystack_model-haystack-api-1   |     mod = importlib.import_module(module)
haystack_model-haystack-api-1   |   File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
haystack_model-haystack-api-1   |     return _bootstrap._gcd_import(name[level:], package, level)
haystack_model-haystack-api-1   |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
haystack_model-haystack-api-1   |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
haystack_model-haystack-api-1   |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
haystack_model-haystack-api-1   |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
haystack_model-haystack-api-1   |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
haystack_model-haystack-api-1   |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/rest_api/application.py", line 13, in <module>
haystack_model-haystack-api-1   |     app = get_app()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/rest_api/utils.py", line 28, in get_app
haystack_model-haystack-api-1   |     from rest_api.controller import file_upload, search, feedback, document, health
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/rest_api/controller/file_upload.py", line 20, in <module>
haystack_model-haystack-api-1   |     indexing_pipeline: Pipeline = get_pipelines().get("indexing_pipeline", None)
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/rest_api/utils.py", line 59, in get_pipelines
haystack_model-haystack-api-1   |     pipelines = setup_pipelines()
haystack_model-haystack-api-1   |   File "/usr/local/lib/python3.7/dist-packages/rest_api/pipeline/__init__.py", line 28, in setup_pipelines
haystack_model-haystack-api-1   |     query_pipeline = Pipeline.load_from_yaml(Path(config.PIPELINE_YAML_PATH), pipeline_name=config.QUERY_PIPELINE_NAME)
haystack_model-haystack-api-1   |   File "/home/user/haystack/pipelines/base.py", line 1890, in load_from_yaml
haystack_model-haystack-api-1   |     strict_version_check=strict_version_check,
haystack_model-haystack-api-1   |   File "/home/user/haystack/pipelines/base.py", line 1947, in load_from_config
haystack_model-haystack-api-1   |     validate_config(pipeline_config, strict_version_check=strict_version_check)
haystack_model-haystack-api-1   |   File "/home/user/haystack/pipelines/config.py", line 242, in validate_config
haystack_model-haystack-api-1   |     validate_schema(pipeline_config=pipeline_config, strict_version_check=strict_version_check, extras=extras)
haystack_model-haystack-api-1   |   File "/home/user/haystack/pipelines/config.py", line 298, in validate_schema
haystack_model-haystack-api-1   |     with open(JSON_SCHEMAS_PATH / f"haystack-pipeline-main.schema.json", "r") as schema_file:
haystack_model-haystack-api-1   | FileNotFoundError: [Errno 2] No such file or directory: '/home/user/haystack/json-schemas/haystack-pipeline-main.schema.json'
haystack_model-haystack-api-1   | [2022-11-15 21:59:31 +0000] [21] [INFO] Worker exiting (pid: 21)
haystack_model-haystack-api-1   | [2022-11-15 21:59:31 +0000] [1] [INFO] Shutting down: Master
haystack_model-haystack-api-1   | [2022-11-15 21:59:31 +0000] [1] [INFO] Reason: Worker failed to boot.
haystack_model-haystack-api-1 exited with code 3

[OMGAmici]

I think this might have something to do with the recent refactoring I noticed that removed the json-schemas/haystack-pipeline-main.schema.json file. Issues 3485 and 3484.

Not surprisingly, if I use the json-schemas folder from version 1.9 of the source code and paste that into my project directory, the errors resolve.

I have tried installing and re-installing haystack both using pip with farm-haystack and by using pip from the cloned directory.

The fact that the error message seems to be looking for python3.7, but that directory is missing in my case might also have something to do with it:

[TuanaCelik]

Hey @OMGAmici there is a PR that I believe are fixing an issue that would cause this scenario: They're #3533 and there's also an issue at #3505

I suggest keeping an eye out on those. Seems that the team is already on this