Using the Haystack Streamlit UI with FAISSDocumentStore instead of Elasticsearch

[shreyanssethi]

I was wondering what changes I would need to make to the code so that when I use docker-compose to launch the UI, it is relying on a FAISS Document Store instead of an ElasticSearch one.

I have already changed:

1. PIPELINE_YAML_PATH environment variable in /haystack/rest_api/config.py to a different yaml file which uses a FAISS Document store instead of an Elasticsearch one

2. The haystack/docker-compose.yml file was changed so that the haystack-api depends on a faissdocumentstore, which is defined as a separate service with a defined docker image (https://hub.docker.com/r/daangn/faiss-server)

3. Uncommented this out in the haystack/docker-compose.yml file so that the custom yaml file for the new pipeline can be loaded in:

      - ./rest_api/pipeline:/home/user/rest_api/pipeline```
     


Despite these changes, when trying to launch the UI using docker-compose, I still get the following error trace so I don't think I'm making changes in all the necessary places. Any suggestions regarding this would be very helpful:

```(base) [root@bdand-1 haystack]# docker-compose up
Creating haystack_faissdocumentstore_1 ... done
Creating haystack_ui_1                 ... 
Creating haystack_haystack-api_1       ... done
Creating haystack_haystack-api_1       ... 
Attaching to haystack_ui_1, haystack_faissdocumentstore_1, haystack_haystack-api_1
faissdocumentstore_1  | 2022-08-30 21:52:31,833:INFO:root - server loading...
faissdocumentstore_1  | 2022-08-30 21:52:31,834:INFO:root - max workers: 1
faissdocumentstore_1  | 2022-08-30 21:52:31,841:INFO:root - server started
haystack-api_1        | [2022-08-30 21:52:42 +0000] [1] [INFO] Starting gunicorn 20.1.0
haystack-api_1        | [2022-08-30 21:52:42 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
haystack-api_1        | [2022-08-30 21:52:42 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
haystack-api_1        | [2022-08-30 21:52:42 +0000] [11] [INFO] Booting worker with pid: 11
haystack-api_1        | [2022-08-30 21:52:42 +0000] [12] [INFO] Booting worker with pid: 12
haystack-api_1        | /usr/local/lib/python3.7/site-packages/transformers/image_utils.py:222: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
haystack-api_1        |   def resize(self, image, size, resample=PIL.Image.BILINEAR, default_to_square=True, max_size=None):
haystack-api_1        | INFO:haystack.pipelines.config:Param 'host' of component 'DocumentStore' overwritten with environment variable 'DOCUMENTSTORE_PARAMS_HOST' value 'faissdocumentstore'.
haystack-api_1        | /usr/local/lib/python3.7/site-packages/transformers/image_utils.py:222: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
haystack-api_1        |   def resize(self, image, size, resample=PIL.Image.BILINEAR, default_to_square=True, max_size=None):
haystack-api_1        | INFO:haystack.pipelines.config:Param 'host' of component 'DocumentStore' overwritten with environment variable 'DOCUMENTSTORE_PARAMS_HOST' value 'faissdocumentstore'.
haystack-api_1        | INFO:haystack.telemetry:Haystack sends anonymous usage data to understand the actual usage and steer dev efforts towards features that are most meaningful to users. You can opt-out at anytime by calling disable_telemetry() or by manually setting the environment variable HAYSTACK_TELEMETRY_ENABLED as described for different operating systems on the documentation page. More information at https://haystack.deepset.ai/guides/telemetry
haystack-api_1        | INFO:haystack.telemetry:Haystack sends anonymous usage data to understand the actual usage and steer dev efforts towards features that are most meaningful to users. You can opt-out at anytime by calling disable_telemetry() or by manually setting the environment variable HAYSTACK_TELEMETRY_ENABLED as described for different operating systems on the documentation page. More information at https://haystack.deepset.ai/guides/telemetry
haystack-api_1        | INFO:haystack.pipelines.config:Param 'host' of component 'DocumentStore' overwritten with environment variable 'DOCUMENTSTORE_PARAMS_HOST' value 'faissdocumentstore'.
haystack-api_1        | INFO:haystack.pipelines.config:Param 'host' of component 'DocumentStore' overwritten with environment variable 'DOCUMENTSTORE_PARAMS_HOST' value 'faissdocumentstore'.
haystack-api_1        | [2022-08-30 21:52:47 +0000] [11] [ERROR] Exception in worker process
haystack-api_1        | Traceback (most recent call last):
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1943, in _load_or_get_component
haystack-api_1        |     component_type=component_type, component_params=component_params, name=name
haystack-api_1        |   File "/home/user/haystack/nodes/base.py", line 158, in _create_instance
haystack-api_1        |     instance = subclass(**component_params)
haystack-api_1        |   File "/home/user/haystack/nodes/base.py", line 48, in wrapper_exportable_to_yaml
haystack-api_1        |     init_func(self, *args, **kwargs)
haystack-api_1        | TypeError: __init__() got an unexpected keyword argument 'host'
haystack-api_1        | 
haystack-api_1        | The above exception was the direct cause of the following exception:
haystack-api_1        | 
haystack-api_1        | Traceback (most recent call last):
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1937, in _load_or_get_component
haystack-api_1        |     cls._load_or_get_component(name=value, definitions=definitions, components=components)
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1956, in _load_or_get_component
haystack-api_1        |     ) from e
haystack-api_1        | haystack.errors.PipelineConfigError: Failed loading pipeline component 'DocumentStore'. See the stacktrace above for more informations.
haystack-api_1        | 
haystack-api_1        | The above exception was the direct cause of the following exception:
haystack-api_1        | 
haystack-api_1        | Traceback (most recent call last):
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
haystack-api_1        |     worker.init_process()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/uvicorn/workers.py", line 66, in init_process
haystack-api_1        |     super(UvicornWorker, self).init_process()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process
haystack-api_1        |     self.load_wsgi()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
haystack-api_1        |     self.wsgi = self.app.wsgi()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
haystack-api_1        |     self.callable = self.load()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
haystack-api_1        |     return self.load_wsgiapp()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
haystack-api_1        |     return util.import_app(self.app_uri)
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app
haystack-api_1        |     mod = importlib.import_module(module)
haystack-api_1        |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
haystack-api_1        |     return _bootstrap._gcd_import(name[level:], package, level)
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
haystack-api_1        |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
haystack-api_1        |   File "/home/user/rest_api/application.py", line 13, in <module>
haystack-api_1        |     app = get_app()
haystack-api_1        |   File "/home/user/rest_api/utils.py", line 28, in get_app
haystack-api_1        |     from rest_api.controller import file_upload, search, feedback, document
haystack-api_1        |   File "/home/user/rest_api/controller/file_upload.py", line 20, in <module>
haystack-api_1        |     indexing_pipeline: Pipeline = get_pipelines().get("indexing_pipeline", None)
haystack-api_1        |   File "/home/user/rest_api/utils.py", line 58, in get_pipelines
haystack-api_1        |     pipelines = setup_pipelines()
haystack-api_1        |   File "/home/user/rest_api/pipeline/__init__.py", line 28, in setup_pipelines
haystack-api_1        |     query_pipeline = Pipeline.load_from_yaml(Path(config.PIPELINE_YAML_PATH), pipeline_name=config.QUERY_PIPELINE_NAME)
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1840, in load_from_yaml
haystack-api_1        |     strict_version_check=strict_version_check,
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1907, in load_from_config
haystack-api_1        |     name=node_config["name"], definitions=component_definitions, components=components
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1956, in _load_or_get_component
haystack-api_1        |     ) from e
haystack-api_1        | haystack.errors.PipelineConfigError: Failed loading pipeline component 'Retriever'. See the stacktrace above for more informations.
haystack-api_1        | [2022-08-30 21:52:47 +0000] [12] [ERROR] Exception in worker process
haystack-api_1        | Traceback (most recent call last):
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1943, in _load_or_get_component
haystack-api_1        |     component_type=component_type, component_params=component_params, name=name
haystack-api_1        |   File "/home/user/haystack/nodes/base.py", line 158, in _create_instance
haystack-api_1        |     instance = subclass(**component_params)
haystack-api_1        |   File "/home/user/haystack/nodes/base.py", line 48, in wrapper_exportable_to_yaml
haystack-api_1        |     init_func(self, *args, **kwargs)
haystack-api_1        | TypeError: __init__() got an unexpected keyword argument 'host'
haystack-api_1        | 
haystack-api_1        | The above exception was the direct cause of the following exception:
haystack-api_1        | 
haystack-api_1        | Traceback (most recent call last):
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1937, in _load_or_get_component
haystack-api_1        |     cls._load_or_get_component(name=value, definitions=definitions, components=components)
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1956, in _load_or_get_component
haystack-api_1        |     ) from e
haystack-api_1        | haystack.errors.PipelineConfigError: Failed loading pipeline component 'DocumentStore'. See the stacktrace above for more informations.
haystack-api_1        | 
haystack-api_1        | The above exception was the direct cause of the following exception:
haystack-api_1        | 
haystack-api_1        | Traceback (most recent call last):
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
haystack-api_1        |     worker.init_process()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/uvicorn/workers.py", line 66, in init_process
haystack-api_1        |     super(UvicornWorker, self).init_process()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process
haystack-api_1        |     self.load_wsgi()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
haystack-api_1        |     self.wsgi = self.app.wsgi()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
haystack-api_1        |     self.callable = self.load()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
haystack-api_1        |     return self.load_wsgiapp()
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
haystack-api_1        |     return util.import_app(self.app_uri)
haystack-api_1        |   File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app
haystack-api_1        |     mod = importlib.import_module(module)
haystack-api_1        |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
haystack-api_1        |     return _bootstrap._gcd_import(name[level:], package, level)
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
haystack-api_1        |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
haystack-api_1        |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
haystack-api_1        |   File "/home/user/rest_api/application.py", line 13, in <module>
haystack-api_1        |     app = get_app()
haystack-api_1        |   File "/home/user/rest_api/utils.py", line 28, in get_app
haystack-api_1        |     from rest_api.controller import file_upload, search, feedback, document
haystack-api_1        |   File "/home/user/rest_api/controller/file_upload.py", line 20, in <module>
haystack-api_1        |     indexing_pipeline: Pipeline = get_pipelines().get("indexing_pipeline", None)
haystack-api_1        |   File "/home/user/rest_api/utils.py", line 58, in get_pipelines
haystack-api_1        |     pipelines = setup_pipelines()
haystack-api_1        |   File "/home/user/rest_api/pipeline/__init__.py", line 28, in setup_pipelines
haystack-api_1        |     query_pipeline = Pipeline.load_from_yaml(Path(config.PIPELINE_YAML_PATH), pipeline_name=config.QUERY_PIPELINE_NAME)
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1840, in load_from_yaml
haystack-api_1        |     strict_version_check=strict_version_check,
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1907, in load_from_config
haystack-api_1        |     name=node_config["name"], definitions=component_definitions, components=components
haystack-api_1        |   File "/home/user/haystack/pipelines/base.py", line 1956, in _load_or_get_component
haystack-api_1        |     ) from e
haystack-api_1        | haystack.errors.PipelineConfigError: Failed loading pipeline component 'Retriever'. See the stacktrace above for more informations.
haystack-api_1        | [2022-08-30 21:52:47 +0000] [11] [INFO] Worker exiting (pid: 11)
haystack-api_1        | [2022-08-30 21:52:47 +0000] [12] [INFO] Worker exiting (pid: 12)```
     

[vblagoje]

Hey @shreyanssethi can you share your docker-compose.yml file? Just paste it here using code blocks.