Issue when running the "explore the world" demo in docker

[MariaDavid30]

I am currently trying to run the "explore the world" demo in my system. I am following readme of https://github.com/deepset-ai/haystack-demos/tree/main/explore_the_world to run the demo. I have installed the 3 containers(elasticsearch,haystack-API,ui). But when I run the 'docker-compose up -d' I get the following elastic search error.

And when I start the streamlit app,

I get the "Haystack is starting" command but does not start.

Elastic search started at port 9200

Guys please help!!

[JonathanLehner]

doesn't work for me either

[MariaDavid30]

Thank you for responding @JonathanLehner

[mayankjobanputra]

@MariaDavid30 @JonathanLehner It generally takes a min before the container becomes healthy. It can also happen that there's very low disk space on the device which is why elasticsearch container is failing.

Would you guys mind sharing the docker logs? So that we can help you debug the issue. You can output them by running the following command. Feel free to adjust the time as per your need.

docker logs --since=5m <container_id>

[MariaDavid30]

Hai @mayankjobanputra , tried pulling it again and I got the following error,

2023-03-24 10:14:46.567 {'_content': b'Traceback (most recent call last):\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call\n await self.app(scope, receive, _send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in call\n await self.app(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call\n raise exc\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call\n await self.app(scope, receive, sender)\n File "/opt/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call\n raise e\n File "/opt/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call\n await self.app(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/routing.py", line 706, in call\n await route.handle(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle\n await self.app(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/routing.py", line 66, in app\n response = await func(request)\n File "/opt/venv/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app\n raw_response = await run_endpoint_function(\n File "/opt/venv/lib/python3.10/site-packages/fastapi/routing.py", line 165, in run_endpoint_function\n return await run_in_threadpool(dependant.call, **values)\n File "/opt/venv/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool\n return await anyio.to_thread.run_sync(func, *args)\n File "/opt/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync\n return await get_asynclib().run_sync_in_worker_thread(\n File "/opt/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread\n return await future\n File "/opt/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run\n result = context.run(func, *args)\n File "/opt/venv/lib/python3.10/site-packages/rest_api/controller/search.py", line 58, in query\n result = _process_request(query_pipeline, request)\n File "/opt/venv/lib/python3.10/site-packages/haystack/telemetry.py", line 116, in wrapper\n return func(*args, **kwargs)\n File "/opt/venv/lib/python3.10/site-packages/rest_api/controller/search.py", line 67, in _process_request\n result = pipeline.run(query=request.query, params=params, debug=request.debug)\nAttributeError: 'NoneType' object has no attribute 'run'\n', '_content_consumed': True, '_next': None, 'status_code': 500, 'headers': {'date': 'Fri, 24 Mar 2023 04:44:45 GMT', 'server': 'uvicorn', 'content-length': '2643', 'content-type': 'text/plain; charset=utf-8'}, 'raw': <urllib3.response.HTTPResponse object at 0x000002C63A446590>, 'url': 'http://localhost:8000/query', 'encoding': 'utf-8', 'history': [], 'reason': 'Internal Server Error', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=99494), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x000002C63A459330>}
Traceback (most recent call last):
File "E:\Maria2\explore_the_world\ui\webapp.py", line 211, in main
st.session_state.results, st.session_state.raw_json = query(
File "E:\Maria2\explore_the_world\ui\utils.py", line 55, in query
raise Exception(f"{vars(response_raw)}")
Exception: {'_content': b'Traceback (most recent call last):\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call\n await self.app(scope, receive, _send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in call\n await self.app(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call\n raise exc\n File "/opt/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call\n await self.app(scope, receive, sender)\n File "/opt/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call\n raise e\n File "/opt/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call\n await self.app(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/routing.py", line 706, in call\n await route.handle(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle\n await self.app(scope, receive, send)\n File "/opt/venv/lib/python3.10/site-packages/starlette/routing.py", line 66, in app\n response = await func(request)\n File "/opt/venv/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app\n raw_response = await run_endpoint_function(\n File "/opt/venv/lib/python3.10/site-packages/fastapi/routing.py", line 165, in run_endpoint_function\n return await run_in_threadpool(dependant.call, **values)\n File "/opt/venv/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool\n return await anyio.to_thread.run_sync(func, *args)\n File "/opt/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync\n return await get_asynclib().run_sync_in_worker_thread(\n File "/opt/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread\n return await future\n File "/opt/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run\n result = context.run(func, *args)\n File "/opt/venv/lib/python3.10/site-packages/rest_api/controller/search.py", line 58, in query\n result = _process_request(query_pipeline, request)\n File "/opt/venv/lib/python3.10/site-packages/haystack/telemetry.py", line 116, in wrapper\n return func(*args, **kwargs)\n File "/opt/venv/lib/python3.10/site-packages/rest_api/controller/search.py", line 67, in _process_request\n result = pipeline.run(query=request.query, params=params, debug=request.debug)\nAttributeError: 'NoneType' object has no attribute 'run'\n', '_content_consumed': True, '_next': None, 'status_code': 500, 'headers': {'date': 'Fri, 24 Mar 2023 04:44:45 GMT', 'server': 'uvicorn', 'content-length': '2643', 'content-type': 'text/plain; charset=utf-8'}, 'raw': <urllib3.response.HTTPResponse object at 0x000002C63A446590>, 'url': 'http://localhost:8000/query', 'encoding': 'utf-8', 'history': [], 'reason': 'Internal Server Error', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(microseconds=99494), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x000002C63A459330>}

Streamlit web app

http://localhost:8000/

[mayankjobanputra]

@MariaDavid30 It seems like the pipeline is not getting initialized inside haystack-api container.

I am assuming you cleaned up old docker containers and images from your system. Can you please confirm that?

If not, after cleaning up everything, can you try running docker compose command again??

If you still face the same issue, would you mind sharing the logs of haystack-api and UI containers?

[MariaDavid30]

@mayankjobanputra I had cleaned up old docker containers and images. I ran it again, and this time I did not get the error, but haystack is not starting.

Haystack logs

C:\Users\user>docker logs --since=5m 02f8680bafdf

INFO:haystack.modeling.model.language_model:Auto-detected model language: english
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'DPRQuestionEncoderTokenizer'.
The class this function is called from is 'DPRContextEncoderTokenizerFast'.
INFO:haystack.modeling.model.language_model:Auto-detected model language: english
INFO:haystack.modeling.utils:Using devices: CPU - Number of GPUs: 0
INFO:haystack.modeling.utils:Using devices: CPU - Number of GPUs: 0

Explore the world Demo containers, did not have any logs

C:\Users\user>docker logs --since=5m b95f249bfcfc

Elastic search container logs

C:\Users\user>docker logs --since=5m 3e7a7c4d1b4f
{"type": "server", "timestamp": "2023-03-24T14:30:36,075Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [66056ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:30:57,032Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@4b84f8ff, interval=5s}] took [7091ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:17,998Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@b346386, interval=1s}] took [10208ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:33,281Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [5.7s/5787ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:37,336Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [5.7s/5786941600ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:41,313Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.1s/8166ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:41,313Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5988/0x00000008019e7390@7e4a0b2e] took [8166ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:45,947Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.1s/8166144500ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:48,363Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [6.6s/6618ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:50,330Z", "level": "WARN", "component": "o.e.m.f.FsHealthService", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "health check of [/usr/share/elasticsearch/data/nodes/0] took [8166ms] which is above the warn threshold of [5s]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:52,196Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [6.6s/6617540900ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:56,638Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.1s/8165ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:31:58,371Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.1s/8165017800ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:03,116Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [6.6s/6642ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:06,996Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [6.6s/6641734900ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:12,641Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [9.1s/9106ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:17,829Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [9.1s/9106091300ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:24,137Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [12.5s/12568ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:25,011Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [30530ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:29,503Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [12.5s/12568486300ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:33,811Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [9.3s/9332ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:36,994Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [9.3s/9331982900ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:43,427Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [7.4s/7409ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:32:52,460Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [7.4s/7408474700ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:00,176Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [18.6s/18606ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:04,234Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [18.6s/18606818800ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:08,568Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.9s/8914ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:12,060Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.6s/8646385500ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:12,893Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [34661ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:18,984Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.7s/8758ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:19,287Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [org.elasticsearch.indices.IndicesService$CacheCleaner@3a652e91] took [8757ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:25,184Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [8.7s/8757789200ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:30,651Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [12.8s/12870ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:39,247Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [12.8s/12870338800ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:46,591Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [15.1s/15129ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:33:57,088Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [15.5s/15523821200ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:34:00,779Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [14.6s/14608ms] on absolute clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:34:00,779Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@4b84f8ff, interval=5s}] took [43002ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:34:03,771Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "timer thread slept for [14.6s/14607883900ns] on relative clock which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:34:12,434Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [6202ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:34:23,266Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [org.elasticsearch.cluster.InternalClusterInfoService$RefreshScheduler$$Lambda$5988/0x00000008019e7390@2c21fb90] took [8358ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:34:34,983Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [5604ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:35:17,539Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [10698ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-24T14:35:55,095Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "3e7a7c4d1b4f", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@650efc2a, interval=5s}] took [5404ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }

[mayankjobanputra]

@MariaDavid30 I don't see any errors either... seems like Elasticsearch container is still loading; please wait for some time. It can take a few mins to spawn and get running.

[MariaDavid30]

Hey @mayankjobanputra , I deleted the containers and images and tried running the demo again. I have waited for 1 and a half hours and this is what I got.

streamlit port

haystack port

Elastic search port

haystack logs

C:\Users\user>docker logs --since=5m 328533821dd9
[2023-03-30 04:41:44 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:39)
[2023-03-30 04:41:52 +0000] [1] [WARNING] Worker with pid 39 was terminated due to signal 6

Elastic search logs

C:\Users\user>docker logs --since=5m 431f38cd988c
{"type": "server", "timestamp": "2023-03-30T04:37:04,308Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "431f38cd988c", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@39a20716, interval=5s}] took [67783ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }
{"type": "server", "timestamp": "2023-03-30T04:41:50,846Z", "level": "WARN", "component": "o.e.t.ThreadPool", "cluster.name": "docker-cluster", "node.name": "431f38cd988c", "message": "execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@39a20716, interval=5s}] took [267418ms] which is above the warn threshold of [5000ms]", "cluster.uuid": "OGF5ISXBRG2DQDrNikNGBw", "node.id": "lYGpRq3cQiShrpNA1w30nQ" }

[JonathanLehner]

How much disk space is required? For me the elasticsearch seems to work, but I get this error (same in explore the world, and in doc-search tutorials):

[mayankjobanputra]

@JonathanLehner seems like it's not haystack issue but gunicorn issue. benoitc/gunicorn#2840

We have changed our docker files to uvicorn but maybe we missed changing it for the demo.