Connection Error. Is Haystack running?

[sieu-tran]

Between October 18, 2021 and October 19, 2021, something has changed and the docker returns the error: "Connection Error. Is Haystack running?" when we start running it.

[grahamschuckman]

Also had an issue with testing Docker on a fresh download of the repo and Docker images (followed exact commands on README).

Log files:
haystack-api_1 | ML Logging is turned off. No parameters, metrics or artifacts will be logged to MLFlow.
haystack-api_1 | [2021-10-19 19:16:54 +0000] [10] [INFO] Started server process [10]
haystack-api_1 | [2021-10-19 19:16:54 +0000] [10] [INFO] Waiting for application startup.
haystack-api_1 | [2021-10-19 19:16:54 +0000] [10] [INFO] Application startup complete.
haystack-api_1 | [2021-10-19 19:16:54 +0000] [10] [ERROR] Exception in ASGI application
haystack-api_1 | Traceback (most recent call last):
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
haystack-api_1 | result = await app(self.scope, self.receive, self.send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call
haystack-api_1 | return await self.app(scope, receive, send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/fastapi/applications.py", line 208, in call
haystack-api_1 | await super().call(scope, receive, send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 112, in call
haystack-api_1 | await self.middleware_stack(scope, receive, send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 181, in call
haystack-api_1 | raise exc
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 159, in call
haystack-api_1 | await self.app(scope, receive, _send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/middleware/cors.py", line 84, in call
haystack-api_1 | await self.app(scope, receive, send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 82, in call
haystack-api_1 | raise exc
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 71, in call
haystack-api_1 | await self.app(scope, receive, sender)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 656, in call
haystack-api_1 | await route.handle(scope, receive, send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 259, in handle
haystack-api_1 | await self.app(scope, receive, send)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 61, in app
haystack-api_1 | response = await func(request)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 227, in app
haystack-api_1 | dependant=dependant, values=values, is_coroutine=is_coroutine
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 161, in run_endpoint_function
haystack-api_1 | return await run_in_threadpool(dependant.call, **values)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
haystack-api_1 | return await anyio.to_thread.run_sync(func, *args)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/anyio/to_thread.py", line 29, in run_sync
haystack-api_1 | limiter=limiter)
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 805, in run_sync_in_worker_thread
haystack-api_1 | return await future
haystack-api_1 | File "/usr/local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 743, in run
haystack-api_1 | result = func(*args)
haystack-api_1 | File "/home/user/rest_api/controller/search.py", line 48, in query
haystack-api_1 | result = _process_request(PIPELINE, request)
haystack-api_1 | File "/home/user/rest_api/controller/search.py", line 66, in _process_request
haystack-api_1 | result = pipeline.run(query=request.query, params=params)
haystack-api_1 | File "/home/user/haystack/pipeline.py", line 300, in run
haystack-api_1 | raise ValueError(f"No node(s) or global parameter(s) named {', '.join(invalid_keys)} found in pipeline.")
haystack-api_1 | ValueError: No node(s) or global parameter(s) named ESRetriever found in pipeline.
ui_1 | 2021-10-19 19:16:54.134 Expecting value: line 1 column 1 (char 0)
ui_1 | Traceback (most recent call last):
ui_1 | File "/home/user/webapp.py", line 134, in main
ui_1 | results, raw_json = retrieve_doc(question, top_k_reader=top_k_reader, top_k_retriever=top_k_retriever)
ui_1 | File "/usr/local/lib/python3.7/site-packages/streamlit/caching.py", line 545, in wrapped_func
ui_1 | return get_or_create_cached_value()
ui_1 | File "/usr/local/lib/python3.7/site-packages/streamlit/caching.py", line 527, in get_or_create_cached_value
ui_1 | return_value = func(*args, **kwargs)
ui_1 | File "/home/user/utils.py", line 30, in retrieve_doc
ui_1 | response_raw = requests.post(url, json=req).json()
ui_1 | File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 910, in json
ui_1 | return complexjson.loads(self.text, **kwargs)
ui_1 | File "/usr/local/lib/python3.7/json/init.py", line 348, in loads
ui_1 | return _default_decoder.decode(s)
ui_1 | File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
ui_1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
ui_1 | File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
ui_1 | raise JSONDecodeError("Expecting value", s, err.value) from None
ui_1 | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Seems to be some sort of JSON issue between the search.py and pipeline.py files with the UI?

[ZanSara]

Hi @sieu-tran and @grahamschuckman, thanks for the bug report! I can see from your stacktrace that this is due to a new procedure for the validation of pipeline parameters tha we merged to master yesterday. I'm going to bugfix this soon and let you know when the fix is out.

Thanks @grahamschuckman for creating the issue (#1618) , let's continue the discussion there :)

[ZanSara]

Just merged the fix, please try again and let me know if it works now 🙂

[rachadbn]

Hi,

I have the same issue, I cloned the latest Demo Repo from Haystack Github. I'm having a connection error.

I'm sharing the terminal log, maybe I can be helpful. Please let me know for any fixes.

Thank you
Rachad

You can now view your Streamlit app in your browser.

Network URL: http://172.17.0.2:8501
External URL: http://92.157.80.44:8501

ERROR:root:HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /initialized (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f129c416680>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn
sock = connection.create_connection(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 395, in request
self.endheaders()
File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/local/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 243, in connect
self.sock = self._new_conn()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 218, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f129c416680>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /initialized (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f129c416680>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/ui/ui/utils.py", line 27, in haystack_is_ready
if requests.get(url).status_code < 400:
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /initialized (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f129c416680>: Failed to establish a new connection: [Errno 111] Connection refused'))
ERROR:root:HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /initialized (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f129c415060>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn
sock = connection.create_connection(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 395, in request
self.endheaders()
File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
self.send(msg)
File "/usr/local/lib/python3.10/http/client.py", line 976, in send
self.connect()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 243, in connect
self.sock = self._new_conn()
File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 218, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f129c415060>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /initialized (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f129c415060>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/ui/ui/utils.py", line 27, in haystack_is_ready
if requests.get(url).status_code < 400:
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /initialized (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f129c415060>: Failed to establish a new connection: [Errno 111] Connection refused'))