SSL error with HTTP bot

[thezolin]

Hello All,

I'm facing some errors when I try to use the http_verify_cert flag in HTTP Collector bots, as described below:

Your help is appreciated.

Bot config inside runtime.yaml file

Bots:

  bot_id: Bots
  description: Fetch reports from an URL
  enabled: true
  group: Collector
  module: intelmq.bots.collectors.http.collector_http
  name: HTTP
  parameters:
    bottype: Collector
    code: ''
    destination_queues:
      _default: [MISPAPI-Output-queue]
    documentation: ''
    extract_files: false
    gpg_keyring: ''
    http_verify_cert: false
    http_header: {}
    http_password: ''
    http_url: https://lists.blocklist.de/lists/bots.txt
    http_url_formatting: ''
    http_username: ''
    provider: Blocklist.de
    rate_limit: 86400
    signature_url: ''
    signature_url_formatting: false
    ssl_client_cert: ''
    ssl_client_certificate: ''
    verify_pgp_signatures: false
  run_mode: continuous

Error:

2025-04-02T10:26:23.583000 - Bots - INFO - Loading destination pipeline and queues {'_default': ['MISPAPI-Output-queue']}.
2025-04-02T10:26:23.608000 - Bots - INFO - Connected to destination queues.
2025-04-02T10:26:23.608000 - Bots - INFO - Downloading report from 'https://lists.blocklist.de/lists/bots.txt'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 363, in start
    self.process()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/collectors/http/collector_http.py", line 84, in process
    resp = self.http_get(http_url)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 103, in http_get
    return self.__session.get(url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 29, in send
    return super().send(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

2025-04-02T10:26:23.905000 - Bots - INFO - Bot will continue in 15 seconds.
2025-04-02T10:26:38.920000 - Bots - INFO - Downloading report from 'https://lists.blocklist.de/lists/bots.txt'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 363, in start
    self.process()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/collectors/http/collector_http.py", line 84, in process
    resp = self.http_get(http_url)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 103, in http_get
    return self.__session.get(url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 29, in send
    return super().send(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

2025-04-02T10:26:39.024000 - Bots - INFO - Bot will continue in 15 seconds.
2025-04-02T10:26:54.039000 - Bots - INFO - Downloading report from 'https://lists.blocklist.de/lists/bots.txt'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 363, in start
    self.process()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/collectors/http/collector_http.py", line 84, in process
    resp = self.http_get(http_url)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 103, in http_get
    return self.__session.get(url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 29, in send
    return super().send(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

2025-04-02T10:26:54.141000 - Bots - INFO - Bot will continue in 15 seconds.
2025-04-02T10:27:09.156000 - Bots - INFO - Downloading report from 'https://lists.blocklist.de/lists/bots.txt'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py", line 363, in start
    self.process()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/collectors/http/collector_http.py", line 84, in process
    resp = self.http_get(http_url)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 103, in http_get
    return self.__session.get(url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/mixins/http.py", line 29, in send
    return super().send(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

2025-04-02T10:27:09.266000 - Bots - INFO - Idling for 86400.0s (1d) now.

[kamil-certat]

Do I understand correctly, that you tried to disable the verification, and then the issue came? (http_verify_cert: false) Or did you copied the configuration after disabling the verification as a workaround?

Could you try to check in the environment, where IntelMQ is installed, the following commands:

python3 -c 'import requests; requests.get("https://lists.blocklist.de/lists/bots.txt", verify=True)'
python3 -c 'import requests; requests.get("https://lists.blocklist.de/lists/bots.txt", verify=False)'

If any of them fails, the issue is rather not with the bot itself

[thezolin]

Hi @kamil-certat,

This is the result of those commands:

root@626c8ab92a4d:/opt# python3 -c 'import requests; requests.get("https://lists.blocklist.de/lists/bots.txt", verify=True)'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.9/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/lib/python3.9/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lists.blocklist.de', port=443): Max retries exceeded with url: /lists/bots.txt (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1137)')))
root@626c8ab92a4d:/opt# python3 -c 'import requests; requests.get("https://lists.blocklist.de/lists/bots.txt", verify=False)'
/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lists.blocklist.de'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(

[sebix]

I just tested http_verify_cert locally and that works fine.

Can you please describe your environment? Which operating system, how did you install IntelMQ, which version of Python and requests do you have installed?

[thezolin]

Hi @sebix,

I'm using Docker image to run IntelMQ, I just follow these instructions: https://intelmq.readthedocs.io/en/latest/user/installation.html#docker

Thanks for your help @sebix and @kamil-certat

[kamil-certat]

I've just did the same in a fresh container, and it worked :/ Could you please use curl (outside of the container or install it inside - in best case check both) and pass the results?

Command: curl -v -s -o /dev/null https://lists.blocklist.de/lists/bots.txt

I'm interested in the IP and certificate data you get, for example I'm getting:

curl -v -s -o /dev/null https://lists.blocklist.de/lists/bots.txt
*   Trying 144.76.114.3:443...
* Connected to lists.blocklist.de (144.76.114.3) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [25 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2897 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.2 (OUT), TLS header, Finished (20):
} [5 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.blocklist.de
*  start date: Apr 22 21:45:16 2024 GMT
*  expire date: May 22 21:45:15 2025 GMT
*  subjectAltName: host "lists.blocklist.de" matched cert's "*.blocklist.de"
*  issuer: C=PL; O=Unizeto Technologies S.A.; OU=Certum Certification Authority; CN=Certum Domain Validation CA SHA2
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
> GET /lists/bots.txt HTTP/1.1
> Host: lists.blocklist.de
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 02 Apr 2025 12:13:14 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 10301
< Connection: keep-alive
< Keep-Alive: timeout=20
< Last-Modified: Wed, 02 Apr 2025 12:13:04 GMT
< ETag: "283d-631ca9347f114"
< Accept-Ranges: bytes
< X-Frame-Options: sameorigin
< Strict-Transport-Security: max-age=31536000
< X-Frame-Options: SAMEORIGIN
< 
{ [10301 bytes data]
* Connection #0 to host lists.blocklist.de left intact

so the IP is 144.76.114.3
and the certificate:

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.blocklist.de
*  start date: Apr 22 21:45:16 2024 GMT
*  expire date: May 22 21:45:15 2025 GMT
*  subjectAltName: host "lists.blocklist.de" matched cert's "*.blocklist.de"
*  issuer: C=PL; O=Unizeto Technologies S.A.; OU=Certum Certification Authority; CN=Certum Domain Validation CA SHA2
*  SSL certificate verify ok.

My suspicious is that in your environment, there is either a proxy required or a firewall performing deep traffic introspection, so the certificate we get is not trusted by default. Could it be the case?

[kamil-certat]

BTW, please use docs.intelmq.org, the readthedocs.io is no longer updated

[thezolin]

Hi @kamil-certat,

Thanks for the tips.

This is my result after run curl outside the container

azureuser@CDCMISPUAT01:~$ curl -v -s -o /dev/null https://lists.blocklist.de/lists/bots.txt

• Trying 144.76.114.3:443...
• Connected to lists.blocklist.de (144.76.114.3) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
  } [512 bytes data]
• TLSv1.2 (IN), TLS header, Certificate Status (22):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Server hello (2):
  { [122 bytes data]
• TLSv1.2 (IN), TLS header, Finished (20):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  { [25 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Certificate (11):
  { [2897 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
  { [264 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Finished (20):
  { [52 bytes data]
• TLSv1.2 (OUT), TLS header, Finished (20):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  } [1 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Finished (20):
  } [52 bytes data]
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server accepted to use http/1.1
• Server certificate:
• subject: CN=*.blocklist.de
• start date: Apr 22 21:45:16 2024 GMT
• expire date: May 22 21:45:15 2025 GMT
• subjectAltName: host "lists.blocklist.de" matched cert's "*.blocklist.de"
• issuer: C=PL; O=Unizeto Technologies S.A.; OU=Certum Certification Authority; CN=Certum Domain Validation CA SHA2
• SSL certificate verify ok.
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
  } [5 bytes data]

GET /lists/bots.txt HTTP/1.1
Host: lists.blocklist.de
User-Agent: curl/7.81.0
Accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [281 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [281 bytes data]
• old SSL session ID is stale, removing
• TLSv1.2 (IN), TLS header, Supplemental data (23):
  { [5 bytes data]
• Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Server: nginx
  < Date: Wed, 02 Apr 2025 12:27:08 GMT
  < Content-Type: text/plain; charset=UTF-8
  < Content-Length: 10247
  < Connection: keep-alive
  < Keep-Alive: timeout=20
  < Last-Modified: Wed, 02 Apr 2025 12:24:05 GMT
  < ETag: "2807-631cabaa9c527"
  < Accept-Ranges: bytes
  < X-Frame-Options: sameorigin
  < Strict-Transport-Security: max-age=31536000
  < X-Frame-Options: SAMEORIGIN
  <
  { [10247 bytes data]
• Connection #0 to host lists.blocklist.de left intact

[thezolin]

I will try to do a fresh install.
And update the info here.

Thanks a lot!

[sebix]

Can you please show the output of intelmqctl --version?

Even if a proxy is in between and uses an untrusted certificate, turning off the certificate verification should still work.

[thezolin]

intelmqctl --version

root@626c8ab92a4d:/opt# intelmqctl --version
3.4.0

[thezolin]

After perform a fresh install, works:

root@70949970d694:/opt# intelmqctl log Bots
2025-04-02T12:42:49.979000 - Bots - INFO - Connected to destination queues.
2025-04-02T12:42:49.980000 - Bots - INFO - PGP signature verification is active.
2025-04-02T12:42:49.980000 - Bots - INFO - Bot initialization completed.
2025-04-02T12:42:49.995000 - Bots - INFO - Loading source pipeline and queue 'Bots-queue'.
2025-04-02T12:42:50.013000 - Bots - INFO - Connected to source queue.
2025-04-02T12:42:50.013000 - Bots - INFO - Loading destination pipeline and queues {'_default': ['MISPAPI-Output-queue']}.
2025-04-02T12:42:50.042000 - Bots - INFO - Connected to destination queues.
2025-04-02T12:42:50.042000 - Bots - INFO - Downloading report from 'https://lists.blocklist.de/lists/bots.txt'.
warnings.warn(

2025-04-02T12:42:50.200000 - Bots - INFO - Report downloaded (10183B).
2025-04-02T12:42:50.214000 - Bots - INFO - Idling for 3600.0s (1h) now.

root@70949970d694:/opt# intelmqctl log Abusech-Feodo-Tracker
2025-04-02T12:42:49.609000 - Abusech-Feodo-Tracker - INFO - Connected to destination queues.
2025-04-02T12:42:49.609000 - Abusech-Feodo-Tracker - INFO - PGP signature verification is active.
2025-04-02T12:42:49.609000 - Abusech-Feodo-Tracker - INFO - Bot initialization completed.
2025-04-02T12:42:49.610000 - Abusech-Feodo-Tracker - INFO - Loading source pipeline and queue 'Abusech-Feodo-Tracker-queue'.
2025-04-02T12:42:49.636000 - Abusech-Feodo-Tracker - INFO - Connected to source queue.
2025-04-02T12:42:49.636000 - Abusech-Feodo-Tracker - INFO - Loading destination pipeline and queues {'_default': ['MISPAPI-Output-queue']}.
2025-04-02T12:42:49.652000 - Abusech-Feodo-Tracker - INFO - Connected to destination queues.
2025-04-02T12:42:49.652000 - Abusech-Feodo-Tracker - INFO - Downloading report from 'https://feodotracker.abuse.ch/downloads/ipblocklist.json'.
warnings.warn(

2025-04-02T12:42:49.975000 - Abusech-Feodo-Tracker - INFO - Report downloaded (2161B).
2025-04-02T12:42:50.027000 - Abusech-Feodo-Tracker - INFO - Idling for 3600.0s (1h) now.

[kamil-certat]

@sebix looking at #2589 (comment) the non-verify variant should work.

Huh, if it works in a fresh installation, then it looks like your installation had a broken certifi package 🤔