SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)'))

[litebito]

Hi,

I'm evaluating Netbox on an internal standard (CentOS Stream 9) server. Netbox setup using git, directly on the server (no docker).
So far, everything seem to work fine, except for one detail.
In the dashboard, the "News" widget, I see the following error:

There was a problem fetching the RSS feed: HTTPSConnectionPool(host='netbox.dev', port=443): Max retries exceeded with url: /rss/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

Now I know how to skip/disable SSL/certification validation when using curl or wget for example. (although I prefer not to)
How can I resolve the issue above?

Thanks

[candlerb]

Have you checked your system clock? (Type date at the shell prompt)

netbox.dev has a valid certificate as far as I can see - it's signed by Letsencrypt and presents a valid chain.

It was created on Feb 9 and expires May 10 20:27:11 2023 GMT - that's only 21 hours away, so it looks like the certificate renewal process is broken - but it is valid for now, just. However, if your clock is a day in the future, it would be broken. But I don't think that can be the problem, as it wouldn't be seen as a "self-signed" certificate.

If that's not it, then it maybe your server doesn't have the relevant root certificate (ISRG Root X1) installed. But that would be weird.

The Netbox requirements include certifi. What does the following show?

cd /opt/netbox
source venv/bin/activate
pip freeze | grep certifi

I get:

certifi==2022.12.7

and I see that /opt/netbox/venv/lib/python3.8/site-packages/certifi/cacert.pem contains

# Issuer: CN=ISRG Root X1 O=Internet Security Research Group
# Subject: CN=ISRG Root X1 O=Internet Security Research Group
# Label: "ISRG Root X1"
# Serial: 172886928669790476064670243504169061120

However, I don't know for sure whether the widget's HTTPS client will use certifi's certificates or your system's root certificate store.

Otherwise: is there a dubious firewall between you and the Internet which does a man-in-the-middle attack on all outgoing HTTPS connections to intercept them?

[candlerb]

Oh, there's one other test you can do:

openssl s_client -connect netbox.dev:443 -showcerts

This will give you several screenfuls of data. If you copy-paste the whole results here - with three backticks on a line of their own (```) before and after the pasted block, to stop them being mangled as Markdown - then we can see what actual certificates your system is being presented with.

[candlerb]

Finally, use "dig" to confirm that netbox.dev resolves to the correct IPs:

$ dig +short netbox.dev a
68.183.150.179
$ dig +short netbox.dev aaaa
2604:a880:800:10::d6c:f001

(where $ represents your shell prompt)

[Alef-Burzmali]

Hello,
As indicated in @candlerb 's answer, the certificate for netbox.dev is now expired, and an error is displayed in the RSS widget.
There was a problem fetching the RSS feed: HTTPSConnectionPool(host='netbox.dev', port=443): Max retries exceeded with url: /rss/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:992)')))

Not sure where the best place to report this is, as it does not fit any of the bug templates.

[candlerb]

Ping @jeremystretch

$ openssl s_client -connect netbox.dev:443
...
depth=0 CN = netbox.dev
verify error:num=10:certificate has expired
notAfter=May 10 20:27:11 2023 GMT
verify return:1

[litebito]

Here is the output of the various checks:

$ pip freeze | grep certifi
certifi==2023.5.7

$ cat /opt/netbox/venv/lib/python3.9/site-packages/certifi/cacert.pem | grep ISRG
# Issuer: CN=ISRG Root X1 O=Internet Security Research Group
# Subject: CN=ISRG Root X1 O=Internet Security Research Group
# Label: "ISRG Root X1"
# Issuer: CN=ISRG Root X2 O=Internet Security Research Group
# Subject: CN=ISRG Root X2 O=Internet Security Research Group
# Label: "ISRG Root X2"

You are right about the MIM, as our firewall is intercepting and inspecting all https traffic:

$ openssl s_client -connect netbox.dev:443 -showcerts
CONNECTED(00000003)
depth=3 C = EU, O = mycorp, OU = IT, CN = mycorpfw-rootca.mycorp.net
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=3 C = EU, O = mycorp, OU = IT, CN = mycorpfw-rootca.mycorp.net
verify return:1
depth=2 C = EU, O = mycorp, OU = IT, CN = mycorpfw-subca.mycorp.net
verify return:1
depth=1 C = EU, O = mycorp, OU = coolio, CN = coolio-decrypt.mycorp.net
verify return:1
depth=0 CN = netbox.dev
verify return:1
---
Certificate chain
 0 s:CN = netbox.dev
   i:C = EU, O = mycorp, OU = coolio, CN = coolio-decrypt.mycorp.net
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: May 10 20:37:35 2023 GMT; NotAfter: Aug  8 20:37:34 2023 GMT
-----BEGIN CERTIFICATE-----
XXX
-----END CERTIFICATE-----
 1 s:C = EU, O = mycorp, OU = coolio, CN = coolio-decrypt.mycorp.net
   i:C = EU, O = mycorp, OU = IT, CN = mycorpfw-subca.mycorp.net
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Nov 28 17:55:47 2021 GMT; NotAfter: Nov 28 17:55:47 2026 GMT
-----BEGIN CERTIFICATE-----
XXX
-----END CERTIFICATE-----
 2 s:C = EU, O = mycorp, OU = IT, CN = mycorpfw-subca.mycorp.net
   i:C = EU, O = mycorp, OU = IT, CN = mycorpfw-rootca.mycorp.net
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Nov 28 17:12:33 2021 GMT; NotAfter: Nov 28 17:12:33 2026 GMT
-----BEGIN CERTIFICATE-----
XXX
-----END CERTIFICATE-----
 3 s:C = EU, O = mycorp, OU = IT, CN = mycorpfw-rootca.mycorp.net
   i:C = EU, O = mycorp, OU = IT, CN = mycorpfw-rootca.mycorp.net
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Nov 28 17:12:00 2021 GMT; NotAfter: Nov 28 17:12:00 2026 GMT
-----BEGIN CERTIFICATE-----
XXX
-----END CERTIFICATE-----
---
Server certificate
subject=CN = netbox.dev
issuer=C = EU, O = mycorp, OU = coolio, CN = coolio-decrypt.mycorp.net
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2634 bytes and written 394 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: XXX
    Session-ID-ctx:
    Resumption PSK: XXX
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - fe aa 33 71 21 55 f3 46-2a fe 3a 27 7d 3d df c8   ................
    0010 - e5 99 ce 30 11 ec 6c 49-be 27 86 17 b0 11 1b 4d   ................

    Start Time: 1683786332
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

$  dig +short netbox.dev aaaa
2604:a880:800:10::d6c:f001
$  dig +short netbox.dev a
68.183.150.179

[candlerb]

You are right about the MIM, as our firewall is intercepting and inspecting all https traffic:

OK, that's why there's the error - your HTTPS connection is being actively attacked.

The easiest solution will be to disable the widget. Otherwise, you'll have to work out how to add your firewall's fake root certificate into the python and/or system trusted root cert stores.

[litebito]

As I don't really need the news widget, I'm ignoring it for now