ssl.get_server_certificate failure with IOS XE/NX-OS

[remingtonc]

Following documentation at https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/1610/b_1610_programmability_cg/gnmi_protocol.html#id_67108

Using the generated certificates yields:

python test_xe.py
Traceback (most recent call last):
  File "test_xe.py", line 14, in <module>
    print(client.capabilities())
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/client.py", line 93, in capabilities
    response = self.service.Capabilities(message, metadata=self._gen_metadata())
  File "/home/remcampb/.local/share/virtualenvs/cisco-gnmi-python-z1MRTrKn/local/lib/python2.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/remcampb/.local/share/virtualenvs/cisco-gnmi-python-z1MRTrKn/local/lib/python2.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1569363736.047551305","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3818,"referenced_errors":[{"created":"@1569363736.047542320","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":395,"grpc_status":14}]}"
>

Attempting to download and use certificate from device:

python test_xe.py
Traceback (most recent call last):
  File "test_xe.py", line 16, in <module>
    client = XEClient('...', attempt_implicit_secure=True).with_authentication('...', '...')
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/client.py", line 78, in __init__
    self.as_secure(root_from_target=True, target_name_from_root=True)
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/base.py", line 113, in as_secure
    root_certificates = get_cert_from_target(self.target_netloc)
  File "/home/remcampb/development/cisco-gnmi-python/src/cisco_gnmi/base.py", line 80, in get_cert_from_target
    (self.target_netloc.hostname, self.target_netloc.port)
  File "/usr/lib/python2.7/ssl.py", line 1007, in get_server_certificate
    with closing(context.wrap_socket(sock)) as sslsock:
  File "/usr/lib/python2.7/ssl.py", line 353, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 830, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)

OpenSSL too has issues.

openssl s_client -state -connect $IP:$PORT
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:unknown state
depth=1 CN = rootCA
verify error:num=19:self signed certificate in certificate chain
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:unknown state
SSL_connect:failed in unknown state
139800832919192:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:

Might be related to SNI?

[remingtonc]

Seeing the same issue on NX-OS.

[remingtonc]

Updating to OpenSSL 1.1.1d changes output to:

  File "test_nx.py", line 4, in <module>
    client = ClientBuilder('...').set_os('NX-OS').set_secure_from_target().set_ssl_target_override().set_call_authentication('...', '...').construct()
  File "/data/src/cisco_gnmi/builder.py", line 191, in set_secure_from_target
    root_certificates = get_cert_from_target(self.__target_netloc)
  File "/data/src/cisco_gnmi/util.py", line 85, in get_cert_from_target
    (target_netloc.hostname, target_netloc.port)
  File "/usr/local/lib/python3.7/ssl.py", line 1314, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/usr/local/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/local/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

"failed in unknown state" is no longer present from openssl client.

(data) /data/src # openssl s_client -state -connect $IP:$PORT
CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
Can't use SSL_get_servername
SSL_connect:SSLv3/TLS read server hello
depth=0 CN = ems.cisco.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = ems.cisco.com
verify error:num=10:certificate has expired
notAfter=Sep 26 22:24:44 2019 GMT
verify return:1
depth=0 CN = ems.cisco.com
notAfter=Sep 26 22:24:44 2019 GMT
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:error in SSLv3/TLS write finished
write:errno=0
---
Certificate chain
 0 s:CN = ems.cisco.com
   i:CN = ems.cisco.com
---
Server certificate
-----BEGIN CERTIFICATE-----
x
-----END CERTIFICATE-----
subject=CN = ems.cisco.com

issuer=CN = ems.cisco.com

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1205 bytes and written 419 bytes
Verification error: certificate has expired
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: x
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1570047758
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: yes
---

[remingtonc]

Does write:errno=0 mean SSL_do_handshake() returned 0? https://github.com/python/cpython/blob/master/Modules/_ssl.c#L1067
https://www.openssl.org/docs/man1.1.1/man3/SSL_do_handshake.html

[remingtonc]

IOS XR

(data) /data/src # no_proxy=$IP openssl s_client -state -bugs -connect $IP:$PORT
CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
Can't use SSL_get_servername
SSL_connect:SSLv3/TLS read server hello
depth=0 C = US, ST = CA, L = San Jose, street = 3700 Cisco Way, postalCode = 95134, O = "Cisco Systems, Inc.", OU = CSG, CN = ems.cisco.com, serialNumber = x
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = CA, L = San Jose, street = 3700 Cisco Way, postalCode = 95134, O = "Cisco Systems, Inc.", OU = CSG, CN = ems.cisco.com, serialNumber = x
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS read server session ticket
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished

Must Crtl+C out of openssl client, might be odd.

IOS XE

(data) /data/src # no_proxy=10.251.98.69 openssl s_client -bugs -connect $IP:$PORT
CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
Can't use SSL_get_servername
SSL_connect:SSLv3/TLS read server hello
depth=1 CN = rootCA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = rootCA
verify return:1
depth=0 CN = x
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:error in SSLv3/TLS write finished
write:errno=0

Erroring as self signed cert, does that effect handshake? Same is present for IOS XR but num=18 vs num=19.

NX-OS

(data) /data/src # no_proxy=$IP openssl s_client -bugs -connect $IP:$PORT
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
Can't use SSL_get_servername
SSL_connect:SSLv3/TLS read server hello
depth=0 CN = ems.cisco.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = ems.cisco.com
verify error:num=10:certificate has expired
notAfter=Sep 26 22:24:44 2019 GMT
verify return:1
depth=0 CN = ems.cisco.com
notAfter=Sep 26 22:24:44 2019 GMT
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:error in SSLv3/TLS write finished
write:errno=0

Certificate is expired, does that effect the handshake?

[remingtonc]

Thank you, SO strangers. https://superuser.com/a/224263

remcampb@brhim-dev:~/development/sandbox/mmkay$ no_proxy=$IP ./test_ciphers.sh $IP:$PORT | grep YES | tee xe.log
Testing ECDHE-RSA-AES256-GCM-SHA384...YES
Testing ECDHE-RSA-AES128-GCM-SHA256...YES
Testing ECDHE-RSA-AES256-SHA384...YES
Testing ECDHE-RSA-AES128-SHA256...YES
remcampb@brhim-dev:~/development/sandbox/mmkay$ no_proxy=$IP ./test_ciphers.sh $IP:$PORT | grep YES | tee xr.log
Testing ECDHE-RSA-AES256-GCM-SHA384...YES
Testing ECDHE-RSA-CHACHA20-POLY1305...YES
Testing ECDHE-RSA-AES128-GCM-SHA256...YES
Testing ECDHE-RSA-AES256-SHA...YES
Testing ECDHE-RSA-AES128-SHA...YES
Testing AES256-GCM-SHA384...YES
Testing AES128-GCM-SHA256...YES
Testing AES256-SHA...YES
Testing AES128-SHA...YES
remcampb@brhim-dev:~/development/sandbox/mmkay$ no_proxy=$IP ./test_ciphers.sh $IP:$PORT | grep YES | tee nx.log
Testing ECDHE-RSA-AES256-GCM-SHA384...YES
Testing ECDHE-RSA-AES128-GCM-SHA256...YES
Testing ECDHE-RSA-AES256-SHA384...YES
Testing ECDHE-RSA-AES128-SHA256...YES

[remingtonc]

Potentially related to OpenSSL vs CiscoSSL?

# XR
[host:~]$ openssl version
OpenSSL 1.0.1j 15 Oct 2014
# NX
bash-4.3$ openssl version
CiscoSSL 1.0.2o.6.2.238

Can't tell with IOS XE, but given support for same cipher suites it is likely XE is also CiscoSSL-based.

[remingtonc]

Only affects ClientBuilder.set_secure_from_target(...).