With SSL renegotiation, openssl CLI fails with "called a function you should not call"

[TobiasEgg]

When using libressl with the command openssl on the command line with -starttls and during the session, a renegotiation occurs, the openssl command line tool aborts with the following error message:

40619255824:error:1404C042:SSL routines:ST_OK:called a function you should not call:/usr/src/lib/libssl/ssl_lib.c:2862:

This should not happen.

Issue reproduced on OpenBSD 7.7 with LibreSSL 4.1.0 (default install) on both ARM64 and x86_64
Issue reproduced on OpenBSD 7.6 with LibreSSL 4.0.0 (default install) on both ARM64 and x86_64
Issue reproduced on OpenBSD 7.6 with LibreSSL 4.0.0 (default install) on x86_64

There are two ways to reproduce.

Variant 1:

$ openssl s_client -starttls ftp -crlf  -connect test.rebex.net:21
CONNECTED(00000003)
(... some stuff removed ...)
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1746077388
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
    See https://test.rebex.net/ for more information and terms of use.
USER ftp
331 Anonymous login OK, send your complete email address as your password.
PASS user@example.com
230 User 'ftp' logged in.
REIN
RENEGOTIATING
17370740397880:error:1404C042:SSL routines:ST_OK:called a function you should not call:/usr/src/lib/libssl/ssl_lib.c:2862:

Variant 2:

Configure SMTP-Server, e.g. OpenSMTPd or any other (tested against several systems), with either self-signed or CA-signed key/cert (irrelevant to trigger the issue), e.g.,

pki openbsd77 key "/etc/ssl/private/testmailkey.pem"
pki openbsd77 cert "/etc/ssl/testmailcert.pem"
listen on lo0 inet4 port 25 hostname openbsd77 tls pki openbsd77

and do:

$ openssl s_client -starttls smtp -crlf -connect 127.0.0.1:25
CONNECTED(00000003)
(... some stuff removed ...)
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1746076742
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
250 HELP
EHLO Test
250-openbsd77 Hello Test [127.0.0.1], pleased to meet you
250-8BITMIME
250-ENHANCEDSTATUSCODES
250-SIZE 36700160
250-DSN
250 HELP
MAIL FROM: <test@example.com>
250 2.0.0 Ok
RCPT TO: <openbsd77@localhost>
RENEGOTIATING
95429802000:error:1404C042:SSL routines:ST_OK:called a function you should not call:/usr/src/lib/libssl/ssl_lib.c:2862:

[botovq]

Thanks. I think this is mostly expected since the generic TLS methods set .ssl_renegotiate to ssl_undefined_function(). If you pass -tls1_2 to force a method supporting renegotiation this works.

We could intercept attempts to renegotiate with TLSv1.3 (which doesn't support renegotiation) either in SSL_renegotiate*() in ssl_lib.c before this handler is called or in the openssl command line tool, or both, to throw a friendlier error like OpenSSL 3 does.

[TobiasEgg]

A friendlier error (ideally pointing to -tls1_2) would be a good idea, however, this does not solve the issue:

$ openssl s_client -starttls smtp -crlf -connect 127.0.0.1:25 -legacy_renegotiation -tls1_2
CONNECTED(00000003)
(... some stuff removed ...)
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 92C9C1584E718B53338BF2DB3AF19F6A5ADA89929F225E47A1F12314C22ED236C476CC905F64A66B539DA3A4D89B57B5
    Start Time: 1746097053
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
250 HELP
EHLO test
250-openbsd77 Hello test [127.0.0.1], pleased to meet you
250-8BITMIME
250-ENHANCEDSTATUSCODES
250-SIZE 36700160
250-DSN
250 HELP
MAIL FROM: <user@example.com>
250 2.0.0 Ok
RCPT TO: <user@localhost>
RENEGOTIATING
147267745808:error:1400444C:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert no renegotiation:/usr/src/lib/libssl/ssl_pkt.c:753:SSL alert number 100
147267745808:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/usr/src/lib/libssl/ssl_pkt.c:495:

The error occurs with and without -legacy_renegotiation

It is a bit confusing, since OpenSMTPd runs on the same machine and therefore uses the same LibreSSL. So why would it at one end of the connection request a renegotiation, if it does not support it on the other end?

$ ldd /usr/sbin/smtpd 
/usr/sbin/smtpd:
	Start            End              Type  Open Ref GrpRef Name
	0000001cffdd0000 0000001cffe85000 exe   1    0   0      /usr/sbin/smtpd
	000000217df0c000 000000217df4a000 rlib  0    1   0      /usr/lib/libevent.so.4.1
	000000216b312000 000000216b358000 rlib  0    1   0      /usr/lib/libutil.so.20.0
	0000002102c3d000 0000002102c81000 rlib  0    1   0      /usr/lib/libtls.so.32.1
	0000002121620000 00000021216b9000 rlib  0    2   0      /usr/lib/libssl.so.59.1
	00000021858a9000 0000002185b00000 rlib  0    3   0      /usr/lib/libcrypto.so.56.0
	00000021b7739000 00000021b7785000 rlib  0    1   0      /usr/lib/libz.so.7.1
	0000002103839000 0000002103964000 rlib  0    1   0      /usr/lib/libc.so.100.3
	00000021c23b0000 00000021c23b0000 ld.so 0    1   0      /usr/libexec/ld.so

# ldd `whereis openssl`
/usr/bin/openssl:
	Start            End              Type  Open Ref GrpRef Name
	0000000230ca0000 0000000230d60000 exe   1    0   0      /usr/bin/openssl
	00000006d79df000 00000006d7a78000 rlib  0    1   0      /usr/lib/libssl.so.59.1
	0000000674dbd000 0000000675014000 rlib  0    2   0      /usr/lib/libcrypto.so.56.0
	000000072c708000 000000072c833000 rlib  0    1   0      /usr/lib/libc.so.100.3
	00000006f3f10000 00000006f3f10000 ld.so 0    1   0      /usr/libexec/ld.so

[botovq]

It's the lovely s_client behavior that hurts you now:

When used interactively (which means neither -quiet nor -ign_eof have been given), the session will be renegotiated if the line begins with an R; if the line begins with a Q or if end of file is reached, the connection will be closed down.

So your line RCPT TO: ... eats the first R and requests a renegotiation instead of sending 'RCPT TO' to the server. The server denies that request with an no_renegotiation(100) alert since OpenSMTPd uses libtls, which sets SSL_OP_NO_CLIENT_RENEGOTIATION on servers because that's a classic attack vector (look up 'renegotiation attack').

You're probably better off experimenting with a nc(1)-like tool (although inserting the CRLF line endings can be slightly annoying with that).

[TobiasEgg]

Well, should s_client really listen to the content? Since any "R" as first letter of a line would then initiate a renegotiation. Which it doesn't on openssl.

[botovq]

Well, it's the historical behavior and OpenSSL changed that only rather recently as far as I can tell (PR 20566 from two years ago, unless I'm mistaken).

Whether it's a feature worth spending time on, I don't know. I would not be against landing a patch that matches this behavior, but I doubt we will be doing this ourselves.