Enter PEM pass phrase does not work correctly

[ioquatix]

> ruby -ropenssl -e 'OpenSSL::PKey.read(File.read(File.expand_path("~/.gem/release.pem")), nil)'
Enter PEM pass phrase: (incorrect password)
Enter PEM pass phrase: (correct password)
Enter PEM pass phrase: (correct password)
-e:1:in 'OpenSSL::PKey.read': Could not parse PKey (OpenSSL::PKey::PKeyError)
	from -e:1:in '<main>'

I'm not sure why yet.

Related to rubygems/rubygems#8896.

[ioquatix]

Here is a minimal reproduction:

require "openssl"
require "tempfile"

passphrase = "secret123"

# 1. Generate RSA key with passphrase
key = OpenSSL::PKey::RSA.new(2048)
encrypted_pem = key.export(OpenSSL::Cipher.new("AES-256-CBC"), passphrase)

# Save to a temporary file
key_file = Tempfile.new("private.pem")
key_file.write(encrypted_pem)
key_file.close

puts "Private key saved to: #{key_file.path}"
puts "Passphrase: #{passphrase}"
puts
puts "Now trying to read with nil passphrase (OpenSSL will prompt):"

# 2. Try reading the key (will trigger 3 OpenSSL prompts)
OpenSSL::PKey.read(File.read(key_file.path), nil)

[rhenium]

I reproduced this with OpenSSL 3.5.1. This is probably an issue with OpenSSL 3.0 or later.

It should ask for the passphrase only once, and if the passphrase turns out to be incorrect, it should fail immediately, similar to how it used to work with OpenSSL 1.1.1 or earlier.

It currently asks for a it 3 times because of this:

openssl/ext/openssl/ossl_pkey.c

Lines 163 to 178 in dfbbac6

	int selections[] = {
	EVP_PKEY_KEYPAIR,
	EVP_PKEY_KEY_PARAMETERS,
	EVP_PKEY_PUBLIC_KEY
	};
	int selection_num = (int)(sizeof(selections) / sizeof(int));
	int i, j;
	for (i = 0; i < input_type_num; i++) {
	for (j = 0; j < selection_num; j++) {
	pkey = ossl_pkey_read(bio, input_types[i], selections[j], pass);
	if (pkey) {
	goto out;
	}
	}
	}

The second and third prompt are shown when it's retrying to decode it as a public key or parameters, after it was not successfully decoded as a private key. I'm not sure why OpenSSL is attempting to decrypt the PEM block in those retries, only to discard the result, but I overlooked this behavior.

[rhenium]

There is another issue. The passphrase is not cached in OSSL_DECODER_CTX, and if the input contains two encrypted PEM blocks, it will prompt for the passphrase for every encountered PEM block:

$ ruby -Ilib -ropenssl -e'pk=OpenSSL::PKey.generate_key("ed25519"); pem=pk.private_to_pem("aes-256-cbc", "secret123"); OpenSSL::PKey.read(pem+pem) {p :asked;"bad"}'

This calls the block 4 times.

[ioquatix]

Should we file upstream bug report to OpenSSL?