openssl-master: rake test_fips failures about keys

[junaruga]

I executed the current latest master branch dfbbac6 with the latest OpenSSL master branch openssl/openssl@b20da23 in my forked repository today. I am seeing the failures in the FIPS tests below.

https://github.com/junaruga/ruby-openssl/actions/runs/16680199093/job/47216748820

$ sed -e "s|OPENSSL_DIR|$HOME/openssl|" tool/openssl_fips.cnf.tmpl > tmp/openssl_fips.cnf
$ export OPENSSL_CONF=$(pwd)/tmp/openssl_fips.cnf
$ bundle exec rake debug
$ bundle exec rake test_fips TESTOPTS="-v --no-show-detail-immediately" OSSL_TEST_ALL=1
...
  1) Error: test_check_key(OpenSSL::TestEC): OpenSSL::PKey::PKeyError: Could not parse PKey
/home/runner/work/openssl/openssl/test/openssl/test_pkey_ec.rb:102:in `read'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_ec.rb:102:in `test_check_key'
      99:     if aws_lc? # AWS-LC automatically does key checks on the parsed key.
     100:       assert_raise(OpenSSL::PKey::PKeyError) { OpenSSL::PKey.read(ec_key_data) }
     101:     else
  => 102:       key4 = OpenSSL::PKey.read(ec_key_data)
     103:       assert_raise(OpenSSL::PKey::ECError) { key4.check_key }
     104:     end
     105: 
Error: OpenSSL::PKey::PKeyError: Could not parse PKey
/home/runner/work/openssl/openssl/test/openssl/test_pkey_ec.rb:102:in `read'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_ec.rb:102:in `test_check_key'

  2) Error: test_dh_compute_key(OpenSSL::TestEC): OpenSSL::PKey::ECError: EC_KEY_generate_key: unable to create drbg
/home/runner/work/openssl/openssl/test/openssl/test_pkey_ec.rb:180:in `generate'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_ec.rb:180:in `test_dh_compute_key'
     177:   end
     178: 
     179:   def test_dh_compute_key
  => 180:     key_a = OpenSSL::PKey::EC.generate("prime256v1")
     181:     key_b = OpenSSL::PKey::EC.generate(key_a.group)
     182: 
     183:     pub_a = key_a.public_key
Error: OpenSSL::PKey::ECError: EC_KEY_generate_key: unable to create drbg
...
278 tests, 686 assertions, 0 failures, 170 errors, 0 pendings, 24 omissions, 0 notifications
33.0709% passed

[junaruga]

The following CI passed with the openssl/openssl@b8c46cb.
https://github.com/ruby/openssl/actions/runs/16656447633/job/47142524897

So, maybe a commit on OpenSSL master branch between the b8c46cba5fa0894fc026f1d965beec8089b79dbf (passed) and b20da2328018107414fe896e59e7d4d6c8af8174 (failed) caused the Ruby OpenSSL's test failures.

[rhenium]

I can't reproduce the errors on my box with OpenSSL b20da2328018107414fe896e59e7d4d6c8af8174 compiled with the fips provider, and all tests passed as expected.

The error is also weird. require "openssl" succeeds, and during it, a DH PEM is decoded.

openssl/lib/openssl/ssl.rb

Lines 36 to 45 in dfbbac6

	DH_ffdhe2048 = OpenSSL::PKey::DH.new <<-_end_of_pem_
	-----BEGIN DH PARAMETERS-----
	MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
	+8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
	87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
	YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
	7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
	ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
	-----END DH PARAMETERS-----
	_end_of_pem_

However, loading the exact same PEM fails later (dh2048_ffdhe2048.pem has the identical content as the heredoc in lib/openssl/ssl.rb).

openssl/test/openssl/test_pkey_dh.rb

Lines 140 to 157 in dfbbac6

	def test_params
	dh = Fixtures.pkey("dh2048_ffdhe2048")
	assert_kind_of(OpenSSL::BN, dh.p)
	assert_equal(dh.p, dh.params["p"])
	assert_kind_of(OpenSSL::BN, dh.g)
	assert_equal(dh.g, dh.params["g"])
	assert_nil(dh.pub_key)
	assert_nil(dh.params["pub_key"])
	assert_nil(dh.priv_key)
	assert_nil(dh.params["priv_key"])
	dhkey = OpenSSL::PKey.generate_key(dh)
	assert_equal(dh.params["p"], dhkey.params["p"])
	assert_kind_of(OpenSSL::BN, dhkey.pub_key)
	assert_equal(dhkey.pub_key, dhkey.params["pub_key"])
	assert_kind_of(OpenSSL::BN, dhkey.priv_key)
	assert_equal(dhkey.priv_key, dhkey.params["priv_key"])
	end

https://github.com/junaruga/ruby-openssl/actions/runs/16680199093/job/47216748820#step:11:683

[junaruga]

I was able to reproduce this issue with openssl/openssl@b20da23 on my local.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.6.0-dev-fips-debug-b20da23280/ssl/openssl_fips.cnf \
  /home/jaruga/.local/openssl-3.6.0-dev-fips-debug-b20da23280/bin/openssl list -providers
Providers:
  base
    name: OpenSSL Base Provider
    version: 3.6.0
    status: active
  fips
    name: OpenSSL FIPS Provider
    version: 3.6.0
    status: active

$ which ruby
~/.local/ruby-3.5.0dev-debug-8d14d6ea2d/bin/ruby

$ bundle list
Gems included by the bundle:
  * date (3.4.1)
  * erb (5.0.2)
  * forwardable (1.3.3)
  * power_assert (2.0.5)
  * prime (0.1.4)
  * psych (5.2.6)
  * rake (13.3.0)
  * rake-compiler (1.3.0)
  * rdoc (6.14.2)
  * singleton (0.3.0)
  * stringio (3.1.7)
  * test-unit (3.7.0)
  * test-unit-ruby-core (1.0.7)
Use `bundle info` to print more detailed information about a gem

$ OPENSSL_CONF=$HOME/.local/openssl-3.6.0-dev-fips-debug-b20da23280/ssl/openssl_fips.cnf \
  bundle exec rake test_fips
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/lib/ruby/3.5.0+2/openssl/ssl.rb:24: warning: already initialized constant OpenSSL::SSL::SSLContext::DEFAULT_PARAMS
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/lib/ruby/3.5.0+2/openssl/ssl.rb:24: warning: previous definition of DEFAULT_PARAMS was here
...
E
===============================================================================
Error: test_check_key(OpenSSL::TestEC): OpenSSL::PKey::PKeyError: Could not parse PKey
/home/jaruga/var/git/ruby/openssl/test/openssl/test_pkey_ec.rb:102:in 'OpenSSL::PKey.read'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_pkey_ec.rb:102:in 'OpenSSL::TestEC#test_check_key'
      99:     if aws_lc? # AWS-LC automatically does key checks on the parsed key.
     100:       assert_raise(OpenSSL::PKey::PKeyError) { OpenSSL::PKey.read(ec_key_data) }
     101:     else
  => 102:       key4 = OpenSSL::PKey.read(ec_key_data)
     103:       assert_raise(OpenSSL::PKey::ECError) { key4.check_key }
     104:     end
     105:-
===============================================================================
E
===============================================================================
Error: test_dh_compute_key(OpenSSL::TestEC): OpenSSL::PKey::ECError: EC_KEY_generate_key: unable to create drbg
/home/jaruga/var/git/ruby/openssl/test/openssl/test_pkey_ec.rb:180:in 'OpenSSL::PKey::EC.generate'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_pkey_ec.rb:180:in 'OpenSSL::TestEC#test_dh_compute_key'
     177:   end
     178:-
     179:   def test_dh_compute_key
  => 180:     key_a = OpenSSL::PKey::EC.generate("prime256v1")
     181:     key_b = OpenSSL::PKey::EC.generate(key_a.group)
     182:-
     183:     pub_a = key_a.public_key
===============================================================================
...
276 tests, 682 assertions, 0 failures, 169 errors, 0 pendings, 24 omissions, 0 notifications
32.9365% passed
-------------------------------------------------------------------------------
34.49 tests/s, 85.22 assertions/s
rake aborted!
Command failed with status (1)
/home/jaruga/var/git/ruby/openssl/Rakefile:22:in 'block in <top (required)>'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/rake-13.3.0/exe/rake:27:in '<top (required)>'
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/bin/bundle:25:in 'Kernel#load'
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/bin/bundle:25:in '<main>'
Tasks: TOP => test_fips_internal
(See full trace by running task with --trace)

[junaruga]

The following test always fails with the openssl/openssl@b20da23.

$ OPENSSL_CONF=$HOME/.local/openssl-3.6.0-dev-fips-debug-b20da23280/ssl/openssl_fips.cnf \
  bundle exec rake test_fips TEST=test/openssl/test_pkey_ec.rb TESTOPTS="--name=test_check_key -v"
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/lib/ruby/3.5.0+2/openssl/ssl.rb:24: warning: already initialized constant OpenSSL::SSL::SSLContext::DEFAULT_PARAMS
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/lib/ruby/3.5.0+2/openssl/ssl.rb:24: warning: previous definition of DEFAULT_PARAMS was here
/bin/gmake install sitearchdir=../../../../lib sitelibdir=../../../../lib target_prefix=
/bin/install -c -m 0755 openssl.so ../../../../lib
cp tmp/x86_64-linux/openssl/3.5.0/openssl.so tmp/x86_64-linux/stage/lib/openssl.so
Loaded suite /home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/rake-13.3.0/lib/rake/rake_test_loader
Started
OpenSSL::TestCase:
  OpenSSL::PKeyTestCase:
    OpenSSL::TestEC:
      test_check_key:																							E
===============================================================================================================================================================================================================================
Error: test_check_key(OpenSSL::TestEC): OpenSSL::PKey::PKeyError: Could not parse PKey
/home/jaruga/var/git/ruby/openssl/test/openssl/test_pkey_ec.rb:102:in 'OpenSSL::PKey.read'
/home/jaruga/var/git/ruby/openssl/test/openssl/test_pkey_ec.rb:102:in 'OpenSSL::TestEC#test_check_key'
      99:     if aws_lc? # AWS-LC automatically does key checks on the parsed key.
     100:       assert_raise(OpenSSL::PKey::PKeyError) { OpenSSL::PKey.read(ec_key_data) }
     101:     else
  => 102:       key4 = OpenSSL::PKey.read(ec_key_data)
     103:       assert_raise(OpenSSL::PKey::ECError) { key4.check_key }
     104:     end
     105:
===============================================================================================================================================================================================================================
: (0.013514)

Finished in 0.014728665 seconds.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 tests, 10 assertions, 0 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
0% passed
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
67.89 tests/s, 678.95 assertions/s
rake aborted!
Command failed with status (1)
/home/jaruga/var/git/ruby/openssl/Rakefile:22:in 'block in <top (required)>'
/home/jaruga/var/git/ruby/openssl/bundle/ruby/3.5.0+2/gems/rake-13.3.0/exe/rake:27:in '<top (required)>'
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/bin/bundle:25:in 'Kernel#load'
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/bin/bundle:25:in '<main>'
Tasks: TOP => test_fips_internal
(See full trace by running task with --trace)

[junaruga]

The following test always fails with the openssl/openssl@b20da23.

Sorry after compiling, the failure disappeared. Weird.

[junaruga]

I am working on the following branch.

https://github.com/junaruga/ruby-openssl/tree/wip/fips-failures-debug

I found the following test.rb fails with error on the openssl-master, but passes on the openssl-3.5.0.

https://github.com/junaruga/ruby-openssl/actions/runs/16729044811/job/47352316360

$ cat test.rb
require 'openssl'

key_data = <<~EOF
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIP+TT0V8Fndsnacji9tyf6hmhHywcOWTee9XkiBeJoVloAoGCCqGSM49
AwEHoUQDQgAEBkhhJIU/2/YdPSlY2I1k25xjK4trr5OXSgXvBC21PtY0HQ7lor7A
jzT0giJITqmcd81fwGw5+96zLcdxTF1hVQ==
-----END EC PRIVATE KEY-----
EOF
OpenSSL::PKey.read(key_data)

$ OPENSSL_CONF="$OPENSSL_DIR/ssl/openssl_fips.cnf" \
  bundle exec rake debug
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/lib/ruby/3.5.0+2/openssl/ssl.rb:24: warning: already initialized constant OpenSSL::SSL::SSLContext::DEFAULT_PARAMS
/home/jaruga/.local/ruby-3.5.0dev-debug-8d14d6ea2d/lib/ruby/3.5.0+2/openssl/ssl.rb:24: warning: previous definition of DEFAULT_PARAMS was here
OpenSSL::OPENSSL_VERSION: OpenSSL 3.6.0-dev
OpenSSL::OPENSSL_LIBRARY_VERSION: OpenSSL 3.6.0-dev
OpenSSL::OPENSSL_VERSION_NUMBER: 30600000
OpenSSL::LIBRESSL_VERSION_NUMBER: undefined
FIPS enabled: true
Providers: fips, base

$ OPENSSL_CONF="$OPENSSL_DIR/ssl/openssl_fips.cnf" \
  bundle exec ruby -I./lib test.rb
test.rb:10:in 'OpenSSL::PKey.read': Could not parse PKey (OpenSSL::PKey::PKeyError)
	from test.rb:10:in '<main>'

[junaruga]

In my case with the above test.rb, I confirmed the test.rb passed with the the commit openssl/openssl@dc5cd6f (dc5cd6f70a0eeb30e272fe885a64f3e3d76b5e42). CI log. So, I ran the git bisect for the b20da2328018107414fe896e59e7d4d6c8af8174 (failed) and dc5cd6f70a0eeb30e272fe885a64f3e3d76b5e42 (passed). The bisect.sh code is here.

$ pwd
/home/jaruga/git/openssl
$ git bisect start b20da2328018107414fe896e59e7d4d6c8af8174 dc5cd6f70a0eeb30e272fe885a64f3e3d76b5e42
$ git bisect run /path/to/bisect.sh

$ git bisect log
# bad: [b20da2328018107414fe896e59e7d4d6c8af8174] Revert "Pairwise check for DH keys import as part of FIPS"
# good: [dc5cd6f70a0eeb30e272fe885a64f3e3d76b5e42] rsa: expose pairwise consistency test API
git bisect start 'b20da2328018107414fe896e59e7d4d6c8af8174' 'dc5cd6f70a0eeb30e272fe885a64f3e3d76b5e42'
# bad: [32ff539daf83cccc15a159fe214cac66acc80fec] changes: add note about PCT on key import to the FIPS provider
git bisect bad 32ff539daf83cccc15a159fe214cac66acc80fec
# bad: [a177798e0b8f3c7ff66b2a609924aafeb66b8b12] ec (fips): add PCT for key import
git bisect bad a177798e0b8f3c7ff66b2a609924aafeb66b8b12
# good: [58ab3b0ffeea2b4c30f8794d281650ce7bfec6d2] ecx (fips): add PCT for key import
git bisect good 58ab3b0ffeea2b4c30f8794d281650ce7bfec6d2
# first bad commit: [a177798e0b8f3c7ff66b2a609924aafeb66b8b12] ec (fips): add PCT for key import

Then I found the openssl/openssl@a177798 was the first bad (failed) commit in my case with the test.rb reading EC PRIVATE KEY.

Seeing the commits of the openssl/openssl, there are similar commits for the other algorithms below. I assume that these commits caused the test failures for the other algorithms

57230da2bd0be6b219cec2995832034b1e09e0e9 rsa (fips): add PCT for key import
a177798e0b8f3c7ff66b2a609924aafeb66b8b12 ec (fips): add PCT for key import
58ab3b0ffeea2b4c30f8794d281650ce7bfec6d2 ecx (fips): add PCT for key import

[rhenium]

Thanks for looking into this.

Then I found the openssl/openssl@a177798 was the first bad (failed) commit in my case with the test.rb reading EC PRIVATE KEY.

This commit would explain the behavior. The PEM block from test_check_key is specially crafted so that it fails a pair-wise consistency test, and OpenSSL self-destructs itself when it encounters this which seems like a requirement by FIPS 140.

[junaruga]

This commit would explain the behavior. The PEM block from test_check_key is specially crafted so that it fails a pair-wise consistency test, and OpenSSL self-destructs itself when it encounters this which seems like a requirement by FIPS 140.

I found the issue ticket openssl/openssl#26572 managing the changes. I still don't understand how to fix our tests by the OpenSSL's changes.

For example, in the case of the test/openssl/test_pkey_ec.rb#test_check_key, how can we change the value of the ec_key_data?