voms-proxy-init fails to decrypt the private key of my EEC

[paulmillar]

Here is voms-proxy-init not working:

paul@sprocket:~$ voms-proxy-init
Enter GRID pass phrase for this identity:
Credentials couldn't be loaded [/home/paul/.globus/userkey.pem, /home/paul/.globus/usercert.pem]: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
No credentials found!
paul@sprocket:~$ 

This is actually voms-proxy-inti3:

paul@sprocket:~$ voms-proxy-init -version
voms-proxy-init v. 3.0.7 (voms-api-java/3.2.0 canl/2.4.1 bouncycastle/1.56.0 bcmail/1.56.0.0)
paul@sprocket:~$ 

Here are other tools successfully creating a proxy from the same EEC:

paul@sprocket:~$ voms-proxy-init2 
Enter GRID pass phrase:
Your identity: /C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar
Creating proxy ................... Done

Your proxy is valid until Tue Jul 17 00:46:22 2018
paul@sprocket:~$ 

paul@sprocket:~$ arcproxy 
Enter pass phrase for private key:
Your identity: /C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar
Proxy generation succeeded
Your proxy is valid until: 2018-07-17 00:46:44
paul@sprocket:~$ 

Here are some environment details:

paul@sprocket:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.5 (stretch)
Release:        9.5
Codename:       stretch
paul@sprocket:~$ 

paul@sprocket:~$ dpkg -l voms-clients-java libvoms-api-java-java libcanl-java libbcpkix-java libbcprov-java
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                        Version                    Architecture               Description
+++-===========================================-==========================-==========================-===========================================================================================
ii  libbcpkix-java                              1.56-1+deb9u2              all                        Bouncy Castle Java API for PKIX, CMS, EAC, TSP, PKCS, OCSP, CMP, and CRMF
ii  libbcprov-java                              1.56-1+deb9u2              all                        Bouncy Castle Java Cryptographic Service Provider
ii  libcanl-java                                2.4.1-1                    all                        EMI Common Authentication library - bindings for Java
ii  libvoms-api-java-java                       3.2.0-1                    all                        Virtual Organization Membership Service Java API
ii  voms-clients-java                           3.0.7-1                    all                        Virtual Organization Membership Service Java clients
paul@sprocket:~$ 

Since the error message doesn't really say what went wrong (perhaps another bug, there), I took the current versions of libraries (available through Debian stretch) and added some simple debugging:

paul@sprocket:~/git/voms-clients$ git diff
diff --git a/src/main/java/org/italiangrid/voms/clients/impl/ProxyInitListenerHelper.java b/src/main/java/org/italiangrid/voms/clients/impl/ProxyInitListenerHelper.java
index bd2bc65..331e27b 100644
--- a/src/main/java/org/italiangrid/voms/clients/impl/ProxyInitListenerHelper.java
+++ b/src/main/java/org/italiangrid/voms/clients/impl/ProxyInitListenerHelper.java
@@ -200,6 +200,7 @@ public class ProxyInitListenerHelper implements InitListenerAdapter {
 
     logger.formatMessage(level, "Credentials couldn't be loaded %s: %s\n",
       Arrays.toString(locations), error.getMessage());
+    error.printStackTrace();
 
   }
 
paul@sprocket:~/git/voms-clients$ 

paul@sprocket:~/git/canl-java$ git diff
diff --git a/src/main/java/eu/emi/security/authn/x509/impl/CertificateUtils.java b/src/main/java/eu/emi/security/authn/x509/impl/CertificateUtils.java
index d309b1c..8098a12 100644
--- a/src/main/java/eu/emi/security/authn/x509/impl/CertificateUtils.java
+++ b/src/main/java/eu/emi/security/authn/x509/impl/CertificateUtils.java
@@ -293,6 +293,7 @@ public class CertificateUtils
        private static PrivateKeyInfo resolvePK(String type, Object src, PasswordFinder pf) throws 
                IOException, OperatorCreationException, PKCSException
        {
+                System.err.println("resolvePK: type=" + type + ", src=" + src.getClass());
                if (src instanceof PrivateKeyInfo)
                        return (PrivateKeyInfo) src;
                
paul@sprocket:~/git/canl-java$ 

Here is the output of voms-proxy-init when run with these patched libraries:

resolvePK: type=PEM, src=class org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo
Enter GRID pass phrase for this identity:
Credentials couldn't be loaded [/home/paul/.globus/userkey.pem, /home/paul/.globus/usercert.pem]: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
        at eu.emi.security.authn.x509.impl.CertificateUtils.convertToPrivateKey(CertificateUtils.java:286)
        at eu.emi.security.authn.x509.impl.CertificateUtils.internalLoadPK(CertificateUtils.java:271)
        at eu.emi.security.authn.x509.impl.CertificateUtils.loadPEMPrivateKey(CertificateUtils.java:242)
        at eu.emi.security.authn.x509.impl.PEMCredential.init(PEMCredential.java:230)
        at eu.emi.security.authn.x509.impl.PEMCredential.<init>(PEMCredential.java:161)
        at org.italiangrid.voms.credential.impl.AbstractLoadCredentialsStrategy.loadPEMCredential(AbstractLoadCredentialsStrategy.java:102)
        at org.italiangrid.voms.credential.impl.DefaultLoadCredentialsStrategy.loadPEMCredentialsFromGlobusDir(DefaultLoadCredentialsStrategy.java:201)
        at org.italiangrid.voms.credential.impl.DefaultLoadCredentialsStrategy.loadCredentials(DefaultLoadCredentialsStrategy.java:132)
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.lookupCredential(DefaultVOMSProxyInitBehaviour.java:603)
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.initProxy(DefaultVOMSProxyInitBehaviour.java:187)
        at org.italiangrid.voms.clients.VomsProxyInit.execute(VomsProxyInit.java:307)
        at org.italiangrid.voms.clients.VomsProxyInit.<init>(VomsProxyInit.java:71)
        at org.italiangrid.voms.clients.VomsProxyInit.main(VomsProxyInit.java:55)
Caused by: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: javax.crypto.BadPaddingException: pad block corrupted
        at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.decryptPrivateKeyInfo(PKCS8EncryptedPrivateKeyInfo.java:72)
        at eu.emi.security.authn.x509.impl.CertificateUtils.resolvePK(CertificateUtils.java:307)
        at eu.emi.security.authn.x509.impl.CertificateUtils.convertToPrivateKey(CertificateUtils.java:280)
        ... 12 more
Caused by: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted
        at javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:121)
        at javax.crypto.CipherInputStream.read(CipherInputStream.java:239)
        at org.bouncycastle.util.io.Streams.pipeAll(Streams.java:114)
        at org.bouncycastle.util.io.Streams.readAll(Streams.java:41)
        at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.decryptPrivateKeyInfo(PKCS8EncryptedPrivateKeyInfo.java:68)
        ... 14 more
Caused by: javax.crypto.BadPaddingException: pad block corrupted
        at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(BaseBlockCipher.java:1215)
        at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(BaseBlockCipher.java:1050)
        at javax.crypto.Cipher.doFinal(Cipher.java:2047)
        at javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:118)
        ... 18 more
java.io.FileNotFoundException: /home/paul/.globus/usercred.p12 (cannot read file)
        at org.italiangrid.voms.credential.impl.AbstractLoadCredentialsStrategy.loadPKCS12Credential(AbstractLoadCredentialsStrategy.java:151)
        at org.italiangrid.voms.credential.impl.DefaultLoadCredentialsStrategy.loadPKCS12CredentialsFromGlobusDir(DefaultLoadCredentialsStrategy.java:192)
        at org.italiangrid.voms.credential.impl.DefaultLoadCredentialsStrategy.loadCredentials(DefaultLoadCredentialsStrategy.java:135)
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.lookupCredential(DefaultVOMSProxyInitBehaviour.java:603)
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.initProxy(DefaultVOMSProxyInitBehaviour.java:187)
        at org.italiangrid.voms.clients.VomsProxyInit.execute(VomsProxyInit.java:307)
        at org.italiangrid.voms.clients.VomsProxyInit.<init>(VomsProxyInit.java:71)
        at org.italiangrid.voms.clients.VomsProxyInit.main(VomsProxyInit.java:55)
No credentials found!

Here's a rough summary of the credentials:

paul@sprocket:~$ cat ~/.globus/userkey.pem
Bag Attributes
    friendlyName: Alexander Paul Millar^Ys GermanGrid ID
    localKeyID: 96 1A 91 EA 36 22 DD 26 BB F2 DF 2B 2A D3 04 21 AF 06 36 5B
Key Attributes: <No Attributes>
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFHDBOBgkqhkiG9w0BBQ0wQTApBgkqhkiG9w0BBQwwHAQInzxyNT6UxBMCAggA
[...]
GRLQOWr+BHG2Q0sugNnQeA==
-----END ENCRYPTED PRIVATE KEY-----
paul@sprocket:~$ 

paul@sprocket:~$ cat ~/.globus/usercert.pem
Bag Attributes
    friendlyName: Alexander Paul Millars GermanGrid ID
    localKeyID: 96 1A 91 EA 36 22 DD 26 BB F2 DF 2B 2A D3 04 21 AF 06 36 5B
subject=/C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar
issuer=/C=DE/O=GermanGrid/CN=GridKa-CA
-----BEGIN CERTIFICATE-----
MIIELTCCAxWgAwIBAgIDAIF1MA0GCSqGSIb3DQEBDQUAMDYxCzAJBgNVBAYTAkRF
[...]
HqQXxWaEGo26n8s+XDGCEQ4=
-----END CERTIFICATE-----
paul@sprocket:~$

I've configured my laptop to use voms-proxy-init2 as a work-around, so this isn't urgent

[andreaceccanti]

Hi @paulmillar , is this issue still bugging you?

[paulmillar]

Well, yes.

I'm currently using a work-around: voms-proxy-init2 (the C-client) works:

paul@celebrimbor:~$ voms-proxy-init2
Enter GRID pass phrase:
Your identity: /C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar
Creating proxy ...................... Done

Your proxy is valid until Sat Feb 23 01:47:40 2019
paul@celebrimbor:~$ 

However, the Java client continues to be completely useless for me:

paul@celebrimbor:~$ voms-proxy-init3
Enter GRID pass phrase for this identity:
Credentials couldn't be loaded [/home/paul/.globus/userkey.pem, /home/paul/.globus/usercert.pem]: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
No credentials found!
paul@celebrimbor:~$ 

For reference, these are the versions I have installed:

paul@celebrimbor:~$ dpkg-query -W|grep voms-client
voms-clients    2.1.0~rc0-2
voms-clients-java       3.0.7-1
paul@celebrimbor:~$ 

[andreaceccanti]

Hi Paul,

as you see from the stack trace

Credentials couldn't be loaded [/home/paul/.globus/userkey.pem, /home/paul/.globus/usercert.pem]: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
        at eu.emi.security.authn.x509.impl.CertificateUtils.convertToPrivateKey(CertificateUtils.java:286)

voms-clients relies on CANL to load credentials (which relies on Bouncycastle).
This seems to be a problem in bouncycastle when decyphering keys issued by openssl 1.1.x:

bcgit/bc-java#400

Taking inspiration from the issue above, I reproduced the issue as follows:

# openssl version
OpenSSL 1.1.1a  20 Nov 2018
# openssl genrsa -out key.pem 2048
# openssl pkcs8 -topk8 -inform PEM -in key.pem -out testkey-1.1.pem
...
$ openssl version
OpenSSL 1.0.2q  20 Nov 2018
$ openssl genrsa -out key.pem 2048
$ openssl pkcs8 -topk8 -inform PEM -in key.pem -out testkey-1.0.2.pem

And then with this small program:

public class TestPrivateKeyParsing {

  public static void main(String[] args) throws IOException {

    FileInputStream key1_0 = new FileInputStream("testkey-1.0.2.pem");
    FileInputStream key1_1 = new FileInputStream("testkey-1.1.pem");
    PasswordSupplier ps = () -> "pippo".toCharArray();
    
    CertificateUtils.loadPEMPrivateKey(key1_0,ps);
    System.out.println("key_1_0 loaded succesfully");
    CertificateUtils.loadPEMPrivateKey(key1_1,ps);
    System.out.println("key_1_1 loaded succesfully");
  }
}

which prints on my machine:

key_1_0 loaded succesfully
Exception in thread "main" java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.
	at eu.emi.security.authn.x509.impl.CertificateUtils.convertToPrivateKey(CertificateUtils.java:286)
	at eu.emi.security.authn.x509.impl.CertificateUtils.internalLoadPK(CertificateUtils.java:271)
	at eu.emi.security.authn.x509.impl.CertificateUtils.loadPEMPrivateKey(CertificateUtils.java:242)
	at it.infn.mw.iam.test.TestPrivateKeyParsing.main(TestPrivateKeyParsing.java:20)
Caused by: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: javax.crypto.BadPaddingException: pad block corrupted
	at org.bouncycastle.pkcs.PKCS8EncryptedPrivateKeyInfo.decryptPrivateKeyInfo(Unknown Source)
	at eu.emi.security.authn.x509.impl.CertificateUtils.resolvePK(CertificateUtils.java:306)
	at eu.emi.security.authn.x509.impl.CertificateUtils.convertToPrivateKey(CertificateUtils.java:280)
	... 3 more
Caused by: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted
	at javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:128)
	at javax.crypto.CipherInputStream.read(CipherInputStream.java:246)
	at org.bouncycastle.util.io.Streams.pipeAll(Unknown Source)
	at org.bouncycastle.util.io.Streams.readAll(Unknown Source)
	... 6 more
Caused by: javax.crypto.BadPaddingException: pad block corrupted
	at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source)
	at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
	at javax.crypto.Cipher.doFinal(Cipher.java:2047)
	at javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:125)
	... 9 more

[andreaceccanti]

@golbi also probably should have a look...

[paulmillar]

This issue has become pressing, as there are now VOMS servers that no longer work with the C-client (voms-proxy-init2). The IAM VOMS server is an example of such a VOMS server. The C-client fails when requesting the AC from such a server.

Via email, Andrea pointed out two work-arounds for this issue:

1. Use the (unsupported) C-client to generate the proxy

First, generate the proxy (without any VOMS AC):

paul@celebrimbor:~$ voms-proxy-init2
Enter GRID pass phrase:
Your identity: /C=DE/O=GermanGrid/OU=DESY/CN=Alexander Paul Millar
Creating proxy ...................................................... Done

Your proxy is valid until Fri Aug  9 01:44:30 2019

Then use the Java-client to add the VOMS AC to the proxy:

paul@celebrimbor:~$ voms-proxy-init3 --voms escape --noregen
Contacting voms-escape.cloud.cnaf.infn.it:15000 [/DC=org/DC=terena/DC=tcs/C=IT/L=Frascati/O=Istituto Nazionale di Fisica Nucleare/CN=voms-escape.cloud.cnaf.infn.it] "escape"...
Remote VOMS server contacted succesfully.

WARNING: proxy lifetime limited to issuing credential lifetime.

Created proxy in /tmp/x509up_u1000.

Your proxy is valid until Fri Aug 09 01:44:30 CEST 2019

2. Re-encrypt the private key

paul@celebrimbor:~$ openssl rsa -in ~/.globus/original-key.pem -des3 -out ~/.globus/userkey.pem 
Enter pass phrase for /home/paul/.globus/GridKa-paulmillar-2019-key.pem:
writing RSA key
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase: