Support for Vendor Defined Key Types and mechanisms

[sandevins]

Is there any plan to add support to vendor defined key types and mechanisms? Right now the openssl engine rejects any key that is not RSA or EC. Is this really necessary? You could simply pass plainly the CKK_VENDOR_DEFINED specific value to C_Sign (for example).

Maybe this should be more flexible looking for future grand modifications in the cryptography landscape like the one we are living with PQC.

[mtrojnar]

Do you have any suggestions for how we should test this feature?

[sandevins]

What I am using is the Quantum Protect simulator from Utimaco, but it's based on propietary software (it's free to use). I don't know any other pkcs11 providers with Vendor Defined Mechanisms, but I can try to research on it if the first option isn't enough.

[dengert]

Google for "Quantum" "PKCS11"
The Google AI points at articles from wolfssl and IBM.
Also listed some work being done by others including osias-open.org that maintains the PKCS11 standards.

[mtrojnar]

Hey folks,

If you've found some software that's free to use and works well, feel free to update your replies with links to its homepage or download page. I can always search by name, but it's helpful to know I'm looking at the exact same thing you tried and recommend.

BTW, an AI once recommended me a library that never existed—so yeah, links are appreciated.

[dengert]

P.S. Libp11 is mainly used to with OpenSSL engine or provider so it cannot do much without OpenSSL support for any mechanisms not supported by OpenSSL.

The OpenSC pkcs11-tool is a utility and example for how to load and call PKCS11 modules So if you can find a Pkcs11 module with some PQC mechanism even if vendor provided, that would be a place to start.
https://github.com/OpenSC/OpenSC/blob/master/src/tools/pkcs11-tool.c#L357C2-L357C3 shows how to enter a vendor CKM as hex.

[dengert]

If you've found some software that's free to use and works well, feel free to update your replies with links to its homepage
I will let @sandevins do the hard work, as it looks there is not much out there, but a lot of talk. The Oasis community appears to be trying to define PKCS11 mechanisms for PQC, rather the leaving it up to every vendor.

I just responded as libp11 can not do anything without OpenSSL and a PKCS11 module that supports some PQC mech.

[mtrojnar]

I looked into our code, and I'm not so sure anymore. I think it might be possible to implement private key operations on EVP_PKEY objects without special OpenSSL support—even for keys that aren't RSA, DH, or EC. We would just need a suitable PKCS#11 module to test with.

If @sandevins requested support for CKK_VENDOR_DEFINED, I assume he already has a PKCS#11 module that provides such keys.

[dengert]

https://openssl-library.org/post/2025-04-08-openssl-35-final-release/

https://developer.ibm.com/tutorials/awb-quantum-safe-openssl/#step-4-install-open-quantum-safe-provider-for-openssl-35
(from February)

Google: "PKCS11" module with "PQC"
Shows even more info.

[mtrojnar]

Thank you for the links. How exactly are OpenSSL providers useful for testing CKK_VENDOR_DEFINED functionality of PKCS#11 modules?

[dengert]

Thank you for the links. How exactly are OpenSSL providers useful for testing CKK_VENDOR_DEFINED functionality of PKCS#11 modules?

Probably, not usefully, unless some PKCS11 module is implementing one of the OpenSSL PQC algorithms and Oasis has not defined
a CKM_ for it yet along with all the other PKCS11 stuff.

Looks like OpenSSL supports some PQC algorithms in software as a provider. Now @sandevins needs to find a PKCS11 module that supports ona mechanisms. (I am assuming @sandevins wants to use libp11 because it uses pkcs11 with OpenSSL to use keys on a smart card or HSM where so crypto is done smart card or HSM.) The only time I use libp11 is to sign a CSR with a key generated on a smart card and not exportable. If @sandevins is willing to use one of the OpenSSL supported PQC in software and does not need to use PKCS11, then there is no need libp11.

Libp11 is the last hurdle in the race, that helps OpenSSL to use hardware for keys and crypto.

[sandevins]

Indeed, I need to do cryptographic operations inside an HSM through a PKCS#11 provider. That is the reason I need libp11. The HSM supports PQC algorithms, OpenSSL supports PQC algorithms in its version 3.5.0, but there is still no support for PQC algorithms in the PKCS#11 standard.

[mtrojnar]

Indeed, I need to do cryptographic operations inside an HSM through a PKCS#11 provider. That is the reason I need libp11. The HSM supports PQC algorithms, OpenSSL supports PQC algorithms in its version 3.5.0, but there is still no support for PQC algorithms in the PKCS#11 standard.

Great. Do you have such a PKCS#11 module that we could use for testing of the requested feature? Send us the software (with hardware if required).

[dengert]

@sandevins If you have the specs for the device, it might be possible to add a driver to https://github.com/OpenSC/OpenSC
but there are PKCS11 issues for example what is the format for a PQC Public key, What OIDs are assigned, And you should try
and follow any guidelines from Oasis. Adding a new mechanism to OpenSC could be done, but it is not easy. (I add the EC mechanism to OpenSC in the early 2000's for PIV card, but PKCS11 specs included EC, making it easier.)

Post you question on https://github.com/OpenSC/OpenSC, maybe someone is already looking at the problem.

It is not clear if you are doing this on your own, or you are employed by the HSM vendor.
If you can make the device available to OpenSC developer, (not me) that would also help.

[sandevins]

What I am using right now is the Utimaco HSM Simulator along with its PKCS#11 Connector

https://support.hsm.utimaco.com/hsm-simulator

I am not a direct employee of Utimaco so I cannot provide the hardware but that simulator can be used freely.

I am surprised that the addition of mechanisms is not a direct task, as libp11 should only redirect the mechanism and keytype as it is to the provider it's using, right?