CMS with encrypted detached data

[ratal]

Hi
I would need to encrypt data, not envelopped in CMS (need to split in blocks, etc. for specific purposes) while keeping the CMS for recipient to decrypt it later.
With openssl command line it is possible to:
openssl cms -encrypt -in <file_not_encrypted> -outform DER -out <metadata.der> -recip <certificate.pem> > <encrypted_file.bin>
However, the cms encrypt(certs: &StackRef<X509>, data: &[u8], cipher: Cipher, flags: CMSOptions) -> Result<CmsContentInfo, ErrorStack> do not allow to return encrypted data of mutate buffer containing encrypted data.

Looking at openssl library documentation for CMS_encrypt:
The data being encrypted is included in the CMS_ContentInfo structure, unless CMS_DETACHED is set in which case it is omitted. This is rarely used in practice and is not supported by SMIME_write_CMS().

Where does come from the limitation? Is it rust openssl crate or openssl library? If openssl, I do not get why you can do in command line but not supported from library, because it is rarely used..

Should I consider workaround, like envelopping in CMS to then later extract it and replace to empty type, or propose a pull request?

[botovq]

The quoted documentation tells you that

1. CMS_encrypt() does the job (it's what the openssl cms "app" used to use before OpenSSL 3 happened. I haven't looked closely, but it shold be possible for you to follow the SMIME_ENCRYPT logic in apps/cms.c.)
2. SMIME_write_CMS() has this limitation of not supporting this (it has many limitations; it should not be used due to lack of proper error checking. The openssl app only uses it if you pass -resign and if the output format is smime).

Added: I assume you will need a different wrapper around CMS_encrypt() than the one currently exposed by rust-openssl. I don't think this is a major problem, though.

[ratal]

Thanks,
My data is not SMIME, Flag is Binary.
Indeed, I am not too familiar with ffi but looking at the code and comparing with sign/verify, it should be possible to add an optional mutable buffer argument for both encrypt&decrypt. Do you think methods signature change would be accepted?

[botovq]

This is ultimately up to the maintainers and not my decision, but I would avoid a breaking change and provide new methods that suit your needs as a first step.