Cluster internal certificates with external PKI infra

[zkutasi]

Hi,

We would like to completely use our proprietary custom PKI infrastructure for everything Strimzi-related, both clusterCA, clientsCA, and derived certificates would be made by our PKI infra Pod.

I have not found any way in the documentation to set the internally used certificates (cluster-CA-issued) by Strimzi, I only have found how to replace the cluster-CA itself.
This is however doable in case of clients-CA: I can make the whole clients-PKI external.

Is it possible for the cluster-CA as well, or not at all?

I have seen that the Operator would like to export the clusterCA+key, and I presume the certificate creation will be fully internal after that.

[scholzj]

You can configure your custom cluster CA - the operator will take it and use it to create the server keys. There is a work-in-progress proposal to make it more pluggable and allow you to integrate your certifcate management: strimzi/proposals#46

In theory, if you create the right secrets with the right certificates (including the right SANs etc.), the operator would just re-use them. But it is not documented, so you would need to reverse engineer it.

[zkutasi]

Hi @scholzj ,

You describe my problem: we cannot expose the ClusterCA key from our Vault, this is not according to our security policy, so we would like to give certificates generated by our certificate management to the Operator... the proposal you linked seems to be interesting, will check on it.

Another question popped up: The documentation says this:
https://strimzi.io/docs/operators/in-development/configuring.html#kafka-listener-certificates-str

"Providing Kafka listener certificates for external listeners allows you to leverage existing security infrastructure, such as your organization’s private CA or a public CA. Kafka clients will connect to Kafka brokers using Kafka listener certificates rather than certificates signed by the cluster CA or clients CA."

Isn't this contradicts what is the written in the next statement: "This procedure shows how to configure a listener to use your own private key and server certificate, called a Kafka listener certificate."

So listener certificates are server certificates, it has nothing to do which certificates the clients connect with... or am I missing something? Or can I specify a per-listener clientCA? Or this sentence is talking about plain-TLS and not mutual TLS? To me "connecting" with a certificate means the client certificate, since that is what a client sends.

[scholzj]

This has nothing to do with mutual TLS -> it is just a server certificate. But I'm not sure which part is confusing for you. The or clients CA is wrong. But without that I do not see any reference to mutual TLS. What from this leads you to think it is related to mutual TLS?

You can provide your own server certificates and private keys for the following types of listeners:

* Internal TLS listeners for communication within the Kubernetes cluster
* External listeners (`route`, `loadbalancer`, `ingress`, and `nodeport` types), which have TLS encryption enabled, for communication between Kafka clients and Kafka brokers

These user-provided certificates are called _Kafka listener certificates_.

Providing Kafka listener certificates for external listeners allows you to leverage existing security infrastructure, such as your organization's private CA or a public CA.
Kafka clients will connect to Kafka brokers using Kafka listener certificates rather than certificates signed by the cluster CA.

You must manually renew Kafka listener certificates when needed.

[zkutasi]

Maybe I am nitpicking, sorry about that... but saying "clients will connect to Kafka brokers using listener certificates" is not right... the listener certificate is not client-side, so how a client could connect using that? The client is using a CA to verify the server certificate, and that CA is not the clientCA/clusterCA.
Consider that I change the clientCA to CAc and set a listenercertificate that is issued by CAs (CAc =/= CAs). So the client is using the certificate issued by CAc as client certificate and using CAs... it is not using the "Kafka listener certificates" that the documentation says.

[scholzj]

Maybe I am nitpicking, sorry about that... but saying "clients will connect to Kafka brokers using listener certificates" is not right... the listener certificate is not client-side, so how a client could connect using that? The client is using a CA to verify the server certificate, and that CA is not the clientCA/clusterCA.

Well, I think strictly speaking it is correct -> the client will "use" this certificate -> it will have to verify it to authenticate the server. But I get that this could be confusing and can mean different things. So would you rephrase it? Or would you leave it out completely? What about saying

Kafka clients will need to trust the CA which was used to sign the listener certificate.

would that work?

[zkutasi]

Exactly, you nailed what my problem is... "certificate" can mean server certificate of CA certificate. Your suggestion is much cleaner.

Returning back to my point of mutual TLS: it is hell of confusing in that case which certificate you are talking about because in that case, you have 4 different ones: client certificate, clientCA, server certificate and serverCA. The client "uses" then all 4: sends the client certificate when the server asks it in the handshake, asks a client certificate from the clientCA (actually this might be a Kubernetes concept because we have CustomResources for CAs and certs, respectfully, and client helm charts indeed ask for a certificate issued by a specific CA), take the server certificate that the server gives in the handshake and uses the serverCA to verify it.

Your suggestion makes it explicit.

[scholzj]

Ok, I opened #6382 => please let me know if there is still something what should be more clear. And thanks for the feedback - this is useful.

[mattia-crypto]

Facing the same scenario, but I am starting at a more basic level - namely without Vault storing a CA, but in stead using cert-manager and a self-signed CA, then creating an intermediary CA for both the clientCA and the clusterCA accordingly.

I would like to use the clientCA and the clusterCA in stead of the Strimzi auto-generated ones thereafter, but only could find docs related to replacing CA, not bootstrapping the Kafka CRD with pre-existing CAs.

Will try to reverse engineer, but also noticed: #929