Grpc over mtls is not working

[J1a-wei]

I deploy mtls grpc server, and try to call sever by ingress. it stuck at some point.

Here is my idea. But I deploy it in Kubernetes. The same program in local env works well.

step1. create two certificates by cert-manager

# server-cert
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mtls-grpc-server
spec:
  dnsNames:
  - $(MyIngressHost)
  - grpc-mtls-server
  - grpc-mtls-server.default
  - grpc-mtls-server.default.svc
  - grpc-mtls-server.default.svc.cluster.local
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned-cluster-issuer
  secretName: mtls-grpc-server-cert
  commonName: grpc-server
  duration: 438000h # 50 * 365 * 24 h

# client cert 
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mtls-grpc-client
spec:
  dnsNames:
  - example.com
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned-cluster-issuer
  secretName: mtls-grpc-client-cert
  commonName: grpc-client
  duration: 438000h # 50 * 365 * 24 h

then I got two certificates.

$ kubectl get secret | grep cert
mtls-grpc-client-cert     kubernetes.io/tls                     3      5h30m
mtls-grpc-server-cert     kubernetes.io/tls                     3      5h33m

step2 deploy pod svc ingress

The pod deployment yaml s ignored, it uses the server as the certificate and the client's certificate as the ca-cert.

apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/client-cert: mtls-grpc-client-cert
    konghq.com/protocol: grpcs
  name: grpc-mtls-server
  namespace: default
spec:
  clusterIP: 10.100.217.8
  clusterIPs:
  - 10.100.217.8
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: grpcs
    port: 10200
    protocol: TCP
    targetPort: 10200
  selector:
    app: grpc-mtls-server
  sessionAffinity: None
  type: ClusterIP

ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/protocols: grpc,grpcs
    kubernetes.io/ingress.class: kong
  name: demo
  namespace: default
spec:
  rules:
  - host: $MyIngressHost
    http:
      paths:
      - backend:
          service:
            name: grpc-mtls-server
            port:
              number: 10200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - $MyIngressHost
    secretName: mtls-grpc-server-cert

Result

local env(Forward pod port)

kubectl port-forward pod/grpc-mtls-server-798bcf695d-jnvcx  10200:10200

# edit local /etc/hosts 
grpc-mtls-server 127.0.0.1

client bring self-signed certificates  server.crt as ca.crt, client.crt as local cert.  Call grpcs://grpc-mtls-server:10200 
it works well
 

When I try call grpc server endpoint by ingress. My client logs

panic: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type

Check kong-proxy log.

2022/08/17 13:23:36 [error] 1101#0: *13152528 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 100.168.65.120, server: kong, request: "POST /greet.Greeting/SayHello HTTP/2.0", upstream: "grpcs://100.168.55.241:10200", host: "$MY_INGRESS_HOST"
--
Wed, Aug 17 2022 9:23:36 pm | 2022/08/17 13:23:36 [error] 1101#0: *13152528 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 100.168.65.120, server: kong, request: "POST /greet.Greeting/SayHello HTTP/2.0", upstream: "grpcs://100.168.55.241:10200", host: "MY_INGRESS_HOST"
Wed, Aug 17 2022 9:23:36 pm | 2022/08/17 13:23:36 [error] 1101#0: *13152528 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 100.168.65.120, server: kong, request: "POST /greet.Greeting/SayHello HTTP/2.0", upstream: "grpcs://100.168.55.241:10200", host: "MY_INGRESS_HOST"
Wed, Aug 17 2022 9:23:36 pm | 2022/08/17 13:23:36 [error] 1101#0: *13152528 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 100.168.65.120, server: kong, request: "POST /greet.Greeting/SayHello HTTP/2.0", upstream: "grpcs://100.168.55.241:10200", host: "MY_INGRESS_HOST"
Wed, Aug 17 2022 9:23:36 pm | 2022/08/17 13:23:36 [error] 1101#0: *13152528 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 100.168.65.120, server: kong, request: "POST /greet.Greeting/SayHello HTTP/2.0", upstream: "grpcs://100.168.55.241:10200", host: "MY_INGRESS_HOST"
Wed, Aug 17 2022 9:23:36 pm | 2022/08/17 13:23:36 [error] 1101#0: *13152528 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 100.168.65.120, server: kong, request: "POST /greet.Greeting/SayHello HTTP/2.0", upstream: "grpcs://100.168.55.241:10200", host: "MY_INGRESS_HOST"

Source Code: https://github.com/islishude/grpc-mtls-example

I have checked a lot of information, but I could not find any case of kong mtls grpc. The official doc configuration of tls can run Thanks in advance!

[shaneutt]

It appears as though the certificates are invalid. This could be for a variety of reasons, including (but not limited to):

• the certificate data is corrupt
• the client certificate is being used with an non-matching private key
• the server is expecting our client cert and didn't receive it

When you said Call grpcs://grpc-mtls-server:10200 it works well how was this done? Did this in fact use the client certificate?

I would recommend verifying that outside of Kong the certificate is working properly with another client. If that all appears to be working, consider using something like pcap to capture the traffic and verify that the client certificate is at least being sent. If all of that checks out then it would appear the problem is within the negotiation itself and that being the case could be a misconfiguration or even a bug.

[J1a-wei]

Yes. It used the client certificate. If I remove bring client certificate code, It will be error.
I write grpc-client with golang. Here is the client source code

package main

import (
	"context"
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"
	"os"

	"github.com/islishude/grpc-mtls-example/greet"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
)

func LoadKeyPair() credentials.TransportCredentials {
	certificate, err := tls.LoadX509KeyPair("certs/client.crt", "certs/client.key")
	if err != nil {
		panic("Load client certification failed: " + err.Error())
	}

	ca, err := ioutil.ReadFile("certs/server.crt")
	if err != nil {
		panic("can't read ca file")
	}

	capool := x509.NewCertPool()
	if !capool.AppendCertsFromPEM(ca) {
		panic("invalid CA file")
	}

	tlsConfig := &tls.Config{
		Certificates: []tls.Certificate{certificate},
		RootCAs:      capool,
	}

	return credentials.NewTLS(tlsConfig)
}

func main() {
	conn, err := grpc.Dial("example.com:10200", grpc.WithTransportCredentials(LoadKeyPair()))
	if err != nil {
		panic(err)
	}
	defer conn.Close()

	client := greet.NewGreetingClient(conn)

	var name = "world"
	if len(os.Args) > 1 {
		name = os.Args[1]
	}
	resp, err := client.SayHello(context.Background(), &greet.SayHelloRequest{Name: name})
	if err != nil {
		panic(err)
	}
	fmt.Println(resp.GetGreet())
}

[J1a-wei]

By the way. Kong chart version was kong:2.7.0

I found a suspicious point.

Here is my service yaml.

apiVersion: v1
kind: Service
metadata:
  annotations:
    konghq.com/client-cert: mtls-grpc-client-cert
    konghq.com/protocol: grpcs
  name: grpc-mtls-server
  namespace: default
spec:
  clusterIP: 10.100.217.8
  clusterIPs:
  - 10.100.217.8
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: grpcs
    port: 10200
    protocol: TCP
    targetPort: 10200
  selector:
    app: grpc-mtls-server
  sessionAffinity: None
  type: ClusterIP

I patch annotation konghq.com/client-cert: mtls-grpc-client-cert
I can find this secret in the default namespace.

When I check kong service by admin URL.

{
    "tls_verify": null,
    "port": 80,
    "client_certificate": null,
    "tags": null,
    "name": "default.grpc-mtls-server.pnum-10200",
    "path": null,
    "connect_timeout": 60000,
    "read_timeout": 60000,
    "created_at": 1660737068,
    "updated_at": 1660737068,
    "enabled": true,
    "host": "grpc-mtls-server.default.10200.svc",
    "ca_certificates": null,
    "id": "8a4cafc6-a3a2-5162-8b4f-bf2fee64bddb",
    "write_timeout": 60000,
    "retries": 5,
    "protocol": "grpcs",
    "tls_verify_depth": null
}

client_certificate and path are null

check kong-ingress-controller log

time="2022-08-18T06:12:16Z" level=error msg="could not update kong admin" error="posting new config to /config: HTTP status 400 (message: \"declarative config is invalid: {services={[89]={[\\\"@entity\\\"]={\\\"failed conditional validation given value of field 'protocol'\\\"},client_certificate=\\\"value must be null\\\"}}}\")" subsystem=proxy-cache-resolver

When I restart KIC, all Routes can't be found. Kong-proxy can't work well.

I don't understand. It seems that konghq.com/client-cert: mtls-grpc-client-cert and konghq.com/protocol: grpcs it unable to appear together. If protocol set https. KIC will not print that error.

[shaneutt]

So after reviewing some of your other comments, it looks like you may have stumbled upon a bug in Kong Gateway.

I've created Kong/kong#9270 to try and fix it and get feedback from the Gateway maintainers, I recommend subscribing to that PR for updates and then hopefully that can get this resolved for you.