failed to ping valkey - TLS connecting to IP

[mbrancato]

The controller is connecting to valkey cluster on IP, and its running into an issue with the self-signed TLS certificate.

2025-03-31T01:35:58Z	ERROR	failed to ping valkey	{"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"myvalkey","namespace":"myapp"}, "namespace": "myapp", "name": "myvalkey", "reconcileID": "18de9661-0fb5-439e-bb5b-925876d31faa", "error": "tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.1.1.8"}
hyperspike.io/valkey-operator/internal/controller.(*ValkeyReconciler).balanceNodes
	internal/controller/valkey_controller.go:1569
hyperspike.io/valkey-operator/internal/controller.(*ValkeyReconciler).Reconcile
	internal/controller/valkey_controller.go:210
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:328
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:288
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.2/pkg/internal/controller/controller.go:249

I'm using a ClusterIssuer to have the TLS cert created. It would make sense to connect to the shard / nodes by hostname. I tried setting clusterPreferredEndpointType to hostname but it did not help.

[mbrancato]

As a workaround to this, you can set the environment variable on the controller deployment:

GODEBUG="x509ignoreCN=1"

The controller currently only supports setting the announce to IP:

valkey-operator/internal/controller/valkey_controller.go

Lines 692 to 703 in 88cd80a

	r.Recorder.Event(valkey, "Normal", "Setting",
	fmt.Sprintf("Setting cluster announce ip %s on pod %s for %s/%s", ip, podName, valkey.Namespace, valkey.Name))
	out, err := clients[podName].Do(ctx, clients[podName].B().ConfigSet().ParameterValue().ParameterValue("cluster-announce-ip", ip).Build()).ToString()
	if err != nil {
	logger.Error(err, "failed to set cluster announce ip "+out)
	return err
	}
	cfgs, err := clients[podName].Do(ctx, clients[podName].B().ConfigGet().Parameter("cluster-announce-ip").Build()).ToMap()
	if err != nil {
	logger.Error(err, "failed to get cluster announce ip")
	}

So this basically means that the clients can't validate the IPs or SANs on the certificate.

[dmolik]

I think I solved this in the "huge" zero-downtime-upgrade PR however I probably won't merge that work as is. I'll see how much I can retrofit into main.