🌱 Allows to redefine ETCD client logger

[dmvolod]

What this PR does / why we need it:
For some reasons, we need to adjust ECTD client logger level to avoid have an unpredictable number of warnings in the log, like

{"level":"warn","ts":"2025-05-22T16:36:08.530926Z","caller":"v3@v3.5.15/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc016cf01e0/etcd-test-control-plane-dx4r5","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: error upgrading connection: error sending request: Post \\\"https://XXX.XX.XX.XX:6443/api/v1/namespaces/kube-system/pods/etcd-test-control-plane-dx4r5/portforward?timeout=10s\\\": EOF\""}
{"level":"warn","ts":"2025-05-22T16:36:23.642869Z","caller":"v3@v3.5.15/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0041314a0/etcd-test-control-plane-dx4r5","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded"}
{"level":"warn","ts":"2025-05-22T16:37:11.809235Z","caller":"v3@v3.5.15/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc01595fa40/etcd-test-control-plane-dx4r5","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: error upgrading connection: error sending request: Post \\\"https://XXX.XX.XX.XX:6443/api/v1/namespaces/kube-system/pods/etcd-test-control-plane-dx4r5/portforward?timeout=10s\\\": EOF\""}
{"level":"warn","ts":"2025-05-22T16:37:30.158754Z","caller":"v3@v3.5.15/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc009898f00/etcd-test-control-plane-dx4r5","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = received context error while waiting for new LB policy update: context deadline exceeded"}

But now, ETCD log level is hardcoded to the zapcore.InfoLevel. This PR allows to redefine it on init in the embedded controllers. If we need to redefine it globally with env variable or run option for any deployment, please let me know.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelspeed for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dmvolod]

/area provider/control-plane-kubeadm

[sbueringer]

Wondering if kubernetes has a similar use case and how are they handling it (e.g. in the kube-apiserver)

[dmvolod]

Wondering if kubernetes has a similar use case and how are they handling it (e.g. in the kube-apiserver)

This is a very good point. It doesn't work in kube-apiserver at the moment, because community copied/pasted (https://github.com/kubernetes/kubernetes/blob/b35c5c0a301d326fdfa353943fca077778544ac6/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go#L97-L111) the etcdClientDebugLevel function from etcd, which is no longer actual for now. I will plan to create a similar PR for Kubernetes also.

I requested an export of this function in the PR etcd-io/etcd#20006. But as far as I understand, there are no plans to backport it to etcd 3.5.

The only downside to this approach is that it won't work natively. We can have both approaches at the same time to get a universal solution. That is, set the logger via a function and change the logging level via ETCD_CLIENT_DEBUG variable. The only thing is that at the moment we will also have to copy/paste the function until the release of etcd 3.6 with this fix is ​​released and we will not switch to it. I could create and track this issue.

[sbueringer]

I was wondering if there is a way to "reuse" our regular log level for the etcd logger

[dmvolod]

I was wondering if there is a way to "reuse" our regular log level for the etcd logger

Seems to with some trick like that, it could be possible, but should be validated

func (r *KubeadmControlPlaneReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager, options controller.Options) error {
	logger, _ := logutil.CreateDefaultZapLogger(zapcore.Level(ctrl.LoggerFrom(ctx).GetV()))
	etcd.SetLogger(logger)

But in this case, it would be nice to redefine log destination also in advance to log level. SetLogger will help with it.

[fabriziopandini]

nit: can we move this right after etcdClientLogger is defined

[fabriziopandini]

What about moving SetLogger to controllers/alias.go (I would prefer to avoid adding a new package for this)

[sbueringer]

To be honest I don't think we should address this issue this way at all. 99% of our users can't use this

[dmvolod]

Generally, I agree with you, it's a very rare case, but if someone to change controller log level to minimal or want to disable it, will see these warnings always in the logs.

I will try to inherit log level from controller-runtime logger.

[sbueringer]

I have to take closer look and currently not much time due to urgent v1beta2 work, but what do you think about introducing an additional command line flag in KCP that allows to set the etcd logger log level?

This way everyone can use it and not just the very few folks that consume the exported KCP controller code

[dmvolod]

Ok, not a problem. In this case, do you mean, that we need separate log levels for controller and etcd client. Now, I converted logr -> zapcore levels and both users (embedded and non embedded) could change it globally for controller and etcd client logging.

[sbueringer]

I have two problems with the current implementation:

1. I don't want to export a public func that allows setting a different etcd logger. I would like to keep this an implementation detail as much as possible
2. I think the --v log levels don't map well to the zap log levels. They seem to follow a different philosophy (--v=0-10 vs debug,info, ...)

So I would suggest:

1. Let's introduce a new command line flag for the KCP controller. Something like "etcd-client-log-level"
2. Create the etcd logger in main.go and pass it through the KubeadmControlPlaneReconciler structs down all the way to where we use it

The flag should have (at least) the following possible values: 'debug', 'info', 'error', 'panic'. (of course we have to map the strings to the zap log level types)

[dmvolod]

Ok, it's not a problem to implement additional option, i.e. etcd-client-log-level. I will do it.

But please have a look at this description, it seems to there is a logic to convert logr level to zapcore
https://github.com/go-logr/zapr/blob/6a622587cb9d0540da5cd87a37621cb08eaf8e67/zapr.go#L50-L53

[sbueringer]

Interesting. I thought zap has the following log levels:

• Debug -1
• Info 0
• Warn 1
• Error 2
• DPanic 3
• Panic 4
• Fatal 5

https://github.com/uber-go/zap/blob/master/zapcore/level.go#L35-L52

What am I misinterpreting? :)

[dmvolod]

Yeah, description and code are bit confusing. Now I understand that your original approach is more suitable for our case, without SetLogger exporting.

Will implement it soon.

[dmvolod]

/test pull-cluster-api-test-main

[dmvolod]

/retest