Conversion Webhook not Available: No endpoint for Webhook Service

[smv5309]

Previously I was using an older version of kubebuilder as part of operator-sdk v1.6 and the latest cert-manager 1.7.0 and I had conversion webhooks working. Recently I have migrated the operator over to he latest kubebuilder v4.6.0 and regenerated my operator. I have made no changes to my cert manager manifests however the newest Kubebuilder removed ca-injection and added significant stuff to config/default/kustomization.yaml.

I followed the setup guide as outlined in https://book-v3.book.kubebuilder.io/multiversion-tutorial/deployment and i noticed several things.

1. The webhook/kustomizations.yaml contains a manifests.yaml which does not exist. Even when you run for example kubebuilder create webhook --group app --version v1 --kind AppSpec --defaulting --programmatic-validation to create a mutilating validating webhook it does not get created. There for it needs to be commented out

2.Currently I am encountering the following error.

[https://x-operator-webhook-service.x-operator-system.svc:443/convert](https://x-operator-webhook-service.x-operator-system.svc/convert)? timeout=30s": no endpoints available for service "x-operator-webhook-service"

This occurs any time i try to deploy a manifest using one of the CRDS the operator manages.

I have cert-manager installed and see the certificates in kubernetes.
NAMESPACE NAME READY SECRET AGE
x-operator-system x-operator-metrics-certs True metrics-server-cert 6m29s
x-operator-system x-operator-serving-cert True webhook-server-cert 6m28s

I am also seeing the following inside the cert-manager-webhook logs
"http:TLS handshake error from 10.150.0.0:35564: EOF"

I have properly uncommented the config/default/kustomization.yaml as outlined in the current documentation.

I am unsure if this is related or not but any assistance solving this lack of endpoint would be greatly appreciated. I am quite stuck, my only recourse at the moment is potentially to revert to an older version of kubebuilder and re generate my entire project again which is not really an option.

I also added the following to the conversion revisions in the config/crd/patches/webhook_<name>.yaml to support a second version i do not know if this is also necessary.

conversionReviewVersions: ["v1", "v2"]

[camilamacedo86]

Hi @smv5309 👋

Following all comments inline — I hope this helps clarify and get things working for you! 💛

---

🛠️ Regarding the changes

I have made no changes to my cert manager manifests however the newest Kubebuilder removed ca-injection and added significant stuff to config/default/kustomization.yaml.

So sorry for the trouble. We truly do our best to avoid burn-style changes, but unfortunately, a few important reasons required us to move forward with these updates:

🔸 Removal of kube-rbac-proxy : #3907

We had to remove kube-rbac-proxy. While it was convenient and generated certs on-the-fly, it’s not a secure approach for production-grade environments — and it’s now deprecated.

Instead, we now rely on cert-manager for both webhook and metrics certificates, and use certwatcher in main.go to handle automatic rotation safely and reliably. See:

• kubebuilder/testdata/project-v4/cmd/main.go

  Lines 81 to 87 in 8ec260a

  	flag.StringVar(&webhookCertPath, "webhook-cert-path", "", "The directory that contains the webhook certificate.")
  	flag.StringVar(&webhookCertName, "webhook-cert-name", "tls.crt", "The name of the webhook certificate file.")
  	flag.StringVar(&webhookCertKey, "webhook-cert-key", "tls.key", "The name of the webhook key file.")
  	flag.StringVar(&metricsCertPath, "metrics-cert-path", "",
  	"The directory that contains the metrics server certificate.")
  	flag.StringVar(&metricsCertName, "metrics-cert-name", "tls.crt", "The name of the metrics server certificate file.")
  	flag.StringVar(&metricsCertKey, "metrics-cert-key", "tls.key", "The name of the metrics server key file.")

• kubebuilder/testdata/project-v4/cmd/main.go

  Lines 113 to 141 in 8ec260a

  	// Create watchers for metrics and webhooks certificates
  	var metricsCertWatcher, webhookCertWatcher *certwatcher.CertWatcher
  	// Initial webhook TLS options
  	webhookTLSOpts := tlsOpts
  	if len(webhookCertPath) > 0 {
  	setupLog.Info("Initializing webhook certificate watcher using provided certificates",
  	"webhook-cert-path", webhookCertPath, "webhook-cert-name", webhookCertName, "webhook-cert-key", webhookCertKey)
  	var err error
  	webhookCertWatcher, err = certwatcher.New(
  	filepath.Join(webhookCertPath, webhookCertName),
  	filepath.Join(webhookCertPath, webhookCertKey),
  	)
  	if err != nil {
  	setupLog.Error(err, "Failed to initialize webhook certificate watcher")
  	os.Exit(1)
  	}
  	webhookTLSOpts = append(webhookTLSOpts, func(config *tls.Config) {
  	config.GetCertificate = webhookCertWatcher.GetCertificate
  	})
  	}
  	webhookServer := webhook.NewServer(webhook.Options{
  	TLSOpts: webhookTLSOpts,
  	})

🔸 Fix for CA Injection Handling

Fixed CA injection for conversion webhooks. Previously, the CA injection patch was not accurate; The injection should occur only for CRDs, which are conversion types and not for all CRDs when a webhook with --conversion option is scaffolded. The issue goes back to release 3.5.0 (where to replace vars for replacements was done and the kustomize/v2-alpha plugin was introduced). It was not previously found, likely because conversion webhook features were incomplete, which is addressed in this release. Now, users can use the tool to generate the conversion webhooks properly (#4254). (#4282). See what is new bellow

---

✨ What's New?

Since this release: https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v4.4.0, you can scaffold a conversion webhook using:

# Create API to check webhook --conversion from v1 to v2
$ kubebuilder create api --group example.com --version v1 --kind Wordpress --controller=true --resource=true
$ kubebuilder create api --group example.com --version v2 --kind Wordpress --controller=false --resource=true
$ kubebuilder create webhook --group example.com --version v1 --kind Wordpress --conversion --spoke v2

This is part of a broader effort to fully support webhook conversions, which weren’t properly handled before — especially when bugs like the one above are involved.

Note: If you run the commands using the tool, it will generate the webhook configuration and scaffolding for you — you just need to plug in your logic.
In the next release, the tool will uncomment all relevant sections by default to simplify setup, since webhook config is often tricky. PR: #4826

---

🧯 Addressing Upgrade Pain

We're also working to automate project upgrades to reduce this kind of "burn":
See: #4302
This is part of Google Summer of Code, and the student has already started contributing. 🌱

---

🔍 Regarding Your Error

We’ve tested the updated scaffolding in our e2e tests, and it's working as expected. The issue you're seeing is likely caused by trying to adapt the new scaffolding into an existing customized project.

Let’s walk through the key areas you can check. You can always refer to the sample project at:
📁 testdata/project-v4 → testdata/project-v4

Regards:

I am also seeing the following inside the cert-manager-webhook logs
"http:TLS handshake error from 10.150.0.0:35564: EOF"

✅ Check 1: TLS Flags in main.go

Ensure the following flags are present:

kubebuilder/testdata/project-v4/cmd/main.go

Lines 81 to 87 in 8ec260a

	flag.StringVar(&webhookCertPath, "webhook-cert-path", "", "The directory that contains the webhook certificate.")
	flag.StringVar(&webhookCertName, "webhook-cert-name", "tls.crt", "The name of the webhook certificate file.")
	flag.StringVar(&webhookCertKey, "webhook-cert-key", "tls.key", "The name of the webhook key file.")
	flag.StringVar(&metricsCertPath, "metrics-cert-path", "",
	"The directory that contains the metrics server certificate.")
	flag.StringVar(&metricsCertName, "metrics-cert-name", "tls.crt", "The name of the metrics server certificate file.")
	flag.StringVar(&metricsCertKey, "metrics-cert-key", "tls.key", "The name of the metrics server key file.")

✅ Check 2: certwatcher usage for webhooks and metrics

Make sure certwatcher is used to load and rotate certificates:

kubebuilder/testdata/project-v4/cmd/main.go

Lines 113 to 141 in 8ec260a

	// Create watchers for metrics and webhooks certificates
	var metricsCertWatcher, webhookCertWatcher *certwatcher.CertWatcher
	// Initial webhook TLS options
	webhookTLSOpts := tlsOpts
	if len(webhookCertPath) > 0 {
	setupLog.Info("Initializing webhook certificate watcher using provided certificates",
	"webhook-cert-path", webhookCertPath, "webhook-cert-name", webhookCertName, "webhook-cert-key", webhookCertKey)
	var err error
	webhookCertWatcher, err = certwatcher.New(
	filepath.Join(webhookCertPath, webhookCertName),
	filepath.Join(webhookCertPath, webhookCertKey),
	)
	if err != nil {
	setupLog.Error(err, "Failed to initialize webhook certificate watcher")
	os.Exit(1)
	}
	webhookTLSOpts = append(webhookTLSOpts, func(config *tls.Config) {
	config.GetCertificate = webhookCertWatcher.GetCertificate
	})
	}
	webhookServer := webhook.NewServer(webhook.Options{
	TLSOpts: webhookTLSOpts,
	})

✅ Check 3: config/default/kustomization.yaml

Uncomment the section to enable webhook + certManager injection:

kubebuilder/testdata/project-v4/config/default/kustomization.yaml

Lines 120 to 155 in 8ec260a

	# - source: # Uncomment the following block if you have any webhook
	# kind: Service
	# version: v1
	# name: webhook-service
	# fieldPath: .metadata.name # Name of the service
	# targets:
	# - select:
	# kind: Certificate
	# group: cert-manager.io
	# version: v1
	# name: serving-cert
	# fieldPaths:
	# - .spec.dnsNames.0
	# - .spec.dnsNames.1
	# options:
	# delimiter: '.'
	# index: 0
	# create: true
	# - source:
	# kind: Service
	# version: v1
	# name: webhook-service
	# fieldPath: .metadata.namespace # Namespace of the service
	# targets:
	# - select:
	# kind: Certificate
	# group: cert-manager.io
	# version: v1
	# name: serving-cert
	# fieldPaths:
	# - .spec.dnsNames.0
	# - .spec.dnsNames.1
	# options:
	# delimiter: '.'
	# index: 1
	# create: true

---

🔁 Verifying CRD Conversion Webhook

CRD Patch Entries

If you generated a webhook using --conversion, ensure your kustomization.yaml has proper CA injection lines:

https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/default/kustomization.yaml#L219-L252

These lines are injected via markers like:

# +kubebuilder:scaffold:crdkustomizecainjectionns

You need to uncomment to allow CA to be injected.

CRD Webhook Patch File

Check the CRD webhook patch file was scaffolded, e.g.: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/crd/patches/webhook_in_firstmates.yaml

Also ensure the CRD kustomization has:

https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/crd/kustomization.yaml#L10-L14

---

💬 Extra Comments to Your Questions

The webhook/kustomizations.yaml contains a manifests.yaml which does not exist.

This file is scaffolded by default:

https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/webhook/kustomization.yaml
It may have been deleted when merging with your custom project — feel free to restore it.

I also added conversionReviewVersions: ["v1", "v2"]

✅ Yes, that is required — and likely the reason you're seeing the "no endpoint" error.
Here’s an example with it properly set: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/crd/patches/webhook_in_firstmates.yaml

---

🙏 Final Thoughts

Really sorry for the disruption this caused.
We’re doing everything we can to ensure that projects are scaffolded following best practices — with stronger focus on security, robust webhooks, and future stability. Beyond what is just out of our control resulted of very old decisions.

We know updates like this can be painful, but the goal is to remove this kind of “burn” going forward and make things simpler.

Things should be much smoother now — especially if you follow the scaffolding closely.

Let us know if we can help with anything else 💛

[smv5309]

Two things:

1. am i reading this correctly I need to uncomment out the following

+kubebuilder:scaffold:crdkustomizecainjectionns
it says in the code: # Do not remove or uncomment the following scaffold marker; required to generate code for target CRD.

2. All of the code that you highlighted exists in my project. I physically made a new kubebuilder project with the newer version and moved my crd, types and controller code into the new project Note using Kubebuilder Version v4.6.0. I set up my webhooks using the new commands and followed the https://book-v3.book.kubebuilder.io/multiversion-tutorial/deployment guide. Specifically I uncommented all of the labeled portions in config/default/kustomization.yaml that you outline in the response, the documentation, and made sure any code following this was uncommented: # Uncomment the following block if you have a ConversionWebhook (--conversion)

These TSL errors still persist in the same form. Are there any alternative ideas?
Examples:
[https://x-operator-webhook-service.x-operator-system.svc:443/convert](https://x-operator-webhook-service.x-operator-system.svc/convert)? timeout=30s": no endpoints available for service "x-operator-webhook-service"

cert-manager-webhook logs:
"http:TLS handshake error from 10.150.0.0:35564: EOF"

I have yet to uncomment out the skaffold line listed above its the only thing i have not tried.

[smv5309]

The webhook/kustomizations.yaml contains a manifests.yaml which does not exist.

This file is scaffolded by default:

https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/webhook/kustomization.yaml
It may have been deleted when merging with your custom project — feel free to restore it.

this comment is incorrect. I just started from scratch with a new webhook and this file was not scaffolded, using any commands listed in documentation. Even when the --defaulting --programmatic-validation flags are set

[camilamacedo86]

Closing as sorted out, if you need feel free to re-open.