conversion webhook for failed: Post : http: server gave HTTP response to HTTPS client

[BradleyMcCallionCOT]

I'm trying to follow along with the official tutorials in creating a conversion webhook for my operator using Kubebuilder. Firstly there are a few hurdles that are poorly documented:

1. The webhook/kustomizations.yaml contains a manifests.yaml which does not exist. This might exist but only if we are scaffolding a validating and mutating webhook as part of the conversion (which isn't always needed). This therefore needs to be commented out along with the kustomizationconfig.yaml in this directory which refers to the Mutating and Validating Webhooks
2. Similarly the default/webhookcainjection_patch.yaml is not needed and should not be commented out along with the replacements defined in the kustomization.yaml which refer to the Mutating and Validating webhooks.

Even once these are completed and the operator will deploy successfully, it then seems to throw an error on reconciliation:

ERROR   Reconciler error ... conversion webhook for test.example.com/v1alpha1, Kind=HelloWorld failed: Post \"https://test-webhook-service.test-operator.svc:443/convert?timeout=30s\": http: server gave HTTP response to HTTPS client

If I'm correct in understanding the architecture the reconciliation will reach out to the conversion webhook at the defined kubernetes service address, which will route this request back into the same pod on the port where the conversion webhook is served? So this sounds to me like the client in the controller making the reconciliation request is using HTTPS but then the webhook responds with an HTTP response.

Obviously this points to a cert manager issue but I have uncommented all the relevant cert manager parts (excluding the above points I noted) and can see that the CRD conversion webhook has the caBundle property as expected.

Is there anything else that I am missing? Is there any insights on this problem that anyone has?

[camilamacedo86]

Hi @BradleyMcCallionCOT,

Thank you for raising this issue. It appears you might be working with a project scaffolded before the fixes introduced in Kubebuilder v4.4.0.

---

🔧 CA Injection Fix for Conversion Webhooks (v4.4.0)

(kustomize/v2, go/v4): Fixed CA injection for conversion webhooks. Previously, the CA injection patch was not accurate; the injection should occur only for CRDs that are conversion types and not for all CRDs when a webhook with the --conversion option is scaffolded. The issue dates back to release 3.5.0, where variable replacements were handled, and the kustomize/v2-alpha plugin was introduced. It was not previously identified, likely because conversion webhook features were incomplete, which is addressed in this release. Now, users can use the tool to generate the conversion webhooks properly.
— Kubebuilder Release Notes

We have been working on stabilizing the implementations of webhooks, certificate management, and metrics. These components had longstanding issues that were not properly surfaced, such as the tool not correctly scaffolding hub versions and missing end-to-end tests. All of these main concerns have been addressed now.

---

🔍 Observations from the Latest Scaffold

• CA Injection Configuration: The injection for each CRD is handled individually. For example:
  

  kubebuilder/testdata/project-v4/config/default/kustomization.yaml

  Lines 219 to 252 in 5d42475

  	# - source: # Uncomment the following block if you have a ConversionWebhook (--conversion)
  	# kind: Certificate
  	# group: cert-manager.io
  	# version: v1
  	# name: serving-cert
  	# fieldPath: .metadata.namespace # Namespace of the certificate CR
  	# targets: # Do not remove or uncomment the following scaffold marker; required to generate code for target CRD.
  	# - select:
  	# kind: CustomResourceDefinition
  	# name: firstmates.crew.testproject.org
  	# fieldPaths:
  	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
  	# options:
  	# delimiter: '/'
  	# index: 0
  	# create: true
  	# +kubebuilder:scaffold:crdkustomizecainjectionns
  	# - source:
  	# kind: Certificate
  	# group: cert-manager.io
  	# version: v1
  	# name: serving-cert
  	# fieldPath: .metadata.name
  	# targets: # Do not remove or uncomment the following scaffold marker; required to generate code for target CRD.
  	# - select:
  	# kind: CustomResourceDefinition
  	# name: firstmates.crew.testproject.org
  	# fieldPaths:
  	# - .metadata.annotations.[cert-manager.io/inject-ca-from]
  	# options:
  	# delimiter: '/'
  	# index: 1
  	# create: true
  	# +kubebuilder:scaffold:crdkustomizecainjectionname

• Main.go Configuration: The main.go file properly allows configuring certificates, metrics with TLS, and webhooks:
  

  kubebuilder/testdata/project-v4/cmd/main.go

  Lines 116 to 205 in 5d42475

  	// Initial webhook TLS options
  	webhookTLSOpts := tlsOpts
  	if len(webhookCertPath) > 0 {
  	setupLog.Info("Initializing webhook certificate watcher using provided certificates",
  	"webhook-cert-path", webhookCertPath, "webhook-cert-name", webhookCertName, "webhook-cert-key", webhookCertKey)
  	var err error
  	webhookCertWatcher, err = certwatcher.New(
  	filepath.Join(webhookCertPath, webhookCertName),
  	filepath.Join(webhookCertPath, webhookCertKey),
  	)
  	if err != nil {
  	setupLog.Error(err, "Failed to initialize webhook certificate watcher")
  	os.Exit(1)
  	}
  	webhookTLSOpts = append(webhookTLSOpts, func(config *tls.Config) {
  	config.GetCertificate = webhookCertWatcher.GetCertificate
  	})
  	}
  	webhookServer := webhook.NewServer(webhook.Options{
  	TLSOpts: webhookTLSOpts,
  	})
  	// Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
  	// More info:
  	// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.20.4/pkg/metrics/server
  	// - https://book.kubebuilder.io/reference/metrics.html
  	metricsServerOptions := metricsserver.Options{
  	BindAddress: metricsAddr,
  	SecureServing: secureMetrics,
  	TLSOpts: tlsOpts,
  	}
  	if secureMetrics {
  	// FilterProvider is used to protect the metrics endpoint with authn/authz.
  	// These configurations ensure that only authorized users and service accounts
  	// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
  	// https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.20.4/pkg/metrics/filters#WithAuthenticationAndAuthorization
  	metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
  	}
  	// If the certificate is not specified, controller-runtime will automatically
  	// generate self-signed certificates for the metrics server. While convenient for development and testing,
  	// this setup is not recommended for production.
  	//
  	// TODO(user): If you enable certManager, uncomment the following lines:
  	// - [METRICS-WITH-CERTS] at config/default/kustomization.yaml to generate and use certificates
  	// managed by cert-manager for the metrics server.
  	// - [PROMETHEUS-WITH-CERTS] at config/prometheus/kustomization.yaml for TLS certification.
  	if len(metricsCertPath) > 0 {
  	setupLog.Info("Initializing metrics certificate watcher using provided certificates",
  	"metrics-cert-path", metricsCertPath, "metrics-cert-name", metricsCertName, "metrics-cert-key", metricsCertKey)
  	var err error
  	metricsCertWatcher, err = certwatcher.New(
  	filepath.Join(metricsCertPath, metricsCertName),
  	filepath.Join(metricsCertPath, metricsCertKey),
  	)
  	if err != nil {
  	setupLog.Error(err, "to initialize metrics certificate watcher", "error", err)
  	os.Exit(1)
  	}
  	metricsServerOptions.TLSOpts = append(metricsServerOptions.TLSOpts, func(config *tls.Config) {
  	config.GetCertificate = metricsCertWatcher.GetCertificate
  	})
  	}
  	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
  	Scheme: scheme,
  	Metrics: metricsServerOptions,
  	WebhookServer: webhookServer,
  	HealthProbeBindAddress: probeAddr,
  	LeaderElection: enableLeaderElection,
  	LeaderElectionID: "da1d9c86.testproject.org",
  	// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
  	// when the Manager ends. This requires the binary to immediately end when the
  	// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
  	// speeds up voluntary leader transitions as the new leader don't have to wait
  	// LeaseDuration time first.
  	//
  	// In the default scaffold provided, the program ends immediately after
  	// the manager stops, so would be fine to enable this option. However,
  	// if you are doing or is intended to do any operation such as perform cleanups
  	// after the manager stops then its usage might be unsafe.
  	// LeaderElectionReleaseOnCancel: true,
  	})

• Kustomize Files: The configurations in the kustomize files have been updated accordingly:
  https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/default/kustomization.yaml#L36-L56

---

🛠️ Regarding Your Scenario

• You're correct that the conversion webhook should be reachable via the Kubernetes service, and the controller reaching out will expect HTTPS.
• The error message you're seeing — server gave HTTP response to HTTPS client — suggests that the webhook server is not serving TLS properly.
• Even if the caBundle is correctly set in the CRD, the conversion webhook itself must be served over HTTPS, with the correct certificates mounted and used in the manager setup.

However, it seems you're using an outdated scaffold where many fixes have already been introduced. To help us verify if there are any other outstanding scenarios that need to be addressed, could you please:

---

✅ Reproduce the Scenario with the Latest Scaffold

1. Create a New Project: Scaffold a new project using the latest version of Kubebuilder.
2. Enable Webhooks and Cert-Manager with TLS: Follow the guidance provided in the Kubebuilder Book to enable webhooks and cert-manager with TLS.
3. Test the Setup: Deploy the operator and test the conversion webhook functionality.

If you encounter any issues:

• Create a New Issue: Please create a new issue in the Kubebuilder GitHub repository.
• Provide Details: Include all steps to reproduce the scenario, what you expected, and what you observed instead.

Thank you for your understanding and assistance in helping us provide a better solution for the entire community.

[camilamacedo86]

Here you can find a more detailed information as well to allow you check it: #4855

[camilamacedo86]

Closing as sorted out, if you need feel free to re-open.