How is webhook bootstrapping supposed to work on upgrade?

[abryant-hv]

We're starting to test more with multiple CRD versions, and we're running into odd behavior, and I can't tell how it was ever supposed to work.

We have several alpha versions of our objects, and we create new versions when we make changes. We have conversion webhooks set up between the different versions (using the hub and spoke model) and they work great on a fresh install. As far as I can tell, we've followed the "Multi-Version API" tutorial exactly. But, when we push updated CRDs and operator deployments (with make install, make deploy) to a cluster that has existing objects, the operator fails to start.

The operator logs indicate that the caches can't be initialized because the operator can't load the objects because the API server can't reach the conversion webhook.

e.g. E1010 15:34:50.452345 3573658 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:137: Failed to watch *v2alpha7.Foo: failed to list *v2alpha7.Foo: conversion webhook for product/v2alpha6, Kind=Foo failed: Post "https://foo-operator-webhook-service.foo-operator-system.svc:443/convert?timeout=30s": dial tcp 10.96.82.112:443: connect: connection refused

But the conversion webhook is in the operator, which can't start!

How is this supposed to work?

[abryant-hv]

Update: this seems to have been caused by an out of date version. We were using controller-runtime 0.8.3, which is quite old, now. Updating to controller-runtime 0.12.3 resolved this issue.

[camilamacedo86]

Closing since it is answered.