Activating operator.networkpolicy breaks the operator on Azure AKS #4411
Description
Component(s)
operator
What happened?
Description
The new support for network policies (when activated) is causing the operator to crash and not work. There is no issue with the operand.networkpolicy flag, which correctly creates the network policy resources for the collectors as expected. However the operator.networkpolicy flag is not working.
I discovered this while working on the helm chart support for this feature (open-telemetry/opentelemetry-helm-charts#1870). I first mentioned the issue on slack to see if this is a k8s vs openshift issue (https://cloud-native.slack.com/archives/C033BJ8BASU/p1759490447012719). Furthermore there's missing documentation that a NAMESPACE env var is required for these features.
Steps to Reproduce
- Add a
NAMESPACEenv variable so the operator knows its own ns. - Add
--feature-gates=operator.networkpolicyas a container args.
Expected Result
A network policy resource for the operator would be created and the operator would function as intended.
Actual Result
Crash loop
Kubernetes Version
1.32.7
Operator version
0.136.0
Collector version
0.136.0
Environment information
Environment
Azure AKS on Ubuntu nodes
Log output
2025-10-03T11:18:38.408830851Z {"level":"INFO","timestamp":"2025-10-03T11:18:38Z","message":"Starting the OpenTelemetry Operator","opentelemetry-operator":"0.136.0","build-date":"2025-09-30T07:12:26Z","go-version":"go1.24.7","go-arch":"amd64","go-os":"linux","feature-gates":"operand.networkpolicy,operator.collector.default.config,operator.collector.targetallocatorcr,-operator.golang.flags,operator.networkpolicy,operator.sidecarcontainers.native,-operator.targetallocator.fallbackstrategy,-operator.targetallocator.mtls","config":{"auto-instrumentation-apache-httpd-image":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.4","auto-instrumentation-dot-net-image":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.2.0","auto-instrumentation-go-image":"ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.22.1","auto-instrumentation-java-image":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.33.6","auto-instrumentation-nginx-image":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.4","auto-instrumentation-node-js-image":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.64.1","auto-instrumentation-python-image":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.58b0","cert-manager-availability":"0","collector-availability":"0","collector-configmap-entry":"collector.yaml","collector-image":"ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s:0.136.0","create-rbac-permissions":"0","create-service-monitor-operator-metrics":"false","enable-apache-httpd-instrumentation":"true","enable-cr-metrics":"false","enable-dot-net-auto-instrumentation":"true","enable-go-auto-instrumentation":"false","enable-java-auto-instrumentation":"true","enable-leader-election":"true","enable-multi-instrumentation":"true","enable-nginx-auto-instrumentation":"false","enable-node-js-auto-instrumentation":"true","enable-python-auto-instrumentation":"true","enable-webhooks":"true","fips-disabled-components":"uppercase","health-probe-addr":":8081","ignore-missing-collector-crds":"false","metrics-addr":"0.0.0.0:8080","opampbridge-availability":"0","open-shift-routes-availability":"1","openshift-create-dashboard":"false","operator-op-amp-bridge-configmap-entry":"remoteconfiguration.yaml","operatoropampbridge-image":"ghcr.io/open-telemetry/opentelemetry-operator/operator-opamp-bridge:0.136.0","pprof-addr":"","prometheus-cr-availability":"0","target-allocator-availability":"0","target-allocator-configmap-entry":"targetallocator.yaml","targetallocator-image":"ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.136.0","webhook-port":"9443"}}
2025-10-03T11:18:38.408871651Z {"level":"INFO","timestamp":"2025-10-03T11:18:38Z","logger":"setup","message":"the env var WATCH_NAMESPACE isn't set, watching all namespaces"}
2025-10-03T11:18:39.812435594Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"Prometheus CRDs are installed, adding to scheme."}
2025-10-03T11:18:39.812508794Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"Openshift CRDs are not installed, skipping adding to scheme."}
2025-10-03T11:18:39.812512094Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"Cert-Manager is available to the operator, adding to scheme."}
2025-10-03T11:18:39.812592094Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"OpenTelemetryCollectorCRDSs are available to the operator"}
2025-10-03T11:18:39.812638194Z {"level":"DPANIC","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"odd number of arguments passed as key-value pairs for logging","ignored key":true,"stacktrace":"main.main\n\t./main.go:253\nruntime.main\n\truntime/proc.go:283"}
2025-10-03T11:18:39.812642094Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"Native sidecar"}
2025-10-03T11:18:39.827376540Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a mutating webhook","GVK":"opentelemetry.io/v1beta1, Kind=OpenTelemetryCollector","path":"/mutate-opentelemetry-io-v1beta1-opentelemetrycollector"}
2025-10-03T11:18:39.827442640Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/mutate-opentelemetry-io-v1beta1-opentelemetrycollector"}
2025-10-03T11:18:39.827465440Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a validating webhook","GVK":"opentelemetry.io/v1beta1, Kind=OpenTelemetryCollector","path":"/validate-opentelemetry-io-v1beta1-opentelemetrycollector"}
2025-10-03T11:18:39.827533640Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/validate-opentelemetry-io-v1beta1-opentelemetrycollector"}
2025-10-03T11:18:39.827641140Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/convert"}
2025-10-03T11:18:39.827646340Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Conversion webhook enabled","GVK":"opentelemetry.io/v1beta1, Kind=OpenTelemetryCollector"}
2025-10-03T11:18:39.827654039Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a mutating webhook","GVK":"opentelemetry.io/v1alpha1, Kind=TargetAllocator","path":"/mutate-opentelemetry-io-v1alpha1-targetallocator"}
2025-10-03T11:18:39.827724439Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/mutate-opentelemetry-io-v1alpha1-targetallocator"}
2025-10-03T11:18:39.827788439Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a validating webhook","GVK":"opentelemetry.io/v1alpha1, Kind=TargetAllocator","path":"/validate-opentelemetry-io-v1alpha1-targetallocator"}
2025-10-03T11:18:39.827801039Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/validate-opentelemetry-io-v1alpha1-targetallocator"}
2025-10-03T11:18:39.827853739Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a mutating webhook","GVK":"opentelemetry.io/v1alpha1, Kind=Instrumentation","path":"/mutate-opentelemetry-io-v1alpha1-instrumentation"}
2025-10-03T11:18:39.827870239Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/mutate-opentelemetry-io-v1alpha1-instrumentation"}
2025-10-03T11:18:39.827916639Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a validating webhook","GVK":"opentelemetry.io/v1alpha1, Kind=Instrumentation","path":"/validate-opentelemetry-io-v1alpha1-instrumentation"}
2025-10-03T11:18:39.827922539Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/validate-opentelemetry-io-v1alpha1-instrumentation"}
2025-10-03T11:18:39.827975138Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/mutate-v1-pod"}
2025-10-03T11:18:39.827979238Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a mutating webhook","GVK":"opentelemetry.io/v1alpha1, Kind=OpAMPBridge","path":"/mutate-opentelemetry-io-v1alpha1-opampbridge"}
2025-10-03T11:18:39.828026538Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/mutate-opentelemetry-io-v1alpha1-opampbridge"}
2025-10-03T11:18:39.828041638Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.builder","message":"Registering a validating webhook","GVK":"opentelemetry.io/v1alpha1, Kind=OpAMPBridge","path":"/validate-opentelemetry-io-v1alpha1-opampbridge"}
2025-10-03T11:18:39.828048938Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Registering webhook","path":"/validate-opentelemetry-io-v1alpha1-opampbridge"}
2025-10-03T11:18:39.828091838Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"setup","message":"starting manager"}
2025-10-03T11:18:39.828174338Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.metrics","message":"Starting metrics server"}
2025-10-03T11:18:39.828197538Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","message":"starting server","name":"health probe","addr":"[::]:8081"}
2025-10-03T11:18:39.828209537Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.metrics","message":"Serving metrics server","bindAddress":"0.0.0.0:8080","secure":false}
2025-10-03T11:18:39.828261937Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Starting webhook server"}
2025-10-03T11:18:39.829211034Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.certwatcher","message":"Updated current TLS certificate"}
2025-10-03T11:18:39.829288034Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.webhook","message":"Serving webhook server","host":"","port":9443}
2025-10-03T11:18:39.829419233Z {"level":"INFO","timestamp":"2025-10-03T11:18:39Z","logger":"controller-runtime.certwatcher","message":"Starting certificate poll+watcher","interval":10}
2025-10-03T11:18:40.628826453Z I1003 11:18:40.628720 1 leaderelection.go:257] attempting to acquire leader lease otel-operator/9f7554c3.opentelemetry.io...
2025-10-03T11:19:28.211822017Z I1003 11:19:28.211707 1 leaderelection.go:271] successfully acquired lease otel-operator/9f7554c3.opentelemetry.io
2025-10-03T11:19:28.212062316Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","logger":"instrumentation-upgrade","message":"looking for managed Instrumentation instances to upgrade"}
2025-10-03T11:19:28.212426315Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.Deployment"}
2025-10-03T11:19:28.212443815Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1beta1.OpenTelemetryCollector"}
2025-10-03T11:19:28.212450015Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.DaemonSet"}
2025-10-03T11:19:28.212454115Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1alpha1.OpAMPBridge"}
2025-10-03T11:19:28.212458515Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1alpha1.TargetAllocator"}
2025-10-03T11:19:28.212463115Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1alpha1.TargetAllocator"}
2025-10-03T11:19:28.212480315Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.ConfigMap"}
2025-10-03T11:19:28.212485215Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.Service"}
2025-10-03T11:19:28.212489515Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.ServiceAccount"}
2025-10-03T11:19:28.212493715Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.Deployment"}
2025-10-03T11:19:28.212497815Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.ConfigMap"}
2025-10-03T11:19:28.212501915Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.StatefulSet"}
2025-10-03T11:19:28.212506115Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.Service"}
2025-10-03T11:19:28.212510315Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.ServiceMonitor"}
2025-10-03T11:19:28.212514315Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.PodMonitor"}
2025-10-03T11:19:28.212518315Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.Deployment"}
2025-10-03T11:19:28.212522615Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.PodDisruptionBudget"}
2025-10-03T11:19:28.212533315Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.ConfigMap"}
2025-10-03T11:19:28.212537815Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.NetworkPolicy"}
2025-10-03T11:19:28.212542415Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.ServiceAccount"}
2025-10-03T11:19:28.212546715Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.ServiceMonitor"}
2025-10-03T11:19:28.212556215Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.Service"}
2025-10-03T11:19:28.212560215Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v2.HorizontalPodAutoscaler"}
2025-10-03T11:19:28.212564515Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.PodDisruptionBudget"}
2025-10-03T11:19:28.212568715Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1beta1.OpenTelemetryCollector"}
2025-10-03T11:19:28.212572915Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.ServiceAccount"}
2025-10-03T11:19:28.212577015Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.Ingress"}
2025-10-03T11:19:28.212848114Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1.PodMonitor"}
2025-10-03T11:19:28.212867914Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting EventSource","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","source":"kind source: *v1beta1.OpenTelemetryCollector"}
2025-10-03T11:19:28.303066088Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Stopping and waiting for non leader election runnables"}
2025-10-03T11:19:28.303108288Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Stopping and waiting for leader election runnables"}
2025-10-03T11:19:28.303114788Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controller-runtime.source.EventHandler","message":"failed to get informer from cache","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:76\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:53\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:54\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:64"}
2025-10-03T11:19:28.303118588Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controller-runtime.source.EventHandler","message":"failed to get informer from cache","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:76\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:53\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:54\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:64"}
2025-10-03T11:19:28.303132788Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting Controller","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge"}
2025-10-03T11:19:28.303171688Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting Controller","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector"}
2025-10-03T11:19:28.303351687Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting workers","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","worker count":1}
2025-10-03T11:19:28.303357787Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Shutdown signal received, waiting for all workers to finish","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge"}
2025-10-03T11:19:28.303362387Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting workers","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","worker count":1}
2025-10-03T11:19:28.303376387Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Shutdown signal received, waiting for all workers to finish","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector"}
2025-10-03T11:19:28.303394887Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controller-runtime.source.EventHandler","message":"failed to get informer from cache","error":"Timeout: failed waiting for *v1alpha1.OpAMPBridge Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:76\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:53\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:54\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:64"}
2025-10-03T11:19:28.303398387Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"All workers finished","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge"}
2025-10-03T11:19:28.303404087Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controller-runtime.source.EventHandler","message":"failed to get informer from cache","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:76\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:53\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/loop.go:54\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/source/kind.go:64"}
2025-10-03T11:19:28.303406887Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting Controller","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator"}
2025-10-03T11:19:28.303414387Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Starting workers","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator","worker count":1}
2025-10-03T11:19:28.303417187Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Shutdown signal received, waiting for all workers to finish","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator"}
2025-10-03T11:19:28.303419987Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"All workers finished","controller":"targetallocator","controllerGroup":"opentelemetry.io","controllerKind":"TargetAllocator"}
2025-10-03T11:19:28.303432887Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","message":"error received after stop sequence was engaged","error":"failed to list: Timeout: failed waiting for *v1alpha1.Instrumentation Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:512"}
2025-10-03T11:19:28.303441787Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controllers.OpenTelemetryCollector","message":"unable to fetch OpenTelemetryCollector","opentelemetrycollector":{"name":"k8sevents","namespace":"monitoring"},"error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\tgithub.com/open-telemetry/opentelemetry-operator/internal/controllers/opentelemetrycollector_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303444887Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","message":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"k8sevents","namespace":"monitoring"},"namespace":"monitoring","name":"k8sevents","reconcileID":"21244f18-5f4e-4973-aace-0ae765f2d2be","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303592086Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controllers.OpenTelemetryCollector","message":"unable to fetch OpenTelemetryCollector","opentelemetrycollector":{"name":"frontend","namespace":"monitoring"},"error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\tgithub.com/open-telemetry/opentelemetry-operator/internal/controllers/opentelemetrycollector_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303616986Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","message":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"frontend","namespace":"monitoring"},"namespace":"monitoring","name":"frontend","reconcileID":"78785e4b-b572-488b-87c0-d899c754f91e","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303620286Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controllers.OpenTelemetryCollector","message":"unable to fetch OpenTelemetryCollector","opentelemetrycollector":{"name":"logs","namespace":"monitoring"},"error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\tgithub.com/open-telemetry/opentelemetry-operator/internal/controllers/opentelemetrycollector_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303624886Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","message":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"logs","namespace":"monitoring"},"namespace":"monitoring","name":"logs","reconcileID":"7565c996-0f14-4237-8b65-7c45763e5a46","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303632786Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controllers.OpenTelemetryCollector","message":"unable to fetch OpenTelemetryCollector","opentelemetrycollector":{"name":"springapps","namespace":"monitoring"},"error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\tgithub.com/open-telemetry/opentelemetry-operator/internal/controllers/opentelemetrycollector_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303649086Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","message":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"springapps","namespace":"monitoring"},"namespace":"monitoring","name":"springapps","reconcileID":"bf9b034f-1cf1-42ff-aaae-853d388befb4","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303691186Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"controllers.OpenTelemetryCollector","message":"unable to fetch OpenTelemetryCollector","opentelemetrycollector":{"name":"waf-event-hub","namespace":"monitoring"},"error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"github.com/open-telemetry/opentelemetry-operator/internal/controllers.(*OpenTelemetryCollectorReconciler).Reconcile\n\tgithub.com/open-telemetry/opentelemetry-operator/internal/controllers/opentelemetrycollector_controller.go:240\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303698886Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","message":"Reconciler error","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"waf-event-hub","namespace":"monitoring"},"namespace":"monitoring","name":"waf-event-hub","reconcileID":"e3315a66-ddc2-463f-989a-70089e3f13e4","error":"Timeout: failed waiting for *v1beta1.OpenTelemetryCollector Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
2025-10-03T11:19:28.303709286Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"All workers finished","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector"}
2025-10-03T11:19:28.303711986Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Stopping and waiting for caches"}
2025-10-03T11:19:28.303849285Z W1003 11:19:28.303778 1 reflector.go:492] k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: watch of *v1alpha1.Instrumentation ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2025-10-03T11:19:28.304479183Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Stopping and waiting for webhooks"}
2025-10-03T11:19:28.304841282Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","logger":"controller-runtime.webhook","message":"Shutting down webhook server with timeout of 1 minute"}
2025-10-03T11:19:28.304857382Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Stopping and waiting for HTTP servers"}
2025-10-03T11:19:28.304861682Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","logger":"controller-runtime.metrics","message":"Shutting down metrics server with timeout of 1 minute"}
2025-10-03T11:19:28.304866082Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"shutting down server","name":"health probe","addr":"[::]:8081"}
2025-10-03T11:19:28.304870182Z {"level":"INFO","timestamp":"2025-10-03T11:19:28Z","message":"Wait completed, proceeding to shutdown the manager"}
2025-10-03T11:19:28.404137424Z {"level":"ERROR","timestamp":"2025-10-03T11:19:28Z","logger":"setup","message":"problem running manager","error":"deployments.apps \"opentelemetry-operator-controller-manager\" not found","stacktrace":"main.main\n\t./main.go:420\nruntime.main\n\truntime/proc.go:283"}Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.