Skip to content

OraOperator v1.2.0 oracle-database-operator-controller-manager pods crashes #183

@andbos

Description

@andbos

OraOperator v1.2.0 oracle-database-operator-controller-manager crashes. Didn't have any similar problems before (v1.1.0).

AWS EKS v1.29.3.

Error:
2025-07-03T09:38:15Z ERROR controllers.database.SingleInstanceDatabase admission webhook "vsingleinstancedatabase.kb.io" denied the request: SingleInstanceDatabase.database.oracle.com "sidb11" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed] {"manageSingleInstanceDatabaseDeletion": {"name":"sidb11","namespace":"oracle-database"}, "error": "admission webhook "vsingleinstancedatabase.kb.io" denied the request: SingleInstanceDatabase.database.oracle.com "sidb11" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed]"}

2025-07-03T09:38:15Z    ERROR   controllers.database.SingleInstanceDatabase     admission webhook "vsingleinstancedatabase.kb.io" denied the request: SingleInstanceDatabase.database.oracle.com "sidb11" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed]  {"manageSingleInstanceDatabaseDeletion": {"name":"sidb11","namespace":"oracle-database"}, "error": "admission webhook \"vsingleinstancedatabase.kb.io\" denied the request: SingleInstanceDatabase.database.oracle.com \"sidb11\" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed]"}
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).manageSingleInstanceDatabaseDeletion
        /workspace/controllers/database/singleinstancedatabase_controller.go:3151
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).Reconcile
        /workspace/controllers/database/singleinstancedatabase_controller.go:158
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
        /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224
2025-07-03T09:38:15Z    INFO    controllers.database.SingleInstanceDatabase     Reconcile queued
E0703 09:38:15.257430       1 panic.go:262] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<
        goroutine 972 [running]:
        k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2f80748, 0x45dcfa0}, {0x25978e0, 0x457aac0})
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/runtime/runtime.go:107 +0xbc
        k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2f80748, 0x45dcfa0}, {0x25978e0, 0x457aac0}, {0x45dcfa0, 0x0, 0x43aaa5?})
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/runtime/runtime.go:82 +0x5e
        k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc0e00?})
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/runtime/runtime.go:59 +0x108
        panic({0x25978e0?, 0x457aac0?})
                /usr/local/go/src/runtime/panic.go:785 +0x132
        github.com/oracle/oracle-database-operator/controllers/database.(*ShardingDatabaseReconciler).SetupWithManager.(*ShardingDatabaseReconciler).eventFilterPredicate.func2({{0x2faaf78?, 0xc00084c8c0?}, {0x2faaf78?, 0xc0002dbb80?}})
                /workspace/controllers/database/shardingdatabase_controller.go:462 +0xb5
        sigs.k8s.io/controller-runtime/pkg/predicate.TypedFuncs[...].Update(...)
                /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/predicate/predicate.go:95
        sigs.k8s.io/controller-runtime/pkg/internal/source.(*EventHandler[...]).OnUpdate(0x2f92600, {0x29d1760?, 0xc00084c8c0}, {0x29d1760, 0xc0002dbb80})
                /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/event_handler.go:117 +0x3ba
        k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
                /root/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:253
        k8s.io/client-go/tools/cache.(*processorListener).run.func1()
                /root/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:976 +0xea
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:226 +0x33
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0004f2f70, {0x2f5ba00, 0xc0012940c0}, 0x1, 0xc000812fc0)
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:227 +0xaf
        k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000afef70, 0x3b9aca00, 0x0, 0x1, 0xc000812fc0)
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:204 +0x7f
        k8s.io/apimachinery/pkg/util/wait.Until(...)
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:161
        k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000e2090)
                /root/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:972 +0x5a
        k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:72 +0x4c
        created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 375
                /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:70 +0x73
 >
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x21de475]

goroutine 972 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2f80748, 0x45dcfa0}, {0x25978e0, 0x457aac0}, {0x45dcfa0, 0x0, 0x43aaa5?})
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc0e00?})
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/runtime/runtime.go:59 +0x108
panic({0x25978e0?, 0x457aac0?})
        /usr/local/go/src/runtime/panic.go:785 +0x132
github.com/oracle/oracle-database-operator/controllers/database.(*ShardingDatabaseReconciler).SetupWithManager.(*ShardingDatabaseReconciler).eventFilterPredicate.func2({{0x2faaf78?, 0xc00084c8c0?}, {0x2faaf78?, 0xc0002dbb80?}})
        /workspace/controllers/database/shardingdatabase_controller.go:462 +0xb5
sigs.k8s.io/controller-runtime/pkg/predicate.TypedFuncs[...].Update(...)
        /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/predicate/predicate.go:95
sigs.k8s.io/controller-runtime/pkg/internal/source.(*EventHandler[...]).OnUpdate(0x2f92600, {0x29d1760?, 0xc00084c8c0}, {0x29d1760, 0xc0002dbb80})
        /root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/source/event_handler.go:117 +0x3ba
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
        /root/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/controller.go:253
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /root/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:976 +0xea
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000aabf70, {0x2f5ba00, 0xc0012940c0}, 0x1, 0xc000812fc0)
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000afef70, 0x3b9aca00, 0x0, 0x1, 0xc000812fc0)
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000e2090)
        /root/go/pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/shared_informer.go:972 +0x5a
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 375
        /root/go/pkg/mod/k8s.io/apimachinery@v0.31.3/pkg/util/wait/wait.go:70 +0x73
$ kubectl -n oracle-database-operator-system get pods
NAME                                                           READY   STATUS             RESTARTS      AGE
oracle-database-operator-controller-manager-59c5c555d6-jncvj   0/1     Error              4 (78s ago)   4m35s
oracle-database-operator-controller-manager-59c5c555d6-m8vb4   0/1     CrashLoopBackOff   4 (44s ago)   4m35s
oracle-database-operator-controller-manager-59c5c555d6-q4hnx   0/1     CrashLoopBackOff   4 (28s ago)   4m35s

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions