Openapi operator send UpdateNotification instead of AddNotifications on operator startup

[etremblay]

Expected Behavior

When the operator start, all reconciled resources should be considered "new" and the onAddFilter should be applied before entering the reconcile() function.

This was the cas with the kubernetes sdk implementation but not the openapi implementation.

Actual Behaviour

The reconcile() method is never called on initial resources because we have a onUpdateFilter that check for generation changes to allow updates.

With a lot of debugging, I think I can explain the behaviour.

• InformerWatcher call the list api and replaceObjectsAndStartWatcher()
• ControllerResourceEventHandler::onAdd() is called on all resources but do nothing since it's disabled.
• The operator win the leader election and the controller is enabled
• DefaultSharedIndexInformer resync the listeners but since the objects already exists in the Indexer, the DeltaConsumer triger an UpdateNotification event instead of AddNotification.

Steps To Reproduce

Start an operator with pre-existing custom resources

Environment Information

No response

Example Application

No response

Version

4.9.2

[msupic]

Could you please give me more details what should be expected behaviour when there are two or more service instances? If the service instance which wins the leader election triggers the 'add' notification, what should happen when that service instances is not leader anymore, should another instance trigger the 'add' notification again after becoming the leader? In this case, each changing of leader will trigger 'add' notifications for all resources from the informer cache. Also, the (not restarted) instance that was leader before can become the leader again after some time. Should 'add' notification be triggered only when a new resource is created?

[etremblay]

That's a good question. We are not in production yet and always have only one instance running.

I'm pretty new to all of this but my guess would be that each instance is completly independant regarding it's initialization.

Let's say we have two instances A and B

A : Cold start fetch all resource from api
A : Add events skipped because election not won yet
A: Win election
A: Should reprocess initially loaded resources as Add Event
B: Cold start fetch all resource from api
B: Add events skipped because election not won
B: Recieve more event (maybe update, delete, ...)
A: Shutdown
B: Win election
B: Should reprocess initially loaded resources as Add Event and all other accumulated events.

Since the reconcile is idempotent, I think Add and Update events sould be processed the same way (and reconciled with the real backend resource).

The only difference I can see is the the onAddFilter and onUpdateFilter than can be different.

Maybe we do not use it for the intended purpose. My workaround is to process all Update event, event if the oldObject equals to the newObject.