Mega Feature: Refactor KRO to use server side apply as the default

[barney-s]

Feature Description

Problem Statement:

1. To avoid handling diffing logic in KRO
2. Fix issues like Handle Secrets that are transformed by server and hence show up as diff #299 , I have Kubernetes Secrets correctly created, but seing with delta by kro #272
3. Handle resources mutated by webhooks without triggering continuous reconciliation by KRO due to diffs seen
4. Fix issues like Handle immutable fields during apply #541 where we are able to update resources with immutable fields (with changes not in immutable fields)
5. ...

Proposed Solution:

Using server side apply in instance reconciler:

1. To let server determine diffs
2. Work nice with mutating webhooks that modify applied resources
3. Allow updates to objects with immutable fields

Consider using using applyset with ssa along with #557

Alternatives Considered:

1. 3 way merge
2. Structured merge diff

• Please vote on this issue by adding a 👍 reaction to the original issue
• If you are interested in working on this feature, please leave a comment

[fabianburth]

I played around with your implementation in #576.

In case you are interested, here's some context:
I applied an RGD with a pod specification, created an instance of that RGD, changed the container image in the pod resource, and expected that the microcontroller would change the container image back to the image specified in the RGD. Instead, I ran into issues with updating immutable fields. After some debugging, I thought the diffing and update logic in pkg/controller/instance/controller_reconcile.go updateResource(...) might be what's causing the issue.
That's how I found your issue here.

While thinking about the server side apply approach in this context, I was wondering whether this is sufficient for drift detection / correction. In particular, I'm concerned about https://kubernetes.io/docs/reference/using-api/server-side-apply/#using-server-side-apply-in-a-controller

• there is no way to remove fields that haven't been applied by the controller before (controller can still send a patch or update for these use-cases).

Would be happy to hear your opinion!

cc @jakobmoellerdev