|
1 | 1 | # Watching Resources |
2 | 2 |
|
3 | | -Inside a `Reconcile()` control loop, you are looking to do a collection of operations until it has the desired state on the cluster. |
4 | | -Therefore, it can be necessary to know when a resource that you care about is changed. |
5 | | -In the case that there is an action (create, update, edit, delete, etc.) on a watched resource, `Reconcile()` should be called for the resources watching it. |
| 3 | +When extending the Kubernetes API, we aim to ensure that our solutions behave consistently with Kubernetes itself. |
| 4 | +For example, consider a `Deployment` resource, which is managed by a controller. This controller is responsible |
| 5 | +for responding to changes in the cluster—such as when a `Deployment` is created, updated, or deleted—by triggering |
| 6 | +reconciliation to ensure the resource’s state matches the desired state. |
6 | 7 |
|
7 | | -[Controller Runtime libraries](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/builder) provide many ways for resources to be managed and watched. |
8 | | -This ranges from the easy and obvious use cases, such as watching the resources which were created and managed by the controller, to more unique and advanced use cases. |
| 8 | +Similarly, when developing our controllers, we want to watch for relevant changes in resources that are crucial |
| 9 | +to our solution. These changes—whether creations, updates, or deletions—should trigger the reconciliation |
| 10 | +loop to take appropriate actions and maintain consistency across the cluster. |
9 | 11 |
|
10 | | -See each subsection for explanations and examples of the different ways in which your controller can _Watch_ the resources it cares about. |
| 12 | +The [controller-runtime][controller-runtime] library provides several ways to watch and manage resources. |
11 | 13 |
|
12 | | -- [Watching Operator Managed Resources](watching-resources/operator-managed.md) - |
13 | | - These resources are created and managed by the same operator as the resource watching them. |
14 | | - This section covers both if they are managed by the same controller or separate controllers. |
15 | | -- [Watching Externally Managed Resources](watching-resources/externally-managed.md) - |
16 | | - These resources could be manually created, or managed by other operators/controllers or the Kubernetes control plane. |
| 14 | +## Primary Resources |
| 15 | + |
| 16 | +The **Primary Resource** is the resource that your controller is responsible |
| 17 | +for managing. For example, if you create a custom resource definition (CRD) for `MyApp`, |
| 18 | +the corresponding controller is responsible for managing instances of `MyApp`. |
| 19 | + |
| 20 | +In this case, `MyApp` is the **Primary Resource** for that controller, and your controller’s |
| 21 | +reconciliation loop focuses on ensuring the desired state of these primary resources is maintained. |
| 22 | + |
| 23 | +When you create a new API using Kubebuilder, the following default code is scaffolded, |
| 24 | +ensuring that the controller watches all relevant events—such as creations, updates, and |
| 25 | +deletions—for (`For()`) the new API. |
| 26 | + |
| 27 | +This setup guarantees that the reconciliation loop is triggered whenever an instance |
| 28 | +of the API is created, updated, or deleted: |
| 29 | + |
| 30 | +```go |
| 31 | +// Watches the primary resource (e.g., MyApp) for create, update, delete events |
| 32 | +if err := ctrl.NewControllerManagedBy(mgr). |
| 33 | + For(&<YourAPISpec>{}). <-- See there that the Controller is For this API |
| 34 | + Complete(r); err != nil { |
| 35 | + return err |
| 36 | +} |
| 37 | +``` |
| 38 | + |
| 39 | +## Secondary Resources |
| 40 | + |
| 41 | +Your controller will likely also need to manage **Secondary Resources**, |
| 42 | +which are the resources required on the cluster to support the **Primary Resource**. |
| 43 | + |
| 44 | +Changes to these **Secondary Resources** can directly impact the **Primary Resource**, |
| 45 | +so the controller must watch and reconcile these resources accordingly. |
| 46 | + |
| 47 | +### Which are Owned by the Controller |
| 48 | + |
| 49 | +These **Secondary Resources**, such as `Services`, `ConfigMaps`, or `Deployments`, |
| 50 | +when `Owned` by the controllers, are created and managed by the specific controller |
| 51 | +and are tied to the **Primary Resource** via [OwnerReferences][owner-ref-k8s-docs]. |
| 52 | + |
| 53 | +For example, if we have a controller to manage our CR(s) of the Kind `MyApp` |
| 54 | +on the cluster, which represents our application solution, all resources required |
| 55 | +to ensure that `MyApp` is up and running with the desired number of instances |
| 56 | +will be **Secondary Resources**. The code responsible for creating, deleting, |
| 57 | +and updating these resources will be part of the `MyApp` Controller. |
| 58 | +We would add the appropriate [OwnerReferences][owner-ref-k8s-docs] |
| 59 | +using the [controllerutil.SetControllerReference][cr-owner-ref-doc] |
| 60 | +function to indicate that these resources are owned by the same controller |
| 61 | +responsible for managing `MyApp` instances, which will be reconciled by the `MyAppReconciler`. |
| 62 | + |
| 63 | +Additionally, if the **Primary Resource** is deleted, Kubernetes' garbage collection mechanism |
| 64 | +ensures that all associated **Secondary Resources** are automatically deleted in a |
| 65 | +cascading manner. |
| 66 | + |
| 67 | +### Which are NOT `Owned` by the Controller |
| 68 | + |
| 69 | +Note that **Secondary Resources** can either be APIs/CRDs defined in your project or in other projects that are |
| 70 | +relevant to the **Primary Resources**, but which the specific controller is not responsible for creating or managing. |
| 71 | + |
| 72 | +For example, if we have a CRD that represents a backup solution (i.e. `MyBackup`) for our `MyApp`, |
| 73 | +it might need to watch changes in the `MyApp` resource to trigger reconciliation in `MyBackup` |
| 74 | +to ensure the desired state. Similarly, `MyApp`'s behavior might also be impacted by |
| 75 | +CRDs/APIs defined in other projects. |
| 76 | + |
| 77 | +In both scenarios, these resources are treated as **Secondary Resources**, even if they are not `Owned` |
| 78 | +(i.e., not created or managed) by the `MyAppController`. |
| 79 | + |
| 80 | +In Kubebuilder, resources that are not defined in the project itself and are not |
| 81 | +a **Core Type** (those not defined in the Kubernetes API) are called **External Types**. |
| 82 | + |
| 83 | +An **External Type** refers to a resource that is not defined in your |
| 84 | +project but one that you need to watch and respond to. |
| 85 | +For example, if **Operator A** manages a `MyApp` CRD for application deployment, |
| 86 | +and **Operator B** handles backups, **Operator B** can watch the `MyApp` CRD as an external type |
| 87 | +to trigger backup operations based on changes in `MyApp`. |
| 88 | + |
| 89 | +In this scenario, **Operator B** could define a `BackupConfig` CRD that relies on the state of `MyApp`. |
| 90 | +By treating `MyApp` as a **Secondary Resource**, **Operator B** can watch and reconcile changes in **Operator A**'s `MyApp`, |
| 91 | +ensuring that backup processes are initiated whenever `MyApp` is updated or scaled. |
| 92 | + |
| 93 | +## General Concept of Watching Resources |
| 94 | + |
| 95 | +Whether a resource is defined within your project or comes from an external project, the concept of **Primary** |
| 96 | +and **Secondary Resources** remains the same: |
| 97 | +- The **Primary Resource** is the resource the controller is primarily responsible for managing. |
| 98 | +- **Secondary Resources** are those that are required to ensure the primary resource works as desired. |
| 99 | + |
| 100 | +Therefore, regardless of whether the resource was defined by your project or by another project, |
| 101 | +your controller can watch, reconcile, and manage changes to these resources as needed. |
| 102 | + |
| 103 | +## Why does watching the secondary resources matter? |
| 104 | + |
| 105 | +When building a Kubernetes controller, it’s crucial to not only focus |
| 106 | +on **Primary Resources** but also to monitor **Secondary Resources**. |
| 107 | +Failing to track these resources can lead to inconsistencies in your |
| 108 | +controller's behavior and the overall cluster state. |
| 109 | + |
| 110 | +Secondary resources may not be directly managed by your controller, |
| 111 | +but changes to these resources can still significantly |
| 112 | +impact the primary resource and your controller's functionality. |
| 113 | +Here are the key reasons why it's important to watch them: |
| 114 | + |
| 115 | +- **Ensuring Consistency**: |
| 116 | + - Secondary resources (e.g., child objects or external dependencies) may diverge from their desired state. |
| 117 | + For instance, a secondary resource may be modified or deleted, causing the system to fall out of sync. |
| 118 | + - Watching secondary resources ensures that any changes are detected immediately, allowing the controller to |
| 119 | + reconcile and restore the desired state. |
| 120 | + |
| 121 | +- **Avoiding Random Self-Healing**: |
| 122 | + - Without watching secondary resources, the controller may "heal" itself only upon restart or when specific events |
| 123 | + are triggered. This can cause unpredictable or delayed reactions to issues. |
| 124 | + - Monitoring secondary resources ensures that inconsistencies are addressed promptly, rather than waiting for a |
| 125 | + controller restart or external event to trigger reconciliation. |
| 126 | + |
| 127 | +- **Effective Lifecycle Management**: |
| 128 | + - Secondary resources might not be owned by the controller directly, but their state still impacts the behavior |
| 129 | + of primary resources. Without watching these, you risk leaving orphaned or outdated resources. |
| 130 | + - Watching non-owned secondary resources lets the controller respond to lifecycle events (create, update, delete) |
| 131 | + that might affect the primary resource, ensuring consistent behavior across the system. |
| 132 | + |
| 133 | +## Why not use `RequeueAfter X` for all scenarios instead of watching resources? |
| 134 | + |
| 135 | +Kubernetes controllers are fundamentally **event-driven**. When creating a controller, |
| 136 | +the **Reconciliation Loop** is typically triggered by **events** such as `create`, `update`, or |
| 137 | +`delete` actions on resources. This event-driven approach is more efficient and responsive |
| 138 | +compared to constantly requeuing or polling resources using `RequeueAfter`. This ensures that |
| 139 | +the system only takes action when necessary, maintaining both performance and efficiency. |
| 140 | + |
| 141 | +In many cases, **watching resources** is the preferred approach for ensuring Kubernetes resources |
| 142 | +remain in the desired state. It is more efficient, responsive, and aligns with Kubernetes' event-driven architecture. |
| 143 | +However, there are scenarios where `RequeueAfter` is appropriate and necessary, particularly for managing external |
| 144 | +systems that do not emit events or for handling resources that take time to converge, such as long-running processes. |
| 145 | +Relying solely on `RequeueAfter` for all scenarios can lead to unnecessary overhead and |
| 146 | +delayed reactions. Therefore, it is essential to prioritize **event-driven reconciliation** by configuring |
| 147 | +your controller to **watch resources** whenever possible, and reserving `RequeueAfter` for situations |
| 148 | +where periodic checks are required. |
| 149 | + |
| 150 | +### When `RequeueAfter X` is Useful |
| 151 | + |
| 152 | +While `RequeueAfter` is not the primary method for triggering reconciliations, there are specific cases where it is |
| 153 | +necessary, such as: |
| 154 | + |
| 155 | +- **Observing External Systems**: When working with external resources that do not generate events |
| 156 | +(e.g., external databases or third-party services), `RequeueAfter` allows the |
| 157 | +controller to periodically check the status of these resources. |
| 158 | +- **Time-Based Operations**: Some tasks, such as rotating secrets or |
| 159 | +renewing certificates, must happen at specific intervals. `RequeueAfter` ensures these operations |
| 160 | +are performed on schedule, even when no other changes occur. |
| 161 | +- **Handling Errors or Delays**: When managing resources that encounter errors or require time to self-heal, |
| 162 | +`RequeueAfter` ensures the controller waits for a specified duration before checking the resource’s status again, |
| 163 | +avoiding constant reconciliation attempts. |
| 164 | + |
| 165 | +## Usage of Predicates |
| 166 | + |
| 167 | +For more complex use cases, [Predicates][cr-predicates] can be used to fine-tune |
| 168 | +when your controller should trigger reconciliation. Predicates allow you to filter |
| 169 | +events based on specific conditions, such as changes to particular fields, labels, or annotations, |
| 170 | +ensuring that your controller only responds to relevant events and operates efficiently. |
| 171 | + |
| 172 | +[controller-runtime]: https://github.com/kubernetes-sigs/controller-runtime |
| 173 | +[owner-ref-k8s-docs]: https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/ |
| 174 | +[cr-predicates]: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate |
| 175 | +[secondary-resources-doc]: watching-resources/secondary-owned-resources |
| 176 | +[predicates-with-external-type-doc]: watching-resources/predicates-with-watch |
| 177 | +[cr-owner-ref-doc]: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil#SetOwnerReference |
0 commit comments