Implement Pruning of removed upstream groups (#166)

* wip: looking into pruning

* wip: actually prune based on syncTime annotation

* change how syncStartTime is used so each group still gets it's own sync time annotation like currently.  other minor cleanup

* refactoring names

* Return prune value

* Add missing prune to CRD for AzureProvider

* Update documentation

* Remove debugging info from logger on prune

* Prune groups and include count of pruned groups in sync complete status log

* Add new metric for groupsPruned

* Restore accidental removal in deepcopy

* Add backticks in docs around prune false

Author: dweebo

README.md

@@ -77,6 +77,7 @@ Groups contained within Azure Active Directory can be synchronized into OpenShif
 | `filter` | Graph API filter | | No |
 | `groups` | List of groups to filter against | | No |
 | `userNameAttributes` | Fields on a user record to use as the User Name | `userPrincipalName` | No |
+| `prune` | Prune Whether to prune groups that are no longer in Azure | `false` | No |
 
 The following is an example of a minimal configuration that can be applied to integrate with a Azure provider:
 
@@ -131,6 +132,7 @@ Teams stored within a GitHub organization can be synchronized into OpenShift. Th
 | `organization` | Organization to synchronize against | | Yes |
 | `teams` | List of teams to filter against | | No |
 | `url` | Base URL for the GitHub or GitHub Enterprise host (Must contain a trailing slash) | | No |
+| `prune` | Prune Whether to prune groups that are no longer in GitHub | `false` | No |
 
 The following is an example of a minimal configuration that can be applied to integrate with a GitHub provider:
 
@@ -199,9 +201,10 @@ Groups stored within a GitLab can be synchronized into OpenShift. The following
 | `ca` | Reference to a resource containing a SSL certificate to use for communication (See below) | | No |
 | `caSecret` | **DEPRECATED** Reference to a secret containing a SSL certificate to use for communication (See below) | | No |
 | `credentialsSecret` | Reference to a secret containing authentication details (See below) | | Yes |
-| `insecure` | Ignore SSL verification | 'false' | No |
+| `insecure` | Ignore SSL verification | `false` | No |
 | `groups` | List of groups to filter against | | No |
 | `url` | Base URL for the GitLab instance | `https://gitlab.com` | No |
+| `prune` | Prune Whether to prune groups that are no longer in GitLab | `false` | No |
 
 The following is an example of a minimal configuration that can be applied to integrate with a GitHub provider:
 
@@ -255,14 +258,15 @@ The configurations of the three primary schemas (`rfc2307`, `activeDirectory` an
 | `ca` | Reference to a resource containing a SSL certificate to use for communication (See below) | | No |
 | `caSecret` | **DEPRECATED** Reference to a secret containing a SSL certificate to use for communication (See below) | | No |
 | `credentialsSecret` | Reference to a secret containing authentication details (See below) | | No |
-| `insecure` | Ignore SSL verification | 'false' | No |
+| `insecure` | Ignore SSL verification | `false` | No |
 | `groupUIDNameMapping` | User defined name mapping | | No |
 | `rfc2307` | Configuration using the [rfc2307](https://docs.openshift.com/container-platform/latest/authentication/ldap-syncing.html#ldap-syncing-rfc2307_ldap-syncing-groups) schema | | No |
 | `activeDirectory` | Configuration using the [activeDirectory](https://docs.openshift.com/container-platform/4.5/authentication/ldap-syncing.html#ldap-syncing-activedir_ldap-syncing-groups) schema | | No |
 | `augmentedActiveDirectory` | Configuration using the [activeDirectory](https://docs.openshift.com/container-platform/4.5/authentication/ldap-syncing.html#ldap-syncing-augmented-activedir_ldap-syncing-groups) schema | | No |
 | `url` | Connection URL for the LDAP server | `https://gitlab.cldap://ldapserver:389om` | No |
 | `whitelist` | Explicit list of groups to synchronize |  | No |
 | `blacklist` | Explicit list of groups to not synchronize |  | No |
+| `prune` | Prune Whether to prune groups that are no longer in LDAP | `false` | No |
 
 The following is an example using the `rfc2307` schema:
 
@@ -361,11 +365,12 @@ Groups stored within Keycloak can be synchronized into OpenShift. The following
 | `caSecret` | **DEPRECATED** Reference to a secret containing a SSL certificate to use for communication (See below) | | No |
 | `credentialsSecret` | Reference to a secret containing authentication details (See below) | | Yes |
 | `groups` | List of groups to filter against | | No |
-| `insecure` | Ignore SSL verification | 'false' | No |
+| `insecure` | Ignore SSL verification | `false` | No |
 | `loginRealm` | Realm to authenticate against | `master` | No |
 | `realm` | Realm to synchronize | | Yes |
 | `scope` | Scope for group synchronization. Options are `one` for one level or `sub` to include subgroups | `sub` | No |
 | `url` | URL Location for Keycloak | | Yes |
+| `prune` | Prune Whether to prune groups that are no longer in Keycloak | `false` | No |
 
 The following is an example of a minimal configuration that can be applied to integrate with a Keycloak provider:
 
@@ -417,7 +422,7 @@ The following table describes the set of configuration options for the Okta prov
 | `extractLoginUsername` | Bool to determine if you should extract username from okta login | `false`  | No |
 | `profileKey` | Attribute field on Okta User Profile you would like to use as identity | `'login'` | No |
 | `groupLimit` | Integer to set the maximum number of groups to retrieve from OKTA per request. | `1000` | No |
-
+| `prune` | Prune Whether to prune groups that are no longer in OKTA | `false` | No |
 
 The following is an example of a minimal configuration that can be applied to integrate with an Okta provider:
 

api/v1alpha1/groupsync_types.go

@@ -185,6 +185,11 @@ type KeycloakProvider struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Keycloak URL",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
 	// +kubebuilder:validation:Required
 	URL string `json:"url"`
+
+	// Prune Whether to prune groups that are no longer in Keycloak. Default is false
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Prune",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:booleanSwitch"}
+	// +kubebuilder:validation:Optional
+	Prune bool `json:"prune"`
 }
 
 // GitHubProvider represents integration with GitHub
@@ -238,6 +243,11 @@ type GitHubProvider struct {
 	// +kubebuilder:validation:Optional
 	// +kubebuilder:default="https://api.github.com/graphql"
 	V4URL *string `json:"v4url,omitempty"`
+
+	// Prune Whether to prune groups that are no longer in GitHub. Default is false
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Prune",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:booleanSwitch"}
+	// +kubebuilder:validation:Optional
+	Prune bool `json:"prune"`
 }
 
 // GitLabProvider represents integration with GitLab
@@ -274,6 +284,11 @@ type GitLabProvider struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="GitLab URL",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
 	// +kubebuilder:validation:Optional
 	URL *string `json:"url,omitempty"`
+
+	// Prune Whether to prune groups that are no longer in GitLab. Default is false
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Prune",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:booleanSwitch"}
+	// +kubebuilder:validation:Optional
+	Prune bool `json:"prune"`
 }
 
 // LdapProvider represents integration with an LDAP server
@@ -335,6 +350,11 @@ type LdapProvider struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Blacklisted groups to not synchronize",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
 	// +kubebuilder:validation:Optional
 	Blacklist *[]string `json:"blacklist,omitempty"`
+
+	// Prune Whether to prune groups that are no longer in LDAP. Default is false
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Prune",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:booleanSwitch"}
+	// +kubebuilder:validation:Optional
+	Prune bool `json:"prune"`
 }
 
 // AzureProvider represents integration with Azure
@@ -374,6 +394,11 @@ type AzureProvider struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Azure UserName Attributes",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:text"}
 	// +kubebuilder:validation:Optional
 	UserNameAttributes *[]string `json:"userNameAttributes,omitempty"`
+
+	// Prune Whether to prune groups that are no longer in Azure. Default is false
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Prune",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:booleanSwitch"}
+	// +kubebuilder:validation:Optional
+	Prune bool `json:"prune"`
 }
 
 // OktaProvider represents integration with Okta
@@ -407,6 +432,11 @@ type OktaProvider struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Group Limit",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:number"}
 	// +kubebuilder:validation:Optional
 	GroupLimit int `json:"groupLimit"`
+
+	// Prune Whether to prune groups that are no longer in OKTA. Default is false
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Prune",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:booleanSwitch"}
+	// +kubebuilder:validation:Optional
+	Prune bool `json:"prune"`
 }
 
 // ObjectRef represents a reference to an item within a Secret

config/crd/bases/redhatcop.redhat.io_groupsyncs.yaml

@@ -77,6 +77,9 @@ spec:
                           insecure:
                             description: Insecure specifies whether to allow for unverified certificates to be used when communicating to Azure
                             type: boolean
+                          prune:
+                            description: Prune Whether to prune groups that are no longer in Azure. Default is false
+                            type: boolean
                           url:
                             description: URL is the location of the Azure platform
                             type: string
@@ -169,6 +172,9 @@ spec:
                           organization:
                             description: Organization represents the location to source teams to synchronize
                             type: string
+                          prune:
+                            description: Prune Whether to prune groups that are no longer in GitHub. Default is false
+                            type: boolean
                           teams:
                             description: Teams represents a filtered list of teams to synchronize
                             items:
@@ -265,6 +271,9 @@ spec:
                           insecure:
                             description: Insecure specifies whether to allow for unverified certificates to be used when communicating to GitLab
                             type: boolean
+                          prune:
+                            description: Prune Whether to prune groups that are no longer in GitLab. Default is false
+                            type: boolean
                           url:
                             description: URL is the location of the GitLab server
                             type: string
@@ -354,6 +363,9 @@ spec:
                           loginRealm:
                             description: LoginRealm is the Keycloak realm to authenticate against
                             type: string
+                          prune:
+                            description: Prune Whether to prune groups that are no longer in Keycloak. Default is false
+                            type: boolean
                           realm:
                             description: Realm is the realm containing the groups to synchronize against
                             type: string
@@ -575,6 +587,9 @@ spec:
                           insecure:
                             description: Insecure specifies whether to allow for unverified certificates to be used when communicating to LDAP
                             type: boolean
+                          prune:
+                            description: Prune Whether to prune groups that are no longer in LDAP. Default is false
+                            type: boolean
                           rfc2307:
                             description: RFC2307Config represents the configuration for a RFC2307 schema
                             properties:
@@ -719,6 +734,9 @@ spec:
                           profileKey:
                             description: ProfileKey the attribute from Okta you would like to use as the user identifier.  Default is "login"
                             type: string
+                          prune:
+                            description: Prune Whether to prune groups that are no longer in OKTA. Default is false
+                            type: boolean
                           url:
                             description: URL is the location of the Okta domain server
                             type: string