how to remove deployed infrastructure from a failed deployment?

[rsignell-usgs]

We tried to deploy nebari using the guided init and the nebari deploy got through the IP creation step but then failed with:

[terraform]: Terraform has created a lock file .terraform.lock.hcl to record the provider
[terraform]: selections it made above. Include this file in your version control repository
[terraform]: so that Terraform can guarantee to make the same selections by default when
[terraform]: you run "terraform init" in the future.
[terraform]:
[terraform]: Terraform has been successfully initialized!
[terraform]:
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]:
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
[terraform]: ╷
[terraform]: │ Error: Invalid credentials
[terraform]: │
[terraform]: │   with module.kubernetes-keycloak-helm.kubernetes_manifest.keycloak-http,
[terraform]: │   on modules/kubernetes/keycloak-helm/main.tf line 37, in resource "kubernetes_manifest" "keycloak-http":[terraform]: │   37: resource "kubernetes_manifest" "keycloak-http" {
[terraform]: │
[terraform]: │ The credentials configured in the provider block are not accepted by the
[terraform]: │ API server. Error: Unauthorized

I was trying to deploy with the GITHUB_USER @rsignell-usgs, but the Oauth client_id and client_secret were created by @jkellndorfer.

I thought I would create my own client_id, client_secret and then rerun the init.

I then thought I would first destroy the existing infrastructure that got deployed.

But when I tried terraform destroy it failed with KeyError: 'realm_id'

I then tried to delete the EKS cluster from the console, but it mentioned there were node groups that needed to be deleted first.

What to do now?

[viniciusdc]

ntioned there were node groups that needed to be deleted first.

Thanks for the updates @rsignell-usgs, its really a unique situation. I am not sure what caused the Keycloak to fail middle deployment... Regarding the error with terraform destroy, if Keycloak is not successfully deployed, the destroy will fail as one of the providers we use always attempts to do a health check in the keycloak realm endpoint...

[viniciusdc]

I am also presenting the Console steps to destroy the K8s cluster and resources, for future ref.

In the aws console:

• Go to Elastic-Kubernetes-service > Cluster > Select you cluster > Delete cluster
  • If you encounter an error message about node groups present in the cluster: open the cluster and go to the compute tab, in there, you will see each node group select then on by one and then press the Delete button. Once they are removed, you can try deleting the cluster again
• After the deletion, search for Load Balancers and select the one under EC2 Feature , there you might encounter a list of resources. Delete the one that has the nebari project name in the tags section (you might need to inspect the associated resource group as well if the tag does not show up)
• Sometimes, all resources are deleted, so you might not find the LB in that case
• this should remove the VM's and the k8s cluster; later I would suggest checking EC2 Volumes and searching for the ones related to the Nebari deployment, they normally linger around, and as PVC mounts, they count as storage.

[rsignell-usgs]

This was a bit of a struggle.

• when it came to deleting the VPCs, they would not delete because they had network interfaces, and it said I can't delete or detach the network interfaces because it says they are being used by a service
• we used "Application Manager" to discover what other resources were associated with our Nebari deployment and realized we had not deleted the EFS volume (we had only deleted the EBS volumes from the EC2 page)
• after deleting the EFS volume, we were able to delete the VPC
• we also discovered other resources via the Application Manager that needed to be deleted, like the DynamoDB table
• We also tried going searching "Resource Group", and there was a Resource Group for the nebari deployment that also showed the resources being used. But when we clicked "delete" there, thinking it would delete all the resources, it just deleted the resource group, which meant we then couldn't figure out which services were left to be deleted! Whoops!

[rsignell-usgs]

I'm in the process of cleaining up an old qhub deployment that didn't get fully destroyed. There was no Resource Group defined, but I was able to create one by choosing "tag based", then specifying the tag as "Project" and the key as "esipfed-maple". This resulted in:

[rsignell-usgs]

Armed with this, I tried to delete the internet gateway, but it said it was in use. I then discovered I needed to search for load balancers and find the one associated with the esipfed-maple-dev VPC. Once that was deleted, I was able to detach and delete the internet gateway. Then delete the VPC. Phew!