Enabling CI on github after deployment

[rsignell-usgs]

@jkellndorfer and I have successfully deployed the latest Nebari on AWS (https://nebari.data.science.earth), and have pushed to a private repo on github. We then pushed a change to the config but no action occurred.

On our older qhub deployment on AWS, the github repo had the .github/workflows dir and several secrets set.
We don't have that dir or any secrets on our Nebari repo.

What should we do to fix ? I did check the docs, including troubleshooting and FAQ and didn't see anything likely (though I might have missed it)

[trallard]

@iameskild do we have any best practices for this? my intuition would be to update the nebari-config.yml to add the relevant GitHub information and redeploy with nebari upgrade?

[jkellndorfer]

We added the ci/cd lines to the config file:

ci_cd:
  type: github-actions
  branch: main

After setting the AWS credential environment variables, we issued the upgrade call with the following result:

nebari upgrade -c nebari-config.yaml
Your config file nebari-config.yaml appears to be already up-to-date for Nebari version 2022.10.1

[trallard]

sorry got my commands messed up see https://github.com/orgs/nebari-dev/discussions/1587#discussioncomment-4321457 for completeness

[rsignell-usgs]

@iameskild We don't want to do nebari deploy again, do we?
I seem to remember we used to do qhub render didn't we?
Is it nebari render that we want when we make changes to the config locally? Should we push changes to the repo first?
I looked in the docs and didn't find info on this.
I guess we are confused!

[trallard]

Hey @rsignell-usgs, I am unfamiliar with your current deployment setup, but you might need to re-deploy to ensure the changes indeed take effect. The needed steps seem to be as follows:

1. Update your nebari-config.yml to add a CI provider
2. Once you made your changes, you can use nebari render to apply these changes and render the Terraform files (note there is now a --dry-run flag in the new CLI, which will render the files without writing or modifying any files)
3. Re-deploy nebari to ensure changes take effect

@viniciusdc or @iameskild can you please double-check to make sure this is correct?

[rsignell-usgs]

Thanks @trallard ! Could we do nebari render and then push the changed terraform to github without doing nebari deploy, then see if Github Actions can deploy the changes?

[trallard]

Yes, it makes sense that this would pick up the changes and trigger the deployment through CI.
LMK how this goes so I can document this accordingly once we have found a resolution 📖

[trallard]

Aha found the old docs https://docs.qhub.dev/en/stable/source/admin_guide/upgrade.html#ci-cd-render-and-commit-to-git and at least these steps are the correct ones.

As per @viniciusdc comment it would be best to inspect your keycloack pod

[jkellndorfer]

We ran the nebari render command and pushed to github. After setting Secrets including a REPOSITORY_ACCESS_TOKEN (which we generated as a personal access token), we could restart the action triggered by the push, but if failed with the following warnings and error message:

[terraform]: No changes. Your infrastructure matches the configuration.
[terraform]: 
[terraform]: Terraform has compared your real infrastructure against your configuration
[terraform]: and found no differences, so no changes are needed.
[terraform]: 
[terraform]: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
[terraform]: 
[terraform]: Outputs:
[terraform]: 
[terraform]: keycloak_credentials = <sensitive>
[terraform]: keycloak_nebari_bot_password = <sensitive>
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nebari.data.science.earth'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
Attempt 1 failed connecting to keycloak master realm
Attempt 2 failed connecting to keycloak master realm
Attempt 3 failed connecting to keycloak master realm
Attempt 4 failed connecting to keycloak master realm
Attempt 5 failed connecting to keycloak master realm
Attempt 6 failed connecting to keycloak master realm
Attempt 7 failed connecting to keycloak master realm
Attempt 8 failed connecting to keycloak master realm
Attempt 9 failed connecting to keycloak master realm
Attempt 10 failed connecting to keycloak master realm
ERROR: unable to connect to keycloak master realm at url=https://nebari.data.science.earth/auth/ with root credentials
Error: Process completed with exit code 1.

[viniciusdc]

Hi @jkellndorfer, the error you are seeing is related to Nebari Keycloak deployment, right after keycloak is deployed, there is a test that checks if the pod is running/accessible. which is the one currently failing.

Could you validate if the Keycloak pods are running already?

kubectl get pods --namespace=<your-namespace>

[costrouc]

We checked the cluster and it was in a weird state where keycloak bits were pending but stages after that were already being provisioned. They had an initial deployment (not from ci) that worked and then the second deployment using CI failed.

[viniciusdc]

Uhm, strange. Did terraform not load the state from remote?

[krassowski]

@viniciusdc this appears the "failed connecting to keycloak master realm" error also is mentioned in #2162.