Service Guru

[usercont-release-bot]

Every Monday (we agreed that it is better to deploy the service always at the same time; we should start around 1PM so that it will be finished, hopefully, before 3PM):

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go.

• Before moving branches change the channel topic on both chats (matrix and slack) that we are going to redeploy packit service (and then change it back when the deployment is finished and the pods are running fine). Changing the topic should automatically create a message in the channel, if not send a message in the channel with the topic. The message could be something like this: We are deploying the new packit service on production. If Packit does not react to your actions for a long time (30 minutes) try retriggering it with /packit build, /packit test, /packit propose-downstream, /packit pull-from-upstream, /packit koji-build, /packit create-update.
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
    • If pushing the stable branch fails for a repo (e.g. because some changes had to be cherry-picked into production), force push the current state of main as stable to sync up the history
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.

Warning

Image build takes around 40 minutes since adding additional architectures (mainly because of the aarch64 emulation).

• Clean up older images from Quay

Note

Roughly once a month check Quay repository and remove older images that take up unnecessary space.

Ideally sort the repos based on “last modified” as the sorting by size doesn't account for different units (e.g., GB vs MB) correctly.

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use title Packit [in] $MONTH $YEAR. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by [make move-stable]. (https://github.com/packit/deployment/blob/main/Makefile). A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
  • Consider posting the most interesting changes on our Mastodon account (but be aware that the limit for a post is 500 characters, so do a shorter version or split the content to multiple posts and post it throughout the week)
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If the deployment was reverted in the previous week, make sure to check the docs and follow the instructions for the manual steps that need to be run.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.
• A couple of minutes after new images are pushed to Quay.io check that prod deployment is OK. If pods are stuck in a terminating status for a long time, kill it manually oc delete pod --force pod-name.
  Newer images are automatically imported from Quay.io and pods are recreated from them in the same way as in the staging environment.
  • Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
  • Sentry link for debugging
  • Dashboard is OK
• Merge/publish your blog post.
• Create a Mastodon post mentioning the top news.

Warning

Don't forget to redo the repositories for moving stable branch or just clone the additional repo with git clone --recurse-submodules git@github.com:packit/production-monorepo.git