Automated TLS certificate rotation with certbot

[createyourpersonalaccount]

Hello,

I'm trying to think about automatic tls cert rotation with my nginx-unit server. I have read the instructions at https://unit.nginx.org/howto/certbot/ but I don't understand the "temporary route" for the webroot method. How is it temporary? In particular, how can I update my TLS cert without shutting down the server? Port 80 is already serving my litestar app. In this letsencrypt forums answer you can see how to use certbots --deploy-hook, so something like this is what I had in mind.

I was hoping I could use this hook without having to shut down my server. Is this possible? Is it a bad idea?

[javorszky]

Hi there,

Thanks for getting in touch! I'm going to tackle different parts of your questions in sections.

The temporary route

The word "temporary" there is sort of a misnomer. The reason it's temporary is because technically we only need this until certbot is happy that it can reach the token file on that route and issues the certificates. After that we could remove the routing to the .well-known path. We don't have to, because it's useful to keep it around when renewal and rollover comes around.

Certbot uses Let's Encrypt's HTTP-01 challenge here.

In order to do this without shutting down Unit:

1. query your current configuration from the Unit instance
2. add the route match rule for the token as described in the temporary route option (ie if the request comes into .well-known/acme-challenge/*, then share whatever is requested from a directory
3. run the certbot command for the webroot authentication passing the directory that the unit rule is sharing from
4. this should result in certificate files that you should bundle up per the Unit documentation
5. and then send the new configuration for your domain at domain uploading the certificates and the changed configuration
6. Unit should then reconfigure itself without you needing to shut down and restart unit

Using the --deploy-hook method

That builds on the above. The prerequisite is that the rule for the .well-known/acme-challenge/* route stays active as long as you want to keep the auto renewal up and running.

If you use the server where certbot is installed to renew certificates for a single domain, then you can put the script you're about to write in the /etc/letsencrypt/renewal-hooks/deploy/ directory, as the forum answer says.

That way when you hae a cron job running every 88 days or so (just a bit shorter than the 90 day renewal period), your script will execute. The cron job would be something like this command (you might need sudo here):

$ certbot certonly --webroot -w /var/www/www.example.com/ -d www.example.com

Then in your script you can do all of the steps that you would manually do:

• bundle certs
• upload the certs with a curl command
• update the configuration with a curl command

This should theoretically work. Let me know if you'd like more guidance, or have any other questions!

And again, thank you for using Unit 🙂

[createyourpersonalaccount]

Thank you for the response, it is very helpful. I'm postponing this for later because my certificate won't expire until November and I'm currently working on other aspects of the server, but I will get back to you with my results.

[ac000]

That way when you hae a cron job running every 88 days

Just to say it's better to run the job weekly, that way if it randomly fails for some reason, you won't end up with an expired certificate. (That's certainly what I do...)

[createyourpersonalaccount]

Thanks. For what it's worth, on Debian the Certbot apt package adds a systemd timer that runs every 12h with certbot renew, which has its own rules for when to request a new certificate (<1/3 of life for >10day certs, <1/2 for <=10day, otherwise nothing is done.) The finer tuning is not necessary for me!

[createyourpersonalaccount]

@javorszky I was finally able to test your suggestion.

I noticed that the Unit docs now have a howto (was it always there?) for Certbot. I followed it and it worked.

I want to add the following comments here for those who want to automate Certbot for their NGINX Unit. Instead of using --deploy-hook, I used the special directories Certbot has for automation. I did not have to set up (on Debian) a cron job because the Certbot apt package adds a systemd timer to run certbot renew at midnight and noon (you can check with systemctl list-timers certbot and systemctl cat certbot.timer). Because I did not want the .well-known path to linger after the certificate is issued, I added it in a pre-hook script and removed it in a post-hook script. I also rotate the bundles in a way that only the original name remains eventually, by updating the listener to a temporary name and then back to the original.

1. Under /etc/letsencrypt/renewal-hooks/pre, I put a script that contains:

#!/bin/sh

# Activate the ACME route.
curl -X PUT \
  --unix-socket /var/run/control.unit.sock \
  --data-binary '"routes/acme"' \
  "http://localhost/config/listeners/*:80/pass"

2. Under /etc/letsencrypt/renewal-hooks/deploy, I put the following script (edit with your website):

#!/bin/sh

# Name of the certificate bundle without the suffix .pem.
bundle=bundle
bundlepath="/root/${bundle}.pem"
# This directory is created by Certbot the first time you
# run Certbot to produce your first certificate.
dir=/etc/letsencrypt/live/www.example.com

# Create the new bundle.
(umask 177; shred -fu "$bundlepath" 2>/dev/null; touch "$bundlepath")
cat "${dir}/privkey.pem" \
    "${dir}/fullchain.pem" \
   >> "$bundlepath"
# Upload the new bundle to NGINX Unit with a temporary name.
curl -X PUT \
  --unix-socket /var/run/control.unit.sock \
  --data-binary "@$bundlepath" \
  "http://localhost/certificates/${bundle}_tmp"
# Configure the port 443 listener to use the uploaded bundle
# with its temporary name.
curl -X PUT \
  --unix-socket /var/run/control.unit.sock \
  --data-binary "\"${bundle}_tmp\"" \
  "http://localhost/config/listeners/*:443/tls/certificate"
# Delete the old bundle.
curl -X DELETE \
  --unix-socket /var/run/control.unit.sock \
  "http://localhost/certificates/$bundle"
# Replace it with the new bundle (upload again).
curl -X PUT \
  --unix-socket /var/run/control.unit.sock \
  --data-binary "@$bundlepath" \
  "http://localhost/certificates/$bundle"
# Configure the port 443 listener to use the new bundle with the original name.
curl -X PUT \
  --unix-socket /var/run/control.unit.sock \
  --data-binary "\"$bundle\"" \
  "http://localhost/config/listeners/*:443/tls/certificate"
# Delete the temporary bundle.
curl -X DELETE \
  --unix-socket /var/run/control.unit.sock \
  "http://localhost/certificates/${bundle}_tmp"
# Securely remove the bundle file.
shred -fu "$bundlepath"

3. Under /etc/letsencrypt/renewal-hooks/post, I put the following script:

#!/bin/sh

# Deactivate the ACME route.
curl -X PUT \
  --unix-socket /var/run/control.unit.sock \
  --data-binary '"routes/static"' \
  "http://localhost/config/listeners/*:80/pass"

4. My unit.json (this is an example static website) looks like this:

{
  "listeners": {
    "*:80": {
      "pass": "routes/static"
    },
    "*:443": {
      "pass": "routes/static",
      "tls": {
        "certificate": "bundle"
      }
    }
  },
  "routes": {
    "acme": [
      {
        "match": {
          "uri": "/.well-known/acme-challenge/*"
        },
        "action": {
          "share": "/var/www/www.example.com$uri"
        }
      },
      {
        "action": {
          "pass": "routes/static"
        }
      }
    ],
    "static": [
      {
        "action": {
          "share": "/home/myuser/www/hello_world.txt"
        }
      }
    ]
  }
}

The first time you run Certbot the "*:80" listener should pass to "routes/acme" instead. Then you can change it to static.

Thanks again.

[javorszky]

Hi @createyourpersonalaccount ! Glad you got it working! I'm unsure if the howto has always been there, but in the past half year or so Unit was moved to a different part of the company, and I'm no longer in that team, so can't tell what their day-to-day is like for Unit.

At the end of the day, it worked! Thanks for getting back to us about it! 🙂