Non-interactively obtain SSH host cert?

[danb35]

I have a Proxmox cluster at home, with a number of templates to make it easy to spin up a VM in moments in a standard configuration. But one thing those templates don't do is obtain a SSH host cert, so that when I ssh to the new VM, I get the dreaded key mismatch error. Consequently, my known_hosts file, which had shrunk significantly once I started using SSH host certificates, is growing again.

So the question is how to set up these templates in such a way that they obtain a SSH host cert (ideally for both the hostname and IP address) on first boot. I saw the piece on IIDs (https://smallstep.com/blog/embarrassingly-easy-certificates-on-aws-azure-gcp/), but I don't see any indication that Proxmox VE has anything comparable. What would be a good way to go about this?

[tashian]

Hi @danb35,

One workflow for this would be to use ACME plus an X5C provisioner.
First you'd use an ACME provisioner to get a TLS certificate for the VM.
(We now support ACME IP SANs, too.)
Then you'd use X5C to get the SSH host certificate, using the cert you got via ACME for authentication.

If you don't want to use ACME, you could alternatively inject a short-lived X509 identity cert into the VM when it's created (which serves as a kind of identity document), and then use X5C to get your SSH host certificate. For this approach you'd want to make sure the X5C provisioner only issues SSH host certificates with the same subjects as are shown in the X5C cert.

A third, and perhaps the simplest, option would be to generate and inject a CA token into the VM on launch, and use the CA token to get an SSH host certificate (via step ssh certificate --host --token $TOKEN) via cloud-init / first boot script. The CA tokens typically have a TTL of 5 minutes.

Does this help?

Carl

BTW does Proxmox have any built-in features that's equivalent to an IID? It could be an interesting future integration, if so!

[tashian]

A third, and perhaps the simplest, option would be to generate and inject a CA token into the VM on launch,

This does sound a bit simpler, but it seems like I still need to authenticate for that token, which means (unless I'm missing something, which is entirely possible) I still need to manually interact with the CA. Maybe I'm not fully understanding how this one would work.

I was imagining the Proxmox host environment would get a CA token via a hookscript that runs when the guest VM is about to launch. Cloud-init would then pass the token to the guest VM as metadata. I've never tested this and I'm not sure if it's even possible.

[danb35]

I think that merits further investigation, but as I can better wrap my head around the ACME/X5C workflow, I think I'm moving in that direction. It's straightforward enough to install acme.sh and use it to get a cert for $(hostname).local. But using the x5c provisioner to get the SSH cert is confusing me:

root@ubuntu22 ➜  ~ step ssh certificate --insecure --no-password --provisioner x5c --x5c-cert=/root/.acme.sh/$(hostname).local/fullchain.cer --host $(hostname).local /etc/ssh/ssh_host_ecdsa_key2
✔ Provisioner: x5c (X5C)
provisioner type 'X5C' requires the '--x5c-key' flag
root@ubuntu22 ➜  ~ step ssh certificate --insecure --no-password --provisioner x5c --x5c-cert=/root/.acme.sh/$(hostname).local/fullchain.cer --k5c-key=/root/.acme.sh/$(hostname).local/$(hostname).local.key --host $(hostname).local /etc/ssh/ssh_host_ecdsa_key2
too many positional arguments were provided in 'step ssh certificate <key-id> <key-file>'

I'm having trouble figuring out the error from what I see on the docs page, particularly when it didn't complain about the syntax when I omitted the --x5c-key flag.

[tashian]

@danb35 I see a typo in your code here — it says --k5c-key instead of --x5c-key on the second command.

[danb35]

Well, that was a silly mistake--and I looked it over several times and missed it. Thanks for the review. So now (once I added SSH to the x5c provisioner, which I'd neglected to do when I first set it up), I'm able to get the cert--which means I can now noninteractively get the x.509 cert using acme.sh, and then use that to (also noninteractively) get the SSH host cert using that x.509 cert. All as you said.

Next step, figuring how to make cloud-init do that when a new instance of a template is created.

[tashian]

Nice! I think this workflow would be a good tutorial for our docs

[danb35]

Another related discussion (though using vSphere rather than Proxmox) is at #446 .