feat(server): Watch the tls certs for changes and update the served certs

[dbason]

Fixes #4617

Watches the certificate file locations (in a manner that is compatible with how k8s links and mounts secrets). Instead of passing the certificate files directly to the http server it uses GetCertificate instead, and updates the certificate when the files are changed.

[netlify]

✅ Deploy Preview for docs-kargo-io canceled.

Name	Link
🔨 Latest commit	cb6361b
🔍 Latest deploy log	https://app.netlify.com/projects/docs-kargo-io/deploys/68d3271edebe680008e1740e

[krancour]

I'd hold off on doing this until what's already here as been reviewed / agreed on, but the external webhooks server is probably in need of this exact same feature.

[dbason]

@krancour is there anything else I need to do to get approval for the CI to run?

[dbason]

internal/server/server.go:253:21: G402: TLS MinVersion too low. (gosec)
			srv.TLSConfig = &tls.Config{
				GetCertificate: certLoader.GetCertificate,
			}
1 issues:

This hasn't actually functionally changed from what the code was doing previously but I think it's being picked up now because I'm explicitly setting the TLSConfig. I can update this.

=== RUN   TestDirectoryWatcher
    directorywatcher_test.go:56: 
        	Error Trace:	/__w/kargo/kargo/internal/server/certwatcher/directorywatcher_test.go:56
        	Error:      	Should be false
        	Test:       	TestDirectoryWatcher
        	Messages:   	channel should be closed
--- FAIL: TestDirectoryWatcher (0.05s)
FAIL
coverage: 87.5% of statements

This passed locally when I was testing but might be flaky so I'll check on it.

[krancour]

is there anything else I need to do to get approval for the CI to run?

They ran. Wonder if @hiddeco authorized it maybe. 🤷‍♂️

This passed locally when I was testing

It's a little bit slower, but make hack-test-unit (as opposed to make test-unit) should execute the tests in as close as we can get to the same environment they're executed in in CI.

[dbason]

I've updated the minimum TLS version to be 1.3 and updated the flaking test to be more resilient.

[dbason]

Is there anything else I can do to move this forward @krancour ?

[dbason]

@krancour we ran into this again recently in our infrastructure. I've rebased and updated the logging package but it would be great to get some idea how how you guys want to handle the certs and push it into the external webhook controller as well.