[ENHANCE] Listen on HTTPS: Support TLS authentication for prometheus

[volker-raschek]

Is your feature request related to a problem? Please describe

Not directly to a problem, but to a security improvement. The application start it's web server via the following implementation:

// https://github.com/stakater/Reloader/blob/master/internal/pkg/cmd/reloader.go#L213
logrus.Fatal(http.ListenAndServe(constants.DefaultHttpListenAddr, nil))

1. The web server will never provide TLS authentication and encryption
2. The option in the helm chart to instruct prometheus to use TLS certificates for authentication can never be used. The tlsConfig property is provided but useless.

Describe the solution you'd like

I want to configure the reloader deployment to provide metrics via HTTPS and TLS authentication. The certificate is created via cert-manager. The secret will be mounted as extra volume mount into the reloader pod. The secret contains ca.crt, tls.key and tls.crt. This secrets are used to start the web server listening on HTTPS. Via environment variables or command line flags can be specified where the TLS certificates are mounted. An additional environment variable instructs the web server to require TLS authentication for http clients. For example HTTP_TLS_AUTH=TRUE. Prometheus must provide a valid TLS certificate for authentication. The prometheus instance has also mounted a TLS certificate issued from the same certificate authority. The defined tlsConfig of the serviceMonitor or podMonitor resource describe who prometheus should be authenticate himself. For example could be the podMonitor described as:

prometheusPodMonitor:
  tlsConfig:
    caFile: /etc/prometheus/tls/ca/ca.crt # location of the ca certificate inside the prometheus container filesystem 
    certFile: /etc/prometheus/tls/app2app/tls.crt # location of the client certificate inside the prometheus container filesystem 
    keyFile: /etc/prometheus/tls/app2app/tls.key  # location of the private key inside the prometheus container filesystem 
    insecureSkipVerify: false
    serverName: reloader # common name or a subject alternative name of the issued certificate of the reloader deployment 

Hint

Based on the fact, that the reloader deployment will than use in future TLS certificates for authentication and encryption, the developers should be aware of the fact, that the TLS certificates can be rotate in the container file system. For example, when the cert-manager issues a new certificate before expiring.

The maintainers should clarify how to proceed. Should be used the reloader deployment to trigger a reload of itself or should be implemented inside the application a file system detection via fsnotify or certificate watcher to avoid a dependency on itself.

Describe alternatives you've considered

No alternative.

Additional context

Precondition: Helm: volumes and volumeMounts