haproxy proxy protocol

[darix]

What would you like to be added

Add support for haproxy's proxy protocol:

https://www.haproxy.org/download/2.4/doc/proxy-protocol.txt

Why this is needed

The documentation for proxying is talking about 2 features you would like to see

1. the client should connect directly to the step-ca daemon because you use the old cert for authentication
2. you would like to see the real IP of the client

right now you need layer 4 load balancing for this. with the proxy protocol implemented, you can use a layer 7 load balancer like haproxy. the protocol is quite simple and there are already a lot of implementations for it

https://www.haproxy.com/blog/haproxy/proxy-protocol/

[darix]

Already existing implementations

• https://github.com/armon/go-proxyproto
• https://github.com/pires/go-proxyproto

[dopey]

Hey @darix we discussed this a bit last week and wanted to get a bit more info.

Main point of curiosity / contention: Why should we support layer 7 load-balancing? Most popular proxies support layer 4 proxying - I haven't checked but I'm guessing haproxy also supports layer 4 proxying based on SNI header. So, if proxying is already supported, why should we integrate further?

[darix]

yes it can do plain tcp proxying. the only reason why you want to implement their proxy protocol, is that your daemon can log the real client address instead of the proxy address.

so instead of having to do http proxying to get x-forwarded-for headers, you can use tcp proxying. see the real client certificate and the real client IP.

[maraino]

One thing to take into account is that a layer 7 proxy even with haproxy won't support the current renew/rekey endpoints. According to the spec, there's some information about the client presenting a certificate, but as I see it it won't be enough to renew or rekey a cert.

[dopey]

Great point @maraino.

We're open to this but we're interested in understanding use cases (as well as trying to gauge benefit to the community). If folks want to chime in with a +1 or tell us about a use case that would be helpful.

[darix]

proxy protocol really just means this:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
<original content of the tcp stream>

Example is v1 protocol, v2 is binary. pires/go-proxyproto implements both.

Maybe this configuration file will help with the understanding:

global
  log stdout format short daemon
  maxconn 32768
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  stats socket /var/lib/haproxy/stats user haproxy group haproxy mode 0640 level admin
  tune.bufsize 32768
  tune.ssl.default-dh-param 2048

  ssl-default-bind-ciphers   ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

  ssl-default-bind-ciphersuites   TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
  ssl-default-server-ciphersuites TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384

  ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 no-tls-tickets
  ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 no-tls-tickets

  nbthread 20
  cpu-map 1/all 0-19

defaults
  log     global
  mode    http
  option  log-health-checks
  option  log-separate-errors
  # option  dontlog-normal
  # option  dontlognull
  option  httplog
  option  splice-auto
  option  socket-stats
  retries 3
  option  redispatch
  maxconn 10000
  timeout connect     5s
  timeout client     50s
  timeout server    450s

listen youdontwantthis
   #
   # HTTP mode. This is bad in the case of step-ca as the step-ca daemon will not see the client cert from the client
   # as such the own protocol for renewing certs will not work.
   #
   bind :8444 tfo ssl crt /etc/ssl/services/letsencrypt/ alpn h2 npn h2 strict-sni
   mode http
   option httplog
 
   option forwardfor
   #      ^ so we can see the real client IP on the backend

   # you probably want sticky sessions per remote IP so we always talk to the same backend from the same host
   # but for the sake of simplicity of the example we leave that out for now.
  
   server step-ca 127.0.0.1:8443 ssl ca-file /etc/ssl/ca-bundle.pem verifyhost step-ca-hostname

listen youwantthis
   #
   # plain TCP proxy, haproxy will not touch the packets from the client but we need
   # a way to passthrough the real client IP. see at the end
   #
   bind :8445 tfo
   mode tcp
   option tcplog

   # you probably want sticky sessions per remote IP so we always talk to the same backend from the same host
   # but for the sake of simplicity of the example we leave that out for now.
  
   server step-ca 127.0.0.1:8443 send-proxy-v2 
   #                             ^ so we can see the real client IP on the backend
   #                               for this we need the proxy protocol support

[maraino]

So it's actually using the transport layer (4) with some metadata in a "header", it is not an layer 7 proxy.

[darix]

correct. and for some business case considerations ... haproxy is not the only proxy which can emit those proxy header. AWS loadbalancer e.g. supports them as well. and there is a k8s ingress controller based on haproxy.

[rmalchow]

this would be a +1 from me. being forced to directly expose the demon seems off. but i think it would be even better to somehow be able to support layer 7 proxying. this would make it possible to only allow public ingress to specific endpoints - rather than all or nothing. this obviously breaks doing the client cert authentication directly in the initial TLS handshake. but haproxy could also do perform the handshake, verify the client cert against the CA, and then pass on the client cert (or a fingerprint thereof) to the step-ca demon in an HTTP header, so this:

frontend client_cert
    bind ::443 ssl crt foobar.pem ca-file ca.pem verify required 
    mode http

    http-request set-header X-SSL-Client-Cert           %[ssl_fc_has_crt]
    http-request set-header X-SSL-Client-Verify         %[ssl_c_verify]
    http-request set-header X-SSL-Client-SHA1           %{+Q}[ssl_c_sha1,hex]
    http-request set-header X-SSL-Client-DN             %{+Q}[ssl_c_s_dn]
    http-request set-header X-SSL-Client-CN             %{+Q}[ssl_c_s_dn(cn)]
    http-request set-header X-SSL-Client-Not-Before     %{+Q}[ssl_c_notbefore]
    http-request set-header X-SSL-Client-Not-After      %{+Q}[ssl_c_notafter]

would yield these headers:

X-SSL: 1
X-SSL-Client-Verify: 0
X-SSL-Client-SHA1: "FF....."
X-SSL-Client-DN: "/C=foo/ST=bar"
X-SSL-Client-CN: "example"
X-SSL-Issuer: "/C=foo/ST=bar/O=smallstep"
X-SSL-Client-Not-Before: "120101100030Z"
X-SSL-Client-Not-After: "160101100030Z"

while at the same time allowing arbitrary proxying (and external inspection of the requests). the question would be wether or not this is good enough to trust - it would be relatively easy to misconfigure this. although misconfiguration on a level to actually make it insecure ... is a bit more difficult.

.rm

[rmalchow]

if one would be thinking about supporting this, then this may need additional config for which proxy's headers we can trust, and expecting a proper client-auth handshake from everyone else?

[darix]

you can do a lot of access control on the tcp proxy already.

[rmalchow]

yes. but, for example, you cannot selectively expose certain endpoints. in my current thinking, i would like to be able to issue tokens, but only allow certificate retrieval (with a token) and renewal for the rest of the world.