Bad Request (400) when accessing NetBox via reverse proxy

[lucakuehne]

Hi

I have installed NetBox on Ubuntu 22.04 using apache (without docker). When I access NetBox using the local IP of the VM it works perfectly fine.

Next I setup nginx on a OPNsense firewall as a reverse proxy and tried accesssing NetBox via the FQDN, but I get a "Bad Request (400)" all the time. (I triedy ALLOWED_HOST set to '*' and to the FQDN + Local IP.)

In the nginx logs there is no error and I got another webserver (Xen Orchestra to be precise) up and running trough the reverse proxy, so I would assume, the error doesn't lie within OPNsense/nginx.

I don't really know where to start looking inside of NetBox. What would be the best log file to take a look at and where is it located?

[candlerb]

So you're going via two reverse proxies: nginx and apache?

The easiest way to diagnose this is going to be to run tcpdump on the Netbox host to look at traffic to and from gunicorn:

tcpdump -i lo -nn -s0 -A tcp port 8001

Make an incoming request, look at the headers of the request, and check that the response 400 is being generated from here (which proves it's not Apache or Nginx that's setting the result code to 400).

There are a few things which can break here, one of which is that Django barfs if "X-Forwarded-Host" has multiple hosts, e.g.

X-Forwarded-Host: 1.2.3.4, 5.6.7.8

If that's what you see, then you need to adjust things.

I've had Netbox running successfully behind a pair of Apache proxies. On the outer host, I prevented it adding X-Forwarded-Host like this:

ProxyPreserveHost On
ProxyAddHeaders Off
RequestHeader set X-Forwarded-For "expr=%{REMOTE_ADDR}"

Then the inner proxy was able to add a single X-Forwarded-Host, and all was OK.

[lucakuehne]

So you're going via two reverse proxies: nginx and apache?

There is the nginx instance (on OPNsense), which acts as a reverse proxy.

And there is apache running on the netbox VM as requested in the netbox documentation. I'm not familiar with python/gunicorn so I'm not quite sure, but if it get it right the local apache instance acts as a reverse proxy to gunicorn.

So technically there are two reverse proxy from my understanding.

The easiest way to diagnose this is going to be to run tcpdump on the Netbox host to look at traffic to and from gunicorn:

tcpdump -i lo -nn -s0 -A tcp port 8001

I did do that, and ran one request to the local IP and one using the public FQDN via the reverse proxy. The local request shows a 200 status code in the packet capture. But the public request results in a 400 status code.

There are a few things which can break here, one of which is that Django barfs if "X-Forwarded-Host" has multiple hosts, e.g.

X-Forwarded-Host: 1.2.3.4, 5.6.7.8

If that's what you see, then you need to adjust things.

Yes I can see two hosts for "X-Forwarded-Host" and two for "X-Forwarded-For". Altough the two values for "X-Forwarded-Host" are the same so like X-Forwarded-Host: netbox.domain.com, netbox.domain.com.

I've had Netbox running successfully behind a pair of Apache proxies. On the outer host, I prevented it adding X-Forwarded-Host like this:

ProxyPreserveHost On
ProxyAddHeaders Off
RequestHeader set X-Forwarded-For "expr=%{REMOTE_ADDR}"

Then the inner proxy was able to add a single X-Forwarded-Host, and all was OK.

Upon first investigations it doens't seem possible to add this for nginx on OPNsense. Is it possible to add this on the inner reverse proxy (= apache on the netbox vm)?

[candlerb]

Sorry I missed this:

Yes I can see two hosts for "X-Forwarded-Host" and two for "X-Forwarded-For". Altough the two values for "X-Forwarded-Host" are the same so like X-Forwarded-Host: netbox.domain.com, netbox.domain.com.

Ah yes, that's the problem that causes Django to barf.

As a quick fix on the inner Apache (the one on the Netbox server), you could try

RequestHeader unset X-Forwarded-Host

(Also, check if the Host: header in tcpdump shows the original FQDN, or the FQDN of the internal Netbox server. Ideally it would be the former. There may be an Nginx setting to tweak for this)

[lucakuehne]

I did do that, and ran one request to the local IP and one using the public FQDN via the reverse proxy. The local request shows a 200 status code in the packet capture. But the public request results in a 400 status code.
No, what I'm suggesting is that you make a request to the public FQDN, but tcpdump the traffic at gunicorn while you do this.

If in this situation you find that gunicorn is generating a 400 response, then you look more carefully at the full set of headers in the inbound request which arrives at gunicorn and shown by tcpdump. Post them here if you need help.

Sorry maybe my English is too bad, so it wasn't clear. Let me try again: Basically you where right. Because the tcpdump looks like this:

• request to local IP
  • status code 200
  • X-Forwarded-For: 1 IP (local/internal)
  • X-Forwarded-Host: 1 IP (local/internal)
• request to public FQDN
  • status code 400
  • X-Forwarded-For: 2 IPs (one public and one local/internal)
  • X-Forwarded-Host: 2 FQDNs (but both the same)

Upon first investigations it doens't seem possible to add this for nginx on OPNsense

I suggest first diagnosing the problem properly, before working on solutions.

That's why I started looking for a way to add the said options to the outer reverse proxy.

[lucakuehne]

Ah yes, that's the problem that causes Django to barf.

As a quick fix on the inner Apache (the one on the Netbox server), you could try

RequestHeader unset X-Forwarded-Host

This seems to work. I'm going to test it a bit further, but thanks a lot already!
So I guess the second value in X-Forwarded-For isn't really a problem for gunicorn?

(Also, check if the Host: header in tcpdump shows the original FQDN, or the FQDN of the internal Netbox server. Ideally it would be the former. There may be an Nginx setting to tweak for this)

It is the original/public FQDN.

[candlerb]

So I guess the second value in X-Forwarded-For isn't really a problem for gunicorn?

X-Forwarded-For is the chain of client IP addresses, and there's no problem with multiple values of that. I believe it's Netbox itself which parses that, in netbox/utilities/request.py - it splits on comma.

X-Forwarded-Host is the chain of server hostnames, and Netbox/Django breaks when that has multiple values. The only reference I see in Netbox code is USE_X_FORWARDED_HOST = True in netbox/netbox/settings.py, so I think this is likely a Django issue - see venv/lib/python3.8/site-packages/django/http/request.py

[The Host: header] is the original/public FQDN.

Good. This means you can put this into ALLOWED_HOSTS instead of *

[lucakuehne]

It seems to work like a charm. Thank you very very much!

Also thank you very much for the additional information on how X-Forwarded-For and X-Forwarded-Host get processed!