Nginx container failing to start with "host not found in upstream" when upstream container is not running

[brtwrst]

Hey,

I use NPM as a reverse proxy for several other docker containers.
When one or more of the upstream containers is not running when NPM is started/restarted, the NPM container will get stuck in a reboot loop with the following error.
(simulated it by running docker stop bitwarden before restarting NPM)

nginx: [emerg] host not found in upstream "bitwarden" in /data/nginx/proxy_host/2.conf:61

This is a bit unfortunate, because in this state, you also can't access the web interface to disable the "missing" proxy host.

Is there any way to make NPM ignore unavailable upstream containers and boot up regardless?

EDIT:
Some more testing shows me that it only happens for containers that have "custom locations".
So the above error seems to be related to the custom location I have set up for this host:

Here are the settings for the proxy host itself:

[kabadisha]

I am also facing exactly the same issue.
Found this guide on how to resolve it with nginx:
https://sandro-keil.de/blog/let-nginx-start-if-upstream-host-is-unavailable-or-down/
But I can't figure out how to do something similar with NPM.

@brtwrst did you ever find a solution?

[brtwrst]

I did not.
I am using traefik instead of NPM at the moment.

[kabadisha]

The youTube algorithm has been pushing traefik at me recently. Worth checking out?
I moved from vanilla nginx as the NPM UI made maintenance so much easier. Is the traefik UI decent?

[brtwrst]

traefik does not have a UI - only a "read only Dashboard". You configure it via labels on docker containers, and give the traefik container access to your docker.sock (like portainer) so it can read these container labels. Then it will just create routes when a container is started and removes them once it is stopped/deleted.
It's definitely not as intuitive as NPM and it took me about half a day to get everything up and moved over but now it just runs.

[kabadisha]

@brtwrst and anyone else who finds this thread in future: I found a workaround!

This issue only occurs with Proxy Hosts in NPM that have custom locations defined. The workaround is to remove the custom location declaration in the GUI and instead declare it manually in the 'Advanced' tab of the host like this:

location / {
    # Block Exploits
    include conf.d/include/block-exploits.conf;
    
    set $custom_upstream example.com;
    proxy_pass http://$custom_upstream:80;
}

Just replace example.com with the host that is not always available.
Even better, if you only have one custom location, you can just use all the variables from the parent Proxy Host so it pulls the details from the GUI:

location / {
    # Block Exploits
    include conf.d/include/block-exploits.conf;
    
    proxy_pass $forward_scheme://$server:$port;
}

Link to thread that helped me find the solution.

[kabadisha]

@brtwrst I raised a PR with a fix for this issue: #2672

Any support you can give to it by testing or even just upvoting would be appreciated 👍