Very poor performance after 3.7.8 to 4.0.5 upgrade

[Kage1]

Deployment Type

Self-hosted

NetBox Version

v4.0.5

Python Version

3.10

Steps to Reproduce

1. Backup database and took VM snap shot of current 3.7.8 version
2. Pull the current netbox git repository against master
3. Run ./upgrade.sh from the /etc/netbox/ install directory
4. Upgrade.sh runs with out an error
5. Restarted the netbox and netbox-rq services
6. Flush cookies/cache and open a new tab to the netbox instance. Web interface is extremely slow, 30+ seconds to navigtate to any page in the system. 3.7.8 version was very snappy, 1 sec or less load times for just about everything.

OS version: Ubuntu 22.04.4 LTS x64
Netbox version: 4.0.5
Redis version: Redis 6.0.16
Postgre db version: PostgreSQL 14.12
Postgre db size: ~ 51 MB

I checked the NB, Redis, and Postgre logs and there are no issues being reported. htop does not show any abnormal process activity, e.g. the python3 environments running are not spiking the processor etc. and memory utilizing is approx 80% free.

Expected Behavior

Similar performance as before.

Observed Behavior

Extremely slow performance to the point of being unusable.

[jeremystretch]

Are you using Safari? If so, see #16447.

[Kage1]

I'm using Firefox 123.01 and tried with Edge.

[Kage1]

It may look like the reindex didnt run... when I try to run it by hand I get:

/opt/netbox# python3 netbox/manage.py reindex
Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

[Kage1]

Finally figured out I needed to run the manage.py through the virtual python environment:

./venv/bin/python3 ./netbox/manage.py reindex

The reindex ran and completed w/o an issue, but the performance is still abysmal.

[markkuleinio]

What does the browser devtools (or whatever they are called) show, where is the time spent?

[Kage1]

During a reload there is no html loaded... from the surface it appears there's is something going on in the backend that is causing the lag. Once the HTML render's it's pretty much immediate.

[Kage1]

Did some timing. From clicking on a menu link in Netbox till data is displayed it's anywhere from 20 - 30 seconds. It doesn't matter if there is nothing in that section (e.g. Circuits) or selecting IP Addresses which loads the first 50 in less than a second. It also doesn't matter if I have been to that section before, clicking between Circuits and IP Addresses always takes 20 - 30 seconds.

I do have one custom link, but even with that disabled it didn't change anything.

[julianstolp]

Have you fixed this problem somehow? I also have this problem coming from 3.7.8 to 4.0.5. Using Edge Browser

[julianstolp]

Problem also disappeared with v4.0.6. Still no idea why it happened.

[markkuleinio]

#16454 - Address DNS lookup bug in django-debug-toolbar

?

[julianstolp]

Thanks, just had a quick look at the patch notes. Yes, downgrading the debug toolbar from 4.4.2 to 4.3.0 in version 4.0.5 fixes the problem.

[Kage1]

#16506 "Roll back django-debug-toolbar version to avoid DNS lookup bug" was merged for v4.0.6. I still need to test it, but @julianstolp confirmed that 4.0.6 fixed the issue for him.

[Kage1]

Wrapping up, I was able to replicate our prod instance into our lab. I upgraded it to 4.0.6 and the performance it back to normal. I can confirm that patch #16506 did indeed fix this issue.

[Delta1977]

Same here, with Edge and Chrome
Coming from 3.0.1 updating through every Version up to 4.0.5.

The Performance Impact appears with v4

[julianstolp]

Can you test with version 4.0.3?