Deivce>Change logs tab loading error

[subhashkoganti]

Hi,

We've been using netbox 3.3.6.
The change logs have grown a lot over time (due to automation), to many hundreds of thousands of entries (for all objects combined).

Specifically for device, interfaces, etc, the change log entries can be upto a few hundreds.

Now, loading the device>>change logs from UI is taking tens of seconds and in many cases it fails after some time.
Below is the error:

<>
<class 'django.db.utils.OperationalError'>

server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Python version: 3.10.6
NetBox version: 3.3.6
<>

The performance of direct sql queries from extras_objectchange table either by id or object_repr, is not bad.
While direct sqls return in reasonable amount of time, when I try to fetch the change logs for a given device (or a specific change log id) from UI, it throws the above error very often.

While deleting the older change logs can mitigate this issue, I'd prefer a solution where I don't have to delete old data, since disk-space is not an issue.

Any suggestions?

[mtinberg]

this may not be the most helpful answer, but I'm on 3.4.10 and have around 3.5m changlog entries (mostly due to automation as well) and loading /extras/changelog/ with 100 items takes 1s at most loading /dcim/devices/1234/changelog/ is about the same. I didn't go back through the changlogs to see if there were any schema changes or improvements between 3.3.6 and 3.4.10 in changlog performance. so that behavior doesn't seem right to me can you see the logs from gunicorn/uwsgi to see if there is a more detailed traceback from the netbox WSGI worker service with journalctl or the text log file and do they provide any enlightenment? — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: subhashkoganti ***@***.***> Sent: Thursday, December 7, 2023 3:15 PM To: netbox-community/netbox ***@***.***> Cc: Subscribed ***@***.***> Subject: [netbox-community/netbox] Deivce>Change logs tab loading error (Discussion #14468) Hi, We've been using netbox 3.3.6. The change logs have grown a lot over time (due to automation), to many hundreds of thousands of entries (for all objects). Specifically for device, interfaces, etc, the change log entries can be upto a few hundreds. Now, loading the device>>change logs from UI is taking tens of seconds and in many cases it fails after some time. Below is the error: <> <class 'django.db.utils.OperationalError'> server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. Python version: 3.10.6 NetBox version: 3.3.6 <> The performance of direct sql queries from extras_objectchange table either by id or object_repr, is not bad. While direct sqls return in reasonable amount of time, when I try to fetch the change logs for a given device (or a specific change log id) from UI, it throws the above error very often. While deleting the older change logs can mitigate this issue, I'd prefer a solution where I don't have to delete old data, since disk-space is not an issue. Any suggestions? — Reply to this email directly, view it on GitHub<#14468>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM7PF7FCDOEHMVVQ373YIIWYJAVCNFSM6AAAAABALWQZNSVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZVHEZTMMZXGM>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[candlerb]

There is a something possibly relevant in v3.3.10:

Bug Fixes

• #11128 - Disable ordering changelog table by object to avoid exception

Whilst the bug isn't directly relevant, it makes me wonder if your changelog table is being sorted in the view you're using, and sorting on a non-indexed field could be very expensive.

Other thoughts:

• The error suggests the database is disconnecting, and it would be good to get some logs from postgres to see why (can you enable query logging? I would expect the query has a LIMIT clause).
• Postgres itself isn't crashing is it? (journalctl -eu postgresql, journalctl -eu postgresql@12-main, less /var/log/postgresql/postgresql-12-main.log)
• Not being killed due to out of memory? (Check output of dmesg)
• You haven't changed anything in postgres.conf, like statement_timeout ?
• You're not using any sort of proxy in front of postgres?

[subhashkoganti]

Thanks Brian.

I was about to update the thread on the log trace.
I see that the netbox application is intermittently not able to connect to the postgres.
It is likely the query is too expensive.

App logs:
port 5432 failed: FATAL: the database system is in recovery mode

Postgres logs:
WARNING: terminating connection because of crash of another server process

I will try increasing the memory for postgres and see if that solves the issue.