Sometimes (randomly) I get the error "netbox@netbox FATAL: could not open file "global/pg_filenode.map": Operation not permitted"

[ane-ruiz]

Hello,

Currently I have Netbox version 3.5.6 (with python version is 3.10.6) and postgresql version 14 (psql 14.8 (Ubuntu 14.8-0ubuntu0.22.04.1)) in Ubuntu 22.04 Server.

The problem I have is that sometimes, without any consistency, I get the following error in postgresql when running more than one query to netbox from an automation environmet (Ansible-Jenkins):

2023-07-13 12:06:21.261 UTC [204463] netbox@netbox FATAL:  could not open file "global/pg_filenode.map": Operation not permitted
2023-07-13 12:18:40.192 UTC [206437] netbox@netbox LOG:  could not receive data from client: Connection reset by peer
2023-07-13 12:20:22.320 UTC [206688] netbox@netbox LOG:  could not receive data from client: Connection reset by peer
2023-07-13 12:20:25.869 UTC [206697] netbox@netbox LOG:  could not receive data from client: Connection reset by peer
2023-07-13 12:26:41.646 UTC [207861] netbox@netbox FATAL:  could not open file "global/pg_filenode.map": Operation not permitted
2023-07-13 12:26:41.646 UTC [207864] netbox@netbox FATAL:  could not open file "global/pg_filenode.map": Operation not permitted
2023-07-13 12:26:41.646 UTC [207863] netbox@netbox FATAL:  could not open file "global/pg_filenode.map": Operation not permitted
2023-07-13 12:26:41.649 UTC [207859] netbox@netbox ERROR:  could not open file "base/16385/16794": Operation not permitted
2023-07-13 12:26:41.649 UTC [207859] netbox@netbox STATEMENT:  SELECT COUNT(*) FROM (SELECT (SELECT count(*) FROM (SELECT U0."id" FROM "dcim_site" U0 INNER JOIN "dcim_region" U1 ON (U0."region_id" = U1."id") WHERE (U1."lft" >= ("dcim_region"."lft") AND U1."lft" <= ("dcim_region"."rght") AND U1."tree_id" = ("dcim_region"."tree_id")) ORDER BY U0."_name" ASC) _count) AS "site_count" FROM "dcim_region") subquery

In the netbox logs, the error I see is the following:

django.request ERROR: Internal Server Error: /api/dcim/regions/

In the postgresql configuration file I have the following, according to the characteristics that my netbox virtual machine has (Postgresql version 14, Linux, RAM 8GB, 4 CPUs, HDD Storage):

# DB Version: 14
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 8 GB
# CPUs num: 4
# Data Storage: hdd

max_connections = 200
shared_buffers = 2GB
effective_cache_size = 6GB
maintenance_work_mem = 512MB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 4
effective_io_concurrency = 2
work_mem = 5242kB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 4
max_parallel_workers_per_gather = 2
max_parallel_workers = 4
max_parallel_maintenance_workers = 2

I do not have any other version of postgresql installed or residual configuration from other versions.

I get this error when doing multiple query tasks continuously. There are times when it occurs when doing a single one, but it is less recurrent.

Can anyone help me to solve this error?

Thank you!!
Ane

[candlerb]

What's the underlying filesystem type that Postgres is storing its data on?

[ane-ruiz]

Hello,

The type is ext4

/dev/sda2 on / type ext4 (rw,relatime)

It is like this by default, is there any problem?
Thank you.

[candlerb]

I was just trying to rule out something weird like using NFS as your data directory. ext4 should be fine.

Have a look at the output of "dmesg", and see if there are any kernel logs which occur around the same time as you see those Operation not permitted messages.

See if you can replicate it using the psql command line tool, by issuing the same sorts of queries as Netbox itself is issuing, concurrently. If you can, then you've ruled out Netbox entirely, and you can raise this as an issue with Ubuntu and/or Postgres communities.

FWIW, I've never seen this issue. Production instances of Netbox have been mostly on 20.04 though; I have access to one on 22.04 but it's not heavily loaded.

If you can find out what REST queries Ansible is making to Netbox then this might also help someone else replicate the problem. At worst you can use tcpdump:

tcpdump -i lo -nn -s0 -A tcp port 8001

If you're able to spin up another VM using 20.04 and postgres-12, then that might solve your problem (and if it does, would help narrow down where the underlying cause lies)

[ane-ruiz]

Hi,

The errors I have seen in similar time to the "Operation not permitted" error are:

[    6.935384] audit: type=1400 audit(1689592280.028:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=509 comm="apparmor_parser"
[    6.935388] audit: type=1400 audit(1689592280.028:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=509 comm="apparmor_parser"
[    7.479058] vmw_vmci 0000:00:07.7: Using capabilities 0xc
[    7.491586] Guest personality initialized and is active
[    7.522118] VMCI host device registered (name=vmci, major=10, minor=123)
[    7.522123] Initialized host personality
[    7.582927] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[   10.402408] NET: Registered PF_VSOCK protocol family
[   11.801636] vmxnet3 0000:03:00.0 ens160: intr type 3, mode 0, 5 vectors allocated
[   11.802670] vmxnet3 0000:03:00.0 ens160: NIC Link is Up 10000 Mbps
[   21.453030] floppy0: disk absent or changed during operation
[   21.453035] blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[   21.453148] floppy0: disk absent or changed during operation
[   21.453150] blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[   21.453183] Buffer I/O error on dev fd0, logical block 0, async page read
[   23.358198] loop8: detected capacity change from 0 to 8
[   87.641674] floppy0: disk absent or changed during operation
[   87.641680] blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[   87.641832] floppy0: disk absent or changed during operation
[   87.641835] blk_update_request: I/O error, dev fd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[   87.641881] Buffer I/O error on dev fd0, logical block 0, async page read
[ 3940.283328] sched: RT throttling activated
[ 4169.200882] Core dump to |/usr/share/apport/apport pipe failed

Previously we had Ubuntu 20 with postgresql version 12, and I think we also got this error. At that time we were not doing as many queries as we do now, as we have developed an automation project that generates a lot of queries. The automation we had previously was less in terms of queries and we did not give it importance....

[candlerb]

The numbers on the left are the time in seconds since system startup, so most of those are from boot-up (and you can ignore errors probing the virtual floppy drive, fd0). The only interesting ones are:

[ 3940.283328] sched: RT throttling activated
[ 4169.200882] Core dump to |/usr/share/apport/apport pipe failed

The last one shows that some application crashed and dumped core, but unfortunately doesn't say which. It could be postgres, or something else.

Since this is a problem which (apparently) affects only you, I think you'll need to push forward with debugging it yourself: find a REST API request and/or SQL request which allows you to reliably replicate the problem, then drill down.

Previously we had Ubuntu 20 with postgresql version 12, and I think we also got this error.

OK. It would be interesting to compare with the system running on a different virtualization platform or on bare metal, although since you see no I/O errors in dmesg, I don't think that the VM environment is likely to be the problem.

Have you done any tweaking to Ubuntu compared to an out-of-box install? You've not installed SELinux, for example? (Again, seems unlikely, in the absence of messages logged to dmesg)

It seems that either you've found some fundamental problem with Postgres handling specific queries, or a fundamental problem with the OS that Postgres is running on, and both sound equally unlikely.

Netbox/Postgres are definitely running in a "full fat" VM, not a Container? (vmxnet3 suggests it's VMware)

I don't think I can help I'm afraid, but good luck in your hunt.