Error while running tracee-ebpf on Debian 11 .

[4LPH4-Z3R0]

I'm trying to run a stand alone tracee-ebpf process on debian 11. However, there are some errors which seems relating to privilege issues. Here are some messages for reproducing the scenario:

$ cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ cat /proc/version

Linux version 5.10.0-17-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.136-1 (2022-08-13)

(root) # ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 257516
max locked memory           (kbytes, -l) 8245095
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 257516
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

$ sudo ./dist/tracee-ebpf --version
Tracee version "v0.8.1"

$ sudo ./dist/tracee-ebpf

TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
libbpf: kprobe perf_event_open() failed: Permission denied
libbpf: prog 'trace_security_inode_unlink': failed to create kprobe 'security_inode_unlink+0x0' perf event: Permission denied

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}
2022/08/17 21:06:55 error initializing Tracee: failed to attach required probe: failed to attach event: security_inode_unlink (failed to attach security_inode_unlink k(ret)probe to program trace_security_inode_unlink: permission denied)

$ sudo ./dist/tracee-ebpf --debug

OSInfo: ID: debian
OSInfo: KERNEL_RELEASE: 5.10.0-17-amd64
OSInfo: ARCH: x86_64
OSInfo: PRETTY_NAME: "Debian GNU/Linux 11 (bullseye)"
OSInfo: VERSION_ID: "11"
OSInfo: VERSION: "11 (bullseye)"
OSInfo: VERSION_CODENAME: bullseye
RuntimeSockets: registered default containerd runtime socket from /var/run/containerd/containerd.sock
RuntimeSockets: registered default docker runtime socket from /var/run/docker.sock
RuntimeSockets: failed to register default crio socket:
failed to register runtime socket stat /var/run/crio/crio.sock: no such file or directory
RuntimeSockets: failed to register default podman socket:
failed to register runtime socket stat /var/run/podman/podman.sock: no such file or directory
OSInfo: Security Lockdown is 'none'
BTF: bpfenv = false, btfenv = false, vmlinux = true
BPF: using embedded BPF object
unpacked CO:RE bpf object file into memory
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
Enricher: error registering enricher: unsupported runtime crio
libbpf: prog 'tc_egress': BPF program load failed: Operation not permitted
libbpf: permission error while running as root; try raising 'ulimit -l'? current value: 512.0 MiB
libbpf: failed to load program 'tc_egress'
libbpf: failed to load object 'embedded-core'

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}
2022/08/17 21:07:16 error initializing Tracee: failed to load BPF object

And for comparison, I can successfully start the official Docker image as follow:

$ docker run  --name tracee --rm -it  --pid=host --cgroupns=host --privileged   -v /etc/os-release:/etc/os-release-host:ro  -e LIBBPFGO_OSRELEASE_FILE=/etc/os-release-host  -e TRACEE_EBPF_ONLY=1   aquasec/tracee:0.8.1

KConfig: warning: could not check enabled kconfig features
(could not read /boot/config-5.10.0-17-amd64: stat /boot/config-5.10.0-17-amd64: no such file or directory)
KConfig: warning: assuming kconfig values, might have unexpected behavior
TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
13:09:19:061502  0      systemd-udevd    408     408     0                kill                 pid: 9858, sig: 15
13:09:19:061596  0      systemd-udevd    408     408     0                kill                 pid: 9867, sig: 15
13:09:19:061647  0      systemd-udevd    408     408     0                kill                 pid: 9859, sig: 15
13:09:19:061951  0      systemd-udevd    9858    9858    0                close                fd: 12
13:09:19:062228  0      systemd-udevd    9858    9858    0                close                fd: 6

So, what is the reason of the errors when running tracee-ebpf on the Host OS? How to fix it?

Thank you for help !

[yanivagman]

It looks like the errors you get with and without debug flag enabled are different.
I think this is because we enable network events debug when the debug flag is enabled, while we shouldn't if no network related feature was chosen (either network event or capture). Tracee will drop the CAP_NET_ADMIN in that case (no net event, but debug enabled), and cause this error (libbpf: prog 'tc_egress': BPF program load failed: Operation not permitted) - this is a bug!

About the first error you got libbpf: prog 'trace_security_inode_unlink': failed to create kprobe 'security_inode_unlink+0x0' perf event: Permission denied, we saw this already when CAP_SYS_ADMIN is dropped and CAP_BPF+CAP_PERMON is used instead. I think it is related to this PR: #2033 which tried to solve #1815.

Can you please cat /proc/sys/kernel/perf_event_paranoid in your environment?
@cdelzotti maybe you can have a look at this as well?

[rafaeldtinoco]

I think this is because we enable network events debug when the debug flag is enabled, while we shouldn't if no network related feature was chosen (either network event or capture). Tracee will drop the CAP_NET_ADMIN in that case (no net event, but debug enabled), and cause this error (libbpf: prog 'tc_egress': BPF program load failed: Operation not permitted) - this is a bug!

FYI, I intend to remove the debug_net "feature" with the new networking code (still keep a way to debug but with the payload and the event themselves). SO this "capability issue" when enabling the networking debug code would be gone.

[cdelzotti]

I still have to look further into this, but the <= 3 threshold for the perf_event_paranoid was set with the particular case of Ubuntu in mind.

For reminder :

/*
 * perf event paranoia level:
 *  -1 - not paranoid at all
 *   0 - disallow raw tracepoint access for unpriv
 *   1 - disallow cpu events for unpriv
 *   2 - disallow kernel profiling for unpriv
 */

In the case of Ubuntu, it worked with a value of < 4, but it is Ubuntu-specific as it's outside of the scope of the kernel documentation. I think the problem here is that we're in a situation where any distribution can do whatever it wants with values above 2. For instance, Debian seems to consider the 3 value here : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994044.

I think we should lower the threshold to 2 (or less if we want to allow running Tracee in non-root mode as discussed in #1946) in order to say within kernel specifications which should (hopefully) be the same for all distribution.

[4LPH4-Z3R0]

Thank you so much for your help!

For further information:

$ cat /proc/sys/kernel/perf_event_paranoid
3

[yanivagman]

Thank you so much for your help!

For further information:

$ cat /proc/sys/kernel/perf_event_paranoid
3

This must be it.
So what @cdelzotti suggested will work if we set the threshold to <=2 instead of <=3. Until we will release such a fix, you can echo 2 to this file, and I believe it should solve the problem

[4LPH4-Z3R0]

Yes, it is exactly as your suggestion: the problem was solved after I change this value from 3 to 2.

Thank you for help!

[yanivagman]

By the way, you can also try to use the --allow-high-capabilities flag.
We added it to workaround these exact cases.

[yanivagman]

By the way, you can also try to use the --allow-high-capabilities flag. We added it to workaround these exact cases.

Actually this was addressed by #2008 with the new --caps cancel-drop flag (which is not available in v0.8.1 and will be part of v0.8.2)

[4LPH4-Z3R0]

Thank you for your help!

I've tried your suggestions, and the results are as follow:

$ sudo ./dist/tracee-ebpf --allow-high-capabilities

TIME             UID    COMM             PID     TID     RET              EVENT                ARGS
libbpf: kprobe perf_event_open() failed: Permission denied
libbpf: prog 'trace_load_elf_phdrs': failed to create kprobe 'load_elf_phdrs+0x0' perf event: Permission denied
libbpf: kprobe perf_event_open() failed: Permission denied
libbpf: prog 'trace_security_sb_mount': failed to create kprobe 'security_sb_mount+0x0' perf event: Permission denied

End of events stream
Stats: {EventCount:0 NetEvCount:0 ErrorCount:0 LostEvCount:0 LostWrCount:0 LostNtCount:0}
2022/08/19 17:37:39 error initializing Tracee: failed to attach required probe: failed to attach event: security_sb_mount (failed to attach security_sb_mount k(ret)probe to program trace_security_sb_mount: permission denied)

Since the --caps cancel-drop flag is not present in this version (0.8.1), I will try it when the 0.8.2 is released.

For more information, I've tried to reproduce this problem on an Ubuntu 22.04 server (kernel 5.15.0-46-generic), and all the problems are disappeared, the tracee-ebpf program works normally, as follow:

$ sudo ./dist/tracee-ebpf
17:41:49:627790  0      irqbalance       887     887     0                security_file_open   pathname: /proc/interrupts, flags: O_RDONLY|O_LARGEFILE, dev: 22, inode: 4026532023, ctime: 1660900889104000000, syscall_pathname: /proc/interrupts
17:41:49:627724  0      irqbalance       887     887     6                openat               dirfd: -100, pathname: /proc/interrupts, flags: , mode: 0
17:41:49:629244  0      irqbalance       887     887     0                close                fd: 6
17:41:49:629344  0      irqbalance       887     887     0                security_file_open   pathname: /proc/stat, flags: O_RDONLY|O_LARGEFILE, dev: 22, inode: 4026532026, ctime: 1660900889128000000, syscall_pathname: /proc/stat
17:41:49:629324  0      irqbalance       887     887     6                openat               dirfd: -100, pathname: /proc/stat, flags: , mode: 0
17:41:49:629583  0      irqbalance       887     887     0                close                fd: 6
17:41:49:629866  1000   tmux: server     1435    1435    0                security_file_open   pathname: /proc/1802/cmdline, flags: O_RDONLY|O_LARGEFILE, dev: 22, inode: 30091, ctime: 1660902106107605813, syscall_pathname: /proc/1802/cmdline
17:41:49:629812  1000   tmux: server     1435    1435    9                openat               dirfd: -100, pathname: /proc/1802/cmdline, flags: , mode: 0

I'm not clear about the reasons (errors on debian 11 and working proper on ubuntu 22.04), is it because some issues of the different kernel version?

[rafaeldtinoco]

@4LPH4-Z3R0 yes, like @cdelzotti said, Ubuntu kernel has a patch that considers a value from /proc/sys/kernel/perf_event_paranoid differently then other distros. @cdelzotti OSInfo package contains information about os-release being used for things like that (and kernel version comes from the same package). We just have to decide if we want to "if ubuntu do this, else do that" approach (or we want something that works for all, which Im more in favor of).

[yanivagman]

@rafaeldtinoco I think the only change needed to fix this is replacing 3 with 2 in this line: https://github.com/aquasecurity/tracee/blob/main/cmd/tracee-ebpf/capabilities.go#L136

[cdelzotti]

See #2077 :)