cgroup: ignore redundant deny dev cgroup rules #1591

giuseppe · 2024-10-25T12:26:04Z

Closes: #1588

giuseppe · 2024-10-25T12:26:21Z

still a Draft as I am not 100% sure yet about the fix

packit-as-a-service · 2024-10-25T12:37:28Z

podman system tests failed. @containers/packit-build please check.

git clean -xdf git submodule sync --recursive git submodule update --recursive git submodule foreach --recursive git clean -xdf tar zcvf ../crun_1.18.orig.tar.gz --exclude=.git . debuild -uc -us cp crun.spec ../crun_1.18-1.spec cp ../crun*1.18*.{gz,xz,spec,dsc} /osc/home\:alvistack/containers-crun-1.18/ rm -rf ../crun*1.18*.* See containers#1591 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

hswong3i · 2024-10-26T04:38:37Z

@giuseppe with Ubuntu 20.04 + Kubernetes 1.31.2 + CRI-O 1.31.1 + crun 1.18 (#1591 patched), with following error message:

Oct 26 04:30:47 kube71-sg crio[3061]: time="2024-10-26 04:30:47.284188993Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=f7825541-d097-4b2f-9c2a-e381b542c3d7 name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:47 kube71-sg crio[3061]: time="2024-10-26 04:30:47.318009718Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=e2d83aff-5ed6-46b2-8261-40c24a73863e name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:47 kube71-sg crio[3061]: time="2024-10-26 04:30:47.355494268Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=46e0b52f-090b-49e0-ba55-83c89c37c9a1 name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:48 kube71-sg crio[3061]: time="2024-10-26 04:30:48.285463329Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=6e36f318-229c-4812-a348-e216082e6957 name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:48 kube71-sg crio[3061]: time="2024-10-26 04:30:48.286444998Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=a05d14e1-ac4b-4d16-935e-9c9eb36fadbd name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:49 kube71-sg crio[3061]: time="2024-10-26 04:30:49.265868792Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/3c1e0d3e492b4e255f2153f727f8ca1d37f66c7782cfc4ab454380cb63b3eba7/merged/etc/passwd: no such file or directory"
Oct 26 04:30:49 kube71-sg crio[3061]: time="2024-10-26 04:30:49.265933253Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/3c1e0d3e492b4e255f2153f727f8ca1d37f66c7782cfc4ab454380cb63b3eba7/merged/etc/group: no such file or directory"
Oct 26 04:30:49 kube71-sg crio[3061]: time="2024-10-26 04:30:49.284000889Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=f0afbbd5-4942-4bab-8f3a-d06001e22e6e name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:51 kube71-sg crio[3061]: time="2024-10-26 04:30:51.261278380Z" level=warning msg="Failed to open /etc/passwd: open /var/lib/containers/storage/overlay/8445931f916d1b74f0dcf70ec9cf083aee8b2574a6218d1498d3a32cbab50754/merged/etc/passwd: no such file or directory"
Oct 26 04:30:51 kube71-sg crio[3061]: time="2024-10-26 04:30:51.261372832Z" level=warning msg="Failed to open /etc/group: open /var/lib/containers/storage/overlay/8445931f916d1b74f0dcf70ec9cf083aee8b2574a6218d1498d3a32cbab50754/merged/etc/group: no such file or directory"
Oct 26 04:30:51 kube71-sg crio[3061]: time="2024-10-26 04:30:51.278357075Z" level=error msg="Container creation error: writing file `devices.allow`: Operation not permitted\n" id=6e9f942f-af12-4fa6-b28b-e82641fc0413 name=/runtime.v1.RuntimeService/CreateContainer
Oct 26 04:30:52 kube71-sg crio[3061]: level=warning msg="Failed to connect to agent socket at unix:///var/run/cilium/cilium.sock." containerID=18d9de178307f861a5b2543606bbdd3e27a486a27fbdc8b574ec094c4986af9c error="failed to create cilium agent client after 10.000000 seconds timeout: Get \"http://localhost/v1/config\": dial unix /var/run/cilium/cilium.sock: connect: connection refused" eventUUID=1193a6f1-d8b4-4622-bc5d-8340a9c61c08 subsys=cilium-cni
Oct 26 04:30:52 kube71-sg crio[3061]: level=info msg="Agent is down, falling back to deletion queue directory" containerID=18d9de178307f861a5b2543606bbdd3e27a486a27fbdc8b574ec094c4986af9c eventUUID=1193a6f1-d8b4-4622-bc5d-8340a9c61c08 subsys=cilium-cni
Oct 26 04:30:52 kube71-sg crio[3061]: level=info msg="Queueing deletion request for endpoint" containerID=18d9de178307f861a5b2543606bbdd3e27a486a27fbdc8b574ec094c4986af9c endpointID="container-id:18d9de178307f861a5b2543606bbdd3e27a486a27fbdc8b574ec094c4986af9c" eventUUID=1193a6f1-d8b4-4622-bc5d-8340a9c61c08 subsys=cilium-cni
Oct 26 04:30:52 kube71-sg crio[3061]: level=info msg="wrote queued deletion file" containerID=18d9de178307f861a5b2543606bbdd3e27a486a27fbdc8b574ec094c4986af9c eventUUID=1193a6f1-d8b4-4622-bc5d-8340a9c61c08 subsys=cilium-cni
Oct 26 04:30:53 kube71-sg crio[3061]: level=warning msg="Failed to connect to agent socket at unix:///var/run/cilium/cilium.sock." containerID=68435fc9f6de40f79d09f91175292ff9f9bd4d8b2f01d274d0c8bcd71e0eaacc error="failed to create cilium agent client after 10.000000 seconds timeout: Get \"http://localhost/v1/config\": dial unix /var/run/cilium/cilium.sock: connect: connection refused" eventUUID=8142fe44-6c35-4fad-a05c-0cf54a1586c0 subsys=cilium-cni
Oct 26 04:30:53 kube71-sg crio[3061]: level=info msg="Agent is down, falling back to deletion queue directory" containerID=68435fc9f6de40f79d09f91175292ff9f9bd4d8b2f01d274d0c8bcd71e0eaacc eventUUID=8142fe44-6c35-4fad-a05c-0cf54a1586c0 subsys=cilium-cni
Oct 26 04:30:53 kube71-sg crio[3061]: level=info msg="Queueing deletion request for endpoint" containerID=68435fc9f6de40f79d09f91175292ff9f9bd4d8b2f01d274d0c8bcd71e0eaacc endpointID="container-id:68435fc9f6de40f79d09f91175292ff9f9bd4d8b2f01d274d0c8bcd71e0eaacc" eventUUID=8142fe44-6c35-4fad-a05c-0cf54a1586c0 subsys=cilium-cni
Oct 26 04:30:53 kube71-sg crio[3061]: level=info msg="wrote queued deletion file" containerID=68435fc9f6de40f79d09f91175292ff9f9bd4d8b2f01d274d0c8bcd71e0eaacc eventUUID=8142fe44-6c35-4fad-a05c-0cf54a1586c0 subsys=cilium-cni

cgroup related information:

root@kube71-sg:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

root@kube71-sg:~# grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2

root@kube71-sg:~# stat -fc %T /sys/fs/cgroup/
tmpfs

root@kube71-sg:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT="0"
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Closes: containers#1588 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

git clean -xdf git submodule sync --recursive git submodule update --recursive git submodule foreach --recursive git clean -xdf tar zcvf ../crun_1.18.orig.tar.gz --exclude=.git . debuild -uc -us cp crun.spec ../crun_1.18-1.spec cp ../crun*1.18*.{gz,xz,spec,dsc} /osc/home\:alvistack/containers-crun-1.18/ rm -rf ../crun*1.18*.* See containers#1591 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

Closes: containers#1589 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe · 2024-10-29T11:41:53Z

@hswong3i added a patch that should fix the issue you've seen in #1589

git clean -xdf git submodule sync --recursive git submodule update --recursive git submodule foreach --recursive git clean -xdf tar zcvf ../crun_1.18.orig.tar.gz --exclude=.git . debuild -uc -us cp crun.spec ../crun_1.18-1.spec cp ../crun*1.18*.{gz,xz,spec,dsc} /osc/home\:alvistack/containers-crun-1.18/ rm -rf ../crun*1.18*.* See containers#1591 Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

giuseppe · 2024-10-29T13:37:08Z

@hswong3i if you can confirm if it works for you, I'll cut a new release after it is merged

hswong3i · 2024-10-29T14:26:05Z

@giuseppe Confirm crun 1.18 + this PR could fix with Ubuntu 20.04, see:

The installed result:

root@kube71-sg:/tmp# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

root@kube71-sg:~# grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2

root@kube71-sg:~# stat -fc %T /sys/fs/cgroup/
tmpfs

root@kube71-sg:/tmp# crun --version
crun version 1.18
commit: 8656b2548509fcc69ea7e8823a870564360a57a1
rundir: /run/user/0/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

root@kube71-sg:/tmp# kubectl get pod --all-namespaces 
NAMESPACE             NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager          cert-manager-5c887c889d-dgrc8              1/1     Running   0          3m1s
cert-manager          cert-manager-cainjector-58f6855565-6d2gt   1/1     Running   0          3m1s
cert-manager          cert-manager-webhook-6647d6545d-lbdsq      1/1     Running   0          3m1s
csi-cephfs            csi-cephfs-provisioner-775856b697-9dx9b    5/5     Running   0          3m1s
csi-cephfs            csi-cephfsplugin-wzszr                     3/3     Running   0          3m
ingress-nginx         ingress-nginx-controller-x8xrt             1/1     Running   0          2m40s
kube-system           cilium-hpskl                               1/1     Running   0          2m58s
kube-system           cilium-node-init-nnc5q                     1/1     Running   0          3m
kube-system           cilium-operator-84b6986cfc-k59dj           1/1     Running   0          3m1s
kube-system           coredns-7c65d6cfc9-dpl2w                   1/1     Running   0          3m1s
kube-system           coredns-7c65d6cfc9-r2ftc                   1/1     Running   0          3m1s
kube-system           kube-addon-manager-kube71-sg               1/1     Running   14         3m1s
kube-system           kube-apiserver-kube71-sg                   1/1     Running   3          3m1s
kube-system           kube-controller-manager-kube71-sg          1/1     Running   3          3m1s
kube-system           kube-proxy-27nm5                           1/1     Running   0          3m
kube-system           kube-scheduler-kube71-sg                   1/1     Running   3          3m
kube-system           snapshot-controller-9b8b6765d-2jhc7        1/1     Running   0          3m1s

giuseppe · 2024-10-29T14:47:56Z

thanks for the confirmation.

@rhatdan @flouthoc PTAL

rhatdan · 2024-10-29T18:30:53Z

LGTM

This was referenced Oct 25, 2024

[1.18] Ubuntu 20.04 + Kubernetes 1.31.2: Container creation error: writing file devices.allow: Operation not permitted #1589

Closed

Docker 27.3.1 cannot start containers using crun 1.18 #1588

Closed

giuseppe force-pushed the fix-cgroup-devices branch from 8cfc530 to 6b661e6 Compare October 25, 2024 15:21

giuseppe force-pushed the fix-cgroup-devices branch from 6b661e6 to 170febe Compare October 28, 2024 09:24

giuseppe added 2 commits October 28, 2024 10:29

cgroup: ignore redundant deny dev cgroup rules

8a30a57

Closes: containers#1588 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

cgroup, systemd: ignore rules before a default deny one

77e4233

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the fix-cgroup-devices branch from 170febe to 77e4233 Compare October 28, 2024 09:29

giuseppe marked this pull request as ready for review October 28, 2024 13:32

cgroup: skip DevicePolicy if all devices are allowed

7c4a3f9

Closes: containers#1589 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

rhatdan merged commit 056a407 into containers:main Oct 29, 2024
57 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cgroup: ignore redundant deny dev cgroup rules #1591

cgroup: ignore redundant deny dev cgroup rules #1591

Uh oh!

giuseppe commented Oct 25, 2024

Uh oh!

giuseppe commented Oct 25, 2024

Uh oh!

packit-as-a-service bot commented Oct 25, 2024

Uh oh!

hswong3i commented Oct 26, 2024

Uh oh!

giuseppe commented Oct 29, 2024

Uh oh!

giuseppe commented Oct 29, 2024

Uh oh!

hswong3i commented Oct 29, 2024

Uh oh!

giuseppe commented Oct 29, 2024

Uh oh!

rhatdan commented Oct 29, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cgroup: ignore redundant deny dev cgroup rules #1591

cgroup: ignore redundant deny dev cgroup rules #1591

Uh oh!

Conversation

giuseppe commented Oct 25, 2024

Uh oh!

giuseppe commented Oct 25, 2024

Uh oh!

packit-as-a-service bot commented Oct 25, 2024

Uh oh!

hswong3i commented Oct 26, 2024

Uh oh!

giuseppe commented Oct 29, 2024

Uh oh!

giuseppe commented Oct 29, 2024

Uh oh!

hswong3i commented Oct 29, 2024

Uh oh!

giuseppe commented Oct 29, 2024

Uh oh!

rhatdan commented Oct 29, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants