SR Linux fails to boot when doing sudo as a non-root user #1356

matthewjmiller1 · 2023-05-01T23:05:55Z

matthewjmiller1
May 1, 2023

I was trying this introductory example to get started with containerlab:
https://containerlab.dev/lab-examples/single-srl/

It comes up when I actually run as root: containerlab deploy --topo srl01.clab.yml.

However, if I run using sudo as a non-root user, it fails:

$ sudo clab deploy --topo srl01.clab.yml
INFO[0000] Containerlab v0.40.0 started                 
INFO[0000] Parsing & checking topology file: srl01.clab.yml 
INFO[0000] Creating lab directory: /u/matthew6/tmp/clab-quickstart/clab-user-srl01 
INFO[0000] Creating container: "srl"                    
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl' node 
ERRO[0121] clab-user-srl01-srl: failed to execute cmd: "sr_cli -d info from state system app-management application mgmt_server state | grep running" with error context deadline exceeded 
ERRO[0122] failed to run postdeploy task for node srl: timed out waiting for SR Linux node srl to boot: <nil> 
INFO[0122] Adding containerlab host entries to /etc/hosts file 
+---+---------------------+--------------+-----------------------+------+---------+----------------+----------------------+
| # |        Name         | Container ID |         Image         | Kind |  State  |  IPv4 Address  |     IPv6 Address     |
+---+---------------------+--------------+-----------------------+------+---------+----------------+----------------------+
| 1 | clab-user-srl01-srl | c5f1c77d0e77 | ghcr.io/nokia/srlinux | srl  | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
+---+---------------------+--------------+-----------------------+------+---------+----------------+----------------------+
$

When I ran with more verbose logging, this is the message that repeatedly fails until a timeout:

DEBU[0095] /clab-srl01-srl exec created clab-srl01-srl  
DEBU[0095] /clab-srl01-srl exec attached clab-srl01-srl 
DEBU[0096] error during checking SR Linux boot status: 22:08:04.753 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds
Error: Server is not running

When I compare the docker logs for the two cases, in the sudo/non-root case I see these messages that aren't there in the root case:

setfacl: /etc/opt/srlinux/ztp: Operation not supported
setfacl: /etc/opt/srlinux/ztp: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux: Operation not supported
setfacl: /etc/opt/srlinux/ztp: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/cli: Operation not supported
setfacl: /etc/opt/srlinux/cli/plugins: Operation not supported
setfacl: /etc/opt/srlinux/devices: Operation not supported
setfacl: /etc/opt/srlinux/eventmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux: Operation not supported
setfacl: /etc/opt/srlinux/ztp: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/cli: Operation not supported
setfacl: /etc/opt/srlinux/cli/plugins: Operation not supported
setfacl: /etc/opt/srlinux/devices: Operation not supported
setfacl: /etc/opt/srlinux/eventmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux: Operation not supported
setfacl: /etc/opt/srlinux/ztp: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/cli: Operation not supported
setfacl: /etc/opt/srlinux/cli/plugins: Operation not supported
setfacl: /etc/opt/srlinux/devices: Operation not supported
setfacl: /etc/opt/srlinux/eventmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux: Operation not supported
setfacl: /etc/opt/srlinux/ztp: Operation not supported
setfacl: /etc/opt/srlinux/warmboot: Operation not supported
setfacl: /etc/opt/srlinux/cli: Operation not supported
setfacl: /etc/opt/srlinux/cli/plugins: Operation not supported
setfacl: /etc/opt/srlinux/devices: Operation not supported
setfacl: /etc/opt/srlinux/eventmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/appmgr: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported
setfacl: /etc/opt/srlinux/tls: Operation not supported

I'm using Ubuntu 22.04 with Docker 23.0.5 and containerlab 0.40.0.

Is there anyway I can run this doing sudo as a non-root user?

hellt · 2023-05-02T07:20:08Z

hellt
May 2, 2023
Maintainer

Hi @matthewjmiller1
Can you try adding -E flag to sudo:

sudo -E containerlab...

5 replies

matthewjmiller1 May 2, 2023
Author

That still doesn't work (gives the same result).

Looking further in the failed container logs, I see:

Log directory is /var/log/srlinux/stdout
  Found supportd: PID 508
  Application supportd is running: PID 508
  Started dev_mgr: source /etc/profile.d/sr_app_env.sh &>/dev/null; bash -c "./sr_device_mgr" >/var/log/srlinux/stdout/dev_mgr.log 2>&1 &
  Started dev_mgr: source /etc/profile.d/sr_app_env.sh &>/dev/null; bash -c "./sr_device_mgr" >/var/log/srlinux/stdout/dev_mgr.log 2>&1 &
  Application dev_mgr is running: PID 1517
  Started dev_mgr: source /etc/profile.d/sr_app_env.sh &>/dev/null; bash -c "./sr_device_mgr" >/var/log/srlinux/stdout/dev_mgr.log 2>&1 &
  Application dev_mgr is running: PID 1876
23-05-02 14:12:29.801 sr_app_mgr: Would reboot with reason 'AppMgr: dev_mgr has failed' but no-reboot is set

Looking at the container:

[root@srl /]# cat /var/log/srlinux/stdout/dev_mgr.log
23-05-02 14:12:24.886868  1876 C common    device_mgr_early_init.cc:350   SRL_ASSERT_MSG  Termination handler
========= SRL_ASSERT_MSG in program: sr_device_mgr =========
=== At: /builds/sr/srlinux/devicemgr/device_mgr_early_init.cc:350
=== Condition failed: (false)
=== Failed to open file /etc/opt/srlinux/devices/hw_details.json

 0x819ec9, ./sr_device_mgr : srlinux::dev_mgr::DeviceManagerEarlyInit::WriteHwDetailsToFile()+0x17b9
 0x7dbce7, ./sr_device_mgr : srlinux::DeviceManager::DoEarlyInit()+0xf7
 0x7dffc3, ./sr_device_mgr : srlinux::DeviceManager::Start(ELoopHandle*)+0x33
 0x7c32e5, ./sr_device_mgr : Start(srlinux::dev_mgr::GlobalOptions*)+0x75
 0x7c8da8, ./sr_device_mgr : main()+0x448
 0x7f117eef5d85, /lib64/libc.so.6 : __libc_start_main()+0xe5
 0x7c310e, ./sr_device_mgr : _start()+0x2e
==============================
=== Program: sr_device_mgr
/builds/sr/srlinux/devicemgr/device_mgr_early_init.cc:350 | condition failed (false) | Failed to open file /etc/opt/srlinux/devices/hw_details.json
*
* Limited COREDUMP report, without core file. /proc/sys/kernel/core_pattern does not have installed sr_host_core_catcher.sh
*

sr_device_mgr: [1876/1876] Direct call to sr_core_dump.sh *****.

/opt/srlinux/bin/sr_core_dump.sh: line 137:  1898 Killed                  /usr/bin/sleep 17
0230502_141224-1876-6 

2023-05-02 14:12:26 sr_core_dump.sh: generated coredump for "sr_device_mgr[no_core]" by direct call at "/var/core/coredump-sr_device_mgr[no_core]-20230502_141224-1876-6.tar.gz" (2 sec)
[root@srl /]# ls -la /etc/opt/srlinux/devices/
total 8
drwxr-xr-x 2 root    root    4096 May  2 14:12 .
drwxrwxrwx 9 srlinux srlinux 4096 May  2 14:12 ..
[root@srl /]#

In the containerlab deploy I did as root, I see that hw_details.json file in the container:

[root@srl /]# cat /etc/opt/srlinux/devices/hw_details.json
{
 "chassis_type": 66,
 "chassis_base_mac": {
  "mac_address": "AAAAAAAA"
 },
 "chassis_serial_number": "",
 "is_active": true,
 "card_details": [],
 "cpm_slot_by_letter": "A",
 "is_singleru": true,
 "slot_num": 0,
 "cpm_card_type": 177,
 "ztp_interfaces": [],
 "chassis_name": "7220 IXR-D3",
 "card_serial_number": ""
}
[root@srl /]#

Any ideas why that file may exist in the container when running as root but not sudo?

hellt May 3, 2023
Maintainer

This is strange. We run CI tests in gh actions for different repos where containerlab is started with sudo, and it works just fine https://github.com/nokia/srlinux-ansible-collection/actions/runs/4851949745/jobs/8646342879#step:5:45

LimeHat May 5, 2023
Maintainer

@matthewjmiller1 maybe you have some additional security mechanisms on this system enabled? selinux, apparmor, etc?

What kind of system is this (baremetal, vm, cloud vm, ...)?

LimeHat May 5, 2023
Maintainer

Also make sure you have cgroups v2 enabled

matthewjmiller1 May 8, 2023
Author

This is a VM deployed within my company using one of their standard images.

It looks like selinux is disabled and cgroups v2 is enabled, but apparmor is enabled. So, maybe that is what's causing this behavior.

$ sestatus
SELinux status:                 disabled
$ sudo aa-status
apparmor module is loaded.
35 profiles are loaded.
35 profiles are in enforce mode.
   /snap/snapd/16010/usr/lib/snapd/snap-confine
   /snap/snapd/16010/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /snap/snapd/17336/usr/lib/snapd/snap-confine
   /snap/snapd/17336/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /snap/snapd/19122/usr/lib/snapd/snap-confine
   /snap/snapd/19122/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/chronyd
   /{,usr/}sbin/dhclient
   docker-default
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   snap-update-ns.lxd
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
   snap.lxd.user-daemon
   tcpdump
0 profiles are in complain mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/chronyd (42038) 
   /usr/sbin/chronyd (42040) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.
$ grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2
$

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SR Linux fails to boot when doing sudo as a non-root user #1356

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

SR Linux fails to boot when doing sudo as a non-root user #1356

Uh oh!

matthewjmiller1 May 1, 2023

Replies: 1 comment · 5 replies

Uh oh!

hellt May 2, 2023 Maintainer

Uh oh!

Uh oh!

matthewjmiller1 May 2, 2023 Author

Uh oh!

hellt May 3, 2023 Maintainer

Uh oh!

Uh oh!

LimeHat May 5, 2023 Maintainer

Uh oh!

LimeHat May 5, 2023 Maintainer

Uh oh!

matthewjmiller1 May 8, 2023 Author

matthewjmiller1
May 1, 2023

Replies: 1 comment 5 replies

hellt
May 2, 2023
Maintainer

matthewjmiller1 May 2, 2023
Author

hellt May 3, 2023
Maintainer

LimeHat May 5, 2023
Maintainer

LimeHat May 5, 2023
Maintainer

matthewjmiller1 May 8, 2023
Author