Why is there no nerdctl0 network interface on host in rootless mode

[vsiravar]

Hi everyone,
I am trying to understand the difference in networking between rootless and rootful mode.

CNI config

{
  "cniVersion": "1.0.0",
  "name": "bridge",
  "nerdctlID": "17f29b073143d8cd97b5bbe492bdeffec1c5fee55cc1fe2112c8b9335f8b6121",
  "nerdctlLabels": {},
  "plugins": [
    {
      "type": "bridge",
      "bridge": "nerdctl0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "ranges": [
          [
            {
              "gateway": "10.4.0.1",
              "subnet": "10.4.0.0/24"
            }
          ]
        ],
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ],
        "type": "host-local"
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "firewall",
      "ingressPolicy": "same-bridge"
    },
    {
      "type": "tuning"
    }
  ]
}

Although the cni config enables the bridge to act as a gateway "isGateway": true, when I check for the network interface nerdctl0 I can't seem to find it on the host.

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.5.15  netmask 255.255.255.0  broadcast 192.168.5.255
        inet6 fe80::5055:55ff:fede:8ab2  prefixlen 64  scopeid 0x20<link>
        inet6 fec0::5055:55ff:fede:8ab2  prefixlen 64  scopeid 0x40<site>
        ether 52:55:55:de:8a:b2  txqueuelen 1000  (Ethernet)
        RX packets 258366  bytes 351742082 (351.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22268  bytes 1637645 (1.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 754  bytes 54169 (54.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 754  bytes 54169 (54.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

However in rootful mode I can see nerdctl0 network interface on the host. Trying to make sense of how the connection is made from the host to the container. Thanks!

[AkihiroSuda]

nerdctl0 is namespaced, as a non-root user can't gain the privilege to create nerdctl in the host netns

$ containerd-rootless-setuptool.sh nsenter -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b2:eb:29:26:a2:c7 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.100/24 scope global tap0
       valid_lft forever preferred_lft forever
    inet6 fe80::b0eb:29ff:fe26:a2c7/64 scope link 
       valid_lft forever preferred_lft forever
3: nerdctl0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a6:3b:04:94:f8:51 brd ff:ff:ff:ff:ff:ff
    inet 10.4.0.1/24 brd 10.4.0.255 scope global nerdctl0
       valid_lft forever preferred_lft forever
    inet6 fe80::a43b:4ff:fe94:f851/64 scope link 
       valid_lft forever preferred_lft forever
4: veth85843e25@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master nerdctl0 state UP group default 
    link/ether a6:d9:35:92:44:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::a4d9:35ff:fe92:4477/64 scope link 
       valid_lft forever preferred_lft forever

eth0 (in the container's netns) → vethXXX (in RootlessKit's netns) → nerdctl0 (in RootlessKit's netns) → tap0 (in RootlessKit's netns) → slirp4netns → e.g., etc0 (in the host's netns)

See also https://github.com/rootless-containers/slirp4netns/ https://github.com/rootless-containers/rootlesskit

[vsiravar]

Thanks for all the details. This helps a lot!

[vsiravar]

An interesting observation in rootful that I found was that the gateway tries to make a connection to a socket which was opened even before a user initiated connection. An example below

$ sudo tcpdump -v -i nerdctl0 -w /tmp/pcap0.log
$ sudo nerdctl run -it -p 11028:80 alpine

/ # echo hello | nc -l -p 80
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 7E:89:35:94:8A:51  
          inet addr:10.4.0.25  Bcast:10.4.0.255  Mask:255.255.255.0
          inet6 addr: fe80::7c89:35ff:fe94:8a51/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:127 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6126 (5.9 KiB)  TX bytes:978 (978.0 B)

The cni gateway(10.4.0.1) makes a connection with the container(10.4.0.25) when it starts without a user initiating a connection from the host.

• Packets 235 and 236 show the three-way handshake being initiated and completed, with packet 235 being the SYN packet sent by the client and packet 236 being the SYN-ACK packet sent by the server in response.
• Packet 237 shows the client acknowledging the server's SYN-ACK packet with an ACK packet.
• Packets 238 and 239 show the transfer of application layer data (“hello” from the server; 6 bytes) from the server to the client, with packet 238 being the PSH-ACK packet sent by the server and packet 239 being the ACK packet sent by the client in response.
• Packet 240 shows the server initiating connection termination by sending a FIN-ACK packet.
• Packet 241 shows the client responding to the server's FIN-ACK packet with a RST-ACK packet, indicating that the connection was abruptly closed by the client which causes the server to shutdown.