Multipass assigns the same IP address to multiple VMs when used by snapcraft

[geekorous]

Describe the bug
Describe what your problem is.
When attempting to run multiple snapcraft builds concurrently, the spawned VMs are allocated the same IP address by the bridge, resulting in the crash of both VMs.

Expected behavior
What did you expect to happen?
Assign the last unassigned address to each VM in a reliable manner.

Logs
Please provide logs from the daemon, see accessing logs on where to find them on your platform.

I guess this portion of the log is more important:

Dec 05 21:18:23 lamborghini dnsmasq-dhcp[3981]: DHCPDISCOVER(mpqemubr0) 52:54:00:97:a4:f9
Dec 05 21:18:23 lamborghini dnsmasq-dhcp[3981]: DHCPOFFER(mpqemubr0) 10.30.202.107 52:54:00:97:a4:f9
Dec 05 21:18:23 lamborghini dnsmasq-dhcp[3981]: DHCPDISCOVER(mpqemubr0) 52:54:00:97:a4:f9
Dec 05 21:18:23 lamborghini dnsmasq-dhcp[3981]: DHCPOFFER(mpqemubr0) 10.30.202.107 52:54:00:97:a4:f9
Dec 05 21:18:23 lamborghini dnsmasq-dhcp[3981]: DHCPREQUEST(mpqemubr0) 10.30.202.107 52:54:00:97:a4:f9
Dec 05 21:18:23 lamborghini dnsmasq-dhcp[3981]: DHCPACK(mpqemubr0) 10.30.202.107 52:54:00:97:a4:f9 snapcraft-app-1
Dec 05 21:18:25 lamborghini multipassd[2303]: QMP: {"timestamp": {"seconds": 1638735505, "microseconds": 227385}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/dev>
Dec 05 21:18:25 lamborghini dnsmasq-dhcp[3981]: DHCPDISCOVER(mpqemubr0) 52:54:00:9b:5f:12
Dec 05 21:18:25 lamborghini dnsmasq-dhcp[3981]: DHCPOFFER(mpqemubr0) 10.30.202.107 52:54:00:9b:5f:12
Dec 05 21:18:25 lamborghini dnsmasq-dhcp[3981]: DHCPREQUEST(mpqemubr0) 10.30.202.107 52:54:00:9b:5f:12
Dec 05 21:18:25 lamborghini dnsmasq-dhcp[3981]: DHCPACK(mpqemubr0) 10.30.202.107 52:54:00:9b:5f:12 snapcraft-app-2

Additional info

• OS: [e.g. macOS 10.15]
• multipass version: 1.8.0
• multipass info --all

Name:           snapcraft-app-2
State:          Unknown
IPv4:           --
Release:        --
Image hash:     7c5c8f24046c (Ubuntu Snapcraft builder for Core 18)
Load:           --
Disk usage:     --
Memory usage:   --
Mounts:         --

Name:           snapcraft-app-1
State:          Running
IPv4:           10.30.202.140
Release:        Ubuntu 18.04.5 LTS
Image hash:     7c5c8f24046c (Ubuntu Snapcraft builder for Core 18)
Load:           0.08 0.06 0.02
Disk usage:     953.6M out of 19.2G
Memory usage:   48.7M out of 5.8G
Mounts:         --

[townsend2010]

Hey @geekorous,

This looks really strange. It kind of looks like dnsmasq didn't do the right thing. Is this easily reproducible for you or is it a one off? Also, I think it would be helpful if you could provide more of the logs and also provide the contents of /var/snap/multipass/common/data/multipassd/network/dnsmasq.leases.

Thanks!

[geekorous]

Hey @townsend2010,

I could easily reproduce the issue from scratch. I create two directories, snap-1 and snap-2, on both I run snapcraft init. I just edit the name of the snaps to come up with simple snapcraft files like this:

name: snap-1 # you probably want to 'snapcraft register <name>'
base: core18 # the base snap is the execution environment for this snap
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil

and similarly:

name: snap-2 # you probably want to 'snapcraft register <name>'
base: core18 # the base snap is the execution environment for this snap
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil

Then I run snapcraft in each directory. Running multipass list hangs up and doesn't respond. However, since I couldn't get useful info from QEMU backend in a predicted manner, I used LXD as the backend, so I could retrieve the info from lxc list --project multipass:

+------------------+---------+----------------------+---------------------------------------------+-----------------+-----------+
|       NAME       |  STATE  |         IPV4         |                    IPV6                     |      TYPE       | SNAPSHOTS |
+------------------+---------+----------------------+---------------------------------------------+-----------------+-----------+
| snapcraft-snap-1 | RUNNING | 10.48.209.209 (eth0) | fd42:d52f:dc55:5f8:5054:ff:fe35:9726 (eth0) | VIRTUAL-MACHINE | 0         |
+------------------+---------+----------------------+---------------------------------------------+-----------------+-----------+
| snapcraft-snap-2 | RUNNING | 10.48.209.209 (eth0) | fd42:d52f:dc55:5f8:5054:ff:fe7a:bd9 (eth0)  | VIRTUAL-MACHINE | 0         |
+------------------+---------+----------------------+---------------------------------------------+-----------------+-----------+

Depending on the progress of the first build when you execute the second one, either you receive:

An error occurred when trying to execute 'sudo -H -i env SNAPCRAFT_BUILD_ENVIRONMENT=managed-host HOME=/root SNAPCRAFT_HAS_TTY=True apt-get dist-upgrade --yes' with 'multipass': returned exit code 255.

or it stalls forever. In the second case I guess the issue avoids SSH to being set up properly since I could see the following in the journal:

multipassd[102910]: Starting Multipass 1.8.0
multipassd[102910]: Daemon arguments: /snap/multipass/6130/bin/multipassd --verbosity debug --logger platform
multipassd[102910]: Using the 'default' storage pool.
multipassd[102910]: Reserving more disk space (274877906944 bytes) than available (115282710528 bytes)
multipassd[102910]: Creating instance with image id: a14b2e755caf397f73e8bf126428fd476cddca68d1f84a41260108a489043c48
multipassd[102910]: Using the 'default' storage pool.
multipassd[102910]: Reserving more disk space (274877906944 bytes) than available (114660392960 bytes)
multipassd[102910]: Creating instance with image id: a14b2e755caf397f73e8bf126428fd476cddca68d1f84a41260108a489043c48
multipassd[102910]: Waiting for SSH to be up
multipassd[102910]: Waiting for SSH to be up

The file /var/snap/multipass/common/data/multipassd/network/dnsmasq.leases has zero-bytes contents:

-rw-r--r-- 1 root root 0 Dec 28 02:43 /var/snap/multipass/common/data/multipassd/network/dnsmasq.leases

[townsend2010]

Hi @geekorous,

Ok, I'm really confused. The log snippet you showed when you first reported this indicates that you were using the qemu driver. But the follow up indicates you are using the lxd driver. Could you please clear up which driver you are using? It will help determine if this is a bug in LXD or potentially in our code or use of dnsmasq.

[geekorous]

Hi @townsend2010, I am sorry for the delay and confusion. You're right, the previous comment is under lxd provider. I changed the provider back to qemu and the issue persists. So it seems it is irrelevant to the backend. I used the same snapcraft files. The logs are as following:

Jan 16 11:58:28 lamborghini multipassd[433813]: process state changed to Running
Jan 16 11:58:28 lamborghini multipassd[433813]: [437485] started: qemu-system-x86_64 --enable-kvm -cpu host -nic tap,ifname=tap-845a8a62c39,script=no,downscript=no,model=virtio-net-pci,mac=52:54:00:7e:30:2c -device virtio-scsi-pci,id=scsi0 -drive file=/var/snap/multipass/common/data/multipassd/vault/instances/snapcraft-snap-1/bionic-server-cloudimg-amd64-disk.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 2 -m 2048M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/snap/multipass/common/data/multipassd/vault/instances/snapcraft-snap-1/cloud-init-config.iso
Jan 16 11:58:28 lamborghini multipassd[433813]: process started
Jan 16 11:58:28 lamborghini multipassd[433813]: Waiting for SSH to be up
Jan 16 11:58:28 lamborghini multipassd[433813]: QMP: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 2, "major": 4}, "package": "Debian 1:4.2-3ubuntu6.18"}, "capabilities": ["oob"]}}
Jan 16 11:58:28 lamborghini multipassd[433813]: QMP: {"return": {}}
Jan 16 11:58:29 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:29 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:29 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:29 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:29 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:29 lamborghini multipassd[433813]: QMP: {"timestamp": {"seconds": 1642330709, "microseconds": 821628}, "event": "RTC_CHANGE", "data": {"offset": 1}}
Jan 16 11:58:31 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:31 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:31 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:31 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:31 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:34 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:34 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:34 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:34 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:34 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:36 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:36 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:36 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:36 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:36 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:38 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:38 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:38 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:38 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:38 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:40 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:40 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:40 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:40 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:40 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:42 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:42 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:42 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:42 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:42 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:44 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:44 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:44 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:44 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:44 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:46 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:46 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:47 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:47 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:47 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:49 lamborghini multipassd[433813]: executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:49 lamborghini multipassd[433813]: Executing 'ip -brief -family inet address show scope global'
Jan 16 11:58:49 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
Jan 16 11:58:49 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 49
Jan 16 11:58:49 lamborghini multipassd[433813]: /build/multipass/parts/multipass/src/src/ssh/ssh_process.cpp:136 read_stream(): num_bytes = 0
Jan 16 11:58:52 lamborghini multipassd[433813]: QMP: {"timestamp": {"seconds": 1642330732, "microseconds": 599357}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[23]/virtio-backend"}}
Jan 16 11:58:52 lamborghini dnsmasq-dhcp[433870]: DHCPDISCOVER(mpqemubr0) 52:54:00:7e:30:2c
Jan 16 11:58:52 lamborghini dnsmasq-dhcp[433870]: DHCPOFFER(mpqemubr0) 10.59.93.135 52:54:00:7e:30:2c
Jan 16 11:58:52 lamborghini dnsmasq-dhcp[433870]: DHCPREQUEST(mpqemubr0) 10.59.93.135 52:54:00:7e:30:2c
Jan 16 11:58:52 lamborghini dnsmasq-dhcp[433870]: DHCPACK(mpqemubr0) 10.59.93.135 52:54:00:7e:30:2c snapcraft-snap-1
Jan 16 11:58:56 lamborghini multipassd[433813]: Executing '[ -e /var/lib/cloud/instance/boot-finished ]'
Jan 16 11:58:58 lamborghini multipassd[433813]: Executing '[ -e /var/lib/cloud/instance/boot-finished ]'
Jan 16 11:58:59 lamborghini multipassd[433813]: Executing '[ -e /var/lib/cloud/instance/boot-finished ]'
Jan 16 11:59:00 lamborghini multipassd[433813]: Executing '[ -e /var/lib/cloud/instance/boot-finished ]'
Jan 16 11:59:25 lamborghini multipassd[433813]: QMP: {"timestamp": {"seconds": 1642330765, "microseconds": 525098}, "event": "RTC_CHANGE", "data": {"offset": 1}}

When I watch for multipass list at some point I observe:

Name                    State             IPv4             Image
snapcraft-snap-1        Running           10.59.93.135     Ubuntu Snapcraft builder for Core 18
snapcraft-snap-2        Running           10.59.93.135     Ubuntu Snapcraft builder for Core 18

Then I purged all the instances and repeated the test to record the contents of dnsmaq.leases. The contents of /var/snap/multipass/common/data/multipassd/network/dnsmasq.leases are as following:

0 52:54:00:3d:a9:c1 10.59.93.157 snapcraft-snap-1 ff:b5:5e:67:ff:00:02:00:00:ab:11:c6:1e:dc:c9:09:1d:06:88

and while the second snapcraft has failed I see in the logs that it was using the first one's IP address.

Name:           snapcraft-snap-2
State:          Delayed Shutdown
IPv4:           10.59.93.157
Release:        Ubuntu 18.04.5 LTS
Image hash:     7c5c8f24046c (Ubuntu Snapcraft builder for Core 18)
Load:           0.36 0.09 0.03
Disk usage:     913.8M out of 247.9G
Memory usage:   55.5M out of 1.9G
Mounts:         --

Then when I restart the second build (while the first one is going through apt-get phase), the file changes its content to:

0 52:54:00:71:9d:a9 10.59.93.157 snapcraft-snap-2 ff:b5:5e:67:ff:00:02:00:00:ab:11:c6:1e:dc:c9:09:1d:06:88

and causes the first one to fail as well.

I suspect there could be a race condition in multipassd, but I don't have a clue where it is happening and why. Let me know if you need more information, or should I repeat the experiments under particular configurations.

[Saviq]

Hi @geekorous, could you please try with sudo snap refresh multipass --channel edge/pr2375? We had a related issue that's fixed in #2375 and now I wonder if it could be the same cause.

[geekorous]

Hi @Saviq. Thanks for the comment, but I'm afraid the issue remains unresolved in 1.9.0-dev.335.pr2375+geca8481c which corresponds to this channel. The outputs are similar so I avoid pasting them down here to keep the discussion concise.

[geekorous]

@Saviq Any news on this issue?

[luis4a0]

Hi @geekorous, to me this smells like a bug in dnsmasq. First, did you try a newer version of Multipass? (you reported 1.8.0). What happens if the calls to snapcraft are serialized (that is, wait for the first run, then run the second)? Is the IP of both instances the same?

[qoreQyaS]

i'am having the same problem on a debian unstable host

multipass   1.10.1
multipassd  1.10.1

the problem seems related to the images of snapcraft.
in my tests only snapcraft:* images were getting the same ip and only in groups of the same image
all of the following were created via
multipass launch -n NAME IMAGE

Name                    State             IPv4             Image
sn18-1                  Running           10.75.254.65     Ubuntu Snapcraft builder for Core 18
sn18-2                  Running           10.75.254.65     Ubuntu Snapcraft builder for Core 18
sn18-3                  Running           10.75.254.65     Ubuntu Snapcraft builder for Core 18
sn1                     Running           10.75.254.164    Ubuntu Snapcraft builder for Core 20
sn2                     Running           10.75.254.164    Ubuntu Snapcraft builder for Core 20
sn3                     Running           10.75.254.164    Ubuntu Snapcraft builder for Core 20
sn22-1                  Running           10.75.254.60     Ubuntu Snapcraft builder for Core 22
sn22-2                  Running           10.75.254.60     Ubuntu Snapcraft builder for Core 22
sndev-1                 Running           10.75.254.66     Ubuntu Snapcraft builder for the devel series
sndev-2                 Running           10.75.254.66     Ubuntu Snapcraft builder for the devel series
foo2                    Running           10.75.254.220    Ubuntu Core 16
foo1                    Running           10.75.254.8      Ubuntu Core 16
mosq-1                  Running           10.75.254.101    Ubuntu Mosquitto Appliance
mosq-2                  Running           10.75.254.131    Ubuntu Mosquitto Appliance

contents of the lease file

0 52:54:00:d8:e0:f2 10.75.254.131 mosq-2 ff:b5:5e:67:ff:00:02:00:00:ab:11:ee:e6:53:56:82:10:6d:f4
0 52:54:00:fe:1f:4d 10.75.254.101 mosq-1 ff:b5:5e:67:ff:00:02:00:00:ab:11:98:41:e2:f1:18:c1:a4:f6
0 52:54:00:1b:bb:a3 10.75.254.66 sndev-2 ff:b5:5e:67:ff:00:02:00:00:ab:11:3a:4a:bd:45:d9:b3:6d:b7
0 52:54:00:6a:aa:f3 10.75.254.60 sn22-2 ff:b5:5e:67:ff:00:02:00:00:ab:11:cb:b7:0f:59:f6:a1:a1:b6
0 52:54:00:ad:89:be 10.75.254.65 sn18-3 ff:b5:5e:67:ff:00:02:00:00:ab:11:c6:1e:dc:c9:09:1d:06:88
0 52:54:00:d7:60:e5 10.75.254.220 foo2 ff:b5:5e:67:ff:00:02:00:00:ab:11:0d:03:8c:18:c0:d8:63:b1
0 52:54:00:fd:ee:32 10.75.254.8 foo1 ff:b5:5e:67:ff:00:02:00:00:ab:11:9d:f1:64:8e:fa:c6:a0:3e
0 52:54:00:44:91:41 10.75.254.164 sn3 ff:b5:5e:67:ff:00:02:00:00:ab:11:24:82:0e:92:80:c8:fa:de

some lines from
journalctl -u sudo journalctl -u snap.multipass.multipassd|grep dnsmasq

Dez 12 01:35:41 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:36:37:52
Dez 12 01:35:41 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.65 52:54:00:36:37:52
Dez 12 01:35:41 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:36:37:52
Dez 12 01:35:41 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.65 52:54:00:36:37:52
Dez 12 01:35:41 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.65 52:54:00:36:37:52
Dez 12 01:35:41 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.65 52:54:00:36:37:52 sn18-1
Dez 12 01:38:08 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:40:4f:bc
Dez 12 01:38:08 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.164 52:54:00:40:4f:bc
Dez 12 01:38:08 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.164 52:54:00:40:4f:bc
Dez 12 01:38:08 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.164 52:54:00:40:4f:bc sn1
Dez 12 01:38:42 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:aa:06:3c
Dez 12 01:38:42 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.164 52:54:00:aa:06:3c
Dez 12 01:38:42 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.164 52:54:00:aa:06:3c
Dez 12 01:38:42 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.164 52:54:00:aa:06:3c sn2
Dez 12 01:39:25 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:44:49:fa
Dez 12 01:39:25 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.65 52:54:00:44:49:fa
Dez 12 01:39:25 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.65 52:54:00:44:49:fa
Dez 12 01:39:25 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.65 52:54:00:44:49:fa sn18-2
Dez 12 01:42:59 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:ad:89:be
Dez 12 01:42:59 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.65 52:54:00:ad:89:be
Dez 12 01:42:59 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.65 52:54:00:ad:89:be
Dez 12 01:42:59 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.65 52:54:00:ad:89:be sn18-3
Dez 12 01:43:41 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:44:91:41
Dez 12 01:43:41 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.164 52:54:00:44:91:41
Dez 12 01:43:41 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.164 52:54:00:44:91:41
Dez 12 01:43:41 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.164 52:54:00:44:91:41 sn3
Dez 12 01:45:16 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:8a:fb:3f
Dez 12 01:45:16 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.60 52:54:00:8a:fb:3f
Dez 12 01:45:16 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.60 52:54:00:8a:fb:3f
Dez 12 01:45:16 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.60 52:54:00:8a:fb:3f sn22-1
Dez 12 01:45:51 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:6a:aa:f3
Dez 12 01:45:51 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.60 52:54:00:6a:aa:f3
Dez 12 01:45:51 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.60 52:54:00:6a:aa:f3
Dez 12 01:45:51 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.60 52:54:00:6a:aa:f3 sn22-2
Dez 12 01:50:10 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:e8:b9:13
Dez 12 01:50:10 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.66 52:54:00:e8:b9:13
Dez 12 01:50:10 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.66 52:54:00:e8:b9:13
Dez 12 01:50:10 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.66 52:54:00:e8:b9:13 sndev-1
Dez 12 01:55:58 debian dnsmasq[2794105]: reading /etc/resolv.conf
Dez 12 01:55:58 debian dnsmasq[2794105]: using local addresses only for domain multipass
Dez 12 01:55:58 debian dnsmasq[2794105]: using nameserver 192.168.42.129#53
Dez 12 01:57:51 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:1b:bb:a3
Dez 12 01:57:51 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.66 52:54:00:1b:bb:a3
Dez 12 01:57:51 debian dnsmasq-dhcp[2794105]: DHCPREQUEST(mpqemubr0) 10.75.254.66 52:54:00:1b:bb:a3
Dez 12 01:57:51 debian dnsmasq-dhcp[2794105]: DHCPACK(mpqemubr0) 10.75.254.66 52:54:00:1b:bb:a3 sndev-2
Dez 12 02:01:31 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:fe:1f:4d
Dez 12 02:01:31 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.101 52:54:00:fe:1f:4d
Dez 12 02:01:31 debian dnsmasq-dhcp[2794105]: DHCPDISCOVER(mpqemubr0) 52:54:00:fe:1f:4d
Dez 12 02:01:31 debian dnsmasq-dhcp[2794105]: DHCPOFFER(mpqemubr0) 10.75.254.101 52:54:00:fe:1f:4d

[qoreQyaS]

i tried to analyze the dhcp packets:
for i in foo1 foo2 sn1 sn2 sn3 sn22-1 sn22-2; do sudo tcpdump -pni mpqemubr0 port 67 and port 68 -w tcpdump.${i}.pcap &> /dev/null & tcpdump_pid=$!;multipass start ${i}; sudo kill ${tcpdump_pid}; done

turns out all the instances using the same snapcraft image are sending the same identifier via option 61.

for i in foo1 foo2 sn1 sn2 sn3 sn22-1 sn22-2; do echo -n ${i}:; tcpdump -vr tcpdump.${i}.pcap 2> /dev/null| grep Client-ID\ \(61\)|sort -u; done

foo1:       Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:9d:f1:64:8e:fa:c6:a0:3e
foo2:       Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:0d:03:8c:18:c0:d8:63:b1
sn1:        Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:24:82:0e:92:80:c8:fa:de
sn2:        Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:24:82:0e:92:80:c8:fa:de
sn3:        Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:24:82:0e:92:80:c8:fa:de
sn22-1:     Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:cb:b7:0f:59:f6:a1:a1:b6
sn22-2:     Client-ID (61), length 19: hardware-type 255, b5:5e:67:ff:00:02:00:00:ab:11:cb:b7:0f:59:f6:a1:a1:b6

[qoreQyaS]

look like /etc/machine-id is the miscreant

for i in foo1 foo2 sn1 sn2 sn3 sn22-1 sn22-2; do echo -n ${i}:; multipass exec ${i} -- bash -c "cat /etc/machine-id 1>&2" > /dev/null; multipass stop ${i} > /dev/null; done

foo1:a63e2d99e68f42debe9caf692e647adf
foo2:943af501340a4e0d89f0942d4f94292d
sn1:0bc8e12adae243ab8d995d19149b4701
sn2:0bc8e12adae243ab8d995d19149b4701
sn3:0bc8e12adae243ab8d995d19149b4701
sn22-1:25829646f1bb29eba9e09ef362686aa1
sn22-2:25829646f1bb29eba9e09ef362686aa1

[qoreQyaS]

after emptying /etc/machine-id and /var/lib/dbus/machine-id by executing
echo | sudo tee /etc/machine-id /var/lib/dbus/machine-id
on /dev/sda1 in the qcow2 image
/var/snap/multipass/common/cache/multipassd/vault/images/snapcraft-core18-20201111/bionic-server-cloudimg-amd64-disk.img
newly created instances using snapcraft:core18 are not assigned the same ip any more.

Name                    State             IPv4             Image
sn18-1                  Running           10.75.254.251    Ubuntu Snapcraft builder for Core 18
sn18-2                  Running           10.75.254.71     Ubuntu Snapcraft builder for Core 18