No ping to SSVM public ip

[dominar250]

ISSUE TYPE

• Other

COMPONENT NAME

SSVM

CLOUDSTACK VERSION

CloudStack 4.19.0.1

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

I'm using nested environment over vmware. promiscuous mode and MAC address change ,Forged Transmit are enabled in port group level. SSVM and console proxy agents are up.I'm having issue with packet loss from kvm machine to SSVM /Console proxy vm both private and public ip address. Unable to ping SSVM public ips .Ping from ssvm to management /kvm is good

root@cloudstack-kvm:~# brctl show
bridge name     bridge id               STP enabled     interfaces
cloud0          8000.fe00a9fe08c6       no              vnet24
                                                        vnet27
cloudbr0                8000.005056b4a6ca       yes             ens192
                                                        vnet25
                                                        vnet28
cloudbr1                8000.005056b48ea4       yes             ens224
                                                        vnet26
                                                        vnet29
root@cloudstack-kvm:~# ip addr show cloudbr0
5: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
    inet 10.158.65.11/25 brd 10.158.65.127 scope global cloudbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb4:a6ca/64 scope link
       valid_lft forever preferred_lft forever
root@cloudstack-kvm:~#
root@cloudstack-kvm:~# ip addr show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
    link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
    altname enp11s0
root@cloudstack-kvm:~#
root@cloudstack-kvm:~# ip addr show cloudbr1
6: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:50:56:b4:8e:a4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:feb4:8ea4/64 scope link
       valid_lft forever preferred_lft forever
root@cloudstack-kvm:~# ip addr show ens224
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr1 state UP group default qlen 1000
    link/ether 00:50:56:b4:8e:a4 brd ff:ff:ff:ff:ff:ff
    altname enp19s0

EXPECTED RESULTS

Able to ping ssvm public/private range from kvm/management server

ACTUAL RESULTS

root@cloudstack-kvm:~# ping -O 10.158.209.11
PING 10.158.209.11 (10.158.209.11) 56(84) bytes of data.
no answer yet for icmp_seq=1
no answer yet for icmp_seq=2
no answer yet for icmp_seq=3
no answer yet for icmp_seq=4

--- 10.158.209.11 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4081ms

[dominar250]

cloud.log-consoleproxy.txt
cloud.log-secondary storage.txt

[weizhouapache]

@dominar250
have you configured any routes to 10.158.209.x on your switch ?

[dominar250]

@weizhouapache No sir . I have assigned this network to a vm for testing , I can able to ssh and ping that ip from 10.158.65.0 network

[weizhouapache]

10.158.209.

@dominar250
does ping to 10.158.209.X gateway work inside the ssvm ?

[dominar250]

@weizhouapache no ping from ssvm to gateway
ping 10.158.209.1
PING 10.158.209.1 (10.158.209.1): 56 data bytes
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable
--- 10.158.209.1 ping statistics ---

[weizhouapache]

@weizhouapache no ping from ssvm to gateway ping 10.158.209.1 PING 10.158.209.1 (10.158.209.1): 56 data bytes 92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable 92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable 92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable 92 bytes from s-10-VM (10.158.209.11): Destination Host Unreachable --- 10.158.209.1 ping statistics ---

I guess it is because the virtual nic (vnet25 ?) for public interface of s-10-VM is on cloudbr0
when you create a vm, the virtual nic is on cloudbr1

you can go to the Zone-> Physical network, and update the traffic label of Public traffic to "cloudbr1" and retry

[dominar250]

@weizhouapache Its already cloudbr1. guest is also in cloudbr1 and managment interface is on cloudbr0

[weizhouapache]

@weizhouapache Its already cloudbr1. guest is also in cloudbr1 and managment interface is on cloudbr0

destroy the system vm and re-check the new system vm ?

[dominar250]

@weizhouapache still having the same issue. no ping to gateways

[DaanHoogland]

@dominar250 can you run /usr/local/cloud/systemvm/ssvm-check.sh in the ssvm and show the output? It may give som clue.

[dominar250]

ssvm-script.txt
@DaanHoogland

[DaanHoogland]

There is obviously a network problem:

root@s-15-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
92 bytes from 10.158.65.14: Destination Host Unreachable
92 bytes from 10.158.65.14: Destination Host Unreachable
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
WARNING: cannot ping DNS server
route follows
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.158.209.1    0.0.0.0         UG    0      0        0 eth2
10.158.65.0     0.0.0.0         255.255.255.128 U     0      0        0 eth1
10.158.209.0    0.0.0.0         255.255.255.128 U     0      0        0 eth2
66.173.144.10   10.158.65.1     255.255.255.255 UGH   0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
================================================
ERROR: DNS not resolving cloudstack.apache.org
resolv.conf follows
nameserver 66.173.144.10
nameserver 139.55.95.240

What are your physical network traffic configurations?

[dominar250]

@DaanHoogland - This is my netplan configuration

network:
version: 2
ethernets:
ens192: {}
ens224: {}
bridges:
cloudbr0:
addresses:
- 10.158.65.11/25
dhcp4: false
routes:
- to: default
via: 10.158.65.1
nameservers:
addresses:
- 139.55.95.240
- 66.173.144.10
search: [winster.com]
interfaces:
- ens192
parameters:
stp: false
cloudbr1:
dhcp4: false
interfaces:
- ens224
parameters:
stp: false

[weizhouapache]

@DaanHoogland - This is my netplan configuration

network:
version: 2
ethernets:
ens192: {}
ens224: {}
bridges:
cloudbr0:
addresses:
- 10.158.65.11/25
dhcp4: false
routes:
- to: default
via: 10.158.65.1
nameservers:
addresses:
- 139.55.95.240
- 66.173.144.10
search: [winster.com]
interfaces:
- ens192
parameters:
stp: false
cloudbr1:
dhcp4: false
interfaces:
- ens224
parameters:
stp: false

@dominar250
Can you share agent.properties on the host?

[dominar250]

@DaanHoogland
#Storage
#Wed Jul 17 15:54:37 UTC 2024
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.network.device=cloudbr1
keystore.passphrase=u8CetwThYSOBkFEy
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr1
local.storage.uuid=24bc3156-6c17-4be9-a256-dfbcb93d60de
host=10.158.65.10@static
guid=80f7ec53-46ad-3d4d-b648-d2bc5203aecb
LibvirtComputingResource.id=1
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

[dominar250]

@DaanHoogland Some how my secondary storage vm is able to ping both gateways . console proxy is failing to reach public gateways and able to ping private gateways. no ping to nfs storage which is on same subnet of private

root@s-16-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
64 bytes from 66.173.144.10: icmp_seq=0 ttl=53 time=15.182 ms
64 bytes from 66.173.144.10: icmp_seq=1 ttl=53 time=15.203 ms
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.182/15.192/15.203/0.000 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
nfs is currently mounted
Mount point is /mnt/SecStorage/c1be951c-81f9-3819-9dfc-ccf19a870369
Good: Can write to mount point
================================================
Management server is 10.158.65.10. Checking connectivity.
Good: Can connect to management server 10.158.65.10 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.
root@s-16-VM:~# ping 10.158.65.1
PING 10.158.65.1 (10.158.65.1): 56 data bytes
64 bytes from 10.158.65.1: icmp_seq=0 ttl=64 time=0.602 ms
64 bytes from 10.158.65.1: icmp_seq=1 ttl=64 time=0.688 ms
--- 10.158.65.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.602/0.645/0.688/0.043 ms
root@s-16-VM:~#
root@s-16-VM:~# ping 10.158.209.1
PING 10.158.209.1 (10.158.209.1): 56 data bytes
64 bytes from 10.158.209.1: icmp_seq=0 ttl=64 time=0.537 ms
64 bytes from 10.158.209.1: icmp_seq=1 ttl=64 time=1.063 ms
--- 10.158.209.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.537/0.800/1.063/0.263 ms

but console proxy vm is failed to reach public gateway

root@v-17-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
64 bytes from 66.173.144.10: icmp_seq=0 ttl=53 time=15.222 ms
64 bytes from 66.173.144.10: icmp_seq=1 ttl=53 time=15.220 ms
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.220/15.221/15.222/0.000 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
ERROR: Storage nfs is not currently mounted
Verifying if we can at least ping the storage
Storage address list is empty, trying to ping storage network gateway instead ()
ping: missing host operand
Try 'ping --help' or 'ping --usage' for more information.
WARNING: Cannot ping nfs storage network gateway
routing table follows
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.158.209.1    0.0.0.0         UG    0      0        0 eth2
10.158.65.0     0.0.0.0         255.255.255.128 U     0      0        0 eth1
10.158.209.0    0.0.0.0         255.255.255.128 U     0      0        0 eth2
66.173.144.10   10.158.65.1     255.255.255.255 UGH   0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
================================================
Management server is 10.158.65.10. Checking connectivity.
Good: Can connect to management server 10.158.65.10 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.
root@v-17-VM:~#
root@v-17-VM:~# ping 10.158.65.1
PING 10.158.65.1 (10.158.65.1): 56 data bytes
64 bytes from 10.158.65.1: icmp_seq=0 ttl=64 time=0.533 ms
64 bytes from 10.158.65.1: icmp_seq=1 ttl=64 time=0.826 ms
--- 10.158.65.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.533/0.679/0.826/0.147 ms
root@v-17-VM:~# ping 10.158.209.1
PING 10.158.209.1 (10.158.209.1): 56 data bytes
92 bytes from v-17-VM (10.158.209.11): Destination Host Unreachable
92 bytes from v-17-VM (10.158.209.11): Destination Host Unreachable
--- 10.158.209.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

[weizhouapache]

@dominar250
can you share the output of ip route command in both system vms ?

[dominar250]

@weizhouapache I found one more clue that the arp table on the un reachable vm has "incomplete entry " on the public gateways . if i added manually to match the same with working one i can able to ping. Is there any changes need to done on kvm host machine ?Thank you in advance

root@v-17-VM:~# ip route
default via 10.158.209.1 dev eth2
10.158.65.0/25 dev eth1 proto kernel scope link src 10.158.65.21
10.158.209.0/25 dev eth2 proto kernel scope link src 10.158.209.11
66.173.144.10 via 10.158.65.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.137.107

root@s-16-VM:~# ip r
default via 10.158.209.1 dev eth2
10.158.65.0/25 dev eth1 proto kernel scope link src 10.158.65.15
10.158.209.0/25 dev eth2 proto kernel scope link src 10.158.209.10
66.173.144.10 via 10.158.65.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.249.72

[dominar250]

@weizhouapache working vm arp table look like this

root@s-16-VM:# arp -an
? (10.158.65.3) at e8:24:a6:25:93:a5 [ether] on eth1
? (10.158.65.2) at e8:24:a6:25:03:a6 [ether] on eth1
? (10.158.65.4) at 00:50:56:b4:67:6b [ether] on eth1
? (169.254.0.1) at fe:00:a9:fe:89:6b [ether] on eth0
? (10.158.209.1) at 00:00:5e:00:01:62 [ether] on eth2
? (10.158.209.3) at e8:24:a6:25:93:a5 [ether] on eth2
? (10.158.209.11) at 1e:00:97:00:00:0d [ether] on eth2
? (10.158.65.10) at 00:50:56:b4:b1:3a [ether] on eth1
? (10.158.65.11) at 00:50:56:b4:a6:ca [ether] on eth1
? (10.158.65.14) at 1e:00:6b:00:00:03 [ether] on eth1
? (10.158.65.1) at 00:00:5e:00:01:63 [ether] on eth1
Non working vm
root@v-17-VM:# arp -an
? (10.158.65.10) at 00:50:56:b4:b1:3a [ether] on eth1
? (10.158.65.1) at 00:00:5e:00:01:63 [ether] on eth1
? (10.158.65.4) at 00:50:56:b4:67:6b [ether] PERM on eth1 == added manually
? (10.158.209.1) at 00:00:5e:00:01:62 [ether] PERM on eth2 == added manually
? (10.158.209.2) at e8:24:a6:25:03:a6 [ether] on eth2
? (10.158.65.11) at 00:50:56:b4:a6:ca [ether] on eth1
? (10.158.209.3) at e8:24:a6:25:93:a5 [ether] on eth2
? (10.158.65.3) at e8:24:a6:25:93:a5 [ether] on eth1
? (169.254.0.1) at fe:00:a9:fe:89:6b [ether] on eth0

[weizhouapache]

@weizhouapache I found one more clue that the arp table on the un reachable vm has "incomplete entry " on the public gateways . if i added manually to match the same with working one i can able to ping. Is there any changes need to done on kvm host machine ?Thank you in advance

root@v-17-VM:~# ip route
default via 10.158.209.1 dev eth2
10.158.65.0/25 dev eth1 proto kernel scope link src 10.158.65.21
10.158.209.0/25 dev eth2 proto kernel scope link src 10.158.209.11
66.173.144.10 via 10.158.65.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.137.107

root@s-16-VM:~# ip r
default via 10.158.209.1 dev eth2
10.158.65.0/25 dev eth1 proto kernel scope link src 10.158.65.15
10.158.209.0/25 dev eth2 proto kernel scope link src 10.158.209.10
66.173.144.10 via 10.158.65.1 dev eth1
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.249.72

It is strange that ssvm works but cpvm does not.

I recalled an issue with CKS which is caused by sysctl settings for arp #9186

You can also check the sysctl settings on kvm host and in the system vms

sysctl -a | grep arp

[dominar250]

@weizhouapache Thanks for helping me to resolve the issue . I have disconnected and reconnected the portgroup in vm level is able to solve this gateway ping issue. I can able to ping both gateways. The only issue i 'm facing right now is console proxy vm is not able to find the nfs storage, even i can able to ping the IP. SSvm looks good. routing table is same for both

root@v-22-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh

First DNS server is 66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
64 bytes from 66.173.144.10: icmp_seq=0 ttl=53 time=15.115 ms
64 bytes from 66.173.144.10: icmp_seq=1 ttl=53 time=15.206 ms
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.115/15.160/15.206/0.046 ms
Good: Can ping DNS server

Good: DNS resolves cloudstack.apache.org

ERROR: Storage nfs is not currently mounted
Verifying if we can at least ping the storage
Storage address list is empty, trying to ping storage network gateway instead ()
ping: missing host operand
Try 'ping --help' or 'ping --usage' for more information.
WARNING: Cannot ping nfs storage network gateway
routing table follows
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.158.209.1 0.0.0.0 UG 0 0 0 eth2
10.158.65.0 0.0.0.0 255.255.255.128 U 0 0 0 eth1
10.158.209.0 0.0.0.0 255.255.255.128 U 0 0 0 eth2
66.173.144.10 10.158.65.1 255.255.255.255 UGH 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0

Management server is 10.158.65.10. Checking connectivity.
Good: Can connect to management server 10.158.65.10 port 8250

Good: Java process is running

Tests Complete. Look for ERROR or WARNING above.
root@v-22-VM:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 466M 0 466M 0% /dev
tmpfs 193M 480K 193M 1% /run
/dev/vda6 4.2G 1.3G 2.9G 32% /
tmpfs 483M 0 483M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda1 366M 46M 321M 13% /boot
tmpfs 97M 0 97M

root@s-23-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh

First DNS server is 66.173.144.10
PING 66.173.144.10 (66.173.144.10): 56 data bytes
64 bytes from 66.173.144.10: icmp_seq=0 ttl=53 time=15.374 ms
64 bytes from 66.173.144.10: icmp_seq=1 ttl=53 time=15.178 ms
--- 66.173.144.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 15.178/15.276/15.374/0.098 ms
Good: Can ping DNS server

Good: DNS resolves cloudstack.apache.org

nfs is currently mounted
Mount point is /mnt/SecStorage/d55dbe2f-0b8e-3a1e-806e-d1551d738c96
Good: Can write to mount point

Management server is 10.158.65.10. Checking connectivity.
Good: Can connect to management server 10.158.65.10 port 8250

Good: Java process is running

Tests Complete. Look for ERROR or WARNING above.
root@s-23-VM:# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.158.209.1 0.0.0.0 UG 0 0 0 eth2
10.158.65.0 0.0.0.0 255.255.255.128 U 0 0 0 eth1
10.158.209.0 0.0.0.0 255.255.255.128 U 0 0 0 eth2
66.173.144.10 10.158.65.1 255.255.255.255 UGH 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
root@s-23-VM:#

[weizhouapache]

@weizhouapache Thanks for helping me to resolve the issue . I have disconnected and reconnected the portgroup in vm level is able to solve this gateway ping issue. I can able to ping both gateways. The only issue i 'm facing right now is console proxy vm is not able to find the nfs storage, even i can able to ping the IP. SSvm looks good. routing table is same for both

@dominar250
the console proxy vm does not have network interface on the storage network (ssvm does)
it is not an issue I think