Can't add host server to cloudstack management server

[subincb30]

I am using 4.19 version on cloudstack management and cloudstack host server. But, I am unable to add the host server to management server. Both are Ubuntu 22.04 server.
Getting the following error:

---

Error 530
Unable to add the host: Cannot find the server resources at http://192.168.200.238

Able to get ping response from host and management server:

---

From management server

root@ubuntu:~# ping 192.168.200.238
PING 192.168.200.238 (192.168.200.238) 56(84) bytes of data.
64 bytes from 192.168.200.238: icmp_seq=1 ttl=64 time=2.00 ms
64 bytes from 192.168.200.238: icmp_seq=2 ttl=64 time=1.42 ms

From the hots server

root@host:~# ping 192.168.200.215
PING 192.168.200.215 (192.168.200.215) 56(84) bytes of data.
64 bytes from 192.168.200.215: icmp_seq=1 ttl=64 time=1.98 ms
64 bytes from 192.168.200.215: icmp_seq=2 ttl=64 time=1.98 ms
64 bytes from 192.168.200.215: icmp_seq=3 ttl=64 time=2.00 ms

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

@subincb30

Could you please take look at the /var/log/cloudstack/management/management-server.log for any other exceptions

and can you provide the log to investigate further

[subincb30]

Here is the recent logs:

---

root@ubuntu:~# tail -f /var/log/cloudstack/management/management-server.log
2024-09-19 08:09:17,523 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:ctx-8f2e2288) (logid:3fc927f1) Starting to remove all volume_stats rows older than [Wed Sep 18 20:09:17 UTC 2024].
2024-09-19 08:09:17,525 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-2:ctx-fe5244f9) (logid:40b04762) Removed a total of [0] vm_stats rows older than [Thu Sep 19 08:08:17 UTC 2024].
2024-09-19 08:09:17,525 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-3:ctx-8f2e2288) (logid:3fc927f1) Removed a total of [0] volume_stats rows older than [Wed Sep 18 20:09:17 UTC 2024].
2024-09-19 08:09:19,255 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-8f547b9c) (logid:38edd7cf) HA health check task is running...
2024-09-19 08:09:23,258 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-765d761d) (logid:bd6be1a8) HA health check task is running...
2024-09-19 08:09:25,242 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-23bae9a7) (logid:1d353076) Skip capacity scan as there is no Primary Storage in 'Up' state
2024-09-19 08:09:25,256 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Enabled non-edge zones available for scan: 1
2024-09-19 08:09:25,257 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Zone 1 has no host available which is enabled and in Up state
2024-09-19 08:09:25,257 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-358eb897) (logid:e61a3b17) Zone [1] is not ready to launch secondary storage VM.
2024-09-19 08:09:27,260 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-3:ctx-243e29db) (logid:feaac53e) HA health check task is running...
2024-09-19 08:09:31,263 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-ad46d590) (logid:232a2af8) HA health check task is running...
2024-09-19 08:09:31,848 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 routers to update status.
2024-09-19 08:09:31,850 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 VPC's to update Redundant State.
2024-09-19 08:09:31,852 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-4d22af35) (logid:e5376347) Found 0 networks to update RvR status.

Also, I checked for the specific error for host server IP: 192.168.200.238:

---

2024-09-19 07:24:47,528 INFO [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Trying to add a new host at http://192.168.200.238 in data center 1
2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted.
2024-09-19 07:30:00,375 WARN [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Cannot find the server resources at http://192.168.200.238
com.cloud.exception.DiscoveryException: Unable to add the host: Cannot find the server resources at http://192.168.200.238
2024-09-19 07:30:00,380 INFO [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Unable to add the host: Cannot find the server resources at http://192.168.200.238
root@ubuntu:~#

Please check and let me know any further details needed.

[subincb30]

Additionally, I found that the port 8250 is not listening in my cloudstack host server.

[weizhouapache]

the logs show

2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted.

there should be some errors in log like CloudStack Agent setup through command ... failed

[subincb30]

@weizhouapache Sorry I didn't get your point. Also, I couldn't find any errors related to the cloud stack agent on the cloud stack management-server.log.

[weizhouapache]

@weizhouapache Sorry I didn't get your point. Also, I couldn't find any errors related to the cloud stack agent on the cloud stack management-server.log.

the logs shows the mgmt server has connected to the host and configured security (for cloudstack-agent and libvirt). so it is not a connectivity issue.

please upload the full log by keyword ctx-6dbc68fe if possible.

[subincb30]

@weizhouapache Here is the full log with keyword: ctx-6dbc68fe:

---

root@ubuntu:~# cat /var/log/cloudstack/management/management-server.log | grep ctx-6dbc68fe
2024-09-19 07:24:47,504 DEBUG [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"3bc44629-66bf-11ef-9691-70b5e860d7ec"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-09-19 07:24:47,513 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-09-19 07:24:47,515 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-09-19 07:24:47,520 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Received unknown parameters for command addHost. Unknown parameters : clustertype
2024-09-19 07:24:47,528 INFO [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Trying to add a new host at http://192.168.200.238 in data center 1
2024-09-19 07:24:47,749 WARN [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Failed to authenticate with ssh key
2024-09-19 07:24:47,750 INFO [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Failed to authenticate with ssh key, retrying with password
2024-09-19 07:24:47,784 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: ls /dev/kvm
2024-09-19 07:24:49,149 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: ls /dev/kvm
2024-09-19 07:24:49,160 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:50,831 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:50,831 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:52,675 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-09-19 07:24:52,725 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties ykT2GOj8BCqvXvQR /etc/cloudstack/agent/
2024-09-19 07:24:55,064 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Succeeded to import certificate in the keystore for agent on the KVM host: 192.168.200.238. Agent secured and trusted.
2024-09-19 07:24:55,082 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Executing cmd: cloudstack-setup-agent -m 192.168.200.215 -z 1 -p 1 -c 1 -g 50638cf5-4698-39d1-b55c-3b7b1bb9647a -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm
2024-09-19 07:25:00,349 DEBUG [c.c.u.s.SSHCmdHelper] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) SSH command: cloudstack-setup-agent -m 192.168.200.215 -z 1 -p 1 -c 1 -g 50638cf5-4698-39d1-b55c-3b7b1bb9647a -a -s --pubNic=cloudbr0 --prvNic=cloudbr0 --guestNic=cloudbr0 --hypervisor=kvm
2024-09-19 07:30:00,373 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Timeout, to wait for the host connecting to mgt svr, assuming it is failed
2024-09-19 07:30:00,375 WARN [c.c.r.ResourceManagerImpl] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Cannot find the server resources at http://192.168.200.238
2024-09-19 07:30:00,375 INFO [c.c.u.e.CSExceptionErrorCode] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2024-09-19 07:30:00,378 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Exception:
2024-09-19 07:30:00,380 INFO [c.c.a.ApiServer] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) Unable to add the host: Cannot find the server resources at http://192.168.200.238
2024-09-19 07:30:00,381 DEBUG [c.c.a.ApiServlet] (qtp341748265-16:ctx-d80d59ca ctx-6dbc68fe) (logid:4e450a45) ===END=== 192.168.232.15 -- POST command=addHost&response=json

[weizhouapache]

@subincb30
can you upload the file /var/log/cloudstack/agent/setup.log on the kvm host ?

[subincb30]

@weizhouapache Please find the log below:

---

root@host:~# tail -f /var/log/cloudstack/agent/setup.log
DEBUG:root:execute:sudo /usr/sbin/service network-manager start
DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager status
DEBUG:root:Failed to execute:Unit network-manager.service could not be found.
DEBUG:root:execute:sudo /usr/sbin/service network-manager start
DEBUG:root:Failed to execute:Failed to start network-manager.service: Unit network-manager.service not found.
DEBUG:root:execute:/etc/init.d/networking stop
DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found
DEBUG:root:execute:/etc/init.d/networking start
DEBUG:root:Failed to execute:/bin/sh: 1: /etc/init.d/networking: not found

[subincb30]

Someone please help me to resolve the issue.

[weizhouapache]

@subincb30
have you configured cloudbr0 on the kvm host ?

[subincb30]

Can you give me complete guide for that? I got some doc for for that but it was not clear. But, I found one new network interface in my server.

---

5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:fe:08:e5 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever

Can you help me setting up cloudbr0?

[weizhouapache]

@subincb30
please refer to https://docs.cloudstack.apache.org/en/4.19.1.1/installguide/hypervisor/kvm.html

[subincb30]

@weizhouapache I already checked this earlier. But, I am little bit confussed on it. I will paste my current netplan settings below.

---

root@host:~# cat /etc/netplan/00-installer-config.yaml

This is the network config written by 'subiquity'

network:
renderer: networkd
ethernets:
enp1s0:
dhcp4: true
enp2s0:
addresses:
- 192.168.200.238/24
nameservers:
addresses:
- 192.168.1.1
- 8.8.8.8
search:
- google.com
routes:
- to: default
via: 192.168.200.1
enp3s0:
dhcp4: true
version: 2

Can you help me adding the additional parameters in this file.

[weizhouapache]

@subincb30
the doc has provided an example of netplan

[subincb30]

@weizhouapache Yes I saw that. But some parameters are confusing for me. Can you check my current settings which I shared earlier and give an example accordingly?

[weizhouapache]

@weizhouapache Yes I saw that. But some parameters are confusing for me. Can you check my current settings which I shared earlier and give an example accordingly?

to be frank, I think it is a good chance for you to learn netplan. Isn't it ?

[subincb30]

Let me try my best, thank you.

[subincb30]

I added cloudbr0. Now I am able to add the host server to management server.

[subincb30]

I added cloudbr0. Now I am able to add the host server to management server.

[xmlgrg]

新版的 Ubuntu 22.04.5 LTS，已经没有这个 /etc/network/interfaces 文件了，/usr/lib/python3/dist-packages/cloudutils/serviceConfig.py 文件中 还是去寻找这个 文件 和 network-manager 服务

[weizhouapache]

Hi @xmlgrg
welcome to the community.

currently there are multiple ways to configure networks in linux servers: netplan, NetworkManager/nmcli, network-scripts, etc
the file serviceConfig.py has not been updated to support all above.
I suggest you to configure networks before adding it to cloudstack. please refer to the cloudstack doc

https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#configure-ubuntu-for-advanced-networks
https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html#configuring-the-network