multiple physical network with guest network issue

[hqming86]

I try to deploy cloudstack guest network for multiple physical network, but It looks like doesn't work, can any one help?

I designed as below:
There are three physical network:
1、physical network1：Three traffic types:Management,Public,Storage. All those types setup as traffic label cloudbr0
2、physical network2：Only one traffic type: Guest which is setup traffic lable cloudbr1
3、physical network3：One more traffic type : Guest traffic with tag "NFS".

The agent deploy 3 bridge: cloudbr0(bridge to eth2), cloudbr1(bridge to eth3),cloudbr2(bridge to bond4)
root@cloudstack-agent02:/etc/cloudstack/agent# brctl show
bridge name bridge id STP enabled interfaces
cloud0 8000.0202e6c8226f no
cloudbr0 8000.1a10af3be1f1 no eth2
cloudbr1 8000.0adad08ce0ff no eth3
cloudbr2 8000.d697c5e4beae no bond4

Now the agent have already added successfully . but the system vm can not deploy automaticly, and the log show that "Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: cloudbr2"

The "agent.properties" as below:
#Thu Jul 10 07:05:43 UTC 2025
cluster=12
LibvirtComputingResource.name=cloudstack-agent42.tpu-42
pod=7
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.cpu.mode=host-model
guest.network.device=cloudbr1
keystore.passphrase=5eNnkdWrq8rF7AV8
hypervisor.type=kvm
port=8250
zone=6
public.network.device=cloudbr0
local.storage.uuid=e44d6ec9-1e50-45a7-b20b-42c825a5bf8f
LibvirtComputingResource.uuid=876cb9d8-905f-4dc2-9db2-5717f1fc4fbf
host=10.10.98.120@static
guid=7024cbd6-7ee5-3134-82e8-5e91aa8d44fe
LibvirtComputingResource.id=31
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

Can anyone assistance to address it?

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[DaanHoogland]

@hqming86 , what is your ratio on creating a “Guest” network with tag “NFS", but have your “Storage” network on the same as the first “Guest” network?
I have no idea why the SSVM cannot find it btw, but it seems confusing.

[weizhouapache]

@hqming86

• do all kvm hosts have the same setting ?
• is bond4 configured with 1 or 2 physical interfaces ? or it is just a virtual nic ?
• can you share the agent.log ?

[hqming86]

do all kvm hosts have the same setting ?
Thare are only one host in the zone
is bond4 configured with 1 or 2 physical interfaces ? or it is just a virtual nic ?
it's 2 physical interfaces
can you share the agent.log ?
session_Management.log
2025-07-10 17:34:07,945 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-4e12bc47]) (logid:361754cd) Unable to start secondary storage VM [5748] for standby capacity, it will be recycled and will start a new one.

session_Agent.log

2025-07-10 09:34:04,569 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) [Ljava.lang.StackTraceElement;@2ea977b8
2025-07-10 09:34:07,566 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-1:[]) (logid:) No existing libvirtd connection found. Opening a new one
2025-07-10 09:34:07,568 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-1:[]) (logid:) Can not find a connection for Instance v-5747-VM. Assuming the default connection.
2025-07-10 09:34:07,685 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-1:[]) (logid:) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2025-07-10 09:34:07,685 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-1:[]) (logid:) Groovy scripting engine is not initialized. Data transformation skipped.
2025-07-10 09:34:07,758 INFO [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:) No existing libvirtd connection found. Opening a new one
2025-07-10 09:34:07,759 WARN [kvm.resource.LibvirtConnection] (AgentRequest-Handler-5:[]) (logid:) Can not find a connection for Instance s-5748-VM. Assuming the default connection.
2025-07-10 09:34:07,874 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-5:[]) (logid:) Groovy script '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available. Transformations will not be applied.
2025-07-10 09:34:07,874 WARN [kvm.resource.LibvirtKvmAgentHook] (AgentRequest-Handler-5:[]) (logid:) Groovy scripting engine is not initialized. Data transformation skipped.
2025-07-10 09:34:27,000 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:) Trying to fetch storage pool 92d75cdb-28df-37cc-9866-bd063b0f52aa from libvirt
2025-07-10 09:34:27,004 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-3:[]) (logid:) Asking libvirt to refresh storage pool 92d75cdb-28df-37cc-9866-bd063b0f52aa
2025-07-10 09:34:27,365 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool 92d75cdb-28df-37cc-9866-bd063b0f52aa from libvirt
2025-07-10 09:34:27,368 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Asking libvirt to refresh storage pool 92d75cdb-28df-37cc-9866-bd063b0f52aa
2025-07-10 09:34:28,421 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Attempting to create storage pool 4ab72685-0c0c-3938-9fc2-4dcb362ab7df (NetworkFilesystem) in libvirt
2025-07-10 09:34:28,423 WARN [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Storage pool 4ab72685-0c0c-3938-9fc2-4dcb362ab7df was not found running in libvirt. Need to create it.
2025-07-10 09:34:28,423 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Didn't find an existing storage pool 4ab72685-0c0c-3938-9fc2-4dcb362ab7df by UUID, checking for pools with duplicate paths
2025-07-10 09:34:28,433 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:) Execution of process [94441] for command [/bin/bash -c mountpoint -q /mnt/4ab72685-0c0c-3938-9fc2-4dcb362ab7df ] failed.
2025-07-10 09:34:28,433 WARN [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:) Process [94441] for command [/bin/bash -c mountpoint -q /mnt/4ab72685-0c0c-3938-9fc2-4dcb362ab7df ] encountered the error: [32].
2025-07-10 09:34:28,454 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec default.oceanstor-9540.glzs.com:/test-nfs/template/tmpl/1/227 /mnt/4ab72685-0c0c-3938-9fc2-4dcb362ab7df) unexpected exit status 32: mount.nfs: Protocol not supported for default.oceanstor-9540.glzs.com:/test-nfs/template/tmpl/1/227 on /mnt/4ab72685-0c0c-3938-9fc2-4dcb362ab7df

2025-07-10 09:34:28,454 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Failed to create netfs mount: default.oceanstor-9540.glzs.com:/test-nfs/template/tmpl/1/227 org.libvirt.LibvirtException: internal error: Child process (/usr/bin/mount -o nodev,nosuid,noexec default.oceanstor-9540.glzs.com:/test-nfs/template/tmpl/1/227 /mnt/4ab72685-0c0c-3938-9fc2-4dcb362ab7df) unexpected exit status 32: mount.nfs: Protocol not supported for default.oceanstor-9540.glzs.com:/test-nfs/template/tmpl/1/227 on /mnt/4ab72685-0c0c-3938-9fc2-4dcb362ab7df

I notice that host can not mount secondray storage.

[weizhouapache]

@hqming86
yes, looks like a secondary storage issue

[hqming86]

I figure out what is going on, I forgot download system template , now the issue solved.

Thanks everyone.