Recurring error in the VR Router in ACS 4.20.1

[KuasarCloud]

Hey community!

After upgrading to 4.20.1 I'm getting this error when trying to start a VM. A VPC clean up is needed everytime, any clue?

2025-07-17 15:44:15,094 WARN  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-45:[ctx-867b7b76, job-50106/job-50110, ctx-3c4763dd]) (logid:0d91769e) Unable to contact resource. com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:2] is unreachable: Unable to apply dhcp entry on router
	at org.apache.cloudstack.network.topology.BasicNetworkTopology.applyRules(BasicNetworkTopology.java:404)
	at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.applyDhcpEntry(AdvancedNetworkTopology.java:173)
	at com.cloud.network.element.VirtualRouterElement.applyDhcpEntries(VirtualRouterElement.java:1087)
	at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:1055)
	at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1831)
	at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:2206)
	at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:2135)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1286)
	at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5486)
	at jdk.internal.reflect.GeneratedMethodAccessor386.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
	at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
	at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

[KuasarCloud]

These are the logs in the KVM Cloudstack Agent

2025-07-17 15:45:00,070 WARN  [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-3:[]) (logid:) Exception [Stream closed] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh update_config.py 169.254.57.165 vm_dhcp_entry.json.a30c8e69-a429-472f-8991-605b1ede4a9c ]. java.io.IOException: Stream closed
        at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
        at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
        at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)
        at com.cloud.utils.script.Script.execute(Script.java:305)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:562)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:303)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:318)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:165)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1958)
        at com.cloud.agent.Agent.processRequest(Agent.java:779)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1194)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

[KuasarCloud]

I normally clean up the vpc but it's happening every day and only one vpc in one account

[rajujith]

@KuasarCloud can you log in to the VR and check its status? Are there any filesystem errors? Is it not booting? unresponsive? Are you using any memory overcommit?
Register a new system VM template and recreate the VRs.

[KuasarCloud]

Hi @rajujith

I'm not seeing any error in the VR, it works for awhile an then start failing trying to insert an entry in the dnsmasq. In the dnsmasq has this error

root@r-1169-VM:/var/log# tail -f dnsmasq.log Jul 21 21:18:34 dnsmasq[950886]: reading /etc/dnsmasq-resolv.conf Jul 21 21:18:34 dnsmasq[950886]: using nameserver 1.1.1.1#53 Jul 21 21:18:34 dnsmasq[950886]: using nameserver 8.8.8.8#53 Jul 21 21:18:34 dnsmasq[950886]: using only locally-known addresses for hisca.local Jul 21 21:18:34 dnsmasq[950886]: read /etc/hosts - 30 names Jul 21 21:18:34 dnsmasq-dhcp[950886]: read /etc/dhcphosts.txt Jul 21 21:18:34 dnsmasq[950886]: cannot read /etc/dhcpopts.txt: No such file or directory Jul 21 21:18:37 dnsmasq[950886]: read /etc/hosts - 30 names Jul 21 21:18:37 dnsmasq-dhcp[950886]: read /etc/dhcphosts.txt Jul 21 21:18:37 dnsmasq[950886]: cannot read /etc/dhcpopts.txt: No such file or directory

That files is inside /etc/dnsmasq.d

[rajujith]

@KuasarCloud can you try a new systemVM template?

[KuasarCloud]

Thanks @rajujith,

How can I do that?

• Register a new template with the same name?
• When I clean up the VPC how I tell the management server that takes the new template?

Regards

[rajujith]

Register the SystemVM template using URL and select the template type as SYSTEM: http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-kvm.qcow2.bz2

Use a new name for the template and update the name at the global settings: router.template.kvm

Restart the management services.
Recreate the VRs.