Replies: 1 comment
-
|
I have now done a fresh install of 4.20.1 using Ubuntu 24.04 with a new database, but I am still experiencing the same issue with not being able to provision the host security keys, due to libvirtd restart is failing. I can execute 3fa4b7a550, name: cloudstack2] |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I asked previously about "assuming server identity" when physically replacing a server, this broadly worked but I could not provision the host security keys. In the end, I removed the previous server from the database and reinstalled the "new" server again with same IP and name, added it as a management server and host. However, I am still unable to provision the security keys, it appears from the logs (below) that libvirtd fails to restart (I can execute sudo service libvirtd restart successfully). I am also unable to launch any VM's on the new server even when unsecure, not sure if this is related and the secure comms needs to be established first. Some help would be appreciated
agent.log snippet
2025-07-28 08:13:56,748 INFO [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] (CertificateRenewalTask-1:[]) (logid:) Restarting libvirt after certificate provisioning/renewal
2025-07-28 08:13:56,764 INFO [cloud.agent.Agent] (AgentShutdownThread:[]) (logid:) Stopping the agent: Reason = sig.kill
2025-07-28 08:13:56,765 WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] (CertificateRenewalTask-1:[]) (logid:) Execution of process [15555] for command [sudo service libvirtd restart ] failed.
2025-07-28 08:13:56,765 WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] (CertificateRenewalTask-1:[]) (logid:) Process [15555] for command [sudo service libvirtd restart ] encountered the error: [].
2025-07-28 08:13:56,767 WARN [cloud.agent.Agent] (CertificateRenewalTask-1:[]) (logid:) Failed to execute post certificate renewal command: java.lang.IllegalStateException: Shutdown in progress
at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:245)
at com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1268)
at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-07-28 08:13:58,190 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Agent started
2025-07-28 08:13:58,192 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Implementation Version is 4.20.1.0
2025-07-28 08:13:58,193 INFO [cloud.agent.AgentShell] (main:[]) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2025-07-28 08:13:58,213 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Defaulting to using properties file for storage
2025-07-28 08:13:58,214 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Defaulting to the constant time backoff algorithm
2025-07-28 08:13:58,229 INFO [cloud.utils.LogUtils] (main:[]) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2025-07-28 08:13:58,229 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Using default Java settings for IPv6 preference for agent connection
2025-07-28 08:13:58,270 INFO [cloud.agent.Agent] (main:[]) (logid:) Initialising agent [id: 10, uuid: aeb26945-0f0d-4fdb-8961-3cd755c35501, name: cloudstack2]
2025-07-28 08:13:58,275 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2025-07-28 08:13:58,293 INFO [kvm.resource.LibvirtConnection] (main:[]) (logid:) No existing libvirtd connection found. Opening a new one
2025-07-28 08:13:58,457 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) IO uring driver for Qemu: enabled
2025-07-28 08:13:58,528 INFO [kvm.storage.KVMStoragePoolManager] (main:[]) (logid:) Adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor
2025-07-28 08:13:58,529 INFO [kvm.storage.KVMStoragePoolManager] (main:[]) (logid:) Adding storage adaptor for com.cloud.hypervisor.kvm.storage.StorPoolStorageAdaptor
2025-07-28 08:13:58,530 INFO [kvm.storage.KVMStoragePoolManager] (main:[]) (logid:) Adding storage adaptor for com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2025-07-28 08:13:58,530 INFO [kvm.storage.KVMStoragePoolManager] (main:[]) (logid:) Adding storage adaptor for com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor
2025-07-28 08:13:58,531 INFO [kvm.storage.KVMStoragePoolManager] (main:[]) (logid:) Adding storage adaptor for com.cloud.hypervisor.kvm.storage.ManagedNfsStorageAdaptor
2025-07-28 08:13:58,537 INFO [kvm.storage.FiberChannelAdapter] (main:[]) (logid:) Loaded FiberChannelAdapter for StorageLayer
2025-07-28 08:13:58,538 INFO [kvm.storage.FiberChannelAdapter] (main:[]) (logid:) Loaded FiberChannelAdapter for StorageLayer on host [cloudstack2]
2025-07-28 08:13:58,538 INFO [kvm.storage.KVMStoragePoolManager] (main:[]) (logid:) Adding storage adaptor for com.cloud.hypervisor.kvm.storage.FiberChannelAdapter
2025-07-28 08:13:58,538 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2025-07-28 08:13:58,642 INFO [cloud.serializer.GsonHelper] (main:[]) (logid:) Default Builder inited.
2025-07-28 08:13:58,643 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) iscsi session clean up is disabled
2025-07-28 08:13:58,644 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) Skipping the memory balloon stats period setting, since there are no VMs (active Libvirt domains) on this host.
2025-07-28 08:13:58,644 INFO [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) The [vm.memballoon.stats.period] property is set to '0', this prevents memory statistics from being displayed correctly. Adjust (increase) the value of this parameter to correct this.
2025-07-28 08:13:58,646 INFO [cloud.agent.Agent] (main:[]) (logid:) Agent [id = 10, uuid = aeb26945-0f0d-4fdb-8961-3cd755c35501, name = cloudstack2, type = LibvirtComputingResource, zone = 1, pod = 1, workers = 5, port = 8250] with host = 192.168.10.221, local id = 1
2025-07-28 08:13:58,647 INFO [utils.nio.NioClient] (main:[]) (logid:) Connecting to 192.168.10.221:8250
2025-07-28 08:13:58,651 INFO [utils.nio.NioClient] (main:[]) (logid:) Connected to 192.168.10.221:8250
2025-07-28 08:13:58,652 INFO [utils.nio.Link] (main:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2025-07-28 08:13:58,803 INFO [utils.nio.NioClient] (main:[]) (logid:) SSL: Handshake done
2025-07-28 08:13:58,812 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching CPU speed from command "lscpu".
2025-07-28 08:13:58,820 ERROR [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Unable to retrieve the CPU speed from lscpu. java.lang.NullPointerException: Cannot invoke "String.trim()" because "in" is null
at java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1838)
at java.base/jdk.internal.math.FloatingDecimal.parseFloat(FloatingDecimal.java:122)
at java.base/java.lang.Float.parseFloat(Float.java:476)
at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeedFromCommandLscpu(KVMHostInfo.java:141)
at org.apache.cloudstack.utils.linux.KVMHostInfo.getCpuSpeed(KVMHostInfo.java:115)
at org.apache.cloudstack.utils.linux.KVMHostInfo.getHostInfoFromLibvirt(KVMHostInfo.java:207)
at org.apache.cloudstack.utils.linux.KVMHostInfo.(KVMHostInfo.java:68)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:3748)
at com.cloud.agent.Agent.sendStartup(Agent.java:542)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1213)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-07-28 08:13:58,829 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Command [lscpu | grep -i 'CPU max MHz' | head -n 1 | sed 's/^.*: //' | xargs] resulted in the value [4367] for CPU speed.
2025-07-28 08:13:58,829 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:[]) (logid:) Fetching host CPU arch
2025-07-28 08:13:58,836 INFO [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Host uses control group [cgroup2fs].
2025-07-28 08:13:58,837 INFO [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) Calculating the max shares of the host.
2025-07-28 08:13:58,837 INFO [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:[]) (logid:) The max shares of the host is [104808].
2025-07-28 08:13:58,868 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [15770] for command [/bin/bash -c virt-v2v --version ] failed.
2025-07-28 08:13:58,868 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15770] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
2025-07-28 08:13:58,877 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Attempting to create storage pool effe48a3-ce83-47be-999b-607501e7851e (Filesystem) in libvirt
2025-07-28 08:13:58,878 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Found existing defined storage pool effe48a3-ce83-47be-999b-607501e7851e, using it.
2025-07-28 08:13:58,878 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:[]) (logid:) Trying to fetch storage pool effe48a3-ce83-47be-999b-607501e7851e from libvirt
2025-07-28 08:13:59,929 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Process agent startup answer, agent [id: 0, uuid: , name: ] connected to the server
2025-07-28 08:13:59,938 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) (logid:) Startup Response Received: agent [id: 0, uuid: , name: ]
2025-07-28 08:14:00,132 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Attempting to create storage pool 9b0a80be-9a4c-3f26-ad9b-7a48a7d37eeb (NetworkFilesystem) in libvirt
2025-07-28 08:14:00,133 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Found existing defined storage pool 9b0a80be-9a4c-3f26-ad9b-7a48a7d37eeb, using it.
2025-07-28 08:14:00,137 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-4:[]) (logid:) Trying to fetch storage pool 9b0a80be-9a4c-3f26-ad9b-7a48a7d37eeb from libvirt
2025-07-28 08:14:00,315 INFO [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:) Processing agent ready command, agent id = 10, uuid = aeb26945-0f0d-4fdb-8961-3cd755c35501, name = cloudstack2
2025-07-28 08:14:00,317 INFO [cloud.agent.Agent] (AgentRequest-Handler-4:[]) (logid:) Ready command is processed for agent [id: 10, uuid: aeb26945-0f0d-4fdb-8961-3cd755c35501, name: cloudstack2]
2025-07-28 08:14:00,449 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Processing agent ready command, agent id = 10, uuid = aeb26945-0f0d-4fdb-8961-3cd755c35501, name = cloudstack2
2025-07-28 08:14:00,458 ERROR [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Unexpected arch null, expected x86_64
2025-07-28 08:14:00,459 INFO [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:) Processed new management server list: 192.168.10.221@static
2025-07-28 08:14:00,459 INFO [cloud.agent.Agent] (A
Beta Was this translation helpful? Give feedback.
All reactions