Issue with Fenced

[leo79901]

HI,
I have configured the host's HA provider and enabled Out-of-band Management (OOBM). I have also tested the OOBM functionality and am sure that it is working properly.

Then the HA state of the host was initially set to Available.

However, when I cut the power of one host, the host enters the Fencing status within 1 minute and remains in that state for more than 1-3 hour. After that, the status changes to fenced, and the VMs on the host are restarted on other hosts by the HA.

At the same time, I saw this on the ShapeBlue's website:

Fencing – the Host-HA framework is trying to Fence the host by issuing OOBM job

Yes, I have seen these logs. ACS try to shutdown the host by OOBM but faild. thers is 2100 try's with a host down.

2024-12-03 14:36:30,219 WARN  [o.a.c.h.t.BaseHATask] (pool-4-thread-82:ctx-af691a2f) (logid:670480bb) Exception occurred while running FenceTask on a resource: org.apache.cloudstack.ha.provider.HAFenceException: OBM service is not configured or enabled for this host idnode10
org.apache.cloudstack.ha.provider.HAFenceException: OBM service is not configured or enabled for this host idnode10
        at org.apache.cloudstack.kvm.ha.KVMHAProvider.fence(KVMHAProvider.java:100)
        at org.apache.cloudstack.kvm.ha.KVMHAProvider.fence(KVMHAProvider.java:42)
        at org.apache.cloudstack.ha.task.FenceTask.performAction(FenceTask.java:42)
        at org.apache.cloudstack.ha.task.BaseHATask$1.call(BaseHATask.java:86)
        at org.apache.cloudstack.ha.task.BaseHATask$1.call(BaseHATask.java:83)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Out-of-band Management action [OFF] on Host {"id":14,"name":"idnode10","type":"Routing","uuid":"b570ac0f-aad5-4f90-a62e-46e710db3763"} failed with error: Set Chassis Power Control to Down/Off failed: Command not supported in present state

        at org.apache.cloudstack.outofbandmanagement.OutOfBandManagementServiceImpl.executePowerOperation(OutOfBandManagementServiceImpl.java:437)
        at jdk.internal.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        ... 21 more

Then the status change to Fenced:

2024-12-03 12:28:53,340 WARN  [c.c.a.AlertManagerImpl] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) alertType=[7] dataCenterId=[1] podId=[1] clusterId=[null] message=[Host is down, name: idnode10 (id:14), availability zone: Zone1, pod: Pod].
2024-12-03 12:28:53,354 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) Notifying HA Mgr of to restart vm 50-i-2-50-VM
2024-12-03 12:28:53,357 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) HA schedule restart
2024-12-03 12:28:53,468 INFO  [c.c.h.HighAvailabilityManagerImpl] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) Schedule vm for HA:  VM instance {"id":50,"instanceName":"i-2-50-VM","type":"User","uuid":"94c170b3-9780-43aa-ab0c-eaaf56fec475"}
2024-12-03 12:28:53,468 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) Wakeup workers HA
2024-12-03 12:28:53,477 INFO  [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-e9d23278 work-3326) (logid:a1f6d350) Processing work HAWork[3326-HA-50-Running-Investigating]
2024-12-03 12:28:53,477 DEBUG [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-e9d23278 work-3326) (logid:a1f6d350) RESTART with HAWORK
2024-12-03 12:28:53,489 INFO  [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-e9d23278 work-3326) (logid:a1f6d350) HA on VM instance {"id":50,"instanceName":"i-2-50-VM","type":"User","uuid":"94c170b3-9780-43aa-ab0c-eaaf56fec475"}
2024-12-03 12:28:53,498 DEBUG [c.c.r.ResourceState] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) Resource state update: [id = 14; name = idnode10; old state = Enabled; event = InternalEnterMaintenance; new state = Maintenance]
2024-12-03 12:28:53,500 WARN  [c.c.a.AlertManagerImpl] (pool-4-thread-139:ctx-efd70258) (logid:60f5a05f) alertType=[30] dataCenterId=[1] podId=[1] clusterId=[null] message=[HA Fencing of host id=14, in dc id=1 performed].
2024-12-03 12:28:53,510 DEBUG [o.a.c.h.HAManagerImpl] (HA-Worker-4:ctx-e9d23278 work-3326) (logid:a1f6d350) HA: Host [14] is fenced.

So my issue are:

Why does the fencing process take such a long time?
How can I make the HA process quicker?

[leo79901]

I don't know much about Java. This is just a conclusion drawn from my little understanding of the code.

I think there may be a bug.

I found these code here:

    @Override
    public boolean fence(Host r) throws HAFenceException {

        try {
            if (outOfBandManagementService.isOutOfBandManagementEnabled(r)){
                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
                return resp.getSuccess();
            } else {
                LOG.warn("OOBM fence operation failed for this host " + r.getName());
                return false;
            }
        } catch (Exception e){
            LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
            throw new HAFenceException("OBM service is not configured or enabled for this host " + r.getName() , e);
        }
    }

ACS wants to shut down the host by IPMI in the fencing step.
from here:

  if (outOfBandManagementService.isOutOfBandManagementEnabled(r)){
                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);

finally , it run this command:
/usr/bin/ipmitool -I lanplus -R 1 -v -H 10.21.0.22 -p 623 -U root -P XXXXXXXXX chassis power off

But it is wrong while shutting down a server which is already shut. I have run it manually and get this error.

[root@idnode1 ~]# /usr/bin/ipmitool -I lanplus -R 1 -v -H 10.21.0.22 -p 623 -U root -P XXXXXXXXX chassis power off
Set Chassis Power Control to Down/Off failed: Command not supported in present state

So it was run again and again.

Maybe we can add some code to resolve it.
Return true if the host is already down

public boolean fence(Host r) throws HAFenceException {

    try {
        // check host status
        if (r.getStatus().equals(Host.Status.DOWN)) {
            LOG.info("Host " + r.getName() + " is already down. Returning success.");
            return true;
        }

        if (outOfBandManagementService.isOutOfBandManagementEnabled(r)) {
            final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
            return resp.getSuccess();
        } else {
            LOG.warn("OOBM fence operation failed for this host " + r.getName());
            return false;
        }
    } catch (Exception e) {
        LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
        throw new HAFenceException("OBM service is not configured or enabled for this host " + r.getName(), e);
    }
}

[slavkap]

Sorry , the checking code should be here：

public boolean fence(Host r) throws HAFenceException {
    try {
        if (outOfBandManagementService.isOutOfBandManagementEnabled(r)) {
            // check host status
            if (r.getStatus().equals(Host.Status.DOWN)) {
                LOG.info("Host " + r.getName() + " is already down. Returning success.");
                return true;
            } else {
                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
                return resp.getSuccess();
            }
        } else {
            LOG.warn("OOBM fence operation failed for this host " + r.getName());
            return false;
        }
    } catch (Exception e) {
        LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
        throw new HAFenceException("OBM service is not configured or enabled for this host " + r.getName(), e);
    }
}

One suggestion here if you're using a build with this change

public boolean fence(Host r) throws HAFenceException {
    try {
        if (outOfBandManagementService.isOutOfBandManagementEnabled(r)) {
            // check host status
            if (r != null && Host.Status.DOWN.equals(r.getStatus())) {
                LOG.info("Host " + r.getName() + " is already down. Returning success.");
                return true;
            } else {
                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
                return resp.getSuccess();
            }
        } else {
            LOG.warn("OOBM fence operation failed for this host " + r.getName());
            return false;
        }
    } catch (Exception e) {
        LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
        throw new HAFenceException("OBM service is not configured or enabled for this host " + r.getName(), e);
    }
}

[DaanHoogland]

not to be pedantic @slavkap but don't you mean

public boolean fence(Host r) throws HAFenceException {
    try {
        // host exists and is managed OOB
        if (r != null && outOfBandManagementService.isOutOfBandManagementEnabled(r)) {
            // check host status
            if (Host.Status.DOWN.equals(r.getStatus())) {
                LOG.info("Host " + r.getName() + " is already down. Returning success.");
                return true;
            } else {
                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
                return resp.getSuccess();
            }
        } else {
            LOG.warn("OOBM fence operation failed for this host " + r.getName());
            return false;
        }
    } catch (Exception e) {
        LOG.warn("OOBM service is not configured or enabled for this host " + r.getName() + " error is " + e.getMessage());
        throw new HAFenceException("OBM service is not configured or enabled for this host " + r.getName(), e);
    }
}

[slavkap]

@DaanHoogland, yes, that would be better :)

[slavkap]

@leo79901, have you tested this? I think that the host status is changed to Down after the host is fenced not before that

[leo79901]

Yes, sometimes the host was shut down due to power, CPU, or board issues, so this problem occurred.

[DaanHoogland]

thanks @leo79901 , could you enter a PR with this change? We can discuss and test it there. At first sight there is nothing wrong with your proposed code.

[leo79901]

I will do it , thanks.

[returntrip]

Am I understanding this correctly that this PR should fix this issue: #10039 and this will be released when 4.21.0 is out?