How to failover better when master and worker down?

[caishunfeng]

Now DS have logic of failoverWorker and failoverMaster, but I think it's not perfect, especially if master down, and I have some question for failover :

• How to judge the taskInstance or processInstance if need fault tolerance?
• When master down, how does other master failover it? May be the failover logic should including process instance and task instance.
• If more than one workers or masters down, how to failover it?

If you have some question or idea about failover, welcome to discussion.

[caishunfeng]

When master lost zk connection:

1. update current server state to wait reconnect
2. send server lost connection alert
3. keep quartz working (it will ensure work normally by quartz and db)
4. stop accepting new request
5. stop handling commands and process instances, clear the local running process instances; (it will be take over by other master)
6. wait to reconnect
7. when reconnect successfully, send server recover alert, update server state to normal and recover working

When worker lost zk connection:

1. update current server state to wait reconnect
2. send server lost connection alert
3. kill the running task (it will be task over by master and rerun)
4. stop accepting new request
5. wait to reconnect within a certain time
6. if reconnect timeout, stop itself
7. when reconnect successfully, send server recover alert, update server state to normal and recover working;