Rest do not respond for ~30sec after Rebooting with leaders having assigned block

[SViksha]

Describe the bug
Rest do not respond for ~30 sec after rebooting with leaders having assigned block.
Passive node work fine this problem is only with leader having assigned block.

Mandatory Information

1. jcli 0.8.14 (HEAD-9ca427ef, release, linux [x86_64]) - [rustc 1.41.1 (f3e1a954d 2020-02-24)]
2. jormungandr 0.8.14 (HEAD-9ca427ef, release, linux [x86_64]) - [rustc 1.41.1 (f3e1a954d 2020-02-24)]

To Reproduce
Steps to reproduce the behavior:

1. Make sure that the leader has blocks assigned more the block better to reproduce.
2. Reboot the node with secrete file
3. Start the script to monitor the node
   ./jcli rest v0 node stats get --host "http://127.0.0.1:<rest port./api"|grep "version|state|lastBlockDate|lastBlockHeight|lastBlockTime|uptime|lastBlockHash"
4. Start the script to delete the leader with 1 sec interval
   watch -n 1 /jcli rest v0 leaders delete 1 --host "http://127.0.0.1:/api"
5. Monitoring will show the status: Bootstrapping but as soon as the node is bootstrapped, the rest stop responding for ~30sec and do not delete for 30sec
   (this issue created the adversarial fork as backup not failed to delete the leaders)

Expected behavior
leader is deleted within a second of bootstraping.

Additional context
If this issue is not resolvable , there should be option in config file to avoid the leaders in enclave after rebooting but load the leader schedule.

[bergr01]

I'm guessing similar issue is what is causing the following entry in leader logs shortly after epoch transition:

inished_at_time: "2020-03-17T19:14:34.538457764+00:00"
scheduled_at_date: "95.12"
scheduled_at_time: "2020-03-17T19:14:01+00:00"
status:
Rejected:
reason: Missed the deadline to compute the schedule
wake_at_time: "2020-03-17T19:14:34.538399753+00:00"

and this error in the node logs around the same time:

Mar 17 20:14:34.538 ERRO Eek... Too late, we missed an event schedule, system time might be off?, event_end: 2020-03-17T19:14:03+00:00, event_start: 2020-03-17T19:14:01+00:00, event_date: 95.12, leader_id: 4, task: leadership

as you can see, the running node was stuck in some processing thread for 20 seconds before it could finally try and action the scheduled leader event, by which point it was too late

[gufmar]

the behaviour of jcli seems different now in 0.9.0
After bootstrap completed jormungandr note sometimes does not immediately react on rest requests to remove the leader secret.
in 0.8.x this resulted in a ~30-60 second timeout error for jcli
now in 0.9.0 jcli immediately end the request but the secret remain active

My watchdog script already covers this and retry to remove the secret until it sees it removes (leaders get) afterwards. Hence why I can say: instead of the long timeout in 0.8.x now a series of removal requests between 30-60 seconds fail until one succeeds.

So it works, if your script retries long enough.
But it's not ideal because the risk window for adversarial forks is longer than required.