Docker pull should be retried infinitely

[aermakov-zalando]

If all 3 attempts to docker pull fail, the boot process is aborted and the instance then proceeds to do nothing. We should either retry the pull infinitely (with a backoff), or implement the solution proposed in #441 and terminate the instance.

[a1exsh]

The suggestion in #441 wasn't to "terminate the instance". It was describing a problem where this doesn't happen, because LB configuration is special.

I wonder in which context did you see this issue? Why wasn't it terminated after "proceeding to do nothing", i.e. not sending success signal to the CF and not responding to health check?

[aermakov-zalando]

Because the instance in question was part of an etcd cluster and the template we're using doesn't have any health checks.

[a1exsh]

OK, that's an important detail. In any case it wouldn't make sense to me to try to terminate instance from taupage (not even sure if that's possible, or at least would require to grant permission to terminate any EC2 instance in the VPC to every instance profile role of every app).

[aermakov-zalando]

I'm not sure if rebooting would be any better — if the instance is somehow broken (e.g. spawned on a faulty hypervisor, ran out of space so the init scripts can't proceed or the volume is corrupted beyond repair) it'd be better to just terminate it and let the ASG to spawn a healthy one instead of just rebooting in a loop. We can make it configurable, though.

[a1exsh]

Making it configurable is a good idea, yes.