redis fault tolerance #1
Description
currently, both workers and the experiment daemon will terminate ungracefully if redis goes offline.
instead we should have a circuit breaker mechanism that allows graceful handling of redis faults. ideally everything comes back online.