[BUG] Low Grade Memory Growth in Cache Controller Refresh #122
Description
Describe the bug
The updateCacheController hook is designed to delete and refresh the cache controllers for all tracked remote clusters. Right now, the goroutines allocated for the cache controllers aren't being released when this happens. Instead, they're being parked (https://golang.org/src/runtime/proc.go). Work is needed to figure out why these goroutines are being parked and resolve that issue. Made #123 to address the symptoms, but needs work to identify and fix the root cause.
Steps To Reproduce
Run admiral with at least one remote cluster (the more there are, the faster the leak) and pprof enabled, and take goroutine dumps periodically. You will see large (and increasing) numbers of parked goroutines and a corresponding memory increase.
Expected behavior
Admiral is able to refresh its controllers without seeing a permanent increase in parked goroutines or memory usage.