Consul Service Discovery Watch AllDatacenters is not working.

[kcajmagic]

Overview

From testing xmidt-org/talaria#136 it appeared the AllDatacenters flag is not working in the watch config.

Version

talaria:
  version:      v0.5.3-1-gc65e045-dirty
  go version:   go1.14.2
  built time:   2020-06-11 23:33:47
  git commit:   c65e045
  os/arch:      linux/amd64

which has pinned webpa-common version of v1.10.2-0.20200611231934-64a391cb4d8d (aka. this commit)

Observed Results

• using docker-compose
• create a additional caducei with each pointing to a different dc
• configure a talaria to watch for caduceus with the config option allDatacenters: true
• start up the containers
• The following line will be observed

{"component":"consulWatcher","event":{"Key":"caduceus[stage=dev flavor=docker]{passingOnly=true}{datacenter=*all*}","Service":"caduceus","Instancer":{"Instancer":{}},"EventCount":2,"Instances":["http://caduceus-1:6000"],"Err":null,"Stopped":false},"level":"debug","msg":"received update route event","ts":"2020-06-11T20:13:07.93670309Z"}

Expected Results

The observed log line should show all caducei not just the caducei from a single datacenter

The Problem

The consul service discovery gets all of the datacenters but fails to watch any of them concurrently and uses the same key for each sd.Instancer

webpa-common/service/consul/environment.go

Line 154 in 64a391c

i.Set(key, newInstancer(l, c, w))