buckyd and bucky don't agree on the carbon hashring

[zerosoul13]

Hello,

I've raised an issue on the incorrect repo and would like to bring it to the right one. Below my original post on jjneely/buckytools. The content below is just added for everyone to have context on my initial issue jjneely/buckytools#38

---

I've found 2 issues which I would love to discuss:

BuckyD and bucky configuration

buckyd will accept the members of the hashring via non-option cli arguments as buckyd <graphite1:port> <graphite2:port> ....
bucky calls for the cluster configuration and it will get graphite1:hashringport instead of graphite1:4242 because of this mismatch, bucky won't be able to reach the buckyd members

/usr/sbin/bucky servers -h go-carbon-0.go-carbon.graphite:4242
2021/11/10 01:01:57 Error retrieving URL: Get "http://go-carbon-0.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:45902->172.16.76.119:2004: read: connection reset by peer
2021/11/10 01:01:57 Cluster unhealthy: go-carbon-0.go-carbon.graphite:2004: Get "http://go-carbon-0.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:45902->172.16.76.119:2004: read: connection reset by peer

2021/11/10 01:01:57 Error retrieving URL: Get "http://go-carbon-1.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:47772->172.16.27.82:2004: read: connection reset by peer
2021/11/10 01:01:57 Cluster unhealthy: go-carbon-1.go-carbon.graphite:2004: Get "http://go-carbon-1.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:47772->172.16.27.82:2004: read: connection reset by peer

2021/11/10 01:01:57 Error retrieving URL: Get "http://go-carbon-2.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:40946->172.16.58.32:2004: read: connection reset by peer
2021/11/10 01:01:57 Cluster unhealthy: go-carbon-2.go-carbon.graphite:2004: Get "http://go-carbon-2.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:40946->172.16.58.32:2004: read: connection reset by peer

2021/11/10 01:01:57 Error retrieving URL: Get "http://go-carbon-3.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:43570->172.16.127.44:2004: read: connection reset by peer
2021/11/10 01:01:57 Cluster unhealthy: go-carbon-3.go-carbon.graphite:2004: Get "http://go-carbon-3.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:43570->172.16.127.44:2004: read: connection reset by peer

2021/11/10 01:01:57 Error retrieving URL: Get "http://go-carbon-4.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:51674->172.16.27.91:2004: read: connection reset by peer
2021/11/10 01:01:57 Cluster unhealthy: go-carbon-4.go-carbon.graphite:2004: Get "http://go-carbon-4.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:51674->172.16.27.91:2004: read: connection reset by peer

2021/11/10 01:01:57 Error retrieving URL: Get "http://go-carbon-5.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:56712-172.16.14.79:2004: read: connection reset by peer
2021/11/10 01:01:57 Cluster unhealthy: go-carbon-5.go-carbon.graphite:2004: Get "http://go-carbon-5.go-carbon.graphite:2004/hashring": read tcp 172.16.14.79:56712->172.16.14.79:2004: read: connection reset by peer`

Buckd daemons are using port: 4242
Hashing algorithm: [carbon: 6 nodes, 100 replicas, 600 ring members go-carbon-0.go-carbon.graphite:2004=None go-carbon-1.go-carbon.graphite:2004=None go-carbon-2.go-carbon.graphite:2004=None go-carbon-3.go-carbon.graphite:2004=None go-carbon-4.go-carbon.graphite:2004=None go-carbon-5.go-carbon.graphite:2004=None]
Number of replicas: 100
Found these servers:
        go-carbon-0.go-carbon.graphite:2004
        go-carbon-1.go-carbon.graphite:2004
        go-carbon-2.go-carbon.graphite:2004
        go-carbon-3.go-carbon.graphite:2004
        go-carbon-4.go-carbon.graphite:2004
        go-carbon-5.go-carbon.graphite:2004

Is cluster healthy: false
2021/11/10 01:01:57 Cluster is inconsistent.

I've tracked the issue to line https://github.com/go-graphite/buckytools/blob/master/cmd/bucky/cluster.go#L88. The the port value for the cluster member is set to the same port as the hashring one instead of 4242 (or whichever port is specified by user).

To test this theory, I've forked and patched the code to set it to default 4242 and cluster is reported as healthy with the correct hashring values as below

/ # /usr/sbin/bucky servers -h go-carbon-5.go-carbon.graphite:4242
Buckd daemons are using port: 4242
Hashing algorithm: [carbon: 6 nodes, 100 replicas, 600 ring members go-carbon-0.go-carbon.graphite:2004=None go-carbon-1.go-carbon.graphite:2004=None go-carbon-2.go-carbon.graphite:2004=None go-carbon-3.go-carbon.graphite:2004=None go-carbon-4.go-carbon.graphite:2004=None go-carbon-5.go-carbon.graphite:2004=None]
Number of replicas: 100
Found these servers:
        go-carbon-0.go-carbon.graphite:4242
        go-carbon-1.go-carbon.graphite:4242
        go-carbon-2.go-carbon.graphite:4242
        go-carbon-3.go-carbon.graphite:4242
        go-carbon-4.go-carbon.graphite:4242
        go-carbon-5.go-carbon.graphite:4242

Is cluster healthy: true

Is this a real issue or just a misconfiguration on my side?

Inconsistent metric count will almost match active metric count

2. bucky is reporting metrics as inconsistent on our cluster and the number is nearly the same as the active metrics one which is very odd. Taking a closer look, this line https://github.com/go-graphite/buckytools/blob/master/cmd/bucky/inconsistent.go#L69 does check the port values and these don't match because one is 2004 and the other is 4242.

The original code does not take the ports into account, just the hostnames
https://github.com/jjneely/buckytools/blob/master/cmd/bucky/inconsistent.go#L64

Is my assumption that these rings won't match because of this correct?

[zerosoul13]

For the inconsistent count, I've patched the code to make it match the original one (https://github.com/jjneely/buckytools/blob/master/cmd/bucky/inconsistent.go#L64). Here are my results

Before patch

2021/11/10 23:03:27 go-carbon-0.go-carbon.graphite:4242 returned 5985241 metrics
2021/11/10 23:03:23 go-carbon-1.go-carbon.graphite:4242 returned 5418552 metrics
2021/11/10 23:03:26 go-carbon-2.go-carbon.graphite:4242 returned 5963922 metrics
2021/11/10 23:03:30 go-carbon-3.go-carbon.graphite:4242 returned 6255700 metrics
2021/11/10 23:03:23 go-carbon-4.go-carbon.graphite:4242 returned 5234422 metrics
2021/11/10 23:03:15 go-carbon-5.go-carbon.graphite:4242 returned 5147329 metrics

2021/11/10 23:03:30 Hashing.
2021/11/10 23:03:57 Hashing time was: 27s..
2021/11/10 23:04:00 5985117 inconsistent metrics found on go-carbon-0.go-carbon.graphite:4242
2021/11/10 23:03:57 5418428 inconsistent metrics found on go-carbon-1.go-carbon.graphite:4242
2021/11/10 23:03:58 5963798 inconsistent metrics found on go-carbon-2.go-carbon.graphite:4242
2021/11/10 23:04:02 6255576 inconsistent metrics found on go-carbon-3.go-carbon.graphite:4242
2021/11/10 23:04:05 5234298 inconsistent metrics found on go-carbon-4.go-carbon.graphite:4242
2021/11/10 23:04:04 5147209 inconsistent metrics found on go-carbon-5.go-carbon.graphite:4242


After patch

2021/11/11 18:55:19 go-carbon-0.go-carbon.graphite:4242 returned 6046211 metrics
2021/11/11 18:55:17 go-carbon-1.go-carbon.graphite:4242 returned 5473367 metrics
2021/11/11 18:55:19 go-carbon-2.go-carbon.graphite:4242 returned 6025582 metrics
2021/11/11 18:55:21 go-carbon-3.go-carbon.graphite:4242 returned 6317835 metrics
2021/11/11 18:55:16 go-carbon-4.go-carbon.graphite:4242 returned 5288551 metrics
2021/11/11 18:54:56 go-carbon-5.go-carbon.graphite:4242 returned 5201150 metrics


2021/11/11 18:55:21 Hashing...
2021/11/11 18:55:43 Hashing time was: 22s
2021/11/11 18:55:43 71411 inconsistent metrics found on go-carbon-0.go-carbon.graphite:4242
2021/11/11 18:55:43 65707 inconsistent metrics found on go-carbon-1.go-carbon.graphite:4242
2021/11/11 18:55:43 71322 inconsistent metrics found on go-carbon-2.go-carbon.graphite:4242
2021/11/11 18:55:43 73072 inconsistent metrics found on go-carbon-3.go-carbon.graphite:4242
2021/11/11 18:55:43 63311 inconsistent metrics found on go-carbon-4.go-carbon.graphite:4242
2021/11/11 18:55:43 62869 inconsistent metrics found on go-carbon-5.go-carbon.graphite:4242

Am I on the right track here?

[zerosoul13]

rebalance command has the same issue with using the same port as hashring to contact the buckyd cluster instead of 4242

2021/11/11 19:27:17 go-carbon-5.go-carbon.graphite:4242 returned 5201150 metrics
2021/11/11 19:27:18 go-carbon-1.go-carbon.graphite:4242 returned 5473367 metrics
2021/11/11 19:27:19 go-carbon-4.go-carbon.graphite:4242 returned 5288551 metrics
2021/11/11 19:27:19 go-carbon-0.go-carbon.graphite:4242 returned 6046211 metrics
2021/11/11 19:27:19 go-carbon-2.go-carbon.graphite:4242 returned 6025582 metrics
2021/11/11 19:27:20 go-carbon-3.go-carbon.graphite:4242 returned 6317835 metrics
2021/11/11 19:27:20 Hashing...
2021/11/11 19:27:42 Hashing time was: 22s
2021/11/11 19:27:42 63311 inconsistent metrics found on go-carbon-4.go-carbon.graphite:4242
2021/11/11 19:27:42 71411 inconsistent metrics found on go-carbon-0.go-carbon.graphite:4242
2021/11/11 19:27:42 71322 inconsistent metrics found on go-carbon-2.go-carbon.graphite:4242
2021/11/11 19:27:42 73072 inconsistent metrics found on go-carbon-3.go-carbon.graphite:4242
2021/11/11 19:27:42 62869 inconsistent metrics found on go-carbon-5.go-carbon.graphite:4242
2021/11/11 19:27:42 65707 inconsistent metrics found on go-carbon-1.go-carbon.graphite:4242
2021/11/11 19:27:43 71411 metrics on go-carbon-0.go-carbon.graphite:4242 must be relocated
2021/11/11 19:27:43 65707 metrics on go-carbon-1.go-carbon.graphite:4242 must be relocated
2021/11/11 19:27:43 71322 metrics on go-carbon-2.go-carbon.graphite:4242 must be relocated
2021/11/11 19:27:43 73072 metrics on go-carbon-3.go-carbon.graphite:4242 must be relocated
2021/11/11 19:27:43 63311 metrics on go-carbon-4.go-carbon.graphite:4242 must be relocated
2021/11/11 19:27:43 62869 metrics on go-carbon-5.go-carbon.graphite:4242 must be relocated
2021/11/11 19:27:43 Syncing 407692 metrics. Copy offload: true.
2021/11/11 19:27:43 Total workers: 750
2021/11/11 19:27:43 Error communicating with server: Post "http://go-carbon-4.go-carbon.graphite:2004/metrics/some.metric.v1.metric?fetch_offload=true": read tcp 172.16.12.115:44786->172.16.0.10:2004: read: connection reset by peer
2021/11/11 19:27:43 Error communicating with server: Post "http://go-carbon-4.go-carbon.graphite:2004/metrics/some.metric.v2.metric?fetch_offload=true": read tcp 172.16.12.115:44810->172.16.0.10:2004: read: connection reset by peer
2021/11/11 19:27:43 Error communicating with server: Post "http://go-carbon-4.go-carbon.graphite:2004/metrics/some.metric.v3.metric?fetch_offload=true": read tcp 172.16.12.115:44788->172.16.0.10:2004: read: connection reset by peer
2021/11/11 19:27:43 Error communicating with server: Post "http://go-carbon-4.go-carbon.graphite:2004/metrics/some.metric.v4.metric?fetch_offload=true": read tcp 172.16.12.115:44808->172.16.0.10:2004: read: connection reset by peer

[deniszh]

Hi @zerosoul13

Looks like something is really fishy with your setup, but I have no idea what exactly. Maybe @bom-d-van have any ideas?

[bom-d-van]

/usr/sbin/buckyd --node go-carbon-5.go-carbon.graphite -b 0.0.0.0:4242 -hash carbon -p /mnt/var/lib/graphite/whisper/ -t /tmp -timeout 7200 go-carbon-0.go-carbon.graphite:2004 go-carbon-1.go-carbon.graphite:2004 go-carbon-2.go-carbon.graphite:2004 go-carbon-3.go-carbon.graphite:2004 go-carbon-4.go-carbon.graphite:2004 go-carbon-5.go-carbon.graphite:2004

Hi @zerosoul13 , is there any reason that you are running one buckyd on 4242 but the other peers on 2004?

In theory, the buckyd should be running as:

/usr/sbin/buckyd \
    --node go-carbon-5.go-carbon.graphite \
    -b 0.0.0.0:4242 \
    -hash carbon \
    -p /mnt/var/lib/graphite/whisper/ \
    -t /tmp \
    -timeout 7200 \
    go-carbon-0.go-carbon.graphite:4242 \
    go-carbon-1.go-carbon.graphite:4242 \
    go-carbon-2.go-carbon.graphite:4242 \
    go-carbon-3.go-carbon.graphite:4242 \
    go-carbon-4.go-carbon.graphite:4242 \
    go-carbon-5.go-carbon.graphite:4242

There was indeed a new change from this commit to make bucky tool use both the peer address and port number returned from the seed buckyd node rather than using just the hostname/fqdn/ip.

[zerosoul13]

is there any reason that you are running one buckyd on 4242 but the other peers on 2004?

I'm not doing so. I'm passing the hashring as set on carbon-relay-ng and expect to be able to use the buckyd/bucky tooling with the port 4242. Right now, I have to go with 1 or the other.

The proposed configuration will make things work for us but it won't match against the hashring on carbon-relay-ng because of the port difference. See results after patch https://gist.github.com/zerosoul13/d92aa97c3c7d72d39dbfb1f942de11e1

/usr/sbin/buckyd \
    --node go-carbon-5.go-carbon.graphite \
    -b 0.0.0.0:4242 \
    -hash carbon \
    -p /mnt/var/lib/graphite/whisper/ \
    -t /tmp \
    -timeout 7200 \
    go-carbon-0.go-carbon.graphite:4242 \
    go-carbon-1.go-carbon.graphite:4242 \
    go-carbon-2.go-carbon.graphite:4242 \
    go-carbon-3.go-carbon.graphite:4242 \
    go-carbon-4.go-carbon.graphite:4242 \
    go-carbon-5.go-carbon.graphite:4242

@bom-d-van, how would this hashring members match my carbon-relay-ng configuration if my original hashrigh from carbon-relay-ng uses another ports or will it not matter at all and still produce the same carbon hashring?

The code I've edited so far, allows me to declare the hashring same as the original one (in my head it does make a difference but could be totally wrong) and still maintain connectivy to the buckyd cluster.

We can take this as example: #31 (comment). The number of inconsistent metrics went down drastically versus the out of the unpatched code.

[zerosoul13]

I started chasing this issue as the count inconsistent metrics was quite high. I saw that the original code from jjnelly/buckytools does not take into account the port https://github.com/jjneely/buckytools/blob/master/cmd/bucky/inconsistent.go#L64 but this fork does take the ports into account https://github.com/go-graphite/buckytools/blob/master/cmd/bucky/inconsistent.go#L69 which is why I believe that this the hashring won't be able to fully match against carbon-relay-ng configuration

[bom-d-van]

how would this hashring members match my carbon-relay-ng configuration if my original hashrigh from carbon-relay-ng uses another ports or will it not matter at all and still produce the same carbon hashring?

Hi @zerosoul13 , the most importation thing is the order, rather than the hostname-port tuple.

We just need to make sure that the order of buckyd daemon is the same as the go-carbon daemon list configured in carbon-relay-ng, and we are good.

For the hashing logics, you can check this out: https://github.com/go-graphite/buckytools/blob/master/hashing/hashing.go#L337-L353

The list of peers of buckyd command is later used by buckytools to communicate with other bucyd peers for rebalancing and buckytools actually don't communicate with go-carbon for rebalancing.

Therefore if you are running your buckyd on port 4242 on all instances, you need to reflect it in the buckyd command by using this:

/usr/sbin/buckyd \
    --node go-carbon-5.go-carbon.graphite \
    -b 0.0.0.0:4242 \
    -hash carbon \
    -p /mnt/var/lib/graphite/whisper/ \
    -t /tmp \
    -timeout 7200 \
    go-carbon-0.go-carbon.graphite:4242 \
    go-carbon-1.go-carbon.graphite:4242 \
    go-carbon-2.go-carbon.graphite:4242 \
    go-carbon-3.go-carbon.graphite:4242 \
    go-carbon-4.go-carbon.graphite:4242 \
    go-carbon-5.go-carbon.graphite:4242

[bom-d-van]

[zerosoul13]

Thank you for the guidance. I was mistaken on how the hashring was being created and the elements it took into account to build it.

[Thorsieger]

bucky calls for the cluster configuration and it will get graphite1:hashringport instead of graphite1:4242 because of this mismatch, bucky won't be able to reach the buckyd members

We just need to make sure that the order of buckyd daemon is the same as the go-carbon daemon list configured in carbon-relay-ng, and we are good.

I think this is a real issue, not in your case because you are using carbon_ch algorithm that just use hostname/ip to calculate the hashring

In my case I use fnv1a_ch that use both ip and port. In that case if a set the instances as :
go-carbon-0.go-carbon.graphite:4242
go-carbon-1.go-carbon.graphite:4242

the cluster will be healthy but it will said that metrics are not at the right position

[bom-d-van]

Hi @Thorsieger I think fnv1a_ch is also counting on host order and is similar to carbon_ch (which uses md5 as hashing algorithm).

https://github.com/go-graphite/buckytools/blob/master/hashing/fnv1a.go#L92-L100

Or do you think I'm missing something important?

[Thorsieger]

Yes the order is important, it use different hashing algorithm but so is data entry :

For carbon_ch it use hostname (+instance) : https://github.com/go-graphite/buckytools/blob/master/hashing/hashing.go#L270-L275

fnv1a_ch use (hostname + port) OR instance : https://github.com/go-graphite/buckytools/blob/master/hashing/fnv1a.go#L34-L39

I first found more info here on carbon-c-relay git

[bom-d-van]

@Thorsieger yeah, I think you are right. Thanks for flagging it. Only jump hash is unaffected by the changes. but carbon_ch and fnv1a_ch are affected: https://github.com/go-graphite/buckytools/blob/master/hashing/hashing.go#L308-L309

cc @zerosoul13 I will reopen the ticket.