Horde OTP 27 issue

[nash8114]

When upgrading our Horde-dependent application to OTP 27 we experienced an issue that requests were not being picked up. This issue should affect all Horde applications which use the default strategy Horde.UniformDistribution

Horde depends on libring to implement Horde.UniformDistribution. Due to changes in OTP 27 libring no longer has a uniform distribution but instead falls back to a "List.first" (actually :gb_trees.smallest(r)). See also bitwalker/libring#35
Because of an implementation choice in libring, the change in OTP causes the algorithm to silently change its behavior. The outcome will always be the same node, regardless of the input.

When all nodes of a DynamicSupervisor agree on the topology, they should choose the same node for a given child_spec. The request is proxied to that node, and its choose_node outcome should pick itself, so it will start the child.
When the algorithms on two nodes disagree, this can cause infinite message proxieing, where node-A selects node-B, and node-B selects node-A. They will proxy to each other infinitely (behavior for which I want to create a PR later).

Nodes with OTP 27 will have different choose_node outcome than OTP 26 or earlier, until libring is patched. Upgrading to OTP 27 is dangerous and can cause messages to bounce around the system, when nodes of OTP 27 and OTP <=26 are alive together. If you do manage to upgrade to OTP 27 on all nodes, then be aware that only a single node will get all start_child requests.

[derekkraan]

This is bizarre. Are you saying that libring simply selects the first node every time on OTP 27?

[nash8114]

That is unfortunately exactly what I'm saying. This is an implementation choice due to a fallback in a case statement in key_to_node. https://github.com/bitwalker/libring/blob/main/lib/ring.ex#L212

Works fine on OTP 26

iex(1)> System.build_info
%{
  version: "1.15.7",
  date: "2023-11-22T00:59:42Z",
  otp_release: "26",
  build: "1.15.7 (compiled with Erlang/OTP 26)",
  revision: ""
}
iex(2)> Mix.install([:libring])
==> libring
Compiling 5 files (.ex)

Generated libring app
:ok
iex(3)> ring = HashRing.new() |> HashRing.add_nodes(~w[a b c d]); Enum.map(1..1000, fn n -> HashRing.key_to_node(ring, n) end) |> Enum.frequencies
%{"a" => 262, "b" => 240, "c" => 246, "d" => 252}

The above gives an approximate fair distribution.

Broken on OTP 27

iex(1)> System.build_info
%{
  version: "1.17.2",
  date: "2024-08-13T20:58:12Z",
  otp_release: "27",
  build: "1.17.2 (compiled with Erlang/OTP 27)",
  revision: ""
}
iex(2)> Mix.install([:libring])
==> libring
Compiling 5 files (.ex)

Generated libring app
:ok
iex(3)> ring = HashRing.new() |> HashRing.add_nodes(~w[a b c d]); Enum.map(1..1000, fn n -> HashRing.key_to_node(ring, n) end) |> Enum.frequencies
%{"b" => 1000}

This time around the result is only "b"

[derekkraan]

Very unfortunate. Have you already raised the issue with the libring project?

[nash8114]

A PR was raised, but there is no issue for it. The PR is still open. PR: bitwalker/libring#35 I don't think it will help to create an issue about it at this point, but I'm willing to do it if you think it does.

…

On Tue, Sep 3, 2024 at 4:45 PM Derek Kraan ***@***.***> wrote: Very unfortunate. Have you already raised the issue with the libring project? — Reply to this email directly, view it on GitHub <#275 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACI4UZ67RUVL4E5SVZL6OTZUXDRDAVCNFSM6AAAAABNRW33YWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRWG4YTONJSGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[derekkraan]

Doesn't seem necessary. I'll subscribe to notifications on this PR and issue an update when it is resolved.

[bluesbettle]

@derekkraan libring has been fixed ☝🏻

[derekkraan]

Thanks! The next version of Horde will depend on the latest version. But anyone facing this issue can also just upgrade libring as well. No action really needed on our side.