Allow constraining dynamic listener ports to a specific port interval (range)

[alex-berger]

A common operating scheme for Kafka and Kafka-compatible systems like for example Redpanda, is to apply rolling upgrades with monotonically increasing broker IDs. In our specific use case, using Redpanda Cloud, it turned out that broker IDs are monotonically increasing with each version upgrade or replacement (e.g. if a broker is move to another machine). Broker IDs are implemented as signed 32-bit integers, and by conventions only zero and positive integers are used. TCP ports are 16-bit unsigned integers and thus we cannot simply increase port numbers beyond what fits into a 16-bit unsigned integer and we can also not use Broker IDs as port numbers (e.g. as is currently implemented for deterministic listeners).

Therefore with this PR we introduce the following backward compatible changes:

• refactor port numbers to be of type uint16 (instead of int).
• introduce a new configuration (and command line) parameter dynamic-sequential-max-ports, specifying the maximum number of ports that can be used for dynamic listeners. Effectively, in combination with dynamic-sequential-min-port allowing users to define a half open port interval [dynamic-sequential-min-port, dynamic-sequential-min-port + dynamic-sequential-max-ports), which is used to allocated dynamic ports.
• adding configuration validation to ensure that:
  • if deterministic-listeners or dynamic-sequential-max-ports is set, dynamic-sequential-min-port must also be set to a positive (non-zero) value.
  • dynamic-sequential-max-ports defaults to 65536 - dynamic-sequential-min-port if dynamic-sequential-min-portis set to a non-zero value.

With those changes in place, dynamic port allocation will now only allocate ports from the above mentioned half open interval and will automatically wrap-over (safely-overflow) to the start of the interval when it reaches the end of the interval. The only exception is if dynamic-sequential-min-port is not set respectively is set to 0, in that case we still delegate allocation of (random) free ephemeral ports to the OS.

Motivation for this PR

Why do we need this changes, which problems are we solving?

• Ensure dynamic deterministic listeners work even if Broker IDs are greater than 65335
• Ensure dynamic sequential listeners work even if the number of allocated ports overflows 65535
• Enable SecOps to limit the port-interval they have to open in firewalls (e.g. SecurityGroups or NetworkPolicies)
• Ease usage of kafka-proxy behind a Kubernetes Service, which needs explicit definition of all ports.