Use Node-IPs for external reachability and localities

[adwk67]

Scope

This issue involves the following topics:

• ensuring that HBase is reachable externally via listener endpoints (see parent issue)
• ensure that these endpoints (or a load balancer that uses them) is consistently used in the UI screens
• ensure the HBase advertises the nodes it is running on so that e.g. an accurate locality can be derived

UI endpoint resolution

The endpoints exposed in the HBase UI (mainly the master) are not resolvable for end-users. This means that, while it is possible to use the listener classes to write discovery information to ConfigMaps (so that users can easily reach master/regionserver/rest-server UIs), links referenced in these webpages - most obviously the list of region servers - are dead.

Context

• Hbase uses the following internally for hostname and port resolution. If these settings are not in hbase-site.xml then a default mechanism is used.
  • hbase.{ master | regionserver| rest }.{ hostname | port }
• the default mechanism involves a DNS lookup (see here and here), which also requires that reverse DNS lookups not be disabled. See here.
• the endpoint information written to Zookeeper is what is used for listing the regionservers, and is also used for internal binding
  • this causes problems as HBase does not have access to the host network
• tweaking internal details for the sake of user-facing endpoints runs the risk of breaking things, so a safer solution would be to patch HBase to use listener information only for the purpose of providing externally-reachable endpoints

Proposed solution

• extend proto structs and use where needed - see below Use Node-IPs for external reachability and localities #641 (comment)

Node advertisement

• the aim is to use the proto struct extensions when calculating localities etc. Not yet researched.

[adwk67]

Notes

HBase master- (HMaster extends HRegionServer) and region-servers (HRegionServer) do not have a direct connection: server data is persisted/shared in ZooKeeper and also in HBase meta tables. The data in ZooKeeper uses protobuf entities which serialize from / deserialize to ServerName objects, which are used in a variety of places in the code to share server data (servlet rendering, RPC calls, ZooKeeper persistence etc. etc.).

Listener data (hostname and port) can be written to custom config settings in hbase-site.xml on start-up, once the listener data has been written to the pods. These settings can then be retrieved for each server. However, the host/port are used for internal RPC calls as well as UI data, and node IPs (in the case of external-stable/external-unstable listener classes) are not reachable without giving access to the host network.

Extending proto entities

Extending ServerName (and the associated protobuf class) to add two new fields - externalHostname and externalPort - has the advantage of keeping that data with other server meta data: when the active master changes (or when region servers are added) the existing mechanisms for tracking these changes are unaffected. However, this is fairly invasive as ServerName is the object used in multiple places in the code. If we want to use these fields in a just-in-time manner, then MasterStatusServlet needs to be patched to replace hostname/port with the "external" values. This is not that simple, though, as the whole ServerName object is used as the key for internal caching, and the link is thus broken between the server name and the metrics that should also be rendered in the UI.

Changes to UI rendering

Alternatively, we can leave the MasterStatusServlet code as-is (i.e. do not attempt to replace host with external host etc.) and make the changes in the .jamon and .jsp files, where the host/port are ready out and displayed directly.

Patch Status

There is a WIP-patch available in this branch (associated changes for hbase-operator changes are here).

[adwk67]

I've asked about this on the dev mailing list: https://lists.apache.org/thread/rx7box0f7b02pg0xnyhlh0sj5b7zp6td

[adwk67]

Moving this into Development:Track with the following comments:

1. the WIP patch works, apart from the parts of the UI that return information from RegionReplicaInfo: it is unclear where the ServerName (which currently is missing the externalHostname and externalPort fields in the patch) is being written for these objects and fixing this will probably require a further change to proto entities (i.e getting ever more invasive)
2. if listener endpoints are used within HBase .jsp code, this will make it difficult to get things to work with a loadbalancer or Ingress, as those will be the URLs that should be displayed in the UI
3. this could be achieved by writing the load balancer endpoint to hbase-site.xml at start-up via the entrypoint script
4. there are various things hard coded in the patch, which the user should be able to "turn off" - doable, but not done
5. most pertinently: if it possible to use Openshift Routes for endpoint resolution (which it is), it may be possible to do a similar thing with non-Openshift clusters

Points 2. and 5. are the ones which make me doubt that this (patching HBase) is the correct/best approach.