Skip to content

feat: HBase Listener integration #639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
05e5c16
output listener refs
adwk67 Mar 26, 2025
776e273
add listener refs to config map
adwk67 Mar 26, 2025
d824b50
Merge branch 'main' into feat/integrate-listener-operator
adwk67 Mar 26, 2025
967239d
changelog
adwk67 Mar 26, 2025
4eb4e35
changed logging statement
adwk67 Mar 26, 2025
96593c7
integration test
adwk67 Mar 27, 2025
ae1654a
changed listener class designation
adwk67 Mar 27, 2025
53ea8bc
update listener class docs
adwk67 Mar 27, 2025
f8cc1c7
working tests
adwk67 Mar 28, 2025
7a2756b
wip: experimental command
adwk67 Mar 31, 2025
2affa65
cleanup start command
adwk67 Mar 31, 2025
652999e
added hostname/port to hbase-env.sh
adwk67 Mar 31, 2025
c8a2644
wip: will move startup logic to hbase-entrypoint.sh
adwk67 Apr 1, 2025
dd793c0
cleaned up/reverted previous wip
adwk67 Apr 1, 2025
bd1db17
cleaned up/reverted previous wip II
adwk67 Apr 1, 2025
80adf71
correct callout
adwk67 Apr 1, 2025
aee4fc2
use pvcs for externally-reachable endpoints
adwk67 Apr 2, 2025
bf969a0
added comment
adwk67 Apr 2, 2025
90efe2b
docs
adwk67 Apr 3, 2025
0030bfb
merge main
adwk67 Apr 3, 2025
8313995
resolve listener-class across role-groups/role
adwk67 Apr 8, 2025
465d0f1
merge conflicts
adwk67 Apr 8, 2025
fb8efde
regenerate nix
adwk67 Apr 8, 2025
5ddbff2
updated tokio and nix packages
adwk67 Apr 9, 2025
6146593
merge main
adwk67 Apr 9, 2025
8acadff
merge main
adwk67 Apr 9, 2025
68ab9cc
added listener-relevant config settings
adwk67 Apr 23, 2025
9a4231a
merged main
adwk67 Apr 24, 2025
61c79a5
Try a different tack at reconfiguring hbase
nightkr May 31, 2025
7b742c0
formatting and partial clean-up
adwk67 Jun 5, 2025
fde3c8a
formatting and partial clean-up
adwk67 Jun 5, 2025
550ea47
merge main and fix conflicts
adwk67 Jun 5, 2025
71015a5
replaced listener-class enum with string
adwk67 Jun 6, 2025
c555448
Merge branch 'main' into feat/integrate-listener-operator
adwk67 Jun 6, 2025
8b3e650
tweak test cases
adwk67 Jun 10, 2025
39740fa
add generic properties for listener endpoints
adwk67 Jun 10, 2025
00ef5c9
write endpoints to hbase-site instead of dedicated config map
adwk67 Jun 10, 2025
74bfc29
remove unused error defs
adwk67 Jun 11, 2025
6c616a1
remove unused error def/function
adwk67 Jun 11, 2025
4309b9d
fix some tests inline with 2.6.1, add UI port type
adwk67 Jun 13, 2025
51b2837
use listener service resolved in entrypoint script rather than passin…
adwk67 Jun 16, 2025
a51003c
Merge branch 'main' into feat/integrate-listener-operator
adwk67 Jun 16, 2025
4ceb588
updated env-vars based on patch changes
adwk67 Jun 25, 2025
4122baf
merge main and fix conflicts
adwk67 Jun 25, 2025
fda8ca3
fix smoke test
adwk67 Jun 26, 2025
2c41938
fixed orphaned-resources test
adwk67 Jun 26, 2025
16e7c45
fixed opa test
adwk67 Jun 26, 2025
88d7458
Update CHANGELOG.md
adwk67 Jun 26, 2025
42148aa
Update docs/modules/hbase/pages/usage-guide/listenerclass.adoc
adwk67 Jun 26, 2025
67d6e54
review feedback
adwk67 Jun 27, 2025
474bdfe
added constant
adwk67 Jun 27, 2025
e8469ab
replace expects with snafu error
adwk67 Jun 27, 2025
1aa786a
correct property usage to match latest image changes
adwk67 Jun 27, 2025
9177c2f
provide bound address to match defaults used in patch
adwk67 Jun 28, 2025
f283715
Update CHANGELOG.md
adwk67 Jul 7, 2025
507d9a8
clarified error
adwk67 Jul 7, 2025
0f396f5
removed unnecessary configs and added trouble-shooting doc page
adwk67 Jul 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

### Added

- BREAKING: Add listener support for HBase ([#639]).
- Adds new telemetry CLI arguments and environment variables ([#652]).
- Use `--file-log-max-files` (or `FILE_LOG_MAX_FILES`) to limit the number of log files kept.
- Use `--file-log-rotation-period` (or `FILE_LOG_ROTATION_PERIOD`) to configure the frequency of rotation.
Expand Down Expand Up @@ -39,6 +40,7 @@
- test: HBase 2.4.18 removed ([#659]):
- Remove operator support for HBase 2.4 including the JMX exporter ([#672]).

[#639]: https://github.com/stackabletech/hbase-operator/pull/639
[#640]: https://github.com/stackabletech/hbase-operator/pull/640
[#645]: https://github.com/stackabletech/hbase-operator/pull/645
[#647]: https://github.com/stackabletech/hbase-operator/pull/647
Expand Down
38 changes: 24 additions & 14 deletions deploy/helm/hbase-operator/crds/crds.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -73,20 +73,6 @@ spec:
hdfsConfigMapName:
description: Name of the [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery) for an HDFS cluster.
type: string
listenerClass:
default: cluster-internal
description: |-
This field controls which type of Service the Operator creates for this HbaseCluster:

* cluster-internal: Use a ClusterIP service

* external-unstable: Use a NodePort service

This is a temporary solution with the goal to keep yaml manifests forward compatible. In the future, this setting will control which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) will be used to expose the service, and ListenerClass names will stay the same, allowing for a non-breaking change.
enum:
- cluster-internal
- external-unstable
type: string
vectorAggregatorConfigMapName:
description: Name of the Vector aggregator [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery). It must contain the key `ADDRESS` with the address of the Vector aggregator. Follow the [logging tutorial](https://docs.stackable.tech/home/nightly/tutorials/logging-vector-aggregator) to learn how to configure log aggregation with Vector.
nullable: true
Expand Down Expand Up @@ -210,6 +196,10 @@ spec:
hbaseRootdir:
nullable: true
type: string
listenerClass:
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose this rolegroup.
nullable: true
type: string
logging:
default:
containers: {}
Expand Down Expand Up @@ -460,6 +450,10 @@ spec:
hbaseRootdir:
nullable: true
type: string
listenerClass:
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose this rolegroup.
nullable: true
type: string
logging:
default:
containers: {}
Expand Down Expand Up @@ -691,6 +685,10 @@ spec:
hbaseRootdir:
nullable: true
type: string
listenerClass:
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose this rolegroup.
nullable: true
type: string
logging:
default:
containers: {}
Expand Down Expand Up @@ -969,6 +967,10 @@ spec:
hbaseRootdir:
nullable: true
type: string
listenerClass:
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose this rolegroup.
nullable: true
type: string
logging:
default:
containers: {}
Expand Down Expand Up @@ -1228,6 +1230,10 @@ spec:
hbaseRootdir:
nullable: true
type: string
listenerClass:
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose this rolegroup.
nullable: true
type: string
logging:
default:
containers: {}
Expand Down Expand Up @@ -1478,6 +1484,10 @@ spec:
hbaseRootdir:
nullable: true
type: string
listenerClass:
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose this rolegroup.
nullable: true
type: string
logging:
default:
containers: {}
Expand Down
6 changes: 6 additions & 0 deletions deploy/helm/hbase-operator/templates/roles.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,12 @@ rules:
verbs:
- create
- patch
- apiGroups:
- listeners.stackable.tech
resources:
- listeners
verbs:
- get
- apiGroups:
- {{ include "operator.name" . }}.stackable.tech
resources:
Expand Down
35 changes: 24 additions & 11 deletions docs/modules/hbase/pages/usage-guide/listenerclass.adoc
Original file line number Diff line number Diff line change
@@ -1,18 +1,31 @@
= Service exposition with ListenerClasses
:description: Configure HBase service exposure using ListenerClasses to control internal and external access for all roles.

Apache HBase offers an API.
The operator deploys a service called `<name>` (where `<name>` is the name of the HbaseCluster) through which HBase can be reached.

This service can have either the `cluster-internal` or `external-unstable` type.
`external-stable` is not supported for HBase at the moment.
Read more about the types in the xref:concepts:service-exposition.adoc[service exposition] documentation at platform level.

This is how the listener class is configured:
The operator deploys a xref:listener-operator:listener.adoc[Listener] for each Master, Regionserver and Restserver pod.
They all default to only being accessible from within the Kubernetes cluster, but this can be changed by setting `.spec.{masters,regionServers,restServers}.config.listenerClass`:

[source,yaml]
----
spec:
clusterConfig:
listenerClass: cluster-internal # <1>
masters:
config:
listenerClass: external-unstable # <1>
regionServers:
config:
listenerClass: external-unstable
restServers:
config:
listenerClass: external-unstable
----
<1> Specify one of `external-stable`, `external-unstable`, `cluster-internal` (the default setting is `cluster-internal`).
This can be set separately for all roles.

Listener endpoints are written to `hbase-site.xml` like this:

[source,xml]
----
<property>
<name>hbase.listener.endpoint</name>
<value>172.19.0.3:32445</value>
</property>
----
<1> The default `cluster-internal` setting.
20 changes: 20 additions & 0 deletions docs/modules/hbase/pages/usage-guide/troubleshooting.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
= Trouble-shooting
:description: Tips and guidelines to help trouble-shoot problems with running HBase.

To add more useful information to stack-traces caused by RPC issues, xref:operators:overrides.adoc[podOverrides] can be used to replace the default Netty implementation:

[source,yaml]
----
masters:
roleGroups:
default:
configOverrides:
hbase-site.xml:
hbase.rpc.client.impl: "org.apache.hadoop.hbase.ipc.BlockingRpcClient"
regionServers:
roleGroups:
default:
configOverrides:
hbase-site.xml:
hbase.rpc.client.impl: "org.apache.hadoop.hbase.ipc.BlockingRpcClient"
----
1 change: 1 addition & 0 deletions docs/modules/hbase/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
** xref:hbase:usage-guide/hbck2.adoc[]
** xref:hbase:usage-guide/snapshot-export.adoc[]
** xref:hbase:usage-guide/adls.adoc[]
** xref:hbase:usage-guide/troubleshooting.adoc[]
** xref:hbase:usage-guide/operations/index.adoc[]
*** xref:hbase:usage-guide/operations/cluster-operations.adoc[]
*** xref:hbase:usage-guide/operations/pod-placement.adoc[]
Expand Down
Loading