feat: HBase Listener integration #639
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can you please run |
…g through as as argument
adwk67
moved this from Development: In Progress
to Development: Waiting for Review
in Stackable Engineering
6 tasks
maltesander
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
maltesander
requested changes
Jun 26, 2025
Co-authored-by: Malte Sander <malte.sander.it@gmail.com>
Co-authored-by: Malte Sander <malte.sander.it@gmail.com>
|
🟢 Re-run local tests Tests |
maltesander
reviewed
Jun 27, 2025
There was a problem hiding this comment.
LGTM! Will wait with approval for on stackabletech/docker-images#1159
|
🟢 Re-run local tests (changed to use the 2.6.1 image) Tests |
nightkr
reviewed
Jul 1, 2025
| template: pod_template, | ||
| volume_claim_templates: Some(vec![pvc]), |
There was a problem hiding this comment.
Sorry, missed this until now!
Master and regionservers should probably use ephemeral listener volumes instead (Pod.spec.volumes[].ephemeral.volumeClaimTemplate), since clients don't actually care about address persistence (they just pull the latest address from ZooKeeper!).
| @@ -929,6 +981,15 @@ fn build_rolegroup_statefulset( | |||
| .service_account_name(service_account.name_any()) | |||
| .security_context(PodSecurityContextBuilder::new().fs_group(1000).build()); | |||
|
|
|||
| // externally-reachable listener endpoints should use a pvc volume... | |||
| let pvc = ListenerOperatorVolumeSourceBuilder::new( | |||
There was a problem hiding this comment.
Suggested change
| let pvc = ListenerOperatorVolumeSourceBuilder::new( | |
| let listener_pvc = ListenerOperatorVolumeSourceBuilder::new( |
NickLarsenNZ
reviewed
Jul 4, 2025
| @@ -4,6 +4,7 @@ | |||
|
|
|||
| ### Added | |||
|
|
|||
| - BREAKING: Adds listener support for HBase ([#639]). | |||
There was a problem hiding this comment.
Suggested change
| - BREAKING: Adds listener support for HBase ([#639]). | |
| - BREAKING: Add listener support for HBase ([#639]). |
NickLarsenNZ
reviewed
Jul 4, 2025
| @@ -696,6 +676,7 @@ impl HbaseRole { | |||
| affinity: get_affinity(cluster_name, self, hdfs_discovery_cm_name), | |||
| graceful_shutdown_timeout: Some(graceful_shutdown_timeout), | |||
| requested_secret_lifetime: Some(requested_secret_lifetime), | |||
| listener_class: Some(DEFAULT_LISTENER_CLASS.to_string()), | |||
There was a problem hiding this comment.
Should these go under spec.{role}.roleConfig, or do they need to be under spec.{role}.config?
There was a problem hiding this comment.
HBase uses listener-per-pod, so rolegroup should be fine.
nightkr
reviewed
Jul 4, 2025
| @@ -102,6 +104,12 @@ pub enum Error { | |||
|
|
|||
| #[snafu(display("incompatible merge types"))] | |||
| IncompatibleMergeTypes, | |||
|
|
|||
| #[snafu(display("role-group is not valid"))] | |||
There was a problem hiding this comment.
This error message says nothing; why is it not valid?
Comment on lines
+211
to
+214
| // Trivial values for role-groups are not allowed | ||
| if role_group.is_empty() { | ||
| return Err(Error::NoRoleGroup); | ||
| } |
There was a problem hiding this comment.
Validating this feels like a broader question across the SDP, not something to sneak in here.
Comment on lines
+547
to
+550
| hbase_site_config.insert( | ||
| "hbase.rpc.client.impl".to_string(), | ||
| "org.apache.hadoop.hbase.ipc.BlockingRpcClient".to_string(), | ||
| ); |
There was a problem hiding this comment.
We shouldn't do this by default; maybe put it as an option for troubleshooting in the docs?
Comment on lines
+552
to
+556
| // Set listener endpoint information | ||
| hbase_site_config.insert( | ||
| "hbase.listener.endpoint".to_string(), | ||
| "${HBASE_LISTENER_ENDPOINT}".to_string(), | ||
| ); |
There was a problem hiding this comment.
Don't think we use this anymore on the hbase side?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
fixes: #618
Warning
Merge this at the same time as stackabletech/docker-images#1159, otherwise tests will break!
How to test
bake -p hbasefrom the branch for this PRkind load docker-image oci.stackable.tech/sdp/hbase:2.6.1-stackable0.0.0-dev --name stackable-data-platformstackable op in hbase=0.0.0-pr639./scripts/run-tests --skip-operator hbase --test ...Implementation notes
Output from the HBase demo:
The Localities indicate that the serving region is the same node as the HDFS data:
CRD change
Note
There does not need to be a decision on this (discussed briefly in planning 09.04.2024) as the implementation is in line with existing implementations for HDFs and Kafka.
old:
new:
(this is inline with the implementations for HDFS and Kafka)
Tests (OpenShift)
Definition of Done Checklist
Author
Reviewer
Acceptance