Skip to content

chore: support for hbase 2.6.1 #627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 12, 2025
Merged

chore: support for hbase 2.6.1 #627

merged 2 commits into from
Feb 12, 2025

Conversation

dervoeti
Copy link
Member

@dervoeti dervoeti commented Feb 11, 2025

Description

Part of stackabletech/docker-images#972

I had to add the --no-checksum-verify flag when exporting snapshots to/from S3, because our snapshot-export test was failing for HBase 2.6.1.
From what I see, checksum verification was merged and released with 2.6.1: https://issues.apache.org/jira/browse/HBASE-28625
However, it seems to have problems when comparing snapshots on different filesystems (like HDFS and S3).

https://issues.apache.org/jira/browse/HBASE-28702 was also merged into 2.6.1 and seems to introduce a flag to skip checksum-checks between snapshots when exporting tables. I'm not sure if this flag is supposed to work with snapshot exports as well, at least I could not make it work. Setting fs.checksum.combine.mode to COMPOSITE_CRC also does not seem to work.

https://issues.apache.org/jira/browse/HBASE-28998 (still open) also mentions that it does not work with COMPOSITE_CRC and it specificly mentions problems with HDFS in combination with S3.

So I decided to add the --no-checksum-verify flag in our tests when exporting snapshots. From what I understand snapshot checksums where not validated before 2.6.1 either. I don't have too much experience with HBase and thus am not sure if this is really the best solution. Could one of our HBase experts (@razvan @siegfriedweber @lfrancke ?) take a look at this as well? If this solution is fine I would adapt the documentation accordingly.

Definition of Done Checklist

# Author
- [x] Changes are OpenShift compatible
- [x] CRD changes approved
- [x] CRD documentation for all fields, following the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [x] Helm chart can be installed and deployed operator works
- [x] Integration tests passed (for non trivial changes)
- [x] Changes need to be "offline" compatible
# Reviewer
- [ ] Code contains useful comments
- [ ] Code contains useful logging statements
- [ ] (Integration-)Test cases added
- [ ] Documentation added or updated. Follows the [style guide](https://docs.stackable.tech/home/nightly/contributor/docs/style-guide).
- [ ] Changelog updated
- [ ] Cargo.toml only contains references to git tags (not specific commits or branches)
# Acceptance
- [ ] Feature Tracker has been updated
- [ ] Proper release label has been added
- [ ] [Roadmap](https://github.com/orgs/stackabletech/projects/25/views/1) has been updated

@razvan razvan self-requested a review February 12, 2025 08:48
@razvan
Copy link
Member

razvan commented Feb 12, 2025

Indeed, it seems there is no workaround to the checksum problem currently.

The tests are green:

--- PASS: kuttl (4600.24s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/resources_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (210.98s)
        --- PASS: kuttl/harness/shutdown_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (503.10s)
        --- PASS: kuttl/harness/smoke_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-3.4.0_zookeeper-3.9.2_listener-class-external-unstable_openshift-false (420.45s)
        --- PASS: kuttl/harness/smoke_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-3.4.0_zookeeper-3.9.2_listener-class-cluster-internal_openshift-false (299.91s)
        --- PASS: kuttl/harness/cluster-operation_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (259.18s)
        --- PASS: kuttl/harness/omid_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false_omid-1.1.2 (262.05s)
        --- PASS: kuttl/harness/logging_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (229.79s)
        --- PASS: kuttl/harness/overrides_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (213.66s)
        --- PASS: kuttl/harness/kerberos_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_listener-class-cluster-internal_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (448.90s)
        --- PASS: kuttl/harness/orphaned_resources_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (240.87s)
        --- PASS: kuttl/harness/kerberos_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_listener-class-external-unstable_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (481.88s)
        --- PASS: kuttl/harness/profiling_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (226.43s)
        --- PASS: kuttl/harness/snapshot-export_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (288.24s)
        --- PASS: kuttl/harness/opa_hbase-opa-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_opa-1.0.0_openshift-false (514.78s)
PASS

Copy link
Member

@razvan razvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with a small fix

@dervoeti dervoeti added this pull request to the merge queue Feb 12, 2025
Merged via the queue into main with commit 63b9bb8 Feb 12, 2025
17 checks passed
@dervoeti dervoeti deleted the chore/add-hbase-2.6.1 branch February 12, 2025 14:13
@dervoeti dervoeti restored the chore/add-hbase-2.6.1 branch February 14, 2025 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants