Skip to content

Commit efd862f

Browse files
docs: Describe how to use ADLS instead of HDFS (#560)
1 parent 7b490a8 commit efd862f

File tree

2 files changed

+93
-0
lines changed

2 files changed

+93
-0
lines changed
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
= Using the Azure Data Lake Storage (ADLS)
2+
:description: Use the Azure Data Lake Storage (ADLS) instead of HDFS for HBase.
3+
:abfs-authentication: https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html#Authentication
4+
:hadoop-18516: https://issues.apache.org/jira/browse/HADOOP-18516
5+
6+
Instead of HDFS, the HBase data can be stored in the Azure Data Lake Storage (ADLS).
7+
HDFS is then not required anymore to run HBase.
8+
9+
A custom ConfigMap for ADLS containing the core-site.xml and an empty hdfs-site.xml must be provided:
10+
11+
[source,yaml]
12+
----
13+
---
14+
apiVersion: v1
15+
kind: ConfigMap
16+
metadata:
17+
name: adls
18+
data:
19+
core-site.xml: |-
20+
<?xml version="1.0"?>
21+
<configuration>
22+
<property>
23+
<name>fs.defaultFS</name> <!--1-->
24+
<value>abfs://<container-name>@<storage-account>.dfs.core.windows.net/</value>
25+
</property>
26+
<property>
27+
<name>fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net</name> <!--2-->
28+
<value>SAS</value>
29+
</property>
30+
<property>
31+
<name>fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net</name> <!--3-->
32+
<value>${env.SAS_TOKEN}</value>
33+
</property>
34+
<!-- Add further properties, e.g. for Kerberos. -->
35+
</configuration>
36+
hdfs-site.xml: |-
37+
<?xml version="1.0"?>
38+
<configuration>
39+
</configuration>
40+
----
41+
<1> The name of the default file system.
42+
The `hadoop-azure` module which provides support for ADLS through the "abfs" connector, is already contained in the Stackable HBase image and accessible on the classpath.
43+
Make sure that the hierarchical namespace is enabled in the storage account.
44+
Replace `<container-name>` and `<storage-account>` accordingly.
45+
<2> The authentication mechanism.
46+
Possible values are `SharedKey`, `OAuth`, `Custom` and `SAS`, see the {abfs-authentication}[documentation] of the `hadoop-azure` module for further information.
47+
<3> If the authentication type is set to `SAS` (shared access signature) and no SAS token provider is given in `fs.azure.sas.token.provider.type`, then a fixed SAS token can be used instead.
48+
The `FixedSASTokenProvider` (see {hadoop-18516}[HADOOP-18516]) is back-ported to all supported HBase versions, so that the SAS token can be configured with this property.
49+
50+
The ConfigMap must be referenced in `hdfsConfigMapName`.
51+
In the example above, the SAS token is read from an environment variable which is taken from a Secret:
52+
53+
[source,yaml]
54+
----
55+
---
56+
apiVersion: hbase.stackable.tech/v1alpha1
57+
kind: HbaseCluster
58+
spec:
59+
clusterConfig:
60+
hdfsConfigMapName: adls
61+
masters:
62+
podOverrides:
63+
spec:
64+
containers:
65+
- name: hbase
66+
env:
67+
- name: SAS_TOKEN
68+
valueFrom:
69+
secretKeyRef:
70+
name: adls-credentials
71+
key: sas-token
72+
regionServers:
73+
podOverrides:
74+
spec:
75+
containers:
76+
- name: hbase
77+
env:
78+
- name: SAS_TOKEN
79+
valueFrom:
80+
secretKeyRef:
81+
name: adls-credentials
82+
key: sas-token
83+
---
84+
apiVersion: v1
85+
kind: Secret
86+
metadata:
87+
name: adls-credentials
88+
type: Opaque
89+
data:
90+
# decoded sas-token: sp=racwdlmeop&st=2024-01-01T00:00:00Z&se=2025-01-01T00:00:00Z&spr=https&sv=2022-11-02&sr=c&sig=xxx
91+
sas-token: c3A9cmFjd2RsbWVvcCZzdD0yMDI0LTAxLTAxVDAwOjAwOjAwWiZzZT0yMDI1LTAxLTAxVDAwOjAwOjAwWiZzcHI9aHR0cHMmc3Y9MjAyMi0xMS0wMiZzcj1jJnNpZz14eHgK
92+
----

docs/modules/hbase/partials/nav.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
** xref:hbase:usage-guide/overrides.adoc[]
1313
** xref:hbase:usage-guide/hbck2.adoc[]
1414
** xref:hbase:usage-guide/snapshot-export.adoc[]
15+
** xref:hbase:usage-guide/adls.adoc[]
1516
** xref:hbase:usage-guide/operations/index.adoc[]
1617
*** xref:hbase:usage-guide/operations/cluster-operations.adoc[]
1718
*** xref:hbase:usage-guide/operations/pod-placement.adoc[]

0 commit comments

Comments
 (0)