Skip to content

Commit e4e7faa

Browse files
docs: Document why we need storage connections (#575)
* docs: Document why we need storage connections * Apply suggestions from code review Co-authored-by: Malte Sander <contact@maltesander.com> --------- Co-authored-by: Malte Sander <contact@maltesander.com>
1 parent 21a1aae commit e4e7faa

File tree

1 file changed

+13
-6
lines changed

1 file changed

+13
-6
lines changed

docs/modules/hive/pages/usage-guide/data-storage.adoc

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,18 @@
11
= Data storage backends
22
:description: Hive supports metadata storage on S3 and HDFS. Configure S3 with S3Connection and HDFS with configMap in clusterConfig.
33

4-
Hive does not store data, only metadata. It can store metadata about data stored in various places. The Stackable Operator currently supports S3 and HFS.
4+
You can operate the Hive metastore service (HMS) without S3 or HDFS.
5+
Its whole purpose is to store metadata such as "Table foo has columns a, b and c and is stored as parquet in local://tmp/hive/foo".
56

6-
== [[s3]]S3 support
7+
However, as soon as you start storing metadata in the HMS that refers to a `s3a://` or `hdfs://` locations, HMS will actually do some operations on the filesystem. This can be e.g. checking if the table location exists, creating it in case it is missing.
78

8-
Hive supports creating tables in S3 compatible object stores.
9+
So if you are storing tables in S3 (or HDFS for that matter), you need to give the HMS access to that filesystem as well.
10+
The Stackable Operator currently supports S3 and HFS.
11+
12+
[s3]
13+
== S3 support
14+
15+
HMS supports creating tables in S3 compatible object stores.
916
To use this feature you need to provide connection details for the object store using the xref:concepts:s3.adoc[S3Connection] in the top level `clusterConfig`.
1017

1118
An example usage can look like this:
@@ -22,10 +29,10 @@ clusterConfig:
2229
secretClass: simple-hive-s3-secret-class
2330
----
2431

32+
[hdfs]
33+
== Apache HDFS support
2534

26-
== [[hdfs]]Apache HDFS support
27-
28-
As well as S3, Hive also supports creating tables in HDFS.
35+
As well as S3, HMS also supports creating tables in HDFS.
2936
You can add the HDFS connection in the top level `clusterConfig` as follows:
3037

3138
[source,yaml]

0 commit comments

Comments
 (0)