Skip to content

Commit 6af2b3f

Browse files
jstirnamanjstirnaman
authored
Merge pull request #5931 from influxdata/jts/iceberg
feat(dedicated): iceberg export for snowflake:
2 parents a013770 + b11788d commit 6af2b3f

File tree

3 files changed

+113
-120
lines changed

3 files changed

+113
-120
lines changed

‎content/influxdb3/cloud-dedicated/process-data/export/_index.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
title: Export data
3+
description: >
4+
Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems
5+
and avoid complex ETL processes.
6+
menu:
7+
influxdb3_cloud_dedicated:
8+
parent: Process & visualize data
9+
weight: 103
10+
---
11+
Use InfluxData export tools to:
12+
13+
- Integrate time series data stored in {{% product-name %}} with other tools and systems
14+
- Avoid complex ETL (Extract, Transform, Load) processes
15+
16+
{{< children >}}

‎content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
title: Export in Iceberg format for Snowflake
3+
seotitle: Export time series data to Iceberg format for Snowflake
4+
description: >
5+
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
6+
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
7+
menu:
8+
influxdb3_cloud_dedicated:
9+
name: Snowflake with Iceberg
10+
parent: Export data
11+
weight: 101
12+
influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export]
13+
source: /shared/influxdb3-query-guides/snapshots/snowflake.md
14+
---
15+
16+
<!-- //FIXME The setup instructions are specific to Dedicated. After the content
17+
is approved, let's move it from content/shared to here -->
18+
<!-- //SOURCE - content/shared/influxdb3-query-guides/snapshots/snowflake.md -->
19+

‎content/shared/influxdb3-query-guides/snapshots/snowflake.md

Lines changed: 78 additions & 120 deletions
Original file line numberDiff line numberDiff line change
@@ -1,185 +1,143 @@
1-
# Snowflake integration
2-
31
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
42
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
53

6-
> **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization.
7-
84
## Key benefits
95

106
- **Efficient data access**: Query your data directly from Snowflake.
117
- **Cost-effective storage**: Optimize data retention and minimize storage costs.
128
- **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake.
139

14-
## Prerequisites
10+
> [!Note]
11+
> #### Contact sales to enable the InfluxDB Iceberg integration
12+
>
13+
> The InfluxDB Iceberg integration is a premium feature.
14+
> [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account.
15+
16+
- [Prerequisite](#prerequisite)
17+
- [Request Iceberg integration](#request-iceberg-integration)
18+
- [Integrate InfluxDB 3 with Snowflake](#integrate-influxdb-3-with-snowflake)
19+
- [Export data to Iceberg format](#export-data-to-iceberg-format)
20+
- [Check snapshot status](#check-snapshot-status)
21+
- [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake)
22+
- [Considerations and limitations](#considerations-and-limitations)
1523

16-
Before you begin, ensure you have the following:
24+
### Prerequisite
1725

18-
- **{{< product-name >}} Cloud Dedicated plan** or compatible environment
26+
Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/cloud-dedicated/get-started/setup/).
1927

20-
## Request Iceberg integration
28+
### Request Iceberg integration
2129

22-
1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration.
30+
1. [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration for your {{% product-name %}} cluster.
2331

24-
2. Be prepared to provide:
32+
2. Be prepared to provide the following information:
2533
- Your organization ID
2634
- Customer name
2735
- Cloud provider and region
2836
- External storage details (bucket name, region, permissions)
2937
- Technical contact information
3038

31-
> **Note**: Iceberg integration is a premium feature that requires setup by the InfluxData team.
32-
33-
## Integrate InfluxDB 3 with Snowflake
34-
35-
Once you've contacted sales and enabled Iceberg integration, your InfluxData support engineers will help you with the following steps:
36-
37-
1. [Create a Snowflake external stage](#create-a-snowflake-external-stage)
38-
2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format)
39-
3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake)
40-
4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake)
41-
42-
### Create a Snowflake external stage
43-
44-
Your support engineer will guide you through setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command. This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata.
45-
46-
#### Example: Configure an S3 stage in Snowflake
47-
48-
```sql
49-
CREATE STAGE my_s3_stage
50-
URL='s3://my-bucket/'
51-
STORAGE_INTEGRATION=my_storage_integration;
52-
```
39+
### Setup Snowflake and configure your cluster
5340

54-
### Set up a catalog integration in Snowflake
41+
The {{% product-name %}} Iceberg integration requires setup by the InfluxData team.
42+
After you have contacted sales and enabled Iceberg integration, you'll specify which
43+
tables you want to make available for exporting.
44+
InfluxData support engineers will help you with the integration steps, which include:
5545

56-
Your support engineer will help you set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently.
46+
- Setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command and your storage provider details.
47+
This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata.
48+
- Setting up a catalog integration in Snowflake to manage and load Iceberg tables efficiently.
49+
For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration).
50+
- Creating an export configuration that tells the InfluxDB Iceberg exporter which tables are available for export.
5751

58-
#### Example: Create a catalog integration in Snowflake
52+
After the setup is complete, you can export data snapshots to Iceberg format and query the Iceberg table from Snowflake.
5953

60-
```sql
61-
CREATE CATALOG INTEGRATION my_catalog_integration
62-
CATALOG_SOURCE = 'OBJECT_STORE'
63-
TABLE_FORMAT = 'ICEBERG'
64-
ENABLED = TRUE;
65-
```
54+
> [!Important]
55+
> #### Export tables are read-only in Snowflake
56+
>
57+
> Tables created through this integration are _read-only_.
58+
> You cannot write directly to these tables using Snowflake or any other engine.
59+
>
6660
67-
For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration).
61+
## Export data to Iceberg format
6862

69-
### Export InfluxDB time series data to Iceberg format
63+
Use the `influxctl` CLI or the HTTP API to export snapshots to Iceberg format.
7064

71-
> **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported.
65+
> [!Important]
66+
> Before you can export data to Iceberg format, the InfluxData Support Team needs
67+
> to set up your tables for exporting.
68+
> For more information, [contact the InfluxData Sales team]({{< cta-link >}}).
7269
7370
{{< tabs-wrapper >}}
7471
{{% tabs %}}
75-
[CLI](#tab-cli)
76-
[API](#tab-api)
72+
[CLI](#cli)
73+
[API](#api)
7774
{{% /tabs %}}
78-
7975
{{% tab-content %}}
80-
{{% tab-pane id="tab-cli" %}}
81-
#### Using the CLI
8276

83-
Use the `influxctl` command to export InfluxDB time-series data to Iceberg format:
84-
85-
```sh
86-
influxctl snapshot export --namespace foo --table bar
77+
{{% code-placeholders "NAMESPACE|TABLE_NAME" %}}
78+
```bash
79+
influxctl snapshot export --namespace NAMESPACE --table TABLE_NAME
8780
```
88-
{{% /tab-pane %}}
89-
{{% tab-pane id="tab-api" %}}
90-
#### Using the API
91-
92-
Use the {{% product-name %}} HTTP API to export snapshots and check status.
81+
{{% /code-placeholders %}}
9382

94-
##### Example: Export a snapshot
83+
Replace the following:
9584

96-
This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API.
85+
- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. <!-- Namespace might need more explanation -->
86+
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export
9787

98-
- **Method**: `POST`
99-
- **Endpoint**: `/snapshots/export`
100-
- **Request body**:
101-
102-
```json
103-
{
104-
"namespace": "foo",
105-
"table": "bar"
106-
}
107-
```
108-
The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported.
109-
110-
##### Example: Check snapshot status
111-
112-
This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API.
113-
114-
- **Method**: `GET`
115-
- **Endpoint**: `/snapshots/status`
116-
117-
The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion.
118-
{{% /tab-pane %}}
11988
{{% /tab-content %}}
120-
{{< /tabs-wrapper >}}
121-
122-
#### Creating a configuration file
89+
{{% tab-content %}}
12390

124-
Before running the export command, you need to create a configuration file that specifies which tables to export:
91+
**Endpoint**:
12592

126-
1. Create a file named `config.json` in your working directory
127-
2. Add the following JSON content, adjusting the namespace and table names to match your data:
93+
{{% api-endpoint method="POST" endpoint="/snapshots/export" %}}
12894

95+
**Request body**:
96+
12997
```json
13098
{
131-
"exports": [
132-
{
133-
"namespace": "company_sensors",
134-
"table_name": "cpu"
135-
}
136-
]
99+
"namespace": "NAMESPACE",
100+
"table": "TABLE_NAME"
137101
}
138102
```
139103

140-
This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array.
104+
The following example shows how to use cURL with the HTTP API:
141105

142-
After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake.
106+
{{% code-placeholders "NAMESPACE|TABLE_NAME" %}}
107+
```bash
108+
curl -X POST https://{{% influxdb/host %}}/snapshots/export \
109+
-d '{"namespace": "NAMESPACE", "table": "TABLE_NAME"}'
110+
```
111+
{{% /code-placeholders %}}
143112

144-
### Create an Iceberg table in Snowflake
113+
Replace the following:
145114

146-
After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step:
115+
- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. <!--Might need more explanation-->
116+
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export
147117

148-
After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step:
118+
{{% /tab-content %}}
119+
{{< /tabs-wrapper >}}
149120

150-
> **Note**: **Tables created through this integration are read-only.**
151-
> **You cannot write directly to these tables using Snowflake or any other engine.**
152-
> They serve as an analytics interface to your InfluxDB data.
121+
## Check snapshot status
153122

154-
1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process.
123+
Use the HTTP API to check the status of an ongoing or completed snapshot export.
155124

156-
2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to:
125+
{{% api-endpoint method="GET" endpoint="/snapshots/status" %}}
157126

158-
```sql
159-
CREATE ICEBERG TABLE my_iceberg_table
160-
EXTERNAL_VOLUME = 'my_external_volume'
161-
METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json';
127+
```bash
128+
curl -X GET https://{{% influxdb/host %}}/snapshots/status
162129
```
163130

164-
3. Your support engineer will help ensure the EXTERNAL_VOLUME and METADATA_FILE_PATH parameters correctly point to your external storage and metadata file.
165-
166-
This creates a table in Snowflake that reads directly from the Iceberg-formatted data exported from your InfluxDB instance.
167-
168-
### Query the Iceberg table from Snowflake
131+
## Query the Iceberg table from Snowflake
169132

170-
Once the Iceberg table is set up, you can query it using standard SQL in Snowflake.
171-
172-
#### Example: Query the Iceberg table
173-
174-
```sql
175-
SELECT * FROM my_iceberg_table
176-
WHERE timestamp > '2025-01-01';
177-
```
133+
Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >}}.
178134

179135
## Considerations and limitations
180136

181137
When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind:
182138

139+
- **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine.
183140
- **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB.
184141
- **Performance**: Query performance may vary based on data size and query complexity.
185142
- **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration.
143+

0 commit comments

Comments
 (0)