You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When migrating from Elastic to ClickStack for observability use cases, we recommend a **parallel operation** approach rather than attempting to migrate historical data. This strategy offers several advantages:
5
5
@@ -11,7 +11,7 @@ When migrating from Elastic to ClickStack for observability use cases, we recomm
11
11
We demonstrate an approach for migrating essential data from Elasticsearch to ClickHouse in the section ["Migrating data"](#migrating-data). This should not be used for larger datasets as it is rarely performant - limited by the ability for Elasticsearch to export efficiently, with only JSON format supported.
12
12
:::
13
13
14
-
### Implementation steps
14
+
### Implementation steps {#implementation-steps}
15
15
16
16
1.**Configure Dual Ingestion**
17
17
<br/>
@@ -35,27 +35,27 @@ Configure Elastic's TTL settings to match your desired retention period. Set up
35
35
- As data naturally expires from Elastic, users will increasingly rely on ClickStack
36
36
- Once confidence in ClickStack is established, you can begin redirecting queries and dashboards
37
37
38
-
### Long-term retention
38
+
### Long-term retention {#long-term-retention}
39
39
40
40
For organizations requiring longer retention periods:
41
41
42
42
- Continue running both systems in parallel until all data has expired from Elastic
43
43
- ClickStack [tiered storage](/engines/table-engines/mergetree-family/mergetree#table_engine-mergetree-multiple-volumes) capabilities can help manage long-term data efficiently.
44
44
- Consider using [materialized views](/materialized-view/incremental-materialized-view) to maintain aggregated or filtered historical data while allowing raw data to expire.
45
45
46
-
### Migration timeline
46
+
### Migration timeline {#migration-timeline}
47
47
48
48
The migration timeline will depend on your data retention requirements:
49
49
50
50
-**30-day retention**: Migration can be completed within a month.
51
51
-**Longer retention**: Continue parallel operation until data expires from Elastic.
52
52
-**Historical data**: If absolutely necessary, consider using [Migrating data](#migrating-data) to import specific historical data.
53
53
54
-
## Migrating settings
54
+
## Migrating settings {#migration-settings}
55
55
56
56
When migrating from Elastic to ClickStack, your indexing and storage settings will need to be adapted to fit ClickHouse's architecture. While Elasticsearch relies on horizontal scaling and sharding for performance and fault tolerance and thus has multiple shards by default, ClickHouse is optimized for vertical scaling and typically performs best with fewer shards.
57
57
58
-
### Recommended settings
58
+
### Recommended settings {#recommended-settings}
59
59
60
60
We recommend starting with a **single shard** and scaling vertically. This configuration is suitable for most observability workloads and simplifies both management and query performance tuning.
61
61
@@ -67,7 +67,7 @@ We recommend starting with a **single shard** and scaling vertically. This confi
67
67
- Using [`ReplicatedMergeTree`](/engines/table-engines/mergetree-family/replication) if high availability is required
68
68
- For fault tolerance, [1 replica of your shard](/engines/table-engines/mergetree-family/replication) is typically sufficient in Observability workloads.
69
69
70
-
### When to shard
70
+
### When to shard {#when-to-shard}
71
71
72
72
Sharding may be necessary if:
73
73
@@ -77,7 +77,7 @@ Sharding may be necessary if:
77
77
78
78
If you do need to shard, refer to [Horizontal scaling](/architecture/horizontal-scaling) for guidance on shard keys and distributed table setup.
79
79
80
-
### Retention and TTL
80
+
### Retention and TTL {#retention-and-ttl}
81
81
82
82
ClickHouse uses [TTL clauses](/use-cases/observability/clickstack/production#configure-ttl) on MergeTree tables to manage data expiration. TTL policies can:
We recommend aligning your ClickHouse TTL configuration with your existing Elastic retention policies to maintain a consistent data lifecycle during the migration. For examples, see [ClickStack production TTL setup](/use-cases/observability/clickstack/production#configure-ttl).
89
89
90
-
## Migrating data
90
+
## Migrating data {#migrating-data}
91
91
92
92
While we recommend parallel operation for most observability data, there are specific cases where direct data migration from Elasticsearch to ClickHouse may be necessary:
93
93
@@ -101,7 +101,7 @@ The following steps allow the migration of a single Elasticsearch index from Cli
101
101
102
102
<VerticalStepperheaderLevel="h3">
103
103
104
-
### Migrate schema
104
+
### Migrate schema {#migrate-scheme}
105
105
106
106
Create a table in ClickHouse for the index being migrated from Elasticsearch. Users can map [Elasticsearch types to their ClickHouse](/use-cases/observability/clickstack/migration/elastic/types) equivalent. Alternatively, users can simply rely on the JSON data type in ClickHouse, which will dynamically create columns of the appropriate type as data is inserted.
107
107
@@ -556,11 +556,11 @@ npm install elasticdump -g
556
556
557
557
Where possible, we recommend running both ClickHouse, Elasticsearch, and `elastic dump` in the same availability zone or data center to minimize network egress and maximize throughput.
Ensure ClickHouse is [installed on the server](/install) on which `elasticdump` is located. **Do not start a ClickHouse server** - these steps only require the client.
562
562
563
-
### Stream data
563
+
### Stream data {#stream-data}
564
564
565
565
To stream data between Elasticsearch and ClickHouse, use the `elasticdump` command - piping the output directly to the ClickHouse client. The following inserts the data into our well structured table `logs_system_syslog`.
See ["Reading JSON as an object"](/integrations/data-formats/json/other-formats#reading-json-as-an-object) for further details.
611
611
:::
612
612
613
-
### Transform data (optional)
613
+
### Transform data (optional) {#transform-data}
614
614
615
615
The above commands assume a 1:1 mapping of Elasticsearch fields to ClickHouse columns. Users often need to filter and transform Elasticsearch data before insertion into ClickHouse.
0 commit comments