Skip to content

elastic migration docs #3928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jun 13, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
106c3ee
elastic migration docs
gingerwizard Jun 11, 2025
3362f54
Merge branch 'main' into elastic_migration
gingerwizard Jun 11, 2025
463249f
spellings
gingerwizard Jun 11, 2025
a42f75d
spellings
gingerwizard Jun 11, 2025
38578de
fixes
gingerwizard Jun 11, 2025
8d8ee82
fix link
gingerwizard Jun 11, 2025
dbcddde
more styles
gingerwizard Jun 11, 2025
e134509
vale issues
gingerwizard Jun 11, 2025
ef4f88c
links
gingerwizard Jun 11, 2025
0baad97
fix spelling
gingerwizard Jun 11, 2025
145fb71
fix vale suggestions
Blargian Jun 11, 2025
8e7e6f2
Update docs/use-cases/observability/clickstack/migration/elastic/migr…
gingerwizard Jun 12, 2025
822abc5
Apply suggestions from code review
gingerwizard Jun 12, 2025
11f18b0
Apply suggestions from code review
gingerwizard Jun 12, 2025
22ce0ad
Apply suggestions from code review
gingerwizard Jun 12, 2025
c541f50
Apply suggestions from code review
gingerwizard Jun 12, 2025
07dd4c1
Apply suggestions from code review
gingerwizard Jun 12, 2025
d27a80a
Apply suggestions from code review
gingerwizard Jun 12, 2025
bb9559d
Update docs/use-cases/observability/clickstack/migration/elastic/conc…
gingerwizard Jun 12, 2025
db6cd88
Update docs/use-cases/observability/clickstack/migration/elastic/conc…
gingerwizard Jun 12, 2025
f883f4d
Apply suggestions from code review
gingerwizard Jun 12, 2025
cf8df28
Merge branch 'main' of https://github.com/clickhouse/clickhouse-docs …
gingerwizard Jun 12, 2025
8a910bb
fix feedback
gingerwizard Jun 12, 2025
5cbdae2
add explicit headers
Blargian Jun 12, 2025
2e50534
Merge branch 'main' into elastic_migration
gingerwizard Jun 13, 2025
a33769c
revert_8a910bb4dc86fe5b1eda2f8f6febd3a3268eecc6
gingerwizard Jun 13, 2025
4bec88a
fix header
gingerwizard Jun 13, 2025
9d7fe35
cleanup for merge
gingerwizard Jun 13, 2025
31f2f81
add frontmatter
gingerwizard Jun 13, 2025
15ce8e0
fix link
gingerwizard Jun 13, 2025
0f747cc
fix spell
gingerwizard Jun 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/best-practices/minimize_optimize_joins.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

## When JOINs are required {#when-joins-are-required}

When JOINs are required, ensure youre using **at least version 24.12 and preferably the latest version**, as JOIN performance continues to improve with each new release. As of ClickHouse 24.12, the query planner now automatically places the smaller table on the right side of the join for optimal performance - a task that previously had to be done manually. Even more enhancements are coming soon, including more aggressive filter pushdown and automatic re-ordering of multiple joins.
When JOINs are required, ensure you're using **at least version 24.12 and preferably the latest version**, as JOIN performance continues to improve with each new release. As of ClickHouse 24.12, the query planner now automatically places the smaller table on the right side of the join for optimal performance - a task that previously had to be done manually. Even more enhancements are coming soon, including more aggressive filter pushdown and automatic re-ordering of multiple joins.

Check notice on line 29 in docs/best-practices/minimize_optimize_joins.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.SentenceLength

Suggestion: Improve readability by using fewer than 25 words in this sentence.

Check notice on line 29 in docs/best-practices/minimize_optimize_joins.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Uppercase

Suggestion: Instead of uppercase for 'JOIN', use lowercase or backticks (`) if possible. Otherwise, ask a Technical Writer to add this word or acronym to the rule's exception list.

Check notice on line 29 in docs/best-practices/minimize_optimize_joins.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.SentenceLength

Suggestion: Improve readability by using fewer than 25 words in this sentence.

Follow these best practices to improve JOIN performance:

Expand Down
4 changes: 2 additions & 2 deletions docs/cloud/changelogs/changelog-25_1-25_4.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@
* Don't fail silently if user executing `SYSTEM DROP REPLICA` doesn't have enough permissions. [#75377](https://github.com/ClickHouse/ClickHouse/pull/75377) ([Bharat Nallan](https://github.com/bharatnc)).
* Add a ProfileEvent about the number of times any of system logs has failed to flush. [#75466](https://github.com/ClickHouse/ClickHouse/pull/75466) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
* Add check and logging for decrypting and decompressing. [#75471](https://github.com/ClickHouse/ClickHouse/pull/75471) ([Vitaly Baranov](https://github.com/vitlibar)).
* Added support for the micro sign (U+00B5) in the `parseTimeDelta` function. Now both the micro sign (U+00B5) and the Greek letter mu (U+03BC) are recognized as valid representations for microseconds, aligning ClickHouse's behavior with Gos implementation ([see time.go](https://github.com/golang/go/blob/ad7b46ee4ac1cee5095d64b01e8cf7fcda8bee5e/src/time/time.go#L983C19-L983C20) and [time/format.go](https://github.com/golang/go/blob/ad7b46ee4ac1cee5095d64b01e8cf7fcda8bee5e/src/time/format.go#L1608-L1609)). [#75472](https://github.com/ClickHouse/ClickHouse/pull/75472) ([Vitaly Orlov](https://github.com/orloffv)).
* Added support for the micro sign (U+00B5) in the `parseTimeDelta` function. Now both the micro sign (U+00B5) and the Greek letter mu (U+03BC) are recognized as valid representations for microseconds, aligning ClickHouse's behavior with Go's implementation ([see time.go](https://github.com/golang/go/blob/ad7b46ee4ac1cee5095d64b01e8cf7fcda8bee5e/src/time/time.go#L983C19-L983C20) and [time/format.go](https://github.com/golang/go/blob/ad7b46ee4ac1cee5095d64b01e8cf7fcda8bee5e/src/time/format.go#L1608-L1609)). [#75472](https://github.com/ClickHouse/ClickHouse/pull/75472) ([Vitaly Orlov](https://github.com/orloffv)).

Check warning on line 277 in docs/cloud/changelogs/changelog-25_1-25_4.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Units

Add a space between the number and the unit in '03B'.

Check warning on line 277 in docs/cloud/changelogs/changelog-25_1-25_4.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Units

Add a space between the number and the unit in '00B'.

Check warning on line 277 in docs/cloud/changelogs/changelog-25_1-25_4.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.Units

Add a space between the number and the unit in '00B'.
* Replace server setting (`send_settings_to_client`) with client setting (`apply_settings_from_server`) that controls whether client-side code (e.g. parsing INSERT data and formatting query output) should use settings from server's `users.xml` and user profile. Otherwise only settings from client command line, session, and the query are used. Note that this only applies to native client (not e.g. HTTP), and doesn't apply to most of query processing (which happens on the server). [#75478](https://github.com/ClickHouse/ClickHouse/pull/75478) ([Michael Kolupaev](https://github.com/al13n321)).
* Keeper improvement: disable digest calculation when committing to in-memory storage for better performance. It can be enabled with `keeper_server.digest_enabled_on_commit` config. Digest is still calculated when preprocessing requests. [#75490](https://github.com/ClickHouse/ClickHouse/pull/75490) ([Antonio Andelic](https://github.com/antonio2368)).
* Push down filter expression from JOIN ON when possible. [#75536](https://github.com/ClickHouse/ClickHouse/pull/75536) ([Vladimir Cherkasov](https://github.com/vdimir)).
Expand Down Expand Up @@ -621,7 +621,7 @@
* The universal installation script will propose installation even on macOS. [#74339](https://github.com/ClickHouse/ClickHouse/pull/74339) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
* Fix build when kerberos is not enabled. [#74771](https://github.com/ClickHouse/ClickHouse/pull/74771) ([flynn](https://github.com/ucasfl)).
* Update to embedded LLVM 19. [#75148](https://github.com/ClickHouse/ClickHouse/pull/75148) ([Konstantin Bogdanov](https://github.com/thevar1able)).
* *Potentially breaking*: Improvement to set even more restrictive defaults. The current defaults are already secure. The user has to specify an option to publish ports explicitly. But when the `default` user doesnt have a password set by `CLICKHOUSE_PASSWORD` and/or a username changed by `CLICKHOUSE_USER` environment variables, it should be available only from the local system as an additional level of protection. [#75259](https://github.com/ClickHouse/ClickHouse/pull/75259) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
* *Potentially breaking*: Improvement to set even more restrictive defaults. The current defaults are already secure. The user has to specify an option to publish ports explicitly. But when the `default` user doesn't have a password set by `CLICKHOUSE_PASSWORD` and/or a username changed by `CLICKHOUSE_USER` environment variables, it should be available only from the local system as an additional level of protection. [#75259](https://github.com/ClickHouse/ClickHouse/pull/75259) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
* Integration tests have a 1-hour timeout for single batch of parallel tests running. When this timeout is reached `pytest` is killed without some logs. Internal pytest timeout is set to 55 minutes to print results from a session and not trigger external timeout signal. Closes [#75532](https://github.com/ClickHouse/ClickHouse/issues/75532). [#75533](https://github.com/ClickHouse/ClickHouse/pull/75533) ([Ilya Yatsishin](https://github.com/qoega)).
* Make all clickhouse-server related actions a function, and execute them only when launching the default binary in `entrypoint.sh`. A long-postponed improvement was suggested in [#50724](https://github.com/ClickHouse/ClickHouse/issues/50724). Added switch `--users` to `clickhouse-extract-from-config` to get values from the `users.xml`. [#75643](https://github.com/ClickHouse/ClickHouse/pull/75643) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
* For stress tests if server did not exit while we collected stacktraces via gdb additional wait time is added to make `Possible deadlock on shutdown (see gdb.log)` detection less noisy. It will only add delay for cases when test did not finish successfully. [#75668](https://github.com/ClickHouse/ClickHouse/pull/75668) ([Ilya Yatsishin](https://github.com/qoega)).
Expand Down
8 changes: 4 additions & 4 deletions docs/cloud/manage/billing.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,7 +417,7 @@
ClickPipes. Compute is shared across all Postgres pipes within a service. **It
is provisioned when the first Postgres pipe is created and deallocated when no
Postgres CDC pipes remain**. The amount of compute provisioned depends on your
organizations tier:
organization's tier:

| Tier | Cost |
|------------------------------|-----------------------------------------------|
Expand All @@ -426,7 +426,7 @@

#### Example {#example}

Lets say your service is in Scale tier and has the following setup:
Let's say your service is in Scale tier and has the following setup:

- 2 Postgres ClickPipes running continuous replication
- Each pipe ingests 500 GB of data changes (CDC) per month
Expand Down Expand Up @@ -540,7 +540,7 @@
**September 1st, 2025**, for all customers—both existing and new. Until then,
usage is free. Customers have a **3-month window** starting from **May 29**
(the GA announcement date) to review and optimize their usage if needed, although
we expect most wont need to make any changes.
we expect most won't need to make any changes.

Check warning on line 543 in docs/cloud/manage/billing.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'won't need', use present tense.

</details>

Expand All @@ -550,7 +550,7 @@

No data ingestion charges apply while a pipe is paused, since no data is moved.
However, compute charges still apply—either 0.5 or 1 compute unit—based on your
organizations tier. This is a fixed service-level cost and applies across all
organization's tier. This is a fixed service-level cost and applies across all

Check warning on line 553 in docs/cloud/manage/billing.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.EOLWhitespace

Remove whitespace characters from the end of the line.
pipes within that service.

</details>
Expand Down
4 changes: 2 additions & 2 deletions docs/cloud/reference/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@

## May 30, 2025 {#may-30-2025}

- Were excited to announce general availability of **ClickPipes for Postgres CDC**
- We're excited to announce general availability of **ClickPipes for Postgres CDC**
in ClickHouse Cloud. With just a few clicks, you can now replicate your Postgres
databases and unlock blazing-fast, real-time analytics. The connector delivers
faster data synchronization, latency as low as a few seconds, automatic schema changes,
Expand Down Expand Up @@ -64,7 +64,7 @@
* Memory & CPU: Graphs for `CGroupMemoryTotal` (Allocated Memory), `CGroupMaxCPU` (allocated CPU),
`MemoryResident` (memory used), and `ProfileEvent_OSCPUVirtualTimeMicroseconds` (CPU used)
* Data Transfer: Graphs showing data ingress and egress from ClickHouse Cloud. Learn more [here](/cloud/manage/network-data-transfer).
- Were excited to announce the launch of our new ClickHouse Cloud Prometheus/Grafana mix-in,
- We're excited to announce the launch of our new ClickHouse Cloud Prometheus/Grafana mix-in,

Check warning on line 67 in docs/cloud/reference/changelog.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.EOLWhitespace

Remove whitespace characters from the end of the line.
built to simplify monitoring for your ClickHouse Cloud services.
This mix-in uses our Prometheus-compatible API endpoint to seamlessly integrate
ClickHouse metrics into your existing Prometheus and Grafana setup. It includes
Expand Down
2 changes: 1 addition & 1 deletion docs/data-modeling/projections.md
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,7 @@ paid prices is streaming 2.17 million rows. When we directly used a second table
optimized for this query, only 81.92 thousand rows were streamed from disk.

The reason for the difference is that currently, the `optimize_read_in_order`
optimization mentioned above isnt supported for projections.
optimization mentioned above isn't supported for projections.

We inspect the `system.query_log` table to see that ClickHouse
automatically used the two projections for the two queries above (see the
Expand Down
4 changes: 2 additions & 2 deletions docs/guides/best-practices/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@
|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Query Optimization Guide](/optimize/query-optimization) | A good place to start for query optimization, this simple guide describes common scenarios of how to use different performance and optimization techniques to improve query performance. |
| [Primary Indexes Advanced Guide](/guides/best-practices/sparse-primary-indexes) | A deep dive into ClickHouse indexing including how it differs from other DB systems, how ClickHouse builds and uses a table's spare primary index and what some of the best practices are for indexing in ClickHouse. |
| [Query Parallelism](/optimize/query-parallelism) | Explains how ClickHouse parallelizes query execution using processing lanes and the max_threads setting. Covers how data is distributed across lanes, how max_threads is applied, when it isnt fully used, and how to inspect execution with tools like EXPLAIN and trace logs. |
| [Query Parallelism](/optimize/query-parallelism) | Explains how ClickHouse parallelizes query execution using processing lanes and the max_threads setting. Covers how data is distributed across lanes, how max_threads is applied, when it isn't fully used, and how to inspect execution with tools like EXPLAIN and trace logs. |
| [Partitioning Key](/optimize/partitioning-key) | Delves into ClickHouse partition key optimization. Explains how choosing the right partition key can significantly improve query performance by allowing ClickHouse to quickly locate relevant data segments. Covers best practices for selecting efficient partition keys and potential pitfalls to avoid. |
| [Data Skipping Indexes](/optimize/skipping-indexes) | Explains data skipping indexes as a way to optimize performance. |
| [PREWHERE Optimization](/optimize/prewhere) | Explains how PREWHERE reduces I/O by avoiding reading unnecessary column data. Shows how its applied automatically, how the filtering order is chosen, and how to monitor it using EXPLAIN and logs. |
| [PREWHERE Optimization](/optimize/prewhere) | Explains how PREWHERE reduces I/O by avoiding reading unnecessary column data. Shows how it's applied automatically, how the filtering order is chosen, and how to monitor it using EXPLAIN and logs. |

Check warning on line 21 in docs/guides/best-practices/index.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.OxfordComma

Use a comma before the last 'and' or 'or' in a list of four or more items.
| [Bulk Inserts](/optimize/bulk-inserts) | Explains the benefits of using bulk inserts in ClickHouse. |
| [Asynchronous Inserts](/optimize/asynchronous-inserts) | Focuses on ClickHouse's asynchronous inserts feature. It likely explains how asynchronous inserts work (batching data on the server for efficient insertion) and their benefits (improved performance by offloading insert processing). It might also cover enabling asynchronous inserts and considerations for using them effectively in your ClickHouse environment. |
| [Avoid Mutations](/optimize/avoid-mutations) | Discusses the importance of avoiding mutations (updates and deletes) in ClickHouse. It recommends using append-only inserts for optimal performance and suggests alternative approaches for handling data changes. |
Expand Down
Loading