From 96b9be05178fd4823f59031811d99dec8c7bf505 Mon Sep 17 00:00:00 2001 From: Tom Schreiber Date: Thu, 10 Apr 2025 12:21:07 +0100 Subject: [PATCH 1/2] Made some of the queries runnable. --- .../best-practices/query-parallelism.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/guides/best-practices/query-parallelism.md b/docs/guides/best-practices/query-parallelism.md index a3fc65cb28e..b6f1d3044e3 100644 --- a/docs/guides/best-practices/query-parallelism.md +++ b/docs/guides/best-practices/query-parallelism.md @@ -83,14 +83,11 @@ SETTINGS send_logs_level='trace'; ``` We can see that - - - * ① ClickHouse needs to read 3,609 granules (indicated as marks in the trace logs) across 3 data ranges. * ② With 59 CPU cores, it distributes this work across 59 parallel processing streams—one per lane. Alternatively, we can use the [EXPLAIN](/sql-reference/statements/explain#explain-pipeline) clause to inspect the [physical operator plan](/academic_overview#4-2-multi-core-parallelization)—also known as the "query pipeline"—for the aggregation query: -```sql runnable=false +```sql runnable=true EXPLAIN PIPELINE SELECT max(price) @@ -99,6 +96,7 @@ FROM ``` ```txt +Static result for the query above from April 2025 ┌─explain───────────────────────────────────────────────────────────────────────────┐ 1. │ (Expression) │ 2. │ ExpressionTransform × 59 │ @@ -130,18 +128,19 @@ Note that the `Resize` operators in the physical plan above [repartition and red ## Why max_threads isn't always respected {#why-max-threads-isnt-always-respected} As mentioned above, the number of `n` parallel processing lanes is controlled by the `max_threads` setting, which by default matches the number of CPU cores available to ClickHouse on the server: -```sql runnable=false +```sql runnable=true SELECT getSetting('max_threads'); ``` ```txt +Static result for the query above from April 2025 ┌─getSetting('max_threads')─┐ 1. │ 59 │ └───────────────────────────┘ ``` -However, the `max_threads` value may be ignored depending on the amount of data selected for processing: -```sql runnable=false +However, the `max_threads` value may be ignored for some plan operators depending on the amount of data selected for processing: +```sql runnable=true EXPLAIN PIPELINE SELECT max(price) @@ -151,6 +150,7 @@ WHERE town = 'LONDON'; ``` ```txt +Static result for the query above from April 2025 ... (ReadFromMergeTree) MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread) × 30 @@ -159,7 +159,7 @@ MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread) × 30 As shown in the operator plan extract above, even though `max_threads` is set to `59`, ClickHouse uses only **30** concurrent streams to scan the data. Now let’s run the query: -```sql runnable=false +```sql runnable=true SELECT max(price) FROM @@ -168,6 +168,7 @@ WHERE town = 'LONDON'; ``` ```txt +Static result for the query above from April 2025 ┌─max(price)─┐ 1. │ 594300000 │ -- 594.30 million └────────────┘ @@ -178,7 +179,7 @@ Peak memory usage: 27.24 MiB. As shown in the output above, the query processed 2.31 million rows and read 13.66MB of data. This is because, during the index analysis phase, ClickHouse selected **282 granules** for processing, each containing 8,192 rows, totaling approximately 2.31 million rows: -```sql runnable=false +```sql runnable=true EXPLAIN indexes = 1 SELECT max(price) @@ -188,6 +189,7 @@ WHERE town = 'LONDON'; ``` ```txt +Static result for the query above from April 2025 ┌─explain───────────────────────────────────────────────┐ 1. │ Expression ((Project names + Projection)) │ 2. │ Aggregating │ @@ -204,7 +206,7 @@ WHERE town = 'LONDON'; └───────────────────────────────────────────────────────┘ ``` -Regardless of the configured `max_threads` value, ClickHouse only allocates additional parallel processing lanes when there’s enough data to justify them. The "max" in `max_threads` refers to an upper limit, not a guaranteed number of threads used. +Regardless of the configured `max_threads` value, ClickHouse only allocates additional parallel processing lanes for operator plans scanning the data when there’s enough data to justify them. The "max" in `max_threads` refers to an upper limit, not a guaranteed number of threads used. What "enough data" means is primarily determined by two settings, which define the minimum number of rows (163,840 by default) and the minimum number of bytes (2,097,152 by default) that each processing lane should handle: From 0e891165768081d8069140d66bcd0c872e1935ec Mon Sep 17 00:00:00 2001 From: Shaun Struwig <41984034+Blargian@users.noreply.github.com> Date: Thu, 10 Apr 2025 13:35:28 +0200 Subject: [PATCH 2/2] Update query-parallelism.md --- docs/guides/best-practices/query-parallelism.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/guides/best-practices/query-parallelism.md b/docs/guides/best-practices/query-parallelism.md index b6f1d3044e3..7fe102394a0 100644 --- a/docs/guides/best-practices/query-parallelism.md +++ b/docs/guides/best-practices/query-parallelism.md @@ -87,7 +87,7 @@ We can see that * ② With 59 CPU cores, it distributes this work across 59 parallel processing streams—one per lane. Alternatively, we can use the [EXPLAIN](/sql-reference/statements/explain#explain-pipeline) clause to inspect the [physical operator plan](/academic_overview#4-2-multi-core-parallelization)—also known as the "query pipeline"—for the aggregation query: -```sql runnable=true +```sql runnable EXPLAIN PIPELINE SELECT max(price) @@ -128,7 +128,7 @@ Note that the `Resize` operators in the physical plan above [repartition and red ## Why max_threads isn't always respected {#why-max-threads-isnt-always-respected} As mentioned above, the number of `n` parallel processing lanes is controlled by the `max_threads` setting, which by default matches the number of CPU cores available to ClickHouse on the server: -```sql runnable=true +```sql runnable SELECT getSetting('max_threads'); ``` @@ -140,7 +140,7 @@ Static result for the query above from April 2025 ``` However, the `max_threads` value may be ignored for some plan operators depending on the amount of data selected for processing: -```sql runnable=true +```sql runnable EXPLAIN PIPELINE SELECT max(price) @@ -159,7 +159,7 @@ MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread) × 30 As shown in the operator plan extract above, even though `max_threads` is set to `59`, ClickHouse uses only **30** concurrent streams to scan the data. Now let’s run the query: -```sql runnable=true +```sql runnable SELECT max(price) FROM @@ -179,7 +179,7 @@ Peak memory usage: 27.24 MiB. As shown in the output above, the query processed 2.31 million rows and read 13.66MB of data. This is because, during the index analysis phase, ClickHouse selected **282 granules** for processing, each containing 8,192 rows, totaling approximately 2.31 million rows: -```sql runnable=true +```sql runnable EXPLAIN indexes = 1 SELECT max(price)