Skip to content

Made some of the queries runnable. #3639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 10, 2025
Merged

Made some of the queries runnable. #3639

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
96b9be0
Made some of the queries runnable.
Apr 10, 2025
0e89116
Update query-parallelism.md
Blargian Apr 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions docs/guides/best-practices/query-parallelism.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,11 @@ SETTINGS send_logs_level='trace';
```

We can see that



* ① ClickHouse needs to read 3,609 granules (indicated as marks in the trace logs) across 3 data ranges.
* ② With 59 CPU cores, it distributes this work across 59 parallel processing streams—one per lane.

Alternatively, we can use the [EXPLAIN](/sql-reference/statements/explain#explain-pipeline) clause to inspect the [physical operator plan](/academic_overview#4-2-multi-core-parallelization)—also known as the "query pipeline"—for the aggregation query:
```sql runnable=false
```sql runnable
EXPLAIN PIPELINE
SELECT
max(price)
Expand All @@ -99,6 +96,7 @@ FROM
```

```txt
Static result for the query above from April 2025
┌─explain───────────────────────────────────────────────────────────────────────────┐
1. │ (Expression) │
2. │ ExpressionTransform × 59 │
Expand Down Expand Up @@ -130,18 +128,19 @@ Note that the `Resize` operators in the physical plan above [repartition and red
## Why max_threads isn't always respected {#why-max-threads-isnt-always-respected}

As mentioned above, the number of `n` parallel processing lanes is controlled by the `max_threads` setting, which by default matches the number of CPU cores available to ClickHouse on the server:
```sql runnable=false
```sql runnable
SELECT getSetting('max_threads');
```

```txt
Static result for the query above from April 2025
┌─getSetting('max_threads')─┐
1. │ 59 │
└───────────────────────────┘
```

However, the `max_threads` value may be ignored depending on the amount of data selected for processing:
```sql runnable=false
However, the `max_threads` value may be ignored for some plan operators depending on the amount of data selected for processing:
```sql runnable
EXPLAIN PIPELINE
SELECT
max(price)
Expand All @@ -151,6 +150,7 @@ WHERE town = 'LONDON';
```

```txt
Static result for the query above from April 2025
...
(ReadFromMergeTree)
MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread) × 30
Expand All @@ -159,7 +159,7 @@ MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread) × 30
As shown in the operator plan extract above, even though `max_threads` is set to `59`, ClickHouse uses only **30** concurrent streams to scan the data.

Now let’s run the query:
```sql runnable=false
```sql runnable
SELECT
max(price)
FROM
Expand All @@ -168,6 +168,7 @@ WHERE town = 'LONDON';
```

```txt
Static result for the query above from April 2025
┌─max(price)─┐
1. │ 594300000 │ -- 594.30 million
└────────────┘
Expand All @@ -178,7 +179,7 @@ Peak memory usage: 27.24 MiB.

As shown in the output above, the query processed 2.31 million rows and read 13.66MB of data. This is because, during the index analysis phase, ClickHouse selected **282 granules** for processing, each containing 8,192 rows, totaling approximately 2.31 million rows:

```sql runnable=false
```sql runnable
EXPLAIN indexes = 1
SELECT
max(price)
Expand All @@ -188,6 +189,7 @@ WHERE town = 'LONDON';
```

```txt
Static result for the query above from April 2025
┌─explain───────────────────────────────────────────────┐
1. │ Expression ((Project names + Projection)) │
2. │ Aggregating │
Expand All @@ -204,7 +206,7 @@ WHERE town = 'LONDON';
└───────────────────────────────────────────────────────┘
```

Regardless of the configured `max_threads` value, ClickHouse only allocates additional parallel processing lanes when there’s enough data to justify them. The "max" in `max_threads` refers to an upper limit, not a guaranteed number of threads used.
Regardless of the configured `max_threads` value, ClickHouse only allocates additional parallel processing lanes for operator plans scanning the data when there’s enough data to justify them. The "max" in `max_threads` refers to an upper limit, not a guaranteed number of threads used.

What "enough data" means is primarily determined by two settings, which define the minimum number of rows (163,840 by default) and the minimum number of bytes (2,097,152 by default) that each processing lane should handle:

Expand Down