Skip to content

Commit 5486e80

Browse files
BlargianBlargian
authored
Merge pull request #3936 from mneedham/dynamic-col-selection
Dynamic column selection
2 parents f18e954 + 472d876 commit 5486e80

File tree

2 files changed

+206
-0
lines changed

2 files changed

+206
-0
lines changed

‎docs/guides/developer/dynamic-column-selection.md

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
---
2+
slug: /guides/developer/dynamic-column-selection
3+
sidebar_label: 'Dynamic column selection'
4+
title: 'Dynamic column selection'
5+
description: 'Use alternative query languages in ClickHouse'
6+
---
7+
8+
[Dynamic column selection](/docs/sql-reference/statements/select#dynamic-column-selection) is a powerful but underutilized ClickHouse feature that allows you to select columns using regular expressions instead of naming each column individually. You can also apply functions to matching columns using the [`APPLY`](/sql-reference/statements/select#apply) modifier, making it incredibly useful for data analysis and transformation tasks.
9+
10+
We're going to learn how to use this feature with help from the [New York taxis dataset](/docs/getting-started/example-datasets/nyc-taxi), which you can also find in the [ClickHouse SQL playground](https://sql.clickhouse.com?query=LS0gRGF0YXNldCBjb250YWluaW5nIHRheGkgcmlkZSBkYXRhIGluIE5ZQyBmcm9tIDIwMDkuIE1vcmUgaW5mbyBoZXJlOiBodHRwczovL2NsaWNraG91c2UuY29tL2RvY3MvZW4vZ2V0dGluZy1zdGFydGVkL2V4YW1wbGUtZGF0YXNldHMvbnljLXRheGkKU0VMRUNUICogRlJPTSBueWNfdGF4aS50cmlwcyBMSU1JVCAxMDA).
11+
12+
<iframe width="768" height="432" src="https://www.youtube.com/embed/moabRqqHNo4?si=jgmInV-u3UxtLvMS" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
13+
14+
## Selecting columns that match a pattern {#selecting-columns}
15+
16+
Let's start with a common scenario: selecting only the columns that contain `_amount` from the NYC taxi dataset. Instead of manually typing each column name, we can use the `COLUMNS` expression with a regular expression:
17+
18+
```sql
19+
FROM nyc_taxi.trips
20+
SELECT COLUMNS('.*_amount')
21+
LIMIT 10;
22+
```
23+
24+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudCcpCkZST00gbnljX3RheGkudHJpcHMKTElNSVQgMTA7&run_query=true)
25+
26+
This query returns the first 10 rows, but only for columns whose names match the pattern `.*_amount` (any characters followed by "_amount").
27+
28+
```text
29+
┌─fare_amount─┬─tip_amount─┬─tolls_amount─┬─total_amount─┐
30+
1. │ 9 │ 0 │ 0 │ 9.8 │
31+
2. │ 9 │ 0 │ 0 │ 9.8 │
32+
3. │ 3.5 │ 0 │ 0 │ 4.8 │
33+
4. │ 3.5 │ 0 │ 0 │ 4.8 │
34+
5. │ 3.5 │ 0 │ 0 │ 4.3 │
35+
6. │ 3.5 │ 0 │ 0 │ 4.3 │
36+
7. │ 2.5 │ 0 │ 0 │ 3.8 │
37+
8. │ 2.5 │ 0 │ 0 │ 3.8 │
38+
9. │ 5 │ 0 │ 0 │ 5.8 │
39+
10. │ 5 │ 0 │ 0 │ 5.8 │
40+
└─────────────┴────────────┴──────────────┴──────────────┘
41+
```
42+
43+
Let’s say we also want to return columns that contain the terms `fee` or `tax`.
44+
We can update the regular expression to include those:
45+
46+
```sql
47+
SELECT COLUMNS('.*_amount|fee|tax')
48+
FROM nyc_taxi.trips
49+
ORDER BY rand()
50+
LIMIT 3;
51+
```
52+
53+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykKRlJPTSBueWNfdGF4aS50cmlwcwpPUkRFUiBCWSByYW5kKCkgCkxJTUlUIDM7&run_query=true)
54+
55+
```text
56+
┌─fare_amount─┬─mta_tax─┬─tip_amount─┬─tolls_amount─┬─ehail_fee─┬─total_amount─┐
57+
1. │ 5 │ 0.5 │ 1 │ 0 │ 0 │ 7.8 │
58+
2. │ 12.5 │ 0.5 │ 0 │ 0 │ 0 │ 13.8 │
59+
3. │ 4.5 │ 0.5 │ 1.66 │ 0 │ 0 │ 9.96 │
60+
└─────────────┴─────────┴────────────┴──────────────┴───────────┴──────────────┘
61+
```
62+
63+
## Selecting multiple patterns {#selecting-multiple-patterns}
64+
65+
We can combine multiple column patterns in a single query:
66+
67+
```sql
68+
SELECT
69+
COLUMNS('.*_amount'),
70+
COLUMNS('.*_date.*')
71+
FROM nyc_taxi.trips
72+
LIMIT 5;
73+
```
74+
75+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIAogICAgQ09MVU1OUygnLipfYW1vdW50JyksCiAgICBDT0xVTU5TKCcuKl9kYXRlLionKQpGUk9NIG55Y190YXhpLnRyaXBzCkxJTUlUIDU7&run_query=true)
76+
77+
```text
78+
┌─fare_amount─┬─tip_amount─┬─tolls_amount─┬─total_amount─┬─pickup_date─┬─────pickup_datetime─┬─dropoff_date─┬────dropoff_datetime─┐
79+
1. │ 9 │ 0 │ 0 │ 9.8 │ 2001-01-01 │ 2001-01-01 00:01:48 │ 2001-01-01 │ 2001-01-01 00:15:47 │
80+
2. │ 9 │ 0 │ 0 │ 9.8 │ 2001-01-01 │ 2001-01-01 00:01:48 │ 2001-01-01 │ 2001-01-01 00:15:47 │
81+
3. │ 3.5 │ 0 │ 0 │ 4.8 │ 2001-01-01 │ 2001-01-01 00:02:08 │ 2001-01-01 │ 2001-01-01 01:00:02 │
82+
4. │ 3.5 │ 0 │ 0 │ 4.8 │ 2001-01-01 │ 2001-01-01 00:02:08 │ 2001-01-01 │ 2001-01-01 01:00:02 │
83+
5. │ 3.5 │ 0 │ 0 │ 4.3 │ 2001-01-01 │ 2001-01-01 00:02:26 │ 2001-01-01 │ 2001-01-01 00:04:49 │
84+
└─────────────┴────────────┴──────────────┴──────────────┴─────────────┴─────────────────────┴──────────────┴─────────────────────┘
85+
```
86+
87+
## Apply functions to all columns {#applying-functions}
88+
89+
We can also use the [`APPLY`](https://clickhouse.com/docs/sql-reference/statements/select#apply) modifier to apply functions across every column.
90+
For example, if we wanted to find the maximum value of each of those columns, we could run the following query:
91+
92+
```sql
93+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(max)
94+
FROM nyc_taxi.trips;
95+
```
96+
97+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkobWF4KQpGUk9NIG55Y190YXhpLnRyaXBzOw&run_query=true)
98+
99+
100+
```text
101+
┌─max(fare_amount)─┬─max(mta_tax)─┬─max(tip_amount)─┬─max(tolls_amount)─┬─max(ehail_fee)─┬─max(total_amount)─┐
102+
1. │ 998310 │ 500000.5 │ 3950588.8 │ 7999.92 │ 1.95 │ 3950611.5 │
103+
└──────────────────┴──────────────┴─────────────────┴───────────────────┴────────────────┴───────────────────┘
104+
```
105+
106+
Or maybe, we’d like to see the average instead:
107+
108+
```sql
109+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg)
110+
FROM nyc_taxi.trips
111+
```
112+
113+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkoYXZnKQpGUk9NIG55Y190YXhpLnRyaXBzOw&run_query=true)
114+
115+
116+
```text
117+
┌─avg(fare_amount)─┬───────avg(mta_tax)─┬────avg(tip_amount)─┬──avg(tolls_amount)─┬──────avg(ehail_fee)─┬──avg(total_amount)─┐
118+
1. │ 11.8044154834777 │ 0.4555942672733423 │ 1.3469850969211845 │ 0.2256511991414463 │ 3.37600560437412e-9 │ 14.423323722271563 │
119+
└──────────────────┴────────────────────┴────────────────────┴────────────────────┴─────────────────────┴────────────────────┘
120+
```
121+
122+
123+
Those values contain a lot of decimal places, but luckily we can fix that by chaining functions. In this case, we’ll apply the avg function, followed by the round function:
124+
125+
```sql
126+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg) APPLY(round)
127+
FROM nyc_taxi.trips;
128+
```
129+
130+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkoYXZnKSBBUFBMWShyb3VuZCkKRlJPTSBueWNfdGF4aS50cmlwczs&run_query=true)
131+
132+
133+
```text
134+
┌─round(avg(fare_amount))─┬─round(avg(mta_tax))─┬─round(avg(tip_amount))─┬─round(avg(tolls_amount))─┬─round(avg(ehail_fee))─┬─round(avg(total_amount))─┐
135+
1. │ 12 │ 0 │ 1 │ 0 │ 0 │ 14 │
136+
└─────────────────────────┴─────────────────────┴────────────────────────┴──────────────────────────┴───────────────────────┴──────────────────────────┘
137+
```
138+
139+
140+
But that rounds the averages to whole numbers. If we want to round to, say, 2 decimal places, we can do that as well. As well as taking in functions, the `APPLY` modifier accepts a lambda, which gives us the flexibility to have the round function round our average values to 2 decimal places:
141+
142+
```sql
143+
SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg) APPLY(x -> round(x, 2))
144+
FROM nyc_taxi.trips;
145+
```
146+
147+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkgYXZnIEFQUExZIHggLT4gcm91bmQoeCwgMikKRlJPTSBueWNfdGF4aS50cmlwcw&run_query=true)
148+
149+
150+
```text
151+
┌─round(avg(fare_amount), 2)─┬─round(avg(mta_tax), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(tolls_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(total_amount), 2)─┐
152+
1. │ 11.8 │ 0.46 │ 1.35 │ 0.23 │ 0 │ 14.42 │
153+
└────────────────────────────┴────────────────────────┴───────────────────────────┴─────────────────────────────┴──────────────────────────┴─────────────────────────────┘
154+
```
155+
156+
## Replacing columns {#replacing-columns}
157+
158+
So far so good. But let’s say we want to adjust one of the values, while leaving the other ones as they are. For example, maybe we want to double the total amount and divide the MTA tax by 1.1. We can do that by using the [`REPLACE`](/sql-reference/statements/select#replace) modifier, which will replace a column while leaving the other ones as they are.
159+
160+
```sql
161+
FROM nyc_taxi.trips
162+
SELECT
163+
COLUMNS('.*_amount|fee|tax')
164+
REPLACE(
165+
total_amount*2 AS total_amount,
166+
mta_tax/1.1 AS mta_tax
167+
)
168+
APPLY(avg)
169+
APPLY(col -> round(col, 2));
170+
```
171+
172+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=RlJPTSBueWNfdGF4aS50cmlwcyAKU0VMRUNUIAogIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykKICBSRVBMQUNFKAogICAgdG90YWxfYW1vdW50KjIgQVMgdG90YWxfYW1vdW50LAogICAgbXRhX3RheC8xLjEgQVMgbXRhX3RheAogICkgCiAgQVBQTFkoYXZnKQogIEFQUExZKGNvbCAtPiByb3VuZChjb2wsIDIpKTs&run_query=true)
173+
174+
175+
```text
176+
┌─round(avg(fare_amount), 2)─┬─round(avg(di⋯, 1.1)), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(tolls_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(mu⋯nt, 2)), 2)─┐
177+
1. │ 11.8 │ 0.41 │ 1.35 │ 0.23 │ 0 │ 28.85 │
178+
└────────────────────────────┴──────────────────────────┴───────────────────────────┴─────────────────────────────┴──────────────────────────┴──────────────────────────┘
179+
```
180+
181+
## Excluding columns {#excluding-columns}
182+
183+
We can also choose to exclude a field by using the [`EXCEPT`](/sql-reference/statements/select#except) modifier. For example, to remove the `tolls_amount` column, we would write the following query:
184+
185+
```sql
186+
FROM nyc_taxi.trips
187+
SELECT
188+
COLUMNS('.*_amount|fee|tax') EXCEPT(tolls_amount)
189+
REPLACE(
190+
total_amount*2 AS total_amount,
191+
mta_tax/1.1 AS mta_tax
192+
)
193+
APPLY(avg)
194+
APPLY(col -> round(col, 2));
195+
```
196+
197+
> [Try this query in the SQL playground](https://sql.clickhouse.com?query=RlJPTSBueWNfdGF4aS50cmlwcyAKU0VMRUNUIAogIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgRVhDRVBUKHRvbGxzX2Ftb3VudCkKICBSRVBMQUNFKAogICAgdG90YWxfYW1vdW50KjIgQVMgdG90YWxfYW1vdW50LAogICAgbXRhX3RheC8xLjEgQVMgbXRhX3RheAogICkgCiAgQVBQTFkoYXZnKQogIEFQUExZKGNvbCAtPiByb3VuZChjb2wsIDIpKTs&run_query=true)
198+
199+
200+
201+
```text
202+
┌─round(avg(fare_amount), 2)─┬─round(avg(di⋯, 1.1)), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(mu⋯nt, 2)), 2)─┐
203+
1. │ 11.8 │ 0.41 │ 1.35 │ 0 │ 28.85 │
204+
└────────────────────────────┴──────────────────────────┴───────────────────────────┴──────────────────────────┴──────────────────────────┘
205+
```

‎sidebars.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1174,6 +1174,7 @@ const sidebars = {
11741174
collapsible: false,
11751175
link: { type: "doc", id: "guides/developer/index" },
11761176
items: [
1177+
"guides/developer/dynamic-column-selection",
11771178
"guides/developer/alternative-query-languages",
11781179
"guides/developer/cascading-materialized-views",
11791180
"guides/developer/debugging-memory-issues",

0 commit comments

Comments
 (0)