ClickHouse · Blargian · Jun 12, 2025 · Jun 12, 2025 · Jun 12, 2025 · Jun 12, 2025
@@ -0,0 +1,205 @@
+---
+slug: /guides/developer/dynamic-column-selection
+sidebar_label: 'Dynamic column selection'
+title: 'Dynamic column selection'
+description: 'Use alternative query languages in ClickHouse'
+---
+
+[Dynamic column selection](/docs/sql-reference/statements/select#dynamic-column-selection) is a powerful but underutilized ClickHouse feature that allows you to select columns using regular expressions instead of naming each column individually. You can also apply functions to matching columns using the APPLY modifier, making it incredibly useful for data analysis and transformation tasks.
+
+We're going to learn how to use this feature with help from the [New York taxis dataset](/docs/getting-started/example-datasets/nyc-taxi), which you can find also find in the [ClickHouse SQL playground](https://sql.clickhouse.com?query=LS0gRGF0YXNldCBjb250YWluaW5nIHRheGkgcmlkZSBkYXRhIGluIE5ZQyBmcm9tIDIwMDkuIE1vcmUgaW5mbyBoZXJlOiBodHRwczovL2NsaWNraG91c2UuY29tL2RvY3MvZW4vZ2V0dGluZy1zdGFydGVkL2V4YW1wbGUtZGF0YXNldHMvbnljLXRheGkKU0VMRUNUICogRlJPTSBueWNfdGF4aS50cmlwcyBMSU1JVCAxMDA).
+
+<iframe width="768" height="432" src="https://www.youtube.com/embed/moabRqqHNo4?si=jgmInV-u3UxtLvMS" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
+
+## Selecting columns that match a pattern  {#selecting-columns}
+
+Let's start with a common scenario: selecting only the columns that contain `_amount` from the NYC taxi dataset. Instead of manually typing each column name, we can use the `COLUMNS` expression with a regular expression:
+
+```sql
+FROM nyc_taxi.trips
+SELECT COLUMNS('.*_amount')
+LIMIT 10;
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudCcpCkZST00gbnljX3RheGkudHJpcHMKTElNSVQgMTA7&run_query=true)
+
+This query returns the first 10 rows, but only for columns whose names match the pattern `.*_amount` (any characters followed by "_amount").
+
+```text
+    ┌─fare_amount─┬─tip_amount─┬─tolls_amount─┬─total_amount─┐
+ 1. │           9 │          0 │            0 │          9.8 │
+ 2. │           9 │          0 │            0 │          9.8 │
+ 3. │         3.5 │          0 │            0 │          4.8 │
+ 4. │         3.5 │          0 │            0 │          4.8 │
+ 5. │         3.5 │          0 │            0 │          4.3 │
+ 6. │         3.5 │          0 │            0 │          4.3 │
+ 7. │         2.5 │          0 │            0 │          3.8 │
+ 8. │         2.5 │          0 │            0 │          3.8 │
+ 9. │           5 │          0 │            0 │          5.8 │
+10. │           5 │          0 │            0 │          5.8 │
+    └─────────────┴────────────┴──────────────┴──────────────┘
+```
+
+Let’s say we also want to return columns that contain the terms `fee` or `tax`. 
+We can update the regular expression to include those:
+
+```sql
+SELECT COLUMNS('.*_amount|fee|tax')
+FROM nyc_taxi.trips
+ORDER BY rand() 
+LIMIT 3;
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykKRlJPTSBueWNfdGF4aS50cmlwcwpPUkRFUiBCWSByYW5kKCkgCkxJTUlUIDM7&run_query=true)
+
+```text
+   ┌─fare_amount─┬─mta_tax─┬─tip_amount─┬─tolls_amount─┬─ehail_fee─┬─total_amount─┐
+1. │           5 │     0.5 │          1 │            0 │         0 │          7.8 │
+2. │        12.5 │     0.5 │          0 │            0 │         0 │         13.8 │
+3. │         4.5 │     0.5 │       1.66 │            0 │         0 │         9.96 │
+   └─────────────┴─────────┴────────────┴──────────────┴───────────┴──────────────┘
+```
+
+## Selecting multiple patterns  {#selecting-multiple-patterns}
+
+We can combine multiple column patterns in a single query:
+
+```sql
+SELECT 
+    COLUMNS('.*_amount'),
+    COLUMNS('.*_date.*')
+FROM nyc_taxi.trips
+LIMIT 5;
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIAogICAgQ09MVU1OUygnLipfYW1vdW50JyksCiAgICBDT0xVTU5TKCcuKl9kYXRlLionKQpGUk9NIG55Y190YXhpLnRyaXBzCkxJTUlUIDU7&run_query=true)
+
+```text
+   ┌─fare_amount─┬─tip_amount─┬─tolls_amount─┬─total_amount─┬─pickup_date─┬─────pickup_datetime─┬─dropoff_date─┬────dropoff_datetime─┐
+1. │           9 │          0 │            0 │          9.8 │  2001-01-01 │ 2001-01-01 00:01:48 │   2001-01-01 │ 2001-01-01 00:15:47 │
+2. │           9 │          0 │            0 │          9.8 │  2001-01-01 │ 2001-01-01 00:01:48 │   2001-01-01 │ 2001-01-01 00:15:47 │
+3. │         3.5 │          0 │            0 │          4.8 │  2001-01-01 │ 2001-01-01 00:02:08 │   2001-01-01 │ 2001-01-01 01:00:02 │
+4. │         3.5 │          0 │            0 │          4.8 │  2001-01-01 │ 2001-01-01 00:02:08 │   2001-01-01 │ 2001-01-01 01:00:02 │
+5. │         3.5 │          0 │            0 │          4.3 │  2001-01-01 │ 2001-01-01 00:02:26 │   2001-01-01 │ 2001-01-01 00:04:49 │
+   └─────────────┴────────────┴──────────────┴──────────────┴─────────────┴─────────────────────┴──────────────┴─────────────────────┘
+```
+
+## Apply functions to all columns  {#applying-functions}
+
+We can also use the [`APPLY`](https://clickhouse.com/docs/sql-reference/statements/select#apply) modifier to apply functions across every column. 
+For example, if we wanted to find the maximum value of each of those columns, we could run the following query:
+
+```sql
+SELECT COLUMNS('.*_amount|fee|tax') APPLY(max)
+FROM nyc_taxi.trips;
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkobWF4KQpGUk9NIG55Y190YXhpLnRyaXBzOw&run_query=true)
+
+
+```text
+   ┌─max(fare_amount)─┬─max(mta_tax)─┬─max(tip_amount)─┬─max(tolls_amount)─┬─max(ehail_fee)─┬─max(total_amount)─┐
+1. │           998310 │     500000.5 │       3950588.8 │           7999.92 │           1.95 │         3950611.5 │
+   └──────────────────┴──────────────┴─────────────────┴───────────────────┴────────────────┴───────────────────┘
+```
+
+Or maybe, we’d like to see the average instead:
+
+```sql
+SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg)
+FROM nyc_taxi.trips
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkoYXZnKQpGUk9NIG55Y190YXhpLnRyaXBzOw&run_query=true)
+
+
+```text
+   ┌─avg(fare_amount)─┬───────avg(mta_tax)─┬────avg(tip_amount)─┬──avg(tolls_amount)─┬──────avg(ehail_fee)─┬──avg(total_amount)─┐
+1. │ 11.8044154834777 │ 0.4555942672733423 │ 1.3469850969211845 │ 0.2256511991414463 │ 3.37600560437412e-9 │ 14.423323722271563 │
+   └──────────────────┴────────────────────┴────────────────────┴────────────────────┴─────────────────────┴────────────────────┘
+```
+
+
+Those values contain a lot of decimal places, but luckily we can fix that by chaining functions. In this case, we’ll apply the avg function, followed by the round function:
+
+```sql
+SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg) APPLY(round)
+FROM nyc_taxi.trips;
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkoYXZnKSBBUFBMWShyb3VuZCkKRlJPTSBueWNfdGF4aS50cmlwczs&run_query=true)
+
+
+```text
+   ┌─round(avg(fare_amount))─┬─round(avg(mta_tax))─┬─round(avg(tip_amount))─┬─round(avg(tolls_amount))─┬─round(avg(ehail_fee))─┬─round(avg(total_amount))─┐
+1. │                      12 │                   0 │                      1 │                        0 │                     0 │                       14 │
+   └─────────────────────────┴─────────────────────┴────────────────────────┴──────────────────────────┴───────────────────────┴──────────────────────────┘
+```
+
+
+But that rounds the averages to whole numbers. If we want to round to, say, 2 decimal places, we can do that as well. As well as taking in functions, the APPLY function takes in a lambda, which gives us the flexibility to have the round function round our average values to 2 decimal places:
+
+```sql
+SELECT COLUMNS('.*_amount|fee|tax') APPLY(avg) APPLY(x -> round(x, 2))
+FROM nyc_taxi.trips;
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=U0VMRUNUIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgQVBQTFkgYXZnIEFQUExZIHggLT4gcm91bmQoeCwgMikKRlJPTSBueWNfdGF4aS50cmlwcw&run_query=true)
+
+
+```text
+   ┌─round(avg(fare_amount), 2)─┬─round(avg(mta_tax), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(tolls_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(total_amount), 2)─┐
+1. │                       11.8 │                   0.46 │                      1.35 │                        0.23 │                        0 │                       14.42 │
+   └────────────────────────────┴────────────────────────┴───────────────────────────┴─────────────────────────────┴──────────────────────────┴─────────────────────────────┘
+```
+
+## Replacing columns  {#replacing-columns}
+
+So far so good. But let’s say we want to adjust one of the values, while leaving the other ones as they are. For example, maybe we want to double the total amount and divide the MTA tax by 1.1. We can do that by using the REPLACE clause, which will replace a column while leaving the other ones as they are.
+
+```sql
+FROM nyc_taxi.trips 
+SELECT 
+  COLUMNS('.*_amount|fee|tax')
+  REPLACE(
+    total_amount*2 AS total_amount,
+    mta_tax/1.1 AS mta_tax
+  ) 
+  APPLY(avg)
+  APPLY(col -> round(col, 2));
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=RlJPTSBueWNfdGF4aS50cmlwcyAKU0VMRUNUIAogIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykKICBSRVBMQUNFKAogICAgdG90YWxfYW1vdW50KjIgQVMgdG90YWxfYW1vdW50LAogICAgbXRhX3RheC8xLjEgQVMgbXRhX3RheAogICkgCiAgQVBQTFkoYXZnKQogIEFQUExZKGNvbCAtPiByb3VuZChjb2wsIDIpKTs&run_query=true)
+
+
+```text
+   ┌─round(avg(fare_amount), 2)─┬─round(avg(di⋯, 1.1)), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(tolls_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(mu⋯nt, 2)), 2)─┐
+1. │                       11.8 │                     0.41 │                      1.35 │                        0.23 │                        0 │                    28.85 │
+   └────────────────────────────┴──────────────────────────┴───────────────────────────┴─────────────────────────────┴──────────────────────────┴──────────────────────────┘
+```
+
+## Excluding columns  {#excluding-columns}
+
+We can also choose to exclude a field by using the EXCEPT clause. For example, to remove the tolls_amount column, we would write the following query:
+
+```sql
+FROM nyc_taxi.trips 
+SELECT 
+  COLUMNS('.*_amount|fee|tax') EXCEPT(tolls_amount)
+  REPLACE(
+    total_amount*2 AS total_amount,
+    mta_tax/1.1 AS mta_tax
+  ) 
+  APPLY(avg)
+  APPLY(col -> round(col, 2));
+```
+
+> [Try this query in the SQL playground](https://sql.clickhouse.com?query=RlJPTSBueWNfdGF4aS50cmlwcyAKU0VMRUNUIAogIENPTFVNTlMoJy4qX2Ftb3VudHxmZWV8dGF4JykgRVhDRVBUKHRvbGxzX2Ftb3VudCkKICBSRVBMQUNFKAogICAgdG90YWxfYW1vdW50KjIgQVMgdG90YWxfYW1vdW50LAogICAgbXRhX3RheC8xLjEgQVMgbXRhX3RheAogICkgCiAgQVBQTFkoYXZnKQogIEFQUExZKGNvbCAtPiByb3VuZChjb2wsIDIpKTs&run_query=true)
+
+
+
+```text
+   ┌─round(avg(fare_amount), 2)─┬─round(avg(di⋯, 1.1)), 2)─┬─round(avg(tip_amount), 2)─┬─round(avg(ehail_fee), 2)─┬─round(avg(mu⋯nt, 2)), 2)─┐
+1. │                       11.8 │                     0.41 │                      1.35 │                        0 │                    28.85 │
+   └────────────────────────────┴──────────────────────────┴───────────────────────────┴──────────────────────────┴──────────────────────────┘
+```
@@ -1174,6 +1174,7 @@ const sidebars = {
       collapsible: false,
       link: { type: "doc", id: "guides/developer/index" },
       items: [
+        "guides/developer/dynamic-column-selection",
         "guides/developer/alternative-query-languages",
         "guides/developer/cascading-materialized-views",
         "guides/developer/debugging-memory-issues",