How to use fields in clickhouse sincs (table, database ?)

[Fritzss]

scenario:
server A sends data to a vector on server B
server B receives and sends to clickhouse
(server A .... n, there can be many)
################
src logs_server cat /etc/vector/logs.toml

data_dir = "/var/lib/vector"

[acknowledgements]
enabled = true

[sources.logs_svc]
type = "file"
file_key = "file"
glob_minimum_cooldown_ms = 1_000
host_key = "host"
ignore_older_secs = 600
include = [
            "/var/log/svc/*.log"
          ]

max_line_bytes = 1024000
max_read_bytes = 1048576
read_from = "beginning"
data_dir = "/var/lib/vector"

[transforms.json_svc]
type = "remap"
inputs = ["logs_svc"]
source = """
        . = merge(object!(parse_json!(.message, max_depth: 1)), . )
        .table = del(.message)
        .table = "svc_logs"
        table = .table
         """
[sinks.vector_ch]
  type = "vector"
  inputs = [ "json_svc" ]
  address = "10.0.0.123:6000"
  compression = true
  buffer.type = "memory"
  buffer.max_events = 5000
  buffer.when_full = "block"
  batch.max_events = 1000
  batch.timeout_secs = 60
  request.concurrency = "adaptive"

#############
dst_server ckickhouse + vector cat /etc/vector/vector.toml

data_dir = "/var/lib/vector"


[acknowledgements]
enabled = true


# Random Syslog-formatted logs
[sources.vector]
  type = "vector"
  version = "2"
  address = "0.0.0.0:6000"


# Print parsed logs to stdout
#[sinks.print]
# type = "console"
# inputs = ["vector"]
# encoding.codec = "json"

[sinks.clickhouse_vault]
  type = "clickhouse"
  inputs = [ "vector" ]
  endpoint = "http://127.0.0.1:8123"
  database = "logs"
  table = {{ table }}
  encoding.timestamp_format = "unix"
  buffer.type = "memory"
  buffer.max_events = 5000
  buffer.when_full = "block"
  batch.max_events = 1000
  batch.timeout_secs = 20
  skip_unknown_fields = true
  request.concurrency = "adaptive"

[neuronull]

Hi @Fritzss ,

What is the error or problematic behavior you are seeing?

It looks to me like the remap transform in server A , is renaming the message field to "table", and then the value of the field table is being set to "svc_logs".

What are you trying to achieve with the transform?

[Fritzss]

DB::Exception: Table logs.{{ table }} doesn't exist.

[Fritzss]

I want to pass the table name to syncs clickhouse, but table name is not set

[Fritzss]

what is the syntax to pass a variable to the sincs and the variable is applied ?

[neuronull]

Thanks for those extra details.

It looks like the template syntax (https://vector.dev/docs/reference/configuration/template-syntax/)
, is not supported yet for the table field of the Clickhouse sink (https://vector.dev/docs/reference/configuration/sinks/clickhouse/#table).

There is an open feature request tracking support for that here: #14283

You can help us prioritize that work by adding a thumbs up ("+1") , to the initial comment of that issue, optionally adding a comment there.