How to Convert Strings of Any Encoding to UTF-8?

[ruoshuixuelabi]

This is my configuration. I need to convert the string encoding with .message = decode_charset!(.message, "GBK"). However, the encoding of my string isn't necessarily GBK. Is there any way to convert it to UTF - 8 regardless of its original encoding?

1. Syslog UDP Source

[sources.syslog_udp]
address = "0.0.0.0:25111"
max_length = 20480
mode = "udp"
receive_buffer_bytes = 16777216
type = "socket"

2. Syslog字段解析 + 字段保留

[transforms.format_log]
inputs = ["syslog_udp"]
source = '''
  if .host != null {
    .host,err = to_string(.host)
    if err != null {
     .host = ""
    }
  } else {
    .host = ""
  }
    .message=decode_charset!(.message, "GBK")
    if .timestamp != null {
    .timestamp,err = to_string(.timestamp)
    if err != null {
     .timestamp = ""
    }
  } else {
    .timestamp = ""
  }
  .host=join!(["host=", "\"", .host, "\""],separator: "")
  .receive_time=join!(["receive_time=", "\"", .timestamp, "\""],separator: "")
  # 构建最终 JSON 对象
  .message=join!([.message,.host,.receive_time],separator: " ")
 to_string(.message)
'''
type = "remap"

5. Kafka输出

[sinks.kafka_file_logs]
batch.max_events = 500000
batch.timeout_secs = 2
bootstrap_servers = "127.0.0.1:9092"
compression = "snappy"
inputs = ["format_log"]
topic = "nbp_kafka_file_logs"
type = "kafka"
[sinks.kafka_file_logs.encoding]
codec = "text"
# 可选：添加缓冲区配置提高吞吐量
[sinks.kafka_file_logs.buffer]
max_events = 1000000
type = "memory"
[sources.metrics_id]
scrape_interval_secs = 1
type = "internal_metrics"
[sinks.sync_metrics_id]
address = "0.0.0.0:9598"
default_namespace = "service"
inputs = ["metrics_id"]
type = "prometheus_exporter"

[pront]

VRL provides https://vector.dev/docs/reference/vrl/functions/#decode_charset but you have to know the original charset. If there are many possible charsets, you can try decode_charset on each one until it passes.