How to Convert Strings of Any Encoding to UTF-8?

[ruoshuixuelabi]

This is my configuration. I need to convert the string encoding with .message = decode_charset!(.message, "GBK"). However, the encoding of my string isn't necessarily GBK. Is there any way to convert it to UTF - 8 regardless of its original encoding?

1. Syslog UDP Source

[sources.syslog_udp]
address = "0.0.0.0:25111"
max_length = 20480
mode = "udp"
receive_buffer_bytes = 16777216
type = "socket"

2. Syslog字段解析 + 字段保留

[transforms.format_log]
inputs = ["syslog_udp"]
source = '''
if .host != null {
.host,err = to_string(.host)
if err != null {
.host = ""
}
} else {
.host = ""
}
.message=decode_charset!(.message, "GBK")
if .timestamp != null {
.timestamp,err = to_string(.timestamp)
if err != null {
.timestamp = ""
}
} else {
.timestamp = ""
}
.host=join!(["host=", """, .host, """],separator: "")
.receive_time=join!(["receive_time=", """, .timestamp, """],separator: "")

构建最终 JSON 对象

.message=join!([.message,.host,.receive_time],separator: " ")
to_string(.message)
'''
type = "remap"

5. Kafka输出

[sinks.kafka_file_logs]
batch.max_events = 500000
batch.timeout_secs = 2
bootstrap_servers = "127.0.0.1:9092"
compression = "snappy"
inputs = ["format_log"]
topic = "nbp_kafka_file_logs"
type = "kafka"
[sinks.kafka_file_logs.encoding]
codec = "text"

可选：添加缓冲区配置提高吞吐量

[sinks.kafka_file_logs.buffer]
max_events = 1000000
type = "memory"
[sources.metrics_id]
scrape_interval_secs = 1
type = "internal_metrics"
[sinks.sync_metrics_id]
address = "0.0.0.0:9598"
default_namespace = "service"
inputs = ["metrics_id"]
type = "prometheus_exporter"

[pront]

VRL provides https://vector.dev/docs/reference/vrl/functions/#decode_charset but you have to know the original charset. If there are many possible charsets, you can try decode_charset on each one until it passes.