How to use Log to metric？

[FengZh61]

This is my configuration

data_dir = "/opt/vector"
[sources.mydev]
type = "file"
ignore_older_secs = 600
include = [ "/opt/fluent-bit/metric.log" ]
read_from = "end"

[transforms.json]
type = "remap"
inputs = [ "mydev" ]
source = ". = parse_json!(string!(.message))"

[transforms.gauge]
type = "log_to_metric"
inputs = [ "json" ]

[[transforms.gauge.metrics]]
type = "gauge"
field = "load_avg_1m"

[transforms.gauge.metrics.tags]
host = "{{host}}"

[[transforms.gauge.metrics]]
type = "gauge"
field = "load_avg_5m"

[transforms.gauge.metrics.tags]
host = "{{host}}"

[[transforms.gauge.metrics]]
type = "gauge"
field = "load_avg_15m"

[transforms.gauge.metrics.tags]
host = "{{host}}"

[sinks.print]
type = "console"
inputs = ["gauge"]
encoding.codec = "json"

[sinks.prom]
type = "prometheus_remote_write"
inputs = [ "gauge" ]
endpoint = "http://192.168.159.133:9090/api/v1/write"
default_namespace = "service"

running

[root@mydev config]# ../bin/vector -c gau.toml
2023-02-03T08:42:51.792741Z INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-02-03T08:42:51.792870Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-02-03T08:42:51.792929Z INFO vector::app: Loading configs. paths=["gau.toml"]
2023-02-03T08:42:51.815819Z INFO vector::topology::running: Running healthchecks.
2023-02-03T08:42:51.816065Z INFO source{component_kind="source" component_id=mydev component_type=file component_name=mydev}: vector::sources::file: Starting file server. include=["/opt/fluent-bit/metric.log"] exclude=[]
2023-02-03T08:42:51.816261Z INFO vector: Vector has started. debug="false" version="0.27.0" arch="x86_64" revision="5623d1e 2023-01-18"
2023-02-03T08:42:51.816704Z INFO vector::app: API is disabled, enable by setting api.enabled to true and use commands like vector top.
2023-02-03T08:42:51.816977Z INFO vector::topology::builder: Healthcheck passed.
2023-02-03T08:42:51.817897Z INFO source{component_kind="source" component_id=mydev component_type=file component_name=mydev}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-02-03T08:42:51.818381Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Unexpected status: 405 Method Not Allowed component_kind="sink" component_type="prometheus_remote_write" component_id=prom component_name=prom
2023-02-03T08:42:51.818522Z INFO source{component_kind="source" component_id=mydev component_type=file component_name=mydev}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/opt/fluent-bit/metric.log file_position=5183

Insert Log

echo '{"load_avg_15m": 48.7,"load_avg_1m": 78.2,"load_avg_5m": 56.2,"host": "10.22.11.222","message": "CPU activity sample"}' >>metric.log

I can't find the data in prometheus！

when endpoint = "http://192.168.159.133:9090/"

2023-02-03T08:55:20.045388Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Unexpected status: 302 Found component_kind="sink" component_type="prometheus_remote_write" component_id=prom component_name=prom

Again, I didn't find the data in prometheus

Is there a mistake in my configuration?

tks！

Version & Command

prometheus, version 2.36.2
vector 0.27.0 (x86_64-unknown-linux-musl 5623d1e 2023-01-18)

/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --web.read-timeout=5m --web.max-connections=10 --storage.tsdb.retention=15d --storage.tsdb.path=/opt/prometheus/data --query.max-concurrency=20 --query.timeout=2m --web.enable-admin-api --web.enable-lifecycle --web.enable-remote-write-receiver

[jszwedko]

Hi @FengZh61 !

I think for endpoint you just want http://192.168.159.133:9090/api/v1 as Vector appears to add /write to that.

[FengZh61]

Hi @FengZh61 !

I think for endpoint you just want http://192.168.159.133:9090/api/v1 as Vector appears to add /write to that.

Hi! @jszwedko
http://192.168.159.133:9090/api/v1/write ，this endpoint is correct.
The problem is in the log_to_metric converted data,when I insert the data, it returns with timestamp.
Like this {"name":"load_avg_1m","tags":{"host":"10.22.11.222"},"timestamp":"2023-02-06T05:34:38.885670167Z","kind":"absolute","gauge":{"value":78.2}}
I need to delete timestamp, then remote_write to endpoint, the correct data is {"name":"load_avg_1m","tags":{"host":"10.22.11.222"},"kind":"absolute","gauge":{"value":78.2}}

[jszwedko]

Ah, I see. You should be able use remap to delete the timestamp using del(.timestamp).