docker_source logs not going to elasticsearch

[agorkiy]

Hi
I have 2 vector instances - first one is collecting docker logs and sends them to another vector, which then modifies them and pushes to ES.
First vector config looks like this:

[sources.docker_logs]
type = "docker_logs"
docker_host = "unix:///var/run/docker.sock"
exclude_containers = [ "node-exporter" ]
include_containers = [ "node", "jibri" ]

[transforms.docker_transform]
type = "remap"
inputs = [ "docker_logs" ]
source = """
.host = "example_hostname"
"""

[sinks.vector_server]
type = "vector"
inputs = [ "docker_transform" ]
address = "vector-server.example.com:6000"
version = "2"
tls.ca_file    = "/etc/vector/ca.pem"
tls.crt_file   = "/etc/vector/client-cert.pem"
tls.enabled    = true
tls.key_file   = "/etc/vector/client-key.pem"
tls.verify_certificate = true

And second vector config looks like this (it collects janus logs from another vector instance as well) :

[sources.janus_logs]
type           = "vector"
address        = "0.0.0.0:6000"
tls.ca_file    = "/etc/vector/ca.pem"
tls.crt_file   = "/etc/vector/server-cert.pem"
tls.enabled    = true
tls.key_file   = "/etc/vector/server-key.pem"
tls.verify_certificate = true

[transforms.janus_transform]
type = "remap"
inputs = [ "janus_logs" ]
source = """
.@timestamp = del(.timestamp)"""

[sinks.es_cluster]
inputs         = ["janus_transform"]
type           = "elasticsearch"
bulk.index     = "vector-{{host}}-%Y-%m-%d"
endpoint       = "http://elasticsearch:9200"
auth.user      = "user"
auth.password  = "password"
auth.strategy  = "basic"

Problem is that nothing appears in ES index. I see it gets created, but it's empty:

green open vector-example_hostname-2022-05-13 xulUKCDARGqgkjVL-Jo_zg 1 0      0      0    226b    226b

I checked with netcat on second vector - it receives messages from first vector with docker logs and it looks like this:

source_type�
�docker
�
�stream�
�stdout
�
	timestamp��
                  ������������^C

So, what am I missing here? Maybe docker logs should be normalized some way? Because at first I had single index for both janus logs and vector logs and ES throw error that can't merge a non object mapping [label.com.docker.compose.project] with an object mapping. So I created separate index for docker logs based on host, but it seems that it didn't help/there is another reason why ES can't receive logs from vector.
Thanks in advance for any help.

[jszwedko]

Hi @agorkiy !

I think you need to set version = "2" on the vector source you have defined as well. Do you see any error logs from Vector?

[agorkiy]

Turned out that docker logs were received by vector in such format:

{"@timestamp":"2022-05-16T12:10:12.191738687Z","container_created_at":"2022-05-16T08:45:07.613962744Z","container_id":"7acda..","container_name":"node","host":"example_hostname","image":"example/node:d2afc212...","label":{"com.docker.compose.config-hash":"f3...","com.docker.compose.container-number":"1","com.docker.compose.oneoff":"False","com.docker.compose.project":"backend","com.docker.compose.project.config_files":"docker-compose.yaml","com.docker.compose.project.working_dir":"/root/backend","com.docker.compose.service":"node","com.docker.compose.version":"1.29.2","traefik.docker.network":"web","traefik.enable":"true","traefik.http.routers.node.entrypoints":"https","traefik.http.routers.node.rule":"Host(example.com) && PathPrefix(/api)","traefik.http.routers.node.service":"node","traefik.http.routers.node.tls.certresolver":"letsEncrypt","traefik.http.services.node.loadbalancer.server.port":"3000"},"message":"\u001b[32m[Nest] 1 - \u001b[39m05/16/2022, 12:10:12 PM \u001b[38;5;3m[getHandle] \u001b[39m\u001b[32mGetting handle 4417326126908923 of 387535687300921 session...\u001b[39m","source_type":"docker","stream":"stdout"}

Not sure what it is wrong here, bu I just deleted label with del(.label) and then it started to work as intended.

[jszwedko]

Ah, yeah, I see. It seems like the data wasn't matching the index mapping. I think the issue might be that Elasticsearch is interpreting the .s in the label names as nested keys. You can see an example of using Lua to replace the .s with _s in #3588 (comment) . We are working on adding support for this to VRL in the next Vector release too which will allow you to do . = map_keys(., recursive: true) -> |key| { replace(key, ".", "_") } in a remap transform.