Vector cannot read the host address of Kafka

[ktpktr0]

1. I read the local log file and send it to Kafka
2. I read the log from Kafka and send it to Loki

After deploying using docker compose, I found that in the first step, it can normally read the address of Kafka, but in the second step, it cannot read the address of Kafka, and always get 127.0.0.1

  vector_agent:
    image: timberio/vector:latest-alpine
    container_name: vector_agent
    ports:
      - 8383:8383
    volumes:
    - ./vector/vector_agent.toml:/etc/vector/vector.toml:ro
    environment:
    - TZ=Asia/Shanghai
    restart: always
    privileged: false

  vector_aggregate:
    image: timberio/vector:latest-alpine
    container_name: vector_aggregate
    ports:
      - 8385:8383
    volumes:
    - ./vector/vector_aggregate.toml:/etc/vector/vector.toml:ro
    environment:
    - TZ=Asia/Shanghai
    restart: always
    privileged: false

# cat vector/vector_agent.toml 
[sources.in]
type         = "file"
include      = ["/var/log/*.log"]
ignore_older = 86400   

[sinks.out]
type = "kafka"
inputs = [ "in" ]
bootstrap_servers = "kafka:9092"
key_field = "user_id"
topic = "test"
compression = "none"

  [sinks.out.encoding]
  codec = "json"

# cat vector/vector_aggregate.toml 
[sources.in]
type = "kafka"
bootstrap_servers = "kafka:9092"
key_field = "user_id"
group_id = "test"
topics = [ "test" ]

[sinks.out]
type = "loki"
inputs = [ "in" ]
endpoint = "http://loki:3100/loki/api/v1/push"
compression = "none"

  [sinks.out.labels]
  forwarder = "vector"
  event = "{{ event_field }}"
  key = "value"
  "\"{{ event_field }}\"" = "{{ another_event_field }}"
  "pod_labels_*" = "{{ kubernetes.pod_labels }}"

  [sinks.out.encoding]
  codec = "json"

agent logs:

2022-07-20T15:02:03.767949Z ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): kafka:9092/bootstrap: Connect to ipv4#172.20.0.9:9092 failed: Connection refused (after 1ms in state CONNECT, 1 identical error(s) suppressed)    
2022-07-20T15:02:05.777883Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Meta data fetch error: BrokerTransportFailure (Local: Broker transport failure) component_kind="sink" component_type="kafka" component_id=out component_name=out
2022-07-20T15:04:43.407641Z  INFO vector: Vector has stopped.
2022-07-20T15:04:43.411215Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="out" time_remaining="59 seconds left"
2022-07-20T15:04:43.824526Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-07-20T15:04:43.824608Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2022-07-20T15:04:43.828500Z  INFO vector::topology::running: Running healthchecks.
2022-07-20T15:04:43.828662Z  INFO vector: Vector has started. debug="false" version="0.23.0" arch="x86_64" build_id="38c2435 2022-07-11"
2022-07-20T15:04:43.828707Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-07-20T15:04:43.829589Z  INFO source{component_kind="source" component_id=in component_type=file component_name=in}:file_server: file_source::checkpointer: Loaded checkpoint data.
2022-07-20T15:04:43.843942Z  INFO vector::topology::builder: Healthcheck: Passed.

aggregate logs:

2022-07-20T15:04:39.331620Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="out, in" time_remaining="59 seconds left"
2022-07-20T15:04:43.365802Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-07-20T15:04:43.365874Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2022-07-20T15:04:43.402433Z  INFO vector::topology::running: Running healthchecks.
2022-07-20T15:04:43.402570Z  INFO vector: Vector has started. debug="false" version="0.23.0" arch="x86_64" build_id="38c2435 2022-07-11"
2022-07-20T15:04:43.402586Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-07-20T15:04:43.408324Z ERROR source{component_kind="source" component_id=in component_type=kafka component_name=in}: rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): GroupCoordinator: 127.0.0.1:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)    
2022-07-20T15:04:43.461853Z  INFO vector::topology::builder: Healthcheck: Passed.
2022-07-20T15:04:43.552739Z ERROR source{component_kind="source" component_id=in component_type=kafka component_name=in}: rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): GroupCoordinator: 127.0.0.1:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)    
2022-07-20T15:05:19.014991Z ERROR source{component_kind="source" component_id=in component_type=kafka component_name=in}: rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): GroupCoordinator: 127.0.0.1:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 8 identical error(s) suppressed)    
2022-07-20T15:05:58.607699Z ERROR source{component_kind="source" component_id=in component_type=kafka component_name=in}: rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): GroupCoordinator: 127.0.0.1:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 4 identical error(s) suppressed)    
2022-07-20T15:06:38.605734Z ERROR source{component_kind="source" component_id=in component_type=kafka component_name=in}: rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): GroupCoordinator: 127.0.0.1:9092: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 4 identical error(s) suppressed) 

[jszwedko]

Hi @ktpktr0 ! This seems like a DNS issue from docker-compose. Do you mind sharing the full docker compose file including the entries for kafka?

[ktpktr0]

Thank you for your attention. I deleted all the data and installed it again. It has no previous failure now. But cloki didn't receive the log. I need to check it.

  vector_agent:
    image: timberio/vector:latest-alpine
    container_name: vector_agent
    ports:
      - 8383:8383
    volumes:
    - ./vector/vector_agent.toml:/etc/vector/vector.toml:ro
    environment:
    - TZ=Asia/Shanghai
    restart: always
    privileged: false

  vector_aggregate:
    image: timberio/vector:latest-alpine
    container_name: vector_aggregate
    ports:
      - 8385:8383
    volumes:
    - ./vector/vector_aggregate.toml:/etc/vector/vector.toml:ro
    environment:
    - TZ=Asia/Shanghai
    restart: always
    privileged: false

  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    volumes: 
        - /etc/localtime:/etc/localtime
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: 120
      KAFKA_MESSAGE_MAX_BYTES: 10000000
      KAFKA_REPLICA_FETCH_MAX_BYTES: 10000000
      KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS: 60000
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_DELETE_RETENTION_MS: 1000

[JeanMertz]

I suspect the issue here is the KAFKA_ADVERTISED_HOST_NAME configuration you've set.

Within the Kafka container itself, that hostname will resolve to 127.0.0.1 (localhost), which then gets advertised by the coordinator as the address for clients to find the broker(s). This is incorrect, since the clients need to use the IP address to which kafka points within their own container.

You can read more about how to configure Kafka in Docker Compose here:

• https://github.com/wurstmeister/kafka-docker/wiki/Connectivity
• https://stackoverflow.com/questions/35861501/kafka-in-docker-not-working

[ktpktr0]

Thank you for your help