Fluentd syslog with Ngnix

[Kevin9n]

By some reason my fluentd doesn't see any logs that has been send by Nginx

Nginx conf:

...

log_format json escape=json '{ "time": "$time_iso8601", '
    '"remote_addr": "$remote_addr", '
    '"remote_user": "$remote_user", '
    '"ssl_protocol_cipher": "$ssl_protocol/$ssl_cipher", '
    '"body_bytes_sent": "$body_bytes_sent", '
    '"request_time": "$request_time", '
    '"status": "$status", '
    '"request": "$request", '
    '"request_method": "$request_method", '
    '"http_referrer": "$http_referer", '
    '"http_x_forwarded_for": "$http_x_forwarded_for", '
    '"http_cf_ray": "$http_cf_ray", '
    '"host": "$host", '
    '"server_name": "$server_name", '
    '"upstream_address": "$upstream_addr", '
    '"upstream_status": "$upstream_status", '
    '"upstream_response_time": "$upstream_response_time", '
    '"upstream_response_length": "$upstream_response_length", '
    '"upstream_cache_status": "$upstream_cache_status", '
    '"http_user_agent": "$http_user_agent" }';


    access_log syslog:server=this_server:5140,tag=nginx_access,severity=info,facility=local7 json;
    error_log syslog:server=this_server:5140,tag=nginx_error,facility=local7;

...

Fluentd conf:

<source>
  type udp
  protocol_type udp
  port 5140
  format none
  tag nginx.local7.info
</source>

<match nginx.local7.info>
  type stdout
</match>

How i run apps:

Fluentd container

docker run -m 64m -d --restart=always -p 24224:24224 -p 24224:24224/udp -p 5140:5140/udp -v /home/user/fluentd_log:/fluentd/log --name internal-fluentd internal-fluentd:latest

Nginx container

docker run --restart always --name nginx -v /home/user/nginx-config/nginx.conf:/etc/nginx/nginx.conf:ro -m 32m -p 80:80 -p 443:443 -d nginx

*Both container on same server

STATUS         PORTS                                                                                                                                     NAMES
Up 6 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                                                                  nginx
Up 3 minutes   0.0.0.0:5140->5140/udp, :::5140->5140/udp, 0.0.0.0:24224->24224/udp, :::24224->24224/udp, 0.0.0.0:24224->24224/tcp, :::24224->24224/tcp   fluentd

if i check 5140 port by
sudo tcpdump -i any -A dst port 5140

i see nginx logs, but my Fluentd not.

What's wrong with my setup?

[daipom]

What is the version of Fluentd?

[Kevin9n]

Here is my docker logs for fluentd

docker logs fluentd
2023-06-06 14:01:45 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-06-06 14:01:45 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-06-06 14:01:45 +0000 [info]: gem 'fluentd' version '1.16.1'
2023-06-06 14:01:45 +0000 [info]: gem 'fluent-plugin-mongo' version '1.6.0'
2023-06-06 14:01:45 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type udp
    port 5140
    tag "test"
    format none
    <parse>
      @type none
    </parse>
  </source>
  <match test>
    @type stdout
  </match>
</ROOT>
2023-06-06 14:01:45 +0000 [info]: starting fluentd-1.16.1 pid=6 ruby="3.1.4"
2023-06-06 14:01:45 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]

[Kevin9n]

Also samples from

sudo tcpdump -i any -A dst port 5140

14:03:18.169081 docker0 In  IP 172.17.0.3.50177 > frontend.5140: UDP, length 712
E...;.@.@.......@.B.......2.<190>Jun  6 14:03:18 589d0c585e84 nginx_access: { "time": "2023-06-06T14:03:18+00:00", "remote_addr": "xx.xxx.xx.xxx", "remote_user": "", "ssl_protocol_cipher": "TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256", "body_bytes_sent": "3226", "request_time": "0.088", "status": "200", "request": "GET /api/test?page=0&size=4&sort=created&sortOrder=DESC HTTP/1.1", "request_method": "GET", "http_referrer": "", "http_x_forwarded_for": "", "http_cf_ray": "", "host": "testsite.com", "server_name": "testsite.com", "upstream_address": "xx.xxx.xx.xxx:8677", "upstream_status": "200", "upstream_response_time": "0.077", "upstream_response_length": "3226", "upstream_cache_status": "", "http_user_agent": "Uptime-Kuma/1.21.3" }
14:03:18.169447 docker0 Out IP frontend.38425 > 172.17.0.2.5140: UDP, length 712
E...C[@.@.................[.<190>Jun  6 14:03:18 589d0c585e84 nginx_access: { "time": "2023-06-06T14:03:18+00:00", "remote_addr": "xx.xxx.xx.xxx", "remote_user": "", "ssl_protocol_cipher": "TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256", "body_bytes_sent": "3226", "request_time": "0.088", "status": "200", "request": "GET /api/test?page=0&size=4&sort=created&sortOrder=DESC HTTP/1.1", "request_method": "GET", "http_referrer": "", "http_x_forwarded_for": "", "http_cf_ray": "", "host": "testsite.com", "server_name": "testsite.com", "upstream_address": "xx.xxx.xx.xxx:8677", "upstream_status": "200", "upstream_response_time": "0.077", "upstream_response_length": "3226", "upstream_cache_status": "", "http_user_agent": "Uptime-Kuma/1.21.3" }

[daipom]

From the logs, the data does not appear to reach Fluentd.

Also samples from

sudo tcpdump -i any -A dst port 5140

Sorry, maybe I don't understand this properly, but does this result show that the data is reaching inside the container?

[Kevin9n]

Seems problem solved. I just increased memory on server.

[daipom]

Oh, I see!
There was not the log: starting fluentd worker pid ....
(I didn't notice it, sorry.)

It would be because there was not enough memory!