logs format (when sent to udp port via syslog)

[charlienegri]

I am trying the exporter in a sidecar on kubernetes,
got the logs sent to the right port in the main container but syslog prepends a header that I am not able to remove
(syslog in my main container image is neither rsyslog nor syslog-ng, it's busybox syslogd)

# HELP nginx_parse_errors_total Total number of log file lines that could not be parsed
# TYPE nginx_parse_errors_total counter
nginx_parse_errors_total 1

which I think it's due to that...

I have set:

kind: ConfigMap
metadata:
  name: nginx-exporter-sidecar-config
data:
  config.hcl: |
    listen {
      port = 4040
    }

    namespace "nginx" {
      source = {
        syslog {
          listen_address = "udp://127.0.0.1:8513"
          format = "auto"
          tags = [
            "nginx"
          ]
        }
      }

      format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\""

      labels {
        app = "hello-world-test"
      }
    }

(tried also format = "rfc3164" there)

should I try to modify the line format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\"" in the configmap above to account for the header?

[charlienegri]

nvm, looks like my problem was not the prepended header but rather a typo in the format I set for logs on nginx

[anhdz-2k1]

Hi, I'm having the same problem as you. If you have successfully solved your problem, can you tell me how to fix it? Thanks very much!! I am using v1.9.2 version.

[charlienegri]

I made sure that the nginx server running in the main container pod did actually stream the logs to the chosen port via udp, that was failling for some reason

[johnny]

I'm facing the same issue right now. Logs don't make it to the log exporter. How did you ensure that the logs were sent out?

[charlienegri]

my main container is just a hello-world kinda nginx server (non-root and I need to expose port 8080 but that's irrelevant here), in the docker image I have

FROM nginx:mainline-alpine

# We need to give all user groups write permission on folder /var/cache/nginx/ and file /var/run/nginx.pid.
# So users with random uid will be able to run NGINX.
RUN chown -R root:nginx /etc/nginx && \
    chmod -R a+rX /etc/nginx && \
    chmod -R a+rwX /tmp

RUN rm /etc/nginx/conf.d/* 
RUN rm /etc/nginx/nginx.conf

COPY files/etc/nginx/override.conf /etc/nginx/override.conf
COPY files/etc/nginx/conf.d/helloworld.conf /etc/nginx/conf.d/helloworld.conf
COPY files/usr/share/nginx/html/index.html /usr/share/nginx/html/index.html
COPY files/logrotate/nginx /etc/logrotate.d/nginx

RUN chmod 755 /var/log/nginx && chmod 644 /var/log/nginx/*.log && chmod 644 /dev/stderr && chmod 644 /dev/stdout

EXPOSE 8080

CMD ["/usr/sbin/nginx", "-c", "/etc/nginx/override.conf"]
USER nginx:nginx

and then I have this files/etc/nginx/override.conf

daemon off;

pid       /tmp/nginx.pid;

events {   
  worker_connections  1024; 
}

error_log /dev/stderr;

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  server_tokens off;

  log_format custom '$remote_addr - $remote_user [$time_local] ' 
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" '
                    '"$http_x_forwarded_for"';

  access_log syslog:server=127.0.0.1:8513,tag=nginx,severity=info custom;

  sendfile off;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout  120;

  # These two should be the same or nginx will start writing
  #  large request bodies to temp files
  client_body_buffer_size 10m;
  client_max_body_size    10m;

  gzip  on;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_proxied any;
  gzip_vary off;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/atom+xml text/javascript application/javascript application/json text/mathml;
  gzip_min_length  1000;
  gzip_disable     "MSIE [1-6]\.";


  variables_hash_max_size 1024;
  variables_hash_bucket_size 64;
  server_names_hash_bucket_size 64;
  types_hash_max_size 2048;
  types_hash_bucket_size 64;
  map_hash_bucket_size 256;

  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;


  # Use same timeout as with old setup
  # apache v2.2 used on thredds2.met.no used default timeout 300s
  # apache v2.4 changed this to 60s - same as default proxy_read_timeout in 
  # nginx
  # Increasing timeout to 1hr due to large single-requests (2018-08-15 arildb)
  proxy_read_timeout 3600s;
  proxy_store off;
  proxy_buffering off;
  
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;

}