Parse Docker log format

[senja7r]

Hello.

I have nginx running in docker. It writes the event log from the container to stdout in docker format.

{"log":"10.5.29.8 - - [04/Jan/2023:09:45:56 +0000] \"GET /statistic HTTP/1.1\" 499 0 \"-\" \"curl/7.61.1\" \"-\" 0.088 88 0.088 \n", "stream":"stdout", "time":"2023-01-04T09:45:56.136555042Z"}

prometheus-nginxlog-exporter cannot parse this format:

Jan 04 15:45:56 drtng-2a-023 prometheus-nginxlog-exporter[23479]: {"log":"10.5.29.8 - - [04/Jan/2023:09:45:56 +0000] \"GET /statistic HTTP/1.1\" 499 0 \"-\" \"curl/7.61.1\" \"-\" 0.088 88 0.088 \n", "stream":"stdout", "time": "2023-01-04T09:45:56.136555042Z"}
Jan 04 15:45:56 drtng-2a-023 prometheus-nginxlog-exporter[23479]: error while parsing line '{"log":"10.5.29.8 - - [04/Jan/2023:09:45:56 +0000] \"GET /statistic HTTP/1.1\" 499 0 \"-\" \"curl/7.61.1" \"-\" 0.088 88 0.088 \n", "stream": "stdout", "time": "2023-01-04T09:45:56.136555042Z"}': text log parsing err: access log line '{"log":"10.5.29.8 - - [04/Jan/2023:09:45:56 +0000] \"GET /statistic HTTP/1.1\" 499 0 \"-\" \"curl/7.61.1" \"-\" 0.088 88 0.088 \n", "stream":"stdout", "time":"2023-01-04T09:45:56.136555042Z"}" does not match given format '^\{(?P<remote_addr>[^ ]*) - (?P<remote_user>[^ ]*) \[(?P<time_local>[^]]*]\] "(?P<request>[^"]*)" (?P<status>[^ ]*) (?P<body_bytes_sent>[^ ]*) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)" "(?P<http_x_forwarded_for>[^"]*)" (?P<upstream_response_time>[^ ]*) (?P<request_length>[^ ]*) (?P<request_time>[^ ]*)'

My config file:

listen {
  port = 4040
}

namespace "oz_nginx" {
  source = {
    files = [
      "/var/lib/docker/containers/503a23cfc382e1731726e3b385a50cab321e373ef5267c31705aec31a52d76f0/503a23cfc382e1731726e3b385a50cab321e373ef5267c31705aec31a52d76f0-json.log"
    ]
  }

  format = "{$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" \"$http_x_forwarded_for\" $upstream_response_time $request_length $request_time"

  print_log = true

  labels {
    app = "oz_api_nginx"
  }

  histogram_buckets = [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]

}

What is the correct format for this case?

Thanks you!

[Nereis]

Hi,

I would suggest to mount the logs in a volume that way you won't have to struggle with the mixed formatting.