Help Parsing Nginx logs on K8S (AWS EKS)

[ghost]

Hi all, I'm trying to setup FluentD on K8S (AWS EKS) to parse out Nginx Logs and send these to AWS Opensearch, but no matter what I try I get the following error:

2024-02-16 15:32:35 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '10.10.54.198 - - [16/Feb/2024:15:32:35 +0000] \"GET /1/spa-settings HTTP/1.1\" 200 6198 \"https://sitename.pre-production.domainname.com/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0\"'" location=nil tag="kubernetes.var.log.containers.appname-api-app-786864f64c-jp45q_appname-v1_nginx-f1aa6a0aeab95dfd516a317f06dbd57f262b430dcad916b8bf3b7b50cde4fe86.log" time=2024-02-16 15:32:35.006497049 +0000 record={"stream"=>"stdout", "log"=>"10.10.54.198 - - [16/Feb/2024:15:32:35 +0000] \"GET /1/spa-settings HTTP/1.1\" 200 6198 \"https://sitename.pre-production.domainname.com/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0\""}

2024-02-16 15:32:35 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '10.10.85.241 - - [16/Feb/2024:15:32:35 +0000] \"GET /1/pages/spa-treatments/en?fallback=1 HTTP/1.1\" 200 719 \"https://sitename.pre-production.domainname.com/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0\"'" location=nil tag="kubernetes.var.log.containers.appname-api-app-786864f64c-jp45q_appname-v1_nginx-f1aa6a0aeab95dfd516a317f06dbd57f262b430dcad916b8bf3b7b50cde4fe86.log" time=2024-02-16 15:32:35.107482340 +0000 record={"stream"=>"stdout", "log"=>"10.10.85.241 - - [16/Feb/2024:15:32:35 +0000] \"GET /1/pages/spa-treatments/en?fallback=1 HTTP/1.1\" 200 719 \"https://sitename.pre-production.domainname.com/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0\""}

The config for FluentD is the following:

env:
  FLUENT_ELASTICSEARCH_HOST: ${elasticsearch_host}
  FLUENT_ELASTICSEARCH_PORT: 443
  FLUENT_ELASTICSEARCH_SCHEME: https
  FLUENT_ELASTICSEARCH_SSL_VERIFY: false
  FLUENT_UID: 0

envFrom:
  - name: FLUENT_ELASTICSEARCH_USER
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: FLUENT_ELASTICSEARCH_USER
  - name: FLUENT_ELASTICSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: FLUENT_ELASTICSEARCH_PASSWORD

externalSecret:
  remoteRef:
    key: "secret/elasticsearch"

priorityClassName: highest

resources:
  limits:
    cpu: 400m
    memory: 300Mi
  requests:
    cpu: 400m
    memory: 300Mi

configMap:
  fluentdKubernetesConf: |
    <source>
      @type tail
      read_from_head true
      tag kubernetes.*
      @label @EKSLOGS
      @id in_tail_container_logs
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      exclude_path ["/var/log/containers/fluent*"]
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_type string
          time_format "%Y-%m-%dT%H:%M:%S.%NZ"
          keep_time_key false
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%NZ'
          keep_time_key false
        </pattern>
      </parse>
      emit_unmatched_lines true
    </source>
    <label @EKSLOGS>
      <filter kubernetes.var.log.containers.**nginx**>
        @type parser
        format /(?<remote_addr>[^ ]*) - - \[(?<time>[^\]]*)\] \\\"(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?\\\" (?<code>[^ ]*) (?<size>[^ ]*)(?: \\\"(?<referer>[^\"]*)\\\" \\\"(?<agent>[^\"]*)\\\")?$/
        time_format %d/%b/%Y:%H:%M:%S %z
        key_name log
      </filter>
      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
        verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
        ca_file "#{ENV['KUBERNETES_CA_FILE']}"
        skip_labels "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_LABELS'] || 'false'}"
        skip_container_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_CONTAINER_METADATA'] || 'false'}"
        skip_master_url "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_MASTER_URL'] || 'false'}"
        skip_namespace_metadata "#{ENV['FLUENT_KUBERNETES_METADATA_SKIP_NAMESPACE_METADATA'] || 'false'}"
      </filter>
      <match **>
        @type elasticsearch
        host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
        port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
        path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
        scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
        ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
        ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
        user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
        password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"
        index_name ${index_name}
        type_name ${type_name}
        include_timestamp true
        suppress_type_name true
        verify_es_version_at_startup false
        default_elasticsearch_version 8
      </match>
    </label>

The format we're using should work. I've tested it in Fluentular, and it works there:
http://fluentular.herokuapp.com/parse?regexp=%28%3F%3Cremote_addr%3E%5B%5E+%5D*%29+-+-+%5C%5B%28%3F%3Ctime%3E%5B%5E%5C%5D%5D*%29%5C%5D+%5C%5C%5C%22%28%3F%3Cmethod%3E%5CS%2B%29%28%3F%3A+%2B%28%3F%3Cpath%3E%5B%5E+%5D*%29+%2B%5CS*%29%3F%5C%5C%5C%22+%28%3F%3Ccode%3E%5B%5E+%5D*%29+%28%3F%3Csize%3E%5B%5E+%5D*%29%28%3F%3A+%5C%5C%5C%22%28%3F%3Creferer%3E%5B%5E%5C%22%5D*%29%5C%5C%22+%5C%5C%5C%22%28%3F%3Cagent%3E%5B%5E%5C%22%5D*%29%5C%5C%5C%22%29%3F%24&input=10.10.65.44+-+-+%5B16%2FFeb%2F2024%3A14%3A43%3A00+%2B0000%5D+%5C%22GET+%2Fstore%2Fpages%2Fsite%2Fsitename%2Fmodules+HTTP%2F1.1%5C%22+200+364+%5C%22https%3A%2F%2Fsitename.pre-production.domainname.com%2F%5C%22+%5C%22Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10.15%3B+rv%3A121.0%29+Gecko%2F20100101+Firefox%2F121.0%5C%22&time_format=%25d%2F%25b%2F%25Y%3A%25H%3A%25M%3A%25S+%25z

Any help is appreciated 🙏