Push logs from multiple directories

[MewanthaB]

I'm running elasticsearch, fluentd, and Kibana in an EKS Kubernetes cluster. I have 3 pods in the same cluster that generate logs which are written to a volume mount, which is mounted in the fluentd pod. I have provided all 3 log paths in the fluent config file, like so:

path /first-pod/*.log,/second-pod/*.log,/third-pod/*.log

However, I only see the logs from the first-pod showing up. If reorder the path like so:

path /second-pod/*.log,/third-pod/*.log,/first-pod/*.log

I will only see the logs of second-pod.

I also tried putting everything in a single folder

/path/to/logs/
  ∟first-pod
      ∟foo.log
      ∟bar.log
  ∟second-pod
      ∟foo.log
      ∟bar.log
  ∟third-pod
      ∟foo.log
      ∟bar.log

And setting the path like this:

path /path/to/logs/**/*.log

But then it only took the logs of third-pod (likely selecting the alphabetically last).

So any ideas on how I can get logs from all directories pushed into elasticsearch?

Full fluentd conf file:

<source>
  @type tail
  @id in_tail_container_logs
  path /first-pod/*.log,/second-pod/*.log,/third-pod/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag inc.log
  exclude_path ["/var/log/containers/fluent*"]
  read_from_head true
  <parse>
    @type grok
    grok_pattern %{TIMESTAMP_ISO8601:timestamp} \| %{GREEDYDATA:jsonbody} \|
  </parse>
</source>
<match inc.log>
  @type elasticsearch
  <buffer>
    flush_thread_count 8
  </buffer>
  host elasticsearch.kube-logging.svc.cluster.local
  port 9200
  logstash_format true
</match>