Missing logs in S3 bucket

[HelenBaoFinnAI]

What is a problem?

I tried to copy logs and output them to both stdout and s3. I can find all logs in stdout and some of them are missing in S3 bucket.

Describe the configuration of Fluentd

<match ai.finn.redacted.analytics.web>
 @type copy
 <store>
   @type stdout
 </store>
 <store>
   @type s3
   @log_level trace
   warn_for_delay 5s
   s3_bucket "#{ENV['S3_ANALYTICS_BUCKET']}"
   s3_region "#{ENV['AWS_REGION']}"
   path web/%Y-%m/%d/
   <buffer time, tag, $.kubernetes.container_name>
     @type file
     path /var/log/fluent/s3/web
     timekey 10
     timekey_zone America/Vancouver
     ## Retrying control
     retry_type exponential_backoff
     retry_forever true
     retry_max_interval 30
     ## buffering params
     chunk_limit_records 5000
     chunk_full_threshold 0.85
     ## flushing params
     flush_thread_count 8
     flush_interval 5s
     flush_mode interval
     overflow_action throw_exception
   </buffer>
 </store>
</match>

Describe the logs of Fluentd

example of missing logs:
2024-05-02T04:37:28+00:00 ai.finn.redacted.analytics.web {"hostname":"web-66fc4f9cc9-zt6zv","logId":"a0679f67-d29f-472e-9169-5ba4eebb1b3f","botInstanceData":{"versions":{"core-api":"7.3.0"},"customer":"abc-bank"},"kubernetes":{"container_image":"648518523462.dkr.ecr.us-east-1.amazonaws.com/finnai/web:stg-7.3.0","pod_name":"web-66fc4f9cc9-zt6zv","namespace_name":"abc-bank-stg","container_name":"web"},"engagementId":"87eebca8-a9da-4368-ad8b-fd68d95ea581","event":"CUI Message From User","properties":{"userGoal":"service-all-find","entities":{"@location":"Northeast Harbor Maine United States XX662","locationType":"atm"},"cuiUserMessageContent":{"text":"my zip code is XX662"},"responseId":"i.fai.branch-atm.locate","contentId":"XX930","nluIntentId":"i.fai.branch-atm.locate","requiresAuth":false,"cuiUserMessageType":"text"},"time_docker_log":"2024-05-02T04:37:28.884465104Z","conversationData":{"currentState":"final"},"userMessageId":"92678b9a-bbba-4693-a18f-baae12b20266","clientData":{"uniquePlatformUserId":"87eebca8-a9da-4368-ad8b-fd68d95ea581","source":"","operatingPlatform":"glia","languageCode":"en-US"},"name":"core-api","userData":{"loggedInState":"neverLoggedIn","id":"87eebca8-a9da-4368-ad8b-fd68d95ea581","connectedToHuman":false}}

Environment

- Fluentd version: v1.16-2
- TD Agent version:
- Fluent Package version: fluent-plugin-s3-1.7.2
- Docker image (tag):
- Operating system: 
- Kernel version: