Fluentd Image issue

[shilpigoelVB]

Describe the bug

Encountering the Fluentd error as attached when running from Oracle cloud

To Reproduce

Executed the cmd
helm install fluentd fluent/fluentd -f fluentd_values.yaml -n logging

Hi Team,

We have started facing fluentd Issue with the error logs as attached.Below are the image tags we are using:-
image:
repository: "fluent/fluentd-kubernetes-daemonset"
pullPolicy: "IfNotPresent"
tag: ""

NOTE:- We don’t have any AWS setup within our environment. The fluentd image was working successfully previously , we are encountering this issue in every environment (DEV,UAT, pre-prod etc.)

Expected behavior

Fluentd container should be running and not in crashloopback state as below
kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
fluentd-b946cf888-2plhd 0/1 CrashLoopBackOff 14 (3m12s ago) 45m

Your Environment

- Fluentd version:fluentd-0.3.9 
- TD Agent version:
- Operating system: Linux
- Kernel version:
helm list -n logging
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/shilpi_goe/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/shilpi_goe/.kube/config
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
fluentd logging         1               2023-04-18 14:10:10.415032717 +0000 UTC deployed        fluentd-0.3.9   v1.14.6

Your Configuration

nameOverride: ""
fullnameOverride: ""

# DaemonSet, Deployment or StatefulSet
kind: "Deployment"

# # Only applicable for Deployment or StatefulSet
# replicaCount: 1

image:
  repository: "fluent/fluentd-kubernetes-daemonset"
  pullPolicy: "IfNotPresent"
  tag: ""

## Optional array of imagePullSecrets containing private registry credentials
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
imagePullSecrets: []

serviceAccount:
  create: true
  annotations: {}
  name: null

rbac:
  create: true

# Configure podsecuritypolicy
# Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
podSecurityPolicy:
  enabled: true
  annotations: {}

## Security Context policies for controller pods
## See https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/ for
## notes on enabling and using sysctls
##
podSecurityContext: {}
  # seLinuxOptions:
  #   type: "spc_t"

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

# Configure the livecycle
# Ref: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
lifecycle: {}
  # preStop:
  #   exec:
  #     command: ["/bin/sh", "-c", "sleep 20"]

# Configure the livenessProbe
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
livenessProbe:
  httpGet:
    path: /metrics
    port: metrics
  initialDelaySeconds: 3
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
  # initialDelaySeconds: 0
  # periodSeconds: 10
  # timeoutSeconds: 1
  # successThreshold: 1
  # failureThreshold: 3

# Configure the readinessProbe
# Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
readinessProbe:
  httpGet:
    path: /metrics
    port: metrics
  initialDelaySeconds: 3
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
  # initialDelaySeconds: 0
  # periodSeconds: 10
  # timeoutSeconds: 1
  # successThreshold: 1
  # failureThreshold: 3

resources: {}
  # requests:
  #   cpu: 10m
  #   memory: 128Mi
  # limits:
  #   memory: 128Mi

## only available if kind is Deployment
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80
  ## see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics
  customRules: []
    # - type: Pods
    #   pods:
    #     metric:
    #       name: packets-per-second
    #     target:
    #       type: AverageValue
    #       averageValue: 1k
  ## see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
  # behavior:
  #   scaleDown:
  #     policies:
  #       - type: Pods
  #         value: 4
  #         periodSeconds: 60
  #       - type: Percent
  #         value: 10
  #         periodSeconds: 60

# priorityClassName: "system-node-critical"

nodeSelector: {}

## Node tolerations for server scheduling to nodes with taints
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
##
tolerations: []
# - key: null
#   operator: Exists
#   effect: "NoSchedule"

## Affinity and anti-affinity
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}

## Annotations to be added to fluentd DaemonSet/Deployment
##
annotations: {}

## Labels to be added to fluentd DaemonSet/Deployment
##
labels: {}

## Annotations to be added to fluentd pods
##
podAnnotations: {}

## Labels to be added to fluentd pods
##
podLabels: {}

## How long (in seconds) a pods needs to be stable before progressing the deployment
##
minReadySeconds:

## How long (in seconds) a pod may take to exit (useful with lifecycle hooks to ensure lb deregistration is done)
##
terminationGracePeriodSeconds:

## Deployment strategy / DaemonSet updateStrategy
##
updateStrategy: {}
#   type: RollingUpdate
#   rollingUpdate:
#     maxUnavailable: 1

## Additional environment variables to set for fluentd pods
env:
- name: "FLUENTD_CONF"
  value: "../../../etc/fluent/fluent.conf"
  # - name: FLUENT_ELASTICSEARCH_HOST
  #   value: "elasticsearch-master"
  # - name: FLUENT_ELASTICSEARCH_PORT
  #   value: "9200"

envFrom: []

volumes:
- name: varlog
  hostPath:
    path: /var/log
- name: varlibdockercontainers
  hostPath:
    path: /var/lib/docker/containers
- name: etcfluentd-main
  configMap:
    name: fluentd-main
    defaultMode: 0777
- name: etcfluentd-config
  configMap:
    name: fluentd-config
    defaultMode: 0777

volumeMounts:
- name: varlog
  mountPath: /var/log
- name: varlibdockercontainers
  mountPath: /var/lib/docker/containers
  readOnly: true
- name: etcfluentd-main
  mountPath: /etc/fluent
- name: etcfluentd-config
  mountPath: /etc/fluent/config.d/

## Only available if kind is StatefulSet
## Fluentd persistence
##
persistence:
  enabled: false
  storageClass: ""
  accessMode: ReadWriteOnce
  size: 10Gi

## Fluentd service
##
service:
  type: "ClusterIP"
  annotations: {}
  # loadBalancerIP:
  # externalTrafficPolicy: Local
  ports:
    - name: "forwarder"
      protocol: TCP
      containerPort: 24224

## Prometheus Monitoring
##
metrics:
  serviceMonitor:
    enabled: false
    additionalLabels:
      release: prometheus-operator
    namespace: ""
    namespaceSelector: {}
    ## metric relabel configs to apply to samples before ingestion.
    ##
    metricRelabelings: []
    # - sourceLabels: [__name__]
    #   separator: ;
    #   regex: ^fluentd_output_status_buffer_(oldest|newest)_.+
    #   replacement: $1
    #   action: drop
    ## relabel configs to apply to samples after ingestion.
    ##
    relabelings: []
    # - sourceLabels: [__meta_kubernetes_pod_node_name]
    #   separator: ;
    #   regex: ^(.*)$
    #   targetLabel: nodename
    #   replacement: $1
    #   action: replace
    ## Additional serviceMonitor config
    ##
    # jobLabel: fluentd
    # scrapeInterval: 30s
    # scrapeTimeout: 5s
    # honorLabels: true

  prometheusRule:
    enabled: false
    additionalLabels: {}
    namespace: ""
    rules: []
    # - alert: FluentdDown
    #   expr: up{job="fluentd"} == 0
    #   for: 5m
    #   labels:
    #     context: fluentd
    #     severity: warning
    #   annotations:
    #     summary: "Fluentd Down"
    #     description: "{{ $labels.pod }} on {{ $labels.nodename }} is down"
    # - alert: FluentdScrapeMissing
    #   expr: absent(up{job="fluentd"} == 1)
    #   for: 15m
    #   labels:
    #     context: fluentd
    #     severity: warning
    #   annotations:
    #     summary: "Fluentd Scrape Missing"
    #     description: "Fluentd instance has disappeared from Prometheus target discovery"

## Grafana Monitoring Dashboard
##
dashboards:
  enabled: "true"
  namespace: ""
  labels:
    grafana_dashboard: '"1"'

## Fluentd list of plugins to install
##
plugins: 
# - fluent-plugin-out-http
- fluent-plugin-opensearch

## Add fluentd config files from K8s configMaps
##
configMapConfigs:
  - fluentd-prometheus-conf
# - fluentd-systemd-conf

## Fluentd configurations:
##
fileConfigs:
  01_sources.conf: |-
    <source>
      @type forward
      @label @KUBERNETES
      port 24224
      bind 0.0.0.0
    </source>
  02_filters.conf: |-
    <label @KUBERNETES>
      <match kube.var.log.containers.fluentd**>
        @type null
      </match>
      <match kube.var.log.containers.fluentbit**>
        @type null
      </match>
      <match kube.var.log.containers.**_kube-system_**>
        @type null
        @id ignore_kube_system_logs
      </match>
      <filter kube.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        skip_labels false
        skip_container_metadata false
        skip_namespace_metadata true
        skip_master_url true
      </filter>
      <match **>
        @type relabel
        @label @DISPATCH
      </match>
    </label>
  03_dispatch.conf: |-
    <label @DISPATCH>
      <filter **>
        @type prometheus
        <metric>
          name fluentd_input_status_num_records_total
          type counter
          desc The total number of incoming records
          <labels>
            tag ${tag}
            hostname ${hostname}
          </labels>
        </metric>
      </filter>
      <match **>
        @type relabel
        @label @OUTPUT
      </match>
    </label>
  04_outputs.conf: |-
    <label @OUTPUT>
      <match **>
        @type opensearch
        @id opensearch
        @log_level info
        include_tag_key true
        type_name _doc
        host <<Open search Hostname>>
        port <<Open search Port>>
        scheme https
        ssl_verify false
        ssl_version TLSv1_2
        suppress_type_name true
        request_timeout 30s
        index_name index-name
        <buffer>
          @type memory
        </buffer>
      </match>
    </label>

Your Error Log

shilpi_goe@cloudshell:~ (ap-mumbai-1)$ kubectl logs fluentd-b946cf888-2plhd -n logging
Successfully installed faraday-net_http-3.0.2
Successfully installed faraday-2.7.4
Successfully installed opensearch-transport-2.1.0
Successfully installed opensearch-api-2.2.0
Successfully installed opensearch-ruby-2.1.0
Successfully installed jmespath-1.6.2
Successfully installed aws-partitions-1.749.0
Successfully installed aws-eventstream-1.2.0
Successfully installed aws-sigv4-1.5.2
Successfully installed aws-sdk-core-3.171.0
Successfully installed faraday-excon-2.1.0
Successfully installed faraday_middleware-aws-sigv4-1.0.1
Successfully installed fluent-plugin-opensearch-1.1.0
13 gems installed
2023-04-18 14:13:33 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.5'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.0'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-04-18 14:13:33 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-04-18 14:13:34 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
/usr/local/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate faraday_middleware-aws-sigv4-1.0.1, because faraday-1.10.0 conflicts with faraday (>= 2.0, < 3) (Gem::ConflictError)
        from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/local/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'
        from /usr/local/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.1.0/lib/fluent/plugin/out_opensearch.rb:58:in `<top (required)>'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in `load'
        from /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in `<main>'
/usr/local/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate faraday_middleware-aws-sigv4-1.0.1, because faraday-1.10.0 conflicts with faraday (>= 2.0, < 3) (Gem::ConflictError)
        from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'
        from /usr/local/lib/ruby/2.7.0/rubygems.rb:211:in `try_activate'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.1.0/lib/fluent/plugin/out_opensearch.rb:58:in `<top (required)>'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in `load'
        from /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in `<main>'
/usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require': cannot load such file -- faraday_middleware/aws_sigv4 (LoadError)
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.1.0/lib/fluent/plugin/out_opensearch.rb:58:in `<top (required)>'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/label.rb:31:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `block in configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `each'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:146:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
        from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
        from /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in `load'
        from /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in `<main>'

Additional context

No response

[ashie]

It's because dependency tree is conflict.

2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
...
2023-04-18 14:13:33 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.0'

fluent-plugin-opensearch v1.1 requires faraday >= 2.0 while fluent-plugin-elasticsearch v5.1 requires faraday < 2.0.
If you don't need fluent-plugin-elasticsearch, Please use the image which doesn't include it.
I'm not sure how to do it because I don't familiar with helm, please ask someone who know it.

[shilpigoelVB]

Thanks Ashie,
We had a successful run for fluentd for past 2 months using the same configuration files.Since last week we have started facing this issue.
Our use case:-

1. Passing of Kubernetes logs to Open search.
2. We are using Fluentd & Fluentbit for the logs movement to Open Search.
3. Can you point to specific image which doesn't have the dependency of fluent-plugin-elasticsearch(not sure here, how the logs will be passed to Open Search).

[anufrievIl]

Hi, I faced the same problem.
Switched to v1.16-debian-opensearch-1 help me.