Needed help to restructure fluentd logs

[Tamannaverma1912]

Hi all, so I have my config map like this
`apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: runai-team-b
labels:
log-app: fluentd
data:
fluent.conf: |

log_level debug

<label @FLUENT_LOG>
  <match fluent.**>
    @type null
  </match>
</label>

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  read_from_head true
  exclude_path ["/var/log/containers/*_kube-system_*.log", "/var/log/containers/*_gpu_*.log", "/var/log/containers/*_monitoring_*.log", "/var/log/containers/*_cert-manager_*.log"]
  # exclude_path ["/var/log/containers/fluent*"]
  tag kube.*
  <parse>
    @type regexp
    expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
    </parse>
</source> 

<filter kube.var.log.containers.**>
@type record_transformer
enable_ruby true
<record>
  tag ${tag}
  namespace_name  ${tag.match(/.*_([^_]+)_([^_]+)$/)[1]}
  pod_name ${tag.match(/.*containers\.([^_]+)_([^_]+)_[^_]+$/)[1]}
  container_name ${tag.match(/.*_([^_]+)$/)[1]}
    </record>
</filter>

<match kube.**>
  @type file
  path /sensei-fs/fluentd-tam/test10/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/output.log
  append true
  <buffer namespace_name,pod_name,container_name,time>
    @type file
    path /var/log/fluentd-buffers/
    flush_interval 10s
    chunk_limit_size 256m
    total_limit_size 1g
    timekey 10m
    timekey_wait 30s
    timekey_use_utc true
    overflow_action block
  </buffer>
</match> 
` 

In this I am using record_transformer to fetch namespace_name, pod_name & container_name from the tag itself as we want to have /sensei-fs/fluentd-tam/test10/${namespace_name}/${pod_name}/${container_name}/%Y-%m-%d-%H-%M-%S/output.log file structure but another requirement that we have is we would also want to remove the metadata from logs too.
Here's an example so I am getting my logs currently like but I want to remove tag, namespace_name, pod_name & container_name from it.
{"stream":"stdout","logtag":"F","log":"level=info ts=2024-05-27T06:59:08.362480623Z caller=reloader.go:237 msg=\"started watching config file and directories for changes\" cfg=/var/lib/grafana-agent/config-in/agent.yml out=/var/lib/grafana-agent/config/agent.yml dirs=","tag":"kube.var.log.containers.loki-logs-57t7z_runai-test-project_config-reloader-0582794d56ac98cdaab2335ee04730becc6b20efacbf40e94ba5f0d9b936eb9f.log","namespace_name":"runai-test-project","pod_name":"loki-logs-57t7z","container_name":"config-reloader-0582794d56ac98cdaab2335ee04730becc6b20efacbf40e94ba5f0d9b936eb9f.log"}
So I tried using remove_keys tag,namespace_name,pod_name,container_name It didn't work. Can anyone help what can we do in here?

[daipom]

So I tried using remove_keys tag,namespace_name,pod_name,container_name It didn't work.

You can't remove keys by remove_keys of filter_record_transformer for that purpose because the output plugin needs the keys for the buffer keys and the placeholders.

You can use fomatter plugins to manage the output format.
Normally, they are used with inject settings.

Example:

<match test>
  @type file
  path /foo/bar
  <buffer>
    ...
  </buffer>
  <inject>
    time_key timestamp
    time_type string
  </inject>
  <format>
    @type csv
    fields timestamp, key1, key2, key3
  </format>
</match>

I could not find a formatter that has the feature of excluding certain keys.
If you need, there may be some, or you can make your own formatter.

[jashwantraj92]

@daipom thanks for the response. We were able to create a custom formatter plugin to remove specific keys from the output file.