How to remove quotes and backslashes in message field?

[soda-pop-ice-cream]

Hi there!
I tried to setup everything as in quick start guide(FluentbitAgent as log collector and SyslogNGFlow & SyslogNGOutput) with Splunk HEC as destination. I receiving logs:

{"ts":"2025-04-16T13:54:40.175887Z","time":"2025-04-16T16:54:40.175887931+01:00","stream":"stdout","logtag":"F","message":"{\"log_source\":\"nginx-unit\",\"remote_addr\":\"172.19.112.49\",\"time_local\":\"16/Apr/2025:16:54:40 +0100\",\"method\":\"PATCH\",\"request_uri\":\"/api/dcim/sites/28287/\",\"status\":\"200\",\"body_bytes_sent\":\"581\",\"request_time\":\"0.152\",\"header_user_agent\":\"python-urllib3/2.4.0\"}","kubernetes":{"pod_name":"netbox-dev-5654d7f9cf-hjmrg","namespace_name":"netbox-dev","pod_id":"56e93116-4d8b-43ce-9e47-ad68e2a9cf34","labels":{"[app.kubernetes.io/instance":"netbox-dev","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"netbox","app.kubernetes.io/version":"4b57b164","helm.sh/chart":"netbox-0.1.7","pod-template-hash":"5654d7f9cf"},"annotations":{"cni.projectcalico.org/containerID":"a111a777d36daaf407f1b41c1a2643251edde343febbc8b21bdec6f10e8291e8","cni.projectcalico.org/podIP":"10.233.109.175/32","cni.projectcalico.org/podIPs":"10.233.109.175/32","kubectl.kubernetes.io/restartedAt":"2025-03-21T15:56:26+07:00"},"host":"test-cluster","container_name":"netbox-dev","docker_id":"5865acc45ddda342e606df1ed8e7401519db75661a9953481e38b33349bfdf09","container_hash":"registry.lan/netbox/netbox-dev@sha256:d39dd6aa2ed9b127376dec1bd58bcd312a2b5000a6098db1db1b2d2bb99da7b2","container_image":"registy.lan/netbox/netbox-dev:4b57b164"},"kubernetes_namespace":{"name":"netbox-dev","labels":{"kubernetes.io/metadata.name":"netbox-dev](http://app.kubernetes.io/instance%22:%22netbox-dev%22,%22app.kubernetes.io/managed-by%22:%22Helm%22,%22app.kubernetes.io/name%22:%22netbox%22,%22app.kubernetes.io/version%22:%224b57b164%22,%22helm.sh/chart%22:%22netbox-0.1.7%22,%22pod-template-hash%22:%225654d7f9cf%22%7D,%22annotations%22:%7B%22cni.projectcalico.org/containerID%22:%22a111a777d36daaf407f1b41c1a2643251edde343febbc8b21bdec6f10e8291e8%22,%22cni.projectcalico.org/podIP%22:%2210.233.109.175/32%22,%22cni.projectcalico.org/podIPs%22:%2210.233.109.175/32%22,%22kubectl.kubernetes.io/restartedAt%22:%222025-03-21T15:56:26+07:00%22%7D,%22host%22:%22test-cluster%22,%22container_name%22:%22netbox-dev%22,%22docker_id%22:%225865acc45ddda342e606df1ed8e7401519db75661a9953481e38b33349bfdf09%22,%22container_hash%22:%22registry.lan/netbox/netbox-dev@sha256:d39dd6aa2ed9b127376dec1bd58bcd312a2b5000a6098db1db1b2d2bb99da7b2%22,%22container_image%22:%22registry.lan/netbox/netbox-dev:4b57b164%22%7D,%22kubernetes_namespace%22:%7B%22name%22:%22netbox-dev%22,%22labels%22:%7B%22kubernetes.io/metadata.name%22:%22netbox-dev)"}}}

(or nicely formatted)

{
  "ts": "2025-04-16T13:54:40.175887Z",
  "time": "2025-04-16T16:54:40.175887931+01:00",
  "stream": "stdout",
  "logtag": "F",
  "message": "{\"log_source\":\"nginx-unit\",\"remote_addr\":\"172.19.112.49\",\"time_local\":\"16/Apr/2025:16:54:40 +0100\",\"method\":\"PATCH\",\"request_uri\":\"/api/dcim/sites/28287/\",\"status\":\"200\",\"body_bytes_sent\":\"581\",\"request_time\":\"0.152\",\"header_user_agent\":\"python-urllib3/2.4.0\"}",
  "kubernetes": {
    "pod_name": "netbox-dev-5654d7f9cf-hjmrg",
    "namespace_name": "netbox-dev",
    "pod_id": "56e93116-4d8b-43ce-9e47-ad68e2a9cf34",
    "labels": {
      "[app.kubernetes.io/instance": "netbox-dev",
      "app.kubernetes.io/managed-by": "Helm",
      "app.kubernetes.io/name": "netbox",
      "app.kubernetes.io/version": "4b57b164",
      "helm.sh/chart": "netbox-0.1.7",
      "pod-template-hash": "5654d7f9cf"
    },
    "annotations": {
      "cni.projectcalico.org/containerID": "a111a777d36daaf407f1b41c1a2643251edde343febbc8b21bdec6f10e8291e8",
      "cni.projectcalico.org/podIP": "10.233.109.175/32",
      "cni.projectcalico.org/podIPs": "10.233.109.175/32",
      "kubectl.kubernetes.io/restartedAt": "2025-03-21T15:56:26+07:00"
    },
    "host": "test-cluster",
    "container_name": "netbox-dev",
    "docker_id": "5865acc45ddda342e606df1ed8e7401519db75661a9953481e38b33349bfdf09",
    "container_hash": "registry.lan/netbox/netbox-dev@sha256:d39dd6aa2ed9b127376dec1bd58bcd312a2b5000a6098db1db1b2d2bb99da7b2",
    "container_image": "registy.lan/netbox/netbox-dev:4b57b164"
  },
  "kubernetes_namespace": {
    "name": "netbox-dev",
    "labels": {
      "kubernetes.io/metadata.name": "netbox-dev"
    }
  }
}

But because of quotes around message field Splunk can't extract values from nested json, can I disable them somehow? Or even maybe specify my own template?
And second question, is it possible to disable quotes escaping for message field?

[pepov]

I beleive there are two ways to solve this:

• there is a flag https://kube-logging.dev/docs/whats-new/#containerd-compatibility that unfortunately turned out to be buggy and will only work with the new release which is coming in the next few days
• this is the solution that is more manual but should work to parse the json logs and avoid escaping: https://github.com/kube-logging/logging-operator/blob/master/config/samples/containerd-merge-log.yaml

[pepov]

also you can use https://docs.fluentd.org/filter/record_transformer#enable_ruby to unescape the message using ruby I beleive