Json With null value

[alex-dengx]

How can I parse json from kafka that some key may have null value ? like following json , the message_key is null so that I get an error message:

ERROR transform{component_kind="transform" component_id=json_parser component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for "parse_json" at (4:25): expected string, got null" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

the json is like this:

{
    "cmdline": "forfiles  /p \"C:\\users\\administrator\\AppData\\Roaming\\Microsoft\\Signatures\\Deloitte short_CN_files\" /c \"cmd /c del /f /q @path\" /d -15/06/2016",
    "company_id": "65b8b9cb881d8861bc00bf82",
    "ctc_version": "2.13.8.4",
    "datetime": 1719931361203,
    "elevation": 1,
    "elevation_sz": "elevated",
    "event_name": "process_create",
    "event_version": 1,
    "hardware_id": "DB2BFB4C-27E9-11B2-A85C-CDF98482CE32-9C2DCD0F899E",
    "headers": {},
    "integrity": 12288,
    "integrity_sz": "high",
    "machine_name": "CNPC2KRXR6",
     "message_key": null,
    "mitre_ids": [
        {
            "categories": [
                "Execution"
            ],
            "id": "T1059",
            "name": "Command and Scripting Interpreter"
        },
        {
            "categories": [
                "Defense Evasion"
            ],
            "id": "T1218",
            "name": "System Binary Proxy Execution"
        },
        {
            "categories": [
                "Privilege Escalation",
                "Defense Evasion"
            ],
            "id": "T1548",
            "name": "Abuse Elevation Control Mechanism",
            "subtechniques": [
                {
                    "id": "T1548.002",
                    "name": "Bypass User Account Control"
                }
            ]
        }
    ],
    "offset": 3308225475,
    "os_family": "windows",
    "os_platform": "x64",
    "os_type": "client",
    "os_version": "Windows 10",
    "parent_cmdline": "C:\\WINDOWS\\system32\\cmd.exe /c \"\"C:\\WINDOWS\\ccmcache\\1k\\Install_user.cmd\"\"",
    "parent_elevation": 1,
    "parent_elevation_sz": "elevated",
    "parent_integrity": 12288,
    "parent_integrity_sz": "high",
    "parent_pid": 4552,
    "parent_process_path": "c:\\windows\\syswow64\\cmd.exe",
    "parent_user_name": "administrator",
    "partition": 29,
    "pid": 23856,
    "process_md5": "d95c443851f70f77427b3183b1619dd3",
    "process_path": "c:\\windows\\syswow64\\forfiles.exe",
    "process_sha": "7074d2a9c3d669a15d5b3a7ba1226dbba05888cc537cf055fed6371f32f0c1f5",
    "product_version": "7.9.12.418",
    "source_type": "kafka",
    "timestamp": "2024-07-02T14:41:59.944Z",
    "topic": "test",
    "user_name": "administrator",
    "user_sid": "S-1-5-21-776561741-1482476501-682003330-2268628"
}

my transform config is this but the del function can not solve this problem

transforms:
  json_parser:
    type: "remap"
    inputs: ["kafka_consumer"]
    source : |
      . = parse_json!(.message)
      del(.message_key)

[jszwedko]

Answered over here: #20770