how can I parse json with null

[alex-dengx]

the Original json is like this :```
{
"cmdline": "forfiles /p "C:\users\administrator\AppData\Roaming\Microsoft\Signatures\Deloitte short_CN_files" /c "cmd /c del /f /q @path" /d -15/06/2016",
"company_id": "65b8b9cb881d8861bc00bf82",
"ctc_version": "2.13.8.4",
"datetime": 1719931361203,
"elevation": 1,
"elevation_sz": "elevated",
"event_name": "process_create",
"event_version": 1,
"hardware_id": "DB2BFB4C-27E9-11B2-A85C-CDF98482CE32-9C2DCD0F899E",
"headers": {},
"integrity": 12288,
"integrity_sz": "high",
"machine_name": "CNPC2KRXR6",
"message_key": null,
"mitre_ids": [
{
"categories": [
"Execution"
],
"id": "T1059",
"name": "Command and Scripting Interpreter"
},
{
"categories": [
"Defense Evasion"
],
"id": "T1218",
"name": "System Binary Proxy Execution"
},
{
"categories": [
"Privilege Escalation",
"Defense Evasion"
],
"id": "T1548",
"name": "Abuse Elevation Control Mechanism",
"subtechniques": [
{
"id": "T1548.002",
"name": "Bypass User Account Control"
}
]
}
],
"offset": 3308225475,
"os_family": "windows",
"os_platform": "x64",
"os_type": "client",
"os_version": "Windows 10",
"parent_cmdline": "C:\WINDOWS\system32\cmd.exe /c ""C:\WINDOWS\ccmcache\1k\Install_user.cmd""",
"parent_elevation": 1,
"parent_elevation_sz": "elevated",
"parent_integrity": 12288,
"parent_integrity_sz": "high",
"parent_pid": 4552,
"parent_process_path": "c:\windows\syswow64\cmd.exe",
"parent_user_name": "administrator",
"partition": 29,
"pid": 23856,
"process_md5": "d95c443851f70f77427b3183b1619dd3",
"process_path": "c:\windows\syswow64\forfiles.exe",
"process_sha": "7074d2a9c3d669a15d5b3a7ba1226dbba05888cc537cf055fed6371f32f0c1f5",
"product_version": "7.9.12.418",
"source_type": "kafka",
"timestamp": "2024-07-02T14:41:59.944Z",
"topic": "test",
"user_name": "administrator",
"user_sid": "S-1-5-21-776561741-1482476501-682003330-2268628"
}

[jszwedko]

You can use a remap transform with the https://vector.dev/docs/reference/vrl/functions/#parse_json function.

[jszwedko]

I'm not sure I follow. Can you provide some example inputs and desired outputs? Maybe you could create an example on https://playground.vrl.dev/ and share that?

[alex-dengx]

My input from Kafka is like this:

{
    "bytes_read": 4096,
    "company_id": "65b8b9cb881d8861bc00bf82",
    "ctc_version": "2.13.8.4",
    "datetime": 1719967862303,
    "entropy": 0,
    "event_name": "file_read",
    "event_version": 1,
    "extra_keys": null,
    "extra_values": null,
    "hardware_id": "9ADDD4CC-217A-11B2-A85C-F4CA8A22C9AE-9C2DCD0F83BA",
    "is_remote": 0,
    "machine_name": "CN-PC2JD8BZ",
    "os_family": "windows",
    "os_platform": "x64",
    "os_type": "client",
    "os_version": "Windows 11",
    "path": "c:\\users\\admin\\appdata\\local\\microsoft\\office\\16.0\\wef\\resources\\ol7jgfcq1kr2jzrs1fo0hw==",
    "pid": 15328,
    "process_md5": "e360fbe011e6d4a5a304f32b50ad66dd",
    "process_path": "c:\\program files\\microsoft office\\root\\office16\\outlook.exe",
    "process_sha": "951fae7f8bd49a63a76f18954ea2ae72f548de01113c0df1adbcbc23cb419702",
    "product_version": "7.9.9.370",
    "type": 999,
    "type_sz": "Unknown",
    "user_name": ""
}

In this input， extra_keys and extra_value are set as null , so I want to filter these keys and them push them to elastic . but vector will cause an error
ERROR transform{component_kind="transform" component_id=json_parser component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for "parse_json" at (4:25): expected string, got null" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
So I want to kwow is there any filter can solve this issue ?

[jszwedko]

By filter the keys, do you mean remove them? You could use the compact function which removes all keys that have a value of null. If that's not what you mean, could you share the output event you'd like?

[alex-dengx]

but in this case . if use parse_json ,it will throw an error and I can not use compact function. So the core question is, is there a way to parse json with a null value? or ignore this value and push it to next step.

[jszwedko]

Do you mean if the value being parsed is null? You could skip the parsing with something like:

if (.message != null) {
  parsed = parse_json!(.message)
}