recursive func for parse_json

[Fritzss]

Hello,
Case:
json: logs = {"time": "2023-02-16T15:57:07.523036823Z", "auth": {"display_name": "root", "policies": ["root"], ' \ '"policy_results": { "allowed": True, "granting_policies": [ {"name": "root", "namespace_id": "root", ' \ '"type": "acl" } ] }, "token_type": "service" }, "request": { "operation": "help", "mount_type": "ldap", ' \ '"namespace": { "id": "root" }, "path": "auth/ldap/" }}

how to get this result with VRL:
{".time": "2023-02-16T15:57:07.523036823Z", ".auth_display_name": "root", ".auth_policies_root": "root", ".auth_policy_results_allowed": "True", ".auth_policy_results_granting_policies_name": "root", ".auth_policy_results_granting_policies_namespace_id": "root", ".auth_policy_results_granting_policies_type": "acl", ".auth_token_type": "service", ".request_operation": "help", ".request_mount_type": "ldap", ".request_namespace_id": "root", ".request_path": "auth/ldap/"}
with python:

from typing import Dict, List
import json

logs = {"time": "2023-02-16T15:57:07.523036823Z", "auth": {"display_name": "root",  "policies": ["root"], ' \
       '"policy_results": {"allowed": True, "granting_policies": [{"name": "root", "namespace_id": "root", ' \
       '"type": "acl"}]}, "token_type": "service"}, "request": {"operation": "help", "mount_type": "ldap", ' \
       '"namespace": {"id": "root"}, "path": "auth/ldap/"}}


path = '.'
keys = {}


def get_dict(json, path):
    if isinstance(json, Dict):
        for key in json:
            p_ath = f'{path}_{str(key).strip()}'
            if isinstance(json.get(key), Dict):
                get_dict(json.get(key), p_ath)
            elif isinstance(json.get(key), List):
                get_list(json.get(key), p_ath)
            else:
                keys.update([(p_ath.replace("._", "."), str(json.get(key)).strip())])
        return keys


def get_list(json, path):
    if isinstance(json, list):
        for lkey in json:
            p_ath = f'{path}_{str(lkey).strip()}'
            if isinstance(lkey, Dict):
                get_dict(lkey, str(path).strip())
            elif isinstance(lkey, List):
                get_list(lkey, p_ath)
            else:
                keys.update([(path.replace("._", "."), str(lkey).strip())])
        return keys


foo = (get_dict(logs, path))

print(json.dumps(foo))

responce:

{".time": "2023-02-16T15:57:07.523036823Z", ".auth_display_name": "root", ".auth_policies_root": "root", ".auth_policy_results_allowed": "True", ".auth_policy_results_granting_policies_name": "root", ".auth_policy_results_granting_policies_namespace_id": "root", ".auth_policy_results_granting_policies_type": "acl", ".auth_token_type": "service", ".request_operation": "help", ".request_mount_type": "ldap", ".request_namespace_id": "root", ".request_path": "auth/ldap/"}

[Sjoertjuh]

Have a look at the flatten vrl function. It should be as easy as calling that function with your object.

[StephenWakely]

Yes, flatten should do the trick. Flatten will name the flattened fields "parent.child". If you need the naming scheme you are suggesting in the question, you will need to run map_keys on the flattened result to update the key names.

[Fritzss]

Thank you

[Fritzss]

it`s Ok

[Fritzss]

but there is an incident
input data:

{
 "time": "2023-02-16T15:57:07.523036823Z",
 "auth": {
         "display_name": "root",
         "policies": ["root"],
         "policy_results": {
                "allowed": True,
                "granting_policies": [
                                       {
                                        "name": "root",
                                        "namespace_id": "root",
                                        "type": "acl"
                                       }
                                      ]
                                  },
          "token_type": "service"
         },
 "request": {
             "operation": "help",
             "mount_type": "ldap",
             "namespace": {
                          "id": "root"
                          },
            "path": "auth/ldap/"
            }
}

my script return:

{
 ".time": "2023-02-16T15:57:07.523036823Z",
 ".auth_display_name": "root",
 ".auth_policies": "root",
 ".auth_policy_results_allowed": "True",
 ".auth_policy_results_granting_policies_name": "root",
 ".auth_policy_results_granting_policies_namespace_id": "root",
 ".auth_policy_results_granting_policies_type": "acl",
 ".auth_token_type": "service",
 ".request_operation": "help",
 ".request_mount_type": "ldap",
 ".request_namespace_id": "root",
 ".request_path": "auth/ldap/"
}

flatten return:

"auth_policy_results_allowed": true,
"auth_policy_results_granting_policies": [
    {
      "name": "default",
      "namespace_id": "root",
      "type": "acl"
    }
  ],
  "auth_token_issue_time": "2023-03-21T19:24:55+03:00",
  "auth_token_policies": [
    "root"
  ]

flatten do not parse list

[jszwedko]

That's correct, flatten doesn't flatten arrays. https://github.com/vectordotdev/vector/issues/8882 is tracking this enhancement. You can follow along there.