How to filed merge

[uglyliu]

My sourcelog is as follows:

{
    "@timestamp":"2023-05-18T12:23:41.983Z",
    "@metadata":{
        "beat":"filebeat",
        "type":"_doc",
        "version":"7.10.0"
    },
    "ecs":{
        "version":"1.6.0"
    },
    "host":{
        "name":"VM-20-8-centos"
    },
    "agent":{
        "type":"filebeat",
        "version":"7.10.0",
        "hostname":"VM-20-8-centos",
        "ephemeral_id":"99fb2668-cf36-4889-a533-9ff4ce7acc63",
        "id":"e0941af7-9260-46ea-8486-7740c5a2ddba",
        "name":"VM-20-8-centos"
    },
    "log":{
        "offset":81,
        "file":{
            "path":"/etc/vector/java.log"
        }
    },
    "message":"2023-05-18 20:21:13.144 httpLog log log2",
    "tags":[
        "aliyun-vector"
    ],
    "input":{
        "type":"log"
    },
    "fields":{
        "kafka_topic":"filter-java"
    }
}

After I use parse_grok to cut the message field, I also want to get the value in the tags field，My transform config is not work,the value of mtags is null

[transforms.parse_remap]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = false
source        = """
           . = parse_json!(.message)
           if contains(string!(.message), "httpLog") {
               . = parse_grok!(string!(.message),"%{TIMESTAMP_ISO8601:datestamp} %{GREEDYDATA:extra}")
             .@timestamp = parse_timestamp!(string!(.datestamp), format: "%F %T%.3f")
             .timestamp = now()
             .unix_time,error = to_int(format_timestamp!(.timestamp, format: "%s"))
             .utc8_time = .unix_time + 28800
             .currentdate = format_timestamp!(to_timestamp!(.utc8_time), format: "%Y.%m.%d")
             .mtags = .tags[0]
             del(.timestamp)
           } else {
                abort
           }
"""

[spencergilbert]

You have a couple of steps that assign the result directly to . which would overwrite the entire event with the parsed result, when that's the case you're likely dropping the .tags path.

[uglyliu]

You have a couple of steps that assign the result directly to . which would overwrite the entire event with the parsed result, when that's the case you're likely dropping the .tags path.

How should the rules be written in this scenario，thank u very much

[uglyliu]

merge

Can you please provide some configuration snippets, I don't understand the usage of merge assign，thank u very much

[spencergilbert]

Have you checked out our examples page?

If you can't get a working program from the examples, I may have time tomorrow to write something out before I go on vacation.

[uglyliu]

examples

Sorry to interrupt your vacation, I'll try again

[spencergilbert]

🙂 vacation next week, I can try and squeeze it in before I go if you're having problems there. Once you do get it, feel free to provide feedback on how we can better explain/additional examples to make it easier to understand.

[uglyliu]

🙂 vacation next week, I can try and squeeze it in before I go if you're having problems there. Once you do get it, feel free to provide feedback on how we can better explain/additional examples to make it easier to understand.

I try to so many times ,it is not work

[transforms.parse_remap]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = false
source        = """
             parsed, err = parse_json(.message)
             . |= object!(parsed)
             .tags = .tags[0]
             . = parse_grok!(string!(.message),"%{TIMESTAMP_ISO8601:datestamp} %{GREEDYDATA:extra}")
             .@timestamp = parse_timestamp!(.datestamp, format: "%F %T%.3f")
             .timestamp = now()
             .unix_time,error = to_int(format_timestamp!(.timestamp, format: "%s"))
             .utc8_time = .unix_time + 28800
             .currentdate = format_timestamp!(to_timestamp!(.utc8_time), format: "%Y.%m.%d")
             del(.timestamp)
"""

can you help me if you are free，thank u very much

[uglyliu]

[transforms.parse_remap]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = false
source        = """
             . = parse_json!(.message)
             . |= parse_grok!(.message,"%{TIMESTAMP_ISO8601:datestamp} %{GREEDYDATA:extra}")
             .@timestamp = parse_timestamp!(.datestamp, format: "%F %T%.3f")
             .timestamp = now()
             .unix_time,error = to_int(format_timestamp!(.timestamp, format: "%s"))
             .utc8_time = .unix_time + 28800
             .currentdate = format_timestamp!(to_timestamp!(.utc8_time), format: "%Y.%m.%d")
             del(.timestamp)
"""

the config is bad ,the error is "this expression must resolve to an object"

[uglyliu]

the following config is good,I have never understood that my log always needs to add this line,. = parse_json!(.message) if not, I can’t achieve the effect I want,the bad result like this photo

But I add this line . = parse_json!(.message), like following,it's can achieve the effect I want

[transforms.parse_remap]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = false
source        = """
             . = parse_json!(.message)
             .message = parse_grok!(.message,"%{TIMESTAMP_ISO8601:datestamp} %{GREEDYDATA:extra}")
             .tags = .tags[0]
             .@timestamp = parse_timestamp!(.message.datestamp, format: "%F %T%.3f")
             .timestamp = now()
             .unix_time,error = to_int(format_timestamp!(.timestamp, format: "%s"))
             .utc8_time = .unix_time + 28800
             .currentdate = format_timestamp!(to_timestamp!(.utc8_time), format: "%Y.%m.%d")
             .datestamp = del(.message.datestamp)
             .extra = del(.message.extra)
             del(.timestamp);del(.ecs);del(.agent);del(.log);del(.fields);del(.input);del(.host);del(.unix_time);del(.utc8_time);del(.@metadata)
"""

the good result is like this photo

[spencergilbert]

Great! It seems like the original log was/is more wrapped than I expected, but I'm glad you got the program working.