How to process this log with remap

[uglyliu]

My source log is as follows:

2023-05-03 13:49:54.004  INFO    87 --- [test-http-epoll-32] c.u.g.s.LogUtil                          : httpLog [#traceId=Root=2-645ww1f602-175bwwc7ac5d320f5244a600f3#][#startTime=2023-05-03 13:49:54#][#time=3#][#code=200#][#uid=#][#extraData=#][#domain=www.baidu.com#]

I have tried many times but failed, can someone help me? thank u very much

[spencergilbert]

I would suggest using parse_grok(), what have you tried so far?

What is producing that log?

[uglyliu]

I would suggest using parse_grok(), what have you tried so far?

What is producing that log?

I'll sort out the errors tomorrow. I've been replacing logstash with vector recently. That log is the log of our business. By the way，I haven't understood that .message means that there is a message field in the event or the meaning of the whole event, so I've been trying to see if the rules I wrote are correct. I've also tried use parse_grok，it's fail,I will try tommorow

[spencergilbert]

Take a look at the VRL expressions documentation on paths. In short . is the root of the event, the entire event, and .message is a top level path on the event. For example, representing an event as JSON...

{
  "message": "contents!",
  "timestamp": "2023-05-03",
  "nested": {
    "key": true
  }
}

• . is the entire object
• .message contains the value contents!
• .timestamp contains "2023-05-03"
• .nested.key contains true

[uglyliu]

Take a look at the VRL expressions documentation on paths. In short . is the root of the event, the entire event, and .message is a top level path on the event. For example, representing an event as JSON...

{

  "message": "contents!",

  "timestamp": "2023-05-03",

  "nested": {

    "key": true

  }

}

• . is the entire object

• .message contains the value contents!

• .timestamp contains "2023-05-03"

• .nested.key contains true

Thank you very much. I now understand why sometimes the rules I write don't match. There are too few Chinese documents. Will you plan to translate them into Chinese documents? The performance of this tool is much better than logstash. In China, most of them still use logstash.

[spencergilbert]

I don't think we have the plans (or the knowledge) to translate documents in the near future. @jszwedko do you think this could be something we accept contributions for, or see if Datadog can help translate - I know they provide their docs in several languages.

[jszwedko]

I think it'd be great to translate the docs, but unfortunately probably not a near term thing as it'll be a significant effort. Datadog does translate their docs into French and Chinese so there is some precedent for us being able to support Chinese translations.

Even user submitted translations would need to be validated. Would you want to open an issue for it? That'll help us gauge how many users would find it useful.

[uglyliu]

It's ok in Grok Debugger，But it is bad in vector,（Grammatical errors),please help me,thank u very much

Source Log

2023-05-11 22:49:44.799  INFO    76 --- [eactor-http-epoll-10] c.u.g.s.LogUtil                          : httpLog [#traceId=i168381658476910753476896#] [#uid=9648056217#]

Remap Transforms

[transforms.parse_remap]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = true
source        = """
            . = parse_json!(.message)
            . = parse_grok!(string!(.message), "%{TIMESTAMP_ISO8601:timestamp} %{SPACE} %{LOGLEVEL:loglevel} %{SPACE} %{INT:instance} --- \[%{DATA:thread}\] %{DATA:class} %{SPACE} : httpLog \[#traceId=%{DATA:traceId}\#\] \[#uid=%{DATA:uid}\#\]")
"""

ERRORS

[root@VM-20-8-centos vector]# vector validate vector.toml
Failed to load ["vector.toml"]
------------------------------
x TOML parse error at line 31, column 141
   |
31 |             . = parse_grok!(string!(.message), "%{TIMESTAMP_ISO8601:timestamp} %{SPACE} %{LOGLEVEL:loglevel} %{SPACE} %{INT:instance} --- \[%{DATA:thread}\] %{DATA:class} %{SPACE} : httpLog \[#traceId=%{DATA:traceId}\#\] \[#uid=%{DATA:uid}\#\]")
   |                                                                                                                                             ^
invalid escape sequence
expected `b`, `f`, `n`, `r`, `t`, `u`, `U`, `\`, `"`

[spencergilbert]

I would suggest using a string literal:

source        = """
            . = parse_json!(.message)
            . = parse_grok!(string!(.message), s'%{TIMESTAMP_ISO8601:timestamp} %{SPACE} %{LOGLEVEL:loglevel} %{SPACE} %{INT:instance} --- [%{DATA:thread}] %{DATA:class} %{SPACE} : httpLog [#traceId=%{DATA:traceId}#] [#uid=%{DATA:uid}#]')

[uglyliu]

I would suggest using a string literal:

source        = """
            . = parse_json!(.message)
            . = parse_grok!(string!(.message), s'%{TIMESTAMP_ISO8601:timestamp} %{SPACE} %{LOGLEVEL:loglevel} %{SPACE} %{INT:instance} --- [%{DATA:thread}] %{DATA:class} %{SPACE} : httpLog [#traceId=%{DATA:traceId}#] [#uid=%{DATA:uid}#]')

Thank u ,it is can be solved as follows:

[transforms.parse_remap]
type = "remap"
inputs = ["dummy_logs"]
drop_on_error = true
source        = """
            . = parse_json!(.message)
           if contains(string!(.message), "httpLog") {
             . = parse_grok!(string!(.message),"%{TIMESTAMP_ISO8601:datestamp} %{GREEDYDATA:invaliad} \\\\[#traceId=%{DATA:traceId}#\\\\]\\\\[#startTime=%{DATA:startTime}#\\\\]\\\\[#time=%{INT:time}#\\\\]\\\\[#code=%{INT:code}#\\\\]\\\\[#uid=%{DATA:uid}#\\\\]\\\\[#extraData=%{DATA:extraData}#\\\\]\\\\[#domain=%{DATA:domain}#\\\\]\\\\[#terminalType=%{INT:terminalType}#\\\\]\\\\[#appVersion=%{DATA:appVersion}#\\\\]\\\\[#clientIp=%{DATA:clientIp}#\\\\]\\\\[#callIp=%{DATA:callIP}#\\\\]\\\\[#instanceName=%{DATA:instanceName}#\\\\]\\\\[#serviceReferer=%{DATA:serviceReferer}#\\\\]\\\\[#url=%{DATA:url}#\\\\]\\\\[#request=%{DATA:request}#\\\\]\\\\[#response=%{DATA:response}#\\\\]%{GREEDYDATA:extra}")
             .@timestamp = parse_timestamp!(string!(.datestamp), format: "%F %T%.3f")
             .timestamp = now()
             .unix_time,error = to_int(format_timestamp!(.timestamp, format: "%s"))
             .utc8_time = .unix_time + 28800
             .currentdate = format_timestamp!(to_timestamp!(.utc8_time), format: "%Y.%m.%d")
             del(.timestamp)
           } else {
                abort
           }
"""