How to add time field

[uglyliu]

Help me please

Architecture

• Filebeat-->Kafka-->Vector-->ES

Version

• vector 0.28.1 (x86_64-unknown-linux-gnu ff15924 2023-03-06)

Content

• How to customize a time field currentdate with a value like 2023.04.30 to replace %Y.%m.%d, and then reference it in the sink

Config

[sinks.elasticsearch]
type = "elasticsearch" # required
inputs = ["parse_logs"] # required
api_version = "auto"
compression = "none" # optional, default
endpoints = ["http://127.0.0.1:9200"] # required
mode = "bulk"
bulk.action = "index"
bulk.index = "vector-java-test-%Y.%m.%d"

#healthcheck.enabled = true # optional, default

[spencergilbert]

What is the behavior of the configuration you provided? The documentation for templates and the elasticsearch sink suggest it should work.

[uglyliu]

I'm not sure how Elasticsearch creating indexes comes into play here, if it's set to auto-create indices Vector's templating of the time is based on it's timezone.

You could also convert your example Logstash config into a remap component.

Maybe the config is effect, I will look at the es index tomorrow,the timestamp is in the form of 2023-05-03 13:49:54.004

[transforms.parse_logs]
type = "remap"
inputs = ["dummy_logs"]
source        = """
     .currentdate = split(replace(.timestamp,"-",".")," ",limit: 2)[0]
"""

[sinks.elasticsearch]
 bulk.index = "vector-accessjava-{{ .currentdate }}"

[spencergilbert]

You should use to_timestamp() and either parse_timestamp() or format_timestamp() rather than trying to manipulate it as a string.

It should be less error prone.

[uglyliu]

You should use to_timestamp() and either parse_timestamp() or format_timestamp() rather than trying to manipulate it as a string.

It should be less error prone.

Thank you for your suggestion. I will try format_timestamp and parse_timestamp today. My method is currently working. Customizing the time suffix of the es index can prevent the es from creating the index 8 hours later.

[uglyliu]

You should use to_timestamp() and either parse_timestamp() or format_timestamp() rather than trying to manipulate it as a string.

It should be less error prone.

use to_timestamp and format_timestamp is done, thank u very much

.timestamp = now()
.unix_time,error = to_int(format_timestamp!(.timestamp, format: "%s"))
.utc8_time = .unix_time
.currentdate = format_timestamp!(to_timestamp!(.utc8_time), format: "%Y.%m.%d")