Convert timezoned timestamp to UTC

[jonathanderham-streamotion]

Our application log contains timestamps and timezone data, so according to vectors doco it will not convert it to UTC. We want to convert this local timestamp to UTC time.

Our elasticsearch instance is expecting UTC time so automatically adds the necessary 10/11 hours onto the timestamp, and the machine is set with the local timezone due to application restrictions. We are trying to convince the developers to log the times in UTC time but this seems to be an uphill battle.

Here is an example of a log entry with a timezone:
[2022-10-07T13:54:27,699+1100] WARN [class] - <log message>

I would like this timezone to appear in the vector output as UTC time so that in Elasticsearch, the messages line up with the correct timestamps.

I have tried to use the parse_timestamp and to_timestamp functions with no success, and have also found the Timestamp format option in the global 'enrichment_tables.file.schema' property, but due to the lack of documentation around this, have not been able to get it working - I don't even know what it would look like in the /etc/vector/vector.toml file. I understand the documentation is automatically generated, but the 'examples' just show the possible values, not how the properties are used, so they don't really help.

When attempting to use parse_timestamp, I did the following:
.timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S,%f%z")
This did not work, even though the format string worked fine in a python datestamp.strptime() function - and even if it did parse the date properly it it would only be the first step anyway as I still need to convert it.

We also tried updating the app logs to remove the timezone offset all together, so now it looks like this:
[2022-10-07T13:54:27,699] WARN [class] - <log message>

According to the doco, this should mean that it detects the timestamp as local time, and behind the scenes converts it to UTC, then passes it to Elasticsearch as UTC, which should work for us. This however did not work, and we continue to see times in Elasticsearch that are 10 hours out of sync.

Can someone please help me work out what is going wrong here? Thanks

[jonathanderham-streamotion]

We managed to work out what was wrong - our parsing regex looked like this:
. = parse_regex!(.message,r'(?s)\[(?P<timestamp>\d+-\d+-\d+T\d+:\d+:\d+).\d+\+\d+\].(?P<severity>\w+)\s+\[(?P<class>(?:.*))\]\s-\s(?P<message>(?:.*))')
Which meant that the timestamp values being grouped into the '.timestamp' grouping did not include either the milliseconds or the timezone information.
By changing the timezone grouping end bracket to just before the end square bracket:
. = parse_regex!(.message,r'(?s)\[(?P<timestamp>\d+-\d+-\d+T\d+:\d+:\d+.\d+\+\d+)\].(?P<severity>\w+)\s+\[(?P<class>(?:.*))\]\s-\s(?P<message>(?:.*))')
it now includes the timezone information, and everything now works as we want it to, with the correct timestamp in ElasticSearch attached to the correct log entries.