ilogtail time是unix时间戳是否支持改为yyyy-mm-dd格式 #1196

LiuShiYa-github · 2023-10-24T07:33:12Z

LiuShiYa-github
Oct 24, 2023

背景

我正在基于Doris构建日志系统。
采用的日志架构： ilogtail ---> kafka ---> Doris ---grafana

遇到的问题

使用ilogtail 作为日志采集端，将采集到的日志输出到Kafka，并最终存入Doris。

要采集的程序日志有多种：journalctl日志、Java程序日志、业务日志、nginx、redisr等等。

我疲惫于处理这些日志格式中的时间。

考虑到未来可能加入新的服务组件，每增加一个新的日志格式，我都要特殊处理时间格式。因此我想要采用ilogtail 自带的time，将此时间存放入Doris中，作为查找日志的条件（指定时间段内的日志）。

请求帮助

{
	"contents": {
		"content": "[com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ServerHttpAgent 167 httpPost - [NACOS ConnectException httpPost] currentServerAddr: http://192.168.0.200:30848, err : Connection refused (Connection refused)",
		"time": "2023-10-23 00:00:01.484"
	},
	"tags": {
		"host.ip": "10.0.0.10",
		"host.name": "DorisLog",
		"log.file.path": "/var/log/server.log"
	},
	"time": 1698131230
}

在这个示例中我虽然取出了content中的"2023-10-23 00:00:01.484"作为time，但是我更想要实现将"time": 1698131230 转为"time":"2023-10-24 15:07:10"存入Doris。

当前配置

root@DorisLog:~/ilogtail-1.7.1# cat user_yaml_config.d/java.yaml 
enable: true
inputs:
  - Type: file_log
    LogPath: /var/log/
    FilePattern: server.log
processors:
  - Type: processor_split_log_regex
    SplitRegex: \d+-\d+-\d+\s\d+:\d+:\d+.\d+\s.*
    SplitKey: content
    PreserveOthers: true
  - Type: processor_regex
    SourceKey: content
    Regex: (\d+-\d+-\d+\s\d+:\d+:\d+.\d+) (.*)
    Keys:
      - time
      - content
flushers:
  - Type: flusher_kafka_v2
    Brokers: 
      - 127.0.0.1:9092
    Topic: javalog

messixukejia · 2023-10-24T08:15:45Z

messixukejia
Oct 24, 2023
Maintainer

可以参考下https://help.aliyun.com/zh/sls/user-guide/extract-log-time?spm=a2c4g.11186623.0.0.5d2466f3r1nJFk#section-3sq-fik-1b7

可以帮忙一起完善下开源文档：https://github.com/alibaba/ilogtail/tree/main/docs/cn/data-pipeline/processor

6 replies

LiuShiYa-github Oct 24, 2023
Author

我参考文档后有些懵，我想可能需要声明下我的环境：idc机房并非阿里云，使用的是ilogtail ：https://ilogtail-community-edition.oss-cn-shanghai.aliyuncs.com/1.7.1/ilogtail-1.7.1.linux-arm64.tar.gz

在文档：https://ilogtail.gitbook.io/ilogtail-docs/data-pipeline/processor/processor-gotime 中没有找到答案

参考您提供的https://help.aliyun.com/zh/sls/user-guide/extract-log-time?spm=a2c4g.11186623.0.0.5d2466f3r1nJFk#section-3sq-fik-1b7后，我不知如何配置。。。因为该文档中写的是json格式的配置并且没有Unix Format的参数，我不清楚Format 应该填什么。

下面是我尝试的配置：（它并没有生效）

root@DorisLog:~/ilogtail-1.7.1# cat user_yaml_config.d/java.yaml 
enable: true
inputs:
  - Type: file_log
    LogPath: /var/log/
    FilePattern: server.log
processors:
  - Type: processor_split_log_regex
    SplitRegex: \d+-\d+-\d+\s\d+:\d+:\d+.\d+\s.*
    SplitKey: content
    PreserveOthers: true
  - Type: processor_regex
    SourceKey: content
    Regex: (\d+-\d+-\d+\s\d+:\d+:\d+.\d+) (.*)
    Keys:
      - logtime
      - message
  - Type: processor_strptime
    SourceKey: time
    Format: "%s"
    DestFormat: "%Y-%m-%d %H:%M:%S"
    EnablePreciseTimestamp: true
flushers:
  - Type: flusher_kafka_v2
    Brokers: 
      - 127.0.0.1:9092
    Topic: javalog

请您帮忙指导下这一块的配置:

  - Type: processor_strptime
    SourceKey: time
    Format: "%s"
    DestFormat: "%Y-%m-%d %H:%M:%S"
    EnablePreciseTimestamp: true

期望效果是：“time”： 1698131230 转为“time”：“2023-10-24 15：07：10”

messixukejia Oct 25, 2023
Maintainer

原始日志的内容，请提供下

LiuShiYa-github Oct 25, 2023
Author

原始日志内容如下：

2023-10-23 00:00:01.484 [com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ServerHttpAgent 167 httpPost - [NACOS ConnectException httpPost] currentServerAddr: http://192.168.0.200:30848, err : Connection refused (Connection refused)
2023-10-23 00:00:01.487 [com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ServerHttpAgent 167 httpPost - [NACOS ConnectException httpPost] currentServerAddr: http://192.168.0.200:30848, err : Connection refused (Connection refused)
2023-10-23 00:00:01.488 [com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ServerHttpAgent 167 httpPost - [NACOS ConnectException httpPost] currentServerAddr: http://192.168.0.200:30848, err : Connection refused (Connection refused)
2023-10-23 00:00:01.488 [com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ServerHttpAgent 167 httpPost - [NACOS ConnectException httpPost] currentServerAddr: http://192.168.0.200:30848, err : Connection refused (Connection refused)
2023-10-23 00:00:01.488 [com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ClientWorker 478 checkUpdateConfigStr - [fixed-192.168.0.200_30848] [check-update] get changed dataId exception java.net.ConnectException: [NACOS HTTP-POST] The maximum number of tolerable server reconnection errors has been reached
	at com.alibaba.nacos.client.config.http.ServerHttpAgent.httpPost(ServerHttpAgent.java:182) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.http.MetricsHttpAgent.httpPost(MetricsHttpAgent.java:68) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.impl.ClientWorker.checkUpdateConfigStr(ClientWorker.java:465) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.impl.ClientWorker.checkUpdateDataIds(ClientWorker.java:432) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.impl.ClientWorker$LongPollingRunnable.run(ClientWorker.java:620) ~[nacos-client-1.3.2.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_282]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]

2023-10-23 00:00:01.488 [com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ClientWorker$LongPollingRunnable 664 run - longPolling error : java.net.ConnectException: [NACOS HTTP-POST] The maximum number of tolerable server reconnection errors has been reached
	at com.alibaba.nacos.client.config.http.ServerHttpAgent.httpPost(ServerHttpAgent.java:182) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.http.MetricsHttpAgent.httpPost(MetricsHttpAgent.java:68) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.impl.ClientWorker.checkUpdateConfigStr(ClientWorker.java:465) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.impl.ClientWorker.checkUpdateDataIds(ClientWorker.java:432) ~[nacos-client-1.3.2.jar!/:?]
	at com.alibaba.nacos.client.config.impl.ClientWorker$LongPollingRunnable.run(ClientWorker.java:620) ~[nacos-client-1.3.2.jar!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_282]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_282]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]

现在的配置文件

root@DorisLog:~/ilogtail-1.7.1# cat  user_yaml_config.d/java.yaml 
enable: true
inputs:
  - Type: file_log
    LogPath: /var/log/
    FilePattern: server.log
processors:
  - Type: processor_split_log_regex
    SplitRegex: \d+-\d+-\d+\s\d+:\d+:\d+.\d+\s.*
    SplitKey: content
    PreserveOthers: true
  - Type: processor_regex
    SourceKey: content
    Regex: (\d+-\d+-\d+\s\d+:\d+:\d+.\d+) (.*)
    Keys:
      - logtime
      - message
  - Type: processor_strptime
    SourceKey: time
    Format: "%s"
    DestFormat: "%Y-%m-%d %H:%M:%S"
    EnablePreciseTimestamp: true
flushers:
  - Type: flusher_kafka_v2
    Brokers: 
      - 127.0.0.1:9092
    Topic: javalog

现在输出到Kafka的内容：

{
	"contents": {
		"logtime": "2023-10-23 00:00:01.484",
		"message": "[com.alibaba.nacos.client.Worker.longPolling.fixed-192.168.0.200_30848] ERROR ServerHttpAgent 167 httpPost - [NACOS ConnectException httpPost] currentServerAddr: http://192.168.0.200:30848, err : Connection refused (Connection refused)"
	},
	"tags": {
		"host.ip": "10.0.0.10",
		"host.name": "DorisLog",
		"log.file.path": "/var/log/server.log"
	},
	"time": 1698140513
}

shalousun Oct 29, 2023
Collaborator

time如果要写入也需要到毫秒级才能出现查询不太会日志错乱。还有time是采集时间，并不等于业务日志产生日志

LiuShiYa-github Oct 29, 2023
Author

正如您所说 time是采集时间，我们已经意识到了这一点，但是我们仍然想要采用time作为时间存放到Doris中（因为日志格式的原因我们很难做到统一，并且有些日志中没有时间）请问如何实现：“time”： 1698131230 转为“time”：“2023-10-24 15：07：10”

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ilogtail time是unix时间戳是否支持改为yyyy-mm-dd格式 #1196

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

ilogtail time是unix时间戳 是否支持改为yyyy-mm-dd格式 #1196

Uh oh!

LiuShiYa-github Oct 24, 2023

背景

遇到的问题

请求帮助

当前配置

Replies: 1 comment · 6 replies

Uh oh!

messixukejia Oct 24, 2023 Maintainer

Uh oh!

LiuShiYa-github Oct 24, 2023 Author

Uh oh!

messixukejia Oct 25, 2023 Maintainer

Uh oh!

LiuShiYa-github Oct 25, 2023 Author

Uh oh!

shalousun Oct 29, 2023 Collaborator

Uh oh!

LiuShiYa-github Oct 29, 2023 Author

ilogtail time是unix时间戳是否支持改为yyyy-mm-dd格式 #1196

LiuShiYa-github
Oct 24, 2023

Replies: 1 comment 6 replies

messixukejia
Oct 24, 2023
Maintainer

LiuShiYa-github Oct 24, 2023
Author

messixukejia Oct 25, 2023
Maintainer

LiuShiYa-github Oct 25, 2023
Author

shalousun Oct 29, 2023
Collaborator

LiuShiYa-github Oct 29, 2023
Author