help request: [file-logger] duplicate log, new log file contains history log message

[kizy008]

Description

In my project, I need split log file by hour and delete logs 7 days ago. So I write a rotate plugin like [log-rotate.lua], and it works.
But I found that every new log file contains the history log message. For example, [log-2022-09-01_01-00-00.log] contains all message in [log-2022-09-01_00-00-00.log]]. So every log contains all history message. I think the new log should contains message in its hour period. But I got all history message.
I want to know how to get incremental logs in new log file.

我的项目中需要将日志按照小时分割并自动淘汰7天前的历史日志。 所以我仿照[log-rotate.lua]写了file-logger的滚动切割日志插件，并且按照预期生效了。但是我发现，所有的心日志文件都包含了全量的历史日志。比如 [log-2022-09-01_01-00-00.log]日志不仅包含了23点到0点的日志， 还包含了所有的历史全量日志。我本以为新日志应该只包含过去1个小时之内的日志。
我想知道如何利用file-logger在每个日志文件中保存这个小时时间段内增量的日志呢？

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[AlphaChen]

i meet same problem。can u show u code？

[monkeyDluffy6017]

1. rename old log file and create new one
2. notify other processes to reopen the file

Do you follow the two steps ?

[AlphaChen]

1. rename old log file and create new one
2. notify other processes to reopen the file

Do you follow the two steps ?

how to notify other processes

[monkeyDluffy6017]

You could use ngx.shared.dict to notify other processes.
For example

-- process 1
local rotate = ngx.shared.rotate
rotate:set("reopen", true)

-- process 2
local rotate = ngx.shared.rotate
local reopen = rotate:get("reopen")
if reopen then
    -- reopen file
end

[kizy008]

The log file in file-logger is not managed by nginx, so send signal to nginx master will not help.

Did you solve this problem by adding shared dict ?

[kizy008]

Here is the process new log file created.
Firstly the file.log rename success. Then create a new file.log, But the new log contains all history log message. I don't know how it happens.

[kizy008]

I add some log when create new log file, and I found that new log file have history log message. I don't know why. I open new log file with mode "w". It should not have any log message.

[kizy008]

Maybe it related to storage medium. I use COS (Cloud Object Storage) to store file logs. And I found that when I use hard disk or CBS , the problem did not occur. When I use COS, I rename a new log file with size 0 to former log name, the log file would be no longger empty.

我感觉这个问题可能跟底层存储介质有关。我使用COS存储日志。当我使用硬盘或云硬盘时，没有出现过这种日志累加的问题。我使用COS来存储时， 当我将一个大小为0的空文件重命名为原先的日志名称时，我发现日志文件不再为空，里面包含了所有的历史日志记录。

Perhaps， I will save history log in COS and real time log in hard disk. Then rotate real time log and save them to history log direcotry.
也许，我应该将历史日志存储在COS上，将实时日志存储在硬盘上，然后切割实时日志，将切割后的日志存储在历史日志的目录里面。

[kizy008]

This issue has been solved by write real time log on hard disk, and rotate the log then move to history log directory on COS.
There is a step need to be attentioned. Because COS and hard disk are diffrent, if we use os.rename, it will report [os.rename Invalid cross-device link]. So here I use os.execute instead.
It works well for me, I hope this solution may help you.

[tzssangglass]

Since this is your own code, not related to the project, I turned it into discussion.