why can't find new file

[summerance]

when my include is
include = "/onlinelogs/Prod/2022/09/19/00/requests*.gz"
"INFO:Found new file to watch"appears .But wen
include = "/onlinelogs/Prod/2022/09/19/**/requests*.gz"
There is no INFO about found new file.
There are 600 logs matched the 2022/09/19/**/requests*.gz. Each gz file with a size of 1GB. Is it because there are too many gz files under this folder, so new files cannot be found?
my config is

data_dir = "/local/vectorLog"
[sources.gz_logs]
type = "file"
max_line_bytes = 102400000
ignore_older_secs = 60
include =[  "/onlinelogs/Prod/2022/09/19/**/requests*.gz"]
......

very urgent!Looking forward for reply!

[jszwedko]

Hi @summerance ,

Is it not finding any files? Or just not new files? I did a quick test locally and Vector didn't seem to have any issues picking up 2000 files. You could also check your ulimits for open file descriptors though, if that is the issue, I believe you would see error logs in Vector.

[summerance]

Hi @summerance ,

Is it not finding any files? Or just not new files? I did a quick test locally and Vector didn't seem to have any issues picking up 2000 files. You could also check your ulimits for open file descriptors though, if that is the issue, I believe you would see error logs in Vector.

Now we tried a folder with 17000 files, 3GB each file, no INFO about "file_server: vector::internal_events::file::source: Found new file to watch".But using a folder with 600 files it can find new file.
my config is:

data_dir = "/local/vectorLog"
[sources.gz_logs]
type = "file"
max_line_bytes = 102400000
include =[  "/onlinelogs/.../2022/09/**/request*.gz"]
multiline.mode = "halt_with"
multiline.condition_pattern = "-----------------------------------------------------------------------"
multiline.timeout_ms = 1000
multiline.start_pattern = '^\{'
# Parse gz_logs
[transforms.parse_logs]
type = "remap"
inputs = ["gz_logs"]
source = """
lines,err = split(.message, "\n")
.message = lines[0] + lines[1]
.message = to_string(.message)
.message = parse_json!(.message)
"""
# Print parsed logs to stdout
[sinks.print]
type = "kafka"
inputs = [ "parse_logs" ]
bootstrap_servers = "10.88.74.41:9092,10.89.215.23:9092,10.161.184.69:9092"
topic = "ESOFS"
compression = "none"
 [sinks.print.encoding]
  codec = "json"

Vector version is 0.23.3

[spencergilbert]

I'm going to close this discussion as it appears to be a duplicate of #14592