Cannot send logs from shared drive

[rrs-llabs]

Hi Team,

Trying to send logs from shared drive but vector not sending any logs, no errors, there are many files but no files getting checkpointed , changed checkpoint strategy but no success, please help

[sources.app_logs]
type = "file"
data_dir = "/opt/vector/local/"
include = ["/gsx/shared/app_logs/ip-10-1-1-97_125614_1.out"]
file_key = "file"
glob_minimum_cooldown_ms = 1000
host_key = ""
ignore_older_secs = 600
fingerprint.strategy = "device_and_inode"
line_delimiter = "\n"
ignore_checkpoints = true

[transforms.enrich_remap]
type = "remap"
inputs = [ "app_logs" ]
drop_on_abort = true
source = """
filename_parts = split!(.file, "_" )
.hostname = filename_parts[0]
.job_id = filename_parts[1]
"""

[sinks.my_sink_id]
type = "axiom"
inputs = [ "enrich_remap" ]
dataset = "app-logs"
token = "xxxxx"

[sinks.test-console]
type = "console"
inputs = [ "app_logs" ]
target = "stdout"
encoding.codec = "json"

vector Logs

2025-01-21T21:03:12.687039Z DEBUG vector::internal_events::file::source: Files checkpointed. count=0 duration_ms=3
2025-01-21T21:03:12.715208Z DEBUG source{component_kind="source" component_id=app_logs component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 7.884506e-5, "other": 4.845607e-5, "reading": 0.00020746754, "sending": 3.4234923e-5, "sleeping": 0.99963105}

[jszwedko]

Can you share the full Vector log output? I'd like to see if it discovers and is able to read the configured file.

[rrs-llabs]

2025-01-21T23:14:49.676324Z INFO source{component_kind="source" component_id=app_logs component_type=file}:file_server: vector::internal_events::file::source: Found new file to watch. file=/gsx/shared/inference_logs/ip-10-1-0-19_125614_0.err
2025-01-21T23:14:49.676585Z INFO source{component_kind="source" component_id=app_logs component_type=file}:file_server: vector::internal_events::file::source: Found new file to watch. file=/gsx/shared/inference_logs/ip-10-1-0-19_125613_0.out
2025-01-21T23:14:49.676876Z INFO source{component_kind="source" component_id=app_logs component_type=file}:file_server: vector::internal_events::file::source: Found new file to watch. file=/gsx/shared/inference_logs/ip-10-1-0-19_125613_0.err
2025-01-21T23:14:50.682300Z DEBUG vector::internal_events::file::source: Files checkpointed. count=0 duration_ms=3
2025-01-21T23:14:50.711461Z DEBUG source{component_kind="source" component_id=app_logs component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 0.0020560923, "other": 9.5312425e-5, "reading": 0.00039705762, "sending": 3.5721016e-5, "sleeping": 0.9974159}
2025-01-21T23:14:51.683125Z DEBUG vector::internal_events::file::source: Files checkpointed. count=0 duration_ms=0
2025-01-21T23:14:51.739252Z DEBUG source{component_kind="source" component_id=app_logs component_type=file}:file_server: file_source::file_server: event_throughput=0.000/sec bytes_throughput=0.000/sec ratios={"discovery": 0.0022521238, "other": 8.363025e-5, "reading": 0.00019917745, "sending": 2.0452862e-5, "sleeping": 0.9974447}
2025-01-21T23:14:52.684057Z DEBUG vector::internal_events::file::source: Files checkpointed. count=0 duration_ms=0
2025-01-21T23:14:53.685103Z DEBUG vector::internal_events::file::source: Files checkpointed. count=0 duration_ms=0

[jszwedko]

Gotcha, thanks, it does look like it finds the files to watch. I assume you don't see any checkpoints being written in /opt/vector/local/?

This is pretty odd. To rule out the possibility it is some oddness with a virtual device, you could try running Vector to consume files from a different, non-virtual, device just to see if you can get that working.

You could also try thechecksum fingerprint strategy in-case device_and_inode isn't working well for virtual devices.

[rrs-llabs]

It works from local dir, tried both fingerprint strategy

[jszwedko]

Interesting. What is backing the shared file system?