--dry-run shows several warnings of parameter <name> in <filter> is not used

[khteh]

What is a problem?

fluentd -c /etc/fluent/fluentd.conf --dry-run shows several warnings of parameter <name> in <filter> is not used

Describe the configuration of Fluentd

<source>
  @type sample
  sample [{"access_log": "[2025-04-04 08:44:16 +0000] [10] [INFO] 192.168.0.149:43472 - - [04/Apr/2025:08:44:16 +0000] \"GET /health/live 2\" 200 2 \"-\" \"kube-probe/1.27\""}]
  tag access_log_sample
</source>
<filter access_log_sample>
  @type parser
  key_name access_log
  reserve_data true
  time_type string
  time_format %Y-%m-%d %H:%M:%S
  keep_time_key true
  # even if we failed to parse the time send the record over to splunk
  emit_invalid_record_to_error false
  reserve_time true
  <parse> 
    @type regexp
      expression /^(?<time>.+?) (?<level>\w{0,8}) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")$/
  </parse>
</filter>

Describe the logs of Fluentd

2025-04-07 13:46:10 +0800 [warn]: parameter 'time_type' in <filter access_log_sample>
  @type parser
  key_name "access_log"
  reserve_data true
  time_type string
  time_format %Y-%m-%d %H:%M:%S
  keep_time_key true
  emit_invalid_record_to_error false
  reserve_time true
  <parse>
    @type "regexp"
    expression /^(?<time>.+?) (?<level>\w{0,8}) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")$/
  </parse>
</filter> is not used.
2025-04-07 13:46:10 +0800 [warn]: parameter 'time_format' in <filter access_log_sample>
  @type parser
  key_name "access_log"
  reserve_data true
  time_type string
  time_format %Y-%m-%d %H:%M:%S
  keep_time_key true
  emit_invalid_record_to_error false
  reserve_time true
  <parse>
    @type "regexp"
    expression /^(?<time>.+?) (?<level>\w{0,8}) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")$/
  </parse>
</filter> is not used.
2025-04-07 13:46:10 +0800 [warn]: parameter 'keep_time_key' in <filter access_log_sample>
  @type parser
  key_name "access_log"
  reserve_data true
  time_type string
  time_format %Y-%m-%d %H:%M:%S
  keep_time_key true
  emit_invalid_record_to_error false
  reserve_time true
  <parse>
    @type "regexp"
    expression /^(?<time>.+?) (?<level>\w{0,8}) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")$/
  </parse>
</filter> is not used.

Environment

- Fluentd version: fluent-package 5.1.0 fluentd 1.17.0 (e763c0761c44d9734b6aa374371387a2e8406522)
- TD Agent version: fluent-package 5.1.0 fluentd 1.17.0 (e763c0761c44d9734b6aa374371387a2e8406522)
- Fluent Package version: ?
- Docker image (tag):
- Operating system: Ubuntu 24.10
- Kernel version: Linux khteh-p17-2i 6.11.0-21-generic #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 19 16:50:40 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[khteh]

Can I use <parse> in sample? According to https://stackoverflow.com/questions/71836203/fluentd-time-format-in-parser-section-doesnt-work, it seems that I can't..?

[daipom]

No. in_sample does not support parser.

You can see it does not have the parser helper.

• https://docs.fluentd.org/input/sample#plugin-helpers

[daipom]

@khteh
Some parameters are not set correctly.

time_type and time_format are parameters of Parser.

• https://docs.fluentd.org/configuration/parse-section#time-parameters

filter_parser is a Filter plugin, not Parser plugin.
So you can not set parameters of Parser directly to filter_parser.
You need to set those parameters under the <parse> section.
Please see examples.

• https://docs.fluentd.org/filter/parser#example-configurations

[khteh]

Sorry, what's the use case of tdlog plugin? I only have these <match in the config which shows these warnings:

<match td.*.*> 
  @type tdlog
  @id output_td
  apikey YOUR_API_KEY

  auto_create_table
  <buffer>
    @type file
    path /var/log/td-agent/buffer/td
  </buffer>

  <secondary>
    @type file
    path /var/log/td-agent/failed_records
  </secondary>
</match>

## match tag=debug.** and dump to console 
<match debug.**>
  @type stdout
  @id output_stdout
</match>

[khteh]

I got it. https://docs.fluentd.org/how-to-guides/http-to-td No, I don't use it and will remove it. I use Elasticsearch instead. Thanks for highlighting that.

[khteh]

After removing tdlog, now I only see these warnings:

2025-04-07 19:36:29 +0800 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2025-04-07 19:36:29 +0800 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2025-04-07 19:36:29 +0800 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2025-04-07 19:36:29 +0800 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.

[daipom]

Sorry, this means there are some input plugins using https://docs.fluentd.org/storage with on-memory state.

Since two warning logs are generated for each one, it seems there are two locations.

sample plugin uses storage_local internally.
This causes the warning log, but you can ignore it.
(storage_local in sample is used for auto_increment_key. You don't use it, so there is no impact at all. We should fix this useless warning...)

If you want to suppress the warning, you need to set https://docs.fluentd.org/deployment/system-config#root_dir.
(It allows storage_local to automatically save data to that directory.)

<system>
  root_dir /path/to/fluentd-root
</system>