Wildcard file monitoring fixes/changes

_data/navigation.yml

@@ -793,7 +793,7 @@ admin-guide-nav:
       subnav:
       - title: "Options of CSV parsers"
         url: /admin-guide/120_Parser/004_CSV_parser/000_CSV_parser_options
-    - title: "Parsing dates and timestamps"
+    - title: "Dates and timestamps parsing"
       url: /admin-guide/120_Parser/005_dates_parser/README
       subnav:
       - title: "Options of date-parser() parsers"
@@ -908,16 +908,16 @@ admin-guide-nav:
         url: /admin-guide/120_Parser/012_Linux_audit_parser/000_Linux_audit_parser_options
     - title: "MariaDB parser"
       url: /admin-guide/120_Parser/013_mariadb_parser
-    - title: "metrics-probe"
+    - title: "metrics probe"
       url: /admin-guide/120_Parser/014_metrics_probe_parser/README
       subnav:
       - title: "metrics-probe options"
         url: /admin-guide/120_Parser/014_metrics_probe_parser/000_metrics_probe_options
-    - title: "Parsing tags"
+    - title: "Tags parser"
       url: /admin-guide/120_Parser/014_tags_parser
     - title: "Netskope parser"
       url: /admin-guide/120_Parser/015_netskope_parser
-    - title: "panos-parser"
+    - title: "panos parser"
       url: /admin-guide/120_Parser/015_panos_parser/README
       subnav:
       - title: "Message format parsed by panos-parser()"

_includes/doc/admin-guide/examples/multi-line.md

@@ -1,6 +1,6 @@
 >**TIP:** To format multi-line messages to your individual needs, consider
 >the following:
->  
+>
 >- To make multi-line messages more readable when written to a file,
 >    use a template in the destination and instead of the ${MESSAGE}
 >    macro, use the following: **$(indent-multi-line ${MESSAGE})**.
@@ -14,12 +14,6 @@
 >            template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
 >        );
 >    };
->    ```
->  
+>    ```  
+>For details on using templates, see Templates and macros.
 {: .notice--info}
-
-For details on using templates, see Templates and macros.
-
-- To actually convert the lines of multi-line messages to single line
-    (by replacing the newline characters with whitespaces), use the
-    **flags(no-multi-line)** option in the source.

_includes/doc/admin-guide/options/follow-freq.md

@@ -3,12 +3,16 @@
 | Type:| number|
 |Default:  | 1|
 
-*Description:* Indicates that the source should be checked periodically.
-This is useful for files which always indicate readability, even though
-no new content were appended (e.g. regular file system files). If this value
-is higher than zero, syslog-ng will not attempt to use kqueue or ivykis file change
-notification methods on the file (poll(), epoll(), etc.), but checks whether
-the file changed every time the follow-freq() interval (in seconds) has elapsed.\
+*Description:* This is useful for files that always indicate readability, even though
+no new content has been appended (e.g., regular file system files).\
+In pre-4.9 versions of {{ site.product.short_name }}, where follow-method() was not yet available,
+this indicated that the source should be checked periodically. If this value is higher than zero,
+{{ site.product.short_name }} will not attempt to use kqueue or ivykis file change notification
+methods on the file (poll(), epoll(), etc.), but instead checks whether the file has changed every time the follow-freq()
+interval (in seconds) has elapsed.\
+In version 4.9 and later, this option only has an effect if follow-method() is set to `legacy` or `poll`,
+ in which case it is used as in pre-4.9 versions. For any other follow-method() values, the value of this option is ignored.
+
 Floating-point numbers (for example, **1.5**) can be used as well.
 
 {% include doc/admin-guide/warnings/file-source-follow-warning.md %}

_includes/doc/admin-guide/options/follow-method.md

@@ -0,0 +1,15 @@
+## follow-method()
+
+|Accepted values:| legacy \| inotify \| poll \| system |
+|Default: | legacy |
+
+*Description:* This option is available in version 4.9 and higher and its value controls how {{ site.product.short_name }} will follow file changes.\
+The default `legacy` mode preserves the pre-4.9 version file follow-mode behavior of {{ site.product.short_name }}, which is based on the value of follow-freq().\
+The `poll` value forces {{ site.product.short_name }} to poll for file changes at the interval specified by the monitor-freq() option, even if a more efficient method (such as `inotify` or `kqueue`) is available.\
+If `inotify` is selected and supported by the platform, {{ site.product.short_name }} uses it to detect changes in source files. This is the most efficient and least resource-consuming option available on Linux for regular files.\
+The `system` value will use system poll methods (via ivykis) like `port-timer` `port` `dev_poll` `epoll-timerfd` `epoll` `kqueue` `ppoll` `poll` and `uring`. For more information about how to control the system polling methods used, see How content changes are followed in file() and wildcard-file() sources.
+
+**NOTE:** Using `inotify` also requires setting monitor-method() to `inotify`.
+{: .notice--info}
+
+{% include doc/admin-guide/warnings/file-source-follow-warning.md %}

_includes/doc/admin-guide/options/multi-line-mode-empty-line-separated.md

@@ -0,0 +1 @@
+- The *empty-line-separated* mode, as its name suggests, reads and treats all messages as one till it receives an empty line (which contains only a \r, \n or \r\n sequence).

_includes/doc/admin-guide/options/multi-line-mode-footer.md

@@ -0,0 +1,2 @@
+>**TIP:** To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the **flags(no-multi-line)** option in the source.
+{: .notice--info}

_includes/doc/admin-guide/options/multi-line-mode-header.md

@@ -0,0 +1,8 @@
+## multi-line-mode()
+
+| Type:  |  indented \| regexp \| empty-line-separated |
+|Default: |  empty string|
+
+*Description:* Use the multi-line-mode() option when processing
+multi-line messages. The {{ site.product.short_name }} application provides the
+following methods to process multi-line messages:

_includes/doc/admin-guide/options/multi-line-mode-network-header.md

@@ -0,0 +1,8 @@
+## multi-line-mode()
+
+| Type:  |  empty-line-separated |
+|Default: |  empty string|
+
+*Description:* Use the multi-line-mode() option when processing
+multi-line messages. For this source the {{ site.product.short_name }} application provides only the
+`empty-line-separated` following method currently to process multi-line messages:

_includes/doc/admin-guide/options/multi-line-mode-network.md

@@ -0,0 +1,5 @@
+{% include doc/admin-guide/options/multi-line-mode-network-header.md %}
+
+{% include doc/admin-guide/options/multi-line-mode-empty-line-separated.md %}
+
+{% include doc/admin-guide/options/multi-line-mode-footer.md %}

_includes/doc/admin-guide/options/multi-line-mode.md

@@ -1,11 +1,4 @@
-## multi-line-mode()
-
-| Type:  |  indented \| regexp|
-|Default: |  empty string|
-
-*Description:* Use the multi-line-mode() option when processing
-multi-line messages. The {{ site.product.short_name }} application provides the
-following methods to process multi-line messages:
+{% include doc/admin-guide/options/multi-line-mode-header.md %}
 
 - The *indented* mode can process messages where each line that
     belongs to the previous line is indented by whitespace, and the
@@ -29,7 +22,7 @@ following methods to process multi-line messages:
     multi-line-mode(prefix-garbage), see the multi-line-prefix() and
     multi-line-garbage() options.
 
-- The prefix-suffix mode uses a string or regular expression (set in
+- The *prefix-suffix* mode uses a string or regular expression (set in
     multi-line-prefix()) that matches the beginning of the log messages,
     ignores newline characters from the source until a line matches the
     regular expression set in multi-line-suffix(), and treats the lines
@@ -43,4 +36,8 @@ following methods to process multi-line messages:
     The prefix-suffix mode is similar to the prefix-garbage mode, but it
     appends the garbage part to the message instead of discarding it.
 
+{% include doc/admin-guide/options/multi-line-mode-empty-line-separated.md %}
+
 {% include doc/admin-guide/examples/multi-line.md %}
+
+{% include doc/admin-guide/options/multi-line-mode-footer.md %}

_includes/doc/admin-guide/options/multi-line-prefix.md

@@ -13,8 +13,6 @@ multi-line messages. If the multi-line-prefix() option is set, {{ site.product.s
 regular expression again, and treats the lines between the matching
 lines as a single message. See also the multi-line-garbage() option.
 
-{% include doc/admin-guide/examples/multi-line.md %}
-
 ### Example: Processing Tomcat logs
 
 The log messages of the Apache Tomcat server are a typical example for

_includes/doc/admin-guide/options/multi-line-timeout.md

@@ -23,3 +23,6 @@ source s_multi {
 ```
 
 For more details see the blog post Multi-line-timeout: making sure your last multi-line message is not lost.
+
+>**NOTE:** This option has effect only if the file follow mode is `syslog-ng poll`, `follow-freq()` > 0. For details of file follow modes, refer to How content changes are followed in file() and wildcard-file() sources.
+{: .notice--info}

_includes/doc/admin-guide/warnings/file-source-follow-warning.md

@@ -1,2 +1,2 @@
-![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), the monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
+![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), follow-method(), monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
 {: .notice--warning}

doc/_admin-guide/060_Sources/005_Default-network-drivers/000_Default-network-drivers_options.md

@@ -18,6 +18,8 @@ The default-network-drivers() source has the following options.
 
 {% include doc/admin-guide/options/max-connections.md %}
 
+{% include doc/admin-guide/options/multi-line-mode-network.md %}
+
 ## rfc5424-tcp-port()
 
 |  Type:|      number|

doc/_admin-guide/060_Sources/020_File/000_File_source_options.md

@@ -17,6 +17,8 @@ The file() driver has the following options:
 
 {% include doc/admin-guide/options/follow-freq.md %}
 
+{% include doc/admin-guide/options/follow-method.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/020_File/001_File_following.md

@@ -19,16 +19,22 @@ Setting the `auto` method will select the best available method on the given OS.
 
 Detecting file content changes involves more factors that can affect resource usage and overall performance. {{ site.product.short_name }} uses two main methods to monitor file content changes.
 
-The first method is automatically selected if the follow-freq() option has a value greater than 0. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires. Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
+The first method is automatically selected in pre-4.9 version of {{ site.product.short_name }} if the follow-freq() option has a value greater than 0, or in version 4.9 or higher if follow-method() `legacy` or `poll` selected. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires.\
+Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
 
-The second method is activated if the follow-freq() option is set to 0. It uses ivykis poll methods, sometimes resembling the poll method for directory change watching described above (with its performance penalties), but often working similarly to the `inotify` or `kqueue` version (with seamless performance).
+The second method is activated in pre-4.9 versions of {{ site.product.short_name }} if the follow-freq() option is set to 0, or in version 4.9 and higher if follow-method() is set to inotify or system.\
+If inotify is available on the system and selected in both follow-method() and monitor-method(), it will be used, resulting in significantly lower resource usage (especially on Linux). This option is accessible only on version 4.9 or later.\
+Otherwise, {{ site.product.short_name }} uses ivykis polling methods, which sometimes resemble the polling method used for directory change watching described above (with its associated performance penalties), but can also operate similarly to the kqueue version (offering seamless performance).
 
 The following table shows which method is selected in different cases.
 
 <table border="0" cellspacing="0" cellpadding="0" width="1397">
   <tr>
-    <td width="99" class="right-edged-col">
-      <p align="center">follow-freq()</p>
+    <td width="174" class="right-edged-col">
+      <p align="center">pre-4.9 version<br>or<br>follow-method(legacy)<br>or<br>follow-method(poll)</p>
+    </td>
+    <td width="151" class="right-edged-col">
+      <p align="center">version 4.9 or higher<br>follow-method()</p>
     </td>
     <td width="200" colspan="2" class="right-edged-col">
       <p align="center">file follow method</p>
@@ -44,8 +50,11 @@ The following table shows which method is selected in different cases.
     </td>
   </tr>
   <tr>
-    <td width="99" rowspan="8" class="right-edged-col">
-      <p align="center">0</p>
+    <td width="174" rowspan="8" class="right-edged-col">
+      <p align="center">follow-freq(0)</p>
+    </td>
+    <td width="150" rowspan="7" class="right-edged-col">
+      <p align="center">system</p>
     </td>
     <td width="150" rowspan="7" class="right-edged-col">
       <p align="center">ivykis poll</p>
@@ -148,7 +157,10 @@ The following table shows which method is selected in different cases.
     </td>
   </tr>
   <tr>
-    <td width="150"  colspan="2" class="right-edged-col">
+    <td width="150" class="right-edged-col">
+      <p align="center">inotify</p>
+    </td>
+    <td width="150" colspan="2" class="right-edged-col">
       <p align="center">inotify from ivykis directory monitor</p>
     </td>
     <td width="151" class="right-edged-col">
@@ -162,8 +174,11 @@ The following table shows which method is selected in different cases.
     </td>
   </tr>
   <tr>
-    <td width="99" rowspan="3" class="right-edged-col">
-      <p align="center">&gt; 0</p>
+    <td width="174" rowspan="3" class="right-edged-col">
+      <p align="center">follow-freq(n)<br> n &gt; 0</p>
+    </td>
+    <td width="150" rowspan="3" class="right-edged-col">
+      <p align="center">poll</p>
     </td>
     <td width="150" colspan="2" rowspan="3" class="right-edged-col">
       <p align="center">syslog-ng poll<br>using an ivykis timer with timer freq set to follow-freq() value</p>
@@ -180,7 +195,7 @@ The following table shows which method is selected in different cases.
   </tr>
 </table>
 
-  **NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-freq() set to 0.
+  **NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-method() set to `inotify`.
   {: .notice--info}
 
 A bit more detail about the notation in the platform columns and what they really mean:

doc/_admin-guide/060_Sources/030_Wildcard-file/000_Wildcard-file_options.md

@@ -76,6 +76,8 @@ source s_files {
 
 {% include doc/admin-guide/options/follow-freq.md %}
 
+{% include doc/admin-guide/options/follow-method.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/070_Network/000_Network_source_options.md

@@ -53,6 +53,8 @@ Available in 3.19 and later.
 
 {% include doc/admin-guide/options/max-connections.md %}
 
+{% include doc/admin-guide/options/multi-line-mode-network.md %}
+
 {% include doc/admin-guide/options/pad-size.md %}
 
 {% include doc/admin-guide/options/port-localport.md %}

doc/_admin-guide/060_Sources/155_stdin/000_stdin_options.md

@@ -18,6 +18,8 @@ The stdin() driver has the following options:
 
 {% include doc/admin-guide/options/follow-freq.md %}
 
+{% include doc/admin-guide/options/follow-method.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/170_Syslog/000_syslog_source_options.md

@@ -53,6 +53,8 @@ Available in 3.19 and later.
 
 {% include doc/admin-guide/options/max-connections.md %}
 
+{% include doc/admin-guide/options/multi-line-mode-network.md %}
+
 {% include doc/admin-guide/options/pad-size.md %}
 
 {% include doc/admin-guide/options/port-localport.md %}