Merge pull request #8420 from naveenpaul1/update_noobaa_log

NSFS | Update noobaa log document and add syslog-ng

Author: naveenpaul1

docs/NooBaaNonContainerized/Logging.md

@@ -19,10 +19,19 @@ This document provides an overview of the logging capabilities within NooBaa, de
 
 ## NooBaa Service Logs
 
-NooBaa service logs are written directly to stderr and to syslog -  
-1. NooBaa writes the logs into stderr and to `/var/log/noobaa.log`.
-2. The `journalctl` service watches stderr and writes it to `journal`.
-3. A `syslog` service, like `rsyslog/syslog_ng` watches `journal` and writes it to `/var/log/messages/`.
+NooBaa service logs are written directly to stderr, syslog, or both.
+1. The `journalctl` service watches stderr and writes into `journal`. That internally updates the `/var/log/messages`.
+2. Noobaa sends logs to a syslog service such as rsyslog or syslog_ng and writes the logs to `/var/log/noobaa.log` if enabled. We can not have more than one service enabled at a time. It might cause some issues.
+
+### Set log config
+
+Set `LOG_TO_STDERR_ENABLED` property to `true` to  send Noobaa logs to stderr. Noobaa logs will be send to syslog when the property `LOG_TO_SYSLOG_ENABLED` set to `true`.
+
+Default configurations are:
+```
+LOG_TO_STDERR_ENABLED = true;
+LOG_TO_SYSLOG_ENABLED = false;
+```
 
 ### Journal logs
 1. NooBaa service logs -
@@ -42,15 +51,15 @@ NooBaa logs are configured using rsyslog and logrotate. NooBaa RPM will configur
 systemctl status rsyslog
 ```
 
-NooBaa logs are pushed to `var/log/noobaa.log` and the log is rotated and compressed daily.
+NooBaa logs are pushed to `/var/log/noobaa.log` and the log is rotated and compressed daily.
 
 The following files contain NooBaa specific configuration for rsyslog and logrotate - 
 1. `etc/rsyslog.d/noobaa_syslog.conf`
 2. `etc/logrotate.d/noobaa-logrotate` 
 
 Verify the rsyslog and logrotate rpm configuration is complete by checking the files above.
 
-#### Logrotate
+### Logrotate
 
 ##### Logrotate automatic rotation
 
@@ -79,16 +88,58 @@ logrotate etc/logrotate.d/noobaa-logrotate
 Set `NOOBAA_LOG_LEVEL` property in config.json to control the amount of debugging information generated by the application.
 For more info about setting custom properties, see - [Non Containerized Config File Customizations](./ConfigFileCustomizations.md)
 
-## NooBaa Logs format
+### NooBaa Logs format
 NooBaa logs are formatted with ANSI color codes. </br>
 Use the `cat <logfile> | less -R` command in order to display the contents of a log file with pagination and the ability to handle ANSI color codes. 
 
-### Set log config
+## How to configure syslog-ng
+
+1. create syslog-ng noobaa specific config file `noobaa-syslog-ng.conf` under `etc/syslog-ng/conf.d/`
+```
+vi etc/syslog-ng/conf.d/noobaa-syslog-ng.conf 
+```
+2. update the file with bellow configuration
 
-Set `LOG_TO_STDERR_ENABLED` property to `true` to  send Noobaa logs to stderr. Noobaa logs will be send to syslog when the property `LOG_TO_SYSLOG_ENABLED` set to `true`. 
+```
+destination d_noobaa_msg { file("/var/log/noobaa.log"); };
+destination d_noobaa_event { file("/var/log/noobaa_events.log"); };
 
-Default configurations are:
+filter f_noobaa_msg     { facility(local0); };
+filter f_noobaa_event     { facility(local2); };
+
+log { source(s_sys); filter(f_noobaa_msg); destination(d_noobaa_msg); };
+log { source(s_sys); filter(f_noobaa_event); destination(d_noobaa_event); };
 ```
-LOG_TO_STDERR_ENABLED = true;
-LOG_TO_SYSLOG_ENABLED = false;
-```
+We are creating two new destination one is `d_noobaa_msg`, point to log file `/var/log/noobaa.log` and another one is `d_noobaa_event` , point to log file `/var/log/noobaa_events.log`
+Filter  `f_noobaa_msg` and `f_noobaa_event` will filter out only noobaa logs and events log. Last created two source that will filter and log only noobaa normal logs and events to previously added destinations.
+
+3. Update existing filter `f_default` in default syslog-ng `etc/syslog-ng/syslog-ng.conf`  to make sure noobaa logs and event are not send to `/var/log/message` log file.
+```
+vi etc/syslog-ng/syslog-ng.conf
+// update file for filter f_default
+filter f_default    { level(info..emerg) and
+                        not (facility(mail)
+                        or facility(local0)
+                        or facility(local2)
+                        or facility(authpriv)
+                        or facility(cron)); };
+```
+ Noobaa logs with facility local0 and local2 are excluded from message logs
+4. restart syslog-ng
+```
+systemctl restart syslog-ng
+```
+
+### Known issues
+
+1. missing logs due to systemd-journald log suppression
+        `Suppressed 6284127 messages from noobaa.service`
+    
+    solution :
+    * Edit /etc/systemd/journald.conf
+    * add below configuration
+        ```
+        RateLimitInterval=0
+        RateLimitBurst=0
+        ```
+    link : https://my.f5.com/manage/s/article/K70501143