Skip to content

Commit c6a5a47

Browse files
committed
OBSDOCS-1972 migrate old content - add in performance_reliability folder
1 parent 8708703 commit c6a5a47

File tree

8 files changed

+110
-0
lines changed

8 files changed

+110
-0
lines changed

_topic_maps/_topic_map.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,3 +98,13 @@ Topics:
9898
File: default-logging-alerts
9999
- Name: Custom logging alerts
100100
File: custom-logging-alerts
101+
---
102+
Name: Performance and reliability tuning
103+
Dir: performance_reliability
104+
Topics:
105+
- Name: Flow control mechanisms
106+
File: logging-flow-control-mechanisms
107+
- Name: Filtering logs by content
108+
File: logging-content-filtering
109+
- Name: Filtering logs by metadata
110+
File: logging-input-spec-filtering

performance_reliability/_attributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../_attributes/

performance_reliability/images

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../images/
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
[id="logging-content-filtering"]
4+
= Filtering logs by content
5+
:context: logging-content-filtering
6+
7+
toc::[]
8+
9+
Collecting all logs from a cluster might produce a large amount of data, which can be expensive to transport and store.
10+
11+
You can reduce the volume of your log data by filtering out low priority data that does not need to be stored. {logging-uc} provides content filters that you can use to reduce the volume of log data.
12+
13+
[NOTE]
14+
====
15+
Content filters are distinct from `input` selectors. `input` selectors select or ignore entire log streams based on source metadata. Content filters edit log streams to remove and modify records based on the record content.
16+
====
17+
18+
Log data volume can be reduced by using one of the following methods:
19+
20+
* xref:../performance_reliability/logging-content-filtering.adoc#logging-content-filter-drop-records_logging-content-filtering[Configuring content filters to drop unwanted log records]
21+
* xref:../performance_reliability/logging-content-filtering.adoc#logging-content-filter-prune-records_logging-content-filtering[Configuring content filters to prune log records]
22+
23+
include::modules/logging-content-filter-drop-records.adoc[leveloffset=+1]
24+
include::modules/logging-content-filter-prune-records.adoc[leveloffset=+1]
25+
26+
[role="_additional-resources"]
27+
[id="additional-resources_logging-content-filtering"]
28+
== Additional resources
29+
* xref:../log_collection_forwarding/configuring-log-forwarding.adoc#cluster-logging-collector-log-forwarding-about_configuring-log-forwarding[About forwarding logs to third-party systems]
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
[id="logging-flow-control-mechanisms"]
4+
= Flow control mechanisms
5+
:context: logging-flow-control-mechanisms
6+
7+
toc::[]
8+
9+
If logs are produced faster than they can be collected, it can be difficult to predict or control the volume of logs being sent to an output.
10+
Not being able to predict or control the volume of logs being sent to an output can result in logs being lost. If there is a system outage and log buffers are accumulated without user control, this can also cause long recovery times and high latency when the connection is restored.
11+
12+
As an administrator, you can limit logging rates by configuring flow control mechanisms for your {logging}.
13+
14+
[id="logging-configuring-flow-control-benefits"]
15+
== Benefits of flow control mechanisms
16+
17+
* The cost and volume of logging can be predicted more accurately in advance.
18+
* Noisy containers cannot produce unbounded log traffic that drowns out other containers.
19+
* Ignoring low-value logs reduces the load on the logging infrastructure.
20+
* High-value logs can be preferred over low-value logs by assigning higher rate limits.
21+
22+
[id="logging-configuring-flow-control-about-rate-limits"]
23+
== Configuring rate limits
24+
25+
Rate limits are configured per collector, which means that the maximum rate of log collection is the number of collector instances multiplied by the rate limit.
26+
27+
Because logs are collected from each node's file system, a collector is deployed on each cluster node. For example, in a 3-node cluster, with a maximum rate limit of 10 records per second per collector, the maximum rate of log collection is 30 records per second.
28+
29+
Because the exact byte size of a record as written to an output can vary due to transformations, different encodings, or other factors, rate limits are set in number of records instead of bytes.
30+
31+
You can configure rate limits in the `ClusterLogForwarder` custom resource (CR) in two ways:
32+
33+
Output rate limit:: Limit the rate of outbound logs to selected outputs, for example, to match the network or storage capacity of an output. The output rate limit controls the aggregated per-output rate.
34+
35+
Input rate limit:: Limit the per-container rate of log collection for selected containers.
36+
37+
include::modules/logging-set-output-rate-limit.adoc[leveloffset=+1]
38+
39+
[role="_additional-resources"]
40+
.Additional resources
41+
* xref:../log_collection_forwarding/logging-output-types.adoc#logging-output-types[Log output types]
42+
43+
include::modules/logging-set-input-rate-limit.adoc[leveloffset=+1]
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
[id="logging-input-spec-filtering"]
4+
= Filtering logs by metadata
5+
:context: logging-input-spec-filtering
6+
7+
toc::[]
8+
9+
10+
You can filter logs in the `ClusterLogForwarder` CR to select or ignore an entire log stream based on the metadata by using the `input` selector. As an administrator or developer, you can include or exclude the log collection to reduce the memory and CPU load on the collector.
11+
12+
[IMPORTANT]
13+
====
14+
You can use this feature only if the Vector collector is set up in your logging deployment.
15+
====
16+
17+
[NOTE]
18+
====
19+
`input` spec filtering is different from content filtering. `input` selectors select or ignore entire log streams based on the source metadata. Content filters edit the log streams to remove and modify the records based on the record content.
20+
====
21+
22+
include::modules/logging-input-spec-filter-namespace-container.adoc[leveloffset=+1]
23+
include::modules/logging-input-spec-filter-labels-expressions.adoc[leveloffset=+1]
24+
include::modules/logging-input-spec-filter-audit-infrastructure.adoc[leveloffset=+1]

performance_reliability/modules

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../modules/

performance_reliability/snippets

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../snippets/

0 commit comments

Comments
 (0)