Skip to content

Commit b693009

Browse files
mburke5678mburke5678
authored
Merge pull request #13344 from mburke5678/logging-configuring-tuning
Add Configuring and Tuning Cluster Logging
2 parents 8359806 + c217c79 commit b693009

File tree

48 files changed

+1083
-270
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+1083
-270
lines changed

‎_topic_map.yml

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -358,25 +358,28 @@ Topics:
358358
File: efk-logging
359359
- Name: Deploying EFK stack
360360
File: efk-logging-deploy
361-
- Name: Uninstalling the EFK stack
362-
File: efk-logging-uninstall
363-
- Name: Working with Elasticsearch
361+
- Name: Configuring cluster logging
362+
File: efk-logging-configuring
363+
- Name: Configuring Elasticsearch
364364
File: efk-logging-elasticsearch
365-
- Name: Working with Fluentd
365+
- Name: Configuring Fluentd
366366
File: efk-logging-fluentd
367-
- Name: Working with Kibana
367+
- Name: Configuring Kibana
368368
File: efk-logging-kibana
369-
- Name: Working with Curator
369+
- Name: Configuring Curator
370370
File: efk-logging-curator
371+
- Name: Configuring the cluster logging driver
372+
File: efk-logging-driver
373+
- Name: Configuring systemd-journald and rsyslog
374+
File: efk-logging-systemd
371375
- Name: Sending logs to external devices
372376
File: efk-logging-external
373-
- Name: Setting the cluster logging driver
374-
File: efk-logging-driver
375377
- Name: Manually rolling out Elasticsearch
376378
File: efk-logging-manual-rollout
377-
- Name: Configuring systemd-journald and rsyslog
378-
File: efk-logging-systemd
379379
- Name: Troubleshooting Kubernetes
380380
File: efk-logging-troubleshooting
381381
- Name: Exported fields
382382
File: efk-logging-exported-fields
383+
- Name: Uninstalling the EFK stack
384+
File: efk-logging-uninstall
385+

‎logging/efk-logging-configuring.adoc

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
:context: efk-logging-configuring
2+
[id='efk-logging-configuring']
3+
= Configuring cluster logging
4+
include::modules/common-attributes.adoc[]
5+
6+
toc::[]
7+
8+
Cluster logging is configurable using a Cluster Logging Custom Resource (CR) deployed
9+
in the **openshift-logging** namespace.
10+
11+
The Cluster Logging Operator watches for changes in the xref:efk-logging-configuring-about-sample[Cluster Logging CR]
12+
and adjusts the logging deployment accordingly.
13+
14+
The Cluster Logging CR is based on the Cluster Logging Custom Resource Definition (CRD), which defines a complete cluster logging deployment
15+
and includes all the components of the logging stack to collect, store and visualize logs.
16+
17+
.Sample Cluster Logging Custom Resource Definition (CRD)
18+
[source,yaml]
19+
----
20+
apiVersion: "logging.openshift.io/v1alpha1"
21+
kind: "ClusterLogging"
22+
metadata:
23+
name: "example"
24+
spec:
25+
managementState: "Managed"
26+
logStore:
27+
type: "elasticsearch"
28+
elasticsearch:
29+
nodeCount: 1
30+
storage: {}
31+
redundancyPolicy: "SingleRedundancy"
32+
visualization:
33+
type: "kibana"
34+
kibana:
35+
replicas: 1
36+
curation:
37+
type: "curator"
38+
curator:
39+
schedule: "30 3 * * *"
40+
collection:
41+
logs:
42+
type: "fluentd"
43+
fluentd: {}
44+
----
45+
46+
You can configure the following components of cluster logging:
47+
48+
* You can place cluster logging into an unmanaged state that allows an administrator to assume full control of individual
49+
component configurations and upgrades.
50+
51+
* You can overwrite the image for each cluster logging component by modifying the appropriate
52+
environment variable in the `cluster-logging-operator` Deployment.
53+
54+
* You can specify specific nodes for the logging components.
55+
56+
* You can specify the Log collectors to deploy to each node in a cluster, either Fluentd or Rsyslog.
57+
58+
[IMPORTANT]
59+
====
60+
The Rsyslog log collector is currently a Technology Preview feature.
61+
====
62+
63+
// The following include statements pull in the module files that comprise
64+
// the assembly. Include any combination of concept, procedure, or reference
65+
// modules required to cover the user story. You can also include other
66+
// assemblies.
67+
68+
include::modules/efk-logging-management-state-changing.adoc[leveloffset=+1]
69+
70+
include::modules/efk-logging-configuring-image-about.adoc[leveloffset=+1]
71+
72+
include::modules/efk-logging-configuring-image.adoc[leveloffset=+2]
73+
74+
include::modules/efk-logging-configuring-node-selector.adoc[leveloffset=+1]

‎logging/efk-logging-driver.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
:context: efk-logging-driver
22
[id='efk-logging-driver']
3-
= Setting the cluster logging driver
3+
= Configuring the cluster logging driver
44
include::modules/common-attributes.adoc[]
55

66
toc::[]

‎logging/efk-logging-elasticsearch-storage.adoc

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
[id='efk-logging-elasticsearch-storage']
2+
= Configuring Elasticsearch storage
3+
include::modules/common-attributes.adoc[]
4+
5+
toc::[]
6+
7+
8+
9+
You can can configure the number of Elasticsearch nodes in your cluster and configure persistent storage.
10+
11+
You can can bind each data node in the Elasticsearch cluster to a persistent volume claim using a Cluster Logging Custom Resource (CR).
12+
13+
[NOTE]
14+
====
15+
If you omit the storage block, the deployment will include ephemeral storage only.
16+
====
17+
18+
The following example specifies each data node in the cluster is bound to a Persistent Volume Claim that requests 200G of General Purpose SSD (gp2) storage.
19+
20+
[source,yaml]
21+
----
22+
apiVersion: "logging.openshift.io/v1alpha1"
23+
kind: "ClusterLogging"
24+
metadata:
25+
name: "clusterlogging"
26+
27+
....
28+
29+
spec:
30+
logStore:
31+
type: "elasticsearch"
32+
elasticsearch:
33+
nodeCount: 3
34+
storage:
35+
storageClass:
36+
name: "gp2"
37+
size: "200G"
38+
----
39+
40+
// The following include statements pull in the module files that comprise
41+
// the assembly. Include any combination of concept, procedure, or reference
42+
// modules required to cover the user story. You can also include other
43+
// assemblies.
44+
45+
46+
include::modules/efk-logging-elasticsearch-persistent-storage-persistent.adoc[leveloffset=+1]
47+
48+
include::modules/efk-logging-elasticsearch-persistent-storage-local.adoc[leveloffset=+1]
49+
50+
include::modules/efk-logging-elasticsearch-persistent-storage-empty.adoc[leveloffset=+1]
51+

‎logging/efk-logging-elasticsearch.adoc

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,33 @@
11
:context: efk-logging-elasticsearch
22
[id='efk-logging-elasticsearch']
3-
= Working with Elasticsearch in {product-title}
3+
= Configuring Elasticsearch in {product-title}
44
include::modules/common-attributes.adoc[]
55

66
toc::[]
77

88
{product-title} uses Elasticsearch (ES) to organize the log data from Fleutnd into datastores, or _indices_.
99

10+
You can add and remove nodes, configure storage for your Elasticsearch cluster, and
11+
define how shards are replicated across data nodes in the cluster, from full replication to no replication.
12+
1013
// The following include statements pull in the module files that comprise
1114
// the assembly. Include any combination of concept, procedure, or reference
1215
// modules required to cover the user story. You can also include other
1316
// assemblies.
1417

18+
include::modules/efk-logging-elasticsearch-limits.adoc[leveloffset=+1]
19+
1520
include::modules/efk-logging-elasticsearch-ha.adoc[leveloffset=+1]
1621

17-
include::modules/efk-logging-elasticsearch-persistent-storage-about.adoc[leveloffset=+1]
22+
include::modules/efk-logging-elasticsearch-storage.adoc[leveloffset=+1]
1823

1924
include::modules/efk-logging-elasticsearch-persistent-storage-persistent.adoc[leveloffset=+2]
2025

26+
include::modules/efk-logging-elasticsearch-persistent-storage-persistent-dynamic.adoc[leveloffset=+2]
27+
2128
include::modules/efk-logging-elasticsearch-persistent-storage-local.adoc[leveloffset=+2]
2229

23-
include::modules/efk-logging-elasticsearch-scaling.adoc[leveloffset=+1]
30+
include::modules/efk-logging-elasticsearch-persistent-storage-empty.adoc[leveloffset=+2]
2431

2532
include::modules/efk-logging-elasticsearch-exposing.adoc[leveloffset=+1]
2633

27-
28-

‎logging/efk-logging-external.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ include::modules/efk-logging-external-elasticsearch.adoc[leveloffset=+1]
1717

1818
include::modules/efk-logging-external-syslog.adoc[leveloffset=+1]
1919

20-
include::modules/efk-logging-external-fluentd.adoc[leveloffset=+1]
20+
include::modules/efk-logging-fluentd-external.adoc[leveloffset=+1]
2121

2222

2323

‎logging/efk-logging-fluentd.adoc

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,34 @@
11
:context: efk-logging-fluentd
22
[id='efk-logging-fluentd']
3-
= Working with Fluentd in {product-title}
3+
= Configuring Fluentd in {product-title}
44
include::modules/common-attributes.adoc[]
55

66
toc::[]
77

88
{product-title} uses Fluentd to collect data about your cluster.
99

10+
You can configure log rotation, log location, use an external log aggregator, and make other configurations.
11+
1012
// The following include statements pull in the module files that comprise
1113
// the assembly. Include any combination of concept, procedure, or reference
1214
// modules required to cover the user story. You can also include other
1315
// assemblies.
1416

1517

18+
1619
include::modules/efk-logging-fluentd-pod-location.adoc[leveloffset=+1]
1720

1821
include::modules/efk-logging-fluentd-log-viewing.adoc[leveloffset=+1]
1922

23+
include::modules/efk-logging-fluentd-limits.adoc[leveloffset=+1]
24+
25+
include::modules/efk-logging-fluentd-collector.adoc[leveloffset=+1]
26+
2027
include::modules/efk-logging-fluentd-log-location.adoc[leveloffset=+1]
2128

2229
include::modules/efk-logging-fluentd-log-rotation.adoc[leveloffset=+1]
2330

24-
include::modules/efk-logging-external-fluentd.adoc[leveloffset=+1]
31+
include::modules/efk-logging-fluentd-external.adoc[leveloffset=+1]
2532

2633
include::modules/efk-logging-fluentd-throttling.adoc[leveloffset=+1]
2734

‎logging/efk-logging-image.adoc

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
:context: efk-logging-configuring
2+
[id='efk-logging-image']
3+
= Setting the cluster logging component image
4+
include::modules/common-attributes.adoc[]
5+
6+
7+
toc::[]
8+
9+
10+
11+
Each cluster logging component uses an image.
12+
13+
By default, cluster logging uses the following images:
14+
15+
* *ELASTICSEARCH_IMAGE* deploys Elasticsearch, by default `docker.io/openshift/origin-logging-elasticsearch5:latest`.
16+
* *FLUENTD_IMAGE* deploys Fluentd, by default `docker.io/openshift/origin-logging-fluentd:latest`.
17+
* *KIBANA_IMAGE* deploys Kibana, by default `docker.io/openshift/origin-logging-kibana5:latest`.
18+
* *CURATOR_IMAGE* deploys Curator, by default `docker.io/openshift/origin-logging-curator5:latest`.
19+
* *OAUTH_PROXY_IMAGE* defines OAUTH for {product-title}, by default `docker.io/openshift/oauth-proxy:latest`.
20+
* *RSYSLOG_IMAGE* deploys Rsyslog, by default `docker.io/viaq/rsyslog:latest`.
21+
* *WATCH_NAMESPACE* allows cluster logging to view all namespaces in the cluster, by default `openshift-logging`.
22+
* *OPERATOR_NAME* configures cluster logging to use the Cluster Logging Operator, by default `cluster-logging-operator`.
23+
24+
25+
[NOTE]
26+
====
27+
The Rsyslog log collector is in Technology Preview.
28+
====
29+
30+
// The following include statements pull in the module files that comprise
31+
// the assembly. Include any combination of concept, procedure, or reference
32+
// modules required to cover the user story. You can also include other
33+
// assemblies.
34+
35+
36+
*Specific information on changing the image through a separate PR*
37+
38+
include::modules/efk-logging-image-configuring.adoc[leveloffset=+1]
39+
40+
41+

‎logging/efk-logging-kibana.adoc

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,23 @@
11
:context: efk-logging-kibana
22
[id='efk-logging-kibana']
3-
= Working with Kibana in {product-title}
3+
= Configuring Kibana in {product-title}
44
include::modules/common-attributes.adoc[]
55

66
toc::[]
77

88
{product-title} uses Kibana to display the log data collected by Fluentd and indexed by Elasticsearch.
99

10-
You can see the user interface by visiting the site specified by the `openshift_logging_kibana_hostname` variable.
10+
Kibana generates a web console specified by the `openshift_logging_kibana_hostname` variable.
1111

12+
You can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes.
1213

1314
// The following include statements pull in the module files that comprise
1415
// the assembly. Include any combination of concept, procedure, or reference
1516
// modules required to cover the user story. You can also include other
1617
// assemblies.
1718

19+
include::modules/efk-logging-kibana-limits.adoc[leveloffset=+1]
20+
1821
include::modules/efk-logging-kibana-console.adoc[leveloffset=+1]
1922

2023
include::modules/efk-logging-kibana-scaling.adoc[leveloffset=+1]

‎logging/efk-logging.adoc

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,6 @@ toc::[]
99

1010
As a {product-title} cluster administrator, you can deploy cluster logging to aggregate logs for a range of {product-title} services.
1111

12-
Cluster logging is distinct from the default systemd logging.
13-
1412
// The following include statements pull in the module files that comprise
1513
// the assembly. Include any combination of concept, procedure, or reference
1614
// modules required to cover the user story. You can also include other
@@ -26,5 +24,9 @@ include::modules/efk-logging-about-kibana.adoc[leveloffset=+2]
2624

2725
include::modules/efk-logging-about-curator.adoc[leveloffset=+2]
2826

27+
include::modules/efk-logging-about-crd.adoc[leveloffset=+2]
28+
29+
30+
2931

3032

0 commit comments

Comments
 (0)