Skip to content

Commit 4de10a0

Browse files
mburke5678mburke5678
authored
Merge pull request #14370 from mburke5678/logging-es-status
[WIP] EO - ClusterLogging Status messages when cluster logging is not functional
2 parents 79dda5c + 2b41341 commit 4de10a0

File tree

3 files changed

+295
-53
lines changed

3 files changed

+295
-53
lines changed

‎logging/config/efk-logging-elasticsearch.adoc

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ or higher memory. Each Elasticsearch node can operate with a lower memory settin
1919

2020
[NOTE]
2121
====
22-
Procedures in this topic require your cluster to be in an unmanaged state. For more information, see _Changing the cluster logging management state_.
22+
Procedures in this topic require your cluster to be in an unmanaged state. For more information, see Changing the cluster logging management state.
2323
====
2424

2525
// The following include statements pull in the module files that comprise
@@ -41,6 +41,10 @@ include::modules/efk-logging-elasticsearch-exposing.adoc[leveloffset=+1]
4141

4242
include::modules/efk-logging-elasticsearch-rules.adoc[leveloffset=+1]
4343

44+
include::modules/efk-logging-elasticsearch-status.adoc[leveloffset=+1]
45+
46+
include::modules/efk-logging-elasticsearch-status-comp.adoc[leveloffset=+1]
47+
4448
////
4549
modules/efk-logging-elasticsearch-persistent-storage-persistent.adoc[leveloffset=+2]
4650

@@ -49,7 +53,7 @@ modules/efk-logging-elasticsearch-persistent-storage-persistent-dynamic.adoc[lev
4953
modules/efk-logging-elasticsearch-persistent-storage-local.adoc[leveloffset=+2]
5054
////
5155

52-
== Additional Resources
56+
// == Additional Resources
5357

54-
//For information on installing Elasticsearch, see xref:../../logging/efk-logging-deploying.adoc[Deploying cluster logging].
58+
//For information on installing Elasticsearch, see xref:../logging/efk-logging-deploy.adoc[Deploying cluster logging].
5559

‎modules/efk-logging-elasticsearch-status-comp.adoc

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * logging/efk-logging-elasticsearch.adoc
4+
5+
[id="efk-logging-elasticsearch-status-example-{context}"]
6+
= Viewing Elasticsearch component status
7+
8+
You can view the status for a number of Elasticsearch components.
9+
10+
Elasticsearch indices::
11+
You can view the status of the Elasticsearch indices.
12+
13+
. Get the name of an Elasticsearch pod:
14+
+
15+
----
16+
$ oc get pods --selector component=elasticsearch -o name
17+
18+
pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
19+
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
20+
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7
21+
----
22+
23+
. Get the status of the indices:
24+
+
25+
----
26+
$ oc exec elasticsearch-cdm-1godmszn-1-6f8495-vp4lw -- indices
27+
28+
Defaulting container name to elasticsearch.
29+
Use 'oc describe pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw -n openshift-logging' to see all of the containers in this pod.
30+
Wed Apr 10 05:42:12 UTC 2019
31+
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
32+
red open .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac N7iCbRjSSc2bGhn8Cpc7Jg 2 1
33+
green open .operations.2019.04.10 GTewEJEzQjaus9QjvBBnGg 3 1 2176114 0 3929 1956
34+
green open .operations.2019.04.11 ausZHoKxTNOoBvv9RlXfrw 3 1 1494624 0 2947 1475
35+
green open .kibana 9Fltn1D0QHSnFMXpphZ--Q 1 1 1 0 0 0
36+
green open .searchguard chOwDnQlSsqhfSPcot1Yiw 1 1 5 1 0 0
37+
----
38+
39+
40+
Elasticsearch pods::
41+
You can view the status of the Elasticsearch pods.
42+
43+
. Get the name of a pod:
44+
+
45+
----
46+
$ oc get pods --selector component=elasticsearch -o name
47+
48+
pod/elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
49+
pod/elasticsearch-cdm-1godmszn-2-5769cf-9ms2n
50+
pod/elasticsearch-cdm-1godmszn-3-f66f7d-zqkz7
51+
----
52+
53+
. Get the status of a pod:
54+
+
55+
----
56+
oc describe pod elasticsearch-cdm-1godmszn-1-6f8495-vp4lw
57+
----
58+
+
59+
The output includes the following status information:
60+
+
61+
----
62+
....
63+
Status: Running
64+
65+
....
66+
67+
Containers:
68+
elasticsearch:
69+
Container ID: cri-o://b7d44e0a9ea486e27f47763f5bb4c39dfd2
70+
State: Running
71+
Started: Mon, 08 Apr 2019 10:17:56 -0400
72+
Ready: True
73+
Restart Count: 0
74+
Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
75+
76+
....
77+
78+
proxy:
79+
Container ID: cri-o://3f77032abaddbb1652c116278652908dc01860320b8a4e741d06894b2f8f9aa1
80+
State: Running
81+
Started: Mon, 08 Apr 2019 10:18:38 -0400
82+
Ready: True
83+
Restart Count: 0
84+
85+
....
86+
87+
Conditions:
88+
Type Status
89+
Initialized True
90+
Ready True
91+
ContainersReady True
92+
PodScheduled True
93+
94+
....
95+
96+
Events: <none>
97+
----
98+
99+
Elasticsearch deployment configuration::
100+
You can view the status of the Elasticsearch deployment configuration.
101+
102+
. Get the name of a deployment configuration:
103+
+
104+
----
105+
$ oc get deployment --selector component=elasticsearch -o name
106+
107+
deployment.extensions/elasticsearch-cdm-1gon-1
108+
deployment.extensions/elasticsearch-cdm-1gon-2
109+
deployment.extensions/elasticsearch-cdm-1gon-3
110+
----
111+
112+
. Get the deployment configuration status:
113+
+
114+
----
115+
$ oc describe deployment elasticsearch-cdm-1gon-1
116+
----
117+
+
118+
The output includes the following status information:
119+
+
120+
----
121+
....
122+
Containers:
123+
elasticsearch:
124+
Image: quay.io/openshift/origin-logging-elasticsearch5:latest
125+
Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
126+
127+
....
128+
129+
Conditions:
130+
Type Status Reason
131+
---- ------ ------
132+
Progressing Unknown DeploymentPaused
133+
Available True MinimumReplicasAvailable
134+
135+
....
136+
137+
Events: <none>
138+
----
139+
140+
Elasticsearch ReplicaSet::
141+
You can view the status of the Elasticsearch ReplicaSet.
142+
143+
. Get the name of a replica set:
144+
+
145+
----
146+
$ oc get replicaSet --selector component=elasticsearch -o name
147+
148+
replicaset.extensions/elasticsearch-cdm-1gon-1-6f8495
149+
replicaset.extensions/elasticsearch-cdm-1gon-2-5769cf
150+
replicaset.extensions/elasticsearch-cdm-1gon-3-f66f7d
151+
----
152+
153+
. Get the status of the replica set:
154+
+
155+
----
156+
$ oc describe replicaSet elasticsearch-cdm-1gon-1-6f8495
157+
----
158+
+
159+
The output includes the following status information:
160+
+
161+
----
162+
....
163+
Containers:
164+
elasticsearch:
165+
Image: quay.io/openshift/origin-logging-elasticsearch5:latest
166+
Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
167+
168+
....
169+
170+
Events: <none>
171+
----
172+
173+

‎modules/efk-logging-elasticsearch-status.adoc

Lines changed: 115 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -13,53 +13,118 @@ You can view the status of your Elasticsearch cluster.
1313

1414
.Procedure
1515

16-
Run the following command to view the Elasticsearch status:
17-
18-
----
19-
$ oc get elasticsearch elasticsearch -o yaml
20-
21-
nodes:
22-
- deploymentName: elasticsearch-clientdatamaster-0-1
23-
podName: elasticsearch-clientdatamaster-0-1-84d764899d-bh7jl
24-
replicaSetName: elasticsearch-clientdatamaster-0-1-84d764899d
25-
roles:
26-
- client
27-
- data
28-
- master
29-
status: Running
30-
upgradeStatus:
31-
underUpgrade: "False"
32-
- deploymentName: elasticsearch-data-1-1
33-
podName: elasticsearch-data-1-1-77ffddbf7b-zdd76
34-
replicaSetName: elasticsearch-data-1-1-77ffddbf7b
35-
roles:
36-
- data
37-
status: Running
38-
upgradeStatus:
39-
underUpgrade: "False"
40-
- podName: elasticsearch-client-2-1-0
41-
roles:
42-
- client
43-
statefulSetName: elasticsearch-client-2-1
44-
status: Running
45-
upgradeStatus:
46-
underUpgrade: "False"
47-
pods:
48-
client:
49-
failed: []
50-
notReady:
51-
- elasticsearch-client-1-1-0
52-
- elasticsearch-client-2-1-0
53-
ready: []
54-
data:
55-
failed: []
56-
notReady:
57-
- elasticsearch-data-1-1-77ffddbf7b-zdd76
58-
ready: []
59-
master:
60-
failed: []
61-
notReady: []
62-
ready: []
63-
shardAllocationEnabled: "True"
64-
----
65-
16+
. Change to the `openshift-logging` project.
17+
+
18+
----
19+
$ oc project openshift-logging
20+
----
21+
22+
. To view the Elasticsearch cluster status:
23+
24+
.. Get the name of the Elasticsearch instance:
25+
+
26+
----
27+
$ oc get Elasticsearch
28+
29+
NAME AGE
30+
elasticsearch 5h9m
31+
----
32+
33+
.. Get the Elasticsearch status:
34+
+
35+
----
36+
$ oc get Elasticsearch <Elasticsearch-instance> -o yaml
37+
----
38+
+
39+
For example:
40+
+
41+
----
42+
$ oc get Elasticsearch elasticsearch -n openshift-logging -o yaml
43+
----
44+
+
45+
The output includes information similar to the following:
46+
+
47+
----
48+
status: <1>
49+
clusterHealth: green <2>
50+
conditions: <3>
51+
.....
52+
nodes: <4>
53+
.....
54+
pods: <5>
55+
.....
56+
shardAllocationEnabled: "True" <6>
57+
----
58+
<1> In the output, the cluster status fields appear in the `status` stanza.
59+
<2> The status of the Elasticsearch cluster, `green`, `red`, `yellow`.
60+
<3> Any status conditions, if present. The Elasticsearch cluster status indicates the reasons from the scheduler if a pod could not be placed. Any events related to the following conditions are shown:
61+
* Container Waiting for both the Elasticsearch and proxy containers,
62+
* Container Terminated for both the Elasticsearch and proxy containers,
63+
* Pod unschedulable.
64+
Also, a condition is shown if the node storage passes the high or low watermark thresholds.
65+
<4> The Elasticsearch nodes in the cluster, with `upgradeStatus`.
66+
<5> The Elasticsearch client, data, and master pods in the cluster, listed under 'failed`, `notReady` or `ready` state.
67+
68+
69+
[id="efk-logging-elasticsearch-status-message-{context}"]
70+
== Example condition messages
71+
72+
The following are examples of some condition messages from the `Status.Nodes` section of the Elasticsearch instance.
73+
74+
75+
// https://github.com/openshift/elasticsearch-operator/pull/92
76+
77+
This status message indicates a node has exceeded the configured low watermark and no shard will be allocated to this node.
78+
79+
----
80+
nodes:
81+
- conditions:
82+
- lastTransitionTime: 2019-03-15T15:57:22Z
83+
message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
84+
be allocated on this node.
85+
reason: Disk Watermark Low
86+
status: "True"
87+
type: NodeStorage
88+
deploymentName: example-elasticsearch-clientdatamaster-0-1
89+
upgradeStatus: {}
90+
----
91+
92+
This status message indicates a node has exceeded the configured high watermark and shard will be relocated to other nodes.
93+
94+
----
95+
nodes:
96+
- conditions:
97+
- lastTransitionTime: 2019-03-15T16:04:45Z
98+
message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
99+
from this node.
100+
reason: Disk Watermark High
101+
status: "True"
102+
type: NodeStorage
103+
deploymentName: example-elasticsearch-clientdatamaster-0-1
104+
upgradeStatus: {}
105+
----
106+
107+
This status message shows the Elasticsearch node selector in the CR does not match any nodes in the cluster:
108+
109+
----
110+
nodes:
111+
- conditions:
112+
- lastTransitionTime: 2019-04-10T02:26:24Z
113+
message: '0/8 nodes are available: 8 node(s) didn''t match node selector.'
114+
reason: Unschedulable
115+
status: "True"
116+
type: Unschedulable
117+
----
118+
119+
This status message indicates that the Elasticsearch CR uses a non-existent PVC.
120+
121+
----
122+
nodes:
123+
- conditions:
124+
- last Transition Time: 2019-04-10T05:55:51Z
125+
message: pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
126+
reason: Unschedulable
127+
status: True
128+
type: Unschedulable
129+
----
130+

0 commit comments

Comments
 (0)