You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
= Configuring alert routing for default platform alerts
8
8
9
-
You can configure Alertmanager to send notifications. Customize where and how Alertmanager sends notifications about default platform alerts by editing the default configuration in the `alertmanager-main` secret in the `openshift-monitoring` namespace.
9
+
You can configure Alertmanager to send notifications to receive important alerts coming from your cluster. Customize where and how Alertmanager sends notifications about default platform alerts by editing the default configuration in the `alertmanager-main` secret in the `openshift-monitoring` namespace.
10
10
11
11
[NOTE]
12
12
====
@@ -16,28 +16,24 @@ All features of a supported version of upstream Alertmanager are also supported
16
16
.Prerequisites
17
17
18
18
* You have access to the cluster as a user with the `cluster-admin` cluster role.
19
+
* You have installed the {oc-first}.
19
20
20
21
.Procedure
21
22
22
-
. Open the Alertmanager YAML configuration file:
23
-
24
-
** To open the Alertmanager configuration from the CLI:
25
-
26
-
.. Print the currently active Alertmanager configuration from the `alertmanager-main` secret into `alertmanager.yaml` file:
23
+
. Extract the currently active Alertmanager configuration from the `alertmanager-main` secret and save it as a local `alertmanager.yaml` file:
27
24
+
28
25
[source,terminal]
29
26
----
30
27
$ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml
31
28
----
32
29
33
-
.. Open the `alertmanager.yaml` file.
30
+
. Open the `alertmanager.yaml` file.
34
31
35
-
** To open the Alertmanager configuration from the {product-title} web console:
32
+
. Edit the Alertmanager configuration:
36
33
37
-
.. Go to the *Administration* -> *Cluster Settings* -> *Configuration* -> *Alertmanager* -> *YAML* page of the web console.
38
-
39
-
. Edit the Alertmanager configuration by updating parameters in the YAML:
34
+
.. Optional: Change the default Alertmanager configuration:
40
35
+
36
+
.Example of the default Alertmanager secret YAML
41
37
[source,yaml]
42
38
----
43
39
global:
@@ -54,54 +50,88 @@ route:
54
50
- "alertname=Watchdog"
55
51
repeat_interval: 2m
56
52
receiver: watchdog
57
-
- matchers:
58
-
- "service=<your_service>" # <5>
59
-
routes:
60
-
- matchers:
61
-
- <your_matching_rules> # <6>
62
-
receiver: <receiver> # <7>
63
53
receivers:
64
54
- name: default
65
55
- name: watchdog
66
-
- name: <receiver>
67
-
<receiver_configuration> # <8>
68
56
----
69
57
<1> If you configured an HTTP cluster-wide proxy, set the `proxy_from_environment` parameter to `true` to enable proxying for all alert receivers.
70
58
<2> Specify how long Alertmanager waits while collecting initial alerts for a group of alerts before sending a notification.
71
59
<3> Specify how much time must elapse before Alertmanager sends a notification about new alerts added to a group of alerts for which an initial notification was already sent.
72
60
<4> Specify the minimum amount of time that must pass before an alert notification is repeated.
73
61
If you want a notification to repeat at each group interval, set the `repeat_interval` value to less than the `group_interval` value.
74
62
The repeated notification can still be delayed, for example, when certain Alertmanager pods are restarted or rescheduled.
75
-
<5> Specify the name of the service that fires the alerts.
76
-
<6> Specify labels to match your alerts.
77
-
<7> Specify the name of the receiver to use for the alerts.
78
-
<8> Specify the receiver configuration.
63
+
64
+
.. Add your alert receiver configuration:
79
65
+
80
-
[IMPORTANT]
81
-
====
82
-
* Use the `matchers` key name to indicate the matchers that an alert has to fulfill to match the node.
83
-
Do not use the `match` or `match_re` key names, which are both deprecated and planned for removal in a future release.
66
+
[source,yaml]
67
+
----
68
+
# ...
69
+
receivers:
70
+
- name: default
71
+
- name: watchdog
72
+
- name: <receiver> # <1>
73
+
<receiver_configuration> # <2>
74
+
# ...
75
+
----
76
+
<1> The name of the receiver.
77
+
<2> The receiver configuration. The supported receivers are PagerDuty, webhook, email, Slack, and Microsoft Teams.
78
+
+
79
+
.Example of configuring PagerDuty as an alert receiver
80
+
[source,yaml]
81
+
----
82
+
# ...
83
+
receivers:
84
+
- name: default
85
+
- name: watchdog
86
+
- name: team-frontend-page
87
+
pagerduty_configs:
88
+
- routing_key: ABCD01234EFGHIJ56789
89
+
http_config: # <1>
90
+
proxy_from_environment: true
91
+
authorization:
92
+
credentials: xxxxxxxxxx
93
+
# ...
94
+
----
95
+
<1> Optional: Add custom HTTP configuration for a specific receiver. That receiver does not inherit the global HTTP configuration settings.
84
96
85
-
* If you define inhibition rules, use the following key names:
97
+
.. Add the routing configuration:
86
98
+
87
-
--
88
-
** `target_matchers`: to indicate the target matchers
89
-
** `source_matchers`: to indicate the source matchers
90
-
--
99
+
[source,yaml]
100
+
----
101
+
# ...
102
+
route:
103
+
group_wait: 30s
104
+
group_interval: 5m
105
+
repeat_interval: 12h
106
+
receiver: default
107
+
routes:
108
+
- matchers:
109
+
- "alertname=Watchdog"
110
+
repeat_interval: 2m
111
+
receiver: watchdog
112
+
- matchers: # <1>
113
+
- "<your_matching_rules>" # <2>
114
+
receiver: <receiver> # <3>
115
+
# ...
116
+
----
117
+
<1> Use the `matchers` key name to specify the matching rules that an alert has to fulfill to match the node.
118
+
If you define inhibition rules, use `target_matchers` key name for target matchers and `source_matchers` key name for source matchers.
119
+
<2> Specify labels to match your alerts.
120
+
<3> Specify the name of the receiver to use for the alerts.
91
121
+
92
-
Do not use the `target_match`, `target_match_re`, `source_match`, or `source_match_re` key names, which are deprecated and planned for removal in a future release.
122
+
[WARNING]
123
+
====
124
+
Do not use the `match`, `match_re`, `target_match`, `target_match_re`, `source_match`, and `source_match_re` key names, which are deprecated and planned for removal in a future release.
93
125
====
94
126
+
95
-
.Example of Alertmanager configuration with PagerDuty as an alert receiver
127
+
--
128
+
.Example of alert routing
96
129
[source,yaml]
97
130
----
98
-
global:
99
-
resolve_timeout: 5m
100
-
http_config:
101
-
proxy_from_environment: true
131
+
# ...
102
132
route:
103
-
group_wait: 30s
104
-
group_interval: 5m
133
+
group_wait: 30s
134
+
group_interval: 5m
105
135
repeat_interval: 12h
106
136
receiver: default
107
137
routes:
@@ -111,31 +141,39 @@ route:
111
141
receiver: watchdog
112
142
- matchers: # <1>
113
143
- "service=example-app"
114
-
routes:
144
+
routes: # <2>
115
145
- matchers:
116
146
- "severity=critical"
117
147
receiver: team-frontend-page
118
-
receivers:
119
-
- name: default
120
-
- name: watchdog
121
-
- name: team-frontend-page
122
-
pagerduty_configs:
123
-
- service_key: "<your_key>"
124
-
http_config: # <2>
125
-
proxy_from_environment: true
126
-
authorization:
127
-
credentials: xxxxxxxxxx
148
+
# ...
128
149
----
129
-
<1> Alerts of `critical` severity that are fired by the `example-app` service are sent through the `team-frontend-page` receiver. Typically, these types of alerts would be paged to an individual or a critical response team.
130
-
<2> Custom HTTP configuration for a specific receiver. If you configure the custom HTTP configuration for a specific alert receiver, that receiver does not inherit the global HTTP config settings.
150
+
<1> This example matches alerts from the `example-app` service.
151
+
<2> You can create routes within other routes for more complex alert routing.
152
+
--
153
+
+
154
+
The previous example routes alerts of `critical` severity that are fired by the `example-app` service to the `team-frontend-page` receiver. Typically, these types of alerts are paged to an individual or a critical response team.
131
155
132
156
. Apply the new configuration in the file:
133
-
134
-
** To apply the changes from the CLI, run the following command:
0 commit comments