Replies: 7 comments 8 replies
-
|
Thanks for raising these concerns! We understand that this experience is not as smooth as we would like it ourselves. This is mostly a result of the operator resources trying to adhere to different constraints and solve additional uses on top of "just" provisioning them (like linking between resources by k8s identifiers). Your input into what you tried & expected is immensely valuable! We'll try to get our documentation updated to better illustrate the configuration model of the operator. For the specific issues of alerts, have you found this page in our documentation: https://grafana.github.io/grafana-operator/docs/alerting/ It contains a full alerting setup including folder, rules, groups and notifications |
Beta Was this translation helpful? Give feedback.
-
Thank you @theSuess Yeah, this is the documentation I was following. Let me outline where I observed apparent deficiencies:
Ok, so it seems like the scheme exported from the legacy side has potential fundamental differences than that of the operated side So I am gonna have to script a way around this I think. More to come. |
Beta Was this translation helpful? Give feedback.
-
OK, I had some time to try and script around this and I was hoping I could just take a straight copy of the ❯ k -n grafana apply -f vmc-1m_folder.yaml -f vmc-1m_masterdb-cpu-alert_alert.yaml
OK: This is DEV...
grafanafolder.grafana.integreatly.org/vmc-1m created
Error from server (BadRequest): error when creating "vmc-1m_masterdb-cpu-alert_alert.yaml": GrafanaAlertRuleGroup in version "v1beta1" cannot be handled as a GrafanaAlertRuleGroup: strict decoding error: unknown field "spec.rules[0].dashboardUid", unknown field "spec.rules[0].notification_settings", unknown field "spec.rules[0].panelId"The scheme in the source rule doesn't match that of what the operator desires it seems. Now, looking at these objects, it makes sense to me why these probably shouldn't carry over as they are likely dynamic fields. We also have not yet attempted to migrate notification policies and contact policies over yet. However, the |
Beta Was this translation helpful? Give feedback.
-
|
@theSuess there is a mildly troubling problem we've just uncovered and I'm hoping you can shed some light on how to overcome this. I've successfully imported alerts and the folders those alerts belong (which are evaluation groups I believe). The problem is that the folders I generated for the alerts, since they are using the notice anything with an evaluation period (1m, 5m, etc) is not technically a dashboard folder, but the following folders are what are showing up as dashboard folders erroneously:
I suspected this might happen. Is this a bug? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @notjames It is up to the user to design their folder structure for their use case. The Grafana Operator, to my knowledge, can represent any valid folder structure which you can create in Grafana. If you want to split Evaluation groups(GrafanaAlertRuleGroup) and Dashboards(GrafanaDashboard) into separate folders(GrafanaFolder), you need to create at least two folder resources. Otherwise, you can have both your Evaluation groups(GrafanaAlertRuleGroup) and dashboards(GrafanaDashboard) reference the same folder and they will appear in different tabs when navigating to the folder.
I hope this clears up some misconceptions 😄 |
Beta Was this translation helpful? Give feedback.
-
OK, that's a fair distinction, but the misconception comes from how our legacy 11.4.x grafana creates alertgroups via UI configuration compared to my trying to create the same using provisioning in the operator. I'm trying to create "sub-folders" or evaluation groups and as I showed above, that is not working. Help me understand how I can do that. I'll show screenshots to show what I'm trying to get. non-operated: alerts: Hopefully it's clear what I'm trying to accomplish. From the two screenshots here, you can see the groups the alerts are in. The operator documentation doesn't make a clear path on how I can replicate my current setup from legacy to operator. |
Beta Was this translation helpful? Give feedback.
-
|
I partially recreated the above. Yaml resourcesapiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
labels:
nested: example
name: grafana-nested
spec:
config:
security:
admin_password: secret
admin_user: root
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
name: testdata
spec:
instanceSelector:
matchLabels:
nested: example
uid: testdata-datasource
datasource:
isDefault: true
name: grafana-testdata-datasource
type: grafana-testdata-datasource
access: proxy
basicAuth: false
---
#
# EKS Folders and alerts
#
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: eks
spec:
title: EKS
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: eks-cluster-metrics
spec:
title: Cluster Metrics
parentFolderRef: eks
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: eks-compute-resources
spec:
title: Compute Resources
parentFolderRef: eks
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: eks-system
spec:
title: System
parentFolderRef: eks
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
name: eks-cluster-metrics-5m
spec:
name: EKS 5m
folderRef: eks-cluster-metrics
instanceSelector:
matchLabels:
nested: example
interval: 5m
rules:
- uid: uid-eks-cluster-metrics
title: Test
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: grafana-testdata
model:
drop: 0
intervalMs: 1000
labels: ""
max: 50
maxDataPoints: 43200
min: -50
noise: 0
refId: A
scenarioId: random_walk
seriesCount: 3
spread: 5
startValue: 0
stringInput: 1,20,90,30,5,0
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: A
hide: false
intervalMs: 1000
maxDataPoints: 43200
reducer: max
refId: B
settings:
mode: dropNN
type: reduce
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 45
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: B
hide: false
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
noDataState: NoData
execErrState: Error
for: 5m
annotations: {}
labels: {}
isPaused: false
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
name: non-eks-system-5m
spec:
name: NON-EKS-5M
folderRef: eks-system
instanceSelector:
matchLabels:
nested: example
interval: 5m
rules:
- uid: uid-non-eks-system-5m
title: Test
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: grafana-testdata
model:
drop: 0
intervalMs: 1000
labels: ""
max: 50
maxDataPoints: 43200
min: -50
noise: 0
refId: A
scenarioId: random_walk
seriesCount: 3
spread: 5
startValue: 0
stringInput: 1,20,90,30,5,0
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: A
hide: false
intervalMs: 1000
maxDataPoints: 43200
reducer: max
refId: B
settings:
mode: dropNN
type: reduce
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 45
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: B
hide: false
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
noDataState: NoData
execErrState: Error
for: 5m
annotations: {}
labels: {}
isPaused: false
#
# SRE Folder
#
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: sre
spec:
title: SRE
instanceSelector:
matchLabels:
nested: example
#
# VMC Folders and alerts
#
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: vmc
spec:
title: VMC
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: vmc-compute-resources
spec:
title: Compute Resources
parentFolderRef: vmc
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
name: vmc-compute-resources-1m
spec:
name: VMC-1m
folderRef: vmc-compute-resources
instanceSelector:
matchLabels:
nested: example
interval: 1m
rules:
- uid: uid-vmc-compute-resources
title: Test
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: grafana-testdata
model:
drop: 0
intervalMs: 1000
labels: ""
max: 50
maxDataPoints: 43200
min: -50
noise: 0
refId: A
scenarioId: random_walk
seriesCount: 3
spread: 5
startValue: 0
stringInput: 1,20,90,30,5,0
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: A
hide: false
intervalMs: 1000
maxDataPoints: 43200
reducer: max
refId: B
settings:
mode: dropNN
type: reduce
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 45
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: B
hide: false
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
noDataState: NoData
execErrState: Error
for: 1m
annotations: {}
labels: {}
isPaused: false
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
name: vmc-system
spec:
title: System
parentFolderRef: vmc
instanceSelector:
matchLabels:
nested: example
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaAlertRuleGroup
metadata:
name: vmc-system-5m
spec:
name: VMC-5m
folderRef: vmc-system
instanceSelector:
matchLabels:
nested: example
interval: 5m
rules:
- uid: uid-vmc-system-5m
title: Test
condition: C
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: grafana-testdata
model:
drop: 0
intervalMs: 1000
labels: ""
max: 50
maxDataPoints: 43200
min: -50
noise: 0
refId: A
scenarioId: random_walk
seriesCount: 3
spread: 5
startValue: 0
stringInput: 1,20,90,30,5,0
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 0
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: A
hide: false
intervalMs: 1000
maxDataPoints: 43200
reducer: max
refId: B
settings:
mode: dropNN
type: reduce
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
conditions:
- evaluator:
params:
- 45
- 0
type: gt
operator:
type: and
query:
params: []
reducer:
params: []
type: avg
type: query
datasource:
name: Expression
type: __expr__
uid: __expr__
expression: B
hide: false
intervalMs: 1000
maxDataPoints: 43200
refId: C
type: threshold
noDataState: NoData
execErrState: Error
for: 5m
annotations: {}
labels: {}
isPaused: false |
Beta Was this translation helpful? Give feedback.
-
|
after setting up alerts/for my org, my advice with Grafana folders and the alerts is Then for each rule under the evaluation group:
|
Beta Was this translation helpful? Give feedback.
-
OK, let me try this out. Thank you. I'll let you know my progress and success. |
Beta Was this translation helpful? Give feedback.
-
|
OK, I've developed a script that is mostly working but the problem I'm running into now is perhaps something you guys eluded to. Situation: my script generates folder manifests from our legacy environment. The operator spec says I can prescribe the folder UID in the manifest so my script is just pulling the UID from the legacy environment. There are two problems.
So I've tried prescribing a UID in both I know that @faidon-passias stated in his bullet In my use-case, this is very less than ideal because we're generating manifests to be applied by argocd. Now there's a chicken-n-egg issue where I can't apply rule manifests without introspecting the operator for folder UIDs, but the operator shouldn't have to be running for me to configure the operator properly. Is this a bug in the operator? If this is confirmed to be a bug in the operator. I am looking at @Baarsgaard 's YAML manifests though, and instead of using |
Beta Was this translation helpful? Give feedback.
-
|
If the The |
Beta Was this translation helpful? Give feedback.
-
Sure enough when our infra team updated our operator, they didn't update the CRDs. So that's been resolved. I've already refactored the code to use the |
Beta Was this translation helpful? Give feedback.
-
|
This was a good conversation which I believe have answeredd most of my questions. I'm going to jump into contact and notification policies now. Is #1529 still relevant? |
Beta Was this translation helpful? Give feedback.
-
|
It's not as clean as we want it to be, but the workaround mentioned in the issue (multiple contact points with the same name) is what works best for now. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your help @theSuess and @Baarsgaard |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
Apologies. This is mildly lengthy, but these are all related to one another:
It seems like the CRs for grafana operator for things like alert rules groups, notification policies, and contact policies are still young? So far I've only tried migrating alert rule groups and that's proving to be pretty complicated. I was hoping that migrating the exported bits from the non operated grafana into a CR would "just work." but that doesn't work at all. In fact, it appears that the concept of rule groups doesn't exist in the CR despite the name of the CR being called
GrafanaAlertRuleGroupWe're using Grafana exclusively for everything to manage TSD visualization and alerting. I was hoping I could just
Export Rulesfrom the alert rules page (which exports all rules groups system wide for readers' context) and place them in the yaml property of the GrafanaAlertRuleGroup. That, of course, doesn't work. The yaml property spec gets violated by several properties of each rule in that yaml output.OK cool, so I guess I need to go through each rule and export each rule into a single GrafanaAlertRuleGroup. But that leaves questions about how to import each group into their respective evaluation groups. I suppose those would be folders? The documentation doesn't address that. Moreover, the folders, while mentioned, are ambiguous with respect to how those coexist with alert folders (or not).
Contact points will be a problem, too, it seems: #1529
I haven't looked into notification policies yet.
However, migrating alerts went great!
Version
5.17.0-debian-12-r0
To Reproduce
Steps to reproduce the behavior:
asked and answered
Expected behavior
Confirmation of my observations
Maybe documentation that exists (hopefully) that disambiguates steps to take to migrate from non operated grafana to operated grafana for notification policies, contact policies, and alert rules/groups
Suspect component/Location where the bug might be occurring
unknown(NA)Screenshots
If applicable, add screenshots to help explain your problem.
Runtime (please complete the following information):
Additional context
NA
Beta Was this translation helpful? Give feedback.
All reactions