Grafana OnCall is sunsetting; is Keep a potential alternative?

[IngwiePhoenix]

Hello there!

Long story short, Grafana has announced that OnCall OSS is entering maintenance mode and will sunset next year. However, at the company I work at, we used OnCall as a way to display all the alrts from all the orgs within Grafana - we even use Git provisioning to deploy them.

So I would like to know if Keep would perhaps be a suitable alternative to migrate to? That is, if it supports receiving alerts from Grafana, which we have built out (20-ish orgs with 3-10 dashboards each, all of them have alert rules attached).

Thank you and kind regards,
Ingwie

[dosubot[bot]]

Hey there, @IngwiePhoenix! I'm a bot here to assist you with bug fixes, questions, and more. Let's tackle this together! 😊

Keep could be a suitable alternative for migrating from Grafana OnCall. It supports integration with Grafana for receiving alerts, allowing you to manage alerts from multiple organizations and dashboards centrally. Keep provides a centralized dashboard for managing alerts, enabling you to review, throttle, mute, and fine-tune alerts from a single console. This can help streamline your alert management across the 20-ish organizations and multiple dashboards you mentioned ^[1][2].

However, I couldn't find specific information about Keep's support for Git provisioning for deploying alerts. You might need to explore this aspect further to ensure it meets your requirements.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[shahargl]

hey @IngwiePhoenix - nice to have you here.

yea this is definitely possible.

You can receive alerts from Grafana and slice and dice the alerts however you want.

You can also create workflows that will automatically enrich your alerts with other data sources - so if you have database or change management systems (e.g. github) that store more data (like commits, versions, etc) - you can enrich the alerts with this data.

You can also create correlation rules to correlate these into incident or deduplication rules to reduce the noise from Grafana Alerts.

You can think of Keep as a "one-stop shop" for all of your alerts.

PS - saw your discussion on alerta (alerta/alerta#2023 (comment)) - basically we see a lot of folks migrating from Alerta to Keep so we basically support everything there.

We have a pretty active Slack community - will be happy to continue the discussion there and see how we can solve your use case, feel free to jump in - https://slack.keephq.dev/

[IngwiePhoenix]

I set up a demo instance at my work's laptop today.

Personally, Alerta seems like more suited for a developer-focused environment (which would be my homelab because I live in the CLI) whilst Keep is, and I genuenly like that quote, "a single pane of glass" more suited for a heavily graphical environment.

So I will probably roll out Keep at work and Alerta at home.

One tiny aside; in the demo, it was showing me the AIOps stuff. We don't have neither the infra, nor API keys. Is it possible to disable entirely unused UI elements?

Further, when configuring Grafana as a data source provider (old habbits die hard...) can I set it to be in a read-only state so I can see what our current alerts would "look like" in Keep? Making some SAs and tokens is hardly a problem, I just wouldn't want to interrupt my coworkers currently still using OnCall while quietly tiptapping away on the rollout.

Hm... I am not sure if I still have my Slack creds, actually. Will see if I can re-find them - I am very used to Matrix and Discord, not so much to Slack. Well, I'll try. :)

(Also, marking Dosu as the answer was fully intended. Where'd you find this thing? That was a spot-on, actually helpful answer. First time an AI support thing actually worked. o.o Makes me quite convinced that your AIOps has some good work behind it!)

Kind regards,
Ingwie

[shahargl]

Cool to hear you tried it! We can hop on a call to show you the product and how people use it and see if I can help.

Regarding "read-only" - yea its possible, you can just send alerts from Grafana without connecting it and it will just work. Go to your grafana, set up a webhook on contact points, and edit your notification policies to send it to Keep.

Regarding UI stuff to disable - what did you have in mind that you want to disable?

[IngwiePhoenix]

Thanks, I will give this a try at my next opportunity!

As for UI, just the bottom left "AIOps" panel. It's no biggie if it can not be removed, just don't really personally like having buttons that effectively do nothing (since we have neither the infra nor the subscription) x)

I'll get back to you about that call once I dug a lot deeper into Keep first; would prefer to come into this with some form of prep.

[Matvey-Kuk]

@IngwiePhoenix Hey, creator of OnCall (not related to Grafana anymore) and a team member at Keep here.

I would like to elaborate more on differences and similarities.

Grafana OnCall is a so-called "Incident Response Tool," its main focus is to be "the last destination for alerts", define the on-call engineer from the on-call schedule, notify him, and if the on-call engineer is absent, escalate.

Keep, as @shahargl mentioned, is a "toolbox for alerts" not focusing on "Incident Response", but providing de-noising, correlation, enrichment, and many other capabilities allowing you to process and access your alerts at scale, including AI. Keep may be, but not limited by design to be "the last destination".

Keep has a feature called "Workflows," allowing you to write almost any scenario for your alerts and execute them on the platform. Theoretically, one could hack a workflow to deliver alerts based on an on-call schedule and escalate.

If you want to send SMS and make Phone Calls to your on-call engineers, you still need some 3'rd party. You may use Keep+Twilio, but I suggest minimalistic use of PagerDuty (Just calendar and phone calling) and Keep for noise reduction, single pane of glass, enrichments, etc.

[IngwiePhoenix]

Oi, that's quite a surprise! Fancy meeting you. :)

We monitor our customer's data and watch for alerts in OnCall's "Grafana app" - it is quite likely not how either OnCall nor Keep is supposed to work - we are mainly a 1st/2nd level support company. But having a good way to see what systems at which customer is failing for what reason exactly is cruicial - and, so far, OnCall did that. But I don't know anymore how many tickets I opened, forum posts I had written or direct E-Mails I had sent - at some point, I was even having a call with the Grafana Team and learned that the enterprise on-prem licence is 40k €/year...

I am aware of Keep's additional features - and I tend to implement those in a few ways, but for the most important bit, most to all of my collegues will just pop it open on a secondary monitor, watch for incomming alerts and act upon seeing them (opening a ticket, calling the affected customer or directly acting by logging in via VPN and solving the issue). That workflow has helped us a lot. And this is what I have been looking for; something I can set up instead of OnCall that can give us this kind of screen. Heck, we have a 60" TV here that we used to just put OnCall on next to all the support desks. xD It looked silly, but it was also kinda nice. Sadly I have not found a single 10ft-UI for this kind of stuff, lol. :)

Thank you for your elaborations, they're very useful! I looked at PagerDuty but could not find a way to host it on-prem (which is a hard-requirement - one of our customers is _extra_picky...)