Healthcheck fails when used with Amazon Managed Grafana

[philipnrmn]

During initiation of the Grafana object, we hit /api/health to ensure the API is available. Unfortunately Amazon Managed Grafana instances return a 400 error from the health endpoint. This makes the snapshot tool unusable with AMG.

We should keep the existing behaviour as default and add an amg configuration option which skips the check or falls back to /healthz instead.

[amotl]

Hi Philip,

thank you for writing in. It looks like this issue has already been addressed within other projects of this organization.

• grafana_client.client.GrafanaClientError: Client Error 404: Not found grafana-client#72
• Doesn't work for simplest example grafana-wtf#113
• Support AWS managed API grafana-wtf#141
• FAILED tests/test_commands.py::test_plugins_status_app - AttributeError: health grafana-wtf#161

Maybe it will be possible to derive a corresponding patch from those insights? Would you be up for submitting one?

Optimally, we'd not need a dedicated configuration option at all, when optimizing for a happy path that works for most users, always assuming most recent Grafana stable, and only optionally falling back to using additional configuration options when applicable.

With kind regards,
Andreas.

/cc @ImerysOTDataAndArchitecture, @interfan7, @squadgazzz, @bhks

[amotl]

Maybe the code at the relevant spot can be relaxed like this?

#* try to connect to the API 
try: 
  res = self.grafana_api.health.check() 
  if res['database'] != 'ok':
    raise Exception('Grafana not healthy')
except: 
  logger.warning('Grafana not healthy, or using AMG')

[peekjef72]

Hi,

@amotl not sure where a fix is required.

Maybe it should be set in grafana-client :

diff --git a/grafana_client/elements/health.py b/grafana_client/elements/health.py
index ca00880..3fc92d6 100644
--- a/grafana_client/elements/health.py
+++ b/grafana_client/elements/health.py
@@ -1,5 +1,5 @@
 from .base import Base
-
+from ..client import GrafanaClientError
 
 class Health(Base):
     def __init__(self, client):
@@ -11,5 +11,22 @@ class Health(Base):
 
         :return:
         """
+        check_frontend = False
         path = "/health"
-        return self.client.GET(path)
+
+        try:
+            client_info = self.client.GET(path)
+        except GrafanaClientError as ex:
+            if ex.status_code == 404:
+                check_frontend = True
+            else:
+                raise
+        if check_frontend:
+            path = "/frontend/settings"
+            try:
+                frontend_info = self.client.GET(path)
+                if frontend_info is not None and isinstance(frontend_info, (dict)) and "buildInfo" in frontend_info:
+                    client_info = frontend_info["buildInfo"]
+            except:
+                raise
+        return client_info

Do you think it is a correct approach ?
If so, i could set an update.

[amotl]

Hi. It might make sense to support it there, yes.

Then, let's possibly have a concrete connotation that it is explicitly about AMG support, implemented turtles all the way up and down that both developers and users can toggle that switch, either per Grafana Client Python API argument, and/or by defining a corresponding environment variable, like GRAFANA_AMG=true ¹, on downstream applications like grafana-wtf, grafana-import, or grafana-snapshots-tool?

Footnotes

1. CLI flag per --amg or --flavor=amg may also be sensible, or even elevated smartness in a future iteration, by learning this fact from the supplied URL itself, when applicable? ↩

[peekjef72]

I don't have access to AWS so to AMG too. So can't check to make my mind.
My first opinion was that if the AMG API is only different for health, maybe it is not necessary to have to flag to distinguish them.
But in fact you are right, it is easier to manage a flag to drive the difference between the two implementation; the options would be set to grafana_client init call via ENV or option like you propose. Maybe --flavor= should be prefered, so we can reuse the flag if there'd others custom grafana in future.

[amotl]

Hi. I am not 100% sure about the design / interface / implementation yet, I am just sharing a few ideas here. I think we may have a few reviewers on grafana-client, who will provide guidance on how to optimally integrate this little "anomaly compensation feature" for AMG users, while not complicating things too much for the majority of others. ¹

While it sounds wild, it may also be possible to toggle the feature using a query parameter to the Grafana API URL you are supplying, i.e. treating it more like a database connection string in the future. We already started to go into this direction by permitting the URL parameter ?verify=false to relay SSL connection options.

In the same spirit, the feature can additionally respect an ?amg=true query parameter, or alternatively, using a more functional-like naming, something like ?healthcheck=false. wdyt?

Footnotes

1. We should possibly continue the discussion there, or, even easier, effectively just transfer this ticket? ↩

[bhks]

for AMG, API based health check is not open and it fails many times, it is actually very deep and also checks plugin servers health I might be wrong here.

However if we are talking about AMG just hit the grafana workspace endpoint and /healthz like below
https://g-xxxxxxxxx.grafana-workspace.us-east-1.amazonaws.com/healthz

[squadgazzz]

For AMG, another endpoint needs to be used: grafana-toolbox/grafana-wtf#144

[amotl]

Hi. Thanks for your swift replies. I've created a dedicated issue over at grafana-client.

• Support for Amazon Managed Grafana (AMG) grafana-client#216

Please let us know over there if you agree with the plan, and feel free to share any specific suggestions. Anyone reading this who would be up for submitting a patch will receive great praise. 🌻

[amotl]

Hi again. Finally, we've submitted a patch over here. Because of potential breaking changes, this is meant to be released as grafana-client 5.0.0.

• AMG compatibility: Fetch Grafana version from /api/frontend/settings grafana-client#229

[amotl]

Hi. grafana-client 5.0.0 has been released, and this patch updates grafana-snapshots to use it:

• Refresh dependencies, update to grafana-client >= 5 #8

@peekjef72: Can you validate, merge, and release?

[amotl]

Dear @philipnrmn. Can we humbly ask you to check if the most recent version of grafana-snapshots works better for you when using AMG?