TooManyRequests errors

[sfairat15]

Hello!

We often receive a lot TooManyRequests errors when we have 12 custom metrics to collect. According documentation there may be only 30 API calls per minute per scope, per user, or application.

Looks like you should use retries or precalculate metrics in background process and save them in memory.

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMEPIOGNTKLE", Request id "0HMEPIOGNTKLE:00000001": An unhandled exception was thrown by the application.
System.Exception: TooManyRequests {"error":{"code":"429","message":"RateLimitingFilter::OnActionExecutingAsync - Too many requests. Please retry."}}
   at AzureBillingExporter.AzureApi.AzureCostManagementClient.ExecuteBillingQuery(String billingQuery, CancellationToken cancel, BillingQueryClient billingQueryClient)+MoveNext() in /app/AzureCostManagementClient.cs:line 56
   at AzureBillingExporter.AzureBillingMetricsGrapper.DownloadFromApi(CancellationToken cancel) in /app/MetricsGrapper.cs:line 60
   at AzureBillingExporter.AzureBillingMetricsGrapper.DownloadFromApi(CancellationToken cancel) in /app/MetricsGrapper.cs:line 60
   at AzureBillingExporter.Startup.<>c__DisplayClass5_0.<<Configure>b__0>d.MoveNext() in /app/Startup.cs:line 72
--- End of stack trace from previous location where exception was thrown ---
   at Prometheus.CollectorRegistry.CollectAndSerializeAsync(IMetricsSerializer serializer, CancellationToken cancel)
   at Prometheus.MetricServerMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)