heroku.router.request.* anomalies?

[mezis]

Hi all!

Are there know issues with the heroku.router.request.* metrics?
We're seeing counts that are completely off (90% of the volume is missing in action), and the median/average connect/service time don't seem to add up with Heroku's own metrics dashboard, either.
This is starting to smell like parsing issues (ignoring many log lines) but I'm not having luck spotting the issue so far.

[w-vi]

I am not aware of any, there was an issue with the sample rate, but that's now 1.0 so it should not drop anything. For the parsing issues there might be some info in the logs https://github.com/apiaryio/heroku-datadog-drain-golang/blob/master/client.go#L91 or in https://github.com/apiaryio/heroku-datadog-drain-golang/blob/master/logproc.go#L54 have you seen any of those?

[mezis]

Nothing comes out at INFO or WARN levels, no. And DEBUG level isn't too useful at our request volumes (north of 120k req/min) - simply too much noise. I'll keep digging, and report here if I manage to figure it out :/

[w-vi]

AFAIK the delivery is done over UDP so maybe there is a big packet loss.

[w-vi]

maybe this is related DataDog/datadog-go#20

[abtris]

Is this still valid?