"Flood" alerting for story volume spikes.

[pgulley]

The downstream story-indexer processes have an edge-case where the fetcher queues can get full and clogged if there is a sudden influx of new stories from a source, especially when those stories throw errors and end up recycling through the re-fetching queues. It would be nice to have an alert in this process when a source starts producing a significantly larger volume of sources, so that we can be alerted ahead of time to keep an eye out.
I'm not personally so familiar with the inner workings of the RSS fetcher, so I'm not sure exactly where the detector would go for this signal. Once we can detect the signal, hooking it up to our existing alerting infrastructure should be straightforward!

[rahulbot]

Maybe a scripts/check_for_url_floods.py nightly cron job that does a count by domains for the ones with the most URLs in the last 3 days. For any domains with a count over X it sends a slack message to an alerts channel? Or any email? that query shouldn't be too costly to run as long as there is an index on the domain column. I think domain here is more meaningful than feed_id

[philbudne]

A simple aggregation (count + group by) query will find url count by domain, and the domain and fetched_at fields both have indices, so it's likely a quick query. Any message logged by a story-indexer program at level "error" will appear in the slack channel, courtesy of the Sentry integration. An alternative/addition would be to send the largest value seen to statsd/graphite/grafana, and have an alert set on it, which would give indication when the condition goes "green" after being red.

[philbudne]

The easiest thing to do was to add a query to the rss-stats-fetcher which does DB queries for display by vitals page.

The stats are:

stats.gauges.mc.REALM.rss-fetcher.rss-fetcher-stats.stories.top-domain.avg.days_3
stats.gauges.mc.REALM.rss-fetcher.rss-fetcher-stats.stories.top-domain.sum.days_3

Where sum.days_3 is the sum over the previous 72 hours and avg.3_days is the average per day over that window.
I added an alert to the rss-fetcher-alerts dashboard for avg.3_days over 50000, which could probably be lowered (agenzianova.com's total over three days is 21242)

[philbudne]

NOTE: agenzianova.com is top domain because I deleted all the story rows from africalog.com (as a brute-force effort to keep them away from the story-indexer).