Metric queries return null data until whisper file exists

[bitprophet]

Scenario

• You have a large Graphite install and/or are just stuck with slow disks, such that data enters carbon-cache and then doesn't reach disk for a number of minutes.
• You add new metric paths to your system that didn't exist before - new hosts or services came online, you renamed some metrics in your collectors, whatever.
• You are querying Graphite-web for these new metrics directly (aka bypassing the 'finder') via direct HTTP calls, or a dashboard, etc.
• Those queries result in no data until the above to-disk lag period passes.
• Users get sad because they made all the changes on their end (re: data collection) but then they have to wait a long time for validation/usefulness.

Cause

Querying the webapp for a specific metric path returns no data until it hits disk even if data is in the cache, due to the short-circuit linked here.

Solution

Update readers.py such that it performs a best-effort check of the cache for the requested metric path before giving up and returning None.

Downsides/challenges

• This may incur a (hopefully minor!) performance hit compared to existing behavior, in the case of truly-bogus metric path queries, but feels quite worth it, at least in my use case where lag is 20-30 minutes at times.
• Unless Carbon/CarbonLink is capable of servicing glob-expression requests (guessing not, given the abovelinked code) this will only solve things for fully explicit queries. Still useful some of the time, but not a full solution.
  • EDIT: Yup, the query is straight-up keying into the carbon-cache's cache dict. Could probably shoehorn in a way to handle globs, however, if core team thinks that's a reasonable thing to do.

[obfuscurity]

I'm really surprised to see that we don't attempt to hit the cache before disk. I'd like to get @mleinart's feedback on this proposed change and why we started returning None in 764bdf7 without checking the cache too.

[g76r]

@obfuscurity: I didn't talk to @mleinart but I had a look in the code to change that behaviour by myself.

IMHO it should be interesting to search the cache for series not yet present on disk, however it needs some changes since carbon cache store series in a hashtable, therefore resolving glob-expression will need another memory structure in addition to the hashtable (or, less likely, instead of the hashtable). This is probably why Graphite-web does not request carbon cache first.

[ocervell]

Any updates on this ? I am running into the same problem.

[deniszh]

@ocervell - unfortunately not yet.
We still need to attack this issue.

[deniszh]

It completely slips off our radars. Developers on my work hitting that constantly, I even using dirty hack for 0.9.x
Will try to make a fix for a master, but adding 1.1.0 milestone just in case.

[ocervell]

Issue is 3 years old. Any update ?
My company moving out of Graphite because of this.
I was still hoping that this issue would get resolved since I'm using Graphite in my own projects.

[DanCech]

The big thing that would be needed for this to work would be the ability for carbon to respond to a find request, since right now if the whisper file doesn't exist then the standard finder is never going to find the series and the code won't even get as far as calling the whisper reader.

If carbon were to provide a find method then it seems like it should be possible to move all the cache functionality out into a finder and reader for the cache, and handle merging the cached data into the final results via new the MultiReader mechanism. The biggest issue there seems to be aligning the data from the cache since it's stored with the raw timestamps and not aligned to a step.

[deniszh]

@ocervell : There're 2 main reasons why this issue is not in active development yet. First - that logic buried quite deep in whisper itself, and fix for this is quite big and massive, as @DanCech said.
On the other hand - if you will use SSDs and increase the number of your caches delay can be quite low and acceptable. I'm not saying that we should fix it, just explaining why even 3rd party graphite implementations still have this issue. :(

[Exocomp]

You have a large Graphite install and/or are just stuck with slow disks, such that data enters carbon-cache and then doesn't reach disk for a number of minutes.

I'm hitting the above scenario. I have to lower MAX_UPDATES_PER_SECOND because it kills the disk's throughput but that causes the cache to increase. Then queries start returning null for a few mintues because the cache is not being checked. ¯\_(ツ)_/¯

[AsenZahariev]

@deniszh , i saw that you included this issue into milestones 1.1.0 and 1.2.0, can you confirm in which version it will be definitely included ? Seems that we are struggling with the same problem.
Thank you!