tc_pybind_example.py occasionally fails on tuning cache check #523

skimo-openhub · 2018-06-18T10:46:54Z

See, e.g., https://ci.pytorch.org/jenkins/job/tensorcomp-builds/job/tc-cuda9.0-cudnn7.1-ubuntu16.04-devel-build-test/259/consoleText

I was able to reproduce it on my system. top10 contains exactly one element, but it is different from top1. The element in top10 corresponds to the one in the protobuf file.

@ftynse

This commit supposedly addresses issue facebookresearch#523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

@ftynse

This commit supposedly addresses issue facebookresearch#523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

@ftynse

This commit supposedly addresses issue facebookresearch#523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

@ftynse

This commit supposedly addresses issue facebookresearch#523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

@ftynse

This commit supposedly addresses issue facebookresearch#523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

@ftynse

This commit supposedly addresses issue facebookresearch#523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

@ftynse

This commit supposedly addresses issue #523 (only supposedly because there is no easy repro). The problem is conjectured to come from the tuner keeping the best time/option in a private field whereas the functions that interact with the cache files operate on the cache. When multiple entries have the same runtime, it is conjectured (by @ftynse) that the ordering of the cache entries do not match the private field. In hindsight this can easily happen with thread/block sizes because once the number of threads/blocks is one per loop element, one can increase the values passed to mapping options but the same code will be generated after tightening. It is not too much of a stretch to imagine that the same code will occasionally have the same runtime. This commit drops the private state and ensures we always fetch the requires values from the options cache (under its lock).

skimo-openhub assigned nicolasvasilache Jun 18, 2018

nicolasvasilache mentioned this issue Jul 20, 2018

Drop duplicate state in tuner / options cache which resulted in intermittent disagreements between tune and load from cache #576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tc_pybind_example.py occasionally fails on tuning cache check #523

tc_pybind_example.py occasionally fails on tuning cache check #523

skimo-openhub commented Jun 18, 2018

tc_pybind_example.py occasionally fails on tuning cache check #523

tc_pybind_example.py occasionally fails on tuning cache check #523

Comments

skimo-openhub commented Jun 18, 2018