pytorch now supports the XDG Base Directory specification pytorch/pytorch#14693.
The pytorchcv module, provided by this repo is still using the hardcoded ~/.torch/models path.
You can see the "correct" logic for finding the .torch cache directory here.