Performance overhead

[peterjc]

Testing with Biopython (which recently finished applying black to the entire code base), numbers on a multi-core Mac:

Black alone (best of three)

$ time black --check Bio/ BioSQL/ Tests/ Scripts/ Doc/
All done! ✨ 🍰 ✨
559 files would be left unchanged.

real	0m0.618s
user	0m0.484s
sys	0m0.129s

Using flake8 with assorted plugins, but without flake8-black (best of three)

$ time flake8 Bio/ BioSQL/ Tests/ Scripts/ Doc/

real	0m56.956s
user	4m1.520s
sys	0m3.470s

Adding the above together gives us an expected run time for running black via flake8.

So, same setup, but with flake8-black (best of three)

$ time flake8 Bio/ BioSQL/ Tests/ Scripts/ Doc/

real	1m36.095s
user	6m51.635s
sys	0m5.928s

That's an overhead of about 40s real and well over 2m user time - not good!

[peterjc]

I didn't get the expected speedup trying this on TravisCI, I now suspect the black cache is the reason why:

$ rm -rf /Users/$USER/Library/Caches/black/19.10b0/ && time black --check Bio/ BioSQL/ Tests/ Scripts/ Doc/
All done! ✨ 🍰 ✨
559 files would be left unchanged.

real	0m29.016s
user	3m7.108s
sys	0m3.005s

So, while we may not be able to dramatically speed up flake8-black on a first run, can we tap into the black cache?

[peterjc]

Or could flake8 include a similar caching mechanism? Black does this with a one cache per mode (covering the relatively short list of black options) per version of black. Doing something similar for flake8 would also have cover the combination of all installed plugins and their versions. This seems like it would be too complicated.

[peterjc]

We can probably take advantage of the black cache by imitating calling the black command line and letting black parse the file (rather than letting flake8 parse the file and giving the data to an internal black function). This will be a performance trade off - I suspect it will be faster for the local use case (e.g. git pre-commit hook), but could be slower for continuous integration (with no black cache present).