Distrust received pack indexes (behind config flag, with perf fixes) #1846
Conversation
The GVFS protocol includes an index file along with pack file in the prefetch workflow (primarily used on a new clone to fetch all commits and trees). Currently, GVFS blindly accepts that index file. This pull request changes GVFS prefetch to discard the index sent by the server and always create an index locally. This provides improved security and verification of the pack file, at the expense of performance for the first clone of a repository on a new drive.
| /* Git's default thread count is Environment.ProcessorCount / 2, with a maximum of 20. | ||
| * Testing shows that we can get a 5% decrease in gvfs clone time for large repositories by using more threads, but | ||
| * we won't go over ProcessorCount or 20. */ | ||
| var threadCount = Math.Min(Environment.ProcessorCount, 20); |
There was a problem hiding this comment.
Is there any negative impact on clone times for machines that have many fewer available cores compared to threads?
There was a problem hiding this comment.
Do you mean, if there are cores with more hyperthreading available than the typical 2 threads per core?
In that case it's possible that there is even more room to raise the thread count, but in my experience Environment.ProcessorCount usually is the number of hardware threads rather than hardware cores (eg it will report 16 on an 8-core, 16-thread processor) so this will attempt to use all the hardware threads. The comments around git code for the default number of threads suggest that number of cores is more important than number of threads though, which is why they default to half (ie they assume 2 threads per core) and is probably why I only observed 5% improvement when doubling that. (Most of the overall improvement comes from running the index tasks for smaller pack files in parallel while the big one is still in its single-threaded phase, not from using more threads in the multi-threaded phase)
This pull request contains 4 things:
This is intended to be a temporary configuration setting, to be removed (or defaulted to false) once more mitigations have been completed for the performance.
git index-packis changed from running in the GVFS enlistment to running outside the enlistment. This was the reason for the unexpected performance issues.It was expected that the first prefetch on a new clone would take longer due to indexing the pack locally; however users who deleted their prefetch cache (but not the rest of the cache or local loose objects) in order to re-fetch it with local indexing enabled experienced many times longer delays than expected, because
git index-packreads all the existing pack indexes and loose objects and considers them when indexing a pack in order to support validating that all referenced objects exist - even when the command-line options to act on nonexistent references are not enabled. Since we aren't using --validate or its variants, we can rungit index-packoutside the enlistment to avoid this issue.