Skip to content

Commit 6cb9616

Browse files
leipertljharb
authored andcommitted
[utils] [perf] Performance of fullResolve
While looking at a larger code base https://gitlab.com/gitlab-org/gitlab, I've came to realize that `fullResolve` takes a lot of CPU cycles, particularly the `hashObject` calls inside it. I applied the following patch locally to see how often this is called and how many unique hashes were produced: ```diff diff --git a/utils/resolve.js b/utils/resolve.js index 4a35c6a..3c28324 100644 --- a/utils/resolve.js +++ b/utils/resolve.js @@ -83,13 +83,28 @@ function relative(modulePath, sourceFile, settings) { return fullResolve(modulePath, sourceFile, settings).path; } +let prevSettings = null; +let nonEqualSettings = 0; +let totalCalls = 0; +let uniqueHashes = new Set(); function fullResolve(modulePath, sourceFile, settings) { // check if this is a bonus core module const coreSet = new Set(settings['import/core-modules']); if (coreSet.has(modulePath)) return { found: true, path: null }; const sourceDir = path.dirname(sourceFile); - const cacheKey = sourceDir + hashObject(settings).digest('hex') + modulePath; + + totalCalls+=1; + const hash = hashObject(settings).digest('hex'); + + if(prevSettings !== settings){ + uniqueHashes.add(hash); + prevSettings = settings; + nonEqualSettings+=1; + console.log(`fullResolve | Total calls:${totalCalls} | Non-Equal settings:${nonEqualSettings} | Unique hashes:${uniqueHashes.size} | dir:${sourceDir}`) + } + + const cacheKey = sourceDir + hash + modulePath; const cacheSettings = ModuleCache.getSettings(settings); ``` For our code base, `fullResolve` is called more than 570 thousand times. The simple in-equality `!==` code path is taken 1090 times. Actually only _four_ unique hashes are produced, meaning we only have four unique settings across our code base. I assume that a full object equality comparison might not be needed, and a simple object comparison with `!==` already would reduce the amount of `hashObject` calls by 570x. This is what is implemented in this commit. Time spend in `fullResolve` was reduced by ~38%: - Before: 17% (19.10s) of our total execution time - After: 11% (11.86s) of our total execution time The effect might even be more pronounced on machines that are slower when calculating `sha256` hashes or that have less memory, as the `hashObject` method tends to create loads of small strings which need to be garbage collected.
1 parent d2b10ae commit 6cb9616

File tree

2 files changed

+14
-1
lines changed

2 files changed

+14
-1
lines changed

utils/CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ This change log adheres to standards from [Keep a CHANGELOG](https://keepachange
55

66
## Unreleased
77

8+
### Fixed
9+
- Improve performance of `fullResolve` for large projects ([#2755], thanks [@leipert])
10+
811
## v2.7.4 - 2022-08-11
912

1013
### Fixed
@@ -123,6 +126,7 @@ Yanked due to critical issue with cache key resulting from #839.
123126
### Fixed
124127
- `unambiguous.test()` regex is now properly in multiline mode
125128

129+
[#2755]: https://github.com/import-js/eslint-plugin-import/pull/2755
126130
[#2523]: https://github.com/import-js/eslint-plugin-import/pull/2523
127131
[#2431]: https://github.com/import-js/eslint-plugin-import/pull/2431
128132
[#2350]: https://github.com/import-js/eslint-plugin-import/issues/2350
@@ -160,6 +164,7 @@ Yanked due to critical issue with cache key resulting from #839.
160164
[@iamnapo]: https://github.com/iamnapo
161165
[@JounQin]: https://github.com/JounQin
162166
[@kaiyoma]: https://github.com/kaiyoma
167+
[@leipert]: https://github.com/leipert
163168
[@manuth]: https://github.com/manuth
164169
[@maxkomarychev]: https://github.com/maxkomarychev
165170
[@mgwalker]: https://github.com/mgwalker

utils/resolve.js

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,13 +83,21 @@ function relative(modulePath, sourceFile, settings) {
8383
return fullResolve(modulePath, sourceFile, settings).path;
8484
}
8585

86+
let prevSettings = null;
87+
let memoizedHash = '';
8688
function fullResolve(modulePath, sourceFile, settings) {
8789
// check if this is a bonus core module
8890
const coreSet = new Set(settings['import/core-modules']);
8991
if (coreSet.has(modulePath)) return { found: true, path: null };
9092

9193
const sourceDir = path.dirname(sourceFile);
92-
const cacheKey = sourceDir + hashObject(settings).digest('hex') + modulePath;
94+
95+
if (prevSettings !== settings) {
96+
memoizedHash = hashObject(settings).digest('hex');
97+
prevSettings = settings;
98+
}
99+
100+
const cacheKey = sourceDir + memoizedHash + modulePath;
93101

94102
const cacheSettings = ModuleCache.getSettings(settings);
95103

0 commit comments

Comments
 (0)