Skip to content

[rush] Optimize the execution speed of Rush #5007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Dec 12, 2024
Merged

[rush] Optimize the execution speed of Rush #5007

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2eb0665
Optimize the execution speed of Rush
L-Qun Nov 18, 2024
1c4e825
update documentation
L-Qun Nov 18, 2024
12c0c46
Update documentation
L-Qun Nov 18, 2024
493e2ab
add enableSubpathScan to control if full scan the entire repository
L-Qun Nov 23, 2024
238d1f3
Merge branch 'microsoft:main' into main
L-Qun Nov 27, 2024
75a0ad0
Update common/changes/@rushstack/package-deps-hash/main_2024-11-18-08…
L-Qun Dec 11, 2024
72940eb
Update libraries/package-deps-hash/src/getRepoState.ts
L-Qun Dec 11, 2024
6236ccd
Update libraries/package-deps-hash/src/getRepoState.ts
L-Qun Dec 11, 2024
4c094f8
Update common/changes/@microsoft/rush/main_2024-11-18-08-13.json
L-Qun Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions common/changes/@microsoft/rush/main_2024-11-18-08-13.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"packageName": "@microsoft/rush",
"comment": "Optimize the execution speed of Rush",
"type": "none"
}
],
"packageName": "@microsoft/rush"
}
10 changes: 10 additions & 0 deletions common/changes/@rushstack/package-deps-hash/main_2024-11-18-08-13.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"packageName": "@rushstack/package-deps-hash",
"comment": "Optimize the execution speed of Rush",
"type": "patch"
}
],
"packageName": "@rushstack/package-deps-hash"
}
2 changes: 1 addition & 1 deletion common/reviews/api/package-deps-hash.api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ export function getRepoChanges(currentWorkingDirectory: string, revision?: strin
export function getRepoRoot(currentWorkingDirectory: string, gitPath?: string): string;

// @beta
export function getRepoStateAsync(rootDirectory: string, additionalRelativePathsToHash?: string[], gitPath?: string): Promise<Map<string, string>>;
export function getRepoStateAsync(rootDirectory: string, additionalRelativePathsToHash?: string[], gitPath?: string, filterPath?: string[]): Promise<Map<string, string>>;

// @beta
export function hashFilesAsync(rootDirectory: string, filesToHash: Iterable<string> | AsyncIterable<string>, gitPath?: string): Promise<Iterable<[string, string]>>;
Expand Down
2 changes: 1 addition & 1 deletion common/reviews/api/rush-lib.api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1129,7 +1129,7 @@ export class ProjectChangeAnalyzer {
// (undocumented)
protected getChangesByProject(lookup: LookupByPath<RushConfigurationProject>, changedFiles: Map<string, IFileDiffStatus>): Map<RushConfigurationProject, Map<string, IFileDiffStatus>>;
// @internal
_tryGetSnapshotProviderAsync(projectConfigurations: ReadonlyMap<RushConfigurationProject, RushProjectConfiguration>, terminal: ITerminal): Promise<GetInputsSnapshotAsyncFn | undefined>;
_tryGetSnapshotProviderAsync(projectConfigurations: ReadonlyMap<RushConfigurationProject, RushProjectConfiguration>, terminal: ITerminal, projectSelection?: ReadonlySet<RushConfigurationProject>): Promise<GetInputsSnapshotAsyncFn | undefined>;
}

// @public
Expand Down
9 changes: 6 additions & 3 deletions libraries/package-deps-hash/src/getRepoState.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,8 @@ export async function hashFilesAsync(
export async function getRepoStateAsync(
rootDirectory: string,
additionalRelativePathsToHash?: string[],
gitPath?: string
gitPath?: string,
filterPath?: string[]
): Promise<Map<string, string>> {
const statePromise: Promise<IGitTreeState> = spawnGitAsync(
gitPath,
Expand All @@ -378,7 +379,8 @@ export async function getRepoStateAsync(
'--full-name',
// As of last commit
'HEAD',
'--'
'--',
...(filterPath ? filterPath : [])
]),
rootDirectory
).then(parseGitLsTree);
Expand All @@ -396,7 +398,8 @@ export async function getRepoStateAsync(
'--ignore-submodules',
// Don't compare against the remote
'--no-ahead-behind',
'--'
'--',
...(filterPath ? filterPath : [])
]),
rootDirectory
).then(parseGitStatus);
Expand Down
3 changes: 2 additions & 1 deletion libraries/rush-lib/src/cli/scriptActions/PhasedScriptAction.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,7 @@ export class PhasedScriptAction extends BaseScriptAction<IPhasedCommandConfig> {
} = options;

const { projectConfigurations } = initialCreateOperationsContext;
const { projectSelection } = initialCreateOperationsContext;

const operations: Set<Operation> = await this.hooks.createOperations.promise(
new Set(),
Expand All @@ -558,7 +559,7 @@ export class PhasedScriptAction extends BaseScriptAction<IPhasedCommandConfig> {

const analyzer: ProjectChangeAnalyzer = new ProjectChangeAnalyzer(this.rushConfiguration);
const getInputsSnapshotAsync: GetInputsSnapshotAsyncFn | undefined =
await analyzer._tryGetSnapshotProviderAsync(projectConfigurations, terminal);
await analyzer._tryGetSnapshotProviderAsync(projectConfigurations, terminal, projectSelection);
const initialSnapshot: IInputsSnapshot | undefined = await getInputsSnapshotAsync?.();

repoStateStopwatch.stop();
Expand Down
7 changes: 5 additions & 2 deletions libraries/rush-lib/src/logic/ProjectChangeAnalyzer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,8 @@ export class ProjectChangeAnalyzer {
*/
public async _tryGetSnapshotProviderAsync(
projectConfigurations: ReadonlyMap<RushConfigurationProject, RushProjectConfiguration>,
terminal: ITerminal
terminal: ITerminal,
projectSelection?: ReadonlySet<RushConfigurationProject>
): Promise<GetInputsSnapshotAsyncFn | undefined> {
try {
const gitPath: string = this._git.getGitPathOrThrow();
Expand Down Expand Up @@ -295,10 +296,12 @@ export class ProjectChangeAnalyzer {
const lookupByPath: IReadonlyLookupByPath<RushConfigurationProject> =
this._rushConfiguration.getProjectLookupForRoot(rootDirectory);

const filterPath: string[] = Array.from(projectSelection ?? []).map((project) => project.projectFolder);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At minimum the choice to perform filtering needs to be behind a flag in experiments.json, because it will break things for consumers with custom plugins, and needs to be a choice whether or not to apply such logic.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a configuration to determine whether to enable this feature.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dmichon-msft, I've addressed the comments and made the updates. When you get a chance, could you please take another look? Thanks!


return async function tryGetSnapshotAsync(): Promise<IInputsSnapshot | undefined> {
try {
const [hashes, additionalFiles] = await Promise.all([
getRepoStateAsync(rootDirectory, additionalRelativePathsToHash, gitPath),
getRepoStateAsync(rootDirectory, additionalRelativePathsToHash, gitPath, filterPath),
getAdditionalFilesFromRushProjectConfigurationAsync(
additionalGlobs,
lookupByPath,
Expand Down
4 changes: 3 additions & 1 deletion libraries/rush-lib/src/logic/incremental/InputsSnapshot.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,9 @@ const { hashDelimiter } = RushConstants;
*/
export interface IInputsSnapshot {
/**
* The raw hashes of all tracked files in the repository.
* The raw hashes of the files relevant to the projects we care about are stored.
* (e.g. when running `rush build`, the hashes of all tracked files in the repository are stored)
* (e.g. when running `rush build --only`, only the hashes of files under the specified project are stored)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work. Computation of operation hashes depends on the entire tree of their dependencies, whether or not you are currently executing said dependencies. So at minimum we always need the expansion of --to, not just the values passed to --only to be able to determine build cache entry ids.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I just want to express that we are no longer storing all the hashes. I’ve removed that description.

*/
readonly hashes: ReadonlyMap<string, string>;

Expand Down
Loading