Skip to content

Optimize yarn workspace discovery #1430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Optimize yarn workspace discovery #1430

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d3d15c0
Optimize yarn workspace discovery
andrian-sevastyanov May 16, 2025
74ed1c2
Change workspace collection back to two levels
andrian-sevastyanov May 16, 2025
89f5473
Release 10.5.0-SIGQA4-andrians.IDETECT-4713-yarn-workspace-glob-optim…
May 16, 2025
c3c18b9
Using the next snapshot post release 10.5.0-SIGQA5-andrians.IDETECT-4…
May 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions ...rc/main/java/com/blackduck/integration/detectable/detectables/yarn/YarnLockExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,7 @@ private ExcludedIncludedWildcardFilter deriveExcludedIncludedWildcardFilter() {

@NotNull
private YarnWorkspaces collectWorkspaceData(File dir) throws IOException {
Collection<YarnWorkspace> curLevelWorkspaces = packageJsonFiles.readWorkspacePackageJsonFiles(dir);
Collection<YarnWorkspace> allWorkspaces = new LinkedList<>(curLevelWorkspaces);
for (YarnWorkspace workspace : curLevelWorkspaces) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is unnecessary to try to find nested workspaces.

Nested workspaces are not supported at this time.

From https://classic.yarnpkg.com/lang/en/docs/workspaces/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That link looks like it covers yarn classic (1.x?). At the top there is a link to I believe more current documentation. It does seem like nested workspaces are supported in more recent versions? Seems like we support up to 4.1 so nested workspaces might be supported?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I'll look into this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't found any mention of nested workspaces in later documentation, so I did an experiment with Yarn 4.1.0:

  1. Create a project with workspaces having 3 levels (root points to workspace sub, which points to workspace sub1, which points to workspace sub2)
  2. In package sub2 package.json specify a dependency
  3. At root level run yarn install
  4. Now yarn.lock contains dependency from sub2

So, it seems like we do need to take nested workspaces into account for at least some versions. Furthermore, we probably need to do this recursively instead of at just two levels as we've been doing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back to two levels for now. We can explore doing more going forward.

Collection<YarnWorkspace> treeBranchWorkspacePackageJsons = packageJsonFiles.readWorkspacePackageJsonFiles(workspace.getWorkspacePackageJson().getDir());
allWorkspaces.addAll(treeBranchWorkspacePackageJsons);
}
Collection<YarnWorkspace> allWorkspaces = packageJsonFiles.readWorkspacePackageJsonFiles(dir);
return new YarnWorkspaces(allWorkspaces);
}
}
65 changes: 42 additions & 23 deletions ...a/com/blackduck/integration/detectable/detectables/yarn/packagejson/PackageJsonFiles.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,42 +38,61 @@ public NullSafePackageJson read(File packageJsonFile) throws IOException {

@NotNull
public Collection<YarnWorkspace> readWorkspacePackageJsonFiles(File workspaceDir) throws IOException {
String forwardSlashedWorkspaceDirPath = deriveForwardSlashedPath(workspaceDir);
File packageJsonFile = new File(workspaceDir, YarnLockDetectable.YARN_PACKAGE_JSON);
List<String> workspaceDirPatterns = extractWorkspaceDirPatterns(packageJsonFile);

if (workspaceDirPatterns.isEmpty()) {
logger.debug("No workspace patterns found in {}", packageJsonFile.getAbsolutePath());
return new LinkedList<>();
}

List<PathMatcher> matchers = convertWorkspaceDirPatternsToPathMatchers(workspaceDirPatterns, workspaceDir);

Collection<YarnWorkspace> workspaces = new LinkedList<>();
for (String workspaceSubdirPattern : workspaceDirPatterns) {
logger.trace("workspaceSubdirPattern: {}", workspaceSubdirPattern);
String globString = String.format("glob:%s/%s/package.json", forwardSlashedWorkspaceDirPath, workspaceSubdirPattern);
logger.trace("workspace subdir globString: {}", globString);
PathMatcher matcher = FileSystems.getDefault().getPathMatcher(globString);
Files.walkFileTree(workspaceDir.toPath(), new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (matcher.matches(file)) {
logger.trace("\tFound a match: {}", file);
NullSafePackageJson packageJson = read(file.toFile());
Path rel = workspaceDir.toPath().relativize(file.getParent());
WorkspacePackageJson workspacePackageJson = new WorkspacePackageJson(file.toFile(), packageJson, rel.toString());
YarnWorkspace workspace = new YarnWorkspace(workspacePackageJson);
workspaces.add(workspace);

Files.walkFileTree(workspaceDir.toPath(), new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (file.getFileName().toString().equals(YarnLockDetectable.YARN_PACKAGE_JSON)) { // no need to try matching if not package.json
for (PathMatcher matcher : matchers) {
if (matcher.matches(file)) {
logger.trace("\tFound a match: {}", file);
NullSafePackageJson packageJson = read(file.toFile());
Path rel = workspaceDir.toPath().relativize(file.getParent());
WorkspacePackageJson workspacePackageJson = new WorkspacePackageJson(file.toFile(), packageJson, rel.toString());
YarnWorkspace workspace = new YarnWorkspace(workspacePackageJson);
workspaces.add(workspace);
break; // no need to match the same file multiple times
}
}
return FileVisitResult.CONTINUE;
}
return FileVisitResult.CONTINUE;
}

@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) {
return FileVisitResult.CONTINUE;
}
});

@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) {
return FileVisitResult.CONTINUE;
}
});
}
if (!workspaceDirPatterns.isEmpty()) {
logger.debug("Found {} matching workspace package.json files for workspaces listed in {}", workspaces.size(), packageJsonFile.getAbsolutePath());
}
return workspaces;
}

private List<PathMatcher> convertWorkspaceDirPatternsToPathMatchers(List<String> workspaceDirPatterns, File workspaceDir) {
List<PathMatcher> matchers = new LinkedList<>();
String forwardSlashedWorkspaceDirPath = deriveForwardSlashedPath(workspaceDir);
for (String workspaceDirPattern : workspaceDirPatterns) {
String globString = String.format("glob:%s/%s/package.json", forwardSlashedWorkspaceDirPath, workspaceDirPattern);
logger.trace("workspace subdir globString: {}", globString);
PathMatcher matcher = FileSystems.getDefault().getPathMatcher(globString);
matchers.add(matcher);
}
return matchers;
}

@NotNull
private String deriveForwardSlashedPath(File file) {
String forwardSlashWorkspaceDirPath;
Expand Down