Merge branch 'main' into add-activerecord-annotate

Author: thiggy1342

docs/codeql/query-help/cpp.rst

@@ -3,7 +3,9 @@ CodeQL query help for C and C++
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/cpp/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/cpp-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/cpp/ql/examples>`__.
 
 .. include:: toc-cpp.rst
 

docs/codeql/query-help/csharp.rst

@@ -3,6 +3,8 @@ CodeQL query help for C#
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/csharp/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/csharp-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/csharp/ql/examples>`__.
 
 .. include:: toc-csharp.rst

docs/codeql/query-help/go.rst

@@ -3,6 +3,8 @@ CodeQL query help for Go
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/go/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/go-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/go/ql/examples>`__.
 
 .. include:: toc-go.rst

docs/codeql/query-help/java.rst

@@ -3,6 +3,8 @@ CodeQL query help for Java
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/java/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/java-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/java/ql/examples>`__.
 
 .. include:: toc-java.rst

docs/codeql/query-help/javascript.rst

@@ -3,6 +3,8 @@ CodeQL query help for JavaScript
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/javascript/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/javascript-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/javascript/ql/examples>`__.
 
 .. include:: toc-javascript.rst

docs/codeql/query-help/python.rst

@@ -3,6 +3,8 @@ CodeQL query help for Python
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/python/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/python-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/python/ql/examples>`__.
 
 .. include:: toc-python.rst

docs/codeql/query-help/ruby.rst

@@ -3,6 +3,8 @@ CodeQL query help for Ruby
 
 .. include:: ../reusables/query-help-overview.rst
 
-For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/main/ruby/ql/examples>`__.
+These queries are published in the CodeQL query pack ``codeql/ruby-queries`` (`changelog <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/src/CHANGELOG.md>`__, `source <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/src>`__).
+
+For shorter queries that you can use as building blocks when writing your own queries, see the `example queries in the CodeQL repository <https://github.com/github/codeql/tree/codeql-cli/latest/ruby/ql/examples>`__.
 
 .. include:: toc-ruby.rst

swift/codegen/schema.yml

@@ -323,6 +323,7 @@ AppliedPropertyWrapperExpr:
   _extends: Expr
 
 Argument:
+  _extends: Locatable
   label: string
   _children:
     expr: Expr

swift/extractor/SwiftExtractor.cpp

@@ -1,11 +1,7 @@
 #include "SwiftExtractor.h"
 
-#include <filesystem>
-#include <fstream>
 #include <iostream>
-#include <sstream>
 #include <memory>
-#include <unistd.h>
 #include <unordered_set>
 #include <queue>
 
@@ -16,8 +12,9 @@
 #include <llvm/Support/Path.h>
 
 #include "swift/extractor/trap/generated/TrapClasses.h"
-#include "swift/extractor/trap/TrapOutput.h"
+#include "swift/extractor/trap/TrapDomain.h"
 #include "swift/extractor/visitors/SwiftVisitor.h"
+#include "swift/extractor/infra/TargetFile.h"
 
 using namespace codeql;
 
@@ -52,7 +49,7 @@ static void archiveFile(const SwiftExtractorConfiguration& config, swift::Source
   }
 }
 
-static std::string getTrapFilename(swift::ModuleDecl& module, swift::SourceFile* primaryFile) {
+static std::string getFilename(swift::ModuleDecl& module, swift::SourceFile* primaryFile) {
   if (primaryFile) {
     return primaryFile->getFilename().str();
   }
@@ -76,60 +73,45 @@ static llvm::SmallVector<swift::Decl*> getTopLevelDecls(swift::ModuleDecl& modul
   return ret;
 }
 
+static void dumpArgs(TargetFile& out, const SwiftExtractorConfiguration& config) {
+  out << "/* extractor-args:\n";
+  for (const auto& opt : config.frontendOptions) {
+    out << "  " << std::quoted(opt) << " \\\n";
+  }
+  out << "\n*/\n";
+
+  out << "/* swift-frontend-args:\n";
+  for (const auto& opt : config.patchedFrontendOptions) {
+    out << "  " << std::quoted(opt) << " \\\n";
+  }
+  out << "\n*/\n";
+}
+
 static void extractDeclarations(const SwiftExtractorConfiguration& config,
                                 swift::CompilerInstance& compiler,
                                 swift::ModuleDecl& module,
                                 swift::SourceFile* primaryFile = nullptr) {
+  auto filename = getFilename(module, primaryFile);
+
   // The extractor can be called several times from different processes with
-  // the same input file(s)
-  // We are using PID to avoid concurrent access
-  // TODO: find a more robust approach to avoid collisions?
-  auto name = getTrapFilename(module, primaryFile);
-  llvm::StringRef filename(name);
-  std::string tempTrapName = filename.str() + '.' + std::to_string(getpid()) + ".trap";
-  llvm::SmallString<PATH_MAX> tempTrapPath(config.getTempTrapDir());
-  llvm::sys::path::append(tempTrapPath, tempTrapName);
-
-  llvm::StringRef tempTrapParent = llvm::sys::path::parent_path(tempTrapPath);
-  if (std::error_code ec = llvm::sys::fs::create_directories(tempTrapParent)) {
-    std::cerr << "Cannot create temp trap directory '" << tempTrapParent.str()
-              << "': " << ec.message() << "\n";
+  // the same input file(s). Using `TargetFile` the first process will win, and the following
+  // will just skip the work
+  auto trapTarget = TargetFile::create(filename + ".trap", config.trapDir, config.getTempTrapDir());
+  if (!trapTarget) {
+    // another process arrived first, nothing to do for us
     return;
   }
-
-  std::ofstream trapStream(tempTrapPath.str().str());
-  if (!trapStream) {
-    std::error_code ec;
-    ec.assign(errno, std::generic_category());
-    std::cerr << "Cannot create temp trap file '" << tempTrapPath.str().str()
-              << "': " << ec.message() << "\n";
-    return;
-  }
-  trapStream << "/* extractor-args:\n";
-  for (auto opt : config.frontendOptions) {
-    trapStream << "  " << std::quoted(opt) << " \\\n";
-  }
-  trapStream << "\n*/\n";
-
-  trapStream << "/* swift-frontend-args:\n";
-  for (auto opt : config.patchedFrontendOptions) {
-    trapStream << "  " << std::quoted(opt) << " \\\n";
-  }
-  trapStream << "\n*/\n";
-
-  TrapOutput trap{trapStream};
-  TrapArena arena{};
+  dumpArgs(*trapTarget, config);
+  TrapDomain trap{*trapTarget};
 
   // TODO: move default location emission elsewhere, possibly in a separate global trap file
-  auto unknownFileLabel = arena.allocateLabel<FileTag>();
   // the following cannot conflict with actual files as those have an absolute path starting with /
-  trap.assignKey(unknownFileLabel, "unknown");
+  auto unknownFileLabel = trap.createLabel<FileTag>("unknown");
+  auto unknownLocationLabel = trap.createLabel<LocationTag>("unknown");
   trap.emit(FilesTrap{unknownFileLabel});
-  auto unknownLocationLabel = arena.allocateLabel<LocationTag>();
-  trap.assignKey(unknownLocationLabel, "unknown");
   trap.emit(LocationsTrap{unknownLocationLabel, unknownFileLabel});
 
-  SwiftVisitor visitor(compiler.getSourceMgr(), arena, trap, module, primaryFile);
+  SwiftVisitor visitor(compiler.getSourceMgr(), trap, module, primaryFile);
   auto topLevelDecls = getTopLevelDecls(module, primaryFile);
   for (auto decl : topLevelDecls) {
     visitor.extract(decl);
@@ -142,28 +124,9 @@ static void extractDeclarations(const SwiftExtractorConfiguration& config,
     // fact that the file was extracted
     llvm::SmallString<PATH_MAX> name(filename);
     llvm::sys::fs::make_absolute(name);
-    auto fileLabel = arena.allocateLabel<FileTag>();
-    trap.assignKey(fileLabel, name.str().str());
+    auto fileLabel = trap.createLabel<FileTag>(name.str().str());
     trap.emit(FilesTrap{fileLabel, name.str().str()});
   }
-
-  // TODO: Pick a better name to avoid collisions
-  std::string trapName = filename.str() + ".trap";
-  llvm::SmallString<PATH_MAX> trapPath(config.trapDir);
-  llvm::sys::path::append(trapPath, trapName);
-
-  llvm::StringRef trapParent = llvm::sys::path::parent_path(trapPath);
-  if (std::error_code ec = llvm::sys::fs::create_directories(trapParent)) {
-    std::cerr << "Cannot create trap directory '" << trapParent.str() << "': " << ec.message()
-              << "\n";
-    return;
-  }
-
-  // TODO: The last process wins. Should we do better than that?
-  if (std::error_code ec = llvm::sys::fs::rename(tempTrapPath, trapPath)) {
-    std::cerr << "Cannot rename temp trap file '" << tempTrapPath.str().str() << "' -> '"
-              << trapPath.str().str() << "': " << ec.message() << "\n";
-  }
 }
 
 static std::unordered_set<std::string> collectInputFilenames(swift::CompilerInstance& compiler) {

swift/extractor/infra/BUILD.bazel

@@ -2,9 +2,11 @@ load("//swift:rules.bzl", "swift_cc_library")
 
 swift_cc_library(
     name = "infra",
+    srcs = glob(["*.cpp"]),
     hdrs = glob(["*.h"]),
     visibility = ["//swift:__subpackages__"],
     deps = [
         "//swift/extractor/trap",
+        "//swift/tools/prebuilt:swift-llvm-support",
     ],
 )