docs: Add more docs on cross-repo (#365)

Author: varungandhi-src

README.md

@@ -4,12 +4,18 @@ scip-clang is a precise code indexer based on Clang 16,
 which supports cross-repository code navigation for C and C++
 in Sourcegraph.
 
-Here are some code navigation examples in [llvm/llvm-project](https://sourcegraph.com/github.com/llvm/llvm-project):
-- [Find references for #include](https://sourcegraph.com/github.com/llvm/llvm-project@97a03eb2eb5acf269db6253fe540626b52950f97/-/blob/llvm/include/llvm/ADT/SmallSet.h?L1:1-1:81#tab=references)
-- [Find references for macros](https://sourcegraph.com/github.com/llvm/llvm-project@daad48d6b236d74c6b29daebba46289b98104241/-/blob/llvm/include/llvm/Support/Debug.h?L101:9-101:19#tab=references)
-- [Find references for types](https://sourcegraph.com/github.com/llvm/llvm-project@daad48d6b236d74c6b29daebba46289b98104241/-/blob/clang/include/clang/AST/ASTContext.h?L1472:34-1472:45#tab=references)
+Here are some code navigation examples:
+- [Cross-repository navigation in Boost](https://sourcegraph.com/github.com/boostorg/assert@f10ddd608e087a89ee5bfc41cf2987cc5ef61473/-/blob/include/boost/assert.hpp?L60:10-60:22#tab=references)
+- In [Chromium](https://sourcegraph.com/github.com/chromium/chromium@b21c706/-/blob/base/atomic_ref_count.h?L19:7-19:21#tab=references).
+- In [llvm/llvm-project](https://sourcegraph.com/github.com/llvm/llvm-project):
+  - [Find references for #include](https://sourcegraph.com/github.com/llvm/llvm-project@97a03eb2eb5acf269db6253fe540626b52950f97/-/blob/llvm/include/llvm/ADT/SmallSet.h?L1:1-1:81#tab=references)
+  - [Find references for macros](https://sourcegraph.com/github.com/llvm/llvm-project@daad48d6b236d74c6b29daebba46289b98104241/-/blob/llvm/include/llvm/Support/Debug.h?L101:9-101:19#tab=references)
+  - [Find references for types](https://sourcegraph.com/github.com/llvm/llvm-project@daad48d6b236d74c6b29daebba46289b98104241/-/blob/clang/include/clang/AST/ASTContext.h?L1472:34-1472:45#tab=references)
 
-You can also test out [code navigation in Chromium](https://sourcegraph.com/github.com/chromium/chromium@b21c706/-/blob/base/atomic_ref_count.h?L19:7-19:21#tab=references).
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://github.com/sourcegraph/scip-clang/assets/93103176/08b3aa95-c3ee-4c56-9920-20dfa4a7070d">
+  <img alt="Boost cross-repository Find References screenshot" src="https://github.com/sourcegraph/scip-clang/assets/93103176/1baf9a40-37ac-4896-bd1b-dc453730f91b">
+</picture>
 
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://github.com/sourcegraph/scip-clang/assets/93103176/54cc557e-16c8-4890-b9d1-b40d6e215084">

docs/CrossRepo.md

@@ -24,10 +24,14 @@ The package map JSON file contains a list of objects in the following format:
 ]
 ```
 
+As an example, you can see [scip-clang's own package map file](/tools/package-map.json).
+
 1. The `path` key may be an absolute path, or a path relative to the current directory
-   (which must be the project root).
+   (which must be the project root). For example:
+   - For projects using Bazel, these paths will generally look like:
+     `./bazel-myproject/external/com_company_libcool`.
 2. The `package` key consists of a `name` followed by an `@` separator and a `version`.
-   - The name and version must only contain characters belonging to `[a-zA-Z0-9_\-.]`.
+   - The name and version must only contain characters belonging to `[a-zA-Z0-9_\-\.]`.
    - The version should be chosen based on release information.
      For example, if you use git tags to mark releases in repos,
      and repositories only depend on tagged releases (instead of arbitrary commits),
@@ -44,7 +48,10 @@ Files under the directories `path/to/package1_root`
 will be treated as belonging to `package1`'s `v1` version.
 
 If one package root is a prefix of another, package information
-is assigned based on the longest match.
+is assigned based on the longest match. For example, if you're
+using git submodules, then packages in subdirectories will be
+recognized correctly if there is a package map entry
+pointing to the subdirectory.
 
 For cross-repository navigation to work,
 `package1` must also be indexed with the same version information:

docs/IndexingProjects.md

@@ -5,6 +5,11 @@
 For the sake of this example, I'll only describe how
 to index scip-clang with cross-repo code navigation support
 for [Abseil](https://github.com/abseil/abseil-cpp/).
+See [tools/package-map.json](/tools/package-map.json) for more entries
+
+> Aside: For a full index, one also needs to run `sed -e 's|$(STACK_FRAME_UNLIMITED)||'` on
+> the compilation database due to the unexpanded Make variable used
+> in compilation commands for LLVM which prevents type-checking from running.
 
 First, index Abseil.
 
@@ -28,6 +33,8 @@ jq '[.[] | select(.file | (contains("indexer/") or startswith("test/") or contai
 ./bazel-bin/indexer/scip-clang --compdb-path=min.json --package-map-path=package-map.json
 ```
 
+> Aside: If indexing `scip-clang` along with 
+
 Here, the `package-map.json` is as follows:
 
 ```json
@@ -40,6 +47,11 @@ Here, the `package-map.json` is as follows:
 The exact versions need to be determined based on current tag
 or hashes provided to Bazel.
 
+Other notes:
+- Indexing rapidjson requires a recursive clone since it uses GTest
+  via a submodule.
+- Indexing spdlog requires pass `-DSPDLOG_ENABLE_TESTING=ON` to cmake.
+
 ## LLVM
 
 Tested environments: Ubuntu 18.04, Ubuntu 22.04, macOS 13.
@@ -130,3 +142,54 @@ meson setup ../postgres-build .
 ninja -C ../postgres-build
 scip-clang --compdb-path=../postgres-build/compile_commands.json
 ```
+
+## Boost
+
+Here are the steps for a cross-repo indexing setup.
+
+> CAVEAT: At the time of writing, scip-clang doesn't have explicit support
+> for indexing multiple projects at once (see https://github.com/sourcegraph/scip-clang/issues/360)
+> so the instructions below will end up duplicating a lot of work,
+> since the compilation database is not pruned based on dependencies.
+>
+> The overall indexing will take about 96 core hours on a GCP N2 instance.
+
+First, do a recursive clone of the boost monorepo.
+
+```bash
+git clone https://github.com/boostorg/boost --recursive
+cd boost
+git checkout boost-1.82.0
+cmake -B build -DENABLE_TESTING=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
+```
+
+Invoke the indexer for each project with this Python script:
+
+```python
+import json
+import os
+import subprocess
+
+cwd = os.getcwd()
+
+dirs = []
+pmap = []
+for dirname in os.listdir("./libs"):
+    abs_path = "{}/libs/{}".format(cwd, dirname)
+    if not os.path.isdir(abs_path):
+        continue
+    dirs.append(abs_path)
+    pmap.append({"path": abs_path, "package": "boost-{}@boost-1.82.0".format(dirname)})
+
+with open('package-map.json', 'w') as f:
+    json.dump(pmap, f)
+
+for d in dirs:
+    if os.path.isfile(d + "/index.scip"):
+        continue
+    print("Indexing {}".format(d))
+    res = subprocess.run(["scip-clang --package-map-path=../../package-map.json --compdb-path=../../build/compile_commands.json"], cwd=d, shell=True)
+    if res.returncode != 0:
+        continue
+    subprocess.run(["src code-intel upload"], cwd=d, shell=True, env={"SRC_ACCESS_TOKEN": os.getenv("SRC_ACCESS_TOKEN"), "PATH": os.getenv("PATH")})
+```

tools/package-map.json

@@ -0,0 +1,12 @@
+[
+    {"path": ".", "package": "scip-clang@134a873"},
+    {"path": "./bazel-scip-clang/external/com_google_absl", "package": "abseil-cpp@4ffaea74"},
+    {"path": "/home/varun/.cache/bazel/_bazel_varun/e45ff3cfa68a557b274978dc134d6467/external/llvm-raw", "package": "llvm-project@e0f3110b854a"},
+    {"path": "./bazel-scip-clang/external/boost/libs/date_time", "package": "boost-date_time@boost-1.82.0"},
+    {"path": "./bazel-scip-clang/external/boost/libs/interprocess", "package": "boost-interprocess@boost-1.82.0"},
+    {"path": "./bazel-scip-clang/external/boost/libs/process", "package": "boost-process@boost-1.82.0"},
+    {"path": "./bazel-scip-clang/external/rapidjson", "package": "rapidjson@a98e9999"},
+    {"path": "./bazel-scip-clang/external/cxxopts", "package": "cxxopts@3.0.0"},
+    {"path": "./bazel-scip-clang/external/spdlog", "package": "spdlog@edc51df1"},
+    {"path": "./bazel-scip-clang/external/doctest", "package": "doctest@2.4.9"}
+]