List all the URLs that Bazel accesses in a build/fetch?

[tmewett]

I am trying to build a Bazel project in a highly controlled environment. One of our requirements is that we do all required downloads ourselves, before the build. In effect, we want to know in advance all URLs that are going to be accessed during the build.

Is there any way at all to get this information out of Bazel? I've trawled through the repository cache and bazel vendor output directories, looking to see if there's enough metadata, but I've been unsuccessful so far.

(It's not viable for us to use such pre-downloaded directories directly, as we want to be able to configure download URL rewriting for mirroring.)

It would be amazing if there was a Bazel sub-command which could just list all the URLs.

Is anyone else working on this problem?

[meteorcloudy]

There is no such feature in Bazel.
But I can think of two way to look into this:

• Inspect the MODULE.bazel.lock file, you'll see at least all registry file URLs needed.
• Use --downloader_config to redirect all URLs to a proxy server (and also add the original one as backup so that the build can continue), and let the proxy server record all URLs requested, it should be very easy to vibe code one.

[tmewett]

Ah, that's a shame.. Those are good ideas to explore, however I also just realised I need more information to construct a cache/vendor dir for Bazel to read.

To create a repository cache, I also need "canonical IDs" for the sources too (by default the list of URLs specified in the rule).

But then I also don't think I can put git_repository's in the repository cache, since they aren't cached there?

There's a lot of undefined information here which makes this difficult to do automatically. Right now we are having to repeatedly, manually:

• attempt the build with no internet
• identify a failing repository target
• find the rule invocation
• manually add it to our repository cache builder

[meteorcloudy]

Theoretically, repository rules can do anything like calling git or some random tool to fetch sources, besides calling repository_ctx.download. There is no way for Bazel to know in advance what kind of network access will be required. And that's exactly why we implemented vendor mode so that you have all repositories pre-fetched, which makes sure you no longer need network access. Can you please explain a bit why https://bazel.build/external/vendor doesn't meet your needs?

[meteorcloudy]

To create a repository cache, I also need "canonical IDs" for the sources too (by default the list of URLs specified in the rule).

For the download URLs, you can mirror then and use --downloader_config to use the mirrored URL, you don't need to manually create a repository cache for it. See https://github.com/bazelbuild/bazel/blob/master/bazel_downloader.cfg as an example

[tmewett]

Can you please explain a bit why https://bazel.build/external/vendor doesn't meet your needs?

1. Vendor has a bug: bazel vendor does not support offline builds on clean installs #26806

2. We want our system to do all downloads itself, for compliance reasons (we know exactly what's being downloaded) and for mirroring (we can systematically change URLs across the entire system). As you note, Bazel has a URL rewriter, but our system does not work via rewriting, so we can't easily apply the configuration to Bazel - and I'm not sure it meets our compliance requirements anyway.

Theoretically, repository rules can do anything like calling git or some random tool to fetch sources, besides calling repository_ctx.download. There is no way for Bazel to know in advance what kind of network access will be required.

I see - I guess we will have to use the MODULE.bazel.lock, and then manually source the rest. I think that could still be a significant improvement.