Skip to content

Do not mark cache generation as local #1533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

DavidANeil
Copy link
Contributor

local=True marks a repository rule to unconditionally re-execute anytime bazel re-evaluates the workspace (almost every invocation) It works well for processes that execute in <10ms, not large ones like the embuilder cache generation.

I have tested this locally and it seems to work fine, but I'll be honest that I don't understand the nuances of repository rules that well, nor do I know much of anything about emscripten or the circumstances in which the cache ought to be rebuilt.

I checked the external/@emscripten_cache.marker file that Bazel uses to store the cache key for if it needs to re-evaluate, and changing any of the options passed into register_emscripten_toolchains resulted in the cache key changing, and it correctly resulted in the cache being regenerated.

The big question I don't have the context to answer: are there files on disk other than these two that need to be watched for changes, so the repository rule should change if they change:

FILE:@@emscripten_bin_linux//BUILD.bazel
FILE:@@emsdk//emscripten_toolchain/default_config

local=True marks a repository rule to unconditionally re-execute anytime bazel re-evaluates the workspace (almost every invocation)
It works well for processes that execute in <10ms, not large ones like the embuilder cache generation.
@sbc100
Copy link
Collaborator

sbc100 commented Feb 21, 2025

unconditionally re-execute anytime bazel re-evaluates the workspace

Does that mean the cache was being completely repopulated every time bazel was run? Is that what you are seeing happen?

This code was added in #1405. Perhaps the creator of that PR has some insight here.

@DavidANeil
Copy link
Contributor Author

Does that mean the cache was being completely repopulated every time bazel was run? Is that what you are seeing happen?

Not necessarily every invocation, but every time the WORKSPACE checks that it is up-to-date (I believe this is called a "fetch" in WORKSPACE-land), which at least happens each time the analysis cache is discarded.
In practice that means I've been seeing it 4-5 times a day during regular development.
Without local = True, then the cache can be vendored by bazel into the cache, which lets it persist until one of the inputs changes.

For us it isn't too bad, because it only takes 10s to create the cache for the few things we need, but it is still annoying.

Good point, maybe @allsey87 knows more about what conditions we would want to regenerate the cache.

@sbc100
Copy link
Collaborator

sbc100 commented Feb 21, 2025

Seems reasonable to me. lgtm if @allsey87 and @walkingeyerobot things this is good idea.

@walkingeyerobot
Copy link
Collaborator

This is a piece of bazel I have minimal experience with, but I'm happy to approve. We should give @allsey87 a few days to respond before merging though.

@allsey87
Copy link
Contributor

This is a problem that I have also noticed so I am glad there is a solution to it! I am not sure why I added local in the first place, but perhaps I just misread the documentation. LGTM.

@walkingeyerobot walkingeyerobot enabled auto-merge (squash) February 23, 2025 21:32
@walkingeyerobot walkingeyerobot merged commit 49ccc1d into emscripten-core:main Feb 23, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants