Filtered watch for large directories

[canada-geek]

TL;DR: Add filters to repository_ctx.watch_tree(); only trigger repo refresh on changes to files matching pattern.

We have a project for which we are prototyping a transition to using bazel. The primary build product is a set of device driver libraries.

The complication arises where the set of devices in the workspace can be fluid:

• our GIT repo has the full set of devices
• a given workspace is able to prune devices, thereby reducing the focus for a particular activity
  • this is not just for convenience; under some conditions a reduced set of sources is sent to customers
• new devices are sometimes introduced as part of ongoing development

We are trying to set up our bazel rule set such that there are no hardcoded references to any specific devices; rather, we would like the bazel rules to remain device agnostic through a discovery mechanism.

For simplicity, let's say that the devices appear as directories with the pattern <root>/driver/<device_name>.

Since per-device information is needed during the build, I have placed a device_info.yaml file in each device directory that contains information pertaining to that device.

I've been able to achieve device discovery (and consumption of these info files) using a repository rule that scans for these files. The repository rule also sets up a repository_ctx.watch() for each such file in order to refresh the local repository when necessary.

The complication comes when devices get added; there is no trigger to rerun the discovery and locate the newly added device(s).

The naive solution would be to use repository_ctx.watch_tree() on the driver subdirectory. However, given that this directory contains many hundreds of files (including the bulk of the source code), any change in source code would trigger a refresh of the repo unnecessarily.

My proposal is to add filtering parameters to watch_tree(), such that only files matching this pattern (which could be a combination of include/exclude patterns similar to glob()) would contribute to the watch.

Note: I'm not an expert in this area; my assumption (please correct me if I've got this wrong) is that bazel is computing some sort of hash on the tree being watched, such that any content change (different hash) triggers a refresh of the repo. If this is the case, then a reduction of the set of files that contribute to the hash should not be a significant departure to the current implementation, while adding value for my use case.

I can probably create a patch for this, but before I make the attempt, I wanted to check in with you to discuss feasibility.

If you have alternate ways of achieving this, please help me learn more.

Thank you for your time and attention.

[fmeum]

Would applying readdir to the directory in watch mode help here or still invalidate too often?

[canada-geek]

Thanks for replying and the pointer. Based on my cursory reading of the documentation for that function, the watch would be applied to all or none of the paths, which would effectively do the same thing as watch_tree(). Further, it would only introduce watching for files that exist at the time readdir() is called, and would not catch the addition of newer files.

The key here is the trigger. I already have watch() posted on existing files. The hard part is triggering a refetch when new devices are added (i.e. a new device directory is added under driver/, and it contains a previously-unseen device_info.yaml file). The subtlety here is that the pattern matching would be applied as part of performing the watch check, to see if any newer files have been introduced since the last bazel build, and only trigger repo refetch upon changes/additions/removals of those specific files, not due to any changes to other files in the subtree).

EDIT: minor addition above

[fmeum]

readdir on the devices directory would trigger if new devices are added, but wouldn't trigger if anything beneath a particular device's directory changes. That does sound like what you want (a non-recursive watch_tree).

[canada-geek]

I see, so readdir() is not recursive. I guess that was a bad assumption on my part. I will try using this. Thanks!