Depfiles for custom rules

[canada-geek]

In bazel, AFAIK there is native support for C/C++ compilers to produce depfiles (using the -M... options such as -MT, -MMD, -MP, -MF and so on). This allows a finer-grained dependency tracking, targeting the specific files that are actually used by the source file(s) being processed, even when the target's actual set of input files (especially headers) is much larger.

Is there a similar mechanism available for custom rules?

In particular, our project has multiple code-generator tools that parse C++ header files. Some examples:

• auto-generate functions for converting enums to strings
• SWIG: auto-generate python wrappers
• auto-generate serialization code
• many more

Each of these tools can be quite expensive to run (at least in our environment). Given that they process header files, the inputs to ctx.actions.run() and ctx.actions.run_shell() end up being supersets of the needed files that are much larger than the actual set of files used when processing the headers.

It is impractical to attempt a reduction in inputs, especially since examination of the files is unavailable during the analysis phase (for good reason).

It would be (very) advantageous in our situation to be able to limit the actual dependencies to the files that actually get used. The depfile mechanism seems like a natural fit, and (IMO) would be a most welcome addition here if it doesn't already exist.

Are there alternatives I may not be aware of? Please help educate me if my understanding doesn't match yours.

One crazy thought that comes to mind... might it be possible to "pretend" that the above are indeed C/C++ compilers, and somehow create C/C++ toolchains that use them? Would this maybe trigger the depfiles mechanism? (this feels to me like a bit of a hack, but I'm open to any ideas that might help here).

Thanks for your time and attention.

[fmeum]

You could implement your own parsing of dep files and then use the unused_inputs_list feature of ctx.actions.run to communicate the unused inputs to Bazel.

[canada-geek]

Hmm, OK I guess I missed that function parameter, thanks for pointing it out.

So I guess I would need to have a (set of) preprocess action(s) that would create this File by doing:

1. run the C/C++ preprocessor to obtain a list of files.
2. take the set of all inputs, subtract from it the list from (1), and write it to a file.
3. Supply a reference to this File to unused_inputs_list.

The main challenge here is ensuring that the execution of the preprocessor is efficient, since its inputs will be the (super)set of all possible header files.

Please verify my understanding, then I can mark this as answered.

Thanks very much!

[fmeum]

That's one way to do it. An alternative could be to wrap your actual actions so that they generate this file as a byproduct.