Feature Requst/Documentation Request: use `build.ninja` stub to bootstrap ninja

[titaniumtraveler]

When reading about how ninja, one of my first thoughts was that why not use ninja itself bootstrap the build.ninja file?

I tried googling this topic, but haven't really found anything, other than some closed issues concerning problems when regenerating build.ninja, but nothing directly about using a build.ninja-stub to bootstrap the project.

Looking at the build.ninja of this project, it at least contains a passage to regenerate its contents when ./configure is changed:

# Regenerate build files if build script changes.
rule configure
  command = ${configure_env}python3 $root/configure.py $configure_args
  generator = 1
build build.ninja: configure | $root/configure.py $root/misc/ninja_syntax.py

So my questions are:

Why not have that part in an otherwise empty build.ninja, so that the user can just run ninja the first time and the project bootstraps itself?

Does the build.ninja get reloaded by the running ninja process when it changes?
As in would running ninja manual succeed in generating doc/manual.html?

If the answer to these questions is "yes" it would be nice for those details to be part of the manual and maybe documented as potential pattern to use to bootstrap projects.

Though arguably writing to a source-controlled file might be a bad idea, instead one could write to an build.ninja file contained in the build directory and include that? (Does changing those cause ninja re-source them? And would ninja run when the file path doesn't exist currently, but is defined as a build target?)

If the answer is "no", it would be good to document that as gotcha and might be a potentially neat feature to consider in the future.

I would of cause be open to discuss this topic either here, or on the mailing list.

[digit-google]

On Tue, Jan 28, 2025 at 2:37 PM Jon Heinritz ***@***.***> wrote: Why not have that part in an otherwise empty build.ninja, so that the user can just run ninja the first time and the project bootstraps itself?

I can think of two reasons: - It is critical to be able to rebuild Ninja *without* an existing Ninja binary. This is what `configure.py --bootstrap` does and what you propose cannot replace this. - Having a build.ninja file in the git tree that gets rewritten every time Ninja is rebuilt is an annoyance during development. It is possible to add it to the project's .gitignore, but then you cannot do `git checkout HEAD build.ninja`. This gets annoying really fast. Does the build.ninja get *reloaded* by the running ninja process when it

changes?

Yes, that's what happens.

As in would running ninja manual succeed in generating doc/manual.html?

`./configure.py --bootstrap && ./ninja manual` doesn't seem much more complex imho.

If the answer to these questions is "yes" it would be nice for those details to be part of the manual and maybe documented as potential pattern to use to bootstrap projects. Yes, feel free to upload a PR for review. Though arguably writing to a source-controlled file might be a bad idea, instead one could write to an build.ninja file contained in the build directory and include that? (Does changing *those* cause ninja re-source them? And would ninja run when the file path doesn't exist currently, but is defined as a build target?)

I don't understand what you are proposing. If you care about the convenience of just invoking `ninja manual` directly, then the `build.ninja` must be in the build directory which is the top-level source directory.

…

If the answer is "no", it would be good to document that as gotcha and might be a potentially neat feature to consider in the future. I would of cause be open to discuss this topic either here, or on the mailing list. — Reply to this email directly, view it on GitHub <#2562>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMNJ4HMCNKGFXP2FTC3ZH332M6B2DAVCNFSM6AAAAABWAPZ6HSVHI2DSMVQWIX3LMV43ASLTON2WKOZSHAYTKNZQGMYDQMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[titaniumtraveler]

I don't understand what you are proposing. If you care about the
convenience of just invoking ninja manual directly, then the
build.ninja must be in the build directory which is the top-level source
directory.

I'm having difficulties to express my proposal in details, but my main idea is basically that a typical contributor shouldn't have to think about the build system when they get started with a project.

So my ideal workflow would be, if they see that the project is build with ninja,
they don't need to care about how generator is invoked, but can just run ninja with some target.

And if we can prove that works for ninja itself, the pattern might get adopted for other projects that use ninja as well.

As context I'm relatively new to C/C++ and with many projects I feel like before being able to even build a project, you have to carefully study the build instruction to make sure you don't do anything wrong, which is a big barrier of entry for C in general in my opinion.
(My first attempt at learning C failed at figuring out cmake for example)

I know a lot of this complexity is somewhat inherent to C, but I feel a lot of them can also be solved with some good defaults.

If the answer to these questions is "yes" it would be nice for those
details to be part of the manual and maybe documented as potential pattern
to use to bootstrap projects.

Yes, feel free to upload a PR for review.

I will. I think I will start with a basic Proof of Concept and see from there.

[titaniumtraveler]

So, I have written a basic POC of how self-bootstrapping ninja file could look like. (See #2563)

It can be tested with

rm -rf build && mkdir build # make sure build directory is empty
touch configure.py          # force the `configure` rule to be run to generate ninja file
touch build/build.ninja     # make sure include target exists so that ninja doesn't error out
ninja <some-target>

The only thing missing to make it actually work is some way conditionally execute includes. (Without the script above that is)

I think the best to achieve that is to allow for include (and subninjas too for that matter) to be able to receive build targets.

So that when ninja encounters an include, it will check, whether the file is in the list of build targets and when necessary build/rebuild that target.

(There might be other solutions to this problem, but this feels most natural)

I think that considering changes of includes, or subninjas when deciding whether to rebuild the manifest would be an good idea anyways, since it feels consistent with ninjas behavior when changing build.ninja itself during build.

[elliotgoodrich]

Personally I have wanted something like this too (I could kill steps 3 and 4 for ninjutsu-build. It would be good to abstract away the configuration script so a user just needs to run ninja and this will work on both a clean checkout and one with build artifacts.

I don't think overwriting build.ninja is the best way to do it because of the reason @digit-google mentioned about wanting to commit build.ninja, but not modifications to it - which git doesn't support at the moment.

Right now running ninja without -f causes it to look for a build.ninja file. One suggestion to support bootstrapping would be to change this behavior to:

1. If there is a build.ninja file, run this as usual
2. Otherwise, look for configure.ninja and run the default target (which should include build.ninja)
3. If this completes successfully, immediately run build.ninja

This way we can hand-write configure.ninja and commit this to git, while keeping build.ninja as an artifact that isn't added to git.

[titaniumtraveler]

Personally I have wanted something like this too (I could kill steps 3 and 4 for ninjutsu-build. It would be good to abstract away the configuration script so a user just needs to run ninja and this will work on both a clean checkout and one with build artifacts.

Yep, that is exactly what my thought was behind it.

I don't think overwriting build.ninja is the best way to do it because of the reason @digit-google mentioned about wanting to commit build.ninja, but not modifications to it - which git doesn't support at the moment.

Technically it does, but exclusively as temporary configuration for your local repository. So not useful for this specific usecase.

Right now running ninja without -f causes it to look for a build.ninja file. One suggestion to support bootstrapping would be to change this behavior to:

1. If there is a `build.ninja` file, run this as usual

2. Otherwise, look for `configure.ninja` and run the default target (which should include `build.ninja`)

3. If this completes successfully, immediately run `build.ninja`

This way we can hand-write configure.ninja and commit this to git, while keeping build.ninja as an artifact that isn't added to git.

I actually think that there is an easier solution for that, that requires less changes: Basically do commit the original build.ninja, but only have it contain the generator instruction.
Those generator instructions then instead of generating the build.ninja, generate a build/build.ninja, which is sourced with subninja.

That build/build.ninja file should then also be able to override its own build configuration to support usecases where the generator script is called with arguments that influence how the project is built.
(That part I roughly demonstrated in #2563)

All that would be needed to support this workflow would be to allow (re)building subninjas based on rules. That feature even has some precedence in the code base considering that build.ninja can rebuild itself.
(I sadly didn't really had the time to investigate further in terms of how it could be implemented)

[nanonyme]

On Tue, Jan 28, 2025 at 2:37 PM Jon Heinritz @.***> wrote:

• It is critical to be able to rebuild Ninja without an existing Ninja
  binary. This is what configure.py --bootstrap does and what you propose
  cannot replace this.

Please make sure this remains as is. We have already in freedesktop-sdk managed to use bootstrapped ninja to be used for bootstrapping cmake (!!!) and the only other way to build ninja in addition to bootstrap.py (without assuming pre-existing ninja) is to use cmake with makefile generator to build ninja. (it also works, we used that at one point of history)

[jhasse]

I was thinking about two ways to solve this:

A)
Having ninja recognize CMakeLists.txt and call cmake -Bbuild -G"Ninja Multi-Config" in that case. Also always look for build/build.ninja, too.

B)
Support a new config.ninja file with a dedicated syntax like:

builddir = build
configure = cmake -B$builddir -G"Ninja Multi-Config"

If such a file is found, ninja would look for $builddir/build.ninja and if that isn't found run the $configure command.

A has the advantage that it would work with every project that is using CMake with no additional changes.
B has the advantage to work with any build system and is a lot more flexible.

A mixture of both could also work, i.e. we implement B and if no config.ninja file is found we use a default for CMake (if CMakeLists.txt found), Meson (if meson.build found), etc.
Generators could even provide these templates so that we don't have to maintain them.

All of this wouldn't affect performance because nothing will be done if a build.ninja is present in the first place (or -f is used).

[eli-schwartz]

As context I'm relatively new to C/C++ and with many projects I feel like before being able to even build a project, you have to carefully study the build instruction to make sure you don't do anything wrong, which is a big barrier of entry for C in general in my opinion. (My first attempt at learning C failed at figuring out cmake for example)

I know a lot of this complexity is somewhat inherent to C, but I feel a lot of them can also be solved with some good defaults.

I'm not so sure this is really true. How do you find out that you're supposed to use the ninja program, for example? It's generally not hard to look at a README.md and see the "Building:" section, though.

It usually has other details too, like what software the package depends on, and that is a problem independent of programming language although build systems like cmake and meson can try to hide this for you by using subprojects / FetchContent.

[titaniumtraveler]

As context I'm relatively new to C/C++ and with many projects I feel like before being able to even build a project, you have to carefully study the build instruction to make sure you don't do anything wrong, which is a big barrier of entry for C in general in my opinion. (My first attempt at learning C failed at figuring out cmake for example)
I know a lot of this complexity is somewhat inherent to C, but I feel a lot of them can also be solved with some good defaults.

I'm not so sure this is really true. How do you find out that you're supposed to use the ninja program, for example? It's generally not hard to look at a README.md and see the "Building:" section, though.

I mean if there is a build.ninja stub that knows how to build the real build-script (in something like build/build.ninja), it might be enough to just have some program that detects that file in the root of the project and just runs ninja based on that.
(Picking all the reasonable defaults for things like platforms and whether to use the system libraries, or to build them from source)

Or just as human lsing a project and seeing a build.ninja stub in the project root would be enough.

It usually has other details too, like what software the package depends on, and that is a problem independent of programming language although build systems like cmake and meson can try to hide this for you by using subprojects / FetchContent.

I definitely think, that those dependencies and/or build options to configure which versions to use, would still have to be documented.
But if the defaults are sane and build-commands can be inferred from the project structure instead of having to explicitly read the documentation to even get started with the project, that is a huge step forward in my opinion.

The core idea all this hinges on for me, is to basically move the build instructions from textual documentation a human has to read, to code.

With a bit of luck one of my next projects will be ninja-based, which means I might have the time to actually implement the features necessary to use the build-stub pattern I described.
(Specifically that includeing/subninjaing a file adds it to the dependencies, which would allow for recursive building of projects)

[eli-schwartz]

I mean if there is a build.ninja stub that knows how to build the real build-script (in something like build/build.ninja), it might be enough to just have some program that detects that file in the root of the project and just runs ninja based on that. (Picking all the reasonable defaults for things like platforms and whether to use the system libraries, or to build them from source)

You're going to write such a program for the ninja project alone? But your entire argument was that contributors know ninja and don't know cmake, so they'd realize they need to manually run ninja?

If you actually wanted even contributors not to know ninja, then... IDEs can already auto detect the CMakeLists.txt and build the project for you.

Or just as human lsing a project and seeing a build.ninja stub in the project root would be enough.

You're expecting that a user doesn't know what cmake is or how to use it, but does know what ninja is and how to use it. This is a bit overly-specific if you ask me.

I definitely think, that those dependencies and/or build options to configure which versions to use, would still have to be documented. But if the defaults are sane and build-commands can be inferred from the project structure instead of having to explicitly read the documentation to even get started with the project, that is a huge step forward in my opinion.

And I'm saying that that's not actually how it works, because you cannot even get started with the majority of projects without first installing dependencies. The build command will just error out.

But build commands can.be inferred from the project structure:

• autotools
• cmake
• meson

All have standardized interfaces and their existence can be programmatically detected from a source tree, so an IDE can pick it up automatically if said IDE wishes to do so.

If you need even more, a lot of people believe it's reasonable to standardize on VS Code. I can tell because it's the number one cause of PRs to projects, adding a repository vscode configuration file telling vscode which recommended plugins to install and what build command to run

[eli-schwartz]

I don't see how any of this helps the ninja project though. You don't have ninja installed until after you build ninja.

If this is all about what projects other than ninja do, which you indicated by recommending that this become a general pattern for projects in general, then I think it's quite fine for Ninja to be an exception to that anyway.

[titaniumtraveler]

If this is all about what projects other than ninja do, which you indicated by recommending that this become a general pattern for projects in general, then I think it's quite fine for Ninja to be an exception to that anyway.

I meant it for any project using ninja. And ninja was an obvious good way to demonstrate how the build-stub-thing could work.

My primary reason to write the POC was to check whether the whole "automatically generate a default build script when running the first time ninja, but still allow for generating it manually if so desired" would work as expected.

[elliotgoodrich]

B) Support a new config.ninja file with a dedicated syntax like:

builddir = build
configure = cmake -B$builddir -G"Ninja Multi-Config"

If such a file is found, ninja would look for $builddir/build.ninja and if that isn't found run the $configure command.

If both config.ninja and $builddir/build.ninja exist, then presumably you'd not run config.ninja again and just run $builddir/build.ninja?

This approach means that you need to check if config.ninja exists and if so, open and parse it, on each ninja invocation in order to figure out the location of the true build.ninja script since it depends on builddir. Granted this is probably a small file, but it's still additional work - and you'd need to check the existence of the config.ninja file for all projects that aren't even using this file.

You'd also need to add new dedicated syntax to ninja for one particular type of file.

My suggestion above,

1. If there is a build.ninja file, run this as usual
2. Otherwise, look for configure.ninja and run the default target (which should include build.ninja)
3. If this completes successfully, immediately run build.ninja

avoids any overhead if build.ninja exists - since it's still the first step, and avoids any syntax or parsing changes for ninja build files.

[jhasse]

No, my approach would also check for build.ninja first. Using a subdirectory allows multiple configurations at the same time, but one might also use builddir = .. I think that shouldn't be done because with dedicated build directory you only need /build*/ in your .gitignore and you can do rm -rf build for a clean rebuild.

Otherwise, look for configure.ninja and run the default target (which should include build.ninja)

So your solution requires opening another file, too. I misunderstood. Your solution relies on using an in-tree build which my solution could also do, when using builddir = .. So they are very similar :)