Source libraries packages, with unit tests - mission impossible?

[ilg-ul]

I am working on a C++ modular project (cross-platform embedded), with various libraries packed as separate projects.

Some of the libraries can be compiled as static libraries, some are headers only, some have sources, but for various reasons must be compiled in the top application context (what CMake calls INTERFACE libraries).

For the static libraries I was able to write a meson.build in the project root folder to create the static libraries, and including them on the application with subdir() was fine.

However, each such library project has a tests folder, with various unit tests. Building the tests requires other such libraries as dependencies, available multiple folders.

I was not able to identify a meson.build configuration that works both for building the unit tests, and also for being included in a top application.

To build the tests, the top meson.build must have a project() as the first call of a project, but when I try to use the library with subdir() in an application, meson complains of a second project() encountered.

I thought of using the tests folder as the root of the tests, but in this case I could not find a way to include the current library code, since it is defined in the parent folder, and subdir() can include only children folders.

I also thought of using subproject(), assuming all dependencies are meson projects. Unfortunately this requires for all sub-projects to be in a single folder, requirement which cannot be met in my environment, since my dependencies are in two folders, and none of them has the hard-coded name subprojects.

What am I missing? Am I trying to use meson in an impossible configuration?

[eli-schwartz]

If they are separate projects e.g. git submodules why can't they be moved to the directory meson uses for subprojects?

The entire point of subprojects is that they are isolated and standalone, and merely export a public interface (cmake interface targets, meson declare_dependency() objects).

Maybe https://mesonbuild.com/Wrap-dependency-system-manual.html would help?

[ilg-ul]

why can't they be moved

There are two groups of dependencies, build configuration dependencies and project wide dependencies, and an automated tool satisfies the dependencies by bringing them in two separate specific folders. This tool has its own naming conventions, which cannot be changed, and moving the folders cannot be done without breaking the functionality. The tool is called xpm and works similarly to npm, it can bring in dependencies from npm servers and a variety of other locations. Dependencies are either source packages, or binary packages (including a meson standalone cross-platform package).

... wrap

I saw this, it is a solution that probably works fine in many other cases, but in my case it doesn't make much sense, the dependent subprojects are already available in my projects, and I can add meson.build to all, so there is not much to wrap around.

[eli-schwartz]

It looks like you're using a package manager, which can build and install meson projects as well as projects in other languages and using other build systems, but you're using it to install... the source repository of other projects, into meson.

How is meson supposed to run xpm (which then runs meson)?

What is in charge of kicking off a build?

How do individual projects communicate with each other? In a typical package manager, other dependencies are built and installed into a workspace, and then meson uses dependency() or c.find_library() to look up the other software.

[ilg-ul]

It looks like you're using a package manager

That's correct, I use xpm.

which can build and install meson projects as well as projects in other languages and using other build systems

xpm can do a lot of things, but relevant for this topic is that is solves dependencies for building and running unit tests for embedded projects with lots of different configurations, each with different dependencies.

but you're using it to install... the source repository of other projects, into meson.

For various reasons, embedded projects are better built from sources. xpm, like npm, does not install plain source repositories, but filtered content, usually from a central server like npmjs.com, but can also do it from a Git repo. By filtered I mean only relevant source files and build metadata/configurations, without documentation, tests, etc.

How is meson supposed to run xpm (which then runs meson)?

meson is not supposed to know anything about xpm, only about the toolchain, ninja, possibly emulators and other such tools.

xpm is called to satisfy dependencies before invoking meson, and it does it in a satisfactory manner, it brings source packages, and binary tools, including the standalone meson binary xPack (which does not need a system Python to run).

What is in charge of kicking off a build?

The workflow is simple, and very CI friendly; for example, for running all tests, after cloning the project repo you have to issue something like:

npm install -g xpm@latest
xpm run install-all
xpm run test-all

The install-all and test-all are names of sequences of other commands that do the actual build and test steps. The mechanism is an extension of npm run xxx.

How do individual projects communicate with each other?

Projects define libraries, intended to be consumed by other projects. Some are static libraries, some are INTERFACE libraries, in CMake parlance. They are all collected to the top project in a single build, performed in a consistent manner, with a specific set of compiler definitions.

In a typical package manager, other dependencies are built and installed into a workspace

For this case the dependencies are source packages, already brought in the source and/or build tree. Each such source package has a meson.build file in the top, and can contribute definitions to an upper build.

From the documentation I understood that only subdir() can be used, since subproject() is not able to look in multiple locations.

This initial use case is ok, I can build applications with multiple such dependencies. The problem is how to build the unit tests associated with each such project, since when the top meson.build is consumed with subdir() it cannot include the project() call.

and then meson uses dependency() or c.find_library() to look up the other software.

I checked the manual, and dependency() seems to have a different purpose, to identify libraries installed in the system, which is not the case here.

If I got it wrong, please help me with some more details.

[jpakkane]

Based on your description it is a bit unclear how your project is set up. However you should note that variables keep existing when you go into subdirectories and back. So at the top level you can do:

my_lib = static_library(..)
subdir('tests')

and then in the tests dir do:

executable(..., link_with: my_lib)

[ilg-ul]

it is a bit unclear how your project is set up

I am building multi-architecture, multi-device, multi-toolchain, etc unit tests, using multiple build configurations.

Below build there are many folders, for tests with different configurations, like native, cortex-m, risc-v, etc, and each configuration has different dependencies.

The dependencies are located in two xpacks folders, like

build
  qemu-mps2-an386-meson-debug
    xpacks
      configuration-dependency
      ...
  qemu-mps2-an386-meson-release
    xpacks
      configuration-dependency
      ...
sources
tests
xpacks
  project-dependency
  ...

variables keep existing when you go into subdirectories and back

Yes, that's fine, I noticed this.

I tried a top meson.build like

project(...)

...

subdir('sources')

if meson.current_source_dir() == meson.project_source_root()

  subdir('xpacks/micro-os-plus-diag-trace') # Dependency

  subdir('tests')

endif

The initial build, without the dependency, passed, but when I tried to include a project with a similar structure as a dependency, I got:

build/qemu-mps2-an386-meson-debug/xpacks/micro-os-plus-diag-trace/meson.build:27:0: ERROR: Second call to project().

The project() call in micro-os-plus-diag-trace/meson.build is needed, since the dependency project has its own tests, and I could not build them without it.

[ilg-ul]

Unless there is another trick which I do not know, in order for the top meson.build to be consumed with subdir() by a parent project, it looks like it must not include a project() call. If so, then the same meson.build cannot include the definitions for the tests, and a separate file must be used.

I tried to find a way to tell meson which file to use, but could not find one, so it looks like in the top folder there can be only one meson configuration file, and it must be called meson.build. If so, the second file can be only in a separate folder, for example in the tests folder.

However, doing this, prevents it for directly accessing the top folders, like sources and the top xpacks, which are also needed to build the tests.

It looks more and more like Mission Impossible :-(