Improve header structure and usage

[gmarull]

Is your enhancement proposal related to a problem? Please describe.

As of today one can observe a few issues with respect to C headers in Zephyr:

• Many C files rely on headers being included somewhere else (indirect dependency)
• Inclusion list does not follow any particular rules (you can find together local, libc, Zephyr, 3rd party lib headers all mixed).
• Some headers are not self-contained, meaning you need to include another one in the right order or your build will fail (e.g. sys/time_units.h.
• Some headers work "à la <windows.h>" way. It happened with device.h and pm/device.h. Another common case is soc.h including HAL headers (see related issue: Improve STM32 LL HAL usage #28822).

All these issues may seem not relevant, but over time this translates to increased compile times, changes in a header that have a domino effect to many source files, etc.

I've opened this issue inspired by https://lwn.net/Articles/880175/.

Describe the solution you'd like

• First establish certain rules. I think the rules used by Google C++ styleguide are a good starting point. Some of the rules:

  • Respect order of inclusion, for example:

    /* local headers */
    #include "mydir/foo.h"
    
    /* libc headers */
    #include <string.h>
    #include <stdlib.h>
    
    /* Zephyr headers */
    #include <kernel.h>
    #include <drivers/gpio.h>
    
    /* 3rd party libraries */
    #include <lvgl.h>

    Note that clang-format can assist on automatically sorting the inclusion list.
  • Headers should be self-contained. Those which are not should error when included standalone (e.g. using a required definition before inclusion).
  • Avoid forward declarations (unless e.g. to break circular dependencies).
  • Include only what is necessary

• Gradually improve the current situation

• Add some minimal CI checks to keep things in good shape

Another interesting change that, I think, would be useful is to place all Zephyr headers into a zephyr folder, so that all Zephyr headers would then be used as:

#include <zephyr/kernel.h>
#include <zephyr/drivers/gpio.h>

This would help a lot to identify Zephyr headers, to create a single clang-format sorting rule, etc.

Describe alternatives you've considered

Let the problem grow and open a PR in the future with 2,297 commits in to fix the issues :-)

Additional context

Include graph for kernel.h as generated by Doxygen (can likely be improved):

[henrikbrixandersen]

I generally like the ideas put forward in the description. However, in my experience we have always opted to put local includes first in the compiler include search path, but last in the list of #include directives. This allows for the local includes to have conditionals based on the scope they are included in. I would be interested to hear arguments for putting them first?

Apart from that, the proposed order is the same as what I would recommend.

[gmarull]

I generally like the ideas put forward in the description. However, in my experience we have always opted to put local includes first in the compiler include search path, but last in the list of #include directives. This allows for the local includes to have conditionals based on the scope they are included in. I would be interested to hear arguments for putting them first?

Apart from that, the proposed order is the same as what I would recommend.

The reason is explained in the Google C++ styleguide:

With the preferred ordering, if the related header dir2/foo2.h omits any necessary includes, the build of dir/foo.cc or dir/foo_test.cc will break. Thus, this rule ensures that build breaks show up first for the people working on these files, not for innocent people in other packages.

dir/foo.cc and dir2/foo2.h are usually in the same directory (e.g., base/basictypes_test.cc and base/basictypes.h), but may sometimes be in different directories too.

Some C projects, e.g. https://github.com/libgit2/libgit2 seem to follow this convention too. Could you provide an example of your usecase?

[marc-hb]

I understand backward compatibility has always been king in C; explains why we're still stuck with the 50-years old #include. However and as highlighted by this issue, #include leaves a large spectrum of possibilities: from disastrous to somewhat maintainable.

From time to time I've (ab)used gcc -E -dI -dM -dZ ... but surely proper tools / compiler plugins exist to help manage this modularity mess? This problem is universal, affects every single C project above a relatively small size. Policies are great but without the automation to implement and maintain them they never go far.

EDIT: I mean tools to not just order includes but to actually analyse dependencies, think jdeps , https://www.jetbrains.com/help/idea/dependencies-analysis.html, https://www.jarchitect.com/Doc_VS_Arch etc.
Doxygen can produce a .dot graph of includes, maybe that's a good start?

[alwa-nordic]

There exists a tool for detecting indirect dependencies we could consider using: https://include-what-you-use.org/

Indirect dependencies / transitive includes are a reliance on a implementation detail and can break in the future. E.g:

//// feature_foo.h:
#include <stdbool.h>
// Declare Foo

//// user_of_feature_foo.c:
#include <feature_foo.h>
bool baz; // Accidentally use `bool` from <feature_foo.h>
// Use Foo

If feature_foo.h drops its dependency on stdbool, user_of_feature_foo.c will unexpectedly break.

[gmarull]

There exists a tool for detecting indirect dependencies we could consider using: include-what-you-use.org

That's an interesting option to look at!