Twister's --enable-asan may hamper GCC bug detection

[LukaszMrugala]

Reproduction

Let us create a simple testcase:

/*
* Copyright (c) 2024 Intel Corporation
*
* SPDX-License-Identifier: Apache-2.0
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <zephyr/ztest.h>


ZTEST_SUITE(asan_tests, NULL, NULL, NULL, NULL, NULL);

int * ptr;

int helper() {
   int loc[10];
   loc[0] = 9;
   ptr = &loc[0];
   return 0;
}

/**
* @brief Test Asserts
*
* This test verifies the asan problem.
*
*/
ZTEST(asan_tests, test_assert)
{
   helper();
   printf("num is: %d\n", ptr[0]);

   zassert_true(1, "1 was false");
}

We can see that the code there is not correct. the helper function sets the ptr pointer to its local array, which goes out of scope as soon as we leave the function. Thus, we use a pointer to freed memory.

Case 1: No asan

I have run this command: twister -T <path_to_test> -p native_sim -i to check what Twister will do without the Address Sanitiser. It turns out that even without it, Twister is runs the test, it errors out with this snippet of logs:

[...]
In function ‘a1_1_tests_test_assert’,
   inlined from ‘_a1_1_tests_test_assert_wrapper’ at <$ZEPHYR_PATH>/tests/asan/src/main.c:50:1:
<$ZEPHYR_PATH>/tests/asan/src/main.c:33:9: error: ‘loc[0]’ is used uninitialized [-Werror=uninitialized]
  33 |         printf("num is: %d\n", ptr[0]);
     |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<$ZEPHYR_PATH>/tests/asan/src/main.c: In function ‘_a1_1_tests_test_assert_wrapper’:
<$ZEPHYR_PATH>/tests/asan/src/main.c:18:13: note: ‘loc’ declared here
  18 |         int loc[10];
     |             ^~~
cc1: all warnings being treated as errors
[...]

Which is great! GCC itself seemed to catch our error. Even without additional options, our end users can get useful information when creating new tests and making a mistake. However, with asan, we should get even better information. After all, this mistake is used as an example of what ASan can detect! (Note that this type of bug requires additional ASan option to actually be caught by ASan, but that's not the main point)

Case 2: Address sanitiser

To this end, I have run such command: twister -T <path_to_test> -p native_sim -i --enable-asan. It differs from the previous one only by the --enable-asan flag.

However, this time Twister run through the test and deemed it correct. The test has passed and no mistakes were reported.

What does --enable-asan actually do?

The relevant parts of Twister code are here:

zephyr/scripts/pylib/twister/twisterlib/handlers.py

Lines 265 to 270 in a48c958

	env = os.environ.copy()
	if self.options.enable_asan:
	env["ASAN_OPTIONS"] = "log_path=stdout:" + \
	env.get("ASAN_OPTIONS", "")
	if not self.options.enable_lsan:
	env["ASAN_OPTIONS"] += "detect_leaks=0"

zephyr/scripts/pylib/twister/twisterlib/testinstance.py

Lines 280 to 282 in a48c958

	if enable_asan:
	if platform.type == "native":
	content = content + "\nCONFIG_ASAN=y"

When we take a look at the buildlog, the difference can be seen in the ccache call, where the asan version has two additional flags: -fsanitize-recover=all -fsanitize=address.

Why is it a problem?

Users expect to have an increased understanding of problems in their code when using additional diagnostic tools. Sanitisers especially should help the users detect more rather than fewer bugs in their code.

I have encountered this problem accidentally, when creating blackbox testcases. I think someone more well-versed in GCC and ASan might shine more light on this.