Closed
Description
Is your enhancement proposal related to a problem? Please describe.
There are too many QA issues with POSIX testing. The primary one being that there are no dedicated integration platforms. This can result in slow CI times.
Describe the solution you'd like
We should probably add one common integration platform per testsuite.
The other QA issue is that there are too many scenarios. However, those scenarios are necessary to ensure that we have coverage across the necessary libc's.
Describe alternatives you've considered
I was under the impression that testing on all tier0 targets was sufficient (and also ideal) in that it would result in fewer regressions.
# 1 tier0 platform per supported architecture
platform_key:
- arch
- simulationHowever, if that's not the case, then we'll add 1 integration platform if one does not already exist.
One thing that isn't clear to me is whether QA considers the platform_key permutations when it reports on missing integration_platforms. From what I understand, using platform_key to run tests for each major architecture in simulation should give sufficient coverage to ensure that there are no regressions.
I have some concerns that adding 1 integration platform might skip tests that are otherwise necessary to ensure that we don't see regressions due to missing coverage.
Additional context
Output of ./scripts/twister -ll qa -y -T tests/posix/ using the qa tag from #86571
INFO - Using Ninja..
INFO - Zephyr version: v4.1.0-rc3-8-gf3c3c2cf7886
INFO - Using 'zephyr' toolchain.
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs.minimal
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs.newlib
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs.tls
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs.tls.newlib
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs.picolibc
QA - filter with no integration platforms in tests/posix/fs/portability.posix.fs.tls.picolibc
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/fs: 7
QA - filter with no integration platforms in tests/posix/c_lib_ext/portability.posix.c_lib_ext
QA - filter with no integration platforms in tests/posix/c_lib_ext/portability.posix.c_lib_ext.armclang_std_libc
QA - filter with no integration platforms in tests/posix/c_lib_ext/portability.posix.c_lib_ext.arcmwdtlib
QA - filter with no integration platforms in tests/posix/c_lib_ext/portability.posix.c_lib_ext.minimal
QA - filter with no integration platforms in tests/posix/c_lib_ext/portability.posix.c_lib_ext.newlib
QA - filter with no integration platforms in tests/posix/c_lib_ext/portability.posix.c_lib_ext.picolibc
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/c_lib_ext: 6
QA - filter with no integration platforms in tests/posix/spinlocks/portability.posix.spinlocks
QA - filter with no integration platforms in tests/posix/spinlocks/portability.posix.spinlocks.minimal
QA - filter with no integration platforms in tests/posix/spinlocks/portability.posix.spinlocks.newlib
QA - filter with no integration platforms in tests/posix/spinlocks/portability.posix.spinlocks.picolibc
QA - filter with no integration platforms in tests/posix/spinlocks/portability.posix.spinlocks.no_spin_validate
QA - filter with no integration platforms in tests/posix/barriers/portability.posix.barriers
QA - filter with no integration platforms in tests/posix/barriers/portability.posix.barriers.minimal
QA - filter with no integration platforms in tests/posix/barriers/portability.posix.barriers.newlib
QA - filter with no integration platforms in tests/posix/barriers/portability.posix.barriers.picolibc
QA - filter with no integration platforms in tests/posix/eventfd/portability.posix.eventfd
QA - filter with no integration platforms in tests/posix/eventfd/portability.posix.eventfd.minimal
QA - filter with no integration platforms in tests/posix/eventfd/portability.posix.eventfd.newlib
QA - filter with no integration platforms in tests/posix/eventfd/portability.posix.eventfd.armclang_std_libc
QA - filter with no integration platforms in tests/posix/eventfd/portability.posix.eventfd.arcmwdtlib
QA - filter with no integration platforms in tests/posix/eventfd/portability.posix.eventfd.picolibc
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/eventfd: 6
QA - filter with no integration platforms in tests/posix/xsi_realtime/portability.posix.xsi_realtime
QA - filter with no integration platforms in tests/posix/xsi_realtime/portability.posix.xsi_realtime.minimal
QA - filter with no integration platforms in tests/posix/xsi_realtime/portability.posix.xsi_realtime.newlib
QA - filter with no integration platforms in tests/posix/xsi_realtime/portability.posix.xsi_realtime.picolibc
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.with_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.without_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.minimal.with_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.minimal.without_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.picolibc.with_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.picolibc.without_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.newlib.with_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.newlib.without_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.arcmwdtlib.with_posix_api
QA - filter with no integration platforms in tests/posix/headers/portability.posix.headers.arcmwdtlib.without_posix_api
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/headers: 10
QA - filter with no integration platforms in tests/posix/rwlocks/portability.posix.rwlocks
QA - filter with no integration platforms in tests/posix/rwlocks/portability.posix.rwlocks.minimal
QA - filter with no integration platforms in tests/posix/rwlocks/portability.posix.rwlocks.newlib
QA - filter with no integration platforms in tests/posix/rwlocks/portability.posix.rwlocks.picolibc
QA - filter with no integration platforms in tests/posix/common/portability.posix.common
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.minimal
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.newlib
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.armclang_std_libc
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.arcmwdtlib
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.tls
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.tls.newlib
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.picolibc
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.no_spin_validate
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.signal.strsignal_no_desc
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.signal.big_nsig
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.dynamic_stack
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.static_stack
QA - filter with no integration platforms in tests/posix/common/portability.posix.common.userspace
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/common: 14
QA - filter with no integration platforms in tests/posix/threads_ext/portability.posix.threads_ext
QA - filter with no integration platforms in tests/posix/threads_ext/portability.posix.threads_ext.minimal
QA - filter with no integration platforms in tests/posix/threads_ext/portability.posix.threads_ext.newlib
QA - filter with no integration platforms in tests/posix/threads_ext/portability.posix.threads_ext.picolibc
QA - filter with no integration platforms in tests/posix/signals/portability.posix.signals
QA - filter with no integration platforms in tests/posix/signals/portability.posix.signals.minimal
QA - filter with no integration platforms in tests/posix/signals/portability.posix.signals.newlib
QA - filter with no integration platforms in tests/posix/signals/portability.posix.signals.picolibc
QA - filter with no integration platforms in tests/posix/signals/portability.posix.signals.strginal_no_desc
QA - filter with no integration platforms in tests/posix/signals/portability.posix.signals.big_nsig
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/signals: 6
QA - filter with no integration platforms in tests/posix/timers/portability.posix.timers
QA - filter with no integration platforms in tests/posix/timers/portability.posix.timers.minimal
QA - filter with no integration platforms in tests/posix/timers/portability.posix.timers.newlib
QA - filter with no integration platforms in tests/posix/timers/portability.posix.timers.picolibc
QA - filter with no integration platforms in tests/posix/xsi_system_logging/portability.xsi.system.logging
QA - filter with no integration platforms in tests/posix/xsi_system_logging/portability.xsi.system.logging.minimal
QA - filter with no integration platforms in tests/posix/xsi_system_logging/portability.xsi.system.logging.newlib
QA - filter with no integration platforms in tests/posix/xsi_system_logging/portability.xsi.system.logging.picolibc
QA - filter with no integration platforms in tests/posix/xsi_threads_ext/portability.posix.xsi_threads_ext
QA - filter with no integration platforms in tests/posix/xsi_threads_ext/portability.posix.xsi_threads_ext.minimal
QA - filter with no integration platforms in tests/posix/xsi_threads_ext/portability.posix.xsi_threads_ext.newlib
QA - filter with no integration platforms in tests/posix/xsi_threads_ext/portability.posix.xsi_threads_ext.picolibc
QA - filter with no integration platforms in tests/posix/xsi_threads_ext/portability.posix.xsi_threads_ext.static_stack
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/net: 6
QA - filter with no integration platforms in tests/posix/semaphores/portability.posix.semaphores
QA - filter with no integration platforms in tests/posix/semaphores/portability.posix.semaphores.minimal
QA - filter with no integration platforms in tests/posix/semaphores/portability.posix.semaphores.newlib
QA - filter with no integration platforms in tests/posix/semaphores/portability.posix.semaphores.picolibc
QA - filter with no integration platforms in tests/posix/single_process/portability.posix.single_process
QA - filter with no integration platforms in tests/posix/single_process/portability.posix.single_process.armclang_std_libc
QA - filter with no integration platforms in tests/posix/single_process/portability.posix.single_process.arcmwdtlib
QA - filter with no integration platforms in tests/posix/single_process/portability.posix.single_process.minimal
QA - filter with no integration platforms in tests/posix/single_process/portability.posix.single_process.newlib
QA - filter with no integration platforms in tests/posix/single_process/portability.posix.single_process.picolibc
QA - Too many scenarios in /home/ubuntu/tt-zephyr-platforms-work/zephyr/tests/posix/single_process: 6
QA - filter with no integration platforms in tests/posix/xsi_streams/portability.posix.xsi_streams
QA - filter with no integration platforms in tests/posix/xsi_streams/portability.posix.xsi_streams.minimal
QA - filter with no integration platforms in tests/posix/xsi_streams/portability.posix.xsi_streams.newlib
QA - filter with no integration platforms in tests/posix/xsi_streams/portability.posix.xsi_streams.picolibc
INFO - Selecting default platforms per testsuite scenario
INFO - Building initial testsuite list...
INFO - Writing JSON report /home/ubuntu/tt-zephyr-platforms-work/zephyr/twister-out/testplan.json
INFO - Completed in 1.5165433883666992 seconds