twister: Report Kconfig symbols in twister.json

[golowanow]

New Twister command line argument --report-kconfig-symbols extends twister.json report with Kconfig symbols which were used to build the test suites selected.
The Kconfig additional test's context allows to track code changes in tests (like HWMv2 migration side effects), helps in regressions' triage and coverage analysis.

[golowanow]

an example snippet from twister.log with --report-kconfig-symbols :

    "testsuites":[
        {
            "name":"tests/benchmarks/footprints/benchmark.kernel.footprints.default",
            "arch":"arm",
            "platform":"frdm_k64f",
            "path":"tests/benchmarks/footprints",
            "run_id":"3f0ce51ec4c3d2566adc17b736176f55",
            "runnable":false,
            "retries":0,
            "status":"passed",
            "execution_time":"0.00",
            "build_time":"28.22",
            "kconfig":{
                "CONFIG_APPLICATION_INIT_PRIORITY":"90",
                "CONFIG_ARCH":"arm",
                "CONFIG_ARCH_HAS_CODE_DATA_RELOCATION":"y",
                "CONFIG_ARCH_HAS_CUSTOM_SWAP_TO_MAIN":"y",
                "CONFIG_ARCH_HAS_EXECUTABLE_PAGE_BIT":"y",
                "CONFIG_ARCH_HAS_EXTRA_EXCEPTION_INFO":"y",
                "CONFIG_ARCH_HAS_NESTED_EXCEPTION_DETECTION":"y",
                "CONFIG_ARCH_HAS_RAMFUNC_SUPPORT":"y",
                "CONFIG_ARCH_HAS_SINGLE_THREAD_SUPPORT":"y",
                "CONFIG_ARCH_HAS_STACK_PROTECTION":"y",
                "CONFIG_ARCH_HAS_SUSPEND_TO_RAM":"y",
                "CONFIG_ARCH_HAS_THREAD_ABORT":"y",
                "CONFIG_ARCH_HAS_THREAD_LOCAL_STORAGE":"y",
                "CONFIG_ARCH_HAS_TIMING_FUNCTIONS":"y",
                "CONFIG_ARCH_HAS_USERSPACE":"y",
...

[hakehuang]

would this be more flexible to add to a separated file to each test link handler.txt, we can add a build_configs.json?

[golowanow]

would this be more flexible to add to a separated file to each test link handler.txt,

@hakehuang, 'handler.log' is not in context of this PR, or I misunderstood your point.

we can add a build_configs.json?

.config is already a separate file, and the goal of this PR's change is to have a single json container with extended and consistent test report data ready to upload, e.g. to ElasticSearch, without a need to merge it with any other artifacts, or rely on some 'foreign' key etc.

[golowanow]

dear reviewers, up to your attention: any comments, objections ?

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

[nashif]

what next? cmake cache? list of compiled filed? All of those small optional additions to address one single usecase is what makes twister very difficult to manage. We have many options already and many unused features that were added at some point because "we can" but getting no attention and no use and make twister almost impossible to maintain.

[golowanow]

.. those small optional additions to address one single usecase

well, at least for these 3 use cases:

The Kconfig additional test's context allows to track code changes in tests (like HWMv2 migration side effects), helps in regressions' triage and coverage analysis.

[github-actions]

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.