test: twister: get_case_or_create : create case by log parse is not correct

[hakehuang]

Summary

in twister testinstance.py we have below logic

    def get_case_or_create(self, name):
        for c in self.testcases:
            if c.name == name:
                return c

        logger.debug(f"Could not find a matching testcase for {name}")
        tc = TestCase(name=name)
        self.testcases.append(tc)
        return tc

and the calls to this api is as below:

in harness.py, we parse the log and find the match test cases and add status.
in testinstance.py, we call the set_case_status_by_name and this api called in

and only when we generate a test plan, we are adding cases one by one. but in harness.py call, the testcase shall not added if it is not expected. this will introduce problem in below situations:

1. a former cases output, which will be wrongly parsed as new test case, which should be ignored.
2. flash failed and the former testcases become current cases log.

see below error json report

       {
            "name":"tests/drivers/pwm/pwm_api/drivers.pwm",
            "arch":"arm",
            "platform":"frdm_mcxn947/mcxn947/cpu0",
            "path":"tests/drivers/pwm/pwm_api",
            "run_id":"b7ade2d084de74e511ec6701f7718a2f",
            "runnable":true,
            "retries":1,
            "toolchain":"zephyr",
            "dut":"ZDDKTUF5SB1OO",
            "status":"passed",
            "execution_time":"15.24",
            "build_time":"0.00",
            "testcases":[
                {
                    "identifier":"drivers.pwm.pwm_basic.pwm_invalid_port",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.pwm_basic.pwm_nsec",
                    "execution_time":"3.01",
                    "status":"passed"
                },
                {
                    "identifier":"drivers.pwm.pwm_basic.pwm_cycle",
                    "execution_time":"3.01",
                    "status":"passed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_long_44100",
                    "execution_time":"0.03",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_08000",
                    "execution_time":"0.03",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_16000",
                    "execution_time":"0.02",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_32000",
                    "execution_time":"0.01",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_44100",
                    "execution_time":"0.01",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_48000",
                    "execution_time":"0.01",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_88200",
                    "execution_time":"0.01",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_transfer_short_96000",
                    "execution_time":"0.01",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_long_44100",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_08000",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_16000",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_32000",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_44100",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_48000",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_88200",
                    "execution_time":"0.00",
                    "status":"failed"
                },
                {
                    "identifier":"drivers.pwm.i2s_dir_both_transfer_short_96000",
                    "execution_time":"0.00",
                    "status":"failed"
                }
            ]
        },

the console log is i2s test case, and its been pared as pwm case log and add none-existing cases

Describe the solution you'd like

change the get_case_or_create() api to below:

def get_case_or_create(self, name, create=False):

and only create cases when create=False, and change the implement in harness.py set this parameter to false.

Alternatives

No response

Additional Context

No response