Applying model into ESP32S3 Problem (TFMIC-58)

[leanhplus2612]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Provided a clear description of your suggestion.
• Included any relevant context or examples.

Issue or Suggestion Description

Hi espressif,
I am researching micro_speech example and trying to replace the google dataset into my own data, in addition, I also want to extend the time for detecting the speech with INMP441 micro.
I had read a google colab link that train for yes/no word by accident. But after training and replace the model in micro speech example, It did not work. Moreover, I also follow an wake word model in the atomic14 tutorial. However, I just seem to work with the lower tensorflow version.

My question is that what is the tensorflow version which is supported by esp32s3 and esp32 in general? Do you have any method for resolve my problem above?

I am really looking forward to your positive response!
Thank you very much in advance!

Best Regards,
Ant Leo

[vikramdattu]

Hi @leanhplus2612

• Does the inference work as expected with python script or similar setup on host?
• Best way to make sure if the model is working as expected for known data is to embed the audio and pass it directly instead of relying on the MIC data.
  • After making sure that, that indeed works, you may enable the actual mic data.
  • This will narrow down your problem to either model OR the data itself.

[leanhplus2612]

Hello @vikramdattu

Thanks for your advice.
I just try to retrain in google colab tutorial without any editting first. Then I replaced the model.cc into the new model.cc which is converted in the final cell.
I realized that the heximal array size is different (18800 vs 19200). I also saw that the depthwiseconv2d turn into conv2d by accident. Although I changed the OpResolver from depthwiseconv2d to conv2d, It not working and return 0,25 for each label. Moreover, I also checking for the model input, It still fluctuate and the raw mic data is not correct!

[maltsev-dev]

Hi @leanhplus2612

Due to the large number of tightly coupled fragile components, it is very difficult to determine who is to blame from your story.

As I understand it, you use ready-made scripts from tutorials for training the model on your data. Probably, your data is already consistent with the original dataset in format, balance, markup, etc.

If after training the model gives a good matrix, just like the quantized model gives good results on test data.
It seems to me that the problem of data fluctuations may be in the excessive sensitivity of the model to input data due to noise.

First of all, (if the previous stages are passed successfully) on the MCU I would try to normalize the raw data and set a suitable threshold, making the entry into the class more strict, conservative filtering so as not to pass garbage data to the inference, etc.

Find the culprit and punish him.
Good luck to you!