Question about inference time improvement (TFMIC-56)

[ssh4]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Provided a clear description of your suggestion.
• Included any relevant context or examples.

Issue or Suggestion Description

Hello,

I'm running an inference task with instance segmentation, on XIAO ESP32S3 Sense.
The input for my model is 64x64x3. The model is quantized to ints, but data type is float.
That's its shape:

-- Model Inputs ---
Input 0:
  Index: 0
  Name: serving_default_images:0
  Shape: [ 1 64 64  3] (batch_size, height, width, channels)
  Data Type: <class 'numpy.float32'>
  Quantization: (0.0, 0)

--- Model Outputs ---
Output 0:
  Index: 438
  Name: PartitionedCall:1
  Shape: [ 1 16 16 32]
  Data Type: <class 'numpy.float32'>
  Quantization: (0.0, 0)
Output 1:
  Index: 486
  Name: PartitionedCall:0
  Shape: [ 1 37 84]
  Data Type: <class 'numpy.float32'>
  Quantization: (0.0, 0)

It is YOLOv8 architecture (YOLOv11 is better in size and inference, but some included OPS are not supported by the esp-tflite-micro framework).
The list of operations my model is built with:
QUANTIZE, DEQUANTIZE, PAD, CONV_2D, LOGISTIC, MUL, STRIDED_SLICE, ADD, CONCATENATION, MAX_POOL_2D, RESIZE_NEAREST_NEIGHBOR, TRANSPOSE, RESHAPE, TRANSPOSE_CONV, SOFTMAX, SUB

My first intergration is Arduino based, and inference time is ~2 seconds.
I wanted to improve the timing and ported my integration to ESP-IDF. I built the project not from examples, but from scratch. Unfortunately, the inference time remained the same, around 2 seconds.

Arduino implementation uses "idf-release_v5.4-2f7dcd86-v1/esp32s3/include/espressif__esp-tflite-micro" version.
ESP-IDF is built with espressif/esp-nn:1.1.1 and espressif/esp-tflite-micro:1.3.3~1

My question is - is it possible to improve the inference time somehow on ESP-IDF?
When I decided to move to ESP-IDF, I was really hoping that ESP-NN will improve the numbers, and some hardware utilization features will make the trick. At least something..

There are a few config options from "idf menuconfig", which seemed to me important:

• optimization for NN functions - Optimized version
• ESP PSRAM Mode - Octal
• PSRAM clock speek - 80MHz
• ESP System Settings - CPU Frequency 240 MHz

Appreciate your help and advice!

[vikramdattu]

For Int8 Quantised YoloV8 model with input/output size mentioned above, the inference time should be smaller, (Few 100s of milliseconds per inference).

1. Have you Quantised to int8 format?
2. Was optimization for NN functions - Optimized version set to ON?
3. If possible do share the link to the ESP-IDF project for me to try out

[ssh4]

Hello @vikramdattu

1. The model is qunatized to ints
2. Yes, it was set to Optimized version. If I set to Ansi-C, the inference time is 3.5 seconds instead of 2 seconds.
3. Please check this repo: https://github.com/ssh4/cltcv-idf-shared. I believe I included everything required for testing the inference.

[vikramdattu]

Your model is huge! (3MB and more than 250 layers!) And only the Conv layer is getting optimisations utilised from esp-nn.

Few things to try:

1. While exporting the model, set input type to int8
2. Small size input with more layers is not generally helpful. Maybe use larger input size (120x120 etc) and drop the number of layers getting the model size below 2M
3. Get rid of the logistic layers if that's possible. This is the layer consuming more than 1 seconds out of 2 seconds you see
4. Increase CACHE sizes (Reduces the time from 2000ms to 1647ms) [Conv is 471ms out of this, rest ~>1sec is logistics]: - Instruction cache: 16K --> 32K
   • Data Cache: 32K --> 64K

[ssh4]

@vikramdattu , thank you for looking into this!
I'll try to follow your recommendations, thanks again!