v62 produces lower framerate than v60 on the same hardware

[v2e]

Hello! After upgrading from ESP32-CAM-Video-Recorder-junior-60x.4.7 to ESP32-CAM-Video-Recorder-junior-62 I noticed that the frame rate had dropped to about 9 frames per second for 800x600 video from OV2640 camera. The older version (v60) produced 25 frames per second easily, with the same settings and the same hardware.

Is it some kind of regression, or does v62 require some additional tuning?

[jameszah]

I think it is just terminology. The new esp32-cam library changed the numbers for framesize, so 720p used to be 11, but now 13 ... something like that. So your config file needs to change going from 60 to 62. I'll post a link below.

[jameszah]

Old file is config.txt, new file is config2 txt
Many changes last few weeks ... some unposted

New

desklens - camera name for files, mdns, etc
13 - framesize 13 is hd 720p 1280x720

• for ov2640 camera 10=vga, 11=svga, 12=xga, 13=hd, 14=sxga, 15=uxga
• for ov5640 camera add 16=fhd, 19=qxga, 20=qhd, 23=qsxga

Old

lapselens // camera name for files, mdns, etc
13 // framesize 8=vga, 9=svga, 10=xga, 11=hd, 12=sxga, 13=uxga, 14=fhd, 17=qxga, 18=qhd, 21=qsxga

[v2e]

I think it is just terminology. The new esp32-cam library changed the numbers for framesize, so 720p used to be 11, but now 13 ... something like that. So your config file needs to change going from 60 to 62. I'll post a link below.

No, I actually took those changes into account, and double checked that the video is 800x600 in both cases.

Here is some additional information from the serial port log:

Ready to start

Start the avi ... at 6554
Framesize 11, quality 6, length 60 seconds

Starting an avi 
File open: /RECORDER003_ESP32_176/RECORDER003_ESP32_176.001.avi

Recording 60 seconds.
Frame     50, len  21829, extra     0, cam time       0,  sd time   91 -- 
So far: 0110 frames, in   12.5 seconds, for last 100 frames: avg frame size   21.7 kb, 8.90 fps ...
Frame    150, len  23090, extra     0, cam time       0,  sd time   63 -- 
So far: 0210 frames, in   22.9 seconds, for last 100 frames: avg frame size   21.6 kb, 9.62 fps ...
Frame    250, len  22028, extra     0, cam time       0,  sd time   63 -- 
So far: 0310 frames, in   33.3 seconds, for last 100 frames: avg frame size   21.6 kb, 9.67 fps ...
Frame    350, len  22091, extra     0, cam time       0,  sd time   60 -- 
So far: 0410 frames, in   43.7 seconds, for last 100 frames: avg frame size   21.7 kb, 9.57 fps ...
Frame    450, len  22193, extra     0, cam time       0,  sd time   59 -- 
So far: 0510 frames, in   54.4 seconds, for last 100 frames: avg frame size   21.7 kb, 9.37 fps ...
Frame    550, len  22201, extra     0, cam time       0,  sd time  113 -- 
End the Avi
End of avi - closing the files

*** Video recorded and saved ***

Recorded   563 frames in    60 seconds
File size is 12504236 bytes
Adjusted FPS is  9.37
Max data rate is 199782 bytes/s
Frame duration is 106751 us
Average frame length is 22196 bytes
Average picture time (ms) 0.87
Average write time (ms)   87.90
Normal jpg % 100.0
Extend jpg % 0.0
Bad    jpg % 0.00000
Slow sd writes 0, 0.000 % 
Writng the index, 563 frames
---

waiting for cam        124ms,  0.2%
Time in camera         487ms,  0.8%
waiting for sd           0ms,  0.0%
Time in sd           49649ms, 82.4%
web (core 1)             0ms,  0.0%
web (core 0)             0ms,  0.0%
time total           60243ms, 100.0%
Card  space: 29844MB
Total space: 29832MB
Used  space:    35MB
Nothing deleted, 0.1% disk full
End the avi at 66888.  It was 563 frames, 66888 ms at 9.38 fps...

And one thing particularly alarms me is the "Average write time (ms) 87.90" (which roughly corresponds to 10 FPS). And the corresponding line in statistics at the end: "Time in sd 49649ms, 82.4%". Are these numbers OK?

[jameszah]

I'll have a look. The camera should be limited to 25 fps for vga, 12.5 fps 720p or 6 fps uxga, so it must be sd issues or some other processing. I did change the jpg queueing a little which did not effect things (I think). My suspicion is the sd ... maybe i've done something strange with svga, as I normally use vga or 720p.

[jameszah]

First test looks okay - sd times bouncing around a little - old no-name 64GB sd card
svga seems bound by sd write rather than camera rate, while other two are definitely camera bound.

720p

09:56:44.532 -> So far: 0610 frames, in   57.0 seconds, for last 100 frames: avg frame size   37.1 kb, 12.15 fps ...
09:56:47.766 -> Frame    650, len  36648, cam time       0,  sd time   54 -- 
09:56:52.654 -> So far: 0710 frames, in   65.1 seconds, for last 100 frames: avg frame size   36.7 kb, 12.34 fps ...
09:56:55.813 -> Frame    750, len  35560, cam time       0,  sd time   62 -- 
09:57:00.528 -> So far: 0810 frames, in   73.0 seconds, for last 100 frames: avg frame size   36.1 kb, 12.64 fps ...
09:57:03.722 -> Frame    850, len  34893, cam time       0,  sd time   51 -- 

vga

78 -> So far: 0510 frames, in   58.6 seconds, for last 100 frames: avg frame size   11.3 kb, 24.99 fps ...
09:59:14.084 -> Frame    550, len  11285, cam time      10,  sd time   16 -- 
09:59:16.496 -> So far: 0610 frames, in   62.6 seconds, for last 100 frames: avg frame size   11.4 kb, 25.06 fps ...
09:59:18.041 -> Frame    650, len  11234, cam time      10,  sd time   17 -- 
09:59:20.5

svga

10:02:22.145 -> So far: 0710 frames, in   72.2 seconds, for last 100 frames: avg frame size   19.1 kb, 20.42 fps ...
10:02:24.153 -> Frame    750, len  20408, cam time       0,  sd time   66 -- 
10:02:27.148 -> So far: 0810 frames, in   77.2 seconds, for last 100 frames: avg frame size   19.0 kb, 20.08 fps ...
10:02:29.025 -> Frame    850, len  18148, cam time       0,  sd time   26 -- 
10:02:32.294 -> So far: 0910 frames, in   82.3 seconds, for last 100 frames: avg frame size   19.3 kb, 19.48 fps ...
10:02:34.225 -> Frame    950, len  17314, cam time       0,  sd time   87 -- 
10:02:37.019 -> So far: 1010 frames, in   87.1 seconds, for last 100 frames: avg frame size   18.3 kb, 21.09 fps ...

[v2e]

Here is the log for the v60 for comparison (same hardware, same settings, 800x600, quality 6):

Start the avi ... at 62635
Framesize 11, quality 6, length 60 seconds

Starting an avi 
File open: /RECORDER003_ESP32_181.002.avi

Recording 60 seconds.
Frame   1250, len  38880, extra     0, cam time      35,  sd time   26 -- 
Total space: 29832MB
Used space: 260MB
Nothing deleted, 0.9% disk full
Frame     50, len  38602, extra     0, cam time      29,  sd time   36 -- 
So far: 0110 frames, in    5.0 seconds, for last 100 frames: avg frame size   38.2 kb, 22.13 fps ...
Frame    150, len  40053, extra     0, cam time      33,  sd time   29 -- 
So far: 0210 frames, in    9.0 seconds, for last 100 frames: avg frame size   39.4 kb, 25.02 fps ...
Frame    250, len  39920, extra     0, cam time      25,  sd time   28 -- 
So far: 0310 frames, in   13.5 seconds, for last 100 frames: avg frame size   39.0 kb, 22.14 fps ...
Frame    350, len  40058, extra     0, cam time      31,  sd time   36 -- 
So far: 0410 frames, in   18.0 seconds, for last 100 frames: avg frame size   39.2 kb, 22.14 fps ...
Frame    450, len  39063, extra     0, cam time      34,  sd time   34 -- 
So far: 0510 frames, in   22.5 seconds, for last 100 frames: avg frame size   37.8 kb, 22.14 fps ...
Frame    550, len  39072, extra     0, cam time      35,  sd time   25 -- 
So far: 0610 frames, in   26.5 seconds, for last 100 frames: avg frame size   38.4 kb, 25.01 fps ...
Frame    650, len  39139, extra     0, cam time      18,  sd time   25 -- 
So far: 0710 frames, in   31.1 seconds, for last 100 frames: avg frame size   38.2 kb, 22.14 fps ...
Frame    750, len  39130, extra     0, cam time      33,  sd time   34 -- 
So far: 0810 frames, in   35.6 seconds, for last 100 frames: avg frame size   38.1 kb, 22.14 fps ...
Frame    850, len  38506, extra     0, cam time      33,  sd time   29 -- 
So far: 0910 frames, in   40.1 seconds, for last 100 frames: avg frame size   37.3 kb, 22.13 fps ...
Frame    950, len  38415, extra     0, cam time      35,  sd time   26 -- 
So far: 1010 frames, in   44.1 seconds, for last 100 frames: avg frame size   38.0 kb, 25.02 fps ...
End the Avi
End of avi - closing the files

*** Video recorded and saved ***

Recorded  1371 frames in    59 seconds
File size is 53604860 bytes
Adjusted FPS is 22.87
Max data rate is 899003 bytes/s
Frame duration is 43732 us
Average frame length is 39085 bytes
Average picture time (ms) 27.14
Average write time (ms)   31.51
Normal jpg % 99.9
Extend jpg % 0.0
Bad    jpg % 0.00000
Writng the index, 1371 frames
---

waiting for cam      43131ms, 71.7%
Time in camera       37213ms, 61.9%
waiting for sd        9425ms, 15.7%
Time in sd           43507ms, 72.3%
web (core 1)             0ms,  0.0%
web (core 0)             0ms,  0.0%
time total           60146ms, 100.0%
End the avi at 123000.  It was 1371 frames, 123000 ms at 22.85 fps...

[jameszah]

ESP32-CAM-Video-Recorder-junior/v62/ESP32-CAM-Video-Recorder-junior-62/ESP32-CAM-Video-Recorder-junior-62.ino

Line 131 in ee65f22

#define fbs 1 // was 64 -- how many kb of static ram for psram -> sram buffer for sd write

My only idea is this issue: Try to change the fbs to 4 or 8 (from 1).

It is the amount of static ram used to write to the sd card using the dma writing. I think it writes 512 byte chunks, and then the driver writes another chunk, and then the user code starts and gives the driver another 1k or 4k. It did not seems to effect my setup. It will be slower if you are doing lots of other things like streaming or file downloading. Streaming would only need sthe cpu, which should be lower prio, but the downloading would strain the sd that the avi writer is using. I lowered it to save precious bytes of ram, but it should have 4kb or 8kb spare.

What kind of SD card are you using? I assume you have the standard old esp32-cam ai-thinker style module.

[sschoombie]

Hi,
First off @jameszah what a fantastic piece of code you have produced, your attention to detail is commendable.

I am having the same problem as @v2e going from v60 to v62 using the following hardware:

• Ai-Thinker ESP32-S
• OV2640 cam
• SanDisk Ultra 16 GB card, class 10 (A1)

I have tried the fbs solution but it did not make much difference (went from ~10 fps to ~13 fps). I tried fbs at 8 as well as 60, see logs below:

v62 - fbs 60 -

Start the avi ... at 6045
Framesize 10, quality 20, length 60 seconds

File open: /desklens028/desklens028.001.avi
Frame     50, len  27334, extra     0, cam time      53,  sd time   15 -- 
So far: 0110 frames, in    8.6 seconds, for last 100 frames: avg frame size   26.6 kb, 12.38 fps ...
Frame    150, len  27375, extra     0, cam time      56,  sd time   13 -- 
So far: 0210 frames, in   16.6 seconds, for last 100 frames: avg frame size   26.7 kb, 12.51 fps ...
Frame    250, len  27403, extra     0, cam time      51,  sd time   14 -- 
So far: 0310 frames, in   24.6 seconds, for last 100 frames: avg frame size   26.7 kb, 12.51 fps ...
Frame    350, len  27208, extra     0, cam time      52,  sd time   14 -- 
So far: 0410 frames, in   32.6 seconds, for last 100 frames: avg frame size   26.7 kb, 12.51 fps ...
Frame    450, len  27368, extra     0, cam time      52,  sd time   13 -- 
Frame    508, sd time very high  180 >>>   14 -- pos D3A9C8,  
So far: 0510 frames, in   40.7 seconds, for last 100 frames: avg frame size   26.6 kb, 12.42 fps ...

v62 - fbs 8

Framesize 10, quality 20, length 60 seconds

File open: /desklens019/desklens019.001.avi
Frame     50, len  28744, extra     0, cam time       0,  sd time   59 -- 
So far: 0110 frames, in    9.5 seconds, for last 100 frames: avg frame size   28.1 kb, 11.78 fps ...
Frame    150, len  28615, extra     0, cam time       0,  sd time   64 -- 
So far: 0210 frames, in   17.3 seconds, for last 100 frames: avg frame size   27.9 kb, 12.76 fps ...
Frame    250, len  28367, extra     0, cam time       4,  sd time   63 -- 
Frame    263, sd time very high 1667 >>>   71 -- pos 732160, Block 1, delay     9 ..., Block 2, delay     8 ..., Block 3, delay   169 ..., Block 4, delay   123 ..., Block 5, delay   126 ..., Block 6, delay   135 ..., Block 7, delay   126 ..., Block 8, delay   128 ..., Block 9, delay   150 ...,  
So far: 0310 frames, in   27.6 seconds, for last 100 frames: avg frame size   27.8 kb, 9.78 fps ...
Frame    350, len  28203, extra     0, cam time       0,  sd time   60 -- 

v60

Framesize 8, quality 12, length 60 seconds

File open: /desklens30.001.avi
Frame     50, len  17130, extra     0, cam time       0,  sd time   31 -- 
So far: 0110 frames, in    5.0 seconds, for last 100 frames: avg frame size   16.9 kb, 21.38 fps ...
Frame    150, len  17542, extra     0, cam time      39,  sd time   12 -- 
So far: 0210 frames, in    9.3 seconds, for last 100 frames: avg frame size   17.2 kb, 23.60 fps ...
Frame    250, len  17589, extra     0, cam time      39,  sd time   12 -- 
So far: 0310 frames, in   13.6 seconds, for last 100 frames: avg frame size   17.2 kb, 23.38 fps ...
Frame    350, len  17518, extra     0, cam time      38,  sd time   13 -- 
So far: 0410 frames, in   17.8 seconds, for last 100 frames: avg frame size   16.8 kb, 23.38 fps ...
Frame    450, len  16827, extra     0, cam time      38,  sd time   13 -- 
So far: 0510 frames, in   22.0 seconds, for last 100 frames: avg frame size   16.5 kb, 24.29 fps ...

What confused me is the average frame size. I am filming the same scene in all these examples, but the frame size is much larger in v62, despite the quality being lower.

When I lower the resolution further in v62 - framesize = 8 (i.e. 296p) I get values comparable to 480p in v60, but obviously this low resolution won't be usable.

v62 - 296p resolution, fbs 8

Framesize 8, quality 12, length 60 seconds

File open: /desklens036/desklens036.001.avi
Frame     50, len  17462, extra     0, cam time      12,  sd time   12 -- 
So far: 0110 frames, in    4.5 seconds, for last 100 frames: avg frame size   17.0 kb, 24.08 fps ...
Frame    150, len  17412, extra     0, cam time      11,  sd time   13 -- 
So far: 0210 frames, in    8.6 seconds, for last 100 frames: avg frame size   16.9 kb, 24.51 fps ...
Frame    250, len  17258, extra     0, cam time      11,  sd time   13 -- 
So far: 0310 frames, in   12.6 seconds, for last 100 frames: avg frame size   16.9 kb, 24.52 fps ...
Frame    350, len  17379, extra     0, cam time       0,  sd time   11 -- 
So far: 0410 frames, in   16.9 seconds, for last 100 frames: avg frame size   16.9 kb, 23.41 fps ...

I am interested in using v62 as v60 seems to have an artefact where it records 5 frames at the start of the video clip at the correct frame rate, followed by a 1 second delay, whereafter it continues recording at the correct frame rate again. I have done several tests on the exact timing of frames within a video clip to map the temporal drift. If it is of interest I can share in a seperate post.

[sschoombie]

Fixed this issue on my side by updating the esp32-arduino board manager to the latest version (3.1.3).

[v2e]

What kind of SD card are you using? I assume you have the standard old esp32-cam ai-thinker style module.

Yes, I use a rather cheap AI-Thinker sort of module. Here are some details from esptool.py:

Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision 3)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
Manufacturer: 5e
Device: 4016
Detected flash size: 4MB

The SD card used is labelled "Kingston CANVAS Select Plus", 32GB, Class 10.

Line 131 in ee65f22
#define fbs 1 // was 64 -- how many kb of static ram for psram -> sram buffer for sd write

My only idea is this issue: Try to change the fbs to 4 or 8 (from 1).

Changing this parameter to 4 and 8 did actually change the frame rate a great deal. Here are my statistics for fbs = 4:

Framesize 11, quality 6, length 60 seconds
...
Frame    150, len  21830, extra     0, cam time       0,  sd time   24 -- 
So far: 0210 frames, in   10.8 seconds, for last 100 frames: avg frame size   21.4 kb, 20.97 fps ...
Frame    250, len  22232, extra     0, cam time       0,  sd time   20 -- 
So far: 0310 frames, in   15.7 seconds, for last 100 frames: avg frame size   21.4 kb, 20.40 fps ...
Frame    350, len  21802, extra     0, cam time       0,  sd time   25 -- 
So far: 0410 frames, in   20.5 seconds, for last 100 frames: avg frame size   21.3 kb, 20.67 fps ...
...
Recorded  1228 frames in    59 seconds
File size is 26959440 bytes
Adjusted FPS is 20.48
Max data rate is 438838 bytes/s
Frame duration is 48839 us
Average frame length is 21940 bytes
Average picture time (ms) 0.90
Average write time (ms)   30.88
...
waiting for cam         52ms,  0.1%
Time in camera        1110ms,  1.8%
waiting for sd           3ms,  0.0%
Time in sd           38121ms, 63.4%
web (core 1)             0ms,  0.0%
web (core 0)             0ms,  0.0%
time total           60144ms, 100.0%

and for fbs = 8:

Framesize 11, quality 6, length 60 seconds
...
Frame    150, len  22843, extra     0, cam time       0,  sd time   17 -- 
So far: 0210 frames, in    9.7 seconds, for last 100 frames: avg frame size   22.4 kb, 21.86 fps ...
Frame    250, len  23113, extra     0, cam time       0,  sd time   19 -- 
So far: 0310 frames, in   13.7 seconds, for last 100 frames: avg frame size   22.4 kb, 24.84 fps ...
Frame    350, len  22927, extra     0, cam time       0,  sd time   20 -- 
So far: 0410 frames, in   18.3 seconds, for last 100 frames: avg frame size   22.4 kb, 21.87 fps ...
...
Recorded  1375 frames in    60 seconds
File size is 31519952 bytes
Adjusted FPS is 22.91
Max data rate is 526966 bytes/s
Frame duration is 43657 us
Average frame length is 22910 bytes
Average picture time (ms) 0.94
Average write time (ms)   20.10
...
waiting for cam         38ms,  0.1%
Time in camera        1286ms,  2.1%
waiting for sd           2ms,  0.0%
Time in sd           28344ms, 46.7%
web (core 1)             0ms,  0.0%
web (core 0)             0ms,  0.0%
time total           60696ms, 100.0%

So, for fbs = 8 the frame rate seems to be essentially the same as for v60. Thank you very much for this solution!

I wonder, to what extent the fbs parameter can be changed? Is there any point in increasing it further? And is it actually possible (e.g. won't it break things randomly)?

[jameszah]

Lots of info here 😄

re: fbs

The program has about 95kb of memory when everything starts, and assigning more to fbs will reduce this, and if it gets too small then the wifi streaming, capturing photos, downloading, and some housekeeping jobs (deleting old files) will fail for lack of memory. Plus c++ is allocating memory and doing its garbage collection to free up the memory. When the memory runs out things just fall apart.

There is no reason to increase it above the framesize - which is about 20kb in these cases. I'll have to do some digging on what is going on there.

re: sd time very high 180 >>> 14 -- pos D3A9C8,

These lines are very interesting. They happening whenever a single frame sd write takes 10 times the average for the video being written. Here 180 ms, when the average is 14ms. Cheap 64GB sd cards seem to have this problem -- sometimes for 20-30 frames in a row (maybe a megabyte), and then recover for 1000 frames. I'm not sure if this happens in expensive brand name cards. I cannot find much on google about it. If it happened with expensive cameras I would think it would be a big issue. Maybe there is enough memory and expensive devices that the writing can catchup after getting to a slow megabyte within the flash memory ???

[v2e]

Hello!
Here is another comparison of v60 vs. v62 for different resolutions and quality factors. I took a 160° camera and did my best to maintain the picture the same in all cases.

I also set the fbs = 8 for v62.

Framesize 10 (640x480), quality 12
v60: Adjusted FPS is 23.64, Average frame length is 22485 bytes
v62: Adjusted FPS is 23.00, Average frame length is 21973 bytes

Framesize 11 (800x600), quality 12
v60: Adjusted FPS is 22.75, Average frame length is 33207 bytes
v62: Adjusted FPS is 20.47, Average frame length is 32900 bytes

Framesize 10 (640x480), quality 6
v60: Adjusted FPS is 22.86, Average frame length is 33616 bytes
v62: Adjusted FPS is 20.89, Average frame length is 32036 bytes

Framesize 11 (800x600), quality 6,
v60: Adjusted FPS is 20.52, Average frame length is 49721 bytes
v62: Adjusted FPS is 15.61, Average frame length is 47881 bytes

I have also noticed that the #define fbs is also present in this line of code:

ESP32-CAM-Video-Recorder-junior/v62/ESP32-CAM-Video-Recorder-junior-62/ESP32-CAM-Video-Recorder-junior-62.ino

Line 2595 in ee65f22

#define fbs 4 // how many kb of static ram for psram -> sram buffer for sd write

I tried to comment it out, but it did not actually change the above statistics.

[jameszah]

re: 2nd fbs on line 2595
Hmmm. I lose marks for that. Probably would not affect things, but will change name of fbs used on lines 2500 to 2700 (about 6 occurances I think)

Those numbers you gave are for original settings of v62 at fbs=1, and v60 at fbs=8 I assume?

They look the same, except the strange case of 800x600 in v62.

If fbs is 8, then there is a dividing line at a 49,152 bytes. It would need 6 blocks below that and 7 blocks above that, and that is about where you are in those averages. So that might be something, but 20 fps to 15 fps still looks quite dramatic.

[v2e]

Those numbers you gave are for original settings of v62 at fbs=1, and v60 at fbs=8 I assume?

Oh, I am sorry, I forgot to mention that I used fbs = 8 for v62. I updated my previous comment accordingly.