Reboots since 14.2.0 on ESP32-C3 / C6 with custom firmware: script, sml, web

[ottelo9]

Hi there. I'm using Gitpod to compile my own Firmware images (with script, sml, homeassistant,...) and offer the builds on my blog. I heavily use the script web function (>W section) together with SML to read an visualize smart meter values directly in tasmota. Up to V14.1.0 everything runs well on all EP32 variants. But from 14.3.0 (and I think 14.2.0 too) I and other users experience reboots every few minutes when we display the Tasmota main page with all the values. Even if I reduce the HTML content (>W script section) to a minimum, the ESP reboots every ~30-60minutes (I can speed up this if I reload the page every 10s). But this happens only on a ESP32-C3 (compiled with tasmota32c3). If I try the same on a ESP32-WROOM board (compiled with -e tasmota32) everything runs well. I spoke to the tasmota script developer @gemu2015 and we went through a few tests. But we have run out of ideas.

I compile the images with these defines:
https://github.com/ottelo9/tasmota-sml-script/blob/main/0_Tasmota_Image_erstellen_ESP32.txt

And use this script (activated auto reload 10s):
https://github.com/ottelo9/tasmota-sml-script/blob/main/2_Tasmota_SML_Script_Chart_PV.txt

I captured 2 reboots, here is the serial log (level 4).
MobaXterm_COM4USBSerialPortCOM4_20250117_192718.txt

reference to the issue on my repo (german):
ottelo9/tasmota-sml-script#1

[Jason2866]

Well, not really something we can help. Blocking graphics output in GUI which takes too long will end in a crash. To save resources reduce drivers to the one really needed and reduce graphics.
The underlying actual core is more restrictive and does not tolerate to long taking tasks as cores before.
To be clear this is not a bug. The script has to take care not to take to long!

If older versions work well for your use cases, stay on this versions. Newer Tasmota versions in general "only" adds new features. As always "don't fix ain't broken"

[Jason2866]

As always "don't fix ain't broken"

every of these old version will become obsolete at some time or become a security nightmare due to a newly discovered vulnerability. Therefore I can't understand such a recommendation considering all the known IT threads!

Well, this versions are far away from being old. AND Tasmota has to be used in an closed WiFi network.

[ottelo9]

@Jason2866 Thanks for your answer. Then the ESP32 WROOM is simply faster than the C3/C6.
And the funny thing is, the Script runs on the ESP8266 without any reboots. Apparently because there are far fewer drivers and features active there.
Then I'll try to deactivate more drivers. Otherwise I'll remove Berry if there's no other option. But I was advised against that.

Whats about this (platformio_override_sample.ini):
; *** comment the following line if you dont use LVGL in a Tasmota32 build. Reduces compile time
lib/libesp32_lvgl
; *** uncomment the following line if you use Bluetooth or Apple Homekit in a Tasmota32 build. Reduces compile time
lib/libesp32_div

[Jason2866]

esp8266 can not directly compared to the esp32. Different chip, SDK, Arduino, less features and core is way more tolerant against too long taking tasks before crashing.

This part does not influence resulting firmware. It just does not compile the unneeded LVGL, BT code when not used in the firmware.

[Jason2866]

There are many other drivers which can be disabled. In your config, i see only enabling SML driver stuff but nowhere disabling driver you will never need for your use case.

[ottelo9]

I now compiled tasmota32 for c3 with almost all drivers/features disabled with this configs. First reboot after 25 min (main menu web page opened with charts and sml data - script enabled). So 20 min later then before.

my_user_config.h

#undef SET_ESP32_STACK_SIZE
#define SET_ESP32_STACK_SIZE (12 * 1024)

#undef USE_DOMOTICZ    
#undef USE_EMULATION_HUE
#undef USE_EMULATION_WEMO
#undef ROTARY_V1
#undef USE_SONOFF_RF
#undef USE_SONOFF_SC
#undef USE_TUYA_MCU
#undef USE_ARMTRONIX_DIMMERS
#undef USE_PS_16_DZ
#undef USE_SONOFF_IFAN
#undef USE_BUZZER
#undef USE_ARILUX_RF
#undef USE_SHUTTER
#undef USE_DEEPSLEEP
#undef USE_EXS_DIMMER
#undef USE_DEVICE_GROUPS
#undef USE_PWM_DIMMER
#undef USE_SONOFF_D1
#undef USE_SHELLY_DIMMER
#undef SHELLY_CMDS
#undef SHELLY_FW_UPGRADE
#undef USE_LIGHT
#undef USE_WS2812
#undef USE_MY92X1
#undef USE_SM16716
#undef USE_SM2135
#undef USE_SM2335
#undef USE_BP1658CJ
#undef USE_BP5758D
#undef USE_SONOFF_L1
#undef USE_ELECTRIQ_MOODL
#undef USE_LIGHT_PALETTE
#undef USE_LIGHT_VIRTUAL_CT
#undef USE_DGR_LIGHT_SEQUENCE
#undef USE_DS18x20
#undef USE_I2C
#undef USE_SERIAL_BRIDGE
#undef USE_ENERGY_SENSOR 
#undef USE_ENERGY_DUMMY
#undef USE_HLW8012      
#undef USE_CSE7766     
#undef USE_PZEM004T
#undef USE_PZEM_AC
#undef USE_PZEM_DC
#undef USE_MCP39F501
#undef USE_BL09XX       
#undef USE_DHT
#undef USE_IR_REMOTE

//-- SML, Script, Google Chart Support
#define USE_SCRIPT
#define USE_SML_M
#undef USE_RULES
#define USE_GOOGLE_CHARTS
#define LARGE_ARRAYS
#define USE_SCRIPT_WEB_DISPLAY
#define USE_SML_AUTHKEY
#define USE_HOME_ASSISTANT
#define USE_ESP32_SW_SERIAL
#define USE_UFILESYS
#undef UFSYS_SIZE
#define UFSYS_SIZE 16384  //ESP32

and this custom build

[env:tasmota32c3-ottelo]
extends                     = env:tasmota32_base
board                       = esp32c3
build_flags                 = ${env:tasmota32_base.build_flags}
build_unflags               = ${env:tasmota32_base.build_unflags}
lib_ignore                  = ${env:tasmota32_base.lib_ignore}
                              Micro-RTSP

[Jason2866]

The exception is as i guessed. The script (GUI) task takes too long. Exactly this crash happens in this case.
The C3 is the slowest from all MCUs. Probably less crashes with an esp32.

[Jason2866]

Why should the faster esp32 fail too? It is a edge case. Most of the time the C3 is fast enough.
The way more faster esp32 which has a different chip architecture too, is fast enough to do the job.
Btw. Tasmota is not running on the H2

[Staars]

@mi-hol
Can you try a custom build with the following addition to your build section? (i.e. in platform_tasmota_cenv.ini)
custom_sdkconfig = '# CONFIG_ESP_SYSTEM_HW_STACK_GUARD is not set'

It could be an unresolved error in the ESP-IDF.

[mi-hol]

it could be an unresolved error in the ESP-IDF.

Great suggestion, we'll try and report back

[Staars]

Edit: Forgot the #

[ottelo9]

@mi-hol Can you try a custom build with the following addition to your build section? (i.e. in platform_tasmota_cenv.ini) custom_sdkconfig = '# CONFIG_ESP_SYSTEM_HW_STACK_GUARD is not set'

It could be an unresolved error in the ESP-IDF.

Unfortunately, that didn't help. Reboot after 6mins.
MobaXterm_COM4USBSerialPortCOM4_20250118_110643.txt

my next steps are:

• remove -DFIRMWARE_TASMOTA32 from build_flags
• add only absolutely necessary drivers/features
  FIRMWARE_TASMOTA32 uses this drivers (extracted from tasmota_configurations_ESP32.h)
  FIRMWARE_TASMOTA32 defines.txt

All drivers actually included in my image are (extracted via decode-status.py):
status 14.3.0 ottelo.txt

[Jason2866]

Anyways i will stop here, everything relevant is told.

[ottelo9]

I figured it out what feature triggers the reboot after some time:

USE_SHUTTER

As soon as I add #undef USE_SHUTTER to my user_config_override.h Tasmota runs without any reboots.
C3 Reboot Test Script.txt
user_config_override.h.txt
platformio_tasmota_cenv.ini.txt

[mi-hol]

heavy use of the GUI stuff pushes the one core MCU to its limits.

@Jason2866 but the same C3 with same script works well on Tasmota 14.1.0 (02.07.2024).
Also the much faster C6 is affected by same instability on Tasmota releases after 14.1.0
How can these facts be explained?

[ottelo9]

I got my C6 yesterday. Sadly the workaround (remove almost all features) doesn't work for the C6. The ESP reboots every ~5min. Only the following features are enabled:

USE_CONFIG_OVERRIDE
USE_COUNTER
USE_DEEPSLEEP
USE_HOME_ASSISTANT
USE_IMPROV
USE_LIGHT
USE_PING
USE_SCRIPT
USE_SML_M
USE_SPI
USE_SUNRISE
USE_TIMERS
USE_TIMERS_WEB
USE_UFILESYS
USE_WEBSERVER
WEBSERVER_ADVERTISE

[mi-hol]

(remove almost all features) doesn't work for the C6.

what about removing USE_DEEPSLEEP as a trial?

Also in in your earlier config USE_ESP32_SW_SERIAL was set, was it now removed?

//-- enables additional software serial channels for ESP32, (receive only), define pin with '-' sign to assign to software serial
#ifndef USE_ESP32_SW_SERIAL
#define USE_ESP32_SW_SERIAL
#endif

[ottelo9]

what about removing USE_DEEPSLEEP as a trial?
#ifndef USE_ESP32_SW_SERIAL
#define USE_ESP32_SW_SERIAL
#endif

I'm testing that right now. 13min without reboot...

Update:
and reboot again :(

[mi-hol]

@gemu2015 @ottelo9

I figured it out what feature triggers the reboot after some time:

USE_SHUTTER

As soon as I add #undef USE_SHUTTER to my user_config_override.h Tasmota runs without any reboots. C3 Reboot Test Script.txt

I referenced your reproduction (but from german repository) in espressif/esp-idf#14456 (comment) and got question back from developer. But I can't answer it.
Would you mind to step in and respond?

[ottelo9]

@Jason2866

Soneone created an issue: espressif/esp-idf#14456 that exactly describe our actual reboot problem. That issue found @gemu2015. So it's an CORE3 error within ESPIDF. Workaround is currently to turn off the stack protector inside sdkconfig.

Update:
I changed the url in platformio_tasmota32.ini to

[core32]
platform = https://github.com/tasmota/platform-espressif32/releases/download/2025.02.30/platform-espressif32.zip

The new framework includes this workaround (see issue above) Jason2866/esp32-arduino-lib-builder@5e7fb8f

[Jason2866]

@ottelo9 Stack protection is off in actual framework (used in Tasmota branch development). So if it crashes it is not this bug. Jason2866/esp32-arduino-lib-builder@5e7fb8f

[gemu2015]

Read the issue , if it were off it would not trigger.
According to the issue it explicitly has to be set to “no”

[Jason2866]

@gemu2015 Thats the way to set to no like it is done ;-) Check the sdkconfig file in the framework

[ottelo9]

@Jason2866
Since I don't use the Dev branch to create the images (I use 14.4.1), I think it's still active in my build?! I will test it later with disabled protection. How can I change the esp32-arduino-lib-builder release inside my VSC project?

[gemu2015]

ok, since it seems that @Jason2866 changed this about 3 weeks ago this might explain that @ottelo9 build still has this bug. anyhow it seems clear now that this caused @ottelo9 s reboot problem.

[gemu2015]

Guru Meditation Error: Core 0 panic'ed (Stack protection fault).
Detected in task "loopTask" at 0x4205130a
Stack pointer: 0x40824750
Stack bounds: 0x40821f58 - 0x40824f50
So why is this error detected when stack protection is off?
it is exactly the same issue stack is within bounds and stack protector triggers which is obviously faulty

[Jason2866]

If for sure this platform is used platform = https://github.com/tasmota/platform-espressif32/releases/download/2025.02.30/platform-espressif32.zip it is time to open an issue in espidf and ask there.

[ottelo9]

If for sure this platform is used platform = https://github.com/tasmota/platform-espressif32/releases/download/2025.02.30/platform-espressif32.zip it is time to open an issue in espidf and ask there.

I used this in my builds:
[core32]
platform = https://github.com/tasmota/platform-espressif32/releases/download/2025.01.31/platform-espressif32.zip

I'm now using the 2025.02.30 framework and testing the image on my C6.

[ottelo9]

Now running for 13h without reboot (Script see first post). So I think the workaround works :)