Polling via REST API leads to outliers in MQTT SENSOR data

[gone-for-coding]

Issue

I'm reading my smart meter via an IR reader and publish the metrics every 60 seconds via MQTT. This has been working rock solid for about 2 years now with no outliers in the data.

Now I try to integrate with OpenDTU to control my micro inverter. I use the REST-API http://<IP>/cm?cmnd=status%208 to poll the data.

When I activate the polling (I tested 1s, 2s and 4s) I get outliers in the data published to the MQTT SENSOR topic. In the last 24 hours I got a reading of 0 for the frequency and a reading of 500 mega volts on phase 1 for exactly one reading. Sometimes I also get a current power reading of multiple tera watts for exactly one reading. As soon as I stop polling via REST, no more outliers are observed.

Is this a bug or a known issue? I have no idea how to pin point it. I upgraded from 12.3.1 (after 795 days of uptime) to 14.6.0 but with no luck.

Another thing I observed is, that for every REST poll an MQTT message is also published. So if I poll every second, I'll also get an MQTT message every second to the STATUS topic.

Setup

• ESP8266 4M
• Tasmota 14.6.0 +Script, SML, UFS
• Teleperiod 60

I'm running the following SML-Script to read my smart meter:

>D
>B
=>sensor53 r
>M 1
+1,3,s,0,9600,pm1
1,77070100010800FF@1,Zählerstand Total,Wh,zaehlerstand_total,1
1,77070100020800FF@1,Einspeisung Total,Wh,einspeisung_total,1
1,770701000E0700FF@1,Netz Frequenz,Hz,netz_frequenz,2
1,77070100100700FF@1,aktuelle Wirkleistung,W,aktuelle_wirkleistung,0
1,77070100240700FF@1,Wirkleistung L1,W,wirkleistung_l1,0
1,77070100380700FF@1,Wirkleistung L2,W,wirkleistung_l2,0
1,770701004C0700FF@1,Wirkleistung L3,W,wirkleistung_l3,0
1,770701001F0700FF@1,Strom L1,A,strom_l1,3
1,77070100330700FF@1,Strom L2,A,strom_l2,3
1,77070100470700FF@1,Strom L3,A,strom_l3,3
1,77070100200700FF@1,Spannung L1,V,spannung_l1,2
1,77070100340700FF@1,Spannung L2,V,spannung_l2,2
1,77070100480700FF@1,Spannung L3,V,spannung_l3,2
#

[s-hadinger]

I'm doing massive polling with Matter and I didn't see any outliers. This is maybe due to this specific driver

[gemu2015]

sml telegrams may be quite large, so you can increase serial irq buffer e.g. to 512 bytes (your complete sml telegrams should fit here)
by adding this line to the descriptor:
1,=so3,48,512

since the sml telegram is guarded by a crc checksum you may also add the compile option
#define USE_SML_CRC
this is default on ESP32 but not an ESP8266

and then add this line to the descriptor:
1,=soC,512,15

you also may also use script?sml[x] to query one sml data x being the decode line number you want so sml[4] would be "aktuelle Wirkleistung" (shorter result)

[gone-for-coding]

Hi @gemu2015 ,
Thanks for your reply, it's highly appreciated.

First I tried with 1,=so3,512 which led to slightly slower updates on the webinterface but everything else worked. But sadly it did not make a difference when I activated the REST polling and I had outliers again.

By the way, what is the difference between 1,=so3,512 and 1,=so3,48,512 you posted? The documentation does not have any information about two parameters.

I then tried to compile with #define USE_SML_CRC and added 1,=soC,1024,15 to the config. This resulted in having just the first value and the rest was all 0. After increasing the buffer size to 1,=soC,1536,15 I got all values but needed to remove the so3-Option because this resulted in all 0.

Do you know if the option soC includes so3, meaning using CRC always increases the buffer size?

So I will test with just 1,=soC,1536,15 over night and report back.

I also tried your advice using script?sml[4] but as I was not able to find any documentation on this I was unable to construct the full URI.

I tested:

• http://<IP>/script?sml[4]
• http://<IP>/cm?cmnd=status%208&script=sml[4]

and a few other combinations. An advice would be highly appreciated!

[sfromis]

You can see the format for issuing console command via web requests here:
https://tasmota.github.io/docs/Commands/#with-web-requests
And don't replace the ? in the command syntax by =

And if you want multiple commands at the same time, use Backlog
(https://tasmota.github.io/docs/Commands/#the-power-of-backlog)

[gemu2015]

form the docs:
M,=so3 sets serial buffer size, serial IRQ buffer size and serial dump buffer size.
thus up to 3 parameters

is is essential that your irq buffer is large enough to fit your meters sml telegram. some meters have very large ones. so try with 1024 or even 1536
since the ESP8266 has only one core and no multitasking this is necessary. on an ESP32 this would be no issue.

IRQ buffer it is not set by the CRC option.
CRC check is an additional option that ensures data integrity. since sml telegrams have only on crc over the complete telegram your crc buffer has to be large enough

this is the correct polling format:
http:///cm?cmnd=script?sml[4]

[gone-for-coding]

Thanks @sfromis and @gemu2015 for being patient with me!

Regarding so3 I also had a look at the code https://github.com/arendst/Tasmota/blob/development/tasmota/tasmota_xsns_sensor/xsns_53_sml.ino#L177-181 which describes it a lot clearer than the docs actually. I might do a PR here to help other people.

Regarding the polling URL: That did the trick. To be honest I have never seen an URL with two ? in it and here as well I wasn't able to get that out of the docs.

Anyway: 24 hours have passed and I have no more outliers after adding 1,=soC,1536,15 to the script yesterday. The so3 option is actually not required in my setup.

[gone-for-coding]

Just for the reference: Here are two PRs:
tasmota/docs#1482
tasmota/docs#1483