Error accessing device after a few hours working

[vjbutt]

I am getting the status from a WiFi Dual Meter every 10s using tinytuya. Everything works fine for a few hours. Then I start getting the following error:

Unexpected error: <class 'KeyError'>

I can still read the meter data using the Smart Life app whilst I am getting this error. However, in order to get the status working again using tinytuya, I have to power cycle the WiFi Dual Meter. Any idea what could be wrong?

[jasonacox]

Hi @vjbutt can you detail or link about the Tuya device (WiFi Dual Meter)?

I have had a few older devices that would stop responding to the local network requests but would continue to work with the cloud (Smart Life). Power cycling them was the only that that would fix them.

It would be interesting to see if you could run a monitoring loop with a persistent connect to see if that helps. See https://github.com/jasonacox/tinytuya/blob/master/examples/monitor.py as an example.

[vjbutt]

@jasonacox thanks for your suggestions, I will try that plus a few other things. It seems that I have bought one of these older devices :(.

The error that I reported above is wrong (that was actually due to a missing DP ID in one of the returned data). The actual error that I am getting when the WiFi Dual Meter gives up on the local connection is:

{'Error': 'Check device key or version', 'Err': '914', 'Payload': None}

The device key is definitely correct, because when I power cycle the device it works. I will next check the version. I am currently setting this to 3.4, as is reported by the device, but maybe I should set this to 3.3.

[vjbutt]

@jasonacox I have tried setting the version 3.3, but tinytuya does not even manage to get the status. So the correct value to set is actually 3.4.

I still need to find some time to check with a monitoring loop over a persistent connection.

BTW the device is this one - https://www.amazon.co.uk/Energy-Electricity-Monitor-Control-Management/dp/B0CDXF3J1K/ref=cm_cr_arp_d_product_top?ie=UTF8

Haven't managed to find a link to the manufacturer though :-(

[uzlonewolf]

I have some v3.4 devices which do this, and my suspicion is memory corruption within the device is causing it to "forget" what the correct local key is. How often it happens is pretty random; one day it may do it 3 times that day, and then it will be fine for a month. I haven't found a solution other than rebooting it.

[vjbutt]

@uzlonewolf that was exactly what I was thinking, that somehow there must be some memory corruption. However, today I was surprised to note that the device started to respond to status updates again after about 1 hour of giving {'Error': 'Check device key or version', 'Err': '914', 'Payload': None} (without necessitating a power cycle). So maybe it is not memory corruption.

Furthermore, just before it started to respond correctly again to status queries, it returned the following status:

{'protocol': 4, 't': 20, 'data': {'dps': {'2': 1887}}, 'dps': {'2': 1887}}

It seems to me that the device is somehow switching to protocol 4. Why should it do this is beyond me. The problem is that tinytuya does not support version 4, so maybe those error messages were being generated because the device switched to version 4. Note that these protocol 4 messages are being received also in between correctly received status updates. They seem to be returned quite at random.

One other thing to note is that the time stamp ('t':20) returned in the above status seems to indicate that the device had rebooted. From previous protocol 4 status messages, it is clear that this is a Unix timestamp. So a timestamp of 20 seems to indicate that not even the real-time clock on board of the device had yet been updated before this status was returned. This may indicate a more serious case of memory corruption.

Will keep monitoring.

[uzlonewolf]

The 'protocol' in the payload does not do anything as far as I can tell and is only present in packets which contain a "protocol version" header directly before the payload. Devices set it to 4 while the SmartLife app sets it to 5.

[vjbutt]

@uzlonewolf, indeed that is now my impression as well. The presence of these protocol 4 packets increases drastically when I've set set_socketPersistent(True). They basically contain a very small subset of the DPS data that eventually follows in a full data payload.

Interestingly, setting a persistent connection also seems to have solved my problem in that the device is still alive and supplying "valid" data after more than 24 hours of continuous operation. However, it has now created a new problem. The status command is returning repeated payloads. I still see the measurements being updated on the Smart Life app, but not in tinytuya. As a consequence the returned data is practically useless. I have tried to force the update of the DPS using the function updatedps() but this does not seem to resolve the issue.

Why would a persistent connection cause repeated payloads and not return the actual data being measured? Note that I am only reading the status every 10s, which is compatible with the value set in the report rate control.

[Dronec]

Got the same situation with this device:
https://www.ebay.com.au/itm/355760971428
Querying it every 5 min, after approximately 12 hours it stops responding with
"Check device key or version"

[dwmw2]

I have a similar problem with a Ciarra cooker hood. I was connected to it and it was happily reporting status, and suddenly it started returning the same error. I was running this:

#!/bin/python3

import tinytuya
d = tinytuya.OutletDevice(dev_id='xxx', address='172.31.31.209', local_key='xxx', version=3.4)


data = d.status()
print('Received Payload: %r' % data)
while (True):
      data = d.receive()
      print('Received Payload: %r' % data)
      if not data:
        print(" > Send Heartbeat Ping < ")
        d.heartbeat()
      print(" > Request Update for DPS indexes 18, 19 and 20 < ")
      result = d.updatedps([4, 5, 6, 18, 19, 20])
      data = d.status()
      print('Received Payload: %r' % data)

.. and it just stopped:

$ python3 hood.py 
Received Payload: {'dps': {'1': False, '2': '1', '6': False, '12': '0', '15': 66, '104': 'Level_1'}}
Received Payload: None
 > Send Heartbeat Ping < 
 > Request Update for DPS indexes 18, 19 and 20 < 
Received Payload: {'dps': {'1': False, '2': '1', '6': False, '12': '0', '15': 66, '104': 'Level_1'}}
Received Payload: None
 > Send Heartbeat Ping < 
 > Request Update for DPS indexes 18, 19 and 20 < 
Received Payload: {'dps': {'1': False, '2': '1', '6': False, '12': '0', '15': 66, '104': 'Level_1'}}
Received Payload: None
 > Send Heartbeat Ping < 
 > Request Update for DPS indexes 18, 19 and 20 < 
Received Payload: {'dps': {'1': False, '2': '1', '6': False, '12': '0', '15': 66, '104': 'Level_1'}}
Received Payload: None
 > Send Heartbeat Ping < 
 > Request Update for DPS indexes 18, 19 and 20 < 
Received Payload: {'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
Received Payload: {'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
 > Request Update for DPS indexes 18, 19 and 20 < 
Received Payload: {'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
Received Payload: {'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
 > Request Update for DPS indexes 18, 19 and 20 < 
Received Payload: {'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
Received Payload: {'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
 > Request Update for DPS indexes 18, 19 and 20 < 

Now it reports the same error every time I try to connect. The Android app was working, and a power cycle seems to have fixed it for tinytuya too.

[Dronec]

Android app is working via the cloud, which doesn't seem to be affected by this issue. I'm starting thinking that emulating the cloud locally might be the way forward https://github.com/kueblc/mocktuyacloud

[jasonacox]

@dwmw2 - If you do not send d.updatedps([4, 5, 6, 18, 19, 20]) does it still occur? That has always been a bit of a hack to try to get these devices to update those DPS values. I wonder if it is tripping some vulnerability in the firmware (e.g. memory leak) that eventually causes it to become unstable.

@Dronec - Yes, valid point. These devices were not designed for local access as we have seen again and again. They are designed for the cloud so that tends to be the most polished and reliable aspect. I think that project you reference has an interesting idea, but doesn't look like it has been touched in 5 years.

[Dronec]

@jasonacox I've tried that without sending updates, same result.

I'm going to play with that project over the weekend, just need to wait for a new device

[jasonacox]

I guess one experiment would be to remove heartbeat to see if that works. There is a high chance that it would hit heartbeat timeout and disconnect the TCP session.

And another experiment would be to stop sending anything at all and test to see if it is just a "running time" related issue vs some bug in their local API handler that traffic aggrevates. You are still having to test it every x minutes/hours, but it would at least tel lus that this is API usage related vs a firmware fundamental flaw.

Another experiment would be to write a timer in your loop to fully disconnect (tear down TCP link) and reconnect. Theory would be that whatever resource leakage that is causing the device to become unresponsive could be averted if it still does a cleanup after a connection close. Of course that also means that once it becomes unresponsive, it is no longer able to recover itself so picking the right timer for the reset before that threshold would be important.

[oeboema]

Same here, with this heater: https://eurom.nl/en/product/permanent-heating/bathroom-heaters-permanent/sani-bathroom-radiator-1000-wifi-white/

Stops responding after a while (several hours), and complains about the key not being valid. 2 Smartplugs that I'm polling work fine. A power cycle for the heater does resolve the situation (for a while).

[jasonacox]

Hi @oeboema can you share what you are using to access the device (script or just scanning)?

[AlphaZulu666]

same problem with a Wifi Power meter, it worked for a month, then started like this.... the bad is that I couldn't find a way to powercycle it, even from the app, and that's a mess, because it is needed a physical powercycle.
Isn't there a way to do the same calls the app do to get the data?

[Dronec]

The app is working via the cloud, so you can extract the data from the cloud. I'm in exactly the same situation: failed wifi circuit breaker, which I'm getting the data from via the cloud.

[jasonacox]

Request: Please share a link to the device (e.g. amazon, aliexpress) if you can. I have found no way to remote power cycle any Tuya device. If anyone ever discovers this, please let us know.

[Dronec]

Request: Please share a link to the device (e.g. amazon, aliexpress) if you can. I have found no way to remote power cycle any Tuya device. If anyone ever discovers this, please let us know.

This is mine: https://www.ebay.com.au/itm/355760971428

[elljay234]

Hi, similar issue with a unit described as "Tuya Smart 2 Way WiFi Energy Meter Bidirection 2 Channel with Clamp" I bought from ebay in 2023. The listing has been removed so I can't post a link.

It normally works for weeks or months then seems to need a power cycle to get it going again. Yesterday and today it's done it several times!

I get a mixture of these errors:
{'Error': 'Check device key or version', 'Err': '914', 'Payload': None}
{'Error': 'Network Error: Unable to Connect', 'Err': '901', 'Payload': None}
{'Error': 'Network Error: Device Unreachable', 'Err': '905', 'Payload': None}

I can't see any model number on the case of the device - only the front is printed:

Shout if you want more details / want me to run some debug code...

[Kowa92]

Same issue with EARU Eelectric Wi-Fi Two-way Energy Meter. The first two weeks they worked well.. Now if they work a few hours without giving me the 914 error, it's already an achievement. The only way to get them to work is to cut off the power and then turn it back on (always praying that they will work at least for a while).

• Has anyone by any chance found a solution to this problem?

Post the debug output of my code:

DEBUG:TinyTuya [1.16.1]

DEBUG:Python 3.9.2 (default, Dec  1 2024, 12:12:57) 
[GCC 10.2.1 20210110] on linux
DEBUG:Using pyca/cryptography 3.3.2 for crypto, GCM is supported
DEBUG:status() entry (dev_type is default)
DEBUG:final payload_dict for 'bf8b89b198c36175bdhf3x' ('v3.4'/'default'): {1: {'command': {'gwId': '', 'devId': '', 'uid': '', 't': ''}}, 7: {'command': {'protocol': 5, 't': 'int', 'data': {}}, 'command_override': 13}, 8: {'command': {'gwId': '', 'devId': ''}}, 9: {'command': {'gwId': '', 'devId': ''}}, 10: {'command': {}, 'command_override': 16}, 13: {'command': {'protocol': 5, 't': 'int', 'data': {}}}, 16: {'command': {}}, 18: {'command': {'dpId': [18, 19, 20]}}, 64: {'command': {'reqType': '', 'data': {}}}}
DEBUG:building command 10 payload=b'{}'
DEBUG:sending payload quick
DEBUG:final payload: b'0123456789abcdef'
DEBUG:payload encrypted=b'000055aa000000010000000300000044fb87b1927b9a5e829bce171f540886e0393d8f2cb924c51ec39b1d8b918945aab01a9ba075e10fddf56ec7b77bfca6c1bfe9266c54c044c407e5407d873ec8b40000aa55'
DEBUG:session key negotiation failed on step 1
DEBUG:ERROR Check device key or version - 914 - payload: null
DEBUG:status() received data={'Error': 'Check device key or version', 'Err': '914', 'Payload': None}

Thanks

[jasonacox]

Thanks, @Kowa92 . I wonder if this is due to a polling problem. If you reduce or disable the polling of the device (no TinyTuya or localtuya calls) does it last longer?

[Dronec]

Thanks, @Kowa92 . I wonder if this is due to a polling problem. If you reduce or disable the polling of the device (no TinyTuya or localtuya calls) does it last longer?

I can confirm that if you call it only occasionally, it lasts much longer.

[jasonacox]

If someone has time to do packet traces, especially comparing TinyTuiya and SmartLife app use (local mode), it would be interesting to see if there is any indication of additional handshakes we are missing (e.g. causing the device ro recover resources). I suspect not and that repeating SmartLife app local traffic use may eventually cause the same thing (indicating bad code on the device for local API but good for cloud API). But maybe some comparison would identify traffic patterns we could adjust. Without seeing the actual code the device is running, we are fishing around in the dark on why it fails (doesn't recover) from basic local API use.

[Dronec]

If someone has time to do packet traces, especially comparing TinyTuiya and SmartLife app use (local mode), it would be interesting to see if there is any indication of additional handshakes we are missing (e.g. causing the device ro recover resources). I suspect not and that repeating SmartLife app local traffic use may eventually cause the same thing (indicating bad code on the device for local API but good for cloud API). But maybe some comparison would identify traffic patterns we could adjust. Without seeing the actual code the device is running, we are fishing around in the dark on why it fails (doesn't recover) from basic local API use.

Jason, can you please tell me more about SmartLife "local" mode? I did packet tracing it once (not too thoroughly though) and didn't notice any local communications apart of the initial registration, the rest was going through the cloud

[uzlonewolf]

My problematic v3.4 devices usually last weeks if not months before dropping offline, however I make a single persistent connection to them that lasts that entire time. When they do drop offline the SmartLife app no longer works in local mode either. I've thought about trying to packet capture them but with my months-long connection it just hasn't been practical. I'll need to see if I can scare up another problematic device that I can put on my test network.

[elljay234]

Not sure if it helps, but just looking in the Tuya cloud Device Log I can match the timing of a locally reported set of "Network Error: Unable to Connect', 'Err': '901'" errors with the device reporting to Tuya that it was offline due to a "keepalive_timeout". The online Tuya log only seems to go back seven days - my last Err 914 was almost two weeks ago! I'll have a look at the log next time I get a 914...

(The online log is here: https://platform.tuya.com/cloud/ - "Open Project", "Devices", "Debug Device", "Device Logs")

[elljay234]

Had a 914 error overnight. The log reports "Event: Device semaphore"; "Event Details: signal intensity:-60||Remaining memory:32240||UTC:1742868113||DST:1742868113", shortly followed by "Device online". The device carried on working and didn't need a power cycle. Smart Life says WiFi signal strength -50 to -70 is "Good", so maybe it was low on memory...?

[jasonacox]

Interesting... - "Device semaphore": This may refer to a synchronization mechanism used to manage access to shared resources. A semaphore is a variable that controls the access to a shared resource by multiple processes or threads. In this case, the device semaphore is likely being used to manage access to the device state? I can only speculate, but I wonder if this means some thread locking mechanism failed and maybe 914 just restarts.

[elljay234]

Had a look at the log in more detail... It only allows you to filter by one event type at a time so I copied and pasted everything except the "EVENT_TYPE_REPORT" which are the measured data into Excel and sorted by date.

First observations are:

• It reports "Online" every 12 hours. Network issues restart that routine.
• Remaining memory remains constant between "Online" events. (Except one occasion 24/3 06:33!)
• It reports "Device restart - power off reboot" when I've not power cycled the device.
• The one 914 Tinytuya error (which didn't need a power cycle) is one second after the device reported an "online" event.

Have attached the Excel file and a screenshot of the file:

TuyaDeviceLog-March.xlsx

[Dronec]

Got this error after 2 days of querying thing:

https://www.aliexpress.com/item/1005004153509348.html

Interesting thing is that I was keeping the connection open to avoid memory leaks and stuff.

{"Error": "Check device key or version", "Err": "914", "Payload": null}

[Dronec]

However, after that something strange and miraculous started happening. I've noticed that when I open the mobile app (Smart Life) the device starts responding again (and stops when I close it).

[sueastside]

@Dronec Can you share a bit more about your setup? I cannot replicate this behavior with the Smart Life app....

I know updating the firmware restarts the device and it starts working again, any way we can trigger this reboot remotely as well?

[Dronec]

I'm starting thinking that this might strongly depend on a brand of the device or even a specific device. I have 4 circuit breakers bought on AliExpress under the brand TONGOU and they all behave differently.
My setup is basically a piece of code from tinytuya examples (in repository, see threading) which writes the responses into a database day and night. One of the devices works totally as expected, other three randomly throw error 914, go offline (so even Smart life app can't see them), and then back online like nothing happened. Happy to share more details if needed