Lots of random missed Tesla data

[rwskinner]

It appears I'm not getting data from the Tesla api at times. On the Ubuntu server that docker running on it, had an expired SSL certificate because I shutdown my personal website and no longer have a public IP. This of course cause certbot to fail renewing the cert. I removed certbot, and set my site up for port 80 just for my local access. This is about the same time frame I noticed lots of dropped data (red line).

Could that have been the cause and if so, best way to fix? Are there any logs or anything to view?

[rwskinner]

It looks like it's at the 5 minute update intervals.

[zigam]

The fact that errors are elevated exactly on the 5-minute mark makes me think it's a rogue scheduled job, or perhaps issue with 5-minute aggregation for energy charts. This has been going on for 2 days now.

[jasonacox]

Great point! That would rule out a single node behind a LB being the fault since the 5m interval is so precise. It could be that they also added new data / scheduled work, or it could be that they hit a tipping point (number of PW installations growing) where the time to process all the sites exceeded the time limit hardcoded at the proxy layer. I see these errors getting passed back:

08/02/2025 04:30:30 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Code 504: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/v2/energy_site/status =\u003e Gateway Timeout","error_description":"","txid":"d68cdffac10aff25c473aaa7b8c99dd4"}
08/02/2025 04:30:30 PM [pypowerwall.fleetapi.fleetapi] [ERROR] Code 504: {"response":null,"error":"https://powergate.prd.sn.tesla.services:443/api/v2/energy_site/status =\u003e Gateway Timeout","error_description":"","txid":"17bb20c2d65b7e0eff54879dcf43ffba"}

If it doesn't get resolved, we can look at adjusting the telegraf polling to avoid the 5m mark, or better yet, instead of doing math on the "no response" data, just store it as a null (gap).

@zigam are these impacting anything on Netzero? I haven't tested any of the control APIs to see if those are impacted at the 5m mark as well.

[zigam]

@zigam are these impacting anything on Netzero? I haven't tested any of the control APIs to see if those are impacted at the 5m mark as well.

It's causing some issues when refreshing energy charts in Netzero. From what I can tell, it doesn't affect other endpoints.

[zigam]

The errors have subsided in the last few hours (after a larger outage that affected the Tesla app as well). Hopefully it sticks.

[rwskinner]

I see that as well. A 15 minute outage around 11:30 am CST then afterwards it appears to be solid.