KNX-Bus stuck with version > 3.1.9

[evazzoler]

Hello Supergiovane, how are you???

I'm opening this issue to continue #393 after a long time. I apologize for this, but I've changed jobs and schedules. I need to perform these tests during the day (otherwise my family remains in the dark, since 90% of my bulbs are HUE), and unfortunately, the sun sets before I get home during this period. 😬

The problem still exists: Node-RED loses a lot of packets, and those that arrive do so very late (1 or 2 minutes). It also seems that the node occasionally disconnects and reconnects. 😟

The CPU doesn't seem to be heavily loaded, except during the startup phase, but that's normal.

This problem has existed since the "delay between each telegram" parameter was introduced, which I set to 30 during tests, as you indicated.

I've discovered that the situation improves significantly if I deselect the checkbox "Echo sent message to all nodes with the same Group Address" and it becomes almost acceptable but I loose some functionalities I implemented. However, I'm very concerned because you've written in the documentation that you'll remove this option and have it selected by default. In that case, I won't be able neither to test the latest version of knx-ultimate and will have to definitively revert to the last version that works for me: 3.1.9. 😰

My architecture is ARM (Raspberry), but the device is mounted on industrial hardware (it was quite a significant investment...) that I don't want to change to make knx-ultimate work. The fact is, if we don't resolve this problem, I'll be forced to. 😵

Do you have any ideas on how I can help you help me?

Thanks in advance!

[Supergiovane]

Hi
That's not an expected behaviour. KNX-Ultimate must run flawlessy. How many KNX Ultimate nodes do you have in place on your flows? You can check it here (this is my home)

Registrazione.schermo.2025-02-09.alle.12.45.59.mov

[evazzoler]

Hi Max, and thank you for the prompt reply!
Here we are:

[Supergiovane]

Hi
91 nodes are 1/4 of the quantity of mines.
38 hue devices are fine.
What type of raspberry is installed? What is the make/model of your industrial device?
And.. i don’t remember your node version.

[evazzoler]

Hi,

• NodeRed 4.0.5 (same issue with previous versions)
• Node v22.11.0
• My Raspi: https://sferalabs.cc/product/strato-pi-cm/
• 91 nodes are not so many, in fact 3.1.9 works like a charm

[Supergiovane]

Wich option have you choosen?

[evazzoler]

4S - 1GB RAM - 32GB eMMC

But... Why investigate the hardware when an older software version has no problems?

[Supergiovane]

Hi
1 GB Ram is way too low. It’s swapping to the MMC.
The older version was using an old Node.js timing mechanism that is deprecated starting v. 20.x, so i need to keep the things up to date with a new timing mechanism.
HUE rest API are also hungry in terms of system resources (RAM).
I suggest you to buy a new CM to replace the old one in the sferalabs. Buy the one with many ram possible.

[evazzoler]

This is very bad news.
This raspberry does everything it is supposed to do, and it has always done so with the old version of knx-ultimate. I wonder if changing modules and having the latest version really gives me advantages.
It is possible that the best choice form me is keeping the old version...
With the old knx-ultimate version I have 220 MB of unused RAM...
I will do a test with the latest version to see if the RAM is actually at 100%.
Thanks.

[Supergiovane]

Be aware, that at certain point in time, you will be forced to upgrade the hardware. The ram you see free, is because your CM is probably already swapping to the eMMC.
If you have a spare well equipped raspberry laying unused, or if you can spin up a VM with debian or raspbian, you’ll see node-red run smoothly.
Otherwise, you can live with the old knxultimate version, if it fits your needs.
Anyway, I’ll investigate further in the event emitter (as you wrote, the performances improves by unchecking that “echo” option).

[Supergiovane]

Hi Enrico
just took a look at the matter.
"Echo sent messages" doesn't really do anything special and it has been forced to "true" in the knxengine module, long time ago. In the knxultimate itself, it takes care of simulating some behaviours.
Please check that the loglevel is set either to "Silent" or "Error" in the knx config node and restart node-red.
I've an unused RevPI3 (industrial grade, like yours, with din rail mount) laying around. I've upgraded the CM Module on that, but i cannot recall the specs (it's a rPi3). If you wish, i can send it to you.
Anyway, the CM module in your device is a SODIMM one, so you if you wish, i can send you only the CM and not the whole RevPI3. You can swap it by yourself, just by opening the Sferalabs.

OR (better), you could buy a standard RaspberryPi 4, with at least 4GB RAM and use an industrial SD Card (you can find it even on amazon). Such SD Cards are built to have a very long MTBF, for use in industrial environments. This, for example, has 1920 TBW with a duration of 526 years (calculating a write rate of 10GB per day).

[evazzoler]

I ordered a CM with 8GB RAM, 32GB. Extimated delivery 24/04/2025... 😆

The standard RaspberryPi 4, compared with the Sferalabs product, misses the sferalab control unit with the hardware wahtchdog (with both soft and hard reboot) connected with a GPIO where nodered sends a keepalive signal. It reboots if nodered freezes.
It's not a simple CM din rail case... I never found a real industrial SD Card. All "industrial" SD are declared industrial (but it seems are the same as non industrial) and are note safe as the DOM of the CM module... Industrial CF are instead really safe.

[Supergiovane]

Good!
I also had that revPi3, with watchdog and so on.
At the end, after many changes, i’ve opted for a Supermicro server with ecc Ram. Low power consumption (30w max)
It runs 3 Node Red VMs, an Homebridge VM and other Stuffs i use for my work.
Backed up by a little UPS.

Please let me know when it arrives. For now, i move this thread to a discussion.

[evazzoler]

Hi, I finally changed today the module (1 to 8GB RAM), after waiting it for 2 months...

Nothing changed.

The ram is almost totally free (7200 MB free), the CPU is 145%.
KNX telegrams (set at 30ms) have a high loss percentage. The system is unusable, lights don't turn on, etc.
I really don't know what to look for in order to find the problem.
Please, give me some directions to investigate.
(I'm available for a remote session with i.e. Anydesk)
Thank you!

[evazzoler]

@Supergiovane, do you think the best way is to remove every reference to knx-ultimate and restart from scratch?
Is it possible that the upgrade don't update a file and the new version is incompatible with it?

[Supergiovane]

Hi Enrico
Go again to your .node-red folder, then issue an “npm install” to update all packages.
Update also node.js and node-red to the latest version.
It can also be, that some custom libraries of Sferalabs must be updated or be re-installed.

[evazzoler]

Anyway, when I upgrade knx-ultimate, I get this warn:

pi@Strato:~/.node-red $ sudo npm install node-red-contrib-knx-ultimate

npm warn EBADENGINE Unsupported engine {

npm warn EBADENGINE   package: 'undici@7.8.0',

npm warn EBADENGINE   required: { node: '>=20.18.1' },

npm warn EBADENGINE   current: { node: 'v18.17.1', npm: '10.9.0' }

npm warn EBADENGINE }

[Supergiovane]

Disable one configuration node at the time, one by one, and see what happens.

[Supergiovane]

You are still on node v18.
Update to at least 22.1.1
Use the PI installation script you find on the node-red website, with the option —node22

[evazzoler]

WTF!
Look at that!

They forget to update the help command missing the 22 version!
In the past I tried (manual install) the v.20 with bad results, and I tried that today, too!
I always reverted to the v.18 because the v.20 fixed nothing.
So, with your suggestion I looked into the source code of the script and discovered the command exists for the v.22, too...
Crazy!
We wrote tons of messages, spent hours, I bought a new 8GB raspi waiting it for 2 months, I uttered a lot of curses and received a ton of scolding from my family, for nothing.
And why that fucking script at the start says that it will manage node.js installation too? So you think that if there is a new version it will install it...
My gosh!
Now all works like a charm with 322 MB of total used RAM and node-red uses 1,2-4,6% of CPU.

There is a mistery in this story: when you asked to me wich version of sw I had (here above), @ 2 of February, I answered "Node: 22.11". It was totally wrong. Where did I get that bad info????????
This has fooled you and I publicly apologize for causing a waste of your time!

Thank you for your support!
(I am always in your debt!)

[Supergiovane]

No problems! Think you’ve learn something new. Anyway, i think you’ll be happy and future proof with the new PI.
;-)

[evazzoler]

It's unbelievable: I performed a deploy after re-adding a debug node into a flow that doesn't even use KNX, and suddenly everything stopped working again as before. At this point, all certainties have collapsed.

I've noticed that if I disable the retrieval of GA states at startup, telegrams get through, and almost everything works (though obviously, the system doesn't know the states and can't function properly), except for HUE—HUE doesn’t work at all. I've tried disabling the configuration nodes one by one, and I found out that the issue is specifically caused by the HUE configuration node. I've deleted and recreated it, reconnected the nodes to the new one, but after upgrading to the latest version, everything stops working again. I’ve attached a screenshot showing the states.

I have several HUE nodes responding to the same GA—could this be causing the issue?
I had to adopt this configuration because HUE groups never worked correctly (I had opened several issues about this) and also because otherwise some particular configurations wouldn't be possible, such as this one:
I have a hallway with two wall lights, each with one bulb pointing upwards and one downwards. During the day, I want all four bulbs at 100%, while at night, the downward-facing bulbs should be off, and the upward-facing ones at 10%. The only workaround to have them behave differently based on night/day was this. I sincerely hope this isn't the cause of the problem, otherwise, I won't be able to do this anymore with the updated knx-ultimate.
Thank you for the support!

[Supergiovane]

I have several HUE nodes responding to the same GA—could this be causing the issue?

No, if all nodes have the same Datapoint.

I've noticed that if I disable the retrieval of GA states at startup, telegrams get through, and almost everything works (though obviously, the system doesn't know the states and can't function properly), except for HUE—HUE doesn’t work at all

I need to see your flow about that

[Supergiovane]

Secondo me c’è un bubbone da qualche parte. Non è normale una cosa del genere.
Solitamente questi problemi nascono quando qualche nodo è registrato aia in modo globale, che in node_modules. Stessa cosa per node.js. Può darsi che venga usata dal sistema, una versione che in realtà non è quella che tu vedi con node -v, perché sei a livello utente.
Se non sei super esperto, ti consiglio di salvarti il progetto (tramite l’export dei flows di node-red), piallare tutto e ripartire con l’installazione di raspbian e poi eseguire lo script che ti avevo linkato. Successivamente, reimporti i tuoi flows tramite il menu import di node-red.
Qui ti darà un sacco si errori perché non troverai le palette che stai usando.
Pazientemente guardi quali mancano (te lo dice lui) le reinstalli una alla volta.

Oppure: mandami il tuo file di configurazione json (quello nella cartella .node-red) che vedo se ci sono caratteri gremlins (sono caratteri che danno fastidio al json del progetto)

[evazzoler]

Ok, proverò a reinstallare tutto. Ci vorrà un po': devo rivedere tutte le configurazioni del sistema operativo, annotarle, etc.
Di export ne ho un centinaio, faccio backup spesso di nodered, ma poi c'é mosquitto, le librerie di gestione del microcontrollore di stratopi, etc.
Mi serve una giornata di vacanza in cui le luci possono stare spente... ;-)
(o un secondo stratopi)

[Supergiovane]

Puoi temporaneamente avviare un’istanza di nodered su un PC di casa con le funzioni base, intanto che lo fai.
Occhio che anch’io avevo comprato all’inizio un “raspberry industriale” che aveva il suo watchdog ed un modulo real time clock. Senza alcuni drivers, mi dava problemi. Si freezzava ogni tanto. Alla fine l’ho buttato nel bidone dei ferri vecchi.

Tornando a te, una volta che hai il nuovo node red funzionante, ti basta importare il tuo progetto, poi andare nella cartella .node-red e da lì dare il comando “npm install”. Verranno installati tutti i nodi di cui il progetto ha bisogno.
Se vuoi mantenere anche tutte le password dei tuoi nodi, basta salvare da qualche parte il file .cred, che contiene tutte le credenziali.

[Supergiovane]

Poi ti do un consiglio da anziano: prendi anche un raspberry normale, con una scheda SD industriale (le trovi su amazon). Le SD industriali hanno un mtbf di decenni.
Su uno dei raspberry installi SOLO KNX es eventualmente anche altri nodi fatti da me ed il nodo mqtt (usa Aedes MQTT).
Sull’altro, installi tutto il resto.
In questo modo hai un “core” con le funzioni base sempre funzionante ed un altro raspberry con tutte le altre funzioni. Io ho fatto così.

[evazzoler]

Escluderei categoricamente che il problema sia dello stratopi, dal momento che ogni volta che rimetto knx-ultimate 3.1.9 il problema scompare, ed ha lavorato egregiamente per oltre un anno prima dell'introduzione del nuovo sistema di timing.

Ho già un node-red su una VM sul NAS (prima usavo Proxmox, poi ho sfoltito le macchine per risparmiare watt inutilmente sprecati) per le funzioni secondarie, ma quelle dove c'é knx-ultimate sono primarie. MQTT serve solo per ricevere i 4 dati fondamentali della stazione meteo, sulla base di irraggiamento e pioggia, faccio ciò che devo con i frangisole, lí deve restare per mantenere l'auto consistenza dell'oggetto, il suo carico è nullo: qualche sensore al minuto.
La casa è tutta esposta a sud, quindi questa funzione è fondamentale per la regolazione delle temperature, che altrimenti andrebbero fuori controllo (le variazioni date dai "vetri", dato l'isolamento e il dimensionamento essenziale delle macchine, sono molto importanti e molto più incisive del lavoro delle macchine stesse).

Detto ciò, posso fare delle prove su una VM o su un docker, ma comunque sarei su un x86, con CPU pressoché infinita e RAM pressoché infinita (anche se posso limitarla e abbiamo comunque visto che non viene usata). Preferisco risolvere il problema lì dov'é e dove dovrà restare node-red.

Non si spiega come knx-ultimate ultima versione abbia girato per un giorno senza problemi. E perché sia la richiesta degli stati della parte HUE a piantare tutto il modulo.

[Supergiovane]

Ultimamente ho modificato alcune cose su HUE, installa sempre l’ultima versione di knx ultimate.
Passami il tuo file di progetto zippato, che vedo se ci sono caratteri gremlins.

[Supergiovane]

Enrico, ricòrdati di mandarmi il tuo progetto, così provo a ricostruire il bubbone.

[evazzoler]

Ok, in questi giorni è un po' un casino perché sono via, fammi avere in privato un indirizzo a cui mandarlo... 😉

[Supergiovane]

maxsupergiovane chiocciola icloud.com