Instability in 0.19.0-beta

[martinatime]

I upgraded to 0.19.0 last week and I have been fighting with it ever since. It will start up and everything seems to be fine and then at some point it decides that it is going to shut down some of the subsystems. If I restart LND then most times it comes back to active but a day or two later something triggers that causes the node to be unusable again. I'm not sure what all of the steps to troubleshoot are at this time but I have included what is a relevant section of the log.

Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.846 [INF] RPCS: New transaction subscription
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.847 [ERR] RPCS: [/lnrpc.Lightning/SubscribeChannelEvents]: subscription server shutting down
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.848 [INF] INVC: New invoice subscription client: id=257
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.848 [INF] INVC: Cancelling invoice subscription for client=257
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.854 [INF] INVC: New invoice subscription client: id=258
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.854 [INF] INVC: Cancelling invoice subscription for client=258
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.849 [ERR] RPCS: [/lnrpc.Lightning/SubscribeChannelBackups]: subscription server shutting down
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.867 [ERR] RPCS: [/routerrpc.Router/SubscribeHtlcEvents]: subscription server shutting down
Jun 10 07:37:59 [redacted] lnd[217664]: 2025-06-10 07:37:59.950 [INF] RPCS: Canceling transaction subscription

This is on a RaspberryPi 4 with 4GB RAM and a 2TB SSD running Debian Bullseye and Bitcoin Client v29.0

[martinatime]

So I did a bunch of research and I figured out that basically the health check was failing and therefore causing the LND client to restart. Apparently, this was happening multiple times. Now that I'm watching for it I see the health check fail but not for the 3 attempts that causes a restart.

I still don't know why the health check fails sometimes so I'm going to keep this open.

[martinatime]

I have now disabled that health check because it is calling Bitcoin Core's "getbestblockhash" which is sometimes taking a very long time

$ time bitcoin-cli getbestblockhash
0000000000000000000208fcdfe388b464d7a7ef9c05b2ad2701488f1eedbe70

real    10m21.332s
user    0m0.000s
sys     0m0.026s

This means the health check will fail at least 3 attempts and restart LND. I have no idea why this is happening, as I don't see anything in the Bitcoin Core log. It could be that in times where I have seen my node restart many times that it was hitting the same issue over and over again.

[guggero]

Do you have other software that connects to the bitcoin core in question? Perhaps you need to increase the value of -rpcthreads= to allow more connections at the same time.

[martinatime]

@guggero Sorry, I lost track of this discussion. I've been attempting to rectify this from the Bitcoin client side. bitcoin/bitcoin#32733

I don't think I was running out of rpc threads. When I shut down everything except for the Bitcoin daemon I could reproduce the timing issue. I think it is some sort of lock contention or other resource constraint that I don't think is expected. I will update the conversation if/when I figure out the root cause on the Bitcoin side. It may have to be taken into consideration in regards to LND's health heartbeat call to Bitcoin.