Unusually high number of inactive channels after upgrade to LND v0.19.0-beta #9870
Replies: 15 comments 20 replies
-
|
This has happened to me with two channels since the update to 0.19.0, with one of them the problem has been solved when his node has been restarted and I suppose that with the second peer it will be solved as soon as he restarts his node. my log, message repeats every 3minutes peer log |
Beta Was this translation helpful? Give feedback.
-
|
My node has been on all 0.19.0-RCs and upgraded to release a few days ago without issue. The above mentioned issues only occurred after channel peers also upgraded to LND 0.19 – and those channels then remained as Adding my logs: Similar from my side as above This looks like the last time the channel connected before the downtime: |
Beta Was this translation helpful? Give feedback.
-
This channel stayed up for about a day, and at some point was offline again remained as I have now removed |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. The very surprising issue I saw on my end -- where two of my OWN nodes, in the SAME datacenter, showed channels between them as |
Beta Was this translation helpful? Give feedback.
-
Yes, exactly– That seems to be the case that simply updating to 0.19 does not cause the issue, but when a peer also updates, it can potentially cause this. However, imposrtant to note that it's not every channel either; some other peers have also updated to 0.19 since I have, and our channels have remained online and active. |
Beta Was this translation helpful? Give feedback.
-
|
I'm just focusing now on two computers, in the same datacenter, both running Checking just a few minutes ago, I'm finding the once again these 5 (five) channels are However the other node DOES show some interesting logs, which DO mention the partner node: |
Beta Was this translation helpful? Give feedback.
-
What do the logs of the other peer show? EOF here just means they hung up the connection while we were trying to read. In the other threads above, I see some connection timeouts, which means they didn't read anything for 5 minutes. The other instance with the "remote commitment dance" means that they never responded to our |
Beta Was this translation helpful? Give feedback.
-
|
One of the nodes, despite CMGR=debug & PEER=debug, showed no mention of the other node in the logs. I have now restarted both the underlying machine and the LND & other docker containers, and the channels came back online, which is the same behavior I have been seeing for days.... the channels operate well at first, but then after some number of hours the channels go This stuff is tricky to debug, so ..... I think we need some OTHER people to report similar problems (ideally in this thread) to really dig into this. If so far it's only myself and @Filouman reporting issues, it still could be something flaky with our networking or machines. |
Beta Was this translation helpful? Give feedback.
-
For these channels, are you able to find any sort of errors on the A channel won't ever go to active if we don't get the channel reest message for it. A channel may go from active to inactive, if we get some error at the link (channel) level, that may have us send an error/warning, but not disconnect. |
Beta Was this translation helpful? Give feedback.
-
|
OK, good, I will put HSWC=debug and then see if I can catch any useful logs when/if the channels go offline again. |
Beta Was this translation helpful? Give feedback.
-
|
@faket0shi are you aware of what node implementation your partner with the problem channel is running? It seems like you are running 0.19, but what about your partner node? |
Beta Was this translation helpful? Give feedback.
-
|
both run lnd 0.19.0, the weird thing is that I am using 0.19.0 since RC3 and the channel with @Filouman was always enabled until I upgraded to final version, at that point channel was disabled until he restarted his node. |
Beta Was this translation helpful? Give feedback.
-
|
Seeing something similar on my node, one node sees the channel as active the other one as inactive trying to pin point the bug. |
Beta Was this translation helpful? Give feedback.
-
|
If one side sends and receive the chan reest, but the other thinks they've only sent it, then you might see something like this. |
Beta Was this translation helpful? Give feedback.
-
|
Here is some history of one of the "problem" channels, as seen from one side: The channel is clearly going up and down a few times per day... and a similar channel with an 0.18.x node, in the same datacenter, doesn't show any intermittency like this. |
Beta Was this translation helpful? Give feedback.
-
|
A disabled channel is related, but a different state from an active channel. A channel goes to disabled (marked as such on channel update on the gossip layer), once a channel inactive for enough time. Once a channel is back active, it needs to be active for a duration before we enable it again. This is what determines if we mark a channel as active or not: Lines 669 to 704 in 3b2af90 What I think might be happening here is:
|
Beta Was this translation helpful? Give feedback.
-
|
For the channels that are going back up, then down again -- do you see anything in the logs re a read/write timeout? |
Beta Was this translation helpful? Give feedback.
-
|
@Roasbeef Please see log file attached, showings logs where the problem peer (which is my own node, in the same datacenter) is mentioned. I do see a lot of |
Beta Was this translation helpful? Give feedback.
-
|
Note that there are 5 parallell (redundant) channels between these two nodes, that's why you see |
Beta Was this translation helpful? Give feedback.
-
|
Looks like you still have the pong disconnection active? Which version is this node on? The v0.18.x series does this pong disconnection, but doesn't have a flag to disable it. The v0.19 implements more conservative practices when it comes to buffering incoming messages off the wire. Then Is see this with a Tor peer: If the peer doesn't send an From there I see the One thing to note is that v0.19 will start to throttle its own outbound gossip messages. By default this is set to 100 KB/s, this can be tuned with these new flags: Lines 1754 to 1763 in 3b2af90 |
Beta Was this translation helpful? Give feedback.
-
|
could you provide also the other node's log corresponding to the log file above ? |
Beta Was this translation helpful? Give feedback.
-
|
can you also set SRVR=debug, ACSM=debug log ? |
Beta Was this translation helpful? Give feedback.
-
|
@faket0shi @MegalithicBTC do you have the setting active ? |
Beta Was this translation helpful? Give feedback.
-
|
yes, it is active |
Beta Was this translation helpful? Give feedback.
-
|
ok I think your problem will be solved with this one: #9872 |
Beta Was this translation helpful? Give feedback.
-
|
@ziggie1984 I have not set I've now restarted both nodes with.. So this should capture sufficient logs that I could see useful logs from both sides next time this happens. thanks |
Beta Was this translation helpful? Give feedback.
-
|
One other thing in v0.19 is that by default, we'll now limit the number of non-channel peers we maintain. Similar to the Lines 569 to 570 in 3b2af90 This might explain some of the EOF's you're seeing, as during the encrypted transport handshake, we'll decide if we even want to accept the peer or not (after they send their pubkey). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks.... My reading of the code (which could be wrong) is that in the case that I have a long-established channel with a peer, then LND should never use the Line 118 in 93a6ab8 restricted slot ( here https://github.com/lightningnetwork/lnd/blob/93a6ab8759391ff70ba6a98196973d17df546667/accessman.go#L138C31-L138C49 ) -- am I wrong about this?
|
Beta Was this translation helpful? Give feedback.
-
|
So if we currently have, or have had a channel in the past with a peer, they'll go to protected. Protected is always allowed, restricted peers eat into that slot limit. You're right that this shouldn't affect peers with a long established channel with you. I mentioned that as some of the logs might be peers w/o any channels at all trying to connect, in which case we'll disconnect them. You can look at this log level on debug or higher for more information: |
Beta Was this translation helpful? Give feedback.
-
|
Yes. Thanks for that heads up. |
Beta Was this translation helpful? Give feedback.
-
|
This pretty interesting, my experience very closely matches @Filouman ...
exact same experience here
exact same experience, it was only when I had ANOTHER node ALSO upgraded to 0.19 that I saw the problem...
Exact same experience... the channel went offline again |
Beta Was this translation helpful? Give feedback.
-
|
Could you share some debug logs? Just DM'ed you on tg |
Beta Was this translation helpful? Give feedback.
-
|
Hi, best logs I could come up with are already in this discussion. |
Beta Was this translation helpful? Give feedback.
-
|
no debug logs? |
Beta Was this translation helpful? Give feedback.
-
|
We now have |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
In the last few days there have been some feedback from node-runners about v0.19.0-beta, where in this version some channels are showing
is_active: falsewhere in previous versions, the channel was reliably active.We have seen this with two of our nodes which we have updated to v.0.19-0-beta, but trying to diagnose it has been tricky.
After first restarting with the new version, we found that a channel between two of our own nodes was down, despite both nodes living in the same datacenter. We solved this by restarting one of the nodes, and now that channel is up.
There are however persistent issues we have been unable to solve.... A few channels with "clearnet" nodes, where we can see (for example on amboss.space) that their other channels are up, but ours are
is_active: false.We've already set
no-disconnect-on-pong-failure: true, inlnd.conf, but that didn't fix the issue.We're now looking for a way to try to get logs from our impacted node to try to understand why the channels have not come back online.
So far, this is what we've tried
Then...
After doing this, nothing of interest appears in LND's log -- even with these subsystems set to
debug, there is no mention of this peer's public key on the logs.So right now we don't have much beyond this anecdotal "it's not working".... But I wanted to open this discussion to see if anyone else could provide more useful evidence about what might or might not be going wrong.... or suggest certain commands we could run to help debug this.
Beta Was this translation helpful? Give feedback.
All reactions