Remove stuck states channels

[heyambob]

Issue and Steps to Reproduce

Due to some bugs I have 2 peers with some bogus states of "AWAITING_UNILATERAL" and "CHANNELD_AWAITING_LOCKIN" which have been like this for months. I know that they can be removed because they were added with non-existing funding bitcoin tx id due to some bugs upon initiating the connection.

And one of them is causing trouble to me. This "AWAITING_UNILATERAL" state prevents my node to connect to him or another way around (saw it from the debug log). Take a look:

{
   "peers": [
      {
         "id": "03xxxx",
         "connected": false,
         "channels": [
            {
               "state": "AWAITING_UNILATERAL",
               "scratch_txid": "e596b3b3d47a68049490d3e2ac18c90bfcf3da11a85c689711961a0bc007a2c0",
               "last_tx_fee": "614000msat",
               "last_tx_fee_msat": "614000msat",
               "feerate": {
                  "perkw": 253,
                  "perkb": 1012
               },
               "channel_id": "5fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc",
               "funding_txid": "1d423ddaae344e4f24cd536af9cda7328b74e14b157308ad761ff49c4a218c85",
               "funding_outnum": 1,
               "initial_feerate": "253perkw",
               "last_feerate": "253perkw",
               "next_feerate": "256perkw",
               "inflight": [
                  {
                     "funding_txid": "xxxxx",
                     "funding_outnum": 1,
                     "feerate": "253perkw",
                     "total_funding_msat": "10000000000msat",
                     "our_funding_msat": "10000000000msat",
                     "scratch_txid": "e596b3b3d47a68049490d3e2ac18c90bfcf3da11a85c689711961a0bc007a2c0"
                  }
               ],
               "close_to_addr": "asdfasdf",
               "close_to": "234242342",
               "private": false,
               "opener": "local",
               "closer": null,
               "features": [
                  "option_static_remotekey",
                  "option_anchor_outputs"
               ],
               "funding_allocation_msat": {
                  "03xxx": 0,
                  "02yyy": 10000000000
               },
               "funding_msat": {
                  "03xxx": "0msat",
                  "02yyy": "10000000000msat"
               },
               "funding": {
                  "local_msat": "10000000000msat",
                  "remote_msat": "0msat",
                  "pushed_msat": "0msat"
               },
               "msatoshi_to_us": 10000000000,
               "to_us_msat": "10000000000msat",
               "msatoshi_to_us_min": 10000000000,
               "min_to_us_msat": "10000000000msat",
               "msatoshi_to_us_max": 10000000000,
               "max_to_us_msat": "10000000000msat",
               "msatoshi_total": 10000000000,
               "total_msat": "10000000000msat",
               "fee_base_msat": "0msat",
               "fee_proportional_millionths": 3000,
               "dust_limit_satoshis": 546,
               "dust_limit_msat": "546000msat",
               "max_htlc_value_in_flight_msat": 18446744073709551615,
               "max_total_htlc_in_msat": "18446744073709551615msat",
               "their_channel_reserve_satoshis": 100000,
               "their_reserve_msat": "100000000msat",
               "our_channel_reserve_satoshis": 100000,
               "our_reserve_msat": "100000000msat",
               "spendable_msatoshi": 4294967295,
               "spendable_msat": "4294967295msat",
               "receivable_msatoshi": 0,
               "receivable_msat": "0msat",
               "htlc_minimum_msat": 0,
               "minimum_htlc_in_msat": "0msat",
               "minimum_htlc_out_msat": "0msat",
               "maximum_htlc_out_msat": "9900000000msat",
               "their_to_self_delay": 144,
               "our_to_self_delay": 2016,
               "max_accepted_htlcs": 30,
               "state_changes": [
                  {
                     "timestamp": "2022-05-14T18:49:26.066Z",
                     "old_state": "DUALOPEND_OPEN_INIT",
                     "new_state": "AWAITING_UNILATERAL",
                     "cause": "protocol",
                     "message": "dualopend: received ERROR error channel 5fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc: Unknown channel for WIRE_CHANNEL_REESTABLISH"
                  }
               ],
               "status": [
                  "AWAITING_UNILATERAL:Will attempt reconnect in 205 seconds"
               ],
               "in_payments_offered": 0,
               "in_msatoshi_offered": 0,
               "in_offered_msat": "0msat",
               "in_payments_fulfilled": 0,
               "in_msatoshi_fulfilled": 0,
               "in_fulfilled_msat": "0msat",
               "out_payments_offered": 0,
               "out_msatoshi_offered": 0,
               "out_offered_msat": "0msat",
               "out_payments_fulfilled": 0,
               "out_msatoshi_fulfilled": 0,
               "out_fulfilled_msat": "0msat",
               "htlcs": []
            }
         ]
      }
   ]
}

and another one which was probably caused by initiating channels without valid tx. It's been sitting for a month already, I think, not sure why CLN hasn't removed it automatically.

{
   "peers": [
      {
         "id": "03cxxxxx",
         "connected": false,
         "channels": [
            {
               "state": "CHANNELD_AWAITING_LOCKIN",
               "scratch_txid": "0304bfd6be3aeb17720ee64524b6bbc38b524a5ff79260309474fd3d54160ee1",
               "last_tx_fee": "2720000msat",
               "last_tx_fee_msat": "2720000msat",
               "feerate": {
                  "perkw": 3758,
                  "perkb": 15032
               },
               "channel_id": "e01d0bc8a82ffd867a6ce262c97fb609b21512de1729287b9644225b5ee4b303",
               "funding_txid": "02b3e45e5b2244967b282917de1215b209b67fc962e26c7a86fd2fa8c80b1de0",
               "funding_outnum": 1,
               "close_to_addr": "bc1yyyy",
               "close_to": "2342342342",
               "private": true,
               "opener": "remote",
               "closer": null,
               "features": [
                  "option_static_remotekey"
               ],
               "funding_allocation_msat": {
                  "03cxxxxx": 16777215000,
                  "02yyyyy": 0
               },
               "funding_msat": {
                  "03cxxxxx": "16777215000msat",
                  "02yyyyy": "0msat"
               },
               "funding": {
                  "local_msat": "0msat",
                  "remote_msat": "16777215000msat",
                  "pushed_msat": "0msat"
               },
               "msatoshi_to_us": 0,
               "to_us_msat": "0msat",
               "msatoshi_to_us_min": 0,
               "min_to_us_msat": "0msat",
               "msatoshi_to_us_max": 0,
               "max_to_us_msat": "0msat",
               "msatoshi_total": 16777215000,
               "total_msat": "16777215000msat",
               "fee_base_msat": "0msat",
               "fee_proportional_millionths": 3000,
               "dust_limit_satoshis": 546,
               "dust_limit_msat": "546000msat",
               "max_htlc_value_in_flight_msat": 18446744073709551615,
               "max_total_htlc_in_msat": "18446744073709551615msat",
               "their_channel_reserve_satoshis": 167772,
               "their_reserve_msat": "167772000msat",
               "our_channel_reserve_satoshis": 167772,
               "our_reserve_msat": "167772000msat",
               "spendable_msatoshi": 0,
               "spendable_msat": "0msat",
               "receivable_msatoshi": 4294967295,
               "receivable_msat": "4294967295msat",
               "htlc_minimum_msat": 0,
               "minimum_htlc_in_msat": "0msat",
               "minimum_htlc_out_msat": "1msat",
               "maximum_htlc_out_msat": "16609443000msat",
               "their_to_self_delay": 144,
               "our_to_self_delay": 1008,
               "max_accepted_htlcs": 30,
               "state_changes": [],
               "status": [
                  "CHANNELD_AWAITING_LOCKIN:Will attempt reconnect in 300 seconds"
               ],
               "in_payments_offered": 0,
               "in_msatoshi_offered": 0,
               "in_offered_msat": "0msat",
               "in_payments_fulfilled": 0,
               "in_msatoshi_fulfilled": 0,
               "in_fulfilled_msat": "0msat",
               "out_payments_offered": 0,
               "out_msatoshi_offered": 0,
               "out_offered_msat": "0msat",
               "out_payments_fulfilled": 0,
               "out_msatoshi_fulfilled": 0,
               "out_fulfilled_msat": "0msat",
               "htlcs": []
            }
         ]
      }
   ]
}

Shouldn't CLN have a deadline to automatically clean up this kind of channels if they are hanging longer than say 3 weeks ?

or CLN should provide a way to remove bogus channels manually. I have investigated the db a bit, it doesn't look very straight forward at all.

getinfo output

They happened when I was around 0.10.x - 0.11.1, but now I'm currently at 0.11.2

[heyambob]

Relating to this: #5255

[grubles]

The second issue seems similar to #5366. Could you try getting the DEBUG logs for that particular node? Specifically what is mentioned here: #5366 (comment)

Can you describe in more detail what you're referring to here too?

I know that they can be removed because they were added with non-existing funding bitcoin tx id due to some bugs upon initiating the connection.

[heyambob]

my bad, CLN has just removed the 2nd stuck channel. It's been there for such a long time that I thought it's become permanent.

Only the first one is still on my db and the debug log on the peer is here: #5255 (comment)

Can you describe in more detail what you're referring to here too?

As you can see, this funding tx: 1d423ddaae344e4f24cd536af9cda7328b74e14b157308ad761ff49c4a218c85 doesn't exist, and the channel was attempted to create on "2022-05-14T18:49:26.066Z" . I'm not sure how it's in "AWAITING_UNILATERAL" state.

[cdecker]

I added a new FaQ entry regarding this situation here #5394

It describes in detail what's wrong, and potential resolutions. In your case you can check if the funding inputs are double-spent, and then use dev-forget-channel to, well, forget the channel. Notice that should the channel confirm anyway, then your funds will likely be lost. This is why we opt to rather keep things in the DB, and not forget them if they could become important later on.

[heyambob]

thank you. This solved my issue

[cdecker]

Glad it worked, thanks for the verifying the faq 👍

[CoderBoogs]

I added a new FaQ entry regarding this situation here #5394

It describes in detail what's wrong, and potential resolutions. In your case you can check if the funding inputs are double-spent, and then use dev-forget-channel to, well, forget the channel. Notice that should the channel confirm anyway, then your funds will likely be lost. This is why we opt to rather keep things in the DB, and not forget them if they could become important later on.

Question about this command. I have two channels that are stuck and they have the same peer_id when I try to run this command I get the following

"code":-1
"message":" Multiple Channels: please specify short_channel_id"

The problem is this id is not available. Both of the funding transactions are not in the mempool. Any idea on how I can clear these pending channels so I can open a new one?