Robosats coordinator node disaster recovery

[TempleOfSats]

Hello,

I'm the node owner of Robosats coordinator Temple Of Sats.
I am experiencing a significant issues with my LND node. After a restart, the ZFS pool hosting the LND data appeared degraded, and my node experienced a large number of force closures.
I'm inexperienced with handling this kind of situation and would greatly appreciate your support.

Node Information: lncli getinfo shows the following, indicating it's unable to sync to the chain and graph:

lncli getinfo
{
  "version":  "0.18.3-beta commit=v0.18.3-beta",
  "commit_hash":  "d72a3aaf261e278fa4aad5be4453df2f74ab50ee",
  "identity_pubkey":  "030a425f5c69a29db30f6740d4e7df8f5612ef9955078ef4497490015464733dc8",
  "alias":  "🏛TempleOfSats🏛",
  "color":  "#ce0b2d",
  "num_pending_channels":  0,
  "num_active_channels":  0,
  "num_inactive_channels":  15,
  "num_peers":  0,
  "block_height":  866932,
  "block_hash":  "0000000000000000000061adba6746690d198a99355be1cb1e3f776f1265d1c2",
  "best_header_timestamp":  "1729651299",
  "synced_to_chain":  false,
  "synced_to_graph":  false,

Details:

Force Closures: Most of the force closures are INITIATOR_REMOTE and "chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator"...
I am not sure if this is correct behaviour but the also have local_txid being the same as closing_txid (not found in mempool) and remote_txid found in the mempool. For example:

{
  "channel": {
    "capacity": "7000000",
    "local_balance": "6923302",
    "remote_balance": "73227",
    "local_chan_reserve_sat": "70000",
    "remote_chan_reserve_sat": "70000",
    "initiator": "INITIATOR_REMOTE",
    "commitment_type": "ANCHORS",
    "num_forwarding_packages": "0",
    "chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
    "private": false,
    "memo": ""
  },
  "limbo_balance": "6923302",
  "commitments": {
    "local_txid": "3d4405dd3a0ce1462192e1035009bab913c93cc8eef8a3b7be5e7d5d6ed6ffe3",
    "remote_txid": "f2b4d3129a282812a162b6e2542ebb09fae4d232acecb9447ddd81d7a8eb93b9",
    "remote_pending_txid": "",
    "local_commit_fee_sat": "2810",
    "remote_commit_fee_sat": "2810",
    "remote_pending_commit_fee_sat": "0"
  },
  "closing_txid": "3d4405dd3a0ce1462192e1035009bab913c93cc8eef8a3b7be5e7d5d6ed6ffe3",
  "closing_tx_hex": ""
}

Upon restart, the LND node keeps trying to broadcast the local_txid but fails:

2024-10-23 01:54:25.605 [INF] CNCT: Re-publishing force close tx(3d4405dd3a0ce1462192e1035009bab913c93cc8eef8a3b7be5e7d5d6ed6ffe3) for channel eb63df27cccf9331634bbeac78f856cde4e62e92589f0a1c74a6e9c17fc8c1dd:1
2024-10-23 01:54:25.630 [INF] FNDG: Announcing ChannelPoint(f8281083de64610035e953f84a56c08340ab916286e1da77e757533b63026731:0), short_chan_id=866827:2687:0
2024-10-23 01:54:25.693 [ERR] BTCN: Broadcast attempt failed: <nil>
2024-10-23 01:54:25.694 [WRN] BTWL: Transaction 3d4405dd3a0ce1462192e1035009bab913c93cc8eef8a3b7be5e7d5d6ed6ffe3 not accepted by mempool: insufficient fee
2024-10-23 01:54:25.694 [WRN] CNCT: Unable to broadcast force close tx(3d4405dd3a0ce1462192e1035009bab913c93cc8eef8a3b7be5e7d5d6ed6ffe3): insufficient fee
2024-10-23 01:54:25.694 [INF] CNCT: Re-publishing force close tx(ab3d8fe4c04a443c6d4e4371cddeb95fe70947d4e824efd36bbaa952a44e7263) for channel 02508d7a1fc80da6ef357d826cd664fe54c4ebf180f196e967b41363d28ae2ac:0
2024-10-23 01:54:25.695 [ERR] BTCN: Broadcast attempt failed: <nil>

Node: The node in use it is still the original one. Initially I only had a couple of force closured, and the other channels were seen as active.
So I naively thought it was a limited issue and kept operating, in the span of 1 day most of the channels were force closed, with the exception of 15 channels that are now seen as inactive.

Questions:

1. Retrieving Funds: For my case what is the best way to proceed to cleanly retrieve the funds from the force-closed channels? Also In some cases, both local_txid and remote_txid are not found in the mempool.
2. Fee Bumping: Is it possible to bump the fees on the transactions pending?
3. Node: Should I shutdown the node for good, and use chantools instead, to force close the inactive channels?

Thank you so so so so much for your assistance.

node startup log: lnd.log
pending channels output: pendingchannels.txt

[yyforyongyu]

Could you do a lncli wallet pendingsweeps and paste the response here? I see some sweeping behaviors in the logs already, and think the sweeper will handle the funds. It's also a good idea to get familiar with the force close handling by reading this doc, which has instructions about how to do fee bumping.

indicating it's unable to sync to the chain and graph
I am not sure if this is correct behaviour but the also have local_txid being the same as closing_txid (not found in mempool) and remote_txid found in the mempool

I think once the closing tx is confirmed and the chain is synced this will get updated to be remote_txid. Some of the txids are not showing in the mempool due to the min relay fee not met. Could you tell us more about the restart process? How long did it take?

[yyforyongyu]

Is there a way to bump them?

There's some commands you can use to bump them. For instance, to trigger a sweep immediately, you could do
lncli wallet bumpfee --immediate 2eb3d8220ad3670d2abe556beb06cd5f6ce6116fe4d0b9703ec989929362baaf:2
To make it sweep faster, you could do,
lncli wallet bumpfee --immediate --conf_target=500 2eb3d8220ad3670d2abe556beb06cd5f6ce6116fe4d0b9703ec989929362baaf:2

Tho I'd suggest waiting for the sweeper to handle them automatically. Can you do a lncli debuglevel --level=info,SWPR=debug, let it run for one or two mintues, then post the logs here?

[TempleOfSats]

Fortunately I have good news, after fixing zmq block/ tx events the node synced and it seems like the force closures started to being broadcasted, I was also lucky that the mempool fees went down.

I attach pendingsweeps: pendingsweep.txt

This show committment_anchor and committment_to_remote_confirmed, I guess the latter are the outputs that will go to the peers?

and pendingchannels: pendingchannels.txt
This is the lnd logs with the level asked for:
lnd.log

Just one thing, isn't the normal behaviour of LND to stop starting up if it cannot connect to zmq block/ tx events? Could this possibly be a bug?

[ziggie1984]

hmm normally the zmq conncection should have been failed here:

lnd/chainreg/chainregistry.go

Lines 363 to 368 in e488002

	// Establish the connection to bitcoind and create the clients
	// required for our relevant subsystems.
	bitcoindConn, err := chain.NewBitcoindConn(bitcoindCfg)
	if err != nil {
	return nil, nil, err
	}

trigger a connection here:
https://github.com/btcsuite/btcwallet/blob/1a8e359f4f4d00a629ac64851db22ffda0acb915/chain/bitcoind_zmq_events.go#L120-L131

if it fails the chaincontrol should not be established.

hmm also for LND 18.3 this should be failing ....

[ziggie1984]

are you sure nothing was running on the ports: 127.0.0.1:18501, 127.0.0.1:18502 when you started the node, because lnd subsrbies to these services, maybe another node ?

[TempleOfSats]

lnd and bitcoind were running under the same docker container stack. However now I am only able to connect to 127.0.0.1:18501, 127.0.0.1:18502 when i bind bitcoin.conf to 0.0.0.0:18501, 0.0.0.0:18502, this must be an issue on my end, but it could it possibly have lead to an edge case where lnd startup did not stop?

[ziggie1984]

Do you have more logs available which you are willing to provide maybe via DM (slack, keybase), to figure out why the force-closes happened in the first place.

[TempleOfSats]

Thank you ziggie!

We can do slack, please!
-> streak-open-guise@duck.com