[bug]: LND and public tracker discreprency

[WalterSmuts]

Background

The saga continues...

Seeing public trackers (here and here) report 5 open channels and not seeing my lnd node report the same. I'm missing around 7M sats after channel.db was lost and I had to do disaster recovery. The 5 open channels only appeared recently (I really think I would've noticed them being listed), and considering the capacity happens to amount to the correct size it seems suspicious.

I'm not seeing any open channels reported by LND:

[walter@fluffy ~]$ lncli listchannels
{
    "channels":  []
}

Seeing too little stats:

[walter@fluffy ~]$ lncli walletbalance
{
    "total_balance":  "6389436", // Should be closer to 14M sats
    "confirmed_balance":  "6389436",
    "unconfirmed_balance":  "0",
    "locked_balance":  "0",
    "reserved_balance_anchor_chan":  "0",
    "account_balance":  {
        "default":  {
            "confirmed_balance":  "6389436",
            "unconfirmed_balance":  "0"
        }
    }
}

Seeing some pending channels although they don't correspond:

[walter@fluffy ~]$ lncli pendingchannels
{
    "total_limbo_balance":  "0",
    "pending_open_channels":  [],
    "pending_closing_channels":  [],
    "pending_force_closing_channels":  [],
    "waiting_close_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "0286ff2f5fe0280ed16a37d69e98817a623b5a3254e98e89099f4314bc99626ab5",
                "channel_point":  "9dc7850ad3c35975149dcfb991878fecff9b82201c89f43877f26bece50a6a89:0",
                "capacity":  "20000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusRestored",
                "private":  true,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "02f5cdd5e831b50c3a3b9553c20a0cac0829fec8cc8b8411a35d450994d6887b29",
                "channel_point":  "e77bac4f5f781ee4fc043003e8bf5835500a73d4d8b020a813f9a215cb745d69:1",
                "capacity":  "250000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusRestored",
                "private":  true,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "0374d63a9db3ab228e9df0edb537e4812f58107998d10fc088b69569389b47c525",
                "channel_point":  "fd02fb80bf9986024f3c297cebd306c9c0e61f82b8361ac824b8be240b149e3c:0",
                "capacity":  "100000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "STATIC_REMOTE_KEY",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusRestored",
                "private":  true,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "",
            "closing_tx_hex":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "03dbc3bdb5cc07a806486de993c543adfe457530ccc272bc717b7bd94eddb48e99",
                "channel_point":  "4393ec199342dee6c8643e02e5f84c4f28ac2c9344c05023bce88543aec93a13:0",
                "capacity":  "25000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusRestored",
                "private":  true,
                "memo":  "",
                "custom_channel_data":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "",
            "closing_tx_hex":  ""
        }
    ]
}

And then also not seeing anything in the closed channels corresponding openly listed channels - e.g. largest one:

[walter@fluffy ~]$ lncli closedchannels | grep 877151893772828673  | wc
      0       0       0

Also nothing in logs(uploaded here):

[walter@fluffy ~]$ zgrep 877151893772828673 .lnd/logs/bitcoin/mainnet/*  | wc
      0       0       0

Your environment

[walter@fluffy ~]$ zgrep bitcoind .lnd/logs/bitcoin/mainnet/* | grep version | tail -n1
.lnd/logs/bitcoin/mainnet/lnd.log.6806.gz:2025-06-02 18:51:04.315 [DBG] RPCC: GetTxSpendingPrevOutAsync: backend version bitcoind v25.0.0 and above
[walter@fluffy ~]$ uname -a
Linux ip-172-31-0-55 6.6.8-1-ec2 #1 SMP PREEMPT_DYNAMIC Thu, 21 Dec 2023 13:05:15 +0000 x86_64 GNU/Linux
[walter@fluffy ~]$ lnd --version
lnd version 0.19.0-beta commit=v0.19.0-beta

Steps to reproduce

1. Setup LND as a router node using neutrino
2. Lose channel.db
3. Not seeing all stats reflect
4. Attempt various recovery methods
5. Wait ~3 years
6. Switch to bitcoind and new lnd versions
7. ???
8. Profit (or hopefully not a loss)

Expected behaviour

Open channels appear in lnd and sats appear in wallet

Actual behaviour

Missing moolah.

Things attempted:

1. Did a couple of restarts with --reset-wallet-transactions
2. Ran using lnd --reset-wallet-transactions a couple of times
   2.1. Using same neutrino backend as I routed with
   2.2. Using a bitcoind backend
3. Opened explicit connections with offending nodes
4. Solicited help previously here

[WalterSmuts]

logs.tar.gz

[guggero]

It looks like at least 3 of the channels in waiting_close_channels correspond to the 5 listed publicly.
If those nodes are still around, I'd try finding out their current IP address and connecting manually. That should initiate the force close. Alternatively you can try chantools triggerforceclose.

For the remaining 2 channels that aren't listed in your waiting_close_channels, those seem to be slightly newer channels. Is it possible they weren't contained in the channel.backup file you used?
You could also try chantools triggerforceclose on those.

[WalterSmuts]

So I did the following:

./chantools triggerforceclose --channel_point c34a46e8d7fd82170c9fc87cfcbc848c477a40799d638512bb60054aafd69cf3:22 --peer 0364913d18a19c671bb36dd04d6ad5be0fe8f2894314c36a9db3f03c2d414907e1@192.243.215.102:9735

And got new closing tx:
https://mempool.space/tx/8cd513cf668d53f3510f2f91ebb9d5228e4864d78cd73c6c2edd5e4d5035902f

After it confirmed I saw nothing reflect in my lnd wallet and cannot find it in the closed channels list:

[walter@fluffy chantools-linux-amd64-v0.13.7]$ lncli closedchannels | grep c34a46e8d7fd82170c9fc87cfcbc848c477a40799d638512bb60054aafd69cf3 | wc
      0       0       0

Is it possible they weren't contained in the channel.backup file you used?

Yes.

[guggero]

Because that channel wasn't tracked by lnd (likely because it wasn't in the channel.backup file you used) need to use chantools sweepremoteclosed as the log output should've told you: You can now use the sweepremoteclosed command to sweep the funds from the channel.

[WalterSmuts]

the log output should've told you

That teaches me for skimming the output 🤦

Hazzah! Looks like we've recovered another large chunk of sats! Thanks a bunch!

[walter@fluffy chantools-linux-amd64-v0.13.7]$ ./chantools sweepremoteclosed --sweepaddr fromseed --publish
2025-06-06 17:00:17.616 [INF] CHAN: chantools version v0.13.7 commit
Input your 24-word mnemonic separated by spaces: not for your eyes ;)
Input your cipher seed passphrase (press enter if your seed doesn't have a passphrase):
2025-06-06 17:01:35.218 [INF] CHAN: Found 1 unspent outputs for address bc1qe6h6qn5wp2ghwaq43sgzqk8z5f5cl7f8z4f0xyvayklwuj4czpashnurdr
2025-06-06 17:01:49.713 [INF] CHAN: Found 1 unspent outputs for address bc1qyy5dev5e7m8mdp4javcd3cgfgqhsrsx9zkxpkcluupfqvyfj47zqxl630x
2025-06-06 17:02:54.900 [INF] CHAN: Fee 4470 sats of 3805610 total amount (estimated weight 596)
2025-06-06 17:02:55.064 [INF] CHAN: Published TX b4b5bae24d6a092c7b988d232ed6e99c192e90e4c18acdefb2cc91d490daa94d, response: b4b5bae24d6a092c7b988d232ed6e99c192e90e4c18acdefb2cc91d490daa94d
2025-06-06 17:02:55.064 [INF] CHAN: Transaction: 0200000000010242c203b08f09ba32924c43a62d962920342ffb429de84daabbac3794738f07f80300000000010000003bdbb0968359cba33f978be2a00f212af706cce78d8d31845df05599a27fb7e30200000000010000000134003a00000000001600143e7902103b1ee1d836bb14d8b969e321dcd5cbb40247304402201c8c2cb1a56a0423972f600f177ec8c092a97fa60d2af1379f99c4151f922c6a02203672c62bba9516d2edf1fdf461e8f0b8eb67d5726e1771a832e6c17317f6ce8d01252102eeca310adb991cc7bdd6ce50d3cab942d85046cdd8ed3ffa01a6708aa237deb7ad51b202483045022100dc4b67767be05690d33d9534a61b98668d1399ee83ff14f284829bfe2353fe220220533d57c2b2554afda93ba521166a1d67649552e3a304742550c2d434ef35d4d10125210365c589ad1cdac01ddecef13555a55651ed10da92c3978292a71acb6e7a107717ad51b200000000
[walter@fluffy chantools-linux-amd64-v0.13.7]$ lncli walletbalance
{
    "total_balance":  "10190576",
    "confirmed_balance":  "6389436",
    "unconfirmed_balance":  "3801140",
    "locked_balance":  "0",
    "reserved_balance_anchor_chan":  "0",
    "account_balance":  {
        "default":  {
            "confirmed_balance":  "6389436",
            "unconfirmed_balance":  "3801140"
        }
    }
}

I see chantools found two unspent outputs. Although suspiciously neither was from the one we just force closed. Although take that statement with a grain of salt as this could just my my lack of bitcoin knowledge talking.

[WalterSmuts]

Mmmm... I attempted to send a force-close message to the other large open channel:

[walter@fluffy chantools-linux-amd64-v0.13.7]$ ./chantools triggerforceclose --channel_point 43d11884e78df667bdc3df61d347cf238344011e9aaa9d174bedef1e140dfbcc:1 --peer 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735 --torproxy localhost:9050
2025-06-06 18:29:54.266 [INF] CHAN: chantools version v0.13.7 commit
Input your 24-word mnemonic separated by spaces: Nope
Input your cipher seed passphrase (press enter if your seed doesn't have a passphrase):
2025-06-06 18:30:02.448 [DBG] CHAN: Attempting to resolve peer address 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735
2025-06-06 18:30:02.451 [DBG] CHAN: Attempting to dial resolved peer address 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735
2025-06-06 18:30:08.141 [INF] CHAN: Attempting to establish p2p connection to peer 03ef68ccd4b33ae540aea5bf91fcfc70137e031e0cf3823a958c3c3d69239eb7cd, dialtimeout is 1m0s
2025-06-06 18:30:08.172 [DBG] CHDB: Populating in-memory channel graph, this might take a while...
2025-06-06 18:30:08.175 [DBG] CHDB: Finished populating in-memory channel graph (took 1.084607ms, num_node_features=0, num_nodes=0, num_channels=0)
2025-06-06 18:30:08.176 [INF] CHDB: Checking for schema update: latest_version=31, db_version=31
2025-06-06 18:30:08.176 [INF] CHDB: Checking for optional update: prune_revocation_log=false, db_version=empty
2025-06-06 18:30:08.182 [DBG] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): Sending Init to 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735
2025-06-06 18:30:09.136 [DBG] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): Received Init from 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735
2025-06-06 18:30:09.138 [DBG] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): Loaded 0 active channels from database
2025-06-06 18:30:09.141 [INF] CHAN: Connection established to peer 03ef68ccd4b33ae540aea5bf91fcfc70137e031e0cf3823a958c3c3d69239eb7cd
2025-06-06 18:30:09.142 [INF] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): Negotiated chan series queries
2025-06-06 18:30:09.141 [INF] CHAN: Sending channel re-establish to peer to trigger force close of channel 43d11884e78df667bdc3df61d347cf238344011e9aaa9d174bedef1e140dfbcc:1
2025-06-06 18:30:09.142 [DBG] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): Sending ChannelReestablish(chan_id=ccfb0d141eefed4b179daa9a1e01448323cf47d361dfc3bd67f68de78418d142, next_local_height=0, remote_tail_height=0) to 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735
2025-06-06 18:30:09.145 [INF] CHAN: Sending channel error message to peer to trigger force close of channel 43d11884e78df667bdc3df61d347cf238344011e9aaa9d174bedef1e140dfbcc:1
2025-06-06 18:30:09.145 [DBG] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): Sending Error(type=17) to 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735
2025-06-06 18:30:09.146 [INF] CHAN: Message sent, waiting for force close transaction to appear in mempool
2025-06-06 18:30:10.504 [INF] CHAN: No spends found yet, waiting 5 seconds...

<SNIP> polling a bit here <\SNIP>

2025-06-06 18:33:05.825 [INF] CHAN: No spends found yet, waiting 5 seconds...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1410749]

goroutine 14 [running]:
github.com/lightningnetwork/lnd/peer.NewBrontide.func1()
	github.com/lightningnetwork/lnd@v0.18.4-beta/peer/brontide.go:645 +0x29
github.com/lightningnetwork/lnd/peer.(*PingManager).pingHandler(0xc000284310)
	github.com/lightningnetwork/lnd@v0.18.4-beta/peer/ping_manager.go:141 +0x1ca
created by github.com/lightningnetwork/lnd/peer.(*PingManager).Start.func1 in goroutine 1
	github.com/lightningnetwork/lnd@v0.18.4-beta/peer/ping_manager.go:105 +0xcc

[saubyk]

Moving this to discussions

[ellemouton]

@WalterSmuts - your last comment's panic trace shows that you potentially need to try updating your chantools version to the latest which points to lnd 0.19 (the panic output shows that the chantools version you're using uses lnd 0.18.4)

[WalterSmuts]

Chantools v0.13.7 is the newest release for chantools according to the releases page. Although I see what you mean WRT the lnd version chantools was compiled with. The latest unreleased version does point to latest lnd.

Will compile self and see what happes.

[WalterSmuts]

Still produces segfault:

...
2025-06-11 15:57:35.177 [INF] CHAN: No spends found yet, waiting 5 seconds...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x178f6e9]

goroutine 15 [running]:
github.com/lightningnetwork/lnd/peer.NewBrontide.func1()
	/home/walter/go/pkg/mod/github.com/lightningnetwork/lnd@v0.19.0-beta/peer/brontide.go:702 +0x29
github.com/lightningnetwork/lnd/peer.(*PingManager).pingHandler(0xc000214a80)
	/home/walter/go/pkg/mod/github.com/lightningnetwork/lnd@v0.19.0-beta/peer/ping_manager.go:175 +0x3bc
created by github.com/lightningnetwork/lnd/peer.(*PingManager).Start.func1 in goroutine 1
	/home/walter/go/pkg/mod/github.com/lightningnetwork/lnd@v0.19.0-beta/peer/ping_manager.go:107 +0xcc

[guggero]

Can you please try with this branch/PR? lightninglabs/chantools#194

[WalterSmuts]

Where we previously segfaulted we now report suspicously that the peer isn't responding with last response at 0s:

2025-06-11 17:42:22.586 [INF] CHAN: No spends found yet, waiting 5 seconds...
2025-06-11 17:42:22.955 [WRN] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): pong response failure for 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735: timeout while waiting for pong response. Time waited for this pong: 1m30.000118321s. Last successful RTT: 0s. -- disconnecting
2025-06-11 17:42:22.956 [INF] PEER: Peer(03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396): disconnecting 03e036c7da199f4dd1f2ea0eaedf2ba6142850553671cdc7d502ec382173ba9396@2hapmorx5prz2saedhwu52y7luay6sozbfrl3jz63fpalcy3qpgqfcyd.onion:9735, reason: pong failure: timeout while waiting for pong response
2025-06-11 17:42:27.740 [INF] CHAN: No spends found yet, waiting 5 seconds...

So perhaps the peer is not complying?

On another note I see this transaction we managed to close previously has two outputs - both of which has been spent... One of them contains a long chain of spends and the other only a single spend. Not too sure how I should interpret it? How do I verify if I own or should own an output there?

[guggero]

It's possible the peer also doesn't have the data for this channel anymore. In which case you'll need to attempt to contact the operator and run through the zombie recovery process.

The transaction you linked is an anchor channel output of a channel where only one party had funds. So it means either your node or their node didn't have any balance in the channel.