LND: Payment failures when amount is over dust limit sats, on testnet

[MegalithicBTC]

I have a strange scenario, between two testnet nodes. My best guess is that it might be due to the current insanely high fees on Testnet (right now about 4200 sats/vbyte), but I'm really not sure.

Both nodes are running:

lncli version 0.18.3-beta commit=lightning-terminal-v0.13.4-alpha

I recently restarted both bitcoind (running testnet3), and also both instances of LND.

I have a channel between the two nodes.

Let's call the two nodes SENDER and RECEIVER.

Here is what the channel looks like on the SENDER node:

{
            "active": true,
            "remote_pubkey": "03a951e02ab549c2ec60d35b5c3bae842fdae176a68e0cfdc2a479063f2b0aa270",
            "channel_point": "551c12684f70a91cc92b94e75460b811215b7f543b85def79b2aa3b2dfd627fb:0",
            "chan_id": "3513908320554123264",
            "capacity": "14000000",
            "local_balance": "13508233",
            "remote_balance": "4428",
            "commit_fee": "486679",
            "commit_weight": "1116",
            "fee_per_kw": "432989",
            "unsettled_balance": "0",
            "total_satoshis_sent": "4428",
            "total_satoshis_received": "0",
            "num_updates": "41",
            "pending_htlcs": [],
            "csv_delay": 1682,
            "private": false,
            "initiator": true,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "140000",
            "remote_chan_reserve_sat": "140000",
            "static_remote_key": false,
            "commitment_type": "ANCHORS",
            "lifetime": "862",
            "uptime": "862",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 1682,
                "chan_reserve_sat": "140000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "13860000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 1682,
                "chan_reserve_sat": "140000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "13860000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [
                "17592186044416000036"
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "Rizful LSP Testnet B",
            "peer_scid_alias": "17592186044416000007",
            "memo": ""
        },

We can see that almost all the balance is on the SENDER side.

Now, let's do a keysend from SENDER to RECEIVER.

A 1 sat keysend works well...

user@aw60:/src$ ./lncli-testnet.sh sendpayment --dest 03a951e02ab549c2ec60d35b5c3bae842fdae176a68e0cfdc2a479063f2b0aa270 --keysend --amt 1   
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| SUCCEEDED  |        0.025 |        0.288 | 1            | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
Amount + fee:   1 + 0 sat
Payment hash:   448f90cca047d007df291b2d55f7f099054f41476fd01609744b2db6a59fde36
Payment status: SUCCEEDED, preimage: ca13bc05fa5c753697951a57fbbe383a6eb77f5ceaffc0e1e4d3e3850745ddea

... and a 100 sat keysend also works well...

user@aw60:/src$ ./lncli-testnet.sh sendpayment --dest 03a951e02ab549c2ec60d35b5c3bae842fdae176a68e0cfdc2a479063f2b0aa270 --keysend --amt 100
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| SUCCEEDED  |        0.024 |        0.277 | 100          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
Amount + fee:   100 + 0 sat
Payment hash:   b55560f482660e091b0b44ab69b0cd80462077608fd70ea98d2278870b13bd2c
Payment status: SUCCEEDED, preimage: abece89898b42fe053640e45aa4a8347eb896ad46595caa8ff0642810c43a567

Also a 200 sat payment works well....
Also a 250 sat payment works well...
... in fact, all payments work well, up until EXACTLY 354 sats... which is the dust limit!

When I send 354 sats or more....

Payment status: SUCCEEDED, preimage: 5337b05056c4ffdeafd1dc4fcf6a2b6ffb2aeb33d573ce076de862b575e00016
user@aw60:/src$ ./lncli-testnet.sh sendpayment --dest 03a951e02ab549c2ec60d35b5c3bae842fdae176a68e0cfdc2a479063f2b0aa270 --keysend --amt 354
+------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+------------+--------------+--------------+--------------+-----+----------+---------------------+----
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+-
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.075 |        0.116 | 354          | 0   |  3196065 | 
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.075 |        0.116 | 354          | 0   |  3196065 | 
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.075 |        0.116 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.075 |        0.116 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.075 |        0.116 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.133 |        0.175 | 354          | 0   |  3196065 | 
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.020 |        0.056 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.075 |        0.116 | 354          | 0   |  3196065 | 3513908320554123264 | Rizful LSP Testnet B |
| TEMPORARY_CHANNEL_FAILURE @ 0th hop |        0.133 |        0.175 | 354          | 0   |  3196065 | 
+-------------------------------------+--------------+--------------+--------------+-----+----------+---------------------+----------------------+
| HTLC_STATE                          | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT            | ROUTE                |

I am attaching the full logs for both SENDER and RECIEVER

lnd-log-RECEIVER.txt

lnd-log-SENDER.txt

[MegalithicBTC]

Sorry, to be clear, at the bottom of the previous message are the full LND logs for both SENDER and RECEIVER

[ziggie1984]

moreover can you fix the outgoing channel so it's using the one you posted

[MegalithicBTC]

I'll try increasing the dust threshold and also fix the outgoing channel ... then try this again and report back here.

[MegalithicBTC]

Yes! Setting both nodes to --dust-threshold=1000000000 fixed it, and I can now send payments. I guess for all of my testnet nodes now I need to set this value.....

[MegalithicBTC]

Incidentally, I guess that anyone running LND now on testnet3, unless for some reason they already set this value, is currently experiencing payment failures.

[ziggie1984]

we are trying to address this misunderstanding with: #9182

[MegalithicBTC]

Note, I tried this also with Bolt 11 invoices, and got the same result. So it's not just a Keysend problem.