Too many channel manager writes in LDK?

[moneyball]

From LDK Discord, @TonyGiorgio:

"One thing I'm realizing now is how often channel manager saves. Like after a single reload it makes dozens of saves under a second apart.

just noticed all the network requests."

...

"channel manager writes by our account look insane atm.

I brought it up once before and it was sort of a "yeah they do that" kind of thing. Not sure why, but I've also chatted with other LDK implementers and that seems to be the consensus. I don't know if there's something wrong on our end, or LDK, or if there's an open issue or not.

But our DB just crapped itself because of it so we need to figure it out. Though it's possible it's not related to the high writes, it's just the only database that had a problem.

i also suspect that's why browser storage policies kick in and stop writing to disk sometimes too."

[TheBlueMatt]

I'd like to see some logs showing the persistence flow here. Indeed, sometimes the ChannelManager persistence may be aggressive, but its quite likely we can tweak it to be less aggressive in some of these cases.

[tnull]

FWIW, I recently logged the KVStore operations of reinitializing an LDK Node object and found it to be ~8 calls base (5 reads, 2 list, 1 write), plus at least ~5 calls per channel (4 reads, 1 write, or so). Note this doesn't include initial syncs etc. However, I'd also be curious where the "dozens of saves under a second apart" might come from.

[AnthonyRonning]

Need to circle back and add some logging so I can check accurately.

Though the "dozens" was incorrect, I hadn't realized at first when we reported it that it was actually half of what I thought because of all the OPTIONS HTTP calls. So more like a single dozen in writes when I load the wallet, including all the resync stuff. I might have a couple channels personally. Will log and test more soon.

[domZippilli]

FWIW, some data from us (c=) ... manager is the hottest key we have by far. Here's a screenshot of a live trace stream:

Note the 16:15:27 second has about 6 writes.

Statisically, we do about 1 write/second on the manager key. It adds up in terms of throughput, we write something like 3GB/hr just of this little guy, ~450KB at a time. It correlates with the amount of LDK events we process, which is our approximation of "traffic."

[TheBlueMatt]

Yea, its definitely hot during normal operation, though it should generally be, roughly what you describe, a write per event/action which happens (sometimes two if that action is a channel update commitment dance). I'm more curious here why we may be seeing 5+ updates during init.

[domZippilli]

Well, this is an aside, then: I do think 5-6 writes of one key per second can be a problem for some k/v stores that are replicated and better than eventually consistent. We already get punishing tail latency and some failed writes with manager and only manager, which I think is due to the key heat. I've considered either rate limiting this key and biting off the FC risk, or some other schemes to spread it across multiple keys. Perhaps table stakes for running with a lot of channels. But, I digress.

About init (I presume you mean starting an LDK node?). I happened to do one of those today, we pummeled that key with 18 writes per second at that point:

That histogram is showing this log line:

2023-08-23 17:32:59.057 TRACE [lightning_background_processor:645] Persisting ChannelManager...

I picked one to give some context:

We can probably get into the logs in more detail out of band, if that would be helpful.

[TheBlueMatt]

Well, this is an aside, then: I do think 5-6 writes of one key per second can be a problem for some k/v stores that are replicated and better than eventually consistent. We already get punishing tail latency and some failed writes with manager and only manager, which I think is due to the key heat.

Oh absolutely, its an important issue, but one that's solved through a larger conversation at #2518, and one which we at have a good understanding of, I think.

I've considered either rate limiting this key and biting off the FC risk, or some other schemes to spread it across multiple keys. Perhaps table stakes for running with a lot of channels. But, I digress.

With today's LDK, sadly, I think so. Absolutely something we need to fix, though.

We can probably get into the logs in more detail out of band, if that would be helpful.

Absolutely would be! While we should fix the fundamental issue in #2518, at least the above log snippet gives me one case where it looks like we're persisting a manager and don't need to be, so at least we can optimize until we do the bigger fix :).