3.0 Candidate: Support for Any Number of Descriptors

[thunderbiscuit]

Overview

This issue is the starting point for discussion on the idea of adding support for any number of keychains in the Wallet type in a future major/breaking release.

I have an PR (#226) digging into what such an API could look like, but it's really not the only way to do it and I invite others to also poke into this if they're interested.

Pros and Cons

✅ The wallet does away with the idea of main/change (external/internal) descriptors, which are fairly narrow-focused and less generally-applicable to all situations and requirements.
✅ Adding to the above: our wallet methods are currently sometimes not explicit enough, and produce unexpected results (for example calling Wallet::peek_address with KeychainKind::Internal will actually return an address on your External keychain if your wallet doesn't have an Internal keychain).
✅ Feature-parity for the Wallet type with application software in the industry that offers this feature. Examples: Blockstream Green, Sparrow Wallet, Blue Wallet, Samourai. All of those can provide a unified balance for the user over a number of descriptors/keychains.
❌ This potentially adds complexity to all wallet methods that deal with the keychains, because users need to handle the multiple keychains more explicitly. I'm not actually sure this is necessarily the case for all methods and under all API designs, just pointing it out here. More keychains = maybe more complexity at the call sites.
❌ Because the workflow of external/internal descriptors is embedded deep into the current Wallet API, the rewrite and review cost for shipping this is likely fairly high. The good thing is (at least so far in my exploration in #226), the reach of the changes is really limited to the wallet module. The separation with bdk_chain structures is really well done, and the inner logic is really built to handle this use case and so work out of the box really well (as far as I can tell). The real changes are mostly all at the Wallet and user-facing API level (not so much at the domain-logic level).

Some questions that still need answering

1. Who needs this? How would they use it?
2. New changesets and persistence. How problematic is that?

[evanlinjin]

The main concern I have is with the changeset design — this might be partly my oversight. I originally designed bdk_wallet::ChangeSet under the assumption that we wouldn't introduce flexible descriptor counts later on.

Some questions on my mind:

• For users who need this flexible descriptor count feature, do they require backward compatibility with the existing 2-descriptor-only Wallet? If not, maybe we can consider introducing another structure.
• Since we're introducing more flexibility, should we also consider use cases where the wallet isn't descriptor-based — e.g., silent payments?

[thunderbiscuit]

Yes the question of whether support for this feature is worth the Changeset type breaks (or break of the backwards-compatibility) across major releases is a good one, and/or whether we could/should look at introducing another structure (I need to understand more what you mean by that though, particularly around how much technical debt it introduces and the value to users of a clean break vs bolting new pieces on top of what we have). I personally think it sets us up nicely moving forward (including for things like your second point) so worth the time/effort, but I do think it'd be great to lay out the production use cases for this before going full steam ahead.

As for extra flexibility, I feel like that might be useful and I'm curious/optimistic we might be able to make this work with the idea of the wallet KeyRing laid out in #226 (a pre-Wallet structure that builds, holds, and controls the descriptors and keys and related data necessary for the Wallet to function). If so that would tie in well with the multi-descriptor wallet idea and in fact become indeed even more general.

[thunderbiscuit]

Keeping track of some of the thinking around this feature after more brainstorms with @ValuedMammal.

• The idea of a KeyRing that allows you to process and verify your keychains prior to actually building your wallet has value. This KeyRing can then be leveraged by the Wallet in all sorts of places. This new type has a few objectives, including thing like (a) making sure that the keychains it holds have certain properties (for example one of them could be declared the default keychain), (b) adding labels to the keychains, (c) helping to retrieve keychains when required by the user or Wallet, (d) ensuring the keychains don't contain private keys, (e) ensuring the consistency of the networks in all keychains in the KeyRing, (f) bundling all of this in a setup ready for consumption by the Wallet. Note that this moves a lot of that logic from the wallet constructors to the KeyRing, and therefore simplifies the wallet constructors (they know they will always get a valid KeyRing).
• It's probably simpler to do a clean break (this is my opinion, feel free to pitch in) on the wallet ChangeSet. If the new ChangeSet includes the KeyRing somehow, we could have utility methods like KeyRing::from_v2_changeset, which would build a fully valid KeyRing from a v1/v2 wallet ChangeSet that included fields like descriptor and change_descriptor. Similarly we can have utility methods to "downgrade" into a v1/v2 ChangeSet from a given KeyRing or Wallet (if they have <= 2 keychains)

[nymius]

* Since we're introducing more flexibility, should we also consider use cases where the wallet isn't descriptor-based — e.g., silent payments?

This may change in the future: discussion about silent payment descriptors at bitcoin core dev tech meeting

And I think encoding key semantics in descriptors is the right way to do it.

I like the idea of KeyRing, if we have silent payments descriptors in the future they will fit with this structure, but not with how Wallet works right now. But that shouldn't stop this change from happening IMHO.