[Pebble DB] Secrets database uses Pebble

[AlexHentschel]

At the moment, the secrets database is the last remaining place in flow-go that still uses badger. Context:

• For each epoch, consensus nodes engage in a DKG to generate a new random beacon key. This is necessary, because the composition of the consensus committee may change at each epoch.
• Each consensus nodes has a secrets database, where it persists its private random beacon keys.

Goal:
Secrets database uses Pebble

Challenges:

• The secrets database must be encrypted. Badger offers functionality to encrypt of the shelf, while pebble does not.

• While running the DKG, each consensus node internally maintains a simplified state machine (see RecoverablePrivateBeaconKeyStateMachine) to enforce consistent behaviour throughout the DKG despite potential crashes. For example, the node must know whether it ended up with a valid private random beacon key or not - we don't want the node to be ignorant, sign with an incorrect key (because it crashed during the DGK) and get slashed.

  The RecoverablePrivateBeaconKeyStateMachine) extensively uses badger's transaction model and snapshot isolation (atomicity of reads and writes). Therefore, migrating this business logic to pebble is challenging, requiring a diligent and methodical approach.