Slim down PersistedForkChoice

[dapplion]

The fork-choice state is persistsed often and it's big, causing a lot of I/O. Currently, its size also grows during non-finality.

Some ideas:

• Don't persist fork-choice nodes, instead re-compute them on start-up from stored blocks. We would need to compute the cost of doing so in long non-finality. Probably we need to store the payload status of them, or maybe start them all as syncing
• Just persist the last votes once in a while. Persisting just votes should be smaller and constant size. We can experiment with how often we need to persist those votes too, or only persisting them on a clean shut-down. Worst case we will get new votes on the epoch after start-up. We can also run a compression algorithm on the votes as most of them have the same roots.

@michaelsproul

[dapplion]

A mainnet node without subscribing all subnets had a PersistedForkChoice entry in the DB of 242 MB. Looking at the type, assuming 1.5M total indices 1M active let's try to account for those 242 MB.

pub struct PersistedForkChoice {
    pub fork_choice: fork_choice::PersistedForkChoice, // 242 - 60 = 166 MB
    pub fork_choice_store: PersistedForkChoiceStoreV17, // ~60 MB
}

pub struct PersistedForkChoice {
    // ProtoArrayForkChoice.as_bytes()
    pub proto_array_bytes: Vec<u8>,
    // slot worth of attestations 0.25 MB
    queued_attestations: Vec<QueuedAttestation>, // <== Double check, could be more depending on aggregation
}

// Written as ProtoArraySszContainer
pub struct ProtoArrayForkChoice {
    pub(crate) proto_array: ProtoArray,
    pub(crate) votes: ElasticList<VoteTracker>,
    pub(crate) balances: JustifiedBalances,
}

// Actual name in the codebase is `SszContainer`
pub struct ProtoArraySszContainer {
    // 108 MB ?? Duplicated list of votes
    pub votes: Vec<VoteTracker>,
    // 12 MB
    pub balances: Vec<u64>,
    pub prune_threshold: usize,
    pub justified_checkpoint: Checkpoint,
    pub finalized_checkpoint: Checkpoint,
    #[superstruct(only(V17))]
    // In 1 month of blocks 98 MB, 456 bytes / item
    pub nodes: Vec<ProtoNodeV17>,  // <== Double check, should be small but maybe not if pruning is broken?
    // In 1 month of blocks 8.6 MB, 40 bytes / item
    pub indices: Vec<(Hash256, usize)>,
    pub previous_proposer_boost: ProposerBoost,
}

pub struct PersistedForkChoiceStore {
    pub balances_cache: BalancesCacheV8, // ~48 MB <== Double check if this has actually 4 entries
    pub time: Slot,
    pub finalized_checkpoint: Checkpoint,
    pub justified_checkpoint: Checkpoint,
    // 12 MB
    pub justified_balances: Vec<u64>,
    pub unrealized_justified_checkpoint: Checkpoint,
    pub unrealized_finalized_checkpoint: Checkpoint,
    pub proposer_boost_root: Hash256,
    pub equivocating_indices: BTreeSet<u64>, // <== Double check, should be zero but maybe not?
}

pub struct BalancesCache {
    pub(crate) items: Vec<CacheItemV8>,
}

// MAX_BALANCE_CACHE_SIZE = 4
pub(crate) struct CacheItem {
    pub(crate) block_root: Hash256,
    pub(crate) epoch: Epoch,
    pub(crate) balances: Vec<u64>,
}

we have unaccounted ~46 MB look at the "Double check" comments