Skip to content

feat: lazy load pending deposit using generator #7584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: unstable
Choose a base branch
from
+95 −52
Draft

feat: lazy load pending deposit using generator #7584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6e55b7b
initial commit
ensi321 Mar 14, 2025
3fc83a1
Further simplify
ensi321 Mar 15, 2025
e399e0a
Lint
ensi321 Mar 15, 2025
4f1b244
use for loop
ensi321 Mar 15, 2025
8eb6b38
Fix type
ensi321 Mar 17, 2025
5d53280
Lint
ensi321 Mar 25, 2025
adacb09
lint
ensi321 Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 45 additions & 52 deletions packages/state-transition/src/epoch/processPendingDeposits.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {CachedBeaconStateElectra, EpochTransitionCache} from "../types.js";
import {increaseBalance} from "../util/balance.js";
import {hasCompoundingWithdrawalCredential, isValidatorKnown} from "../util/electra.js";
import {computeStartSlotAtEpoch} from "../util/epoch.js";
import {pendingDepositIterator} from "../util/generator.js";
import {getActivationExitChurnLimit} from "../util/validator.js";

/**
Expand All @@ -25,67 +26,59 @@ export function processPendingDeposits(state: CachedBeaconStateElectra, cache: E
let isChurnLimitReached = false;
const finalizedSlot = computeStartSlotAtEpoch(state.finalizedCheckpoint.epoch);

let startIndex = 0;
// TODO: is this a good number?
const chunk = 100;
const pendingDepositsLength = state.pendingDeposits.length;
outer: while (startIndex < pendingDepositsLength) {
const deposits = state.pendingDeposits.getReadonlyByRange(startIndex, chunk);
const depositIterator = pendingDepositIterator(state);

for (const deposit of deposits) {
// Do not process deposit requests if Eth1 bridge deposits are not yet applied.
if (
// Is deposit request
deposit.slot > GENESIS_SLOT &&
// There are pending Eth1 bridge deposits
state.eth1DepositIndex < state.depositRequestsStartIndex
) {
break outer;
}
for (const deposit of depositIterator) {
// Do not process deposit requests if Eth1 bridge deposits are not yet applied.
if (
// Is deposit request
deposit.slot > GENESIS_SLOT &&
// There are pending Eth1 bridge deposits
state.eth1DepositIndex < state.depositRequestsStartIndex
) {
break;
}

// Check if deposit has been finalized, otherwise, stop processing.
if (deposit.slot > finalizedSlot) {
break outer;
}
// Check if deposit has been finalized, otherwise, stop processing.
if (deposit.slot > finalizedSlot) {
break;
}

// Check if number of processed deposits has not reached the limit, otherwise, stop processing.
if (nextDepositIndex >= MAX_PENDING_DEPOSITS_PER_EPOCH) {
break outer;
}
// Check if number of processed deposits has not reached the limit, otherwise, stop processing.
if (nextDepositIndex >= MAX_PENDING_DEPOSITS_PER_EPOCH) {
break;
}

// Read validator state
let isValidatorExited = false;
let isValidatorWithdrawn = false;
// Read validator state
let isValidatorExited = false;
let isValidatorWithdrawn = false;

const validatorIndex = state.epochCtx.getValidatorIndex(deposit.pubkey);
if (isValidatorKnown(state, validatorIndex)) {
const validator = state.validators.getReadonly(validatorIndex);
isValidatorExited = validator.exitEpoch < FAR_FUTURE_EPOCH;
isValidatorWithdrawn = validator.withdrawableEpoch < nextEpoch;
}
const validatorIndex = state.epochCtx.getValidatorIndex(deposit.pubkey);
if (isValidatorKnown(state, validatorIndex)) {
const validator = state.validators.getReadonly(validatorIndex);
isValidatorExited = validator.exitEpoch < FAR_FUTURE_EPOCH;
isValidatorWithdrawn = validator.withdrawableEpoch < nextEpoch;
}

if (isValidatorWithdrawn) {
// Deposited balance will never become active. Increase balance but do not consume churn
applyPendingDeposit(state, deposit, cache);
} else if (isValidatorExited) {
// Validator is exiting, postpone the deposit until after withdrawable epoch
depositsToPostpone.push(deposit);
} else {
// Check if deposit fits in the churn, otherwise, do no more deposit processing in this epoch.
isChurnLimitReached = processedAmount + deposit.amount > availableForProcessing;
if (isChurnLimitReached) {
break outer;
}
// Consume churn and apply deposit.
processedAmount += deposit.amount;
applyPendingDeposit(state, deposit, cache);
if (isValidatorWithdrawn) {
// Deposited balance will never become active. Increase balance but do not consume churn
applyPendingDeposit(state, deposit, cache);
} else if (isValidatorExited) {
// Validator is exiting, postpone the deposit until after withdrawable epoch
depositsToPostpone.push(deposit);
} else {
// Check if deposit fits in the churn, otherwise, do no more deposit processing in this epoch.
isChurnLimitReached = processedAmount + deposit.amount > availableForProcessing;
if (isChurnLimitReached) {
break;
}

// Regardless of how the deposit was handled, we move on in the queue.
nextDepositIndex++;
// Consume churn and apply deposit.
processedAmount += deposit.amount;
applyPendingDeposit(state, deposit, cache);
}

startIndex += chunk;
// Regardless of how the deposit was handled, we move on in the queue.
nextDepositIndex++;
}

const remainingPendingDeposits = state.pendingDeposits.sliceFrom(nextDepositIndex);
Expand Down
50 changes: 50 additions & 0 deletions packages/state-transition/src/util/generator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import {CompositeViewDU, ListCompositeTreeViewDU} from "@chainsafe/ssz";
import {BeaconStateElectra} from "../types.js";

type LatestBeaconState = BeaconStateElectra;

type BeaconStateIterableKey = Extract<
keyof LatestBeaconState,
"pendingDeposits" | "pendingConsolidations" | "pendingPartialWithdrawals"
>;

type ElementType = LatestBeaconState[BeaconStateIterableKey] extends ListCompositeTreeViewDU<infer T> ? T : never;
type DepositType = LatestBeaconState[Extract<
BeaconStateIterableKey,
"pendingDeposits"
>] extends ListCompositeTreeViewDU<infer T>
? T
: never;

export function* pendingDepositIterator(
state: BeaconStateElectra,
startIndex?: number,
chunkSize?: number
): Generator<CompositeViewDU<DepositType>> {
for (const deposit of chunkedIterator(state.pendingDeposits, startIndex, chunkSize)) {
yield deposit as CompositeViewDU<DepositType>;
}
}
Comment on lines +19 to +27
Copy link

@graphite-app graphite-app bot Jul 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exported function pendingDepositIterator would benefit from JSDoc documentation that describes its purpose, parameters, and return value. This would maintain consistency with the documentation standards in the codebase and make the API more accessible to other developers. Consider adding a comment similar to the one provided for the chunkedIterator function below it.

GitHub: Fixes #1234

/**
 * Generator function that yields pending deposits from the beacon state
 * @param state - The beacon state containing pending deposits
 * @param startIndex - Optional starting index for iteration (default: 0)
 * @param chunkSize - Optional number of items to retrieve per chunk (default: 100)
 * @returns A generator that yields deposit objects
 */
export function* pendingDepositIterator(...)
Suggested change
export function* pendingDepositIterator(
state: BeaconStateElectra,
startIndex?: number,
chunkSize?: number
): Generator<CompositeViewDU<DepositType>> {
for (const deposit of chunkedIterator(state.pendingDeposits, startIndex, chunkSize)) {
yield deposit as CompositeViewDU<DepositType>;
}
}
/**
* Generator function that yields pending deposits from the beacon state
* @param state - The beacon state containing pending deposits
* @param startIndex - Optional starting index for iteration (default: 0)
* @param chunkSize - Optional number of items to retrieve per chunk (default: 100)
* @returns A generator that yields deposit objects
*/
export function* pendingDepositIterator(
state: BeaconStateElectra,
startIndex?: number,
chunkSize?: number
): Generator<CompositeViewDU<DepositType>> {
for (const deposit of chunkedIterator(state.pendingDeposits, startIndex, chunkSize)) {
yield deposit as CompositeViewDU<DepositType>;
}
}

Spotted by Diamond (based on custom rules)

Is this helpful? React 👍 or 👎 to let us know.


/**
* Generator function that abstracts away getReadonlyByRange. Should be a generator version of `getAllReadonly()` but lazy load by chunks
* @param iterable - An iterable property of the latest beacon state that has `getReadonlyByRange`
* @param chunkSize - Number of items to retrieve per iteration (default: 100)
* @param startIndex - Starting index for iteration (default: 0)
*/
function* chunkedIterator(
iterable: ListCompositeTreeViewDU<ElementType>,
startIndex = 0,
chunkSize = 100
): Generator<CompositeViewDU<ElementType>> {
const iterableLength = iterable.length;
let chunkStartIndex = startIndex;

while (chunkStartIndex < iterableLength) {
const currentChunk = iterable.getReadonlyByRange(chunkStartIndex, chunkSize);
for (const element of currentChunk) {
yield element;
}
chunkStartIndex += chunkSize;
}
}
Loading