Draft: New EraE implementation #32157

shazam8253 · 2025-07-07T10:54:58Z

Here is a draft for the New EraE implementation. The code follows along with the spec listed at this link.

MariusVanDerWijden

lint is failing, you can check on your machine with make lint

internal/era2/accumulator.go

internal/era2/builder2.go

MariusVanDerWijden · 2025-07-07T10:56:55Z

internal/era2/era2.go

@@ -0,0 +1,429 @@
+package era2


lightclient

Left a bunch of comments here - we still need to figure out how to call this module since era2 isn't very elegant. In the meantime should rename builder2.go to just builder.go and era2.go to era.go.

internal/era2/builder2.go

lightclient · 2025-07-09T14:22:53Z

internal/era2/builder2.go

+	}
+}
+
+func (b *Builder) Add(header types.Header, body types.Body, receipts types.Receipts, blockhash common.Hash, blocknum uint64, td *big.Int, proof *Proof) error {


blockhash and blocknum are already available via header, so no need to duplicate them

lightclient · 2025-07-09T14:25:15Z

internal/era2/builder2.go

+	}
+}
+
+func (b *Builder) Add(header types.Header, body types.Body, receipts types.Receipts, blockhash common.Hash, blocknum uint64, td *big.Int, proof *Proof) error {


I would separate this into Add and AddRLP

lightclient · 2025-07-09T14:26:55Z

internal/era2/builder2.go

+	headersRLP  [][]byte
+	bodiesRLP   [][]byte
+	receiptsRLP [][]byte
+	proofsRLP   [][]byte


would remove RLP suffix since it is pretty explanatory by the type [][]byte

internal/era2/builder2.go

lightclient · 2025-07-09T15:35:42Z

internal/era2/era2.go

+	"github.com/klauspost/compress/snappy"
+)
+
+type meta struct {


Suggested change

type meta struct {

type metadata struct {

lightclient · 2025-07-09T15:35:55Z

internal/era2/era2.go

+type meta struct {
+	start     uint64 // start block number
+	count     uint64 // number of blocks in the era
+	compcount uint64 // number of properties


Suggested change

compcount uint64 // number of properties

components uint64 // number of properties

lightclient · 2025-07-09T15:36:04Z

internal/era2/era2.go

+	start     uint64 // start block number
+	count     uint64 // number of blocks in the era
+	compcount uint64 // number of properties
+	filelen   int64  // length of the file in bytes


Suggested change

filelen int64 // length of the file in bytes

length int64 // length of the file in bytes

lightclient · 2025-07-09T15:36:36Z

internal/era2/era2.go

+	mu                                                *sync.Mutex
+	headeroff, bodyoff, receiptsoff, tdoff, proofsoff []uint64 // offsets for each entry type
+	indstart                                          int64
+	rootheader                                        uint64 // offset of the root header in the file if present


what is the use of this?

when reading each era file I load in the index table into cache to make lookup faster, indstart is where the byte where the index table starts, and the root header is where the accumulator root is present when reading so it can seek there quickly since it is its own e2store object. The mutex should be removed though, forgot to do so very early on when I didn't understand what the file was doing I thought I wouldn't want it to read and write at the same time so put a lock.

lightclient · 2025-07-09T15:36:56Z

internal/era2/era2.go

+	m                                                 meta // metadata for the era2 file
+	mu                                                *sync.Mutex
+	headeroff, bodyoff, receiptsoff, tdoff, proofsoff []uint64 // offsets for each entry type
+	indstart                                          int64


what's this?

lightclient · 2025-07-09T15:39:52Z

Also, please write a description for your PRs and try to fix the CI errors so your code is all green.

Fixed all issues including extra logic regarding proof types, modularizing some functions and refactoring code for correctness and readability.

…ments

MariusVanDerWijden · 2025-07-14T18:20:08Z

internal/era2/era.go

+
+func (*BlockProofHistoricalSummariesDeneb) Variant() proofvar { return proofDeneb }
+
+func proofVariantOf(p Proof) proofvar {


Suggested change

func proofVariantOf(p Proof) proofvar {

func variantOf(p Proof) proofvar {

MariusVanDerWijden · 2025-07-14T18:21:25Z

internal/era2/builder.go

@@ -57,41 +57,33 @@ const (

 type proofvar uint16


Suggested change

type proofvar uint16

type variant uint16

We know that the variant is for Proofs, should be at most a comment, not part of the variable name, otherwise you end up with the types like mev-boost :D

gotcha will make the change :)

lightclient

Left a lot of style nits, but the main problems we'll need to work through are:

utilize proof interface more, avoid handling variant values directly
remove unneeded struct fields
avoid building entire era file in memory, write incremental work to disk

lightclient · 2025-07-17T18:31:36Z

internal/era2/builder.go

+	buff buffer
+	off  offsets
+
+	prooftype    variant


This should be removed. If you store the proofs as the interface object you will be able to determine the variant at anytime / alternatively if it makes better sense to store the bytes, then the bytes will already have included the variant type so it won't matter anymore.

lightclient · 2025-07-17T18:32:26Z

internal/era2/builder.go

+	off  offsets
+
+	prooftype    variant
+	tdsint       []*big.Int


This should be removed and tds in buffer should be big.Ints. Serialize them all at once during finalize.

lightclient · 2025-07-17T18:32:59Z

internal/era2/builder.go

+type Builder struct {
+	w   *e2store.Writer
+	buf *bytes.Buffer
+	sn  *snappy.Writer


Suggested change

sn *snappy.Writer

snappy *snappy.Writer

internal/era2/builder.go

lightclient · 2025-07-17T18:57:41Z

internal/era2/era.go

+}
+
+// retrieves the raw body frame in bytes of a specific block
+func (e *Era) GetRawBodyFrameByNumber(blockNum uint64) ([]byte, error) {


what do you mean by "frame" here?

It is the raw bytes that are written, without snappy decoding it and rlp decoding

I think you should decode the snappy here (and other methods like this) and return the RLP bytes

lightclient · 2025-07-17T18:58:04Z

internal/era2/era.go

+	return io.ReadAll(r)
+}
+
+// retrieves the raw receipts frame in bytes of a specific block


Suggested change

// retrieves the raw receipts frame in bytes of a specific block

// GetRawReceiptsFrameByNumber retrieves the raw receipts frame in bytes of a specific block.

lightclient · 2025-07-17T18:58:12Z

internal/era2/era.go

+	return io.ReadAll(r)
+}
+
+// retrieves the raw proof frame in bytes of a specific block proof


Suggested change

// retrieves the raw proof frame in bytes of a specific block proof

// GetRawProofFrameByNumber retrieves the raw proof frame in bytes of a specific block proof.

lightclient · 2025-07-17T18:58:31Z

internal/era2/era.go

+	return io.ReadAll(r)
+}
+
+// loads in the index table containing all offsets and caches it


Suggested change

// loads in the index table containing all offsets and caches it

// loadIndex loads in the index table containing all offsets and caches it.

lightclient · 2025-07-17T19:00:50Z

internal/era2/era.go

+// Getter methods to calculate offset of a specific component in the file.
+func (e *Era) headerOff(num uint64) (uint64, error) { return e.indexOffset(num, compHeader) }
+func (e *Era) bodyOff(num uint64) (uint64, error)   { return e.indexOffset(num, compBody) }
+func (e *Era) rcptOff(num uint64) (uint64, error)   { return e.indexOffset(num, compReceipts) }


Suggested change

func (e *Era) rcptOff(num uint64) (uint64, error) { return e.indexOffset(num, compReceipts) }

func (e *Era) receiptOff(num uint64) (uint64, error) { return e.indexOffset(num, compReceipts) }

Really no reason to ever abbreviate by taking out vowels in golang.

lightclient · 2025-07-22T21:22:07Z

cmd/utils/cmd.go

@@ -403,8 +410,8 @@ func ExportAppendChain(blockchain *core.BlockChain, fn string, first uint64, las

 // ExportHistory exports blockchain history into the specified directory,
 // following the Era format.
-func ExportHistory(bc *core.BlockChain, dir string, first, last, step uint64) error {
-	log.Info("Exporting blockchain history", "dir", dir)
+func ExportHistory(bc *core.BlockChain, dir string, first, last, step uint64, f ExportFormat) error {


I would not use an enum to dictate the output format, this should be done via the method naming, e.g.

ExportHistoryEra1(..)
ExportHistoryEraE(..)

lightclient · 2025-07-22T21:23:59Z

cmd/utils/cmd.go

+	if f == Era1 {
+		filename = era.Filename
+		newBuilder = func(w io.Writer) any { return era.NewBuilder(w) }
+		add = func(b any, blk *types.Block, rcpt types.Receipts, td *big.Int) error {
+			return b.(*era.Builder).Add(blk, rcpt, td)
+		}
+	} else {
+		filename = era2.Filename
+		newBuilder = func(w io.Writer) any { return era2.NewBuilder(w) }
+		add = func(b any, blk *types.Block, rcpt types.Receipts, td *big.Int) error {
+			return b.(*era2.Builder).Add(*blk.Header(), *blk.Body(), rcpt, td, nil)
+		}
+	}


This is kind of impressive, but also is what an Interface is for :)

If you want different builders with the same methods (like Add) you can create the interface type and implement the method for both.

lightclient · 2025-07-22T21:24:26Z

cmd/utils/cmd.go

-				receipts := bc.GetReceiptsByHash(block.Hash())
-				if receipts == nil {
-					return fmt.Errorf("export failed on #%d: receipts not found", n)
+				rcpt := bc.GetReceiptsByHash(blk.Hash())


Suggested change

rcpt := bc.GetReceiptsByHash(blk.Hash())

receipts := bc.GetReceiptsByHash(blk.Hash())

lightclient · 2025-07-22T21:24:32Z

cmd/utils/cmd.go

+
+			for j := uint64(0); j < step && batch+j <= last; j++ {
+				n := batch + j
+				blk := bc.GetBlockByNumber(n)


Suggested change

blk := bc.GetBlockByNumber(n)

block := bc.GetBlockByNumber(n)

lightclient · 2025-07-22T21:24:38Z

cmd/utils/cmd.go

-				)
-				if block == nil {
-					return fmt.Errorf("export failed on #%d: not found", n)
+			bldr := newBuilder(f)


Suggested change

bldr := newBuilder(f)

builder := newBuilder(f)

lightclient · 2025-07-22T21:28:56Z

internal/era2/builder.go

+	tds      []*big.Int
+}
+
+// The offsets holds the offsets of the different block components in the e2store file. Eventually these offsets will be used to write the index table at the end of the file.


Suggested change

// The offsets holds the offsets of the different block components in the e2store file. Eventually these offsets will be used to write the index table at the end of the file.

// offsets holds the offsets of the different block components in the e2store file. Eventually these offsets will be used to write the index table at the end of the file.

lightclient · 2025-07-22T21:29:40Z

internal/era2/builder.go

+	buf *bytes.Buffer
+
+	buff buffer


we still have buf and buff, I think we discussed removing buf since it isn't used until the end?

lightclient · 2025-07-22T21:32:36Z

internal/era2/builder.go

+// Add writes a block entry, its reciepts, and optionally its proofs as well into the e2store file.
+func (b *Builder) Add(header types.Header, body types.Body, receipts types.Receipts, td *big.Int, proof Proof) error {
+	if len(b.buff.headers) == 0 { // first block determines wether proofs are expected
+		b.expectsProofs = proof != nil


i don't think you need to track this explicitly, just check if b.buff.proofs != nil and if proof == nil or vice versa. only special case is first block, which you allow for the b.buff.proofs to be nil even if proof is non-nil.

lightclient · 2025-07-22T21:35:11Z

internal/era2/builder.go

+		if err != nil {
+			return common.Hash{}, fmt.Errorf("compute accumulator: %w", err)
+		}
+		if n, err := b.w.Write(TypeAccumulatorRoot, accRoot[:]); err != nil {


looks like this still needs to be addressed

lightclient · 2025-07-22T21:36:33Z

internal/era2/era.go

+}
+
+// retrieves the raw body frame in bytes of a specific block
+func (e *Era) GetRawBodyFrameByNumber(blockNum uint64) ([]byte, error) {


I think you should decode the snappy here (and other methods like this) and return the RLP bytes

lightclient · 2025-07-24T19:11:35Z

cmd/utils/era_interface.go

+type Iterator interface {
+	Next() bool
+	Number() uint64
+	Block() (*types.Block, error)
+	Receipts() (types.Receipts, error)
+	Error() error
+}


This interface looks right, but it's in the wrong place. We should define it in the era package. I will take a stab at this.

lightclient · 2025-07-24T19:12:27Z

cmd/utils/era_interface.go

+func (era1Format) Filename(n string, e int, h common.Hash) string { return era.Filename(n, e, h) }
+func (era1Format) NewBuilder(w io.Writer) Builder                 { return &era1Builder{era.NewBuilder(w)} }
+func (era1Format) ReadDir(dir, net string) ([]string, error)      { return era.ReadDir(dir, net) }
+func (era1Format) NewIterator(f *os.File) (Iterator, error) {
+	e, err := era.From(f)
+	if err != nil {
+		return nil, err
+	}
+	return era.NewIterator(e)
+}


you shouldn't need to redefine these methods to satisfy the interface - if the type implements the interface methods then it can be accepted anywhere the interface is accepted

lightclient · 2025-07-24T19:53:11Z

cmd/utils/cmd.go

@@ -248,11 +250,11 @@ func readList(filename string) ([]string, error) {
 // ImportHistory imports Era1 files containing historical block information,
 // starting from genesis. The assumption is held that the provided chain
 // segment in Era1 file should all be canonical and verified.
-func ImportHistory(chain *core.BlockChain, dir string, network string) error {
+func ImportHistory(chain *core.BlockChain, dir string, network string, format Format) error {


So on this method and ExportHistory we can avoid using this Format type by just passing in the functions we need. For example, here we need a way to read all the entries and a way to create an iterator from an open file. If we just pass those two functions in, we can create an if statement in the cli handling so that we can reuse the function.

The issue with format is it is kind of a superfluous type.

shantichanal added 5 commits June 26, 2025 14:47

Working on implementation

04baf07

updated some things made a section writer

f44aa8a

finished builder

34640ad

readers for single reads

db4d4dc

sequential access completed (without iterator)

ba32a61

MariusVanDerWijden reviewed Jul 7, 2025

View reviewed changes

shantichanal added 3 commits July 7, 2025 15:21

simplified proof builder structure

6b876c7

adding testing and refining functions

2865b06

refactored and updated e2store framing for objects

a0fffe1

lightclient reviewed Jul 9, 2025

View reviewed changes

Updated with all comments.

6618a4e

Fixed all issues including extra logic regarding proof types, modularizing some functions and refactoring code for correctness and readability.

lightclient self-assigned this Jul 10, 2025

Implemented the proof interface, refactored code, and implemented com…

19ae280

…ments

MariusVanDerWijden reviewed Jul 14, 2025

View reviewed changes

shantichanal and others added 2 commits July 15, 2025 13:46

formatting changes and lint

7837839

internal/era2: add correct license headers

ad938e6

lightclient reviewed Jul 17, 2025

View reviewed changes

shantichanal added 5 commits July 17, 2025 23:06

added cmd

f3f12df

small update

61103ef

fix

bdc2987

added all changes from comments

4d538b3

testing and made code more readable

5a40e87

lightclient reviewed Jul 22, 2025

View reviewed changes

shantichanal added 5 commits July 23, 2025 02:26

built cmd

0b08fe8

added some abstraction

7cffbdf

small changes to test for era export

751c728

small fix

0669bb1

trying another small fix

430a6fe

lightclient reviewed Jul 24, 2025

View reviewed changes

all: refactor era1 and era2 into era subpackages of onedb and execdb

02d498b

lightclient requested a review from rjl493456442 as a code owner July 24, 2025 20:01

lightclient and others added 2 commits July 24, 2025 14:03

internal/era: rename era2 to execdb package

091211d

all cmd changes to pass in functions in chaincmd

acd77a9

	compcount uint64 // number of properties
	components uint64 // number of properties

	filelen int64 // length of the file in bytes
	length int64 // length of the file in bytes


		func (*BlockProofHistoricalSummariesDeneb) Variant() proofvar { return proofDeneb }

		func proofVariantOf(p Proof) proofvar {

	func proofVariantOf(p Proof) proofvar {
	func variantOf(p Proof) proofvar {

	// retrieves the raw receipts frame in bytes of a specific block
	// GetRawReceiptsFrameByNumber retrieves the raw receipts frame in bytes of a specific block.

	// retrieves the raw proof frame in bytes of a specific block proof
	// GetRawProofFrameByNumber retrieves the raw proof frame in bytes of a specific block proof.

	// loads in the index table containing all offsets and caches it
	// loadIndex loads in the index table containing all offsets and caches it.

	func (e *Era) rcptOff(num uint64) (uint64, error) { return e.indexOffset(num, compReceipts) }
	func (e *Era) receiptOff(num uint64) (uint64, error) { return e.indexOffset(num, compReceipts) }

	rcpt := bc.GetReceiptsByHash(blk.Hash())
	receipts := bc.GetReceiptsByHash(blk.Hash())

	blk := bc.GetBlockByNumber(n)
	block := bc.GetBlockByNumber(n)

	// The offsets holds the offsets of the different block components in the e2store file. Eventually these offsets will be used to write the index table at the end of the file.
	// offsets holds the offsets of the different block components in the e2store file. Eventually these offsets will be used to write the index table at the end of the file.

Draft: New EraE implementation #32157

Are you sure you want to change the base?

Draft: New EraE implementation #32157

Conversation

shazam8253 commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MariusVanDerWijden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightclient left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightclient commented Jul 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightclient left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

shazam8253 commented Jul 7, 2025 •

edited

Loading