Prysm panics when --enable-light-client is set #15657
Description
Describe the bug
Hello Developers,
I am currently conducting API testing on an Electra PoS devnet chain and have observed inconsistencies in the Beacon-API responses across different CL clients: Prysm, Lighthouse, Nimbus, Lodestar, and Grandine.
During testing of the light-client related APIs, I encountered an issue where, upon starting Prysm with the --enable-light-client flag, Prysm panics and disconnects from the EL client. The error message indicates a panic caused by an "invalid memory address or nil pointer dereference." If the flag is removed, Prysm works fine.
2025-09-05 17:08:20 [2025-09-05 09:08:20.12] ERROR sync: Panic occurred error=runtime error: invalid memory address or nil pointer dereference recoveredAt=subscribeWithBase stack=goroutine 52076 [running]:runtime/debug.Stack()GOROOT/src/runtime/debug/stack.go:26 +0x5egithub.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.func1.1()beacon-chain/sync/subscriber.go:294 +0x24fpanic({0x1261400?, 0x4652c10?})GOROOT/src/runtime/panic.go:792 +0x132github.com/OffchainLabs/prysm/v6/beacon-chain/core/light-client.(*Store).LastOptimisticUpdate(0x1c1adc0?)beacon-chain/core/light-client/store.go:199 +0x29github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).processLightClientOptimisticUpdate(0xc0003f8140, {0x1c02780, 0xc002d9a850}, {0x1c1acb0, 0xc002538900}, {0x1c39e60, 0xc00291b348})beacon-chain/blockchain/process_block_helpers.go:256 +0x365github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).processLightClientUpdates(0xc0003f8140, 0xc001eac000)beacon-chain/blockchain/process_block_helpers.go:140 +0x1dbgithub.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).postBlockProcess(0xc0003f8140, 0xc001eac000)beacon-chain/blockchain/process_block.go:115 +0xc25github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).ReceiveBlock(0xc0003f8140, {0x1c02780?, 0xc002d9a850?}, {0x1c1adc0, 0xc002d9aa10}, {0xa2, 0xbe, 0x1f, 0xee, 0x7e, ...}, ...)beacon-chain/blockchain/receive_block.go:135 +0xba8github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).beaconBlockSubscriber(0xc000174848, {0x1c02780, 0xc002d9a850}, {0x1be7540?, 0xc002d8ae10?})beacon-chain/sync/subscriber_beacon_blocks.go:42 +0x2f4github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.func1(0xc003230780)beacon-chain/sync/subscriber.go:307 +0x3f1created by github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.func2 in goroutine 48965beacon-chain/sync/subscriber.go:334 +0xf9 topic=/eth2/d09a044a/beacon_block/ssz_snappy
2025-09-05 17:08:28 [2025-09-05 09:08:28.01] ERROR message-handler: Panicked when handling p2p message! Recovering... msg=data:"\350\007L\254\000\000\000\000\001\203\202\311H|\316\000\004\014\n&#\3619\001\023.\001\000\0003>!\000\360q\003\0200(\230\214\304\347\014@\300\240`2\022\245\355AZ\366\344\277\303#\377\337\205\201H\001\273\367\007\021\220\2037_\026\223y \035\232\3205z\372B\037hR/_6\212\336`\276\005\256\267\357\006T`\243\254\243\030e\301L\001\334,\307\345E\2138\324\204\225\242I&\2455\002$-\262\350t\300O\317\360T\260)}\207\022\202\236Yf\2263\001\000\000.\217\000\242\001\000\360\311(\346\370\277\2367\345Fv\270\321\000\336\277|\0205xP\333$\276\304k#\237\220(B\317\252\337\312O\230\211\013\311\212Y\360\025\320cu\245\340\005F\270\362\254\036\210\323\033\027t\352(\324\263\347\321\364\000\000\000B\260RT\035\316EU}\203\323F4\244ZV\322\026\3247^Z\225\204\366D\\\344\3463$\257\264o\014\001\200_\342\022\341Y\007\230\033u~lIk\014\260fd\"FUa=\316\310%\005\273m\323\271\225]\211-\2223\213\031\227o\320p\204\277\350\212v\303\0064\202\267\363\016\346\017\353*X\342\343\323WO\301\333\375\007\3045\235\0310\305\205W\024\3308\037\254\351y7Z\364--\371\013\267\000\000\000\000\000\000\242\363\000\376\001\000\376\001\000\376\001\000\376\001\000\376\001\000\376\001\000\005\001\004H\002\356\207\001)\207\360O\177\376$\036\246\001\207\375\260\030{\372\"\3365\321\371\276\327\253\006\035\224\001\375G\343JT\373\355\341y)0\273\325\272\254C\274\307\230\356I\252\201\205\357v\273;D\272b\271\035\206\256V\236K\2655\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" topic:"/eth2/d09a044a/light_client_optimistic_update/ssz_snappy" r=runtime error: invalid memory address or nil pointer dereference
2025-09-05 17:08:28 goroutine 52960 [running]:
2025-09-05 17:08:28 runtime/debug.Stack()
2025-09-05 17:08:28 GOROOT/src/runtime/debug/stack.go:26 +0x5e
2025-09-05 17:08:28 runtime/debug.PrintStack()
2025-09-05 17:08:28 GOROOT/src/runtime/debug/stack.go:18 +0x13
2025-09-05 17:08:28 github.com/OffchainLabs/prysm/v6/runtime/messagehandler.HandlePanic({0x1c02710, 0xc000193a40}, 0xc00228acc0)
2025-09-05 17:08:28 runtime/messagehandler/messagehandler.go:47 +0x205
2025-09-05 17:08:28 panic({0x1261400?, 0x4652c10?})
2025-09-05 17:08:28 GOROOT/src/runtime/panic.go:792 +0x132
2025-09-05 17:08:28 github.com/OffchainLabs/prysm/v6/beacon-chain/core/light-client.(*Store).LastOptimisticUpdate(0xc226c803009af76f?)
2025-09-05 17:08:28 beacon-chain/core/light-client/store.go:199 +0x29
2025-09-05 17:08:28 github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).validateLightClientOptimisticUpdate(0xc000174848, {0x1c02780, 0xc0029e0d90}, {0xc0008446f0, 0x27}, 0xc00228acc0)
2025-09-05 17:08:28 beacon-chain/sync/validate_light_client.go:67 +0x5b1
2025-09-05 17:08:28 github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.(*Service).wrapAndReportValidation.func3({0x1c02710?, 0xc000193a40?}, {0xc0008446f0, 0x27}, 0xc00228acc0)
2025-09-05 17:08:28 beacon-chain/sync/subscriber.go:376 +0x2ab
2025-09-05 17:08:28 github.com/libp2p/go-libp2p-pubsub.(*validatorImpl).validateMsg(0xc002ee9740, {0x1c02710, 0xc000193a40}, {0xc0008446f0, 0x27}, 0xc00228acc0)
2025-09-05 17:08:28 external/com_github_libp2p_go_libp2p_pubsub/validation.go:485 +0x18b
2025-09-05 17:08:28 github.com/libp2p/go-libp2p-pubsub.(*validation).validateSingleTopic(...)
2025-09-05 17:08:28 external/com_github_libp2p_go_libp2p_pubsub/validation.go:463
2025-09-05 17:08:28 github.com/libp2p/go-libp2p-pubsub.(*validation).validateTopic(0xc0004a95e0, {0xc003e43640?, 0xc001b0dfd0?, 0x34f5cbf?}, {0xc0008446f0, 0x27}, 0xc00228acc0)
2025-09-05 17:08:28 external/com_github_libp2p_go_libp2p_pubsub/validation.go:412 +0x118
2025-09-05 17:08:28 github.com/libp2p/go-libp2p-pubsub.(*validation).doValidateTopic(...)
2025-09-05 17:08:28 external/com_github_libp2p_go_libp2p_pubsub/validation.go:383
2025-09-05 17:08:28 github.com/libp2p/go-libp2p-pubsub.(*validation).validate.func1()
2025-09-05 17:08:28 external/com_github_libp2p_go_libp2p_pubsub/validation.go:348 +0x50
2025-09-05 17:08:28 created by github.com/libp2p/go-libp2p-pubsub.(*validation).validate in goroutine 464
2025-09-05 17:08:28 external/com_github_libp2p_go_libp2p_pubsub/validation.go:347 +0x84a
2025-09-05 17:08:32 [2025-09-05 09:08:32.00] INFO sync: Unsubscribed from topic=/eth2/d09a044a/beacon_attestation_1/ssz_snappy
2025-09-05 17:08:32 [2025-09-05 09:08:32.05] ERROR sync: Panic occurred error=runtime error: invalid memory address or nil pointer dereference recoveredAt=subscribeWithBase stack=goroutine 53151 [running]:runtime/debug.Stack()GOROOT/src/runtime/debug/stack.go:26 +0x5egithub.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.func1.1()beacon-chain/sync/subscriber.go:294 +0x24fpanic({0x1261400?, 0x4652c10?})GOROOT/src/runtime/panic.go:792 +0x132github.com/OffchainLabs/prysm/v6/beacon-chain/core/light-client.(*Store).LastOptimisticUpdate(0x1c1adc0?)beacon-chain/core/light-client/store.go:199 +0x29github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).processLightClientOptimisticUpdate(0xc0003f8140, {0x1c02780, 0xc002c380e0}, {0x1c1acb0, 0xc001e4e6c0}, {0x1c39e60, 0xc001de58c8})beacon-chain/blockchain/process_block_helpers.go:256 +0x365github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).processLightClientUpdates(0xc0003f8140, 0xc002478400)beacon-chain/blockchain/process_block_helpers.go:140 +0x1dbgithub.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).postBlockProcess(0xc0003f8140, 0xc002478400)beacon-chain/blockchain/process_block.go:115 +0xc25github.com/OffchainLabs/prysm/v6/beacon-chain/blockchain.(*Service).ReceiveBlock(0xc0003f8140, {0x1c02780?, 0xc002c380e0?}, {0x1c1adc0, 0xc002c381c0}, {0x9b, 0xac, 0x86, 0xd, 0x3a, ...}, ...)beacon-chain/blockchain/receive_block.go:135 +0xba8github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).beaconBlockSubscriber(0xc000174848, {0x1c02780, 0xc002c380e0}, {0x1be7540?, 0xc0026d6370?})beacon-chain/sync/subscriber_beacon_blocks.go:42 +0x2f4github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.func1(0xc003ba7a80)beacon-chain/sync/subscriber.go:307 +0x3f1created by github.com/OffchainLabs/prysm/v6/beacon-chain/sync.(*Service).subscribeWithBase.func2 in goroutine 48965beacon-chain/sync/subscriber.go:334 +0xf9 topic=/eth2/d09a044a/beacon_block/ssz_snappy
2025-09-05 17:08:36 [2025-09-05 09:08:36.00] ERROR message-handler: Panicked when handling p2p message! Recovering... msg=data:"\201\010\020\254\000\000\000\377z\001\000<G\273\356~u\371w\360\035\355\272\373\325\345\337\301>/\000\360c\265\353`\212\314\025\024v9Y\230\306\353/\273<\333Cj80/\205\333\254+\243\004\253\212\017\"Iz\2142\357\245\037\240\2449,\016\322#\022d\004\337_%\231v\226\006\0029\353\013\206\3355\207\220\301u|\350\303\010\372+>\\\221\374\367\263\037}i\321\007\037j\376\037\247\227\020\327\236\255\"\334\002\000\000\000\001\003\000\001\001\005\014\000\000\0000\001\005\364^\001\000\000\000\354b6j\3133\274+\221\314#\\\346\262\357&/P\276\314\344\352\343\016\3001\021\032\262bcTTb\275\023+\n\333\217\243xG'\262\310\026\210e\025\020\260\344x\343Tgf\256\357io9\226\241C\t\353\243\252\177\244{\244\310\n\363p'E7\022\3053\0149\242:5\177J\334\210\370\345\303\364\000\000\000\010\217R\213\032\013\224|\0275=S\202]\260\211c\300\344\324w\320\nNl>r\357\312\316/F\264o\014\001\200_\342\022\341Y\007\230\033u~lIk\014\260fd\"FUa=\316\310%\005\273m\323\271\225]\211-\2223\213\031\227o\320p\204\277\350\212v\303\0064\202\267\363\016\346\017\353*X\032\315\373~\341\367!\350\344\333\357\336kq\0342\374\373\374t\272\304\246\254\334\017\201\324\331b\344T\342G\017A\24123\222.al\340\341wy :8\2751K\370\234X\313Ts;\220\365\245h\211CTQw\200n\321{\237#\360\242\036\345\224\216\312\247v\t\007fI=\233L\330@\313\021\356\357\244/\005\212\007\255\026Cu'\271\355\262\220%o\302\277\331V\350\037\027\033\314U\246\377\203E\346\222\300\370n[H\340\033\231l\255\300\001b/\265\343c\264!\000\000\000\000!`\376\001\000\376\001\000\376\001\000\336\001\000\004\342Gvt\001Q\210\014 \242\222\003.$\001\014\204\250\272h\001\020\034H\002\000\000\300p'4\001\014^\001\000\360`\261zh\205\002\232 \201\335\207\016\305\374h)\274\222.\020\326[\361\345R\370\004P\010\304>\211%\177\376$\036\246\001\207\375\260\030{\372\"\3365\321\371\276\327\253\006\035\224\001\375G\343JT\373\355\341y)0\273\325\272\254C\274\307\230\356I\252\201\205\357v\273;D\272b\271\035\206\256V\236K\2655\000:y\000`\330\203\001\020\002\204geth\210go1.24.5\205linux" topic:"/eth2/d09a044a/light_client_optimistic_update/ssz_snappy" r=runtime error: invalid memory address or nil pointer dereference
Full node log: prysmlog.txt
Thanks for your time!
Has this worked before in a previous version?
Yes. At least in 6.0.4.
🔬 Minimal Reproduction
Just start Prysm with --enable-light-client, If removed it works fine.
Error
invalid memory address or nil pointer dereference
Platform(s)
Linux (x86)
What version of Prysm are you running? (Which release)
Prysm/v6.0.4 (linux amd64)
Anything else relevant (validator index / public key)?
Specific versions of prysm and validator images:
gcr.io/offchainlabs/prysm/beacon-chain:HEAD-278b79
gcr.io/offchainlabs/prysm/validator:HEAD-278b79