[ZK Stack] - column "predicted_gas_limit" of relation "eth_txs_history" does not exist

[moneta]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.85.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I have the ecosystem deployed with the branch core-v27.3.0. After I bridged a base from L1 to L2, new blocks/batches are added. Things looks normal, then the sequencer node crashes with this error

error returned from database: column "predicted_gas_limit" of relation "eth_txs_history" does not exist 

Here are the log messages when I tried to restart the node:

2025-05-29T15:53:19.060866Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading state changes for l1 batch 1                                                             
2025-05-29T15:53:19.090308Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading factory deps for L1 batch 1                                                              
2025-05-29T15:53:19.102049Z  INFO zksync_merkle_tree::domain: Extending Merkle tree with batch #1 with 164 ops in full mode                                                                                 
2025-05-29T15:53:19.105920Z  INFO EthWatch::loop_iteration: zksync_db_connection::instrument: Slow query update_next_block_to_process(event_type=PriorityTransactions, chain_id=SLChainId(11155111)) called 
at /usr/src/zksync/core/lib/dal/src/eth_watcher_dal.rs:94 [requested by `eth_watch` at /usr/src/zksync/core/node/eth_watch/src/lib.rs:156] has finished after 253.336977ms                                  
2025-05-29T15:53:19.117572Z  INFO zksync_merkle_tree::domain: Processed batch #1; root hash is 0x87dc…316d, 170 leaves in total                                                                             
2025-05-29T15:53:19.125250Z  INFO catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_db_connection::instrument: Slow query get_l1_batch_factory_deps(l1_batch_number=L1BatchNumber(0)) called at /usr/
src/zksync/core/lib/dal/src/blocks_dal.rs:2346 [requested by `state_keeper` at /usr/src/zksync/core/lib/state/src/catchup.rs:174] has finished after 166.482871ms                                           
2025-05-29T15:53:19.125590Z  INFO TreeUpdater::step{next_l1_batch_to_process=L1BatchNumber(1)}: zksync_metadata_calculator::updater: Saved witnesses for L1 batch #1 to object storage at `merkle_tree_paths
_1.cbor`                                                                                                                                                                                                    
2025-05-29T15:53:19.130184Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading state changes for l1 batch 2                                                             
2025-05-29T15:53:19.131015Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading state changes for l1 batch 1                                                             
2025-05-29T15:53:19.135838Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading factory deps for L1 batch 2                                                              
2025-05-29T15:53:19.202248Z  INFO NamedFuture{name=state_keeper/rocksdb_catchup_task}:catch_up{target_l1_batch=None}: zksync_db_connection::instrument: Slow query get_l1_batch_factory_deps(l1_batch_number
=L1BatchNumber(0)) called at /usr/src/zksync/core/lib/dal/src/blocks_dal.rs:2346 [requested by `state_keeper` at /usr/src/zksync/core/lib/state/src/catchup.rs:174] has finished after 241.919616ms         
2025-05-29T15:53:19.212913Z DEBUG NamedFuture{name=state_keeper/rocksdb_catchup_task}:catch_up{target_l1_batch=None}: zksync_state::rocksdb: Loading state changes for l1 batch 1                           
2025-05-29T15:53:19.226790Z  INFO catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Secondary storage for L1 batch #2 initialized, size is 181                                       
2025-05-29T15:53:19.231103Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading factory deps for L1 batch 1                                                              
2025-05-29T15:53:19.231369Z DEBUG NamedFuture{name=state_keeper/rocksdb_catchup_task}:catch_up{target_l1_batch=None}: zksync_state::rocksdb: Loading factory deps for L1 batch 1                            
2025-05-29T15:53:19.285863Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading state changes for l1 batch 2                                                             
2025-05-29T15:53:19.300105Z DEBUG NamedFuture{name=state_keeper/rocksdb_catchup_task}:catch_up{target_l1_batch=None}: zksync_state::rocksdb: Loading state changes for l1 batch 2                           
2025-05-29T15:53:19.302054Z DEBUG catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Loading factory deps for L1 batch 2                                                              
2025-05-29T15:53:19.305038Z  INFO catch_up{target_l1_batch=Some(L1BatchNumber(2))}: zksync_state::rocksdb: Secondary storage for L1 batch #2 initialized, size is 181                                       
2025-05-29T15:53:19.306285Z DEBUG NamedFuture{name=state_keeper/rocksdb_catchup_task}:catch_up{target_l1_batch=None}: zksync_state::rocksdb: Loading factory deps for L1 batch 2                            
2025-05-29T15:53:19.311179Z  INFO TreeUpdater::step{next_l1_batch_to_process=L1BatchNumber(1)}: zksync_metadata_calculator::updater: Updated metadata for L1 batch #1 in Postgres                           
2025-05-29T15:53:19.313037Z  INFO zksync_merkle_tree::domain: Extending Merkle tree with batch #2 with 40 ops in full mode                                                                                  
2025-05-29T15:53:19.315981Z  INFO zksync_merkle_tree::domain: Processed batch #2; root hash is 0x7604…df00, 174 leaves in total                                                                             
2025-05-29T15:53:19.316171Z  INFO NamedFuture{name=state_keeper/rocksdb_catchup_task}:catch_up{target_l1_batch=None}: zksync_state::rocksdb: Secondary storage for L1 batch #2 initialized, size is 181     
2025-05-29T15:53:19.317059Z  INFO TreeUpdater::step{next_l1_batch_to_process=L1BatchNumber(1)}: zksync_metadata_calculator::updater: Saved witnesses for L1 batch #2 to object storage at `merkle_tree_paths
_2.cbor`                                                                                                                                                                                                    
2025-05-29T15:53:19.368214Z  INFO TreeUpdater::step{next_l1_batch_to_process=L1BatchNumber(1)}: zksync_metadata_calculator::updater: Updated metadata for L1 batch #2 in Postgres                           
2025-05-29T15:53:19.368359Z  INFO zksync_merkle_tree::domain: Flushing L1 batches #[1, 2] to RocksDB                                                                                                        
2025-05-29T15:53:19.369127Z  INFO TreeUpdater::step{next_l1_batch_to_process=L1BatchNumber(1)}: zksync_shared_metrics::tree: L1 batches #1..=2 processed in tree                                            
2025-05-29T15:53:19.446595Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Sending 1 Blob new transactions                                                                          
2025-05-29T15:53:19.448809Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Sending Blob tx 2 (nonce 2) at block 8432968 with base_fee_per_gas 1112151090, priority_fee_per_gas 20000
00000, blob_fee_per_gas Some(2),max_gas_per_pubdata_price None,gas_limit 15000000,                                                                                                                          
thread 'tokio-runtime-worker' panicked at /usr/src/zksync/core/node/eth_sender/src/eth_tx_manager.rs:242:14:                                                                                                
called `Result::unwrap()` on an `Err` value: error returned from database: column "predicted_gas_limit" of relation "eth_txs_history" does not exist                                                        
Caused by:                                                                                                                                                                                                  
    column "predicted_gas_limit" of relation "eth_txs_history" does not exist                                                                                                                               
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                                                                                                               
2025-05-29T15:53:19.469762Z DEBUG zksync_health_check: Changed health of `eth_tx_manager` from {"status":"ready"} to {"status":"panicked"}                                                                  
2025-05-29T15:53:19.469883Z ERROR zksync_node_framework::service: Task eth_tx_manager panicked: called `Result::unwrap()` on an `Err` value: error returned from database: column "predicted_gas_limit" of r
elation "eth_txs_history" does not exist                                                                                                                                                                    
Caused by:                                                                                                                                                                                                  
    column "predicted_gas_limit" of relation "eth_txs_history" does not exist                                                                                                                               
2025-05-29T15:53:19.470221Z  INFO zksync_node_framework::service: One of the task has exited, shutting down the node   

Expected Behavior

There should not be database error

Current Behavior

Database error trying to access non-existing column

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.