Skip to content

Move unpacking related code to agave_snapshots #8428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Move unpacking related code to agave_snapshots #8428

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
64cf4e0
Move unpacking related code to agave_snapshots
kskalski Oct 10, 2025
8ab6164
Update lock
kskalski Oct 10, 2025
8df8e46
Bring back lz4
kskalski Oct 10, 2025
8ca704f
Missing files
kskalski Oct 10, 2025
3808ae2
Update handling of set perms
kskalski Oct 12, 2025
d4874bd
Fix cfg on import
kskalski Oct 13, 2025
000014e
Move some deps to dev
kskalski Oct 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 0 additions & 2 deletions accounts-db/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ blake3 = { workspace = true }
bv = { workspace = true, features = ["serde"] }
bytemuck = { workspace = true }
bytemuck_derive = { workspace = true }
bzip2 = { workspace = true }
crossbeam-channel = { workspace = true }
dashmap = { workspace = true, features = ["rayon", "raw-api"] }
indexmap = { workspace = true }
Expand Down Expand Up @@ -101,7 +100,6 @@ solana-transaction-error = { workspace = true }
solana-vote-program = { workspace = true, optional = true }
spl-generic-token = { workspace = true }
static_assertions = { workspace = true }
tar = { workspace = true }
tempfile = { workspace = true }
thiserror = { workspace = true }

Expand Down
6 changes: 4 additions & 2 deletions accounts-db/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ mod bucket_map_holder_stats;
mod buffered_reader;
pub mod contains;
mod file_io;
pub mod hardened_unpack;
mod io_uring;
pub mod is_loadable;
mod is_zero_lamport;
Expand All @@ -46,7 +45,10 @@ pub mod utils;
pub mod waitable_condvar;

pub use {
buffered_reader::large_file_buf_reader, file_io::validate_memlock_limit_for_disk_io,
buffered_reader::large_file_buf_reader,
file_io::{
file_creator, set_path_permissions, validate_memlock_limit_for_disk_io, FileCreator,
},
obsolete_accounts::ObsoleteAccounts,
};

Expand Down
10 changes: 6 additions & 4 deletions core/src/validator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,19 @@ use {
tpu::{ForwardingClientOption, Tpu, TpuSockets, DEFAULT_TPU_COALESCE},
tvu::{Tvu, TvuConfig, TvuSockets},
},
agave_snapshots::SnapshotInterval,
agave_snapshots::{
hardened_unpack::{
open_genesis_config, OpenGenesisConfigError, MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
Copy link

@roryharr roryharr Oct 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What a mess MAX_GENESIS_ARCHIVE_UNPACKED_SIZE makes this!

Is hardened unpack even the right place for it to live?

I guess it started out as a hard limit on the genesis size, but then became a configurable parameter, but was left in the hardened_unpack function.

Should it be part of genesis_config instead? I'm not familiar with that part of the code at all, so my suggestion may be very off base.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a kind of sanity check limit, but you are right, this is technically a global constraint that could as well be exposed as public constant in genesis_config...
Moving it there seems like a few steps, but I will prepare the change if that is desired, @steviez what do you think?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, actually this file doesn't really need this constant - it's only used in tests and can be replaced by a more aggressive constant easily (also can add test that actually hits the limit)

so for now I will extract it to a place closer to where it's actually used

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't had the chance to review this yet, but happened to see #8489; will try to chime in tomorrow

},
SnapshotInterval,
},
anyhow::{anyhow, Context, Result},
crossbeam_channel::{bounded, unbounded, Receiver},
quinn::Endpoint,
serde::{Deserialize, Serialize},
solana_accounts_db::{
accounts_db::{AccountsDbConfig, ACCOUNTS_DB_CONFIG_FOR_TESTING},
accounts_update_notifier_interface::AccountsUpdateNotifier,
hardened_unpack::{
open_genesis_config, OpenGenesisConfigError, MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
},
utils::move_and_async_delete_path_contents,
},
solana_client::{
Expand Down
2 changes: 1 addition & 1 deletion genesis-utils/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ crate-type = ["lib"]
name = "solana_genesis_utils"

[dependencies]
agave-snapshots = { workspace = true }
log = { workspace = true }
solana-accounts-db = { workspace = true }
solana-download-utils = { workspace = true }
solana-genesis-config = { workspace = true }
solana-hash = { workspace = true }
Expand Down
2 changes: 1 addition & 1 deletion genesis-utils/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
use {
agave_snapshots::hardened_unpack::unpack_genesis_archive,
log::*,
solana_accounts_db::hardened_unpack::unpack_genesis_archive,
solana_download_utils::download_genesis_if_missing,
solana_genesis_config::{GenesisConfig, DEFAULT_GENESIS_ARCHIVE},
solana_hash::Hash,
Expand Down
2 changes: 1 addition & 1 deletion genesis/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ path = "src/main.rs"

[dependencies]
agave-feature-set = { workspace = true }
agave-snapshots = { workspace = true }
base64 = { workspace = true }
bincode = { workspace = true }
clap = { workspace = true }
Expand All @@ -29,7 +30,6 @@ serde = { workspace = true }
serde_json = { workspace = true }
serde_yaml = { workspace = true }
solana-account = "=3.1.0"
solana-accounts-db = { workspace = true }
solana-clap-utils = { workspace = true }
solana-cli-config = { workspace = true }
solana-clock = "=3.0.0"
Expand Down
2 changes: 1 addition & 1 deletion genesis/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@

use {
agave_feature_set::FEATURE_NAMES,
agave_snapshots::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
base64::{prelude::BASE64_STANDARD, Engine},
clap::{crate_description, crate_name, value_t, value_t_or_exit, App, Arg, ArgMatches},
itertools::Itertools,
solana_account::{Account, AccountSharedData, ReadableAccount, WritableAccount},
solana_accounts_db::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
solana_clap_utils::{
input_parsers::{
cluster_type_of, pubkey_of, pubkeys_of, unix_timestamp_from_rfc3339_datetime,
Expand Down
2 changes: 1 addition & 1 deletion ledger-tool/src/args.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ pub fn hardforks_of(matches: &ArgMatches<'_>, name: &str) -> Option<Vec<Slot>> {

#[cfg(test)]
mod tests {
use {super::*, solana_accounts_db::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE};
use {super::*, agave_snapshots::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE};

#[test]
fn test_max_genesis_archive_unpacked_size_constant() {
Expand Down
6 changes: 3 additions & 3 deletions ledger-tool/src/ledger_utils.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
use {
crate::LEDGER_TOOL_DIRECTORY,
agave_snapshots::hardened_unpack::open_genesis_config,
clap::{value_t, value_t_or_exit, values_t_or_exit, ArgMatches},
crossbeam_channel::unbounded,
log::*,
solana_accounts_db::{
hardened_unpack::open_genesis_config,
utils::{create_all_accounts_run_and_snapshot_dirs, move_and_async_delete_path_contents},
solana_accounts_db::utils::{
create_all_accounts_run_and_snapshot_dirs, move_and_async_delete_path_contents,
},
solana_clock::Slot,
solana_core::validator::BlockVerificationMethod,
Expand Down
2 changes: 1 addition & 1 deletion ledger-tool/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1778,7 +1778,7 @@ fn main() {
create_new_ledger(
&output_directory,
&genesis_config,
solana_accounts_db::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
agave_snapshots::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
LedgerColumnOptions::default(),
)
.unwrap_or_else(|err| {
Expand Down
1 change: 1 addition & 0 deletions ledger/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ agave-unstable-api = []
[dependencies]
agave-feature-set = { workspace = true }
agave-reserved-account-keys = { workspace = true }
agave-snapshots = { workspace = true }
anyhow = { workspace = true }
assert_matches = { workspace = true }
bincode = { workspace = true }
Expand Down
8 changes: 4 additions & 4 deletions ledger/src/blockstore.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ use {
transaction_address_lookup_table_scanner::scan_transaction,
},
agave_feature_set::FeatureSet,
agave_snapshots::hardened_unpack::unpack_genesis_archive,
assert_matches::debug_assert_matches,
bincode::{deserialize, serialize},
crossbeam_channel::{bounded, Receiver, Sender, TrySendError},
Expand All @@ -34,7 +35,6 @@ use {
rayon::iter::{IntoParallelIterator, ParallelIterator},
rocksdb::{DBRawIterator, LiveFile},
solana_account::ReadableAccount,
solana_accounts_db::hardened_unpack::unpack_genesis_archive,
solana_address_lookup_table_interface::state::AddressLookupTable,
solana_clock::{Slot, UnixTimestamp, DEFAULT_TICKS_PER_SECOND},
solana_entry::entry::{create_ticks, Entry},
Expand Down Expand Up @@ -5274,14 +5274,14 @@ pub mod tests {
leader_schedule::{FixedSchedule, IdentityKeyedLeaderSchedule},
shred::{max_ticks_per_n_shreds, MAX_DATA_SHREDS_PER_SLOT},
},
agave_snapshots::hardened_unpack::{
open_genesis_config, MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
},
assert_matches::assert_matches,
bincode::{serialize, Options},
crossbeam_channel::unbounded,
rand::{seq::SliceRandom, thread_rng},
solana_account_decoder::parse_token::UiTokenAmount,
solana_accounts_db::hardened_unpack::{
open_genesis_config, MAX_GENESIS_ARCHIVE_UNPACKED_SIZE,
},
solana_clock::{DEFAULT_MS_PER_SLOT, DEFAULT_TICKS_PER_SLOT},
solana_entry::entry::{next_entry, next_entry_mut},
solana_hash::Hash,
Expand Down
4 changes: 1 addition & 3 deletions ledger/src/blockstore/error.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
//! The error that can be produced from Blockstore operations.

use {
log::*, solana_accounts_db::hardened_unpack::UnpackError, solana_clock::Slot, thiserror::Error,
};
use {agave_snapshots::hardened_unpack::UnpackError, log::*, solana_clock::Slot, thiserror::Error};

#[derive(Error, Debug)]
pub enum BlockstoreError {
Expand Down
2 changes: 1 addition & 1 deletion ledger/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ extern crate solana_frozen_abi_macro;

#[doc(hidden)]
pub mod macro_reexports {
pub use solana_accounts_db::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE;
pub use agave_snapshots::hardened_unpack::MAX_GENESIS_ARCHIVE_UNPACKED_SIZE;
}

mod wire_format_tests;
6 changes: 2 additions & 4 deletions local-cluster/tests/local_cluster.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#![allow(clippy::arithmetic_side_effects)]
use {
agave_snapshots::SnapshotInterval,
agave_snapshots::{hardened_unpack::open_genesis_config, SnapshotInterval},
assert_matches::assert_matches,
crossbeam_channel::{unbounded, Receiver},
gag::BufferRedirect,
Expand All @@ -9,9 +9,7 @@ use {
rand::seq::SliceRandom,
serial_test::serial,
solana_account::AccountSharedData,
solana_accounts_db::{
hardened_unpack::open_genesis_config, utils::create_accounts_run_and_snapshot_dirs,
},
solana_accounts_db::utils::create_accounts_run_and_snapshot_dirs,
solana_client_traits::AsyncClient,
solana_clock::{
self as clock, Slot, DEFAULT_SLOTS_PER_EPOCH, DEFAULT_TICKS_PER_SLOT, MAX_PROCESSING_AGE,
Expand Down
13 changes: 10 additions & 3 deletions programs/sbf/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading