Skip to content

Commit b50554b

Browse files
committed
Merge bitcoin/bitcoin#29370: assumeutxo: Get rid of faked nTx and nChainTx values
9d9a745 assumeutxo: Remove BLOCK_ASSUMED_VALID flag (Ryan Ofsky) ef174e9 test: assumeutxo snapshot block CheckBlockIndex crash test (Ryan Ofsky) 0391458 test: assumeutxo stale block CheckBlockIndex crash test (Ryan Ofsky) ef29c8b assumeutxo: Get rid of faked nTx and nChainTx values (Ryan Ofsky) 9b97d5b doc: Improve comments describing setBlockIndexCandidates checks (Ryan Ofsky) 0fd915e validation: Check GuessVerificationProgress is not called with disconnected block (Ryan Ofsky) 63e8fc9 ci: add getchaintxstats ubsan suppressions (Ryan Ofsky) f252e68 assumeutxo test: Add RPC test for fake nTx and nChainTx values (Ryan Ofsky) Pull request description: The `PopulateAndValidateSnapshot` function introduced in f6e2da5 from #19806 has been setting fake `nTx` and `nChainTx` values that can show up in RPC results (bitcoin/bitcoin#29328) and make `CBlockIndex` state hard to reason about, because it is difficult to know whether the values are real or fake. Revert to previous behavior of setting `nTx` and `nChainTx` to 0 when the values are unknown, instead of faking them. Also drop no-longer needed `BLOCK_ASSUMED_VALID` flag. Dropping the faked values also fixes assert failures in the `CheckBlockIndex` `(pindex->nChainTx == pindex->nTx + prev_chain_tx)` check that could happen previously if forked or out-of-order blocks before the snapshot got submitted while the snapshot was being validated. The PR includes two commits adding tests for these failures and describing them in detail. Compatibility note: This change could cause new `-checkblockindex` failures if a snapshot was loaded by a previous version of Bitcoin Core and not fully validated, because fake `nTx` values will have been saved to the block index. It would be pretty easy to avoid these failures by adding some compatibility code to `LoadBlockIndex` and changing `nTx` values from 1 to 0 when they are fake (when `(pindex->nStatus & BLOCK_VALID_MASK) < BLOCK_VALID_TRANSACTIONS`), but a little simpler not to worry about being compatible in this case. ACKs for top commit: Sjors: re-ACK 9d9a745 achow101: ACK 9d9a745 mzumsande: Tested ACK 9d9a745 maflcko: ACK 9d9a745 🎯 Tree-SHA512: b1e1e2731ec36be30d5f5914042517219378fc31486674030c29d9c7488ed83fb60ba7095600f469dc32f0d8ba79c49ff7706303006507654e1762f26ee416e0
2 parents 69ddee6 + 9d9a745 commit b50554b

File tree

8 files changed

+231
-156
lines changed

8 files changed

+231
-156
lines changed

doc/design/assumeutxo.md

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -51,18 +51,12 @@ The utility script
5151

5252
## Design notes
5353

54-
- A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block
55-
index entries that are required to be assumed-valid by a chainstate created
56-
from a UTXO snapshot. This flag is used as a way to modify certain
57-
CheckBlockIndex() logic to account for index entries that are pending validation by a
58-
chainstate running asynchronously in the background.
59-
6054
- The concept of UTXO snapshots is treated as an implementation detail that lives
6155
behind the ChainstateManager interface. The external presentation of the changes
6256
required to facilitate the use of UTXO snapshots is the understanding that there are
63-
now certain regions of the chain that can be temporarily assumed to be valid (using
64-
the nStatus flag mentioned above). In certain cases, e.g. wallet rescanning, this is
65-
very similar to dealing with a pruned chain.
57+
now certain regions of the chain that can be temporarily assumed to be valid.
58+
In certain cases, e.g. wallet rescanning, this is very similar to dealing with
59+
a pruned chain.
6660

6761
Logic outside ChainstateManager should try not to know about snapshots, instead
6862
preferring to work in terms of more general states like assumed-valid.

src/chain.h

Lines changed: 19 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -98,16 +98,20 @@ enum BlockStatus : uint32_t {
9898

9999
/**
100100
* Only first tx is coinbase, 2 <= coinbase input script length <= 100, transactions valid, no duplicate txids,
101-
* sigops, size, merkle root. Implies all parents are at least TREE but not necessarily TRANSACTIONS. When all
102-
* parent blocks also have TRANSACTIONS, CBlockIndex::nChainTx will be set.
101+
* sigops, size, merkle root. Implies all parents are at least TREE but not necessarily TRANSACTIONS.
102+
*
103+
* If a block's validity is at least VALID_TRANSACTIONS, CBlockIndex::nTx will be set. If a block and all previous
104+
* blocks back to the genesis block or an assumeutxo snapshot block are at least VALID_TRANSACTIONS,
105+
* CBlockIndex::nChainTx will be set.
103106
*/
104107
BLOCK_VALID_TRANSACTIONS = 3,
105108

106109
//! Outputs do not overspend inputs, no double spends, coinbase output ok, no immature coinbase spends, BIP30.
107-
//! Implies all parents are either at least VALID_CHAIN, or are ASSUMED_VALID
110+
//! Implies all previous blocks back to the genesis block or an assumeutxo snapshot block are at least VALID_CHAIN.
108111
BLOCK_VALID_CHAIN = 4,
109112

110-
//! Scripts & signatures ok. Implies all parents are either at least VALID_SCRIPTS, or are ASSUMED_VALID.
113+
//! Scripts & signatures ok. Implies all previous blocks back to the genesis block or an assumeutxo snapshot block
114+
//! are at least VALID_SCRIPTS.
111115
BLOCK_VALID_SCRIPTS = 5,
112116

113117
//! All validity bits.
@@ -124,21 +128,8 @@ enum BlockStatus : uint32_t {
124128

125129
BLOCK_OPT_WITNESS = 128, //!< block data in blk*.dat was received with a witness-enforcing client
126130

127-
/**
128-
* If ASSUMED_VALID is set, it means that this block has not been validated
129-
* and has validity status less than VALID_SCRIPTS. Also that it may have
130-
* descendant blocks with VALID_SCRIPTS set, because they can be validated
131-
* based on an assumeutxo snapshot.
132-
*
133-
* When an assumeutxo snapshot is loaded, the ASSUMED_VALID flag is added to
134-
* unvalidated blocks at the snapshot height and below. Then, as the background
135-
* validation progresses, and these blocks are validated, the ASSUMED_VALID
136-
* flags are removed. See `doc/design/assumeutxo.md` for details.
137-
*
138-
* This flag is only used to implement checks in CheckBlockIndex() and
139-
* should not be used elsewhere.
140-
*/
141-
BLOCK_ASSUMED_VALID = 256,
131+
BLOCK_STATUS_RESERVED = 256, //!< Unused flag that was previously set on assumeutxo snapshot blocks and their
132+
//!< ancestors before they were validated, and unset when they were validated.
142133
};
143134

144135
/** The block chain is a tree shaped structure starting with the
@@ -173,21 +164,16 @@ class CBlockIndex
173164
//! (memory only) Total amount of work (expected number of hashes) in the chain up to and including this block
174165
arith_uint256 nChainWork{};
175166

176-
//! Number of transactions in this block.
167+
//! Number of transactions in this block. This will be nonzero if the block
168+
//! reached the VALID_TRANSACTIONS level, and zero otherwise.
177169
//! Note: in a potential headers-first mode, this number cannot be relied upon
178-
//! Note: this value is faked during UTXO snapshot load to ensure that
179-
//! LoadBlockIndex() will load index entries for blocks that we lack data for.
180-
//! @sa ActivateSnapshot
181170
unsigned int nTx{0};
182171

183172
//! (memory only) Number of transactions in the chain up to and including this block.
184-
//! This value will be non-zero only if and only if transactions for this block and all its parents are available.
173+
//! This value will be non-zero if this block and all previous blocks back
174+
//! to the genesis block or an assumeutxo snapshot block have reached the
175+
//! VALID_TRANSACTIONS level.
185176
//! Change to 64-bit type before 2024 (assuming worst case of 60 byte transactions).
186-
//!
187-
//! Note: this value is faked during use of a UTXO snapshot because we don't
188-
//! have the underlying block data available during snapshot load.
189-
//! @sa AssumeutxoData
190-
//! @sa ActivateSnapshot
191177
unsigned int nChainTx{0};
192178

193179
//! Verification status of this block. See enum BlockStatus
@@ -262,15 +248,14 @@ class CBlockIndex
262248
}
263249

264250
/**
265-
* Check whether this block's and all previous blocks' transactions have been
266-
* downloaded (and stored to disk) at some point.
251+
* Check whether this block and all previous blocks back to the genesis block or an assumeutxo snapshot block have
252+
* reached VALID_TRANSACTIONS and had transactions downloaded (and stored to disk) at some point.
267253
*
268254
* Does not imply the transactions are consensus-valid (ConnectTip might fail)
269255
* Does not imply the transactions are still stored on disk. (IsBlockPruned might return true)
270256
*
271-
* Note that this will be true for the snapshot base block, if one is loaded (and
272-
* all subsequent assumed-valid blocks) since its nChainTx value will have been set
273-
* manually based on the related AssumeutxoData entry.
257+
* Note that this will be true for the snapshot base block, if one is loaded, since its nChainTx value will have
258+
* been set manually based on the related AssumeutxoData entry.
274259
*/
275260
bool HaveNumChainTxs() const { return nChainTx != 0; }
276261

@@ -318,14 +303,6 @@ class CBlockIndex
318303
return ((nStatus & BLOCK_VALID_MASK) >= nUpTo);
319304
}
320305

321-
//! @returns true if the block is assumed-valid; this means it is queued to be
322-
//! validated by a background chainstate.
323-
bool IsAssumedValid() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
324-
{
325-
AssertLockHeld(::cs_main);
326-
return nStatus & BLOCK_ASSUMED_VALID;
327-
}
328-
329306
//! Raise the validity level of this block index entry.
330307
//! Returns true if the validity was changed.
331308
bool RaiseValidity(enum BlockStatus nUpTo) EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
@@ -335,12 +312,6 @@ class CBlockIndex
335312
if (nStatus & BLOCK_FAILED_MASK) return false;
336313

337314
if ((nStatus & BLOCK_VALID_MASK) < nUpTo) {
338-
// If this block had been marked assumed-valid and we're raising
339-
// its validity to a certain point, there is no longer an assumption.
340-
if (nStatus & BLOCK_ASSUMED_VALID && nUpTo >= BLOCK_VALID_SCRIPTS) {
341-
nStatus &= ~BLOCK_ASSUMED_VALID;
342-
}
343-
344315
nStatus = (nStatus & ~BLOCK_VALID_MASK) | nUpTo;
345316
return true;
346317
}

src/test/util/chainstate.h

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -91,13 +91,16 @@ CreateAndActivateUTXOSnapshot(
9191
// these blocks instead
9292
CBlockIndex *pindex = orig_tip;
9393
while (pindex && pindex != chain.m_chain.Tip()) {
94-
pindex->nStatus &= ~BLOCK_HAVE_DATA;
95-
pindex->nStatus &= ~BLOCK_HAVE_UNDO;
96-
// We have to set the ASSUMED_VALID flag, because otherwise it
97-
// would not be possible to have a block index entry without HAVE_DATA
98-
// and with nTx > 0 (since we aren't setting the pruned flag);
99-
// see CheckBlockIndex().
100-
pindex->nStatus |= BLOCK_ASSUMED_VALID;
94+
// Remove all data and validity flags by just setting
95+
// BLOCK_VALID_TREE. Also reset transaction counts and sequence
96+
// ids that are set when blocks are received, to make test setup
97+
// more realistic and satisfy consistency checks in
98+
// CheckBlockIndex().
99+
assert(pindex->IsValid(BlockStatus::BLOCK_VALID_TREE));
100+
pindex->nStatus = BlockStatus::BLOCK_VALID_TREE;
101+
pindex->nTx = 0;
102+
pindex->nChainTx = 0;
103+
pindex->nSequenceId = 0;
101104
pindex = pindex->pprev;
102105
}
103106
}

src/test/validation_chainstatemanager_tests.cpp

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -276,9 +276,6 @@ struct SnapshotTestSetup : TestChain100Setup {
276276
BOOST_CHECK_EQUAL(
277277
*node::ReadSnapshotBaseBlockhash(found),
278278
*chainman.SnapshotBlockhash());
279-
280-
// Ensure that the genesis block was not marked assumed-valid.
281-
BOOST_CHECK(!chainman.ActiveChain().Genesis()->IsAssumedValid());
282279
}
283280

284281
const auto& au_data = ::Params().AssumeutxoForHeight(snapshot_height);
@@ -410,7 +407,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_activate_snapshot, SnapshotTestSetup)
410407
//! - First, verify that setBlockIndexCandidates is as expected when using a single,
411408
//! fully-validating chainstate.
412409
//!
413-
//! - Then mark a region of the chain BLOCK_ASSUMED_VALID and introduce a second chainstate
410+
//! - Then mark a region of the chain as missing data and introduce a second chainstate
414411
//! that will tolerate assumed-valid blocks. Run LoadBlockIndex() and ensure that the first
415412
//! chainstate only contains fully validated blocks and the other chainstate contains all blocks,
416413
//! except those marked assume-valid, because those entries don't HAVE_DATA.
@@ -421,7 +418,6 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
421418
Chainstate& cs1 = chainman.ActiveChainstate();
422419

423420
int num_indexes{0};
424-
int num_assumed_valid{0};
425421
// Blocks in range [assumed_valid_start_idx, last_assumed_valid_idx) will be
426422
// marked as assumed-valid and not having data.
427423
const int expected_assumed_valid{20};
@@ -456,35 +452,30 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
456452
reload_all_block_indexes();
457453
BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), 1);
458454

459-
// Mark some region of the chain assumed-valid, and remove the HAVE_DATA flag.
455+
// Reset some region of the chain's nStatus, removing the HAVE_DATA flag.
460456
for (int i = 0; i <= cs1.m_chain.Height(); ++i) {
461457
LOCK(::cs_main);
462458
auto index = cs1.m_chain[i];
463459

464-
// Blocks with heights in range [91, 110] are marked ASSUMED_VALID
460+
// Blocks with heights in range [91, 110] are marked as missing data.
465461
if (i < last_assumed_valid_idx && i >= assumed_valid_start_idx) {
466-
index->nStatus = BlockStatus::BLOCK_VALID_TREE | BlockStatus::BLOCK_ASSUMED_VALID;
462+
index->nStatus = BlockStatus::BLOCK_VALID_TREE;
463+
index->nTx = 0;
464+
index->nChainTx = 0;
467465
}
468466

469467
++num_indexes;
470-
if (index->IsAssumedValid()) ++num_assumed_valid;
471468

472469
// Note the last fully-validated block as the expected validated tip.
473470
if (i == (assumed_valid_start_idx - 1)) {
474471
validated_tip = index;
475-
BOOST_CHECK(!index->IsAssumedValid());
476472
}
477473
// Note the last assumed valid block as the snapshot base
478474
if (i == last_assumed_valid_idx - 1) {
479475
assumed_base = index;
480-
BOOST_CHECK(index->IsAssumedValid());
481-
} else if (i == last_assumed_valid_idx) {
482-
BOOST_CHECK(!index->IsAssumedValid());
483476
}
484477
}
485478

486-
BOOST_CHECK_EQUAL(expected_assumed_valid, num_assumed_valid);
487-
488479
// Note: cs2's tip is not set when ActivateExistingSnapshot is called.
489480
Chainstate& cs2 = WITH_LOCK(::cs_main,
490481
return chainman.ActivateExistingSnapshot(*assumed_base->phashBlock));

0 commit comments

Comments
 (0)