Data race in IAVL nodeDB storage version access during concurrent read/write operations #455
Description
A data race occurs when the IAVL tree's nodeDB storage version field is accessed concurrently without synchronization. Read operations (via getStorageVersion() during query handling) and write operations (via SetFastStorageVersionToBatch() during commit) are accessing the same memory location (0x00c001b8bac8) simultaneously.
The race manifests when:
Read path: A gRPC query (GetLatestSpan) attempts to read from the store, which checks the storage version to determine if fast storage is available
Write path: A commit operation is saving a new version and updating the fast storage version in the database
This can lead to inconsistent reads of the storage version state and potential undefined behavior. The issue requires adding proper synchronization (likely a mutex) around access to the storage version field in the nodeDB struct.
==================
WARNING: DATA RACE
Read at 0x00c001b8bac8 by goroutine 41650:
github.com/cosmos/iavl.(*nodeDB).getStorageVersion()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/nodedb.go:332 +0x73
github.com/cosmos/iavl.(*nodeDB).hasUpgradedToFastStorage()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/nodedb.go:337 +0x8a
github.com/cosmos/iavl.(*nodeDB).GetFastNode()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/nodedb.go:199 +0x1d
github.com/cosmos/iavl.(*ImmutableTree).Get()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/immutable_tree.go:186 +0xd0
cosmossdk.io/store/iavl.(*Store).Get()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/iavl/store.go:200 +0x144
cosmossdk.io/store/iavl.(*Store).Get()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/iavl/store.go:200 +0x144
cosmossdk.io/store/cachekv.(*Store).Get()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/cachekv/store.go:61 +0x26a
cosmossdk.io/store/cachekv.(*Store).Has()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/cachekv/store.go:82 +0x44
cosmossdk.io/store/gaskv.(*Store).Has()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/gaskv/store.go:60 +0xb0
github.com/cosmos/cosmos-sdk/runtime.coreKVStore.Has()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk@v0.2.6-polygon/runtime/store.go:68 +0x6f
github.com/cosmos/cosmos-sdk/runtime.(*coreKVStore).Has()
<autogenerated>:1 +0x1f
cosmossdk.io/collections.Map[go.shape.uint64,go.shape.832bca78113e242a6f39b3b7d7418354c940d3d008c6882c4f76810998e5608c].Has()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/collections@v0.4.0/map.go:104 +0x101
github.com/0xPolygon/heimdall-v2/x/bor/keeper.(*Keeper).GetSpan()
x/bor/keeper/keeper.go:133 +0x116
github.com/0xPolygon/heimdall-v2/x/bor/keeper.(*Keeper).GetLastSpan()
x/bor/keeper/keeper.go:202 +0x31c
cosmossdk.io/collections.Item[go.shape.uint64].Has()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/collections@v0.4.0/item.go:42 +0x147
github.com/0xPolygon/heimdall-v2/x/bor/keeper.(*Keeper).GetLastSpan()
x/bor/keeper/keeper.go:187 +0x7e
github.com/0xPolygon/heimdall-v2/x/bor/keeper.queryServer.GetLatestSpan()
x/bor/keeper/grpc_query.go:46 +0x18f
github.com/0xPolygon/heimdall-v2/x/bor/types._Query_GetLatestSpan_Handler.func1()
x/bor/types/query.pb.go:1037 +0xcb
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk@v0.2.6-polygon/baseapp/grpcserver.go:65 +0x4b7
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.2.1()
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0xcd
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1()
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/recovery/interceptors.go:33 +0x174
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.2()
/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53 +0x222
github.com/0xPolygon/heimdall-v2/x/bor/types._Query_GetLatestSpan_Handler()
x/bor/types/query.pb.go:1039 +0x22f
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk@v0.2.6-polygon/baseapp/grpcserver.go:79 +0x1ad
google.golang.org/grpc.(*Server).processUnaryRPC()
/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1400 +0x1bc2
google.golang.org/grpc.(*Server).processUnaryRPC()
/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1356 +0x15e7
google.golang.org/grpc.(*Server).handleStream()
/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1810 +0x1372
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/go/pkg/mod/google.golang.org/grpc@v1.70.0/server.go:1030 +0x158
Previous write at 0x00c001b8bac8 by goroutine 315:
github.com/cosmos/iavl.(*nodeDB).SetFastStorageVersionToBatch()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/nodedb.go:327 +0x35d
github.com/cosmos/iavl.(*MutableTree).saveFastNodeVersion()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/mutable_tree.go:798 +0x86
github.com/cosmos/iavl.(*MutableTree).SaveVersion()
/go/pkg/mod/github.com/cosmos/iavl@v1.2.2/mutable_tree.go:742 +0xb84
cosmossdk.io/store/iavl.(*Store).Commit()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/iavl/store.go:126 +0x109
cosmossdk.io/store/cache.(*CommitKVStoreCache).Commit()
<autogenerated>:1 +0x43
cosmossdk.io/store/rootmulti.commitStores()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/rootmulti/store.go:1196 +0x199
cosmossdk.io/store/rootmulti.commitStores()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/rootmulti/store.go:1196 +0x199
cosmossdk.io/store/rootmulti.commitStores()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/rootmulti/store.go:1196 +0x199
cosmossdk.io/store/rootmulti.(*Store).Commit()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk/store@v1.1.2-0.20241126102051-89dc71d02611/rootmulti/store.go:482 +0x36b
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk@v0.2.6-polygon/baseapp/abci.go:931 +0x35a
github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.Commit()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk@v0.2.6-polygon/server/cmt_abci.go:55 +0x4a
github.com/cosmos/cosmos-sdk/server.(*cometABCIWrapper).Commit()
<autogenerated>:1 +0x1b
github.com/cosmos/cosmos-sdk/server.cometABCIWrapper.Commit()
/go/pkg/mod/github.com/0x!polygon/cosmos-sdk@v0.2.6-polygon/server/cmt_abci.go:55 +0x4a
github.com/cosmos/cosmos-sdk/server.(*cometABCIWrapper).Commit()
<autogenerated>:1 +0x1b
github.com/cometbft/cometbft/abci/client.(*localClient).Commit()
/go/pkg/mod/github.com/0x!polygon/cometbft@v0.3.0-polygon/abci/client/local_client.go:113 +0x14e
github.com/cometbft/cometbft/proxy.(*appConnConsensus).Commit()
/go/pkg/mod/github.com/0x!polygon/cometbft@v0.3.0-polygon/proxy/app_conn.go:109 +0x2bd
github.com/cometbft/cometbft/state.(*BlockExecutor).Commit()
{
"IPT_bytes_out": 34941824,
"output_text": "WARNING: DATA RACE",
"source": {
"container": "l2-cl-1-heimdall-v2-bor-validator--0652d832865847b4a61dd4dda37e4bc8",
"name": "l2-cl-1-heimdall-v2-bor-validator--0652d832865847b4a61dd4dda37e4bc8",
"stream": "info"
},
"moment": {
"input_hash": "-2198210237136624424",
"_vtime_ticks": 1842702253946,
"session_id": "387aac0404473becedb39ef62b17f934-37-2"
}
}