Improve indexing speed

[konstantinabl]

Hi there,

Im part of hedera hashgraph team and we are currently exploring using blockscout with our network through our json-rpc-relay
Im running blockscout with the following configuration

ETHEREUM_JSONRPC_VARIANT=geth
ETHEREUM_JSONRPC_HTTP_URL=https://previewnet.hashio.io/api
DATABASE_URL=postgresql://blockscout:ceWb1MeLBEeOIfk65gU8EjF8@db:5432/blockscout


ETHEREUM_JSONRPC_TRANSPORT=http
ETHEREUM_JSONRPC_DISABLE_ARCHIVE_BALANCES=false
ETHEREUM_JSONRPC_TRACE_URL=https://previewnet.hashio.io/api
ETHEREUM_JSONRPC_WS_URL=ws://previewnet.hashio.io/ws

NETWORK=Hedera
SUBNETWORK=Awesome chain
LOGO=/images/blockscout_logo.svg
IPC_PATH=
NETWORK_PATH=/
BLOCKSCOUT_HOST=
BLOCKSCOUT_PROTOCOL=
SECRET_KEY_BASE=56NtB48ear7+wMSf0IQuWDAAazhpb31qyc7GiyspBP2vh7t5zlCsF5QDv76chXeN
PORT=4000
COIN_NAME=HBAR
EMISSION_FORMAT=DEFAULT
COIN=HBAR
EXCHANGE_RATES_COIN=
POOL_SIZE=80
POOL_SIZE_API=10
ECTO_USE_SSL=false
HEART_BEAT_TIMEOUT=30
RELEASE_LINK=
BLOCK_TRANSFORMER=base
FOOTER_LOGO=/images/blockscout_logo.svg
FOOTER_LINK_TO_OTHER_EXPLORERS=false
FOOTER_OTHER_EXPLORERS={}
SUPPORTED_CHAINS={}
CACHE_BLOCK_COUNT_PERIOD=7200
CACHE_TXS_COUNT_PERIOD=7200
CACHE_ADDRESS_SUM_PERIOD=3600
CACHE_TOTAL_GAS_USAGE_PERIOD=3600
CACHE_ADDRESS_TRANSACTIONS_GAS_USAGE_COUNTER_PERIOD=1800
CACHE_TOKEN_HOLDERS_COUNTER_PERIOD=3600
CACHE_TOKEN_TRANSFERS_COUNTER_PERIOD=3600
CACHE_ADDRESS_WITH_BALANCES_UPDATE_INTERVAL=1800
CACHE_AVERAGE_BLOCK_PERIOD=1800
CACHE_MARKET_HISTORY_PERIOD=21600
CACHE_ADDRESS_TRANSACTIONS_COUNTER_PERIOD=1800
CACHE_ADDRESS_TOKENS_USD_SUM_PERIOD=3600
CACHE_ADDRESS_TOKEN_TRANSFERS_COUNTER_PERIOD=1800
TOKEN_METADATA_UPDATE_INTERVAL=172800
CONTRACT_VERIFICATION_ALLOWED_SOLIDITY_EVM_VERSIONS=homestead,tangerineWhistle,spuriousDragon,byzantium,constantinople,petersburg,istanbul,berlin,london,paris,shanghai,cancun,default
CONTRACT_VERIFICATION_ALLOWED_VYPER_EVM_VERSIONS=byzantium,constantinople,petersburg,istanbul,berlin,paris,shanghai,cancun,default
CONTRACT_MAX_STRING_LENGTH_WITHOUT_TRIMMING=2040
UNCLES_IN_AVERAGE_BLOCK_TIME=false
DISABLE_WEBAPP=true
ADMIN_PANEL_ENABLED=false
API_V2_ENABLED=true
API_V1_READ_METHODS_DISABLED=false
API_V1_WRITE_METHODS_DISABLED=false
API_RATE_LIMIT_TIME_INTERVAL=1s
API_RATE_LIMIT_BY_IP_TIME_INTERVAL=5m
API_RATE_LIMIT=50
API_RATE_LIMIT_BY_KEY=50
API_RATE_LIMIT_BY_WHITELISTED_IP=50
API_RATE_LIMIT_WHITELISTED_IPS=
API_RATE_LIMIT_STATIC_API_KEY=
API_RATE_LIMIT_UI_V2_WITH_TOKEN=5
API_RATE_LIMIT_BY_IP=3000
API_NO_RATE_LIMIT_API_KEY=
DISABLE_INDEXER=false
DISABLE_REALTIME_INDEXER=false
DISABLE_CATCHUP_INDEXER=false
INDEXER_DISABLE_ADDRESS_COIN_BALANCE_FETCHER=false
INDEXER_DISABLE_TOKEN_INSTANCE_REALTIME_FETCHER=false
INDEXER_DISABLE_TOKEN_INSTANCE_RETRY_FETCHER=false
INDEXER_DISABLE_TOKEN_INSTANCE_SANITIZE_FETCHER=false
INDEXER_DISABLE_TOKEN_INSTANCE_LEGACY_SANITIZE_FETCHER=false
SHOW_ADDRESS_MARKETCAP_PERCENTAGE=true
CHECKSUM_ADDRESS_HASHES=true
CHECKSUM_FUNCTION=eth
DISABLE_EXCHANGE_RATES=true
TXS_STATS_ENABLED=true
SHOW_PRICE_CHART=false
SHOW_PRICE_CHART_LEGEND=false
SHOW_TXS_CHART=true
TXS_HISTORIAN_INIT_LAG=0
TXS_STATS_DAYS_TO_COMPILE_AT_INIT=10
COIN_BALANCE_HISTORY_DAYS=90
APPS_MENU=true
EXTERNAL_APPS=[]
SHOW_MAINTENANCE_ALERT=false
MAINTENANCE_ALERT_MESSAGE=
CHAIN_ID=
MAX_SIZE_UNLESS_HIDE_ARRAY=50
HIDE_BLOCK_MINER=false
DISPLAY_TOKEN_ICONS=false
RE_CAPTCHA_SECRET_KEY=
RE_CAPTCHA_CLIENT_KEY=
RE_CAPTCHA_V3_SECRET_KEY=
RE_CAPTCHA_V3_CLIENT_KEY=
RE_CAPTCHA_DISABLED=false
JSON_RPC=
API_RATE_LIMIT_UI_V2_TOKEN_TTL_IN_SECONDS=18000
FETCH_REWARDS_WAY=trace_block
MICROSERVICE_SC_VERIFIER_ENABLED=true
MICROSERVICE_SC_VERIFIER_URL=https://eth-bytecode-db.services.blockscout.com/
MICROSERVICE_SC_VERIFIER_TYPE=eth_bytecode_db
MICROSERVICE_ETH_BYTECODE_DB_INTERVAL_BETWEEN_LOOKUPS=10m
MICROSERVICE_ETH_BYTECODE_DB_MAX_LOOKUPS_CONCURRENCY=10
MICROSERVICE_VISUALIZE_SOL2UML_ENABLED=true
MICROSERVICE_VISUALIZE_SOL2UML_URL=http://visualizer:8050/
MICROSERVICE_SIG_PROVIDER_ENABLED=true
MICROSERVICE_SIG_PROVIDER_URL=http://sig-provider:8050/
MICROSERVICE_ACCOUNT_ABSTRACTION_ENABLED=false
MICROSERVICE_ACCOUNT_ABSTRACTION_URL=http://user-ops-indexer:8050/
DECODE_NOT_A_CONTRACT_CALLS=true
ACCOUNT_CLOAK_KEY=
ACCOUNT_ENABLED=false
ACCOUNT_REDIS_URL=redis://redis-db:6379
EIP_1559_ELASTICITY_MULTIPLIER=2
SOURCIFY_INTEGRATION_ENABLED=false
SOURCIFY_SERVER_URL=http://host.docker.internal:5555
SOURCIFY_REPO_URL=http://host.docker.internal:10000
SHOW_TENDERLY_LINK=false
TENDERLY_CHAIN_PATH=
NFT_MEDIA_HANDLER_ENABLED=false
NFT_MEDIA_HANDLER_REMOTE_DISPATCHER_NODE_MODE_ENABLED=false
RELEASE_NODE=producer@172.18.0.4
RELEASE_DISTRIBUTION=name
RELEASE_COOKIE=secret_cookie
INDEXER_DISABLE_PENDING_TRANSACTIONS_FETCHER=true
INDEXER_DISABLE_INTERNAL_TRANSACTIONS_FETCHER=true
INDEXER_CATCHUP_BLOCKS_BATCH_SIZE = 100
INDEXER_CATCHUP_BLOCKS_CONCURRENCY = 15
INDEXER_RECEIPTS_BATCH_SIZE = 500
INDEXER_MEMORY_LIMIT=4
INDEXER_DISABLE_EMPTY_BLOCKS_SANITIZER=true
INDEXER_DISABLE_BLOCK_REWARD_FETCHER =true

However indexing is still quite slow (takes around 6 hours) and the database is around 1,3 GB. The issue is the previewnet network has around 800k blocks currently and mainnet has 1000x this, so this means if we extrapolate the numbers that mainnet will take aroun 6000 hours to index and the DB will be around 1TB. Thats quite a lot, so I am seeking suggestions on how to improve the indexing time via configuration. Increasing the batch size results in a lot of additional requests and latency on our infrastructure, but if theres no other way we may go down this path.

[Mojmir-R]

Hey @konstantinabl, Mojmir from Blockscout's BD team here. Glad to hear that Hedera team is exploring using Blockscout. I have asked our support team to advise here, and in the meantime, it would be great to connect directly to learn more about your use case and how we could help outside of this indexing issue.

You can reach me at mojmir@blockscout.com or TG at Mojmir_R

[konstantinabl]

Hi there @Mojmir-R ,thank you for your fast response!
Will connect over email : )

[GimluCom]

Hi @konstantinabl, have you checked if there are any errors being output in the Blockscout backend?
Yes, increasing the batch_size, concurrency and pool_size may help with indexing speed. Make sure the indexer, database and RPC node have sufficient resources (CPU, RAM, IO, etc) to avoid one potentially acting as a bottleneck.
Keep in mind public RPC endpoints may be rate limited, which would severely affect indexing. It looks like the websocket endpoint you are currently using should be using the wss:// scheme.
When querying the public mainnet endpoint, it reports ~75M blocks which would be 100x of the previewnet.

[GimluCom]

Hi @konstantinabl, yes that would likely be a bottleneck. There are some minimum hardware requirements listed at https://docs.blockscout.com/setup/requirements/resource-requirements however in this case you will need something better than the minimum, however exact specs recommendation vary between each chain.

The token balance and coin balance fetchers indexes balances of addresses. One use of them is at this page https://eth.blockscout.com/address/0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045?tab=coin_balance_history

[konstantinabl]

@GimluCom Thank you for the response!
Thank you for the clarification about minimum requirements, I didnt see it, its useful.
So I guess once we deploy a VM with more ram and more vCPU indexing should be faster.
Two more questions, if you dont mind:

1. What is the POOL_SIZE for. I saw in the documentation its about DB connections, but is it DB connections of blockscout to the DB or something else :?
2. We will mainly use token transfer events and this endpoint https://docs.blockscout.com/devs/apis/rpc/account#get-list-of-tokens-owned-by-address. As per your answer, I understand coin balances fetcher is not needed for these?

[GimluCom]

@konstantinabl 1. Yes, that would be the number of connections Blockscout may use to the DB. 2. Correct, coin balance refers to the balance of the native coin.

[konstantinabl]

@GimluCom
Thank you and last thing that is not so clear for me. What exactly is it meant by accounts here in the docs for env variables
https://docs.blockscout.com/setup/env-variables/backend-env-variables#account-related ? And is redis only used for this functionality?

[GimluCom]

@konstantinabl Although slightly outdated, the account feature is described in https://docs.blockscout.com/using-blockscout/my-account
Redis can also be used for rate limiting.