Measure block processing time with a separate service #179
Description
We need to continuously measure the SLO of the form "how many blocks were processed under 500ms"? By "processed" we mean the time between the block was first read from the RPC provider to the time it became available to portal users.
Right now the portal is making HTTP requests to the data service to find out the first of those two timestamps. Which complicates both components unnecessarily.
The idea is to develop a new simple service that
- Subscribes to one or multiple RPC nodes for block emission events and uses the first time of seeing the block as "block reference timestamp", keeping those values for the last N blocks in memory
- Subscribes to the specified service (which can be the portal or the hotblocks service because they have similar API) and emulates the user's behavior of staying at the head. Once it receives the block, it calculates the processing time for this block compared to its "reference timestamp".
- Exposes the results as prometheus metrics.
The first part will require separate implementations for different kinds of chains — for EVM there is websocket subscription and for Solana there is Geyser plugin to fetch block headers.