fix: optimize address transactions query to prevent timeouts #2347

iamramtin · 2025-08-26T17:30:24Z

[WIP] Optimize getAddressTransactions query to eliminate timeouts

Status: Work in progress, seeking feedback on approach before finalizing

The getAddressTransactions endpoint times out for addresses with high transaction volumes, causing 500 errors for wallet apps and block explorers.

I've analyzed query performance using the latest database archive from archive.hiro.so/mainnet/stacks-blockchain-api-pg and identified bottlenecks in:

Late and repeated canonical filtering creating expensive nested loops
Separate count calculation requiring full table scans
Correlated subqueries using inefficient bitmap operations

Approach

Query Restructuring - Added getAddressTransactionsV2 function with:
- Early canonical filtering within UNION ALL branches
- Two-stage CTE processing (collect → limit → details)
- Deduping first removes the costly Aggregate
- Better parallel worker utilization across UNION ALL branches for the expensive principal transaction scan
Database Indexes - Migration script creates optimized indexes for:
- Canonical transaction filtering with built-in ordering
- Event table subqueries enabling index-only scans

Performance Results

Query optimization alone: ~4x speedup (20.8s → ~5s)
With database indexes: Additional 2x improvement (~5s → 2.4s)
Total improvement: 8.7x faster execution

Testing Environment

Local PostgreSQL instance with production data from archive.hiro.so
Tested against a medium volume address SPDSMZGNMVKK48BHX45VKETX4VF8264NV0C9J692
Tested against problematic high-volume address SP3SBQ9PZEMBNBAWTR7FRPE3XK0EFW9JWVX4G80S2 (65k+ transactions) → original query exceeds 2.6 minutes when run locally on M3 MacBook Air, updated query takes around 24 seconds in comparison
Verified backward compatibility and identical result sets

Next Steps

Review query optimization approach
Validate index strategy for production deployment
Plan migration timeline and rollback procedures

Signed-off-by: Ramtin Mesgari 26694963+iamramtin@users.noreply.github.com

iamramtin · 2025-08-27T11:08:31Z

Another potential optimization is to materialize the address_txs CTE into its own indexed table. Precomputing and indexing the results could add another layer of improvement, it would reduce the execution time as it would avoid repeated heavy joins and the indexing would help

This would trade off some storage and update complexity for faster reads, but my hunch says it's worthwhile given the scale and volume of data

iamramtin · 2025-08-27T11:25:48Z

I ran EXPLAIN (ANALYZE, BUFFERS, TIMING OFF) on the original query and the optimized query, below is a a copy of the results on the third run of each analysis

You can see that the planning time is slower on the optimized version, however, the execution time is around 6x faster.

Original Query

Limit  (cost=1026458.77..1026509.82 rows=1 width=2457) (actual time=14029.501..14084.731 rows=50 loops=1)
  CTE address_txs
    ->  Unique  (cost=30386.56..31219.25 rows=83269 width=96) (actual time=71.446..101.970 rows=65659 loops=1)
          ->  Sort  (cost=30386.56..30594.73 rows=83269 width=96) (actual time=71.444..84.683 rows=65663 loops=1)
                Sort Key: principal_stx_txs.tx_id, principal_stx_txs.index_block_hash, principal_stx_txs.microblock_hash
                Sort Method: external merge  Disk: 4960kB
                ->  Append  (cost=0.69..19311.19 rows=83269 width=96) (actual time=0.828..27.588 rows=65663 loops=1)
                      ->  Index Only Scan using unique_principal_tx_id_index_block_hash_microblock_hash on principal_stx_txs  (cost=0.69..7365.82 rows=80522 width=70) (actual time=0.826..19.745 rows=65659 loops=1)
                            Index Cond: (principal = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                            Heap Fetches: 0
                      ->  Bitmap Heap Scan on stx_events  (cost=18.71..4649.56 rows=1198 width=69) (actual time=1.991..2.112 rows=4 loops=1)
                            Recheck Cond: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                            Heap Blocks: exact=4
                            ->  BitmapOr  (cost=18.71..18.71 rows=1198 width=0) (actual time=1.848..1.848 rows=0 loops=1)
                                  ->  Bitmap Index Scan on stx_events_sender_index  (cost=0.00..9.11 rows=606 width=0) (actual time=1.071..1.071 rows=0 loops=1)
                                        Index Cond: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                  ->  Bitmap Index Scan on stx_events_recipient_index  (cost=0.00..9.01 rows=593 width=0) (actual time=0.776..0.776 rows=4 loops=1)
                                        Index Cond: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                      ->  Bitmap Heap Scan on ft_events  (cost=19.80..5189.44 rows=1333 width=68) (actual time=1.098..1.098 rows=0 loops=1)
                            Recheck Cond: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                            ->  BitmapOr  (cost=19.80..19.80 rows=1333 width=0) (actual time=1.097..1.097 rows=0 loops=1)
                                  ->  Bitmap Index Scan on ft_events_sender_index  (cost=0.00..11.53 rows=929 width=0) (actual time=0.561..0.561 rows=0 loops=1)
                                        Index Cond: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                  ->  Bitmap Index Scan on ft_events_recipient_index  (cost=0.00..7.60 rows=405 width=0) (actual time=0.536..0.536 rows=0 loops=1)
                                        Index Cond: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                      ->  Bitmap Heap Scan on nft_events  (cost=10.71..857.33 rows=216 width=70) (actual time=1.468..1.468 rows=0 loops=1)
                            Recheck Cond: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                            ->  BitmapOr  (cost=10.71..10.71 rows=216 width=0) (actual time=1.467..1.467 rows=0 loops=1)
                                  ->  Bitmap Index Scan on nft_events_sender_index  (cost=0.00..5.59 rows=154 width=0) (actual time=0.598..0.598 rows=0 loops=1)
                                        Index Cond: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                  ->  Bitmap Index Scan on nft_events_recipient_index  (cost=0.00..5.01 rows=61 width=0) (actual time=0.869..0.869 rows=0 loops=1)
                                        Index Cond: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
  InitPlan 13 (returns $37)
    ->  Aggregate  (cost=334393.75..334393.76 rows=1 width=4) (actual time=4327.488..4327.489 rows=1 loops=1)
          ->  Nested Loop  (cost=0.56..334393.75 rows=1 width=0) (actual time=0.014..4321.034 rows=65654 loops=1)
                ->  CTE Scan on address_txs address_txs_1  (cost=0.00..1665.38 rows=83269 width=96) (actual time=0.001..48.594 rows=65659 loops=1)
                ->  Index Only Scan using idx_txs_canonical_optimized on txs txs_1  (cost=0.56..4.00 rows=1 width=70) (actual time=0.065..0.065 rows=1 loops=65659)
                      Index Cond: ((tx_id = address_txs_1.tx_id) AND (index_block_hash = address_txs_1.index_block_hash) AND (microblock_hash = address_txs_1.microblock_hash))
                      Heap Fetches: 5
  ->  Result  (cost=660845.76..660896.80 rows=1 width=2457) (actual time=14029.499..14084.723 rows=50 loops=1)
        ->  Sort  (cost=660845.76..660845.76 rows=1 width=2453) (actual time=14027.722..14027.729 rows=50 loops=1)
              Sort Key: txs.block_height DESC, txs.microblock_sequence DESC, txs.tx_index DESC
              Sort Method: top-N heapsort  Memory: 110kB
              ->  Nested Loop  (cost=0.56..660845.75 rows=1 width=2453) (actual time=4399.648..13992.493 rows=65654 loops=1)
                    ->  CTE Scan on address_txs  (cost=0.00..1665.38 rows=83269 width=96) (actual time=71.448..82.483 rows=65659 loops=1)
                    ->  Index Scan using idx_txs_canonical_optimized on txs  (cost=0.56..7.92 rows=1 width=2353) (actual time=0.145..0.145 rows=1 loops=65659)
                          Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash))
        SubPlan 2
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=32) (actual time=0.358..0.358 rows=1 loops=50)
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_1  (cost=0.56..4.58 rows=1 width=8) (actual time=0.358..0.358 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash))
                      Filter: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                      Heap Fetches: 0
        SubPlan 3
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=50)
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_2  (cost=0.56..4.58 rows=1 width=8) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash))
                      Filter: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                      Heap Fetches: 0
        SubPlan 4
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=50)
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_3  (cost=0.56..4.59 rows=1 width=0) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 1))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 5
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=50)
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_4  (cost=0.56..4.59 rows=1 width=0) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 2))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 6
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=50)
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_5  (cost=0.56..4.59 rows=1 width=0) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 3))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 7
          ->  Aggregate  (cost=4.72..4.73 rows=1 width=4) (actual time=0.391..0.391 rows=1 loops=50)
                ->  Index Only Scan using idx_ft_events_subquery_optimized on ft_events ft_events_1  (cost=0.69..4.72 rows=1 width=0) (actual time=0.391..0.391 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 1))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 8
          ->  Aggregate  (cost=4.72..4.73 rows=1 width=4) (actual time=0.006..0.006 rows=1 loops=50)
                ->  Index Only Scan using idx_ft_events_subquery_optimized on ft_events ft_events_2  (cost=0.69..4.72 rows=1 width=0) (actual time=0.006..0.006 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 2))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Rows Removed by Filter: 1
                      Heap Fetches: 0
        SubPlan 9
          ->  Aggregate  (cost=4.72..4.73 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=50)
                ->  Index Only Scan using idx_ft_events_subquery_optimized on ft_events ft_events_3  (cost=0.69..4.72 rows=1 width=0) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 3))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 10
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual time=0.372..0.372 rows=1 loops=50)
                ->  Index Only Scan using idx_nft_events_subquery_optimized on nft_events nft_events_1  (cost=0.56..4.59 rows=1 width=0) (actual time=0.372..0.372 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 1))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 11
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=50)
                ->  Index Only Scan using idx_nft_events_subquery_optimized on nft_events nft_events_2  (cost=0.56..4.59 rows=1 width=0) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 2))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
        SubPlan 12
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=50)
                ->  Index Only Scan using idx_nft_events_subquery_optimized on nft_events nft_events_3  (cost=0.56..4.59 rows=1 width=0) (actual time=0.001..0.001 rows=0 loops=50)
                      Index Cond: ((tx_id = address_txs.tx_id) AND (index_block_hash = address_txs.index_block_hash) AND (microblock_hash = address_txs.microblock_hash) AND (asset_event_type_id = 3))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
Planning Time: 3.165 ms
Execution Time: 14085.615 ms

Optimized Query

Sort  (cost=163765.09..163765.10 rows=1 width=2457) (actual rows=50 loops=1)
  Sort Key: txs.block_height DESC, txs.microblock_sequence DESC, txs.tx_index DESC
  Sort Method: quicksort  Memory: 61kB
  Buffers: shared hit=300312 read=60945, temp read=1414 written=1416
  CTE deduped_txs
    ->  Unique  (cost=163670.70..163670.78 rows=5 width=80) (actual rows=65654 loops=1)
          Buffers: shared hit=298262 read=60394, temp read=708 written=710
          ->  Sort  (cost=163670.70..163670.71 rows=5 width=80) (actual rows=65658 loops=1)
                Sort Key: t.tx_id, t.index_block_hash, t.microblock_hash, t.block_height, t.microblock_sequence, t.tx_index
                Sort Method: external merge  Disk: 5664kB
                Buffers: shared hit=298262 read=60394, temp read=708 written=710
                ->  Gather  (cost=1011.27..163670.64 rows=5 width=80) (actual rows=65658 loops=1)
                      Workers Planned: 2
                      Workers Launched: 2
                      Buffers: shared hit=298262 read=60394
                      ->  Parallel Append  (cost=11.27..162670.14 rows=4 width=80) (actual rows=21886 loops=3)
                            Buffers: shared hit=298262 read=60394
                            ->  Nested Loop  (cost=11.27..1851.69 rows=1 width=80) (actual rows=0 loops=1)
                                  Buffers: shared read=9
                                  ->  Bitmap Heap Scan on nft_events ne  (cost=10.71..857.33 rows=216 width=70) (actual rows=0 loops=1)
                                        Recheck Cond: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                                        Buffers: shared read=9
                                        ->  BitmapOr  (cost=10.71..10.71 rows=216 width=0) (actual rows=0 loops=1)
                                              Buffers: shared read=9
                                              ->  Bitmap Index Scan on nft_events_sender_index  (cost=0.00..5.59 rows=154 width=0) (actual rows=0 loops=1)
                                                    Index Cond: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                                    Buffers: shared read=4
                                              ->  Bitmap Index Scan on nft_events_recipient_index  (cost=0.00..5.01 rows=61 width=0) (actual rows=0 loops=1)
                                                    Index Cond: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                                    Buffers: shared read=5
                                  ->  Index Only Scan using idx_txs_canonical_optimized on txs t  (cost=0.56..4.60 rows=1 width=80) (never executed)
                                        Index Cond: ((tx_id = ne.tx_id) AND (index_block_hash = ne.index_block_hash) AND (microblock_hash = ne.microblock_hash))
                                        Heap Fetches: 0
                            ->  Nested Loop  (cost=20.36..8771.14 rows=1 width=80) (actual rows=0 loops=1)
                                  Buffers: shared read=10
                                  ->  Parallel Bitmap Heap Scan on ft_events fe  (cost=19.80..5181.21 rows=784 width=68) (actual rows=0 loops=1)
                                        Recheck Cond: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                                        Buffers: shared read=10
                                        ->  BitmapOr  (cost=19.80..19.80 rows=1333 width=0) (actual rows=0 loops=1)
                                              Buffers: shared read=10
                                              ->  Bitmap Index Scan on ft_events_sender_index  (cost=0.00..11.53 rows=929 width=0) (actual rows=0 loops=1)
                                                    Index Cond: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                                    Buffers: shared read=5
                                              ->  Bitmap Index Scan on ft_events_recipient_index  (cost=0.00..7.60 rows=405 width=0) (actual rows=0 loops=1)
                                                    Index Cond: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                                    Buffers: shared read=5
                                  ->  Index Only Scan using idx_txs_canonical_optimized on txs t_1  (cost=0.56..4.58 rows=1 width=80) (never executed)
                                        Index Cond: ((tx_id = fe.tx_id) AND (index_block_hash = fe.index_block_hash) AND (microblock_hash = fe.microblock_hash))
                                        Heap Fetches: 0
                            ->  Nested Loop  (cost=19.27..7869.88 rows=1 width=80) (actual rows=4 loops=1)
                                  Buffers: shared hit=16 read=17
                                  ->  Parallel Bitmap Heap Scan on stx_events se  (cost=18.71..4642.17 rows=705 width=69) (actual rows=4 loops=1)
                                        Recheck Cond: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                                        Buffers: shared hit=1 read=13
                                        ->  BitmapOr  (cost=18.71..18.71 rows=1198 width=0) (actual rows=0 loops=1)
                                              Buffers: shared read=10
                                              ->  Bitmap Index Scan on stx_events_sender_index  (cost=0.00..9.11 rows=606 width=0) (actual rows=0 loops=1)
                                                    Index Cond: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                                    Buffers: shared read=5
                                              ->  Bitmap Index Scan on stx_events_recipient_index  (cost=0.00..9.01 rows=593 width=0) (actual rows=4 loops=1)
                                                    Index Cond: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                                    Buffers: shared read=5
                                  ->  Index Only Scan using idx_txs_canonical_optimized on txs t_2  (cost=0.56..4.58 rows=1 width=80) (actual rows=1 loops=4)
                                        Index Cond: ((tx_id = se.tx_id) AND (index_block_hash = se.index_block_hash) AND (microblock_hash = se.microblock_hash))
                                        Heap Fetches: 0
                                        Buffers: shared hit=15 read=4
                            ->  Nested Loop  (cost=1.25..144177.37 rows=1 width=80) (actual rows=21885 loops=3)
                                  Buffers: shared hit=298246 read=60358
                                  ->  Parallel Index Only Scan using unique_principal_tx_id_index_block_hash_microblock_hash on principal_stx_txs p  (cost=0.69..6896.11 rows=33551 width=70) (actual rows=21886 loops=3)
                                        Index Cond: (principal = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                                        Heap Fetches: 0
                                        Buffers: shared hit=41674 read=1119
                                  ->  Index Only Scan using idx_txs_canonical_optimized on txs t_3  (cost=0.56..4.09 rows=1 width=80) (actual rows=1 loops=65659)
                                        Index Cond: ((tx_id = p.tx_id) AND (index_block_hash = p.index_block_hash) AND (microblock_hash = p.microblock_hash))
                                        Heap Fetches: 5
                                        Buffers: shared hit=256572 read=59239
  InitPlan 13 (returns $47)
    ->  Aggregate  (cost=0.11..0.12 rows=1 width=4) (actual rows=1 loops=1)
          Buffers: temp read=706 written=1
          ->  CTE Scan on deduped_txs deduped_txs_1  (cost=0.00..0.10 rows=5 width=0) (actual rows=65654 loops=1)
                Buffers: temp read=706 written=1
  ->  Nested Loop  (cost=0.72..94.18 rows=1 width=2457) (actual rows=50 loops=1)
        Buffers: shared hit=300312 read=60945, temp read=1414 written=1416
        ->  Limit  (cost=0.16..0.17 rows=5 width=106) (actual rows=50 loops=1)
              Buffers: shared hit=298262 read=60394, temp read=708 written=1415
              ->  Sort  (cost=0.16..0.17 rows=5 width=106) (actual rows=50 loops=1)
                    Sort Key: deduped_txs.block_height DESC, deduped_txs.microblock_sequence DESC, deduped_txs.tx_index DESC
                    Sort Method: top-N heapsort  Memory: 36kB
                    Buffers: shared hit=298262 read=60394, temp read=708 written=1415
                    ->  CTE Scan on deduped_txs  (cost=0.00..0.10 rows=5 width=106) (actual rows=65654 loops=1)
                          Buffers: shared hit=298262 read=60394, temp read=708 written=1415
        ->  Index Scan using unique_tx_id_index_block_hash_microblock_hash on txs  (cost=0.56..8.58 rows=1 width=2353) (actual rows=1 loops=50)
              Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash))
              Buffers: shared hit=89 read=161
        SubPlan 2
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=32) (actual rows=1 loops=50)
                Buffers: shared hit=71 read=129
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events  (cost=0.56..4.58 rows=1 width=8) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash))
                      Filter: (sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                      Heap Fetches: 0
                      Buffers: shared hit=71 read=129
        SubPlan 3
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=32) (actual rows=1 loops=50)
                Buffers: shared hit=200
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_1  (cost=0.56..4.58 rows=1 width=8) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash))
                      Filter: (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text)
                      Heap Fetches: 0
                      Buffers: shared hit=200
        SubPlan 4
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=200
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_2  (cost=0.56..4.59 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 1))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=200
        SubPlan 5
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=200
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_3  (cost=0.56..4.59 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 2))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=200
        SubPlan 6
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=200
                ->  Index Only Scan using idx_stx_events_subquery_optimized on stx_events stx_events_4  (cost=0.56..4.59 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 3))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=200
        SubPlan 7
          ->  Aggregate  (cost=4.72..4.73 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=111 read=139
                ->  Index Only Scan using idx_ft_events_subquery_optimized on ft_events  (cost=0.69..4.72 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 1))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=111 read=139
        SubPlan 8
          ->  Aggregate  (cost=4.72..4.73 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=250 read=1
                ->  Index Only Scan using idx_ft_events_subquery_optimized on ft_events ft_events_1  (cost=0.69..4.72 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 2))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Rows Removed by Filter: 1
                      Heap Fetches: 0
                      Buffers: shared hit=250 read=1
        SubPlan 9
          ->  Aggregate  (cost=4.72..4.73 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=250
                ->  Index Only Scan using idx_ft_events_subquery_optimized on ft_events ft_events_2  (cost=0.69..4.72 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 3))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=250
        SubPlan 10
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=79 read=121
                ->  Index Only Scan using idx_nft_events_subquery_optimized on nft_events  (cost=0.56..4.59 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 1))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=79 read=121
        SubPlan 11
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=200
                ->  Index Only Scan using idx_nft_events_subquery_optimized on nft_events nft_events_1  (cost=0.56..4.59 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 2))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=200
        SubPlan 12
          ->  Aggregate  (cost=4.59..4.60 rows=1 width=4) (actual rows=1 loops=50)
                Buffers: shared hit=200
                ->  Index Only Scan using idx_nft_events_subquery_optimized on nft_events nft_events_2  (cost=0.56..4.59 rows=1 width=0) (actual rows=0 loops=50)
                      Index Cond: ((tx_id = deduped_txs.tx_id) AND (index_block_hash = deduped_txs.index_block_hash) AND (microblock_hash = deduped_txs.microblock_hash) AND (asset_event_type_id = 3))
                      Filter: ((sender = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text) OR (recipient = 'SP21EK0KSQG7HEHBGCVRJGPGFMV8SCA2B85X01DK2'::text))
                      Heap Fetches: 0
                      Buffers: shared hit=200
Planning:
  Buffers: shared hit=116 read=90
Planning Time: 26.490 ms
Execution Time: 2221.355 ms

- Restructure query with early canonical filtering and optimized CTEs - Add database indexes for correlated subqueries - Add optimized function and migration script Signed-off-by: Ramtin Mesgari <26694963+iamramtin@users.noreply.github.com>

iamramtin mentioned this pull request Aug 26, 2025

Timeout for account with many transactions #2337

Open

iamramtin force-pushed the fix/address-transactions-timeout branch from 4da659b to 4db2943 Compare August 27, 2025 11:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: optimize address transactions query to prevent timeouts #2347

fix: optimize address transactions query to prevent timeouts #2347

Uh oh!

iamramtin commented Aug 26, 2025 •

edited

Loading

Uh oh!

iamramtin commented Aug 27, 2025

Uh oh!

iamramtin commented Aug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fix: optimize address transactions query to prevent timeouts #2347

Are you sure you want to change the base?

fix: optimize address transactions query to prevent timeouts #2347

Uh oh!

Conversation

iamramtin commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

[WIP] Optimize getAddressTransactions query to eliminate timeouts

Approach

Performance Results

Testing Environment

Next Steps

Uh oh!

iamramtin commented Aug 27, 2025

Uh oh!

iamramtin commented Aug 27, 2025

Original Query

Optimized Query

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

iamramtin commented Aug 26, 2025 •

edited

Loading