Fixes #1246 and #1756 - Removed Hall of Fame task

[bwbroersma]

• Fixes Improve Hall of Fame task #1246 and Optimize run_stats_queries #1756

Todo:

• Migration

• New: use of django-pgtrigger (note this does add some overhead/boilerplate to support the pgtrigger.ignore decorator to dynamically Ignoring Execution of the triggers)

• Rewrite update.py to use Fame model

• New: rewrite update.py + Redis cache + Optimize run_stats_queries #1756 to interface/views/__init__.py
  Already cached via @simple_cache_page in Redis

• New: check if instead of requesting once from the DB and sorting in application space the DB can be requested 3 times with the sorted output, so the iterator() can be used without sorting in Python, this would save memory (and probably faster to sort at the DB, since there it's also possible to add indices). Check if permalink formatting can be done generic and done at a point to minimize string copying.

• Optional: put slow-worker in some sort of compose batch profile (since the 2-5GB memory container won't be needed in Single Test mode), or move remaining slow-worker tasks (registering and generating async results) to another container

• Find a way to remove the SELECT "checks_fame"."id" (while .only("domain")) for Index Only Scan performance for HoF queries (fixed by using domain as primary key in django, note this does create an extra "checks_fame_domain_HASHHASH_like" btree (domain varchar_pattern_ops) index)

• Test

  • Enable PostgreSQL query log (make sure the ForeignKey doesn't actually fetch the related report model)

  docker compose --env-file docker/develop.env exec postgres psql -Uinternetnl internetnl_db1 -c "ALTER SYSTEM SET log_statement TO 'all';" -c "SELECT pg_reload_conf();"
  docker compose --env-file=docker/defaults.env --env-file=docker/develop.env logs postgres -f

  Seems okay:

  postgres-1  | 2025-05-19 21:55:23.082 UTC [1603] LOG:  statement: BEGIN
  postgres-1  | 2025-05-19 21:55:23.082 UTC [1603] LOG:  statement: DECLARE "_django_curs_140058792129024_sync_1" NO SCROLL CURSOR WITHOUT HOLD FOR SELECT "checks_fame"."id", "checks_fame"."domain", "checks_fame"."site_report_id", "checks_fame"."site_report_timestamp", "checks_fame"."mail_report_id", "checks_fame"."mail_report_timestamp" FROM "checks_fame"
  postgres-1  | 2025-05-19 21:55:23.084 UTC [1603] LOG:  statement: FETCH FORWARD 2000 FROM "_django_curs_140058792129024_sync_1"
  postgres-1  | 2025-05-19 21:55:23.085 UTC [1603] LOG:  statement: FETCH FORWARD 2000 FROM "_django_curs_140058792129024_sync_1"
  postgres-1  | 2025-05-19 21:55:23.085 UTC [1603] LOG:  statement: CLOSE "_django_curs_140058792129024_sync_1"
  postgres-1  | 2025-05-19 21:55:23.087 UTC [1603] LOG:  statement: COMMIT
  

  • Stress test populate dataset query with 'real' dataset (that is: checks_{domain,mail}testreport (id, timestamp, domain, score) dump, or anonymized set)

• Use single MERGE instead of DELETE + UPDATE:

• Use OR UPDATE OF score ON trigger

Note for the initial population: maybe PARTITION BY domain ORDER BY id DESC should be changed to PARTITION BY domain ORDER BY timestamp DESC, although the change will probably be minimal, but the current situation is ordered by timestamp.

[bwbroersma]

Testing concluded:

• If the (page) cache is outdated, the view should run spiffy, it does not, this is mainly due to the slow counts.
• Even after adding an extra index on the domain field, the resulting Index Only Scans (that can run as a Parallel Scan by PostgreSQL) will run 'slow' (as in seconds) for tables with multi M records (see PostgreSQL count performance reference), these all run in serial in the (new) code (that is by Django, not talking about the Parallel PostgreSQL part), although running in parallel with async is probably not the hack desired (see async Django since it might require raw SQL).

Steps forward:

• Create index on domain for test reports
• (Minor improvement of the trigger) Add NEW.Score != OLD.Score to Trigger (and make sure if both old and new are NULL the query of the trigger also does not execute, e.g. on registrar updates).
• Create some distinct_count model/table, with initial count fill for all 3 models, with total & fame + champions count
• Triggers on test reports to update total, triggers on fame to update fame:
  • Test report: on create, was there a earlier report, if not: update total +1, on delete/update domain (not sure if this is a valid case: check if there is still a report with this domain, if not: total -1.
  • Fame: on insert / update / delete: and for both site/mail: if old is null, fame +1, if new is null, fame -1
• Redo-testing with double the batch count (since the HoF SQL is not optional and 'enabled by default')
  Numbers are 11.5-14M per table: 11 520 239 and 13 791 292
• Ideally prevent edits from Django / PostgreSQL to fame and counts