Need help with understanding metrics in InfluxDB

[psihonavt]

Hello,

I inherited a monitoring system for our PostgreSQL DB which is an official pgwatch2 docker image (pgwatch2 + influxdb + grafana). There is an alarm in Grafana for an abnormally high number of inserts in a period of time (15m):

SELECT non_negative_derivative(mean("tup_inserted"), 1m) FROM "db_stats" WHERE ("dbname" = 'nimble') AND $timeFilter GROUP BY time(15m) fill(null)

If to slightly modify this query and run it in the influxdb-cli:

> select "time", non_negative_derivative(mean("tup_inserted"), 1m) from "db_stats" where "dbname" = 'nimble' and time >= '2021-09-28T16:31:00Z' and time <= '2021-09-28T16:36:00Z' group by time(1m) fill(null) order by time asc;
name: db_stats
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 2384
2021-09-28T16:32:00Z 16294
2021-09-28T16:33:00Z 1998713
2021-09-28T16:34:00Z 924
2021-09-28T16:35:00Z 1606

And indeed, there is a huge spike in the number of inserts.
I wanted to understand what table had the most inserts, and I ran the next query:

> select non_negative_derivative(mean("n_tup_ins"), 1m) from "table_stats" where time >= '2021-09-28T16:31:00Z' and time <= '2021-09-28T16:36:00Z' group by table_name, time(1m) fill(null) order by time asc;
name: table_stats
tags: table_name=alembic_version
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=assigned_partners
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=calls_trashbin_contacts
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=companies_cloud_usage
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contact_files
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 4
2021-09-28T16:32:00Z 3
2021-09-28T16:33:00Z 2
2021-09-28T16:34:00Z 4
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts__addresses
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 1
2021-09-28T16:32:00Z 1
2021-09-28T16:33:00Z 3
2021-09-28T16:34:00Z 6
2021-09-28T16:35:00Z 4

name: table_stats
tags: table_name=contacts__aggregated
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 4
2021-09-28T16:32:00Z 3
2021-09-28T16:33:00Z 2
2021-09-28T16:34:00Z 4
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts__aggregated_only_ft
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 4
2021-09-28T16:32:00Z 3
2021-09-28T16:33:00Z 2
2021-09-28T16:34:00Z 4
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts__autofilled_search_fields
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts__employments
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 4
2021-09-28T16:33:00Z 5
2021-09-28T16:34:00Z 6
2021-09-28T16:35:00Z 5

name: table_stats
tags: table_name=contacts__facets_counters
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts__timestamps
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_avatars
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 2
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 1
2021-09-28T16:34:00Z 3
2021-09-28T16:35:00Z 1

name: table_stats
tags: table_name=contacts_count
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_enrichment_process_state
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_available_actions
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_choices
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_groups
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_kinds
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_tabs
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_tabs_available_actions
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_fields_tabs_types
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_import_status
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_importance
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 4
2021-09-28T16:32:00Z 1
2021-09-28T16:33:00Z 4
2021-09-28T16:34:00Z 3
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts_last_contacted
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 11
2021-09-28T16:32:00Z 23
2021-09-28T16:33:00Z 11
2021-09-28T16:34:00Z 10
2021-09-28T16:35:00Z 7

name: table_stats
tags: table_name=contacts_link_types
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_links
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 1
2021-09-28T16:33:00Z 1
2021-09-28T16:34:00Z 3
2021-09-28T16:35:00Z 1

name: table_stats
tags: table_name=contacts_links_many_to_many
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 2
2021-09-28T16:33:00Z 2
2021-09-28T16:34:00Z 3
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts_notes
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 2
2021-09-28T16:32:00Z 4
2021-09-28T16:33:00Z 3
2021-09-28T16:34:00Z 4
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts_notes_associations
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 2
2021-09-28T16:32:00Z 9
2021-09-28T16:33:00Z 3
2021-09-28T16:34:00Z 4
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts_reminders
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 1
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_reminders__trigger_ts
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 1
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_suggestions
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 4
2021-09-28T16:32:00Z 3
2021-09-28T16:33:00Z 2
2021-09-28T16:34:00Z 4
2021-09-28T16:35:00Z 2

name: table_stats
tags: table_name=contacts_tags
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 2
2021-09-28T16:32:00Z 2
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=contacts_tags_associations
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 623
2021-09-28T16:32:00Z 535
2021-09-28T16:33:00Z 22
2021-09-28T16:34:00Z 17
2021-09-28T16:35:00Z 46

name: table_stats
tags: table_name=contacts_trashbin
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 2
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 1
2021-09-28T16:34:00Z 3
2021-09-28T16:35:00Z 3

name: table_stats
tags: table_name=contacts_trashbin_avatars
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 1
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 1

name: table_stats
tags: table_name=contacts_values
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 26
2021-09-28T16:32:00Z 21
2021-09-28T16:33:00Z 23
2021-09-28T16:34:00Z 39
2021-09-28T16:35:00Z 33

name: table_stats
tags: table_name=counters_contact_files_size
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=deal_import_statuses
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=deals_trashbin_contacts
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=events_meetings
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=freetext_feedbacks
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=leads_pipeline_log
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 3
2021-09-28T16:32:00Z 1
2021-09-28T16:33:00Z 1
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=leads_pipeline_lost_reasons
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=leads_pipeline_stage_roles
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=leads_pipeline_stages
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=leads_pipelines
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=leads_pipelines_templates
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=partners
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=posts_import_statuses
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=promo_packages
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=promo_promotions
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=saml_relations
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=schema_import_statuses
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=sync_configurations
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=tasks_import_statuses
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=users_admin_groups
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=users_groups_to_users_association
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=users_permission_groups
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=users_permissions
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=web_risk_hashes_malware
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=web_risk_hashes_social_engineering
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 399
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=web_risk_hashes_unwanted_software
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 1
2021-09-28T16:33:00Z 0
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

name: table_stats
tags: table_name=web_risk_sync
time                 non_negative_derivative
----                 -----------------------
2021-09-28T16:31:00Z 0
2021-09-28T16:32:00Z 0
2021-09-28T16:33:00Z 1
2021-09-28T16:34:00Z 0
2021-09-28T16:35:00Z 0

As you can see, not a single table has a spike in the number of inserts compared to the number in the "db_stats" series. I want to understand when pgwatch adds the tup_inserted metric to the db_stats and why the number is so vastly different from the number of inserts on separate tables. Can it be some system tables or triggers? We do have triggers but they modify the same tables you can see ^^.

Thanks.