[Neutron] Add PodMonitor and replace uwsgi-datadog exporter with uWSG…

[sven-rosenzweig]

…I stats exporter

This change enables the uWSGI stats server, allowing metrics to be fetched via HTTP. It replaces the deprecated uwsgi-datadog exporter, which was based on an unmaintained open-source project.

As before, each uWSGI worker thread now exposes its own metrics, which can be aggregated without any loss of functionality.

The new uWSGI exporter is forked from https://github.com/timonwong/uwsgi_exporter and written in Go. It scrapes the uWSGI stats endpoint when the /metrics endpoint is called.

A PodMonitor resource has been added to configure Prometheus to scrape these new endpoints.

[sven-rosenzweig]

Sample uwsgi stat server response:

curl http://localhost:9104
{
	"version":"2.0.30",
	"listen_queue":0,
	"listen_queue_errors":0,
	"signal_queue":0,
	"load":0,
	"pid":7,
	"uid":42435,
	"gid":42435,
	"cwd":"/",
	"locks":[
		{
			"user 0":0
		},
		{
			"signal":0
		},
		{
			"filemon":0
		},
		{
			"timer":0
		},
		{
			"rbtimer":0
		},
		{
			"cron":0
		},
		{
			"thunder":21
		},
		{
			"rpc":0
		},
		{
			"snmp":0
		}
	],
	"sockets":[
		{
			"name":"127.0.0.1:42603",
			"proto":"uwsgi",
			"queue":0,
			"max_queue":4096,
			"shared":0,
			"can_offload":0
		}
	],
	"workers":[
		{
			"id":1,
			"pid":21,
			"accepting":1,
			"requests":1,
			"delta_requests":1,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":248557568,
			"vsz":462704640,
			"running_time":1308772,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":1247,
			"avg_rt":654386,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":1,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":2,
			"pid":22,
			"accepting":1,
			"requests":1,
			"delta_requests":1,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":250613760,
			"vsz":464826368,
			"running_time":1256377,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":1016,
			"avg_rt":628188,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":1,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":3,
			"pid":23,
			"accepting":1,
			"requests":1,
			"delta_requests":1,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":250396672,
			"vsz":463810560,
			"running_time":786147,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":2457,
			"avg_rt":393073,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":1,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":4,
			"pid":24,
			"accepting":1,
			"requests":39,
			"delta_requests":39,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":247554048,
			"vsz":461385728,
			"running_time":10045721,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":520698,
			"avg_rt":111881,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":28,
					"requests":39,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":5,
			"pid":25,
			"accepting":1,
			"requests":2,
			"delta_requests":2,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":248717312,
			"vsz":462794752,
			"running_time":1200495,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":1379,
			"avg_rt":454091,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":2,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":6,
			"pid":26,
			"accepting":1,
			"requests":2,
			"delta_requests":2,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":251895808,
			"vsz":465883136,
			"running_time":574821,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":732,
			"avg_rt":144171,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":2,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":7,
			"pid":27,
			"accepting":1,
			"requests":3,
			"delta_requests":3,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":251289600,
			"vsz":465858560,
			"running_time":1344762,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":3295,
			"avg_rt":425011,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":3,
					"exceptions":0,
					"chdir":""
				}
			]
		},
		{
			"id":8,
			"pid":28,
			"accepting":1,
			"requests":1,
			"delta_requests":1,
			"exceptions":0,
			"harakiri_count":0,
			"signals":0,
			"signal_queue":0,
			"status":"idle",
			"rss":255582208,
			"vsz":469594112,
			"running_time":1280284,
			"last_spawn":1761033782,
			"respawn_count":1,
			"tx":1069,
			"avg_rt":640142,
			"apps":[
				{
					"id":0,
					"modifier1":0,
					"mountpoint":"",
					"startup_time":39,
					"requests":1,
					"exceptions":0,
					"chdir":""
				}
			]
		}
	]
}

[felix-kaestner]

The latest tag would be 20251021090402. Also, should we make this a value (e.g. imageVersionUSWSGIExporter), such that we can bump the image via values instead of having a change here in the charts?

[mutax]

I discussed out of band that it might make sense to have the change behind a feature flag, so we can do a staged rollout or disable the exporter in case of performance issues.

[sven-rosenzweig]

Before we merge this PR:

• Update CI pipeline: https://github.wdf.sap.corp/cc/secrets/pull/18617
• Change to Makefile Maker: Add go-makefile-maker support uwsgi-exporter#10
• Merge all renovate pull requests