Feat: Allow label dropping/relabeling within server groups

[Wing924]

Problem/Motivation

A common and recommended pattern for setting up Prometheus High Availability (HA) involves configuring external_labels on each Prometheus instance to uniquely identify them.
This is crucial for distinguishing data sources, especially when using tools like Thanos or Cortex for long-term storage and global query views.

Example Prometheus Configuration with external_labels:

# prometheus-instance-A.yml
global:
  external_labels:
    cluster: my-prod-cluster
    prometheus_replica: prometheus-a # Unique identifier for this instance

# prometheus-instance-B.yml (identical scrape config, different external_label)
global:
  external_labels:
    cluster: my-prod-cluster
    prometheus_replica: prometheus-b # Unique identifier for this instance

When promxy is configured to aggregate metrics from such an HA pair within a server_group:

# promxy.yaml
promxy:
  server_groups:
    - static_configs:
        - targets:
            - prometheus-instance-A:9090
            - prometheus-instance-B:9090
      labels:
        sg: my_ha_cluster
      # ... other configurations ...

Because of the prometheus_replica label, each metric fetched from the upstream Prometheus instances will carry this label. As a result, the deduplication logic in promxy will treat metrics from each instance as distinct, even if they represent the same underlying data.

[jacksontj]

If I'm understanding the ask correctly this should be supported using the metric_relabel_configs section. This allows mutation of labels of the metrics coming through a servergroup.