Failure to send alerts to Pagerduty through a squid proxy (error:invalid-request)

[sajoupa]

Bug Description

Configuring cos-alerter to use a squid proxy to reach the internet (and pagerduty), yields an invalid request in Squid:
1710247929.228 0 <IP adress> NONE/400 3824 NONE error:invalid-request - HIER_NONE/- text/html
And then cos-alerter logs an error:
Mar 12 10:31:19 prod-is-cos-alerter-1 docker[3288655]: 2024-03-12T10:31:19.847Z [cos-alerter] WARNING:apprise:A Connection error occurred sending Pager Duty notification to events.pagerduty.com.

Whereas, when using the apprise application from the same machine, through the same proxy, the alerts are successfully sent.
(Yielding 1710257497.415 783 <IP address> TCP_TUNNEL/200 4124 CONNECT events.pagerduty.com:443 - HIER_DIRECT/44.233.86.211 -)

$ https_proxy="http://<proxy adress>:3128" apprise -vvvv -t "Test Alert, please ignore" -b "Test Message Body" "pagerduty://<routing key>@events.pagerduty.com/"
[...]
2024-03-12 15:31:36,624 - DEBUG - Loaded Pager Duty URL: pagerduty://****@****/A...e/N...n?region=us&image=yes&format=text&overflow=upstream&rto=4.0&cto=4.0&verify=yes
2024-03-12 15:31:36,625 - DEBUG - Pager Duty POST URL: https://events.pagerduty.com/v2/enqueue (cert_verify=True)
2024-03-12 15:31:36,625 - DEBUG - Pager Duty Payload: {'routing_key': '<REDACTED>', 'payload': {'summary': 'Test Alert, please ignore\r\nTest Message Body', 'severity': 'info', 'source': 'Apprise', 'component': 'Notification'}, 'client': 'Apprise', 'event_action': 'trigger', 'images': [{'src': 'https://github.com/caronc/apprise/raw/master/apprise/assets/themes/default/apprise-info-128x128.png', 'alt': 'info'}]}
2024-03-12 15:31:37,410 - INFO - Sent Pager Duty notification.

To Reproduce

(see above)

Environment

cos-alerter running in a docker container:

$ systemctl cat docker.cos-alerter.service 
# /etc/systemd/system/docker.cos-alerter.service
[Unit]
Description=cos-alerter service
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker exec %n stop
ExecStartPre=-/usr/bin/docker rm %n
ExecStartPre=/usr/bin/docker pull ghcr.io/canonical/cos-alerter:latest
ExecStart=/usr/bin/docker run --env HTTP_PROXY="http://<proxy address>:3128" --env HTTPS_PROXY="https://<proxy address>:3128" --rm --name %n \
    -p 8080:8080 \
    --mount type=bind,source=/home/ubuntu/cos-alerter-config.yaml,target=/etc/cos-alerter.yaml,readonly \
    ghcr.io/canonical/cos-alerter:latest

[Install]
WantedBy=default.target

Relevant log output

(see above)

Additional context

No response

[lucabello]

We should make sure we correctly pass the proxy variables to whatever is doing the request.