Skip to content

Commit 89bd806

Browse files
dcupifdcupif
authored
Merge pull request #15 from qonto/add-alerting-on-rds-ca-certificates
feat(rds): add alerting on CA certificates expiration
2 parents 0d58716 + 99e908e commit 89bd806

File tree

3 files changed

+124
-0
lines changed

3 files changed

+124
-0
lines changed

‎charts/prometheus-rds-alerts/prometheus_tests/RDSCACertificateCloseToExpiration.yml

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
rule_files:
2+
- rules.yml
3+
4+
evaluation_interval: 1m
5+
6+
tests:
7+
8+
- name: RDSCACertificateCloseToExpiration
9+
interval: 1d
10+
input_series:
11+
- series: 'rds_certificate_expiry_timestamp_seconds{aws_account_id="111111111111",aws_region="eu-west-3",dbidentifier="db1"}'
12+
values: '1728000x40' # 1728000 seconds = 20 days
13+
- series: 'rds_certificate_expiry_timestamp_seconds{aws_account_id="111111111111",aws_region="eu-west-3",dbidentifier="db2"}'
14+
values: '2629800x40' # 2629800 seconds = 1 month
15+
- series: 'rds_certificate_expiry_timestamp_seconds{aws_account_id="111111111111",aws_region="eu-west-1",dbidentifier="db1"}'
16+
values: '1728000x40' # 1728000 seconds = 20 days
17+
- series: 'rds_certificate_expiry_timestamp_seconds{aws_account_id="222222222222",aws_region="eu-west-3",dbidentifier="db1"}'
18+
values: '2629800x40' # 2629800 seconds = 1 month
19+
alert_rule_test:
20+
- alertname: RDSCACertificateCloseToExpiration
21+
eval_time: 4d
22+
exp_alerts: []
23+
- alertname: RDSCACertificateCloseToExpiration
24+
eval_time: 6d
25+
exp_alerts:
26+
- exp_labels:
27+
aws_account_id: 111111111111
28+
aws_region: eu-west-3
29+
severity: warning
30+
exp_annotations:
31+
description: "1 instance(s) of the AWS account ID=111111111111 in region=eu-west-3 use(s) a certificate with an expiration date inferior to 15 days"
32+
summary: "RDS instance(s) use(s) a certificate with an expiration date inferior to 15 days"
33+
runbook_url: "https://qonto.github.io/database-monitoring-framework/0.0.0/runbooks/rds/RDSCACertificateCloseToExpiration"
34+
- exp_labels:
35+
aws_account_id: 111111111111
36+
aws_region: eu-west-1
37+
severity: warning
38+
exp_annotations:
39+
description: "1 instance(s) of the AWS account ID=111111111111 in region=eu-west-1 use(s) a certificate with an expiration date inferior to 15 days"
40+
summary: "RDS instance(s) use(s) a certificate with an expiration date inferior to 15 days"
41+
runbook_url: "https://qonto.github.io/database-monitoring-framework/0.0.0/runbooks/rds/RDSCACertificateCloseToExpiration"

‎charts/prometheus-rds-alerts/values.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,3 +184,13 @@ rules:
184184
description: "{{ $labels.dbidentifier }} has forced maintenance"
185185
pintComments:
186186
- disable promql/series
187+
188+
RDSCACertificateCloseToExpiration:
189+
expr: |
190+
# 1296000 seconds = 15 days
191+
count by (aws_account_id, aws_region) (rds_certificate_expiry_timestamp_seconds - time() <= 1296000) > 0
192+
labels:
193+
severity: warning
194+
annotations:
195+
summary: "RDS instance(s) use(s) a certificate with an expiration date inferior to 15 days"
196+
description: "{{ $value }} instance(s) of the AWS account ID={{ $labels.aws_account_id}} in region={{ $labels.aws_region }} use(s) a certificate with an expiration date inferior to 15 days"

‎content/runbooks/rds/RDSCACertificateCloseToExpiration.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: CA Certificate Close to Expiration
3+
---
4+
5+
# RDSCACertificateCloseToExpiration
6+
7+
## Meaning
8+
9+
Alert is triggered when an RDS instance is detected using a CA certificate which is going to expire in less than 15 days.
10+
11+
## Impact
12+
13+
If the certificate is not renewed before expiration, all attempts to initiate an SSL/TLS connection to the RDS instance will fail.
14+
15+
{{< hint warning >}}
16+
**Important**
17+
18+
The `Amazon RDS Root 2019 CA` certificate expires on **Aug 22 17:08:50 2024 UTC**.
19+
20+
- Starting January 25th 2024, RDS instances created without specifying the CA will use `rds-ca-rsa2048-g1``.
21+
- In August 2024, AWS will enforce the CA rotation on all RDS instances on the expiring CA during a window maintenance
22+
{{< /hint >}}
23+
24+
## Diagnosis
25+
26+
- Identify the instance(s) concerned by either:
27+
- opening the `RDS instances` dashboard
28+
- or using the following AWS CLI command
29+
30+
```bash
31+
aws rds describe-db-instances | jq '
32+
[
33+
.DBInstances[] |
34+
{
35+
db_instance_identifier: .DBInstanceIdentifier,
36+
ca_certificate_identifier: .CACertificateIdentifier,
37+
ca_certificate_valid_until: .CertificateDetails.ValidTill
38+
} |
39+
(now + 1296000) as $date |
40+
select (
41+
(.ca_certificate_valid_until | split("+")[0] + "Z" | fromdate) < $date
42+
)
43+
]'
44+
```
45+
46+
Note: `1296000` seconds = 15 days
47+
48+
## Mitigation
49+
50+
Renew your certificate for the instances retrieved above by running:
51+
52+
```bash
53+
aws rds modify-db-instance \
54+
--db-instance-identifier <your_db_instance> \
55+
--ca-certificate-identifier <your_new_certificate>
56+
```
57+
58+
Use the `--apply-immediately` flag if you wish to change the certificate immediately, otherwise it will apply during your next scheduled maintenance window.
59+
60+
{{< hint info >}}
61+
**Tips**
62+
63+
We recommend using the `rds-ca-rsa2048-g1` certificate authority which:
64+
65+
- Has the same properties as `rds-ca-2019` (2048 private key, SHA256 signing alg.) so no risk of incompatibility
66+
- Is valid until 2061
67+
- Change can be done without restarting the instances
68+
{{< /hint >}}
69+
70+
## Additional resources
71+
72+
- [Using SSL with RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)
73+
- [SSL Certificate Rotation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html)

0 commit comments

Comments
 (0)