cloudwatch-alarm

Name: cloudwatch-alarm
Description: Amazon CloudWatch Alarm 2.0. CloudWatch to Slack smart notification
GitHub: https://github.com/korniichuk/cloudwatch-alarm

Table of Contents

• Introduction
  • Amazon CloudWatch Alarm 2.0
  • Standard Amazon CloudWatch Alarm
• How it works
• Requirements
• Python lib versions
• Create Slack webhook
• Create Amazon SNS topic
• Create Amazon CloudWatch Alarm
• Create Amazon S3 bucket
• Create Amazon Lambda function
  • Create function base from blueprint
  • Upgrade function
• Edit AWS IAM role
• Test Amazon Lambda function
• Change Slack message retention

Introduction

Amazon CloudWatch Alarm 2.0

Standard Amazon CloudWatch Alarm

How it works

Requirements

Please, install Python packages:

$ pip install -t . -r requirements.txt

Python lib versions

• boto3 ver. 1.9.41
• botocore ver. 1.12.41

Create Slack webhook

Navigate to https://<your-team-domain>.slack.com/apps, like https://example.slack.com/apps. Search for and select Incoming WebHooks. Click Add Configuration button:

Choose the default channel where messages will be sent (like #example) and click Add Incoming WebHooks integration. Copy and save the webhook URL (like https://hooks.slack.com/services/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) from the setup instructions.

Note: You need admin rights to your Slack.

Note: Slack marks public channels with # (e.g. #public) and private channels without # (e.g. private).

Create Amazon SNS topic

Create new Amazon SNS topic: https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html#CreateTopic. Like example topic with arn:aws:sns:eu-west-1:539199393808:example ARN.

Create Amazon CloudWatch Alarm

Navigate to https://console.aws.amazon.com/cloudwatch/ and create new Amazon CloudWatch Alarm. Click Select metric button:

Select metric, like EC2 -> Per-Instance Metrics -> CPUUtilization:

Choose Source tab and select Image API checkbox. Copy and save json code. Click Select metric button:

Saved json code example:

{
    "metrics": [
        [ "AWS/EC2", "CPUUtilization", "InstanceId", "i-0d91fdf2dbb765977", { "period": 300, "stat": "Average" } ]
    ],
    "title": "The percentage of CPU utilization",
    "start": "-PT3H",
    "end": "P0D",
    "timezone": "+0100"
}

Set up Alarm details. Enter Name (e.g. CPUUtilization) and Description (e.g. The percentage of CPU utilization). Specify the alarm condition. For example: select >=, enter 80, and enter 3 datapoints.

Connect CloudWatch Alarm to created SNS topic and finally click Create Alarm button:

AWS docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html

Create Amazon S3 bucket

Navigate to https://console.aws.amazon.com/s3/ and create new S3 bucket (e.g. example). Create new lifecyle rule for bucket (e.g. DeleteTmpAfter72h) with tmp/ prefix filter:

Configure expiration as below. Expire after 3 days. Permanently delete after 3 days. Clean up incomplete multipart uploads after 1 day.

Edit public access settings to your S3 bucket. Select your S3 bucket (e.g. example) and click Edit public access settings button. Deselect all checkboxes and click Save button:

Enter confirm to confirm field and click Confirm button:

Create Amazon Lambda function

Create function base from blueprint

Navigate to https://console.aws.amazon.com/lambda/. Click Create function. Click Blueprints. Search for and select cloudwatch-alarm-to-slack-python3. Click Configure button:

Enter function name (e.g. example) to Name field. In Role select Create a new role from one or more templates. In Role name field enter role name (e.g. example).

Go to SNS trigger section and select your SNS topic. For example: example topic with arn:aws:sns:eu-west-1:539199393808:example ARN. Select the Enable trigger checkbox.

Create KMS Key or use an existing KMS Key. For example: arn:aws:kms:eu-west-1:539199393808:key/4c3126f2-f3bf-453e-b32d-58c9356d84b7.

Go to Lambda function code section. Go to Encryption configuration subsection and select the Enable helpers for encryption in transit checkbox. Select AWS KMS key to encrypt in transit (e.g. arn:aws:kms:eu-west-1:539199393808:key/4c3126f2-f3bf-453e-b32d-58c9356d84b7).

Paste Slack channel into the slackChannel environment variable, like example. The Slack channel does not contain private info, so do NOT click Encrypt button.

Paste Slack webhook URL into the kmsEncryptedHookUrl environment variable. You must exclude the protocol from the URL (e.g. hooks.slack.com/services/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). Click Encrypt button. Finally click Create function button:

Upgrade function

Clone cloudwatch-alarm repository:

$ git clone https://github.com/korniichuk/cloudwatch-alarm.git

Modify cloudwatch-alarm/lambda_function.py file. Replace confluence_url and cloudwatch_url var values. For example, from:

confluence_url = 'http://www.korniichuk.com'
cloudwatch_url = 'http://www.korniichuk.com'

to:

confluence_url = 'https://bit.ly/1d3LdqJ'
cloudwatch_url = 'https://amzn.to/2NeZooo'

Update metric var value with saved Image API json. For example, from:

metric = """{
    "metrics": [[
        "LogMetrics",
        "MetricName",
        {"period": 3600, "stat": "Sum"}]],
    "title": "Title",
    "start": "-P1D",
    "end": "P0D",
    "timezone": "+0100"}"""

to:

metric = """{
    "metrics": [[
        "AWS/EC2",
        "CPUUtilization",
        "InstanceId",
        "i-0d91fdf2dbb765977",
        {"period": 300, "stat": "Average"}]],
    "title": "The percentage of CPU utilization",
    "start": "-PT3H",
    "end": "P0D",
    "timezone": "+0100"}"""

Docs: Metric widget structure and syntax

Replace bucket_name var value to your bucket name. For example, from:

bucket_name = 'korniichuk'

to:

bucket_name = 'example'

Navigate to cloned cloudwatch-alarm repository and install Python packages:

$ cd cloudwatch-alarm
$ pip install -t . -r requirements.txt

Prepare zip.zip with lambda_function.py file and Python dependencies:

$ rm -r *.dist-info __pycache__
$ zip -r zip.zip .

Navigate to your lambda function (e.g. example). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:

Note: We use boto3 ver. 1.9.41 and botocore ver. 1.12.41 because default Lambda's packages do not support get_metric_widget_image function.

Edit AWS IAM role

Navigate to https://console.aws.amazon.com/iam/. Select your role (e.g. example). Select Permissions tab. Click Attach policies button:

Search for and select AmazonS3FullAccess and CloudWatchFullAccess. Click Attach policy button. See final result below.

Note: Do not use *FullAccess roles on production. You can grant access to concrete S3 bucket (e.g. example). You can precisize CloudWatchFullAccess policy to cloudwatch:* plus logs:* only.

Test Amazon Lambda function

Navigate to your lambda function (e.g. example). Click Select a test event.. and select Configure test events:

Enter event name to Event name field (e.g. Example). Copy json code from test-event.json file to code area or enter your own. Click Create button:

Select created event (e.g. Example) and click Test button:

Note: You can maximize CPU utilization with $ dd if=/dev/zero of=/dev/null command.

Change Slack message retention

Navigate to Slack channel. Click Channel Settings icon. Select Edit message retention...:

Change default message retention to 3 Days. Click Save button: