Skip to content

Commit 75aeafe

Browse files
kundusokunduso
authored
Merge pull request #40 from kunduso/verify-access
Create AWS cloud resources to access the ElastiCache cluster from Amazon EC2 instances.
2 parents 1ab35de + 2e6c5a5 commit 75aeafe

File tree

7 files changed

+292
-9
lines changed

7 files changed

+292
-9
lines changed

‎README.md

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,18 +3,28 @@
33
[![terraform-infra-provisioning](https://github.com/kunduso/amazon-elasticache-redis-tf/actions/workflows/terraform.yml/badge.svg?branch=main)](https://github.com/kunduso/amazon-elasticache-redis-tf/actions/workflows/terraform.yml)[![checkov-static-analysis-scan](https://github.com/kunduso/amazon-elasticache-redis-tf/actions/workflows/code-scan.yml/badge.svg?branch=main)](https://github.com/kunduso/amazon-elasticache-redis-tf/actions/workflows/code-scan.yml)
44

55

6-
![Image](https://skdevops.files.wordpress.com/2023/10/85-image-0-1.png)
6+
![Image](https://skdevops.files.wordpress.com/2023/12/87-image-0-1.png)
77
# Motivation
8-
Amazon ElastiCache service supports Redis and Memcached. If you want in an in-memory caching solution for your application, check out the [AWS-Docs](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html). In this repository I have the Terraform code to provision an Amazon ElastiCache for Redis cluster and all the supporting infrastructure components like Amazon VPC, subnets, security group, AWS KMS key, and AWS Secrets Manager secret.
9-
<br />The process of provisioning is automated using GitHub Actions. I also followed a few best practices while creating the Amazon ElastiCache service, like enabling multi-availability zone, multi-node, and encryption in transit and at rest.
8+
Amazon ElastiCache service supports Redis and Memcached. If you want in an in-memory caching solution for your application, check out the [AWS-Docs](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/WhatIs.html). In this repository I cover **two use cases.**
109

11-
<br />I discussed the concept in detail in my notes at [-create-an-amazon-elasticache-for-redis-cluster-using-terraform](https://skundunotes.com/2023/10/21/create-an-amazon-elasticache-for-redis-cluster-using-terraform/).
10+
<br />**Use-Case 1:** Create an Amazon ElastiCache for Redis cluster using Terraform, and
11+
<br />**Use-Case 2:** Create an Amazon ElastiCache for Redis cluster and Amazon EC2 instances to access the cluster using Terraform.
1212

13-
<br />I used Bridgecrew Checkov to scan the Terraform code for security vulnerabilities. Here is a link if you are interested in adding code scanning capabilities to your GitHub Actions pipeline [-automate-terraform-configuration-scan-with-checkov-and-github-actions](https://skundunotes.com/2023/04/12/automate-terraform-configuration-scan-with-checkov-and-github-actions/).
14-
<br />I also used Infracost to generate a cost estimate of building the architecture. If you want to learn more about adding Infracost estimates to your repository, head over to this note [-estimate AWS Cloud resource cost with Infracost, Terraform, and GitHub Actions](https://skundunotes.com/2023/07/17/estimate-aws-cloud-resource-cost-with-infracost-terraform-and-github-actions/).
15-
<br />Lastly, I also automated the process of provisioning the resources using GitHub Actions pipeline and I discussed that in detail at [-CI-CD with Terraform and GitHub Actions to deploy to AWS](https://skundunotes.com/2023/03/07/ci-cd-with-terraform-and-github-actions-to-deploy-to-aws/).
13+
<br />If you are interested in Use-case 1, please refer to the [create-amazon-elasticache branch.](https://github.com/kunduso/amazon-elasticache-redis-tf/tree/create-amazon-elasticache)
14+
15+
For Use-case 2, this repository has the Terraform code to provision an Amazon ElastiCache for Redis cluster and all the supporting infrastructure components like Amazon VPC, subnets, security group, AWS KMS key, and AWS Secrets Manager secret. It also has addition AWS cloud resources like:
16+
<br />- an **internet gateway** and update the path in the route table attached to the **public subnet**
17+
<br />- an **IAM instance profile** and attach an **IAM role** with the two existing **IAM policies** to read from the **SSM parameter store** and **AWS Secrets manager**. These resources have the ElastiCache endpoint and auth_token stored that was created in Use-case 1.
18+
<br />- two **Amazon EC2 instances** in the public subnet with separate user data scripts to install **Python libraries** and create Python files inside the instances.
19+
<br />The process of provisioning is automated using **GitHub Actions**.
20+
21+
<br />I discussed the concept in detail in my notes at [-Connect to an Amazon ElastiCache cluster from an Amazon EC2 instance using Python](https://skundunotes.com/2023/12/13/connect-to-an-amazon-elasticache-cluster-from-an-amazon-ec2-instance-using-python/).
22+
23+
<br />I used **Bridgecrew Checkov** to scan the Terraform code for security vulnerabilities. Here is a link if you are interested in adding code scanning capabilities to your GitHub Actions pipeline [-automate-terraform-configuration-scan-with-checkov-and-github-actions](https://skundunotes.com/2023/04/12/automate-terraform-configuration-scan-with-checkov-and-github-actions/).
24+
<br />I also used **Infracost** to generate a cost estimate of building the architecture. If you want to learn more about adding Infracost estimates to your repository, head over to this note [-estimate AWS Cloud resource cost with Infracost, Terraform, and GitHub Actions](https://skundunotes.com/2023/07/17/estimate-aws-cloud-resource-cost-with-infracost-terraform-and-github-actions/).
25+
<br />Lastly, I also automated the process of provisioning the resources using **GitHub Actions** pipeline and I discussed that in detail at [-CI-CD with Terraform and GitHub Actions to deploy to AWS](https://skundunotes.com/2023/03/07/ci-cd-with-terraform-and-github-actions-to-deploy-to-aws/).
1626
## Prerequisites
17-
For this code to function without errors, I created an OpenID connect identity provider in Amazon Identity and Access Management that has a trust relationship with this GitHub repository. You can read about it [here](https://skundunotes.com/2023/02/28/securely-integrate-aws-credentials-with-github-actions-using-openid-connect/) to get a detailed explanation with steps.
27+
For this code to function without errors, I created an **OpenID connect** identity provider in **Amazon Identity and Access Management** that has a trust relationship with this GitHub repository. You can read about it [here](https://skundunotes.com/2023/02/28/securely-integrate-aws-credentials-with-github-actions-using-openid-connect/) to get a detailed explanation with steps.
1828
<br />I stored the ARN of the IAM Role as a GitHub secret which is referred in the [`terraform.yml`](https://github.com/kunduso/amazon-elasticache-redis-tf/blob/eb148db2b9ff37cff9f1fb469d0c14b6479bd57a/.github/workflows/terraform.yml#L42) file.
1929
<br />Since I used Infracost in this repository, I stored the `INFRACOST_API_KEY` as a repository secret. It is referenced in the [`terraform.yml`](https://github.com/kunduso/amazon-elasticache-redis-tf/blob/eb148db2b9ff37cff9f1fb469d0c14b6479bd57a/.github/workflows/terraform.yml#L52) GitHub actions workflow file.
2030
<br />As part of the Infracost integration, I also created a `INFRACOST_API_KEY` and stored that as a GitHub Actions secret. I also managed the cost estimate process using a GitHub Actions variable `INFRACOST_SCAN_TYPE` where the value is either `hcl_code` or `tf_plan`, depending on the type of scan desired.

‎ec2.tf

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
resource "aws_internet_gateway" "this-igw" {
2+
vpc_id = aws_vpc.this.id
3+
tags = {
4+
"Name" = "app-4-gateway"
5+
}
6+
}
7+
resource "aws_route" "internet-route" {
8+
destination_cidr_block = "0.0.0.0/0"
9+
route_table_id = aws_route_table.public.id
10+
gateway_id = aws_internet_gateway.this-igw.id
11+
}
12+
# create a security group
13+
resource "aws_security_group" "ec2_instance" {
14+
name = "app-4-ec2"
15+
description = "Allow inbound to and outbound access from the Amazon EC2 instance."
16+
ingress {
17+
from_port = 0
18+
to_port = 0
19+
protocol = "-1"
20+
cidr_blocks = [var.vpc_cidr]
21+
description = "Enable access from any resource inside the VPC."
22+
}
23+
egress {
24+
from_port = 0
25+
to_port = 0
26+
protocol = "-1"
27+
cidr_blocks = ["0.0.0.0/0"]
28+
description = "Enable access to the internet."
29+
}
30+
vpc_id = aws_vpc.this.id
31+
}
32+
33+
#create an EC2 in a public subnet
34+
data "aws_ami" "amazon_ami" {
35+
filter {
36+
name = "name"
37+
values = var.ami_name
38+
}
39+
filter {
40+
name = "virtualization-type"
41+
values = ["hvm"]
42+
}
43+
most_recent = true
44+
owners = ["amazon"]
45+
}
46+
resource "aws_instance" "app-server-read" {
47+
instance_type = var.instance_type
48+
ami = data.aws_ami.amazon_ami.id
49+
vpc_security_group_ids = [aws_security_group.ec2_instance.id]
50+
iam_instance_profile = aws_iam_instance_profile.ec2_profile.name
51+
associate_public_ip_address = true
52+
#checkov:skip=CKV_AWS_88: Required for Session Manager access
53+
subnet_id = aws_subnet.public[0].id
54+
ebs_optimized = true
55+
monitoring = true
56+
root_block_device {
57+
encrypted = true
58+
}
59+
metadata_options {
60+
http_endpoint = "enabled"
61+
http_tokens = "required"
62+
}
63+
tags = {
64+
Name = "app-4-server-read"
65+
}
66+
user_data = templatefile("user_data/read_elasticache.tpl",
67+
{
68+
Region = var.region,
69+
elasticache_ep = aws_ssm_parameter.elasticache_ep.name,
70+
elasticache_ep_port = aws_ssm_parameter.elasticache_port.name,
71+
elasticache_auth_token = aws_secretsmanager_secret.elasticache_auth.name
72+
})
73+
}
74+
resource "aws_instance" "app-server-write" {
75+
instance_type = var.instance_type
76+
ami = data.aws_ami.amazon_ami.id
77+
vpc_security_group_ids = [aws_security_group.ec2_instance.id]
78+
iam_instance_profile = aws_iam_instance_profile.ec2_profile.name
79+
associate_public_ip_address = true
80+
#checkov:skip=CKV_AWS_88: Required for Session Manager access
81+
subnet_id = aws_subnet.public[0].id
82+
ebs_optimized = true
83+
monitoring = true
84+
root_block_device {
85+
encrypted = true
86+
}
87+
metadata_options {
88+
http_endpoint = "enabled"
89+
http_tokens = "required"
90+
}
91+
tags = {
92+
Name = "app-4-server-write"
93+
}
94+
user_data = templatefile("user_data/write_elasticache.tpl",
95+
{
96+
Region = var.region,
97+
elasticache_ep = aws_ssm_parameter.elasticache_ep.name,
98+
elasticache_ep_port = aws_ssm_parameter.elasticache_port.name,
99+
elasticache_auth_token = aws_secretsmanager_secret.elasticache_auth.name
100+
})
101+
}

‎ec2_role.tf

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# #https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role
2+
resource "aws_iam_role" "ec2_role" {
3+
name = "app-4-ec2-role"
4+
5+
# Terraform's "jsonencode" function converts a
6+
# Terraform expression result to valid JSON syntax.
7+
assume_role_policy = jsonencode({
8+
Version = "2012-10-17"
9+
Statement = [
10+
{
11+
Action = "sts:AssumeRole"
12+
Effect = "Allow"
13+
Sid = ""
14+
Principal = {
15+
Service = "ec2.amazonaws.com"
16+
}
17+
},
18+
]
19+
})
20+
}
21+
#Attach role to policy
22+
#https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment
23+
resource "aws_iam_role_policy_attachment" "custom" {
24+
role = aws_iam_role.ec2_role.name
25+
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
26+
}
27+
28+
resource "aws_iam_role_policy_attachment" "ssm_policy_attachement" {
29+
role = aws_iam_role.ec2_role.name
30+
policy_arn = aws_iam_policy.ssm_parameter_policy.arn
31+
}
32+
resource "aws_iam_role_policy_attachment" "secret_policy_attachement" {
33+
role = aws_iam_role.ec2_role.name
34+
policy_arn = aws_iam_policy.secret_manager_policy.arn
35+
}
36+
#Attach role to an instance profile
37+
#https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_instance_profile
38+
resource "aws_iam_instance_profile" "ec2_profile" {
39+
name = "app-4-ec2-profile"
40+
role = aws_iam_role.ec2_role.name
41+
}

‎random.tf

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
#https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/auth.html#auth-overview
2+
#https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/password
23
resource "random_password" "auth" {
34
length = 128
45
special = true

‎user_data/read_elasticache.tpl

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
#!/bin/bash
2+
yum update -y
3+
yum install python-pip -y
4+
yum install python3 -y
5+
pip3 install redis-py-cluster
6+
pip3 install boto3
7+
pip3 install botocore
8+
echo "The region value is ${Region}"
9+
AWS_REGION=${Region}
10+
local_elasticache_ep=${elasticache_ep}
11+
local_auth_token=${elasticache_auth_token}
12+
local_elasticache_ep_port=${elasticache_ep_port}
13+
cat <<EOF >> /var/read_cache.py
14+
from rediscluster import RedisCluster
15+
from botocore.exceptions import ClientError
16+
import logging
17+
import boto3
18+
19+
def main():
20+
session = boto3.Session(region_name='$AWS_REGION')
21+
auth_token = get_secret(session)
22+
elasticache_endpoint = get_elasticache_endpoint(session)
23+
elasticache_port = get_elasticache_port(session)
24+
read_from_redis_cluster(elasticache_endpoint, elasticache_port, auth_token)
25+
26+
def get_secret(session):
27+
secret_client = session.client('secretsmanager')
28+
try:
29+
get_secret_value_response = secret_client.get_secret_value(
30+
SecretId='$local_auth_token'
31+
)
32+
except ClientError as e:
33+
raise e
34+
return get_secret_value_response
35+
36+
def get_elasticache_endpoint(session):
37+
ssm_client = session.client('ssm')
38+
return ssm_client.get_parameter(
39+
Name='$local_elasticache_ep', WithDecryption=True)
40+
41+
def get_elasticache_port(session):
42+
ssm_client = session.client('ssm')
43+
return ssm_client.get_parameter(
44+
Name='$local_elasticache_ep_port', WithDecryption=True)
45+
46+
def read_from_redis_cluster(endpoint, port, auth):
47+
logging.basicConfig(level=logging.INFO)
48+
redis = RedisCluster(startup_nodes=[{
49+
"host": endpoint['Parameter']['Value'],
50+
"port": port['Parameter']['Value']}],
51+
decode_responses=True,skip_full_coverage_check=True,ssl=True,
52+
password=auth['SecretString'])
53+
if redis.ping():
54+
logging.info("Connected to Redis")
55+
print("The city name entered is "+redis.get("City"))
56+
redis.close()
57+
main()
58+
EOF

‎user_data/write_elasticache.tpl

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
#!/bin/bash
2+
yum update -y
3+
yum install python-pip -y
4+
yum install python3 -y
5+
pip3 install redis-py-cluster
6+
pip3 install boto3
7+
pip3 install botocore
8+
echo "The region value is ${Region}"
9+
AWS_REGION=${Region}
10+
local_elasticache_ep=${elasticache_ep}
11+
local_auth_token=${elasticache_auth_token}
12+
local_elasticache_ep_port=${elasticache_ep_port}
13+
cat <<EOF >> /var/write_cache.py
14+
from rediscluster import RedisCluster
15+
import logging
16+
import boto3
17+
import sys
18+
19+
def main():
20+
CityName = input("Enter a City Name: ")
21+
session = boto3.Session(region_name='$AWS_REGION')
22+
auth_token = get_secret(session)
23+
elasticache_endpoint = get_elasticache_endpoint(session)
24+
elasticache_port = get_elasticache_port(session)
25+
write_into_redis_cluster(
26+
elasticache_endpoint,
27+
elasticache_port,
28+
auth_token,
29+
CityName)
30+
31+
def get_secret(session):
32+
secret_client = session.client('secretsmanager')
33+
try:
34+
get_secret_value_response = secret_client.get_secret_value(
35+
SecretId='$local_auth_token'
36+
)
37+
except ClientError as e:
38+
raise e
39+
return get_secret_value_response
40+
41+
def get_elasticache_endpoint(session):
42+
ssm_client = session.client('ssm')
43+
return ssm_client.get_parameter(
44+
Name='$local_elasticache_ep', WithDecryption=True)
45+
46+
def get_elasticache_port(session):
47+
ssm_client = session.client('ssm')
48+
return ssm_client.get_parameter(
49+
Name='$local_elasticache_ep_port', WithDecryption=True)
50+
51+
def write_into_redis_cluster(endpoint, port, auth, cityname):
52+
logging.basicConfig(level=logging.INFO)
53+
redis = RedisCluster(startup_nodes=[{
54+
"host": endpoint['Parameter']['Value'],
55+
"port": port['Parameter']['Value']}],
56+
decode_responses=True,skip_full_coverage_check=True,ssl=True,
57+
password=auth['SecretString'])
58+
if redis.ping():
59+
logging.info("Connected to Redis")
60+
redis.set('City', cityname)
61+
print("The city name entered is updated in the Redis cache cluster.")
62+
redis.close()
63+
main()
64+
EOF

‎variable.tf

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,15 @@ variable "subnet_cidr_public" {
3232
default = ["10.20.32.96/27"]
3333
type = list(any)
3434
}
35-
35+
variable "ami_name" {
36+
description = "The ami name of the image from where the instances will be created"
37+
default = ["amzn2-ami-amd-hvm-2.0.20230727.0-x86_64-gp2"]
38+
type = list(string)
39+
}
40+
variable "instance_type" {
41+
description = "The instance type of the EC2 instances"
42+
default = "t3.medium"
43+
}
3644
variable "replication_group_id" {
3745
description = "The name of the ElastiCache replication group."
3846
default = "app-4-redis-cluster"

0 commit comments

Comments
 (0)