System Documentation

Technical Architecture

Overview

This system uses AWS Terraform to provision and manage a scheduled data pipeline. A Docker container running a Python application is executed every Wednesday at 6:30 PM IST (13:00 UTC) using AWS EventBridge to trigger an ECS Fargate task.

---

System Components

1. Docker Container

• A Docker container with a Python application is built and stored in Amazon ECR (Elastic Container Registry).
• This container runs the data processing workload.

2. AWS Infrastructure

The infrastructure includes:

• EventBridge Rule: Schedules task execution using CRON expression
• ECS Cluster: Managed container environment
• ECS Task Definition: Defines container parameters
• IAM Roles: Permission controls for ECS and EventBridge
• Security Group: Network traffic control
• CloudWatch Logs: Container logging and monitoring

---

Dockerization Process

1. Dockerfile in code

# Use official Python 3.11 slim image
FROM python:3.11-slim

# Set working directory inside the container
WORKDIR /app

# Copy requirements file and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the code
COPY . .

# Run the main Python script
CMD ["python", "main.py"]

2. Build the Docker image locally

docker build -t python-app:latest .

3. Authenticate with ECR

aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin <your-aws-account>.dkr.ecr.ap-south-1.amazonaws.com

4. Tag and push to ECR

docker tag python-app:latest <your-aws-account>.dkr.ecr.ap-south-1.amazonaws.com/python-app:latest
docker push <your-aws-account>.dkr.ecr.ap-south-1.amazonaws.com/python-app:latest

---

Terraform Infrastructure Breakdown

The Terraform code is organized into logical modules:

1. Backend Configuration (backend.tf)

• Uses S3 for storing Terraform state
• Bucket: client-bucket
• Region: ap-south-1
• State file path: terraform-files/ecs-scheduler/terraform.tfstate

2. IAM Configuration (iam.tf)

Creates three IAM roles:

• ecs_task_role: Permissions for the task itself
• ecs_execution_role: Permissions for ECS to execute the task
• eventbridge_ecs_invoke_role: Permissions for EventBridge to invoke ECS

Attaches managed and custom policies to these roles.

3. ECS Configuration (ecs.tf)

• Creates ECS cluster: ecs-cluster
• Defines task definition with:
  • CPU: 2048 units (2 vCPU)
  • Memory: 4096 MiB (4 GB)
  • Container image: From ECR
  • Log configuration: CloudWatch

4. Logging Configuration (logging.tf)

• Creates CloudWatch log group: python-app
• Sets retention period: 30 days

5. Network Configuration (network.tf)

• Creates security group with:
  • Outbound: All traffic allowed
  • Inbound: HTTP (80) and HTTPS (443) allowed

6. Scheduler Configuration (scheduler.tf)

• Creates EventBridge rule using CRON expression
• Configures ECS task as rule target
• Sets network configuration for the task

---

Key Variables Explained

Variable	Description	Value
ecr_image_url	Latest ECR image URL with tag	<your-aws-account>.dkr.ecr.ap-south-1.amazonaws.com/python-app:latest
schedule_expression_cron	CRON schedule in UTC	cron(0 13 ? * 4 *) (Wednesday 13:00 UTC)
ecs_task_cpu	CPU allocation for task	2048 (2 vCPU)
ecs_task_memory	Memory allocation for task	4096 (4 GB)
vpc_id	VPC for deployment	Set in tfvars
subnet_ids	Subnets for task	Set in tfvars
assign_public_ip_to_task	Public IP assignment	true

---

VPC and Subnet Configuration

The ECS task runs in a specified VPC and subnet(s). These determine:

• Internet or private resource access (ensure public subnet with Internet Gateway)
• Security boundaries
• IP address ranges
• Network routing for outbound connections

Public IP is assigned: assign_public_ip_to_task = true, allowing internet access.

---

Security Considerations

• IAM roles with least-privilege permissions
• Security group with restricted network access
• Resource tagging for organization
• CloudWatch logging for monitoring and troubleshooting

---

Applying the Terraform Configuration

1. Initialize Terraform

terraform init

2. Create or update .tfvars file

vpc_id     = "vpc-0123456789abcdef0"
subnet_ids = ["subnet-01bcb35c4328cee27"]

3. Plan the deployment

terraform plan -var-file="your-vars.tfvars"

4. Apply the configuration

terraform apply -var-file="your-vars.tfvars"

---

Execution Flow

1. Every Wednesday at 6:30 PM IST (13:00 UTC), EventBridge triggers the rule
2. EventBridge assumes eventbridge_ecs_invoke_role
3. EventBridge calls ECS RunTask API to start the task
4. ECS starts a Fargate task with:
   • Task role permissions
   • Execution role permissions
   • Container from ECR
   • Network configuration
5. The container runs the Python application
6. Logs are sent to CloudWatch
7. Task terminates upon completion

---

The entire process is fully automated, requiring no manual intervention once deployed.