Skip to content

Completed - Assignment 2 Pull Request: Extend Automation for Log Management with EC2 & S3 #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Completed - Assignment 2 Pull Request: Extend Automation for Log Management with EC2 & S3 #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fed863e
Added feature to use 2nd EC2 instance(read only role) to pull logs to…
git-user-9 Jul 2, 2025
144f457
Delete terraform/.terraform.lock.hcl
git-user-9 Jul 2, 2025
c793f73
Readme changes
git-user-9 Jul 2, 2025
b914ae4
Minor Improvements
git-user-9 Jul 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 114 additions & 55 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AWS EC2 Auto Deployment with Terraform & Shell Scripts

This project automates the provisioning of an EC2 instance and the deployment of your application on AWS using Terraform and shell scripts. It supports different environments (Dev, Prod) through configuration files and scripts.
This project automates the provisioning of an EC2 instance and the deployment of your application on AWS using Terraform and shell scripts. It supports different environments (Dev, Prod) through configuration files and deployment scripts.

---

Expand All @@ -9,50 +9,58 @@ This project automates the provisioning of an EC2 instance and the deployment of
```
tech_eazy_devops_git-user-9/
├── README.md # Project documentation
├── terraform/ # Contains Terraform configurations
├── terraform/ # Terraform configurations
│ ├── main.tf # Main Terraform configuration file
│ ├── outputs.tf # Defines Terraform outputs (e.g., EC2 public IP)
│ ├── variables.tf # Declares input variables for Terraform
│ ├── dev_config.tfvars # Terraform variable values for the 'dev' environment
│ └── prod_config.tfvars # Terraform variable values for the 'prod' environment
└── scripts/ # Shell scripts for deployment and configuration
├── deploy.sh # Automates the Terraform apply process for a given environment
├── dev_script.sh # User-data script for configuring EC2 in the 'dev' environment
└── prod_script.sh # User-data script for configuring EC2 in the 'prod' environment
│ ├── variables.tf # Public variables file for EC2 and other details
│ ├── dev_config.tfvars # Variable values for 'Dev' environment
│ ├── prod_config.tfvars # Variable values for 'Prod' environment
├── scripts/ # Shell scripts for provisioning and deployments
│ ├── deploy.sh # Automates provisioning with Terraform
│ ├── dev_script.sh # Dev-specific configuration script for EC2
│ ├── prod_script.sh # Production-specific script for EC2
│ ├── verify_logs.sh # Validates and uploads logs
├── mylogs/ # Application and system logs
│ ├── app/ # Stores runtime application logs
│ │ └── my-app.log # Main application log
│ └── system/ # Tracks provisioning/system logs
│ └── cloud-init.log # Logs of initialization processes
└── .gitignore # Lists files to exclude from version control
```

---

## ⚙️ **Prerequisites**

Before you begin, ensure you have the following:
Ensure the following tools and resources are configured before deploying:

- **AWS Account** with EC2 access and permissions to create resources
- **IAM User** with programmatic access (access key ID and secret access key)
- **AWS CLI** installed and configured
- **Terraform** installed (version >= 1.0 recommended)
- **Git** (optional, for repository operations)
- **AWS Account** with IAM permissions for creating EC2, S3, and other resources.
- **IAM User** with access keys for programmatic access.
- **AWS CLI** installed and configured on your machine.
- **Terraform** (version >= 1.0 recommended).
- **Git** installed for version control.
- An **EC2 Key Pair** set up in AWS Console for securely accessing instances (see [Key Pair Section](#ec2-key-pair-requirement)).

---

## 🔐 **AWS Credentials Setup**

You must set your AWS credentials in your local environment so Terraform can authenticate with AWS.
Terraform authenticates with AWS using your configured credentials.

### Option 1: Using AWS CLI (Recommended)
### Option 1: AWS CLI (Recommended)

```bash
aws configure
```

You'll be prompted to input your:
Provide the following inputs:
- AWS Access Key ID
- AWS Secret Access Key
- Default region name (e.g., `ap-south-1`)
- Default AWS region (e.g., `ap-south-1`)
- Default output format (e.g., `json`)

### Option 2: Set environment variables directly
### Option 2: Environment Variables

Set environment variables explicitly:
```bash
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
Expand All @@ -61,67 +69,118 @@ export AWS_DEFAULT_REGION=ap-south-1

---

## EC2 Key Pair Requirement

Ensure you have an EC2 Key Pair set up in the AWS Console. Update the key pair's name in these files:

**`terraform/variables.tf`**
```hcl
variable "key_name" {
default = "your-key-name-here"
}
```

**`terraform/dev_config.tfvars`**
```hcl
key_name = "your-key-name-here"
```

**`terraform/prod_config.tfvars`**
```hcl
key_name = "your-key-name-here"
```

The Key Pair ensures secure SSH access to the instances.

---

## 🚀 **How to Deploy**

### 1️⃣ Clone the Repository

```bash
git clone https://github.com/git-user-9/tech_eazy_devops_git-user-9.git app
cd app
git clone https://github.com/git-user-9/tech_eazy_devops_git-user-9.git
cd tech_eazy_devops_git-user-9
```

### 2️⃣ Run the Deployment Script

```bash
./scripts/deploy.sh dev # For Dev environment
./scripts/deploy.sh prod # For Prod environment
./scripts/deploy.sh dev # For Development Environment
./scripts/deploy.sh prod # For Production Environment
```

The script will:
- Load the corresponding Terraform variable file
- Initialize and apply the Terraform configuration
This will:
- Apply Terraform configurations for selected environment
- Output the public IP of the created EC2 instance
- Upload logs to S3 bucket
- Shutdown the instance after 10-15 minutes (configurable)
- Upload logs to S3 automatically
- Terminate the instance after 10-15 minutes if configured

### 3️⃣ Access the Application

Visit the public IP shown in the terminal:

Navigate to:
```
http://<your-ec2-ip>:80
http://<ec2-public-ip>:80
```

---

## 🛠️ **What Happens Behind the Scenes**
## 🛠️ **Details of Automation**

### Terraform provisions:
- EC2 instance in default VPC
- Security group with ports 22 (SSH) and 80 (HTTP) open
### Terraform Provisions:
- **EC2 Instances** within the default VPC.
- **Security Groups** with HTTP (80) and SSH (22) access.
- **IAM Roles** for instances to access S3.

### User Data (inside shell script):
- Updates the system
- Installs AWS CLI, curl, unzip, Java 21, Git, and Maven
- Clones and builds your application
- Launches the application on port 80
- Uploads logs to S3 bucket
- Shuts down the instance after 10-15 minutes (configurable)
### Shell Scripts:
- Update operating system packages.
- Install required tools such as Java, Git, Maven, AWS CLI, etc.
- Clone, build, and run the application on Port 80.
- Upload logs to the S3 bucket.

---

## ✅ **Security Notes**
## Note on Pulling Logs from EC2 to Local

To enable **log pulling from EC2 to your local machine,** follow these steps:

1. **Uncomment Lines in the Script:**
*This step is only if you want logs to be fetched from s3 to your local directory*
Locate the following lines in your deployment script between **lines 52–59** and uncomment them:

```bash
# Wait a while for logs to upload
sleep 100
cd .. # Save logs at the root level
PRIVATE_KEY_PATH="/Users/default/CS/DevOps/AWS/ssh-key-ec2.pem" # Change this to your SSH private key path and ensure `chmod 400` on your key
echo "Trying to SCP logs to local"
scp -r -i "$PRIVATE_KEY_PATH" ubuntu@$VERIFIER_IP:/mylogs/ . # Pull logs from EC2 to /mylogs/ in your local directory
cd $TERRAFORM_DIR # Return to Terraform directory for destroy commands
```

2. **Specify Your Private Key Path:**
- Replace the placeholder `"/Users/default/CS/DevOps/AWS/ssh-key-ec2.pem"` under the variable `PRIVATE_KEY_PATH` with the actual path to your EC2 key's private key file.
- Before using, ensure the private key has the appropriate permissions by running:
```bash
chmod 400 /path/to/your/private-key.pem
```

3. **Save the logs locally:**
After successfully setting this up, the script will pull logs from `/mylogs/` on your EC2 instance to a local `/mylogs/` directory at the repository's root level.

This addition ensures your logs are saved to your local environment automatically.


- No sensitive keys are stored in the repo
- Make sure your security group is restricted to specific IPs if used in production
- Rotate AWS credentials regularly

---

## 💬 **FAQ**
## 💬 **FAQs**

**Q: Can I deploy to a different AWS region?**
Yes. Modify the `aws_region` value in `terraform/variables.tf`, `dev_config.tfvars` and `prod_config.tfvars`
**Q: How can I deploy in a different region?**
Modify the `aws_region` variable in the `terraform/variables.tf` file and update it in the `.tfvars` files.

**Q: What happens if deployment fails?**
Terraform maintains a state file. Retry by running the deployment script again.

**Q: Where can I find the logs?**
Logs are stored in the `mylogs/` directory or uploaded to the configured S3 bucket.

---


19 changes: 19 additions & 0 deletions mylogs/app/my-app.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@

. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/

:: Spring Boot :: (v3.4.6)

2025-07-08T23:23:28.441Z INFO 4086 --- [techeazy-devops] [ main] c.t.devops.TecheazyDevopsApplication : Starting TecheazyDevopsApplication v0.0.1-SNAPSHOT using Java 21.0.7 with PID 4086 (/home/ubuntu/app/target/techeazy-devops-0.0.1-SNAPSHOT.jar started by root in /home/ubuntu/app)
2025-07-08T23:23:28.445Z INFO 4086 --- [techeazy-devops] [ main] c.t.devops.TecheazyDevopsApplication : No active profile set, falling back to 1 default profile: "default"
2025-07-08T23:23:30.381Z INFO 4086 --- [techeazy-devops] [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port 80 (http)
2025-07-08T23:23:30.411Z INFO 4086 --- [techeazy-devops] [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2025-07-08T23:23:30.411Z INFO 4086 --- [techeazy-devops] [ main] o.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/10.1.41]
2025-07-08T23:23:30.612Z INFO 4086 --- [techeazy-devops] [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2025-07-08T23:23:30.615Z INFO 4086 --- [techeazy-devops] [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2078 ms
2025-07-08T23:23:31.650Z INFO 4086 --- [techeazy-devops] [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port 80 (http) with context path '/'
2025-07-08T23:23:31.676Z INFO 4086 --- [techeazy-devops] [ main] c.t.devops.TecheazyDevopsApplication : Started TecheazyDevopsApplication in 4.078 seconds (process running for 4.942)
Loading