I recently completed a DevOps project that focused on building and deploying a microservices-based application to convert MP4 videos to MP3 audio files. The project was deployed on AWS Elastic Kubernetes Service (EKS), utilizing several microservices for authentication, video conversion, and notification delivery. It leveraged RabbitMQ for message queuing and PostgreSQL and MongoDB for database management.
The project gave me hands-on experience with cloud-based microservices architecture and Kubernetes. Here’s a detailed breakdown of how I approached and deployed the project.
The application was designed using a microservices architecture with the following components:
- Auth Service: Handles user login and authentication using JWT tokens.
- Converter Module: Converts video files (MP4) into audio files (MP3).
- Database Server: Uses PostgreSQL to store user data and MongoDB to store file metadata.
- Notification Service: Sends an email notification to users after a successful conversion.
RabbitMQ was used to manage queues between the video and MP3 conversion process.
The first step was to set up the infrastructure. I used AWS EKS to deploy the Kubernetes cluster. This involved:
- Creating IAM roles for both the EKS cluster and nodes.
- Provisioning the EKS cluster via the AWS Console and configuring VPC and subnets.
- Adding a node group to ensure the cluster had sufficient compute resources.
aws eks update-kubeconfig --name <cluster_name> --region <aws_region>For data storage, I deployed two different databases:
- PostgreSQL: Handles user authentication data.
- MongoDB: Stores video and audio file metadata.
I used Helm to deploy both databases:
# Install PostgreSQL
cd Helm_charts/Postgres
helm install postgres .
# Install MongoDB
cd Helm_charts/MongoDB
helm install mongo .Next, I deployed RabbitMQ to manage the messaging queues between the video and MP3 conversion process. I created two queues (mp3 and video) to handle different stages of conversion.
helm install rabbitmq .I verified that RabbitMQ was running and the queues were properly set up.
With the infrastructure ready, I deployed the core microservices:
-
Auth Service: Manages user login and issues JWT tokens.
kubectl apply -f auth-service/manifest
-
Converter Module: Performs the actual video-to-audio conversion.
kubectl apply -f converter-service/manifest
-
Notification Service: Sends email notifications after conversions.
kubectl apply -f notification-service/manifest
-
Gateway Service: Acts as a gateway for routing API requests.
kubectl apply -f gateway-service/manifest
To ensure secure email notifications, I implemented 2-factor authentication (2FA) using Gmail for sending out notifications. I generated an application-specific password from Gmail and stored it in the Kubernetes secrets for the notification service.
Once the microservices were up, I tested the API endpoints to make sure everything was functioning correctly.
-
Login Endpoint:
curl -X POST http://<nodeIP>:30002/login -u <email>:<password>
-
Upload Video (MP4):
curl -X POST -F 'file=@./video.mp4' -H 'Authorization: Bearer <JWT Token>' http://<nodeIP>:30002/upload
-
Download MP3: After conversion, I verified the MP3 file could be downloaded using:
curl --output video.mp3 -X GET -H 'Authorization: Bearer <JWT Token>' "http://<nodeIP>:30002/download?fid=<Generated_fid>"
Here’s an overview of the application's architecture:
Through this project, I gained practical experience in deploying a full-fledged microservices application on AWS EKS, integrating RabbitMQ, and managing data with PostgreSQL and MongoDB. I also enhanced my skills in Kubernetes and secured API services using JWT tokens. This project allowed me to build a complete end-to-end DevOps solution, emphasizing scalability, automation, and security.
Feel free to check out the code and ask any questions!
If you would like to connect or have questions, feel free to reach out via my LinkedIn.