In this Project created an ETL(Extract,Transform,Load) pipeline by using Spotify API on AWS cloud. The Project aims at using the playlist from Spotify API and transforming and storing cleaned data in AWS cloud storage and utilizing the data in snowflake using snowpipe.
-
AWS S3(Simple Storage Service): AWS S3 is a highly saclable object storage service used for storing and retrieving any amount of data from anywhere using web. It is commonly used to store and distribute large files.
-
AWS Lambda: AWS Lambda is a serverless computing service that lets you run code without managing servers.
-
Cloud Watch: AWS Cloudwatch is a monitoring service for AWS Resources and the applications you run on them. It is used to collect and tracks metrics, collect and monitor log files and set alarms.
-
AWS Data Catalog: AWS Data Catalog is a centralized metadata repository for all your data assets across various data sources.
-
AWS GLue Crawler: AWS Data Catalog consists of AWS Glue Crawler which crawls the data sources and identifies data formats, infer schemas and create AWS Glue Data Catalog.
-
AWS Athena: Amazon Athena is a interactive query service that makes it easy to analyze data stored in S3 using standard SQL. It is also used to analyze stored in Glue Data Catalog.
-
Snowflake: Snowflake is a cloud-based data warehouse platform that allows users to store, analyze, and exchange data securely.
-
PowerBI: Power BI is a business analytics service by Microsoft that allows organizations to connect to various data sources, transform and clean data, create interactive visualizations, and share insights with others.
