- 👨💻 I’m currently working as a Database Administrator, building strong foundations in data management and reliability
- 🌱 Transitioning into Data Engineering by designing end‑to‑end batch and streaming pipelines
- 🛠️ Passionate about building scalable, reliable data pipelines that turn raw data into actionable insights
- 👯 Open to collaborating on Data Engineering & Open Source projects
- 👨💻 Explore my work here: My Portfolio
- 📫 Reach me at arjunmpec101@gmail.com
- ⚡ Fun fact: I debug pipelines the way I play games — with persistence and strategy
-
🗄️ YouTube Data Engineering Pipeline (Batch Processing)
End‑to‑end batch ETL pipeline implementing the Medallion Architecture (Bronze → Silver → Gold).- Orchestrated with Apache Airflow (3.x)
- Transformations with Apache Spark
- Data lake layers on local filesystem (Bronze/Silver/Gold)
- Serving layer in Postgres (analytics‑ready tables)
- Interactive Streamlit + Altair dashboard via SQLAlchemy
- Ingests raw YouTube trending data (CSV/JSON), cleans, enriches, and computes derived metrics for BI
-
📊 StockPulse (Streaming Pipeline)
Real‑time streaming pipeline simulating stock ticks and processing them end‑to‑end.- Ingestion via Kafka producer publishing to
stock_tickstopic - Processing with Spark Structured Streaming (schema enforcement + derived metrics)
- Dual sinks: Postgres (serving layer) + Parquet (partitioned by index/date)
- Interactive Streamlit + Altair dashboard for real‑time visualization
- Fully orchestrated with Apache Airflow
- Ingestion via Kafka producer publishing to
Note: Top languages is only a metric of the languages my public code consists of and doesn't reflect experience or skill level.
