This organization will keep all my sandbox projects for research and develop any data tools and solutions.
If you want to know about all services that data engineer should to know, you can follow 📖 Data Developer & Engineer document.
This part create my interface object for easy to use the target API tools.
- 🔎 Elastic: Create the best practice interface to Elastic cloud.
- 🌊 LakeHouse: Interact with any lake storage like DeltaLake, IceBerg, Hudi etc.
- 🐋 Docker: Create the best practice interface to Docker via Python.
- 📩 Notify (Comming Soon): Interact with any notification services like Jira, MS Team, etc.
- ✨ Databricks (Comming Soon)
- ⚡ PySpark (Comming Soon)
Note
All interface projects use the same files structure:
root/
├─ docs/
│ ╰─ index.md
├─ src/
│ ├─ modules/
│ │ ├─ __init__.py
│ │ ╰─ sub-module.py
│ ╰─ module.py
├─ tests/
│ ╰─ test_module.py
├─ .env
├─ main.py
╰─ requirements.txt
- 🌬️ Airflow: Create the Best practice of Airflow.
- 🌬️🧰 Airflow Generator: Create the Best practice of Airflow with Minimal Generator DAG Concept.
- 🛠️ DBT (Comming Soon): Create the Best practice for DBT.
- ✅ Great Expectation (Comming Soon): Create the Best practice for Great Expectation.
- 🤖 AI Agent (Comming Soon)
Note
This module has setup file for the local installation.
Note
Comming Soon ...