For this project, I built a complete data infrastructure from the ground up, mirroring a real-world ecommerce environment. I managed diverse data sources and orchestrated an end-to-end ELT (Extract, Load, Transform) pipeline. The process involved extracting raw data, loading it into a PostgreSQL data warehouse, and then running automated transformations using Pandas and dbt. To ensure reliability, I orchestrated the entire workflow with Apache Airflow and even implemented a Telegram bot for real-time notifications on pipeline status. Finally, I brought the data to life by creating insightful dashboards in Metabase. The entire system was containerized using Docker and Docker Compose for easy deployment and scalability.
- Technologies Used: Apache Airflow, dbt, PostgreSQL, Metabase, Docker, Pandas, Telegram API
- View on GitHub