I built an end-to-end data pipeline that ingests customer reviews from e-commerce platforms (like Amazon and AliExpress), processes and transforms the data using Databricks and PySpark, and stores the cleaned output in PostgreSQL, exposed securely via ngrok. The final data is visualized in Power BI, enabling insightful analysis of customer sentiment and review trends. What sets this project apart is the full integration of Azure Data Lake, Databricks, and PostgreSQL with automation-ready infrastructure demonstrating real-world data engineering from raw ingestion to business intelligence.

Step 1 - Data Collection

Step 2 - Upload to GitHub(Individual submission)

Step 3 - Local Consolidation & Raw Folder Creation

Step 4 - Upload to Azure Data Lake (via VS Code + Python)

Step 5 - Mount Data Lake Data & Data Transformation in Databricks

Step 6 - Load into PostgreSQL