Redfin Analytics: Python ETL Pipeline with Apache Airflow

This project demonstrates an automated ETL (Extract, Transform, Load) pipeline for Redfin Analytics, leveraging Python, Apache Airflow, AWS, and Snowflake. The pipeline extracts real estate data from Redfin's API, processes it, and loads it into Snowflake for real-time analysis.

Data Extraction: Raw data, including property prices and features, is extracted from Redfin's API and stored in Amazon S3.

Data Transformation: Python scripts clean and transform the data, ensuring it is ready for analysis, and the processed data is stored in a separate S3 bucket.

Orchestration: Apache Airflow manages the ETL workflow on AWS EC2, scheduling tasks and ensuring fault tolerance.

Loading into Snowflake: Cleaned data is loaded into Snowflake using SnowPipe for real-time ingestion.

Analysis and Visualization: Analysts leverage SQL queries in Snowflake and integrate with BI tools for dashboard creation, enabling data-driven decisions.

This pipeline streamlines data collection and analysis, allowing stakeholders to focus on insights from real-time data.