Freelancers using Apache Airflow in KarachiFreelancers using Apache Airflow in Karachi
ProjectsPeople
Muhammad Arsalan's avatar
Muhammad Arsalan
Karachi, Pakistan
Data Engineer | Web Scraping | SQL | BI Dashboards
Data Engineer | Web Scraping | SQL | BI Dashboards
Cover image for Problem: Financial data from stocks and
Problem: Financial data from stocks and crypto APIs was scattered, refreshed manually, and not ready for analytics or ML use. Solution: Built an Apache Airflow pipeline to collect, transform, validate, and load real-time financial data from multiple APIs into PostgreSQL, MongoDB, AWS RDS, and Qdrant. Tools: Apache Airflow, Python, PostgreSQL, MongoDB, AWS RDS, Qdrant, APIs Result: Automated sub-hourly data refresh, processed thousands of records daily, and delivered clean data for dashboards, analytics, and vector search.
0
67
Cover image for Problem: Batch data processing was not
Problem: Batch data processing was not suitable for real-time analytics and scalable cloud-based data ingestion. Solution: Created a real-time streaming pipeline using Kafka on AWS EC2, stored processed data in S3, cataloged it with AWS Glue, and queried it with Amazon Athena. Tools: Python, Apache Kafka, AWS EC2, Amazon S3, AWS Glue, Amazon Athena, Pandas Result: Built an end-to-end cloud data streaming workflow that supports real-time ingestion, storage, cataloging, and SQL-based analytics.
0
64
Cover image for Problem: Finance data was difficult to
Problem: Finance data was difficult to analyze across sales, profit, orders, discounts, countries, and customer segments. Solution: Created an interactive Power BI dashboard with KPI cards, sales trends, profit analysis, country performance, and segment-level insights. Tools: Power BI, Power Query, DAX, Excel, Data Modeling Result: Delivered a clear executive dashboard for tracking financial performance, identifying trends, and making faster business decisions.
0
53
Cover image for Problem: Users needed a faster way
Problem: Users needed a faster way to find the most relevant counselor based on specialization and available data. Solution: Built a recommendation workflow using PySpark for data processing and Redis for fast lookup and recommendation serving. Tools: Python, PySpark, Redis, Docker, Docker Compose Result: Created a containerized recommendation system that processes counselor data and returns relevant matches efficiently.
0
48
Apache Airflow(1)