A ETL pipeline that extract Flight data from Tribhuvan International Airport.

Data is extracted from tia

Load to s3

Transform the data

Load the data in RDS

Set the values inside .env and terraform/variables.tf

Now you can see ec2 ip and rds endpoint in terminal

terraform output private_key and copy the rsa key into tia_key.pem in project root directory

Connect to the ec2 instance using ssh -i tia_key.pem {user_name}@{ec2_ip}

sudo snap install docker to install docker in ec2

Username and password used by airflow and local postgres server are hardcoded in docker-compose.yaml

Copy all files and folders to ec2 except terraform

Now docker container will run and data will be extracted every 30 minutes and stored in RDS.

To remove all the services in aws terrafrom destroy

I used docker and ec2 in this pipeline to learn about them. One can simply run this pipeline by removing docker and airflow in lambda. Using lambda will be quick and cost effictive.