Data Engineer, Horizon Media
Ankit B
Execute OAuth and REST API calls using Postman API Platform to acquire client data in JSON format.
Develop Python scripts for data extraction and categorization into S3 buckets for incremental and full load using Airflow, AWS infrastructure and Big Data (Hive, Presto, Spark).
Manage codebase on GitHub, ensuring proper version control and documentation in Confluence.
Establish Fivetran connections to facilitate ETL processes from S3 to Snowflake Data Warehouse.
Collaborate on data transformations within Snowflake, optimizing query performance for analytics.
Utilize DBT for efficient data modeling and transformations in both Dev and Prod environments.
Tools: SQL Server, Airflow, DBT, AWS (S3, Lambda, Glue, Step Functions, Redshift), Fivetran, Snowflake, GitHub, Confluence, Postman, Python (JSON, PyTest, PyCharm, Pandas), Jira
Like this project
Posted Feb 12, 2024
Develop data pipelines to ingest / automate WebAPI data from USA based retail clients and populate data warehouses, data marts for various business teams.
