Data Analytics Platform

In this pivotal role, I spearheaded the development and operationalisation of sophisticated ETL pipelines leveraging a cutting-edge tech stack comprising AWS Elastic Container Registry (ECR) for Docker container management, AWS Batch for efficient job scheduling and execution, PySpark for big data processing running inside Docker , AWS S3 for scalable storage, AWS Redshift for data warehousing, and Apache Airflow for orchestrating complex workflows also running inside Docker by consuming the data from Workday SOAP API and Salesforce REST API.

Responsibilities and Accomplishments:

Design and Implementation of ETL Pipelines: Led the architecture and creation of robust ETL pipelines to process vast datasets efficiently. Utilising Workday's SOAP and REST APIs, I extracted valuable data, transforming it through PySpark in Databricks, and loaded it into AWS Redshift for analytics and insights, all within Docker containers managed in AWS ECR and orchestrated by Apache Airflow.

Stakeholder Engagement: Acted as the primary liaison between technical teams and business stakeholders. I ensured clear communication and alignment of project objectives, timelines, and deliverables. My proactive engagement strategy led to the successful delivery of projects that met and exceeded stakeholder expectations.

Mentoring and Team Leadership: Dedicated to fostering a culture of continuous learning and improvement, I mentored junior team members and peers in best practices for cloud architecture, data engineering, and DevOps. My efforts significantly uplifted the team's technical capabilities and contributed to a collaborative and productive work environment.

Innovative Solutions from Scratch: Demonstrated exceptional problem-solving skills by designing and building innovative solutions from the ground up. My ability to conceptualise and implement effective data processing frameworks significantly enhanced our data analytics capabilities and supported strategic decision-making processes.

Through these responsibilities and accomplishments, I played a crucial role in enhancing our organisation's data infrastructure, enabling advanced analytics, and driving data-driven decision-making. My contributions not only streamlined operations but also fostered a culture of innovation.