Data Engineering Pipeline Development

Grant Fraley

(0)

Starting at

100

/hr

About this service

Summary

I offer Data Engineering services that focus on automating data pipelines, building data warehouses, and designing data lakes. This consulting service helps clients to build efficient, scalable, and reliable data processing systems that can handle large volumes of data.

Services include:

Consultation on choosing the appropriate technology stack, data storage systems, and processing frameworks to meet the your business objectives.

Designing and building automated data pipelines that can ingest, process, and transform data from a variety of sources into structured data models.

Building data warehouses that provide a centralized repository for storing structured data, and designing data lakes that provide a centralized repository for storing raw, unstructured data.

Configuring and optimizing the data processing and storage environment, including setting up data processing frameworks, data storage systems, and cloud services such as AWS or GCP.

Ongoing maintenance and support to ensure the data processing systems are running smoothly and continue to meet the client's business objectives.

Overall, my goal is to help you build efficient, scalable, and reliable data processing systems that can provide meaningful insights for decision-making. I can also support business intelligence and data visualization efforts using Looker, Tableau, or other BI tools.

What's included

Automated Data Pipelines
I will provide code that sources data from APIs or external systems into a data lake (e.g. AWS S3 or Google Cloud Storage), loads relevant data into the warehouse (e.g. BigQuery, Redshift, or Snowflake) and performs transformations and analytics to create production-ready tables in the warehouse. Pipelines may be written in SQL and Python and automated using workflow solutions such as dbt, Airflow, and AWS Step Functions. These automated workflows may be triggered on a schedule or upon events such as new data arriving.
Data Warehouse
Based on a review of business requirements, I'll use data engineering best practices to design a data model that can help answer a variety of questions and support ad-hoc analytic queries. The infrastructure and schemas will be created through code so that it is repeatable and maintainable, and I can work within existing database systems and cloud accounts or create new secure environments.
Data Lake
Using my experience designing petabyte scale data lakes, I can work with you to design a cloud storage layout and file formats to support ease of testing and loading data into your data warehouse.

Skills and tools

Cloud Infrastructure Architect

Data Engineer

Software Engineer