Trucost: Environmental and ESG Risk Assessment

Trucost has been a leader in assessing risks related to climate change, natural resource constraints, and broader environmental, social, and governance (ESG) factors since 2000.

In this project, I developed a robust ETL loader written in C# to process and manage extensive environmental datasets provided by Trucost. The pipeline was designed to automate the ingestion, transformation, and storage of data into a centralized Data Management System (DMS) for downstream analytics and reporting.

Automated Ingestion: Built a process to fetch CSV files from a secure SFTP location at scheduled intervals using a custom batching strategy to ensure reliability and efficiency.

Data Transformation: Applied complex data transformations to clean and structure the raw data, including handling missing values, normalizing metrics, and ensuring compliance with ESG reporting standards.

Centralized Storage: Stored the transformed data in a Data Warehouse (DMS) for efficient querying and analysis by various stakeholders. The warehouse schema was optimized for analytics and integrated seamlessly with reporting tools.

Batching Strategy: Implemented a batching mechanism to process large datasets in manageable chunks, reducing memory overhead and ensuring fault tolerance.

Scalability and Maintainability: The loader was designed with modularity in mind, making it easy to extend for future data sources and transformation requirements.

C#, .NET Core for ETL pipeline development.

SFTP for secure file transfer.

SQL Server for the Data Management System.

Power BI for visualizing key metrics from the processed data.

Git and Azure DevOps for version control and CI/CD pipelines.

This project not only streamlined Trucost’s data handling but also provided critical ESG insights that empowered clients to make informed and sustainable investment decisions.