PySpark & Databricks migration for a European grid operator

A European electricity grid operator ran the ingestion pipelines of its grid modelling system as plain Python scripts. As data volumes grew, they became slow and hard to operate. I migrated them to PySpark on Databricks, with Delta Lake as the storage layer, while keeping the output identical for every downstream team.

Rewrote the ingestion logic in PySpark, with sources coming from Azure Event Hubs and bulk file transfers

Matched the legacy results exactly by aligning Delta merge keys with the original logic

Handled nested data safely, keeping parallel arrays aligned and deduplicating rows after explosion

Orchestrated the pipelines with Azure Data Factory

Pipelines that scale with data volume and are easier to monitor and maintain, without disrupting the teams that rely on the data.

PySpark, Databricks, Delta Lake, Azure Data Factory, Azure Event Hubs