Rebuilt ELT workflows with PySpark incremental loading on Databricks and Snowpipe Streaming on Snowflake, replacing full reloads. Ingestion time dropped from 2 hours to 15 minutes. Also built Delta Lake CDC ingestion for a Phase 3 clinical trial (69K+ records/day) with sync between PHI and non-PHI environments.
0 likes · 0 views · Nov 10, 2026
Replaced full reloads with PySpark incremental loading on Databricks and Snowpipe Streaming, cutting ingestion from 2 hours to 15 minutes.
