Data Ingestion Migration Cuts Costs and Improves ReliabilityData Ingestion Migration Cuts Costs and Improves Reliability
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Data Ingestion Migration: $170K a Year Saved (cover: work_6.png)
# Problem:
At a $2B SaaS unicorn, Fivetran’s new pricing model was about to raise ingestion costs from ~$13K to ~$23K a month, with no change in data volume. Syncs from Google Cloud SQL read replicas also broke whenever a replica lagged or was rebuilt, and each break needed manual re-syncs and chat with Fivetran team (due to table size).
# What I did
- Benchmarked ELT alternatives on cost, connector coverage and sync reliability
- Migrated production connectors to a modern ELT platform in about 30 days, running old and new pipelines in parallel
- Built an end-to-end validation framework comparing every internal and customer-facing data product before cutover: row counts, values and schemas
- Found and resolved data type inconsistencies between the two tools (timestamps, decimals, JSON and nested fields) plus edge cases in deletes and soft-deletes
- Preserved full history: migrated SCD Type 2 tables and past deletes from the existing Snowflake tables, so no history was lost and nothing had to be re-synced from source
- Re-pointed downstream dbt sources only after the outputs matched
- Tuned sync frequency and schemas to stop loading tables nobody used
- Set up automatic recovery for replica issues, so syncs resume on their own instead of failing and waiting for a manual re-sync
# Result:
- Ingestion cost down from ~$13K to ~$9K a month (about 30% lower)
~$170K a year saved against the projected new pricing
- Zero changes to internal or customer-facing data products: every output validated identical before cutover
- Full history kept: SCD Type 2 history and past deletes carried over with no historical re-sync
- More reliable pipelines: replica hiccups now recover automatically instead of breaking syncs and paging the team
Post image
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started