Freelancers using Snowflake in Fort CollinsFreelancers using Snowflake in Fort Collins

Results that are similar to your search

Similar results

Cover image for Data Ingestion Migration: $170K a
Data Ingestion Migration: $170K a Year Saved (cover: work_6.png) # Problem: At a $2B SaaS unicorn, Fivetran’s new pricing model was about to raise ingestion costs from ~$13K to ~$23K a month, with no change in data volume. Syncs from Google Cloud SQL read replicas also broke whenever a replica lagged or was rebuilt, and each break needed manual re-syncs and chat with Fivetran team (due to table size). # What I did - Benchmarked ELT alternatives on cost, connector coverage and sync reliability - Migrated production connectors to a modern ELT platform in about 30 days, running old and new pipelines in parallel - Built an end-to-end validation framework comparing every internal and customer-facing data product before cutover: row counts, values and schemas - Found and resolved data type inconsistencies between the two tools (timestamps, decimals, JSON and nested fields) plus edge cases in deletes and soft-deletes - Preserved full history: migrated SCD Type 2 tables and past deletes from the existing Snowflake tables, so no history was lost and nothing had to be re-synced from source - Re-pointed downstream dbt sources only after the outputs matched - Tuned sync frequency and schemas to stop loading tables nobody used - Set up automatic recovery for replica issues, so syncs resume on their own instead of failing and waiting for a manual re-sync # Result: - Ingestion cost down from ~$13K to ~$9K a month (about 30% lower) ~$170K a year saved against the projected new pricing - Zero changes to internal or customer-facing data products: every output validated identical before cutover - Full history kept: SCD Type 2 history and past deletes carried over with no historical re-sync - More reliable pipelines: replica hiccups now recover automatically instead of breaking syncs and paging the team
1
12