Optimizing Snowflake MERGE Performance with Incremental LoadingOptimizing Snowflake MERGE Performance with Incremental Loading
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
A slow MERGE in Snowflake usually comes down to one of three things:
1. Scanning everything. If the source is the whole staging table, every run reprocesses history. Filter on a high-water mark (updated_at > last load) so you only merge new rows.
2. Duplicate keys in the source. When several source rows match one target row, the MERGE either errors or does extra work. Deduplicate first with QUALIFY ROW_NUMBER() = 1.
3. No pruning on the target. Snowflake can only skip micro-partitions when the ON or filter conditions touch a column that the target is clustered or naturally ordered by, such as a date. Adding a date-range condition can cut the scan dramatically, as long as late updates can't fall outside that window.
Fix these three and you'll often see a MERGE that took minutes finish in seconds.
#Snowflake #SQL #DataEngineering
Post image
Benjamin's avatar
TRUEFRAME logo
qualify row_number in one line is the dedupe trick people forget. pinning the date range tip too
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started