CSV Duplicate Removal & Quality Check | Demo Project

Synthetic order-line data processed with an AI-assisted workflow and independent quality checks. This is a demonstration, not a client engagement.

The input is a seven-row synthetic CSV order export. The goal was to remove repeated order lines without accidentally merging different lines from the same order. The acceptance rule was explicit: compare the exact pair order_id + line_id, keep the first occurrence of each pair and preserve the original order of retained records.

Deduplication starts with a business rule, not a guess. An order ID alone is insufficient because one order can contain several distinct lines. In this example the composite key identifies a repeated line, and the retention rule determines which occurrence survives. Leading-zero line identifiers stay strings throughout processing.

The declared transformation ran in an isolated Docker worker. No name matching, approximate similarity or inferred merge was used. The original CSV, specification, result and change report were retained so the transformation can be inspected. The separate validator independently derived duplicate groups and row lineage from the source.

Seven input rows became five retained rows. Two repeated key occurrences were removed. Different lines belonging to the same order remained separate. All retained values, leading-zero line IDs and relative row order were preserved. The comparison image shows every source row and every output row so the removal decision can be checked directly.

All nine validation criteria passed. The report checks schema, row count, deduplication, value preservation, types, encoding, change accounting, formula safety and the output hash. The validator compares actual output values with the independently expected retained rows; a success message from the worker alone is not sufficient.

An additional negative check deliberately altered a retained identifier. That output was rejected. This demonstrates detection of a damaged result for the tested example, rather than a claim that every possible dataset or matching rule has been validated.

The package contains the original CSV, deduplicated CSV, exact key and retention rules, a separate QA report and the full visual comparison. Source and output are linked to the report by SHA-256 hashes.

This synthetic demonstration establishes a concrete technical result, not client acceptance, revenue or time savings. For a real export, duplicate keys and the retention policy must be agreed before removing records. Fuzzy matching, inferred corrections, live database changes and production imports are separate scopes requiring their own checks.