IIX Lab / Guides

Remove duplicate rows before they break your CSV import

Every store importer rejects a whole file for one duplicate SKU, and almost every duplicate is a whitespace or case difference that looks identical on screen. This is the order that actually clears them.

Last verified 2026-11-01 · 4 min read

Do this, in this order

Trim first, dedupe second. A leading space makes "00127" and " 00127" two different keys. Trim before comparing, never after.
Decide whether case is a duplicate. SKUs are case-sensitive; names and emails usually are not. Lowercase only the columns where case is noise.
Dedupe on the identifier column, not the whole row. Two rows with the same SKU but different prices are a real error, not a duplicate. Deduping on the SKU surfaces it; deduping on the whole row hides it.
Re-check the count against the source. Compare row counts before and after. A drop larger than your expected duplicates means the key was too loose.

Where people get stuck. Excel silently converts 00127 to 127 on paste. If you dedupe after that, the SKU no longer matches your warehouse and the file imports as a new product.

Where duplicates actually come from

Symptom	Cause	Fix
Same SKU twice, one with a space	Trailing whitespace from a copy-paste	Trim every cell before comparing
Importer says duplicate handle	Two rows map to the same product slug	Dedupe on the handle column
Product imported twice	Leading zero lost, so 00127 became 127	Store identifiers as text before pasting
Nothing deduped at all	Case difference (SKU vs sku)	Lowercase the key only, keep the original for output

Questions we keep getting

Can I dedupe inside Shopify?

Not on import. The importer rejects the file before it touches your catalogue, so the cleanup has to happen on the file you upload.

Does it matter which duplicate I keep?

Yes for anything with a price or stock count. Keep the newest row by timestamp, not the first one you find.

What about invisible characters?

Non-breaking spaces (U+00A0) survive normal trimming in some tools. If a row still looks like a duplicate, check for 0xC2 0xA0 in the raw bytes.

Official references

Official CSV format reference: Shopify product CSV documentation