Remove Duplicate Lines Free — Dedupe

How to remove duplicate lines

Paste your list. One item per line, or drop a text file.
Set the options. Ignore case, trim whitespace, sort, and keep first or last occurrence.
Copy the result. With a count of how many duplicates were removed.

About Remove Duplicate Lines

Lists accumulate duplicates. Email addresses merged from two exports, keywords collected from three research tools, URLs pasted from a crawl, product codes assembled from different systems: anywhere data comes from more than one source, the same entry shows up more than once. Deduplicating by hand is tedious and unreliable past a couple of dozen lines.

The options matter more than the core operation. Ignore case is the one people most often need and most often forget: "[email protected]" and "[email protected]" are the same address to a mail server but different strings to a computer, so without that setting you keep both. Trim whitespace catches the equally common case where one export added a trailing space and another did not, making two identical-looking lines technically distinct.

Keep first or keep last decides which copy survives when order carries meaning. If your list is chronological and later entries are more current, keeping the last occurrence preserves the newest version. If the list is in priority order, keeping the first preserves the ranking. When order is arbitrary it makes no difference, but it is worth a moment's thought when it is not.

Sorting is offered alongside because the two operations almost always travel together. A deduplicated list is usually about to be reviewed or imported, and both are easier alphabetically. The tool reports how many duplicates were removed, which is often the more interesting number: a large count usually means your source data has a problem worth understanding rather than just cleaning.

Remove Duplicate Lines FAQ

Does it preserve the original order?

Yes, unless you turn on sorting. Deduplication keeps the first occurrence by default and leaves everything else in place.

Can it ignore capitalisation?

Yes. Turn on ignore case so that "[email protected]" and "[email protected]" are treated as the same entry.

What about lines that differ only by a trailing space?

Turn on trim whitespace and they will be treated as duplicates. This is the second most common cause of missed matches after capitalisation.

Can I keep the last occurrence instead of the first?

Yes. That is the right choice when your list is chronological and later entries are more current.

Is there a size limit?

No fixed limit. Very large lists may slow the browser down, since the work happens on your device.

Is my list uploaded?

No. Everything runs in this browser tab, which matters if the list is email addresses or customer data.

Remove Duplicate Lines