Remove Duplicate Lines

The Remove Duplicate Lines tool strips repeated lines from any list of text, leaving only unique entries. It sounds simple but turns up constantly in real work: an email list accidentally concatenated twice, a log file with the same error repeating hundreds of times, a scraped URL list that pulled from multiple pages, a CSV with duplicate rows from a bad export, a list of IP addresses that blocks keep re-adding. In every case you want one clean list with each value appearing exactly once.

This tool goes beyond naive deduplication. Choose between case-sensitive matching (Admin and admin treated as different) and case-insensitive (treated as same). Choose whether to preserve the original input order (keeping the first occurrence of each unique value) or to sort the output alphabetically. Trim whitespace before matching so that \"abc\" and \"abc \" are treated as the same line. Ignore blank lines entirely or keep them. Count both unique and duplicate entries so you know how many duplicates were removed. Everything runs in your browser and handles lists up to hundreds of thousands of lines without breaking a sweat.

Deduplication is conceptually trivial but the implementation details matter for correctness and performance.

The simplest correct approach is a Set: iterate through lines, add each to a Set (which ignores duplicates), then output the Set. This preserves first-occurrence ordering in modern JavaScript (Set maintains insertion order) and runs in O(n) time with O(n) space. This tool uses that approach.

Case-insensitive deduplication requires normalizing the key while preserving the original value. Internally the tool maintains a Set of lowercased lines for comparison and a separate array of original lines for output; if the lowercased form is already in the Set, the original line is skipped. This way \"Admin\" stays \"Admin\" in output even though it was matched against \"admin\".

Whitespace trimming is applied to the comparison key only (by default) — the output can preserve the original whitespace or use the trimmed value, depending on user preference. The first variant keeps data fidelity; the second normalizes visually.

Sort-then-dedupe is different from dedupe-then-sort. Sort-then-dedupe is what Unix sort -u does: sort all lines, then collapse adjacent duplicates. Dedupe-then-sort preserves first occurrence then sorts the unique set. The tool offers both modes because different workflows want different behaviors (keeping a specific first-occurrence entry vs pure canonical output).

For very large lists (multi-million lines), memory usage is a concern because the entire list plus the Set live in browser RAM. Modern browsers handle million-line lists; beyond that the tool starts to struggle — split into chunks if you need to dedupe truly huge files.

Natural-order handling: if you sort before dedupe, lexicographic order places \"item-10\" before \"item-2\" which may surprise users. The sort-and-dedupe workflow offers a natural-sort mode that sorts numbers embedded in strings correctly.

Performance: a million-line dedupe takes about 100-500 ms in modern browsers depending on line length. Memory peaks at roughly 2-3x the input size because the Set stores string references alongside the array.

Compared to spreadsheet Remove Duplicates, this tool is faster for simple line deduplication without opening a spreadsheet. For column-based or multi-column dedup, a spreadsheet is the right choice.

Compared to Unix uniq (which only removes adjacent duplicates) and sort -u (which sorts first), this tool offers both behaviors plus case-insensitive matching and whitespace handling in one interface. CLI remains ideal for scripting; this tool is faster for interactive work.

Compared to the Sort Lines tool in this suite, deduplication happens with or without sorting. If you need both, either tool can do it — use Sort Lines when sort is the primary intent, this one when dedup is the primary intent.