Skip to content

Text Tools

Remove duplicate lines

Strip repeated lines and keep the first occurrence of each, in the order they appeared.

  • Free, no signup
  • No upload — runs on your device
  • No watermark
  • Unlimited use

Remove Duplicate Lines

Processed locally

Sorting to deduplicate destroys information you may need

The classic Unix one-liner for deduplication is sort | uniq, and it works because uniq only removes adjacent duplicates. Sorting first brings them together. It also destroys the original order, permanently.

That matters more often than people expect. A list of URLs in crawl order, a list of errors in the sequence they occurred, a list of names in the order they signed up — the order is data, and sorting throws it away to work around a limitation of a 1970s utility. Deduplicating with a hash set, as this tool does, removes repeats while keeping every remaining line exactly where it was.

Case insensitivity is the other thing worth turning on. Email lists, URL lists and tag lists routinely contain the same value in three capitalisations, and an exact-match dedupe keeps all three.

How it works

  1. Paste your list
  2. Choose whether to ignore case, spaces and blank lines
  3. Copy the deduplicated result
Why nothing uploads. Every operation on this page happens inside your browser using JavaScript and WebAssembly. Your file is read into memory, processed, and offered back as a download. It is never transmitted. Disconnect from the internet after this page loads and the tool keeps working.

Frequently asked questions

Which occurrence of a duplicate line is kept?
The first. Every subsequent repetition is removed, and the surviving lines stay in their original positions.
Why not just sort the list and use uniq?
Because sorting destroys the original order, which is often meaningful — a crawl sequence, a signup order, an error log. Hash-based deduplication removes repeats without reordering anything.
Does it ignore leading and trailing spaces?
If you tick the trim option, yes — and the surviving lines are also trimmed in the output. Two lines differing only by a trailing space are almost always the same line.
What happens to blank lines?
By default they are removed, since a list of items rarely wants them. Tick the option to keep the first blank line and remove the rest.
How many lines can it handle?
Hundreds of thousands. Matching uses a hash set, so it scales linearly with the number of lines.

Related tools