What is a Remove Duplicate Lines?
Removing duplicate lines is the first step in almost every list-cleaning job: a pasted column of tags, a subscriber export, a set of keywords, a list of URLs. The task sounds trivial until you decide what counts as a duplicate, and that decision changes the result more than the deduplication itself.
The three questions are whether case matters — is Apple the same as apple? — whether surrounding whitespace should be ignored, so that a trailing space does not create a second entry, and whether blank lines are content or noise. Each of those is a toggle here, because the right answer depends on the data. Tag lists usually want case-insensitive matching; a list of file paths does not, since paths are case-sensitive on Linux and macOS.
The tool also runs in the other direction. Instead of keeping one of each line, you can ask which lines were duplicated — the view you want when the goal is to find a problem rather than remove it — or keep the full list and annotate each line with how many times it appears.
How to remove duplicate lines from a list
- Paste the list into the left box, one entry per line.
- Choose a mode: Keep one of each removes repeats, Show only the duplicates lists just the lines that appeared more than once, and Keep everything, count duplicates leaves the list intact with an occurrence count.
- Decide the matching rules. Ignore capitalisation groups Apple with apple, and Ignore leading and trailing spaces stops a stray space from creating a second entry.
- Leave Remove blank lines on unless the empty lines matter — it strips them from the output and keeps the counts meaningful.
- Turn on Sort the output alphabetically if you want a tidy list rather than the original order, then copy or download the result.
- Check the Occurrences table to see which lines repeated and how many times. It is capped at the first 500 rows, but the counts above it cover the whole input.
Remove Duplicate Lines limits and rules
| Item | Limit |
|---|---|
| Comparison | Exact line matching — case and whitespace are only ignored if you say so |
| Preserved | The first occurrence of each line keeps its original position unless you sort |
| Order | Original order by default; alphabetical sorting is optional |
| Blank lines | Removed from the output by default; turn the option off to keep them |
| Case folding | Uses full Unicode case folding, so sizes and accented letters compare correctly |
| Occurrences table | First 500 distinct lines are listed; the summary counts cover everything |
| Whitespace inside a line | Always preserved — only leading and trailing spaces are trimmed, and only if enabled |
| Not covered | Fuzzy matching, near-duplicate detection, CSV column handling and sorting by frequency |
| Large lists | Handled in memory in the browser; a few hundred thousand lines is comfortable |
When to use it
- Cleaning a keyword list before an SEO analysis, so a repeated phrase does not inflate its apparent frequency.
- Deduplicating an email or contact export pasted straight out of a spreadsheet, where the same address often appears in several formats.
- Trimming a tag, category or technology list that has accumulated the same entry with different capitalisation over time.
- Finding what is duplicated rather than removing it — for example checking whether a sitemap has the same URL twice.
- Producing a sorted, unique list of values to use as a filter, a dropdown or a test fixture.
Is it private?
Everything runs inside your browser. Your text and files are never uploaded, never stored and never seen by anyone else — including me. You can use the tool with the network tab open and watch no request leave the page. Closing the tab erases the input.
Frequently asked questions
Only if Ignore capitalisation is on, which it is by default. That is the right choice for tags and keywords, where the difference is meaningless. Turn it off for file paths, IDs or anything case-sensitive.
The first one, in its original position. The output keeps the order of first appearance, so removing duplicates never shuffles the list — although whitespace is trimmed from the kept line if that option is on. Sorting is a separate, optional step.
The work is done in memory in your browser, and a few hundred thousand lines is comfortable on a modern machine. Beyond that the page may pause briefly while it counts. Nothing is uploaded, so there is no server limit to hit.
No. Every comparison happens in JavaScript in your tab, with no request made. That matters here more than usual, because the lists people deduplicate are often contact or keyword exports holding personal data.
Not directly. This tool works line by line, so it treats each whole row as one item. For CSV, convert or extract the column you want first — the CSV to JSON converter and a spreadsheet's own unique filter are both easier routes.
Related tools
Written and maintained by Md. Nur Hossain Repon.