How to delete duplicate lines in text in Linux

This article will explain in detail how to delete duplicate lines in the text in Linux. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

First, use sort+uniq. Note that uniq alone will not work.

Shell > sort-k2n file | uniq

Here I do a simple test. When the duplicate lines in file are no longer together, uniq removes all duplicate lines from the service. After sorting, all the same rows are adjacent, so unqi can delete duplicate rows normally.

Second, use the sort+awk command. Note that awk alone will not work either, for the same reason as above.

Shell > sort-k2n file | awk'{if ($0,000line) print;line=$0}'

Of course, if you redesign the code behind the pipe yourself, you may not need the sort command to sort it first.

Third, with the sort+sed command, you also need the sort command to sort first.

Shell > sort-K2n file | sed'$! n; / ^. ∗\ n\ 1 $/! P; D'

Finally, attach an example of the text that must be sorted with sort first, of course, the reason for this need to sort with sort is very simple, that is, the "locality" of the later algorithm design, the same lines may be scattered in different regions, once a new peer appears, then the previous records that have already appeared will be overwritten, after seeing this example, it is easy to understand.

Ffffffffffffffffff

Ffffffffffffffffff

Eeeeeeeeeeeeeeeeeeee

Fffffffffffffffffff

Eeeeeeeeeeeeeeeeeeee

Eeeeeeeeeeeeeeeeeeee

Gggggggggggggggggggg

Linux on how to delete duplicate lines in the text to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.