How to delete duplicate lines in Linux

This article introduces you how to delete duplicate lines in Linux, the content is very detailed, interested friends can refer to, hope to be helpful to you.

First, use sort+uniq. Note that uniq alone will not work.

Sort-n test.txt | uniq

Second, use the sort+awk command. Note that awk alone will not work either, for the same reason as above.

Sort-n $file | awk'{if ($0,000lines) print; line=$0}'

Third, with the sort+sed command, you also need the sort command to sort first.

Sort-n $file | sed'$! n; / ^ .∗\ n\ 1 $/! P; D'

Shell script

#! / bin/shfile='test.txt'sort-n $file | uniqsort-n $file | awk'{if ($0Secretline) print; line=$0} 'sort-n $file | sed' $! n; / ^\ (. *\)\ n\ 1 $/! P; D'

Test file:

Yanggang@barry$ cat test.txt

Aaa

Bbbbb

Ccccc

one hundred and twenty three

Aaaaa

one hundred and twenty three

Bbb

Aaa

Execution result:

Yanggang@barry$. / diffRow.sh

Aaa

Aaaaa

Bbb

Bbbbb

Ccccc

one hundred and twenty three

On how to delete duplicate lines in Linux to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.