Use the uniq command of linux to remove duplicate lines from a file
This article mainly introduces the method of using linux's uniq command to remove duplicate lines in the file, which has a certain reference value, and friends who need it can refer to it. Let's take a look at it with me.
The full name of the uniq command is "unique", and the Chinese definition is "unique and unique". The purpose of this command is to remove consecutive duplicate lines from a text file without other lines of text. Get rid of the repetition, what is retained is unique, that is, unique.
We should note that the difference between it and sort is that sort removes duplicate lines as long as there are duplicate lines, while uniq duplicate lines must be continuous, or you can use it to ignore duplicate lines in the file.
Syntax format: uniq [parameters] [file]
Common parameters:
-c print the number of times each line is repeated in the text-d shows only duplicate records, each duplicate record appears only once-u only shows records that are not duplicated
Reference example
Delete consecutive duplicate lines in a continuous file:
[root@linuxcool ~] # cat testfile
Test 30
Test 30
Test 30
Hello 95
Hello 95
Hello 95
Hello 95
Linux 85
Linux 85
[root@linuxcool ~] # uniq testfile
Test 30
Hello 95
Linux 85
Print the number of times each line is repeated in the file:
[root@linuxcool ~] # uniq-c testfile
3 test 30
4 Hello 95
2 Linux 85
Only duplicate records are shown, and each record appears only once:
[root@linuxcool] # uniq-d testfile
Test 30
Hello 95
Linux 85
Only records that are not duplicated are displayed:
[root@linuxcool] # uniq-u testfile
[root@linuxcool] # above are the details of the method of using linux's uniq command to remove duplicate lines in the file. have you learned anything after reading it? If you want to know more about it, welcome to the industry information!