What are the ways to remove duplicates from Shell scripts?
This article introduces the relevant knowledge of "what are the ways to remove the repetition of Shell scripts". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!
Test file
The code is as follows:
[root@bogon ~] # cat > test
Jason
Jason
Jason
Fffff
Jason
Press Ctr + D to save
1. Sort-u
The code is as follows:
[root@bogon] # sort-u test
Fffff
Jason
2 、 uniq
The code is as follows:
[root@bogon ~] # uniq test
Jason
Fffff
Jason
It can be seen that uniq needs to be sorted before it can be duplicated.
3. The combined use of sort, pipeline and uniq
Remove duplicates and retain a duplicate copy of data
The code is as follows:
[root@bogon ~] # sort test | uniq
Fffff
Jason
Remove all duplicate rows (do not keep duplicate rows) and count the number of rows
The code is as follows:
[root@bogon ~] # sort test | uniq-uc
1 fffff
Find all duplicate rows and count the number of rows
The code is as follows:
[root@bogon ~] # sort test | uniq-dc
4 jason
To repeat according to a certain field
Change the original data to
The code is as follows:
[root@bogon ~] # more test
Jason 1
Jason 2
Jason 1
Fffff 2
Jason 3
[root@bogon] # sort-k 1 test 1-u
Fffff 2
Jason 1
Man sort looks at the usage of sort. The-k parameter is described as follows:
The code is as follows:
-k,-- key=POS1 [, POS2]
Start a key at POS1 (origin 1), end it at POS2 (default end of line)
This is the end of the content of "what are the ways to duplicate Shell scripts". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!
