How to realize text de-duplication and count the number of repetitions in linux

This article mainly introduces how to achieve text de-duplication and count the number of repetitions in linux, which has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

The linux command line provides very powerful text processing capabilities, and the combination of linux commands can achieve many powerful functions. This article gives an example of how to use the Linux command line to de-duplicate text by line and sort by number of repeats. The main command used is sort,uniq. Among them, the main function of sort is sorting, and the main function of uniq is to remove the duplicates of adjacent text lines.

The test file test.txt for the demonstration is as follows:

Hello World. Apple and Nokia. Hello World. I wanna buy an Apple device. The Iphone of Apple company. Hello World. The Iphone of Apple company. My name is Friendfish. Hello World. Apple and Nokia.

The command is as follows:

$sort test.txt | uniq-c | sort-rn 4 Hello World. 2 The Iphone of Apple company. 2 Apple and Nokia. 1 My name is Friendfish.

The uniq command plus the-c option allows you to count the number of repetitions, and sort-n recognizes the number at the beginning of each line and sorts the lines of text by their size. The default is to sort in ascending order, and add the-r option (sort-rn) if you want to sort in descending order.

Thank you for reading this article carefully. I hope the article "how to remove duplicates in linux and count the number of repetitions" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!