Shell solves the problem of Redis Slow Log weight removal

The old employer monitors the slow query log of Redis, uses Crontab to sample the slow query log at regular intervals, and then writes a text file.

So there are many duplicate slow query logs in the text file.

How to deduplicate the whole paragraph of repeated slow query logs?

The way to use the Shell command is the simplest and most straightforward.

Awk'/ ^ 1\) / {print sscatter = ""}; {s "" $0)}; END {print s} 'b.txt | sort | uniq | sed's / /\ n / g' | awk'/ ^ 1\)\ (integer\) / {rc = rc+1;rowdata=rc ")\ t" $1 "" $2 "$3"\ n\ t "4" $5 "$6" $7 "\ n\ t" $8 "$9" $10 " Print rowdata}; / ^ / {print "\ t" $0} END {print "No duplicate logs Total:" rc}'

The core is to merge the entire paragraph of redis logs into one line, remove the duplicates, and restore the format.

Here are the attachments for testing

RedisSlowLog.txt

With Wang Gong's description and explanation of the demand for this problem.

Https://blog.csdn.net/wzy0623/article/details/83652539?tdsourcetag=s_pcqq_aiomsg